. t:
61...?

.5: ._

I. .41
.u‘dxvxh.

 

:n‘
.. 9.411;...

 

 

 

 

 

 

 

 

 

Len-fit:-

m

This is to certify that the
dissertation entitled

MAKING SENSE OF THE LINKS: FROM GOVERNMENT
POLICY TO STUDENT ACHIEVEMENT

presented by
MARJORIE CAROL RHODES WALLACE

has been accepted towards fulfillment
of the requirements for the

PhD. degree in Curriculum, Teaching, and
Edu tion Policy

 

 

 

W {a

V Major Professor‘s Signature
I I / y/és"

Date

MSU is an Affirmative Action/Equal Opportunity institution

t__‘ if i _ _—~ _—

LlBFiAFgYtate
Michigan .
Universfly

 

 

 

 

 

PLACE IN RETURN BOX to remove this checkout from your record.
TO AVOID FINES return on or before date due.
MAY BE RECALLED with earlier due date if requested.

 

DATE DUE DATE DUE DATE DUE

 

 

 

 

 

 

 

 

 

 

 

 

 

 

2/05 p:/C|RC/DateDue.indd-p.1

 

 

MAKING SENSE OF THE LINKS:
FROM GOVERNMENT POLICY TO STUDENT ACHIEVEMENT

By

Marjorie Carol Rhodes Wallace

A DISSERTATION
Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of

DOCTOR OF PHILOSOPHY
Department of Teacher Education

2005

ABSTRACT

MAKING SENSE OF THE LINKS:
FROM GOVERNMENT POLICY TO STUDENT ACHIEVEMENT

By

Marjorie Carol Rhodes Wallace

Using the Beginning Teacher Preparation Study (BTPS) in Connecticut and
Tennessee and the National Assessment of Educational Progress in Mathematics and
Reading 1999-2000 (NAEP), this study provides evidence on the strength of relationships
and relative contributions of four important elements in the education system to student
achievement in mathematics and reading: teacher preparation, teacher characteristics,
professional development, and teacher practice. Results from structural equation
modeling (SEM) indicate that teacher practices, teacher characteristics, and professional
development across teachers are generally linked to teacher practice and student
achievement in consistent ways. They have moderate to large direct effects on teacher
practice and small but significant indirect effects on student achievement in mathematics
and reading (5 = .05). Findings vary based on curriculum and policy context.

Hierarchical linear modeling (HLM) results indicate that teacher characteristics,
professional development, and teacher practices can differentially affect student
achievement by ethnicity and gender. Differential effects on gain scores (estimated at the
state level) portray effects on learning while effects on achievement status confound
teacher effects with prior differences in achievement at the national level. Black mean

achievement varies across teachers in mathematics (Connecticut, NAEP 1996) and

reading (Connecticut, NAEP 1998). The mean for Black (Connecticut reading) varies
across teachers but teacher variables do not account for variation. Female mean
mathematics achievement varies across teachers in two data sets. Hispanic mean
mathematics (NAEP 2000) and Black female mean mathematics achievement (NAEP
1996) vary across teachers in one data set each. Teacher variables exhibiting differential
effects (increasing achievement for one group while simultaneously decreasing
achievement for another) are teacher drill and lecture activities, sensemaking activities,
knowledge, preparation, reform orientation, and professional development for
mathematics. For reading, the list includes traditional activities, computer activities,
reading preparation, assessment preparation, and professional development. The
differential effects sometimes split by ethnicity and sometimes by gender.

In the three-level HLM for Connecticut, effects of teacher preparation programs
do not vary significantly across teachers when student gains are the outcome. Using
SEM, teacher preparation programs as a whole have effects on student achievement but

those effects cannot be attributed to individual teacher preparation programs using HLM.

Copyright by
MARJORIE CAROL RHODES WALLACE

2005

To my family and friends, especially my husband, Charles E. Wallace, and my

mother, Marian J. Rhodes (1922-1995). Who knew?

ACKNOWLEDGEMENTS

This research was supported by a grant from the American Educational Research
Association which receives funds for its “AERA Grants Program” from the National
Center for Education Statistics and the Office of Education Sciences (US Department of
Education), and the National Science Foundation under NSF Grant #REC-99805 73.
Opinions reflect those of the author and do not necessarily reflect those of the granting
agencies.

A great many individuals deserve thanks for their input and support of this work.
The first acknowledgment goes to my family and friends who continued to ask, “How is
the dissertation coming?” even when they knew they would not want to hear the answer.
Over the years, my children patiently waited when I was late to pick them up, late for
school events, and late to say good night. Now that Casey and Charlie are in graduate
school and Lesley working toward her bachelors, they understand the level of
commitment involved as well as the rewards. I couldn't be more proud and I thank them
for their love, support, and tolerance. Chuck, my husband, kept me grounded and
wouldn't let me give up golf even when the dissertation preoccupied nearly every waking
and sleeping minute. Without him, I would have imploded long ago.

I had the best committee ever assembled: Robert Floden, Mark Reckase, Kenneth
Frank, Mary Kennedy, and Kimberly Maier. There is no way to really thank them for
their vast knowledge, encouragement, and support. Special kudos to Bob and Kim who
modeled very important lessons in mentoring and patience. Additional thanks go to Gail

Nutter and Lynne F rechen who made it possible for me to do this.

vi

Back in the past, thanks to Dr. Stephen Raudenbush, Dr. Alex Sedlacek, and my
colleagues from LAMMP who surprised me with a new way of seeing the world, related
and multilevel.

And last but far from least, without dissertation support from American
Educational Research Association, I would not have had time to start or finish the
dissertation. It was a surprise and a great pleasure to get to know Dr. Jeremy Kilpatrick,
my AERA board mentor. Jeanie Murdock (I have to wonder how she manages all the
fledgling researchers year after year) took some of the sting out of having to admit that
my revision aspirations outpaced the time available as the grant period stretched and
stretched again. Now that I am finished, I hope this and my future work lives up to the

honor you all have bestowed on me.

vii

TABLE OF CONTENTS

LIST OF TABLES ...................................................... xi
LIST OF FIGURES ................................................... xvii
INTRODUCTION ....................................................... 1
1. PREVIOUS RESEARCH ............................................... 7
Accountability and Assessment. ...................................... 7
Teacher Preparation ................................................ 8
Teacher Characteristics ............................................. 9
Teacher Professional Development. .................................. 10
Teacher Practice .................................................. 10
Ongoing BTPS studies ............................................. 11

. EDUCATION POLICY ENVIRONMENTS ............................... 13
State Level ...................................................... 13
Connecticut ............................................... 15

Tennessee ................................................. 17

National Level ................................................... 18

. CONCEPTUAL MODEL .............................................. 21
. DATA SETS ........................................................ 24
State Data ....................................................... 24
Connecticut Teacher and Student Samples ....................... 25

Tennessee Teacher and Student Samples ......................... 26

National Data .................................................... 27
NAEP Teacher and Student samples ............................ 27

Scale comparisons for BTPS and NAEP ......................... 28

. LATENT VARIABLES AND THEIR INDICATORS ....................... 33
Teacher Preparation ............................................... 33
Teacher Characteristics ............................................ 33
Professional Development .......................................... 37
Teacher Practices ................................................. 38
Student Achievement .............................................. 39

. METHODS & PROCEDURES ......................................... 41
. STRUCTURAL EQUATION MODELING RESULTS ...................... 47
Connecticut Base Model for Mathematics .............................. 48

viii

Connecticut Base Model for Reading ................................. 53

Tennessee Mathematics and Reading Results ........................... 57
NAEP Mathematics and Reading Models .............................. 58
Comparisons Across All Data Sets ................................... 68
8. RESULTS FROM THE HLM ANALYSIS OF
CONNECTICUT 4‘h-8‘“ GRADE MATHEMATICS GAINS .............. 74
1. Setting the Stage: Variation in Mathematics Achievement Gains by Student
Demographic Groups and Grade Level .......................... 76
11. Effects of Teacher Practices, Teacher Characteristics, and Professional
Development on Student Achievement Gains ..................... 88
III. IHE Teacher Preparation, Teachers, and Student Achievement ......... 115
Results Summary ................................................ 127
9. RESULTS FROM THE HLM ANALYSIS
OF CONNECTICUT 4‘h-8th READING GAINS ....................... 133
1. Setting the Stage: Variation in Reading Achievement Gains by Student
Demographic Groups ....................................... 135

II. Differential Effects of Teacher Practices, Teacher Characteristics,
and Professional Development on Student Reading

Achievement Gains ........................................ 145
HI. IHE Teacher Preparation, Teachers, and Student Achievement ......... 152
Results Summary ................................................ 158
10. RESULTS OF THE HLM ANALYSIS
FOR 1996 NAEP 4T“ GRADE MATHEMATICS ...................... 161
1. Setting the Stage: Variation in Mathematics Achievement Gains by Student
Demographic Groups and SES ............................... 163
II. Differential Effects of Teacher Characteristics and Teacher Practices on
Student Achievement ....................................... 174

11. RESULTS OF THE HLM ANALYSIS

FOR 2000 NAEP 4TH GRADE MATHEMATICS ...................... 196

1. Setting the Stage: Variation in Mathematics Achievement by Student
Demographic Groups and SES ............................... 198

II. Differential Effects of Teacher Characteristics and Teacher Practices on
Student Achievement ....................................... 209

12. RESULTS OF THE HLM ANALYSIS

FOR 1998 NAEP 4TH GRADE READING ........................... 218

1. Setting the Stage: Variation in Reading Achievement Gains by Student
Demographic Groups and SES ............................... 220

II. Differential Effects of Teacher Characteristics and Teacher Practices on
Student Achievement ....................................... 231

ix

13. RESULTS OF THE HLM ANALYSIS

FOR 2000 NAEP 4TH GRADE READING ................................. 244
1. Setting the Stage: Variation in NAEP 2000 Reading Achievement Status by
Student Demographic Groups and SES ......................... 246
II. Differential Effects of Teacher Characteristics and Teacher Practices on
Student Achievement ....................................... 257

14. COMPARISON OF HLM RESULTS

FOR MATHEMATICS AND READING ACROSS DATA SETS ......... 264

15. DISCUSSION AND IMPLICATIONS .................................. 277
The Conceptual Model ............................................ 277

SEM Analysis ............................................ 279

HLM Analysis ............................................ 280

Gains Versus Achievement Status ............................. 281

State Versus National Data Sets .............................. 281

Relationships Across Models ....................................... 282
Contexts ....................................................... 283
Differential Effects ............................................... 285
Implications for the Present and Future ............................... 289
APPENDICES ........................................................ 293
APPENDD( A: Correlation Tables .................................. 294
APPENDD( B: Miscellaneous Tables ................................ 305
REFERENCES ....................................................... 307

LIST OF TABLES

4. DATA SETS ........................................................ 24
Table 4.1 Sample Comparisons by Data Set and Analysis ................. 24
Table 4.2 BTPS Scales and Properties ................................. 29
Table 4.3 NAEP 1996 Mathematics Scales ............................. 30
Table 4.4 NAEP 2000 Mathematics Scales ............................. 30
Table 4.5 1998 NAEP 4th Grade Reading Scales ........................ 31
Table 4.7 2000 NAEP 4th Grade Reading Scales ........................ 31
6. METHODS & PROCEDURES ......................................... 41
Table 6.1 Analysis Matrix .......................................... 43
Table 6.2 Variable Availability Across Data Sets ....................... 44
7. STRUCTURAL EQUATION MODELING RESULTS ...................... 47
Table 7.1 Connecticut Standardized Total Effects for Mathematics ......... 50
Table 7.2. Connecticut Standardized Total Effects for Reading ............. 55
Table 7.3 Comparison of r2 for Mathematics and Reading Constructs
Across Data Sets ........................................... 63
Table 7.4. Standardized Total Effects for Mathematics Across Data Sets ..... 64
Table 7.5. Standardized Total Effects for Reading Across Data Sets ......... 66
8. RESULTS FROM THE HLM ANALYSIS OF
CONNECTICUT 4th-8th GRADE MATHEMATICS GAINS .............. 74
Table 8.1 Description of Variables Used in the Connecticut Mathematics
Analyses .................................................. 75
Table 8.2 Connecticut’s Average Mathematics Gain 1999-2000 ............ 77
Table 8.3 Regression Coefficients for Connecticut Mathematics Gain 1999-2000
by Ethnicity ............................................... 79
Table 8.4 Predicted Mathematics Gains by Student Demographic Group ..... 80
Table 8.5 Regression Coefficients for Connecticut 4th Grade Mathematics Gain
1999-2000 by Gender ........................................ 81
Table 8.6 Regression Coefficients for Connecticut Mathematics Gain
1999-2000 by Gender and Ethnicity ............................ 82
Table 8.7 Predicted Mathematics Gains by Student Demographic Group with
Differences by Gender ....................................... 84
Table 8.8 Regression Coefficients for Connecticut Mathematics Gain
1999-2000 by Gender, Ethnicity, Grade ......................... 86
Table 8.9 Predicted Mathematics Gains by Student Demographic Group and
Grade .................................................... 87
Table 8.10 Regression Coefficients for Connecticut Mathematics Gain
1999-2000 by Gender ........................................ 91
Table 8.11 Regression Coefficients for Average Gains by Gender Predicted by
Average District SES ........................................ 93

xi

Table 8.12 Variance Decomposition for Gender by District SES ............ 94
Table 8.13 Variance Decomposition for Gender Controlling SES -

Teacher Practices ........................................... 97
Table 8.14 Variance Decomposition by Gender for Teacher Characteristics
Controlling for SES ......................................... 98
Table 8.15 Variance Decomposition for Gains by Gender Controlling for SES -
Teacher Professional Development ............................ 100
Table 8.16 Regression Coefficients for Gains by Ethnicity 1999-2000 ...... 104
Table 8.17 Regression Coefficients for Black/White Gains by Average District
SES ..................................................... 105
Table 8.18 Variance Decomposition for Black/White Gains by SES ........ 106
Table 8.19 Variance Decomposition for White/Black Controlling SES -
Teacher Practices .......................................... 109
Table 8.20 Variance Decomposition for Black/White Controlling for SES -
Teacher Characteristics ..................................... 110
Table 8.21 Variance Decomposition for Black/White Controlling for SES -
Teacher Professional Development ............................ 1 12
Table 8.22 Description of Teacher Preparation Variables ................ 117
Table 8.23 Connecticut’s Average Mathematics Knowledge by Teacher
Preparation Institutions ..................................... l 18
Table 8.24 Connecticut’s Average Mentoring Frequency by Teacher Preparation
Institutions ............................................... 119
Table 8.25 Average Mathematics Knowledge Predicted by Program
Characteristics ............................................ 1 20
Table 8.26 Average Mentoring Frequency Predicted by Program
Characteristics ............................................ 120
Table 8.27 Average Mathematics Knowledge Distributed Across Teacher
Preparation Institutions ..................................... 122
Table 8.28 Distribution of Predicted Mathematics Knowledge across Teacher
Preparation Programs (low to high) ............................ 123
Table 8.29 Average Mentoring Frequency Distributed Across Teacher Preparation
Institutions (low to high) .................................... 124
Table 8.30 Distribution of Mentoring Frequency Across Teacher Preparation
Programs (low to high) ..................................... 125
Table 8.31 Teacher Preparation Programs Ranked on Teacher Graduates’
Mathematics Knowledge and Mentoring Frequency ............... 131
9. RESULTS FROM THE HLM ANALYSIS
OF CONNECTICUT 4th-8th READING GAINS ....................... 133
Table 9.1 Description of Variables Used in the Connecticut Reading
Analyses ................................................. 134
Table 9.2 Connecticut’s Average Reading Gain 1999-2000 .............. 136
Table 9.3 Regression Coefficients for Connecticut Reading Gain 1999-2000
by Ethnicity .............................................. 138
Table 9.4 Predicted Reading Gains by Student Demographic Group ........ 139

xii

Table 9.5 Regression Coefficients for Reading Gains by Gender (CT) ...... 140
Table 9.6 Regression coefficients for Connecticut Reading Gain 1999-2000 by

Gender and Ethnicity ....................................... 141
Table 9.7 Predicted Reading Gain by Student Demographic Groups with
Differences by Gender ...................................... 142
Table 9.8 Regression Coefficients for Connecticut Reading Gain 1999-2000 by
Gender, Ethnicity, and Grade ................................. 144
Table 9.9 Predicted Mathematics Gains by Student Demographic Group and
Grade ................................................... 144
Table 9.10 Regression Coefficients for Average Connecticut Reading Gain
1999-2000 Across Teachers .................................. 148
Table 9.11 Regression Coefficients for Average Reading Gain by Average
District SES .............................................. 149
Table 9.12 Variance in Average Reading Gains Explained by SES ......... 150
Table 9.13 Variance Decomposition Table for Average Reading Gains Across
Teachers In Connecticut ..................................... 151
Table 9.14 Unconditional Models for Levels of Teacher Variables Across
Teacher Preparation Institutions .............................. 154
Table 9.15 Connecticut’s Average English Preparation by Teacher Preparation
Institutions ............................................... 1 55
Table 9.16 Regression Coefficients for Teacher English Preparation by Teacher
Preparation Institutions ..................................... 156
Table 9.17 Teacher English Preparation by 13 Connecticut Teacher Preparation
Programs (low to high) ..................................... 156
10. RESULTS OF THE HLM ANALYSIS
FOR 1996 NAEP 4TH GRADE MATHEMATICS ...................... 161
Table 10.1 Description of Variables Used in the NAEP 1996 4th Grade
Mathematics Analyses ............................................ 162
Table 10.2 NAEP 1996 Average Mathematics Achievement .............. 164
Table 10. 3 Regression Coefficients for NAEP 4th Grade Mathematics 1996
by Ethnicity .............................................. 166

Table 10.4 Predicted Mathematics Achievement Student Demographic Group 167
Table 10.5 Regression coefficients for NAEP Mathematics Achievement 1996

by gender ................................................ 168
Table 10. 6 Regression Coefficients for NAEP 4th Grade Mathematics

Achievement 1996 by Gender and Ethnicity ..................... 169
Table 10.7 Predicted Mathematics Achievement by Student Demographic

Group with Differences by Gender ............................ 170
Table 10.8 Regression Coefficients for NAEP 4th Mathematics Achievement

1996 by Gender, Ethnicity, and SES ........................... 172
Table 10.9 Predicted Mathematics Achievement by Student Demographic Group

and SES ................................................. 173
Table 10.10 Regression Coefficients for NAEP Mathematics Achievement 1996

by Gender ................................................ 177

xiii

Table 10.11 Regression Coefficients for NAEP Mathematics Achievement 1996

by Gender and Average Classroom SES ........................ 179
Table 10.12 Variance Decomposition for Male/Female Controlling SES -
Teacher Practices .......................................... 181
Table 10.13 Variance Decomposition for Male/Female Controlling for SES -
Teacher Characteristics ..................................... 182
Table 10.14 Regression Coefficients for 1996 NAEP 4th Mathematics
Achievement by Ethnicity ................................... 184
Table 10.15 Regression Coefficients for Black/White Achievement Predicted by
Classroom SES ........................................... 186

Table 10.16 Variance Decomposition for Black/White achievement by SES . 187
Table 10.17 Teacher Practices - Variance Decomposition for Average

White/Black Achievement Across Teachers Controlling SES ....... 189
Table 10.18 Variance Decomposition for Black/White controlling for SES -

Teacher Characteristics ..................................... 191
Table 10.19 Variance Decomposition for Black Female Controlling for SES -

Teacher Practice, Characteristics, and Professional Development . . . . 193

11. RESULTS OF THE HLM ANALYSIS

FOR 2000 NAEP 4TH GRADE MATHEMATICS ...................... 196
Table 11.1 Description of Variables Used in the 2000 NAEP 4th Grade
Mathematics Analyses ............................................ 197
Table 11.2 2000 NAEP 4th Grade Average Mathematics Achievement ...... 199
Table 11.3 Regression Coefficients for 2000 NAEP 4th Grade Mathematics by
Ethnicity ................................................. 201

Table 11.4 Predicted Mathematics Achievement Student Demographic Group 202
Table 11.5 Regression Coefficients for 2000 NAEP Mathematics Achievement

by Gender ................................................ 203
Table 11.6 Regression Coefficients for 2000 NAEP 4th Grade Mathematics
Achievement by Gender and Ethnicity ......................... 204
Table 11.6 Predicted Mathematics Achievement by Student Demographic Group
with Differences by Gender .................................. 205
Table 11.8 Regression Coefficients for 2000 NAEP 4th Mathematics
Achievement by Gender, Ethnicity, and SES .................... 207
Table 11.9 Predicted Mathematics Achievement by Student Demographic Group
and SES ................................................. 208
Table 11.10 Regression Coefficients for 2000 NAEP 4th Mathematics
Achievement by Ethnicity ................................... 212
Table 11.11 Regression Coefficients for Black/White Achievement Predicted by
Average Classroom SES .................................... 214
Table 11.12 Variance Decomposition for Hispanic/White Achievement
by SES .................................................. 215
Table 11.13 Variance Decomposition for Hispanic/White controlling SES -
Teacher Practices .......................................... 216
xiv

 

12. RESULTS OF THE HLM ANALYSIS

FOR 1998 NAEP 4TH GRADE READING ........................... 218
Table 12.1 Description of Variables Used in the 1998 NAEP 4th Grade

Reading Analyses .......................................... 219
Table 12.2 1998 NAEP 4th Grade Average Reading Achievement .......... 221
Table 12.3 Regression Coefficients for 1998 NAEP 4th Grade Reading

by Ethnicity .............................................. 223

Table 12.4 Predicted Reading Achievement Student Demographic Group . . . 224
Table 12.5 Regression Coefficients for 1998 NAEP Reading Achievement

by Gender ................................................ 225
Table 12.6 Regression Coefficients for 1998 NAEP 4th Grade Reading
Achievement by Gender and Ethnicity ......................... 226
Table 12.7 Predicted Reading Achievement by Student Demographic Group with
Differences by Gender ...................................... 227
Table 12.8 Regression Coefficients for 1998 NAEP 4th Reading Achievement
by Gender, Ethnicity, and SES ................................ 229
Table 12.9 Predicted Reading Achievement by Student Demographic Group and
SES ..................................................... 230
Table 12.10 Regression Coefficients for 1998 NAEP 4th Reading Achievement by
Ethnicity ................................................. 234
Table 12.11 Regression Coefficients for Black/White Achievement Predicted by
Average District SES ....................................... 236

Table 12.12 Variance Decomposition for Black/White Achievement by SES . 237
Table 12.13 Variance Decomposition for White/Black controlling SES -

Teacher Practices .......................................... 239
Table 12.14 Variance Decomposition for Black/White Controlling for SES -

Teacher Characteristics ..................................... 240
Table 12.15 Variance Decomposition for Black/White Controlling for SES -

Teacher Professional Development ............................ 242

13. RESULTS OF THE HLM ANALYSIS

FOR 2000 NAEP 4TH GRADE READING ........................... 244
Table 13.1 Description of Variables Used in the 2000 NAEP 4th Grade
Reading Analyses .......................................... 244
Table 13.2 2000 NAEP 4th Grade Average Reading Achievement .......... 247
Table 13.3 Regression Coefficients for 2000 NAEP 4th Grade Reading by
Ethnicity ................................................. 249

Table 13.4 Predicted Reading Achievement Student Demographic Group . . . . 250
Table 13.5 Regression Coefficients for 2000 NAEP Reading Achievement

by Gender ................................................ 251
Table 13.6 Regression Coefficients for 2000 NAEP 4th Grade Reading

Achievement by Gender and Ethnicity ......................... 252
Table 13.7 Predicted Reading Achievement by Student Demographic Group

with Differences by Gender .................................. 253

Table 13.8 Regression Coefficients for 2000 NAEP 4th Grade Reading

XV

Achievement by Gender, Ethnicity, and SES .................... 255
Table 13.9 Predicted Reading Achievement by Student Demographic Group and

SES ..................................................... 256
Table 13.10 Regression Coefficients for 2000 NAEP 4th Grade Reading

Achievement by Ethnicity ................................... 259
Table 13.11 Regression Coefficients Mean Achievement Predicted by Average

District SES .............................................. 260
Table 13.12 Variance Decomposition for Mean Achievement by SES ....... 261
Table 13.13. Teacher Effects on Mean Achievement Controlling SES ...... 263

14. COMPARISON OF HLM RESULTS

FOR MATHEMATICS AND READING ACROSS DATA SETS ......... 264
Table 14.1 Comparison of Variation Across Data Sets ................... 264

Table 14.2 Group Means Which Varied Across Teachers (x) for All Data Sets 266
Table 14.3 Variance in Mean Achievement by Ethnicity Explained by Teacher

Variables Across All Data Sets ............................... 267
Table 14.4 Variance in Mean Achievement by Gender Explained by Teacher
Variables ................................................ 273
APPENDDC A: Correlation Tables ........................................ 294
Table A.1 Connecticut 2000 4th-8th Mathematics (n=168) ............... 295
Table A2 Connecticut 2000 4th-8th Reading (n=168) ................... 297
Table A3 Tennessee 2000 4th-8th Mathematics and Reading Correlations
(n=442) .................................................. 299
Table A4 1996 4th Grade NAEP Mathematics Correlations (n=584 teachers) 301
Table A5 NAEP 2000 4th Mathematics Correlations (n=1029) ............ 302
Table A6 NAEP 1998 Reading (n=757) .............................. 303
Table A7 NAEP Reading 2000 (n=867) .............................. 304
APPENDIX B: Miscellaneous Tables ...................................... 305
Table 8.1 Student Demographics - Dummy Coding .................... 306

xvi

LIST OF FIGURES

3. CONCEPTUAL MODEL .............................................. 21
Figure 3.1 Conceptual Model ........................................ 21
7. STRUCTURAL EQUATION MODELING RESULTS ...................... 47
Figure 7.1 Structural Equation Model for Mathematics in Connecticut and
Tennessee ................................................. 49
Figure 7.2 Structural Equation Model for Reading in Connecticut and
Tennessee ................................................. 54

Figure 7.3 1996 NAEP 4th Grade Mathematics Structural Equation Model . . . . 59
Figure 7.4 2000 NAEP 4th Grade Mathematics Structural Equation Model . . . . 60

Figure 7.5 1998 NAEP 4th Grade Reading Structural Equation Model ........ 61

Figure 7.6 2000 NAEP 4th Grade Reading Structural Equation Model ........ 62
14. COMPARISON OF HLM RESULTS

FOR MATHEMATICS AND READING ACROSS DATA SETS ......... 264

Figure 14.1 Conceptual Model (repeated) ............................ 278

Figure 14.2 Three Level HLM model ................................ 280

xvii

INTRODUCTION

Ideas about the importance of assessment, accountability, standards, and teacher
quality to student achievement in education are changing. Although education reform has
often foundered due to a lack of system-wide cohesion, the focus of current federal policy
is on system-wide accountability tied to student scores. The underlying premise is that
the system is or can be made cohesive, that pushing button A will trigger button B to
produce the desired result, but is it really so simple? Can accountability based on
students’ adequate yearly progress (AYP) really force the system to improve? Is there
evidence that teacher preparation, teacher characteristics, professional development,
and/or teacher practice are sufficiently connected to K-12 student outcomes so that
change in one is related to change in the other?

The lack of research which ties the elements of the education system to student
achievement in a single model has fueled arguments on both Sides of the education
controversy. Those who believe that the current system is “broken,” that high barriers
and low standards prohibit the best applicants from entering teaching focus on an
expensive system that doesn’t produce increased student achievement (Angus, 2001;
Paige, 2002; Ravitch, 2002). Those who believe that the lack of highly qualified teachers
is tied to the weakening of certification standards, an underfunded public education
system, and lack of teacher professionalization, also point to low student achievement as
the end result (Darling-Hammond & Youngs, 2002). In either case, empirical evidence
that teacher preparation, teacher characteristics, professional development, and teacher

practices are or are not related to each other and to student achievement is lacking.

No one denies that education in the United States is complicated by a world of
competing expectations, demands, and separately administered but intertwined levels of
responsibility. A long list of research, both qualitative and quantitative, attests to a
resistant and fragmented education system fraught with uncertainty and conflicting goals,
where students, teachers, administrators, and districts work toward different ends (Cusick,
1992; Floden & Buchmann, 1993; Labaree, 1997; Lortie, 1975; Muncey & McQuillan,
1996). At the local level, principals and teachers may or may not understand and follow
state standards and practice guidelines (Spillane, 2005; Youngs & King, 2002). Teachers,
especially beginning teachers, often default to the way they were taught when they were
students {Lortie, 1975 #57}. When teachers do seek to improve their practice based on
research results, evidence is oflen lacking on what constitutes high-quality research and
what practices actually improve student scores {What Works Clearinghouse, 2002 #488}.
Conflicting goals, resistence, and lack of proven teaching practices surely place time on
the side of education tradition (Tyack & Cuban, 1995).

Teacher shortages and underfunded state and district budgets tangle the quest for
teacher quality with the need to fill classrooms. In some cases at the school district level,
this means hiring teachers who lack appropriate credentials or who come from alternative
certification programs that are sometimes less rigorous routes into teaching {Darling-
Hammond, 1999 #498}. In other cases, teaching positions are left unfilled while class
size climbs. Additionally, rising costs curtail academic support programs, giving teachers
and administrators more to do with less.

Politicians are under great pressure to prove they are spending the taxpayers

money wisely in times of shrinking budgets and term limits. They cannot afford to wait

for positive news about failing schools in their districts so many are embracing the push
for accountability tied to standards, assessment, and teacher quality at all levels. Under
President Clinton, the Title 11 State Report Card (Reauthorization of HEA 1998)(U S
Department of Education, 1998a) was the first federal legislation aimed at improving
teacher quality and teacher preparation by requiring all states to report teacher
certification pass rates by teacher preparation institution to the Department of Education
for public distribution. It strongly encouraged communication between teacher
preparation programs and state government, as well as alignment among teacher
preparation program content, teacher certification examinations, and K-12 student
standards and assessments.

Even newer is the federal law that requires all states to develop their own
accountability systems for public education. “No Child Left Behind” (NCLB) Act of
2001, formerly known as ESEA, mandated that all states test annually in mathematics and
reading for all students in grades 3-8. NCLB, the flagship of President Bush’s education
policy, requires that states align student tests and teacher professional development with
state standards. By the end of the 2006 school year, all teachers must be “highly
qualified.” {, 2002 #257}

While the federal government has gradually increased its presence in public
education since the Elementary and Secondary School Act (ESEA) of 1988 (US
Department of Education, 1998b), the most recent NCLB assessment and accountability
provisions have proven far-reaching and expensive. States must balance increased
student testing costs against programs and services as federal funding dwindles.

The idea that large-scale assessments and accountability will improve the quality

of instruction and increase student learning for all students enjoys widespread support
(Amrein & Berliner, 2002; Robert L Linn, Baker, & Betebenner, 2002; Traub, 2002).
The pressure and stakes are high for all participants in the education system but it is
unclear how strongly the teacher education, teacher quality, teacher professional
development, teacher practices are related to the fifth component, K-12 student scores, or
what policy instruments help or hinder student gains.

Qualitative and quantitative research connects teacher preparation, teacher
characteristics, and professional development to teacher practice, but consistent findings
are rare. There is little empirical research that specifically traces the effects from teacher
preparation, teacher characteristics, or professional development through teacher practice
to student achievement. The most convincing evidence supporting the idea that alignment
among standards, professional development, and teacher practice can improve student
achievement is the Cohen and Hill (D. K. Cohen & Hill, 2001) study of mathematics
reform in California. They found that teachers who had the opportunity to study the
reform and received professional development in the supported mathematics framework
improved student scores on the state test.

The history of education suggests a system resistant to change and lacking
consistent connections, yet accountability, which relies on a connected system, is already
in place demanding swift, positive results for all children. Can student improvement be
attributed to changes in teacher practice? Can changes in teacher practice be traced to
changes in teacher preparation and professional development? Can changes in student
scores be attributed to teacher personal and professional characteristics? Do all children

benefit from new teaching practices associated with the standards movement or does this

reform, like so many other efforts, unwittingly benefit the advantaged groups in our
society?

To shed light on these concerns, this research examines the effects of teacher
preparation, teacher characteristics, professional development, and teaching practice on
student mathematics and reading achievement using data from the “Beginning Teacher
Preparation Survey (BTPS)”(Valli, 2000) and consecutive administrations of the National
Assessment of Educational Progress (N AEP). Employing a combination of qualitative
and quantitative methods, the project asks the following series of questions at both state
and national levels:

1) What characteristics of teacher preparation programs are strongly related to:

0 Teacher practice?
0 Student math and literacy achievement?
- Differences in achievement across student demographic groups?

2) What characteristics of teacher professional development are strongly related

to:

- Teacher practice?

- Student math and literacy achievement?

- Differences in achievement across student demographic groups?

3) What teacher characteristics are strongly related to:

0 Teacher practice?
- Student math and literacy achievement?
0 Differences in achievement across student demographic groups?

 

4) When all three are included in the model, what are their relative contributions

to:

- Teacher practice?

- Student math and literacy achievement?

0 Differences in achievement across student demographic groups?

Sections 1-2 review research on the constructs of interest, their relationships to
each other and student achievement, as well as related education policy at the state and
national levels. The third section develops the conceptual model that includes the four
constructs and their hypothesized connections to student achievement in mathematics and
reading, This is followed by a description of data sets, scales, and variables. Section 6
presents the methods and procedures used in the analyses followed by the section on
structural equation modeling (SEM) which determines the relative importance of teacher
preparation, teacher characteristics and professional development on teacher practice and
on subsequent student achievement. Sections 8-14 present the results for hierarchical
linear modeling (HLM) analyses which specifically looks at differential outcomes of
teacher variables on average achievement by gender and ethnicity across data sets.
Finally, Section 15 discusses overarching research results and implications for future

research.

l. PREVIOUS RESEARCH

In the emerging atmosphere of accountability linked to student assessments, the
research focused on five high profile targets for education policy: teacher preparation at
institutions of higher education (hereafter referred to as teacher preparation or teacher
preparation program), teacher quality, teacher professional development, teacher
practices, and their relationships to Student gains/achievement.

Accountability and Assessment. Large scale accountability systems which hold
students, teachers, schools, and districts accountable for student scores on state
assessments are relatively new to American education. Accountability systems must
answer three basic questions: who is to be held accountable, for what will they be held
accountable, and what will be the consequences (Goertz, Duffy, & with Le F loch, 2001).
NCLB (2001) holds students, schools, districts and states accountable for the pass rates of
all students on the required state student assessment; HEA (1998) holds states and teacher
preparation programs accountable for the pass rates of their students state teacher
certification examinations. States face the threat of bad publicity on “report cards” and
possible withdrawal of federal funding if they, or those under them, do not comply with
the law. In addition to negative publicity and loss of funds, failing K-12 schools face a
series of increasing consequences beginning with “being identified” for improvement and
culminating in takeover or reconstitution. Schools of education face adverse publicity on
the State Report Card, the loss of accreditation, and funding.

The idea that large-scale assessments and accountability will improve the quality

of instruction and increase student learning for all students enjoys widespread support

(Amrein & Berliner, 2002; Robert L. Linn, 2001; "No Child Left Behind Act of 2001 ,"
2002; Paige, 2002, 2003, 2004; Traub, 2002). Unfortunately, it is far easier to agree with
policy than to implement it. It is well documented that policy is modified by local
interpretations and contexts during the implementation process (David K. Cohen, 1990;
D. K. Cohen & Spillane, 1993; Kingdon, 1995; Tyack & Cuban, 1995); modifications are
more likely to occur the further one gets from the source of authority and closer to the
point of delivery (Lipsky, 1980).

The pressure and stakes are high for all participants in the education system but it
is unclear how strongly the four components (teacher education, teacher quality, teacher
professional development, teacher practices) are related to the fifth component, K-12
student scores, or what policy instruments help or hinder student gains. In part, the lack
of large scale databases which span all five measures tied to student gains/achievement
has limited the research community’s ability to tackle such questions. This is not to say
that the relationships among teacher preparation, teacher characteristics, professional
development, and teacher practice have not been studied, but rather that only a small
proportion of studies have extended those relationships to include student achievement or
gains.

Teacher Preparation. Policy makers have focused on teacher education as a
pressure point to improve K-12 student learning although they lack strong evidence about
what teacher education practices should be supported (M. M. Kennedy, 1999; S. Wilson,
Darling-Hammond, & Berry, 2000; S. M. Wilson, Floden, & F errini-Mundy, 2001).
Nonetheless, it is important to discover which elements or combinations of elements

actually make a difference in teaching practice and student learning. If government

wishes to push the educational system by requiring higher student scores, then it is
imperative to know which factors contribute to the desired student achievement outcomes
for all students.

Wilson, F loden and F errini-Mundy (S. M. Wilson et al., 2001) examined fifty-
seven rigorous, peer-reviewed studies that addressed the key issues in initial teacher
preparation. Little research directly assessed prospective teacher subject matter or
pedagogical knowledge and then evaluated the relationships between teacher preparation
and student learning. There were inconsistent among those that did. The review called for
research which was more specific about the features of program content and quality.

Teacher Characteristics. Teacher characteristics include both professional and
personal attributes. High quality teachers make a large difference in student learning
(Darling-Hammond, 2000; Ferguson, 1991; Sanders & Rivers, 1996; Spellings, 2005).
Indicators of teacher quality typically include scores on tests, degrees, course work, and
certification status. Wayne and Youngs (Wayne & Youngs, 2003) reviewed twenty-one
studies that connected teacher characteristics to student achievement. The authors found
that positive relationships existed between college ratings of teachers’ undergraduate
institutions and K-12 student achievement. The review could not determine the
importance of degrees and course work in general, although mathematics achievement in
high school was consistently positive. One cited study (Eberts & Stone, 1984) found no
relationship between number of college level mathematics courses in the last three years
and 4th grade math achievement. Certification studies also indicated that subject specific
qualifications matter but there were differences in interpretation across studies.

Other studies have looked at the impact of teacher personal characteristics on

student achievement. Barton & Oja (Barton & Oja, 1999), found that efficacy, teachers’
beliefs that they can affect Student achievement, is small but positively related to student
performance although others have reported mixed results (Brian Rowan, Chiang, &
Miller, 1997; Valli, Reckase, & Raths, 2003).

Teacher Professional Development. Somewhat more research was done to
connect in-service teacher preparation to student achievement. Kennedy (M.M. Kennedy,
1998) and Cohen & Hill (D. K. Cohen & Hill, 2001) focused on the relationship between
teacher professional development/in-service teacher education and student achievement.
Kennedy found twelve studies that used student achievement as the outcome measure for
the effectiveness of professional development. She noted the lack of research on teacher
professional development, intermediate outcomes such as teacher practices, and their
ultimate effect on student outcomes.

Cohen & Hill's study (2001) centered on the implementation of mathematics
reform in California. Their findings indicated that although learning opportunities varied
among teachers, certain types of learning opportunities influenced practice, the amount of
professional development influenced practice, using the student test as a learning
opportunity for teachers influenced practice, and certain types of teacher learning
opportunities affected student achievement. Student achievement was the dependent
measure; teachers’ practice was both an intermediate dependent measure of policy
enactment and a direct influence on students’ performance; teachers were the key
connection between policy and practice.

Teacher Practice. While there is evidence that teacher practice is related to

student achievement, there is little evidence that teacher practice actually increases

10

student achievement (Supovitz, 2001). In his review of the literature, Supovitz found
strong evidence, replicated across contexts, that connected teacher professional
development to teacher practice. There were a handful of studies that connected teacher
practice to student achievement. The results from those studies showed that, although
positive, the impact of teacher practice on student achievement was hard to detect.

As in mathematics, experts disagree about what teacher reading practices are most
desirable. The effects of teacher reading practices may be more difficult to interpret since
they are confounded by issues of culture and power (Delpit, 1988; Foster & Peele, 1999;
Heath, 1983; Hirsch, 1996). Although most educators agree that a combination of
traditional and reform instructional strategies produce the best overall results for most
children (Carnilli, Vargas, & Yurecko, 2003; Pearson, 1996), it is also possible that
specific instructional practices benefit student demographic groups differentially. The
National Reading Panel (N RP)(N ational Reading Panel, 2000) found that direct
instruction in phonics was especially useful for children who had not developed these
skills during the initial phases of reading acquisition. This lends support to Delpit's idea
that when instruction relies heavily on implicit components, it may be difficult or
impossible for children from different cultures and ethnic backgrounds to succeed unless
these hidden components are addressed directly. While more children Show achievement
gains when a blend of instructional practices is used those gains may not be significant
for specific demographic subgroups of children.

Ongoing BTPS studies. Important work on the connections among teacher
preparation, professional development, and student achievement in mathematics has been

done by the developers of BTPS. Valli, Reckase, and Raths (Valli et al., 2003) focused

11

on teacher education’s relationship to student gains in Tennessee. They found modest but
significant relationships among teacher preparation program variables, program
outcomes, teaching practice and student mathematics achievement. They also noted a
negative effect of preparation to teach mathematics on student achievement, possibly due
to the study’s grade level span or the inexperience of beginning teachers to assess how
well prepared they, in fact, were. In a later article (Reckase, 2002), university faculty
were found to be critical components of teacher preparation for mathematics instruction
in Tennessee when student math achievement was the outcome.

The literature points to a need for more studies that include student scores on state
assessments as the indicator of the effectiveness of teacher preparation, teacher quality,
professional development, and teacher practice. Now that policy makers, politicians,
educators, and parents believe that these connections can realistically be made and
measured, they will continue to insist that states, teacher preparation institutions,
researchers, districts, schools, teachers, and students use K-12 student scores as the litmus
test of system effectiveness. In the hopes of providing baseline evidence, it is this study’s
challenge to make sense of the links between government policy and student achievement

in a complex, pre-NCLB world.

12

2. EDUCATION POLICY ENVIRONMENTS

Government policy affects education at all levels. Three state level policy choices
believed to influence education systems within states are the decisions to institute low
stakes versus high stakes testing, centralized versus decentralized responsibility for
education, and level of per pupil spending (Fuhrman, 2001; Greenwald, Hedges, & Laine,
1996; Sanders & Rivers, 1996). Connecticut and Tennessee approached improvement in
K-12 student achievement from decidedly different angles while existing in the same
national environment.

In 1999-2000, the national environment was rapidly changing from one that
traditionally lefi matters of education to the states to one that exerted strong pressure on
states to improve student achievement in prescribed ways. It is important to note that
many of the national changes had been introduced and were being discussed but had not
yet taken effect. This section describes the policy environments at the state and national
levels during the 19905 and early 20005.

State Level:

Connecticut and Tennessee made major changes to education during the 19903.
The most striking difference in their policy environments was system focus. Connecticut
concentrated on inputs such as raising teacher quality with low stakes K-12 assessment
and Tennessee focused on outputs by instituting a high-stakes accountability system
based on K-12 student gains. State focus affected how pre-service teachers were prepared,
how in-service teachers taught and were professionally supported, as well as affecting the

content that students were expected to know and demonstrate.

l3

There were serious disparities between Connecticut and Tennessee that played a
part in teacher quality and student achievement. Connecticut outspent Tennessee and
Connecticut students outperformed their Tennessee counterparts on NAEP. In many
ways, Connecticut had the financial luxury to focus on inputs. An Education Week
analysis of 1999-2000 SASS data (Education Counts Research Center, 2002) revealed
that 99.9% of Connecticut students attended schools in districts that have per pupil
expenditures at or above the national median ($5,385) compared to 25.3% students in
Tennessee. Willis Hawley (1991), then director of the Center for Education Policy at
Vanderbilt, commented that the Tennessee tax structure didn’t yield enough revenue to
make massive changes to education. TVAAS (Sanders & Horn, 1998) offered a
reasonable and cost-effective alternative to Connecticut’s centralized, multifaceted
reforms.

During this period, the courts mandated that both states address educational equity
based on district funding differences. In Sheff v. O’Neill ("Sheff v O'Neill," 1996), the
Connecticut state supreme court ruled that poor, urban schools were receiving an inferior
and unequal education. In Tennessee ("Tennessee Small School Systems v. McWherter,"
1993), the court ruled that wealthier urban districts were advantaged in comparison to
poor rural districts.

To fill in background for the state policy environments, additional information
from the Education Week analysis of 1999-2000 SASS data indicated that Connecticut
and Tennessee stacked up in the following ways: Both states were reported to have
similar teacher/staff ratios and about the same percentage of total education dollars spent

on teachers. Tennessee was more likely to have teachers without a major or minor in the

14

subject and/or without certification. While both states offered educational assistance to
teachers in the form of loans, scholarships and the waiving of license fees, Connecticut
additionally offered housing assistance (loans, low interest mortgages, tax credits)
targeted toward subject area shortages and high need schools.

In terms of achievement, 32% of Connecticut 4th grade students scored at
proficient or higher on NAEP mathematics compared to 18% in Tennessee. 6% of black
4th graders were proficient or higher in Connecticut compared to 4% in Tennessee and
9% of hispanic 4th grades were proficient or higher in both state. A discussion of state
specific education policy follows.

Connecticut:

The BTPS survey took place during a time of comprehensive, standards-based
education reform in Connecticut. The ambitious goals, outlined in Connecticut’s
Comprehensive Plan for Education 1996-2000, aimed “to set and meet high standards for
the performance of teachers and administrators leading to and evidenced by improved
student learning” (Sergi, Commissioner of Eduction,1998). Expectations and supports
for teachers undergirded expectations for student learning. There was a concerted effort
to align all levels of the system with K-12 standards.

The emphasis on teacher quality included higher standards for entry into teaching.
Along with the increased expectations for teachers came increased teacher salaries (S.
Wilson et al., 2000). All prospective teachers had to pass both the Connecticut
Competency Examination for Prospective Teachers (CONNCEPT) which tested basic
knowledge and skills in reading, writing and mathematics plus either the National

Teacher Examination (NTE) or the Connecticut Elementary Certification Test

15

(CONNECT) demonstrating subject matter proficiency. To further ensure teacher
quality, all prospective teachers had to pass PRAXIS 11 content area examinations before
graduation.

Teacher preparation programs were not only required to obtain National Council
for the Accreditation of Teacher Education (N CATE) approval but also incorporate
instruction on the aligned policy instruments into their programs. These instruments
included the Common Core of Teaching (CCT) and the Code of Responsibility for
Teachers aimed at teachers and the Common Core of Learning (CCL), the Connecticut
Mastery Test (CMT), and the Connecticut Academic Performance Test (CAPT) which
impact K-12 students.

Professional development in Connecticut came under the heading BEST,
Beginning Educator Support and Training. During their first year, all beginning teachers
were assessed six times using the Connecticut Competency Instrument (CCI), an
observation instrument designed to measure effective teaching. District-based mentoring
and state supported professional development were available to support new teachers
undergoing observations. Teachers unable to meet CCI standards were refused
provisional certification and prohibited from teaching.

The CMT, a criterion-referenced test aligned with K-12 state standards, measured
student achievement against standards-based performance levels. As a low stakes
assessment, it was meant to inform student progress and teacher practice.

NAEP results from 1998 showed that Connecticut students were among the best
in the nation on reading. There were, however, documented achievement gaps between

white and minority students. To correct the problem, a 1996 Supreme Court decision

16

(Sheff v. O’Neill) determined that a 1909 law created a system of economic and racial
segregation. Consequently, the state began a number of new initiatives to address
inequity as it overhauled education. The state increased school funding by fifiy million
dollars (1996-97). Districts were mandated to develop minority recruitment plans and
create magnate schools to increase movement across district lines.

Neither education nor policy is static so it was not surprising that policy
environment changed during the study year. Just as the final pieces of Connecticut’s
Comprehensive Plan for Education 1996-2000 were implemented, Greater Expectations:
Connecticut’s Comprehensive Plan for Education 2001-2005 emerged.

Tennessee:

At the core of Tennessee education reform was the adoption of Tennessee Value-
Added Assessment System (TVAAS) in 1992. System focus was on the product of the
educational experience rather than the process by which it was to be achieved (Sanders &
Horn, 1998, p 339). TVAAS defined academic progress in terms of yearly student gains
on the Tennessee Competency Assessment Program (TCAP). Teachers and schools were
rated on whether their students make more or less progress than a typical student is
expected to make in a given subject and grade, after adjusting for the prior achievement
of each child (Archer, 1999). Rewards and sanctions were in place for those teachers
who either exceeded or failed to meet the minimum progress.

The State Board of Education adopted new professional licensure standards for all
teacher candidates in 1998. The standards were aligned with the Interstate New Teacher
Assessment and Support Consortium (INTASC).

School-based decision making was encouraged and was tied to student academic

17

 

progress. Teacher professional development activities and teacher practices were also
expected to be results-driven. TVAAS allowed teachers and schools to use whatever
methods resulted in achieving student academic progress (Sanders & Horn, 1998). The
state role was to enhance the capacity of each school and district to take responsibility for
the professional development of it’s own personnel. By focusing on test results, other

elements in the system were expected to fall into line.

National Level:

NAEP has been the Nation’s Report Card since 1969. It’s purpose from inception
was to reflect current educational and assessment practices and measure change reliably
over time. Mandated by Congress, the voluntary assessment was informational with no
stakes attached. Rather than representing one state’s content standards, it included
nationally representative content identified by the Council of Chief State School Officers
(CCSSO). The National Assessment Governing Board (NAGB) hired ACT, Inc., to
identify appropriate achievement standards for each subject and grade tested.

While arguments and plans for the improvement of K-12 education raged at all
levels of government, the authority to act in these matters rested with state and local
government. In 1998, public and special interest group pressure, such as the National
Commission for Teaching and America’s Future (N CTAF) which focused in issues of
teacher quality (National Center for Research on Teaching and Learning, 1993),
combined to create a new expectation that the federal government could improve the
quality of K-12 education by improving the quality of teacher preparation. This pressure
opened a window that eased the way for Title II and NCLB reporting requirements to

come into existence.

18

Title H of the 1998 Amendments to Higher Education Act (HEA) of 1965 was
designed to recruit new teachers to areas with shortages of qualified personnel; to prepare
them in both subject matter and pedagogical knowledge; and to gather information on
state and university policies and procedures related to teacher quality. Section 207 of this
amendment mandated that all states and institutions of higher education (lHEs) receiving
any funds directly or indirectly through the HEA must provide the Secretary of Education
with data on teacher standards and preparation. In addition to the reporting of pass rates
on state teacher examinations by subject and by population subgroups, other information
in the state report was to be determined by the Secretary of Education in conjunction with
the National Center for Educational Statistics (N CES).

Virtually all states, territories, and the District of Columbia received HEA funds
directly or indirectly and therefore had to provide this information on their teacher
preparation programs. With the possible exception of a few, small, private IHEs, most
colleges and universities enrolled students who received federal financial assistance
authorized under this Act. Consequently, information on nearly all US teacher
preparation programs was gathered and reported to the Secretary of Education in 2001.
The Initial Report of the Secretary on the Quality of Teacher Preparation (otherwise
known as the State Report card) was voluntarily collected in the Spring of 1999 to
provide a baseline measure of teacher quality. While reporting was not mandatory during
the BTPS survey year, the initial report card heralded impending changes to relationship
between the federal government and states.

Measures of state and federal policy were not available in the data sets used for

this study so it was not possible to link most policies to the model statistically. The one

19

exception was that teacher pass rates from the Initial State Report Card could be

appended to the Connecticut data and tied to IHE teacher preparation program name.

20

3. CONCEPTUAL MODEL

In order to answer the questions posed in this study, it was necessary to develop a
conceptual model that depicted the hypothesized relationships among government policy,
teacher preparation, teacher characteristics, professional development, teaching practice,
and student achievement. It was posited, as did BTPS, that there were direct linear
relationships between teacher preparation, teacher characteristics, professional
development and teacher practice, as well as a direct linear relationship between teacher
practice and student achievement. Given the standards-based climates of Connecticut and
Tennessee, this linear relationship was extended from government policy through teacher
preparation, teacher characteristics, and professional development to teacher practice and
from teacher practice through to student achievement as the following model suggests.

Figure 3.1 Conceptual Model

A\
/' IHE \
Teacher 1
Preparation _

\
fl \\\ fl\

, overnment , .
\ Policy l > Teacher 7930"" )—>Student Gains!

,I—Pl
V/

Characteristics x w Achievement

   
 
   

 

    

 

 

 

A

     
     

Professional
Development

Quantitative analysis

 

Qualitative analysis ______

21

Elements of the model:

Government policy refers to state policies for teacher qualifications and
professional development, standards alignment, equity, and state assessments. The state
policy environment is set in a federal policy environment which, in 1999-2000,
influenced state education policies primarily using inducements in the form of grants
(McDonald & Elmore, 1987).1 Dashed lines represent the suspected effects of policies
that can be reviewed but not tested in this study.

Teacher preparation encompasses the quality of teachers’ formal pre-service
teacher education experiences. Teacher characteristics cover individual qualities, both
personal and professional, that teachers bring to their teaching positions. Examples
include subject matter credentials, efficacy (belief that they can make a difference for
students), and specialized training in areas like assessment and diversity. Professional
development includes time spent in professional development as well as its perceived
impact. Teacher practices for both reading and math include traditional and reform
activities. An example of a traditional activity would be mathematics drill and lecture
activities while group and project activities would represent a reform activity. Depending
on the data set used, the outcome variable for student achievement is either student
gains over the course of one year (1999-2000) or the plausible value scores from the 4th
grade NAEP reading (1998, 2000) or 4th grade mathematics (1996, 2000). Note that only

teacher practice is hypothesized to be directly related to gains, that the variables in the

model must go through student classroom experiences to affect student achievement.

 

1 In 2005, the model would be the same but federal No Child Left Behind
policies are now believed to affect state education policy strongly and in direct ways.

22

Solid lines represent the hypothesized paths tested in this study.
Full descriptions of these variables follow in Section 5, Latent Variables and

Their Indicators. First, however, come descriptions of the BTPS and NAEP data sets.

23

4. DATA SETS

The study used six existing databases, two from BTPS and four from NAEP. All
offered similar variables spanning the policy to student achievement continuum. Samples
varied by the analysis performed and are presented in Table 2.

Table 4.1 Sample Comparisons by Data Set and Analysis

 

 

 

EEEEEE z - zMathz C - L - , - , ,,Reading-w,-wl
CT TN NAEP 4th NAEP 4th CT TN NAEP 4th NAEP 4th
2000 1996 2000 1998
ISEM Teachern 168 442 1029 584 168 442 867 757
ELMSIPQENLL 1.550.. EX, .‘ 6202 L 4109, ..--90- XL D 4§4§L-r EFL
,HLM Teacher n 81 X 1245 584 1776 X 918 828

 

The Connecticut and Tennessee SEM correlation matrices included all teachers
and average student gains for both mathematics and reading. Due to the presence of
individual student data nested withing teachers in Connecticut, it was possible to split the
mathematics and reading samples for separate HLM analysis.

AM Sofiware, necessary for creating weighted NAEP correlation matrices, used
listwise deletion for missing values. HLM, which also needs complete data, used a
refined set of variables based on findings from the SEM analyses. This allowed some
cases to be recovered. Consequently, NAEP n for HLM is larger than that for SEM.
More information on methods and procedures are included in Chapter 6.

State data. NPEAT researchers conducted BTPS during the 1999-2000 school
year in Connecticut and Tennessee. There were three related goals (Valli, Reckase, Raths,
& Rennert-Ariev, 2001):

1. To develop a survey that would provide information from beginning

teachers about their preservice and induction experiences; their teaching

24

beliefs, knowledge and practices; and their teaching context
2. To develop theoretically and empirically based constructs of teacher
preparation, induction, teacher knowledge and beliefs, and teaching
practices
3. To determine the relationships among three sets of variables: a) teacher
preparation, b) teaching knowledge, practice, and beliefs, and c) student
learning.
The survey was designed to study the linear relationship of teacher preparation and
professional development to teacher knowledge, practice, and beliefs to student
achievement. Valli and colleagues targeted 3rd through 8th grade math and
reading/language arts teachers in their first three years of practice in public schools.
BTPS selected the states of Connecticut and Tennessee based on their national
reputations for innovative K-12 educational practices. The focus on student outcomes
required pre- and post-test student scores which could be directly tied to specific teachers.
BTPS took advantage of an existing sample (n=168) selected for a prior study of
beginning teachers by the Connecticut State Department of Education (CSDE). 100% of
the CSDE sample responded to the BTPS questionnaire. The Tennessee sample consisted
of 442 beginning math and reading teachers from 3rd-8th grade. Since all teachers in
Tennessee have student gain scores attached to their performance, researchers compiled
mailing lists to select their sample.
Connecticut Teacher and Student Samples. Teacher samples were compared to
a national profile of beginning teachers, the 1998 Teacher Survey on Professional

Development and Training conducted by NCES. While the study does not plan to

25

generalize from the Connecticut and Tennessee samples to the rest of the United States, it
is interesting to note that Connecticut and Tennessee teachers were demographically
much like US teachers overall with a few exceptions: they were more likely to be white,
female (no high school teachers were included in the survey), and possess masters
degrees. There were two major differences between Connecticut and Tennessee: 22% of
the Connecticut sample was male in comparison to in 7.6% in Tennessee. The Tennessee
sample was similar to the national profile on diversity (21% minority) but the Connecticut
sample was predominantly white (94.5%) (Valli, Reckase et al., 2001). The outcome
variable, student achievement, was defined as student gain scores from the beginning to
the end of the school year based on two administrations of the state assessment. In
Connecticut, demographic information and scores exist for approximately 2500 students
nested within the 168 teachers. The primary sampling unit for the survey was teachers.

Tennessee Teacher and Student Samples. Due to confidentiality concerns,
Tennessee did not include individual student scores tied to teachers as part of the BTPS
data. After aggregating students by teacher using the value-added framework, Dr.
William Sanders, director of the TVAAS program, provided student outcome correlations
to BTPS researchers who used them to create a full correlation matrix for entry into SEM.
Specific demographic information at the student level was absent.

The BTPS data included variables that did not exist in NAEP but that were
nevertheless important for studying the effects of teacher preparation on teacher practice
and student achievement. Not only were factual measures available for teacher
preparation, knowledge, and practice, BTPS also included attitudinal measures. For

example, BTPS researchers found that a teacher's sense of self-efficacy was an important

26

predictor of student gains in their preliminary multivariate analyses.

National Data. NAEP regularly assesses a nationally representative sample of
students in grades 4, 8, and 12 in most subjects. While students are the primary sampling
unit, teachers fill out questionnaires regarding their education, certification, and teaching
practices for each class that includes assessed students. At the national level, gain scores
do not exist for the same samples of children and classrooms; however, one purpose of
NAEP and its complex design is to serve as an effective barometer of change over time.
Two consecutive administrations of 4th grade math (Mathzl996 & 2000) and two
consecutive administrations of 4th grade reading assessments (Reading: 1998 & 2000)
provide a measure of change in student achievement and teaching practice over time.

NAEP Teacher and Student samples. For all four NAEP data sets, the public
school sample with no accommodations (psamp) was selected. The national weight
(origwt) was applied to produce national samples of students. The teacher sample exists
only as part of the student record so the data for teachers who have multiple students
assessed appeared multiple times at the student level. In order to create a teacher level
sample, teacher variables were aggregated by teacher booklet. Consequently, the teacher
sample is not random. Analyses using these data must be interpreted as random samples
of students who have teachers with certain characteristics or who employ certain
practices.

The single most important characteristic of both BTPS and NAEP data is the
existence of a measure of student achievement that is nested within specific teachers.
Teachers, then, provide the all-important link between the four latent variables and

student achievement. These six data sets provide state and national assessment results as

27

a measure upon which to explore the effects of teacher characteristics, professional
development, teaching practices, and, in the case of Connecticut, teacher preparation.
Scale comparisons for BTPS and NAEP. No comparison of multiple data sets
can be completed without a comparison of scales that are used. BTPS data included
many scales important to this study. It was necessary to create similar scales for NAEP
data. Table 4.2 presents the scales created for BTPS. Tables 4.3 through 4.7 contain

information on similar scales created for NAEP mathematics and reading.

28

Table 4.2 BTPS Scales and Properties

 

Beginning Teacher Preparation Survey #of Range Mean Standard Internal

 

 

 

 

 

 

 

 

Scales ltems scores Deviation consistency
i-IE Teacher Preparation T
Faculty Characteristics 6 9-30 15.20 4.9 0.77
Program Quality 4 5-25 17.70 4.10 0.73
Field Experience 4 ”5-20 __ _ 15.20 _ _ 3.70 _ _ 0.77
Teacher characteristics A , _. _ _ _
MA TH
Traditional Orientation 4 4-20 13.60 3.00 0.64
Reform Orientation 4 9-20 15.30 2.20 0.45
Math Preparation 6 0-12 3.50 1.80 0.70
Math Instruction Prep 13 14-65 45.50 11.20 0.96
Math knowledge 6 9-30 24.30 17.80 0.90
BOTH
Efficacy 7 9-35 28.37 29.00 0.66
Diversity Prep 7 7-35 19.70 6.20 0.92
Assessment PreparationNariety 8 8-40 29.00 6.80 0.95
READING
Balanced Literacy Orientation 8 10-40 34.60 3.30 0.65
ReadingNVriting Preparation' 1 0-4 0.80
Reading/Writing Instruction Preparation 12 15-60 41.80 10.00 0.94
Literacy knowledge 10 3-50 37.10 7.10 0.90
EFofessional Development T ' "
Mentoring worth 11 11-33 23.70 7.80 0.96
Professional development frequency 10 10-50 21 .60 8.30 0.90
Professional development impact 12 12-72 46.20 1 1.90 0.90
fiber Practice ' '
Math Drill and Lecture activities 6 11-30 26.00 2.80 0.64
Math group and Project activities 7 8-35 20.00 5.10 0.79
Math sensemaking activities 7 18-35 29.30 3.60 0.75
READING
Reform activities 8 15-40 30.60 5.00 0.82
Traditional Activities 7 13-35 31 .70 3.20 0.70
_Litera_cy materials_ -_ 4 4-20 10.10 __ 4.0_0_ 0.68

* only one variable available

29

Table 4.3 NAEP 1996 Mathematics Scales

 

#of Range Mean Standard Alpha

 

 

 

 

 

Items scores Deviation

{catcher characteristics TI
Reform Orientation 4 1 1.53 2.17 0.67
Math Preparation 2
Math Instruction Prep 2 7.49 0.9 0.89
Math knowledge 4 3.25 1 .14 0.74
Diversity Prep 3 1.58 1.15 0.68

flashed Development ‘
Professional development frequency" 1

Eacher Practice

 

Math group and Project activities 4-16 11.97 2.69 0.74

Math sensemaking activities 5-20 10.35 2.74 0.73

Emphasize Group & Project Assessment 3-12 0.69
" only one variable available

(”P

Table 4.4 NAEP 2000 Mathematics Scales

 

#of Range Mean Standard Alpha

Items scores Deviation
Teacher characteristics ‘ TY

Math Preparation 6 38.67 6.66 0.61
Math Instruction Prep 8 27.88 3.38 0.85
NCTM standards knowledge‘ 1

1

State standards knowledge"
fimfoml Development
Professional development frequency" 1
ice cher Practice ‘ ' “’5
Math group and Project activities 3 4-16 9.45 2.21 0.74
Math sensemaking activities 6 5-20 13.53 3.07 0.70
Math Drill & Lecture Activities 3 4-9 8.63 0.78 0.67
" only one variable available

30

Table 4.5 1998 NAEP 4th Grade Reading Scales

 

#of Range Mean Standard Alpha

 

 

 

 

 

 

Items scores Deviation

Teacher characteristiés 7
Diversity Prep 2 3.97 1.24 0.69
Assessment PreparationNariety 11 29.01 5.29 0.71
Reading/Writing Preparation 4 16.55 8.27 0.92
Reading/Writing Instruction Prep 9 23.54 3.26 0.85
Technology preparation 3 5.68 1.56 0.71

Efessional Development
Professional development frequency* 1

Echer Practice I
Reform activities 14 38.16 6.53 0.84
Traditional Activities 12 37.57 3.79 0.64
Computer Activities

 

* only one variable available

Table 4.7 2000 NAEP 4th Grade Reading Scales

 

#of Range Mean Standard Alpha

 

 

 

 

 

 

Items scores Deviation

Lacher characteristics
Diversity Prep 3 6.19 1.68 0.72
Assessment PreparationNariety 6 16. 35 3.19 0.63
ReadingNVriting Preparation 4 17.3 8.61 0.92
ReadingNVriting Instruction Prep 8 21.04 3.05 0.86
Technology preparation 4 7.72 2.12 0.8

Eofessional Development '
Professional development frequency‘ 1

Teacher Practice '
Reform activities 5 14.83 2.81 0.74
Traditional Activities 3 9.44 1.84 0.75
Computer Activities 3 7.17 2.96 0.83

 

* only one variable available

31

Internal consistency using Cronbach’s Alpha was computed for NAEP scales in
SPSS. In most cases, the number of items and therefore the range of the scales is
considerably smaller for NAEP but sample sizes are much larger than BTPS.
Additionally, results are intended to reflect national samples of students and their teachers
rather than smaller state samples.

NAEP alphas ranged from .61 to .92, with eight in the .605, ten in the .705, seven
in the .805, and two in the .905. BTPS alphas ranged from .45 to .96, with five in the 605,
seven in the 705, two in the 805, and nine in the nineties. It is reasonable that an
instrument such as BTPS designed to capture these particular scales would have better
reliabilities overall than NAEP which created for other purposes. In spite of these
differences in intent, the NAEP scales performed reasonably well and in similar fashion

to the BTPS scales across analyses.

32

5. LATENT VARIABLES AND THEIR INDICATORS

Three key latent variables (teacher preparation, professional development, and
teacher characteristics) were hypothesized to be related to a fourth latent variable, teacher
practices. Each latent variable had three or more indicator variables, a requirement for
analysis stability in SEM. A discussion of the latent variables and their indicators
follows.

Teacher Preparation. A5 of the dates of this study, both Connecticut and
Tennessee required graduation from an accredited IHE teacher preparation program to be
licensed as a teacher. While the importance of IHE teacher preparation as the only route
to teaching is under debate, the qualities of IHE teacher preparation programs which are
related to K-12 teacher practice and subsequent student gains has not been studied. In
order to look at these connections, BTPS asked teachers to rate their preparation
programs on a number of factors. Three scales are included as indicators of teacher
preparation: faculty characteristics, field experiences, and program quality. These
constructs are unique to BTPS and one of the characteristics that makes the data so
attractive.

The faculty characteristics scale was designed to measure whether or not teacher
preparation faculty practices are consistent with the ideals they espouse in the classroom.
Field experience is an attendee rating of the quality of the internship experience. The
program quality scale includes items related to the foundation and strength of disciplinary
preparation as well as a rating by graduates of their teacher program quality.

Teacher Characteristics. Indicators of teacher characteristics include personal

33

and professional qualities of individual teachers: subject matter, pedagogical, and
diversity preparation; teacher efficacy and orientation; and assessment
preparation/variety. Teachers with greater subject matter knowledge tend to stress
conceptual understanding and ask higher order questions (Ball, 1997; Grossman, 1989)
These teacher understandings are particularly relevant when the goal is student
understanding (Borko & Putnam, 1996). Pedagogical preparation is also related to student
understanding (Shulman, 1987). It allows teachers to maximize student learning by
applying subject specific strategies for teaching the subject. Scales for subject matter and
pedagogical preparation are included for both mathematics and reading below.

Scales used for both mathematics and reading - Teachers will always have prior
knowledge and beliefs that filter their learning and teaching (Borko & Putnam, 1996;
Lortie, 1975). Teacher efficacy is one such belief. The Valli et a1. (Valli, Raths, Reckase,
& Rennert-Ariev, 2001) work using the BTPS in Tennessee found that teacher efficacy, a
scale which assessed the belief that a teacher can make a difference for all students, was
related to increased student achievement. It has also been related to improvement in
teaching practice and student motivation and performance (Barton & Oja, 1999).

The diversity preparation scale includes measures on that how well teachers
believe they are prepared to teach students from other cultures and/or with different
languages. When teachers choose instructional strategies, they must also be able to tailor
that knowledge to the diverse learners represented in their classroom (Borko & Putnam,
1996; Shulman, 1987).

The National Research Council (1999) argued the need for matching assessments

with student learning goals. They claimed that effective teachers learn about students’

34

learn about students’ understanding and assess students often. Assessment preparation
covers teacher readiness to evaluate student learning, to align assessments and
expectations, and to use assessments to guide decisions about teaching.

Mgthemgtics only - Teacher characteristics applicable only to mathematics
preparation includes traditional orientation, reform orientation, mathematics preparation,
mathematics knowledge, and knowledge of national and state mathematics standards.

The data includes two orientation scales, traditional and reform. Mathematics
reform orientation is measured by asking whether the respondents agreed with items such
as “a teacher’s primary goal is to help students achieve a deeper conceptual understanding
of mathematics.” Mathematics traditional orientation reflects agreement with items such
as “a teacher’s primary goal is to help students learn mathematical terms and master
computational skills.” The scales are not mutually exclusive and both reflect elements of
the NCTM standards (National Council of Teachers of Mathematics, 2000) which
advocate a mixture of conceptual and computational instruction for students.

The mathematics knowledge scale measures teachers’ self-reported understanding
in five areas of mathematics (general knowledge; fractions and decimals; estimation and
measurement; patterns, functions, and algebraic thinking; statistics and probability; and
geometric concepts) with ratings from “poor” to “excellent.” Teacher self—reports of
mathematics knowledge correlate with student achievement (Robert L Linn & Baker,
1998)

The preparation for teaching mathematics scale combines responses (“not at all”
to “very well prepared”) on a series of questions regarding a teacher’s pedagogical

training in computations, problem solving, reasoning, assessment, etc. A separate scale

35

combines scores for mathematics major, passed mathematics course work, and possession
of mathematics license. These are indicators of a teacher’s pre-service mathematics
preparation.

In NAEP 4th grade mathematics analyses, knowledge of NC T M standards and
knowledge of state standards knowledge were included in the analyses. Knowledge of
NCTM standards is believed to result in teaching with a combination of conceptual and
computational methods which will improve student achievement. According to Cohen
and Hill, knowledge Of state standards helps teachers align their instruction to the goals of
their state which result in better scores on assessment (D. K. Cohen & Hill, 2001). Both
are single variables based on a 4 point Likert scale from “no knowledge” to “very
knowledgeable.”

Reading Only- There is some overlap between reading, English, and language arts
in these scales. Teacher characteristics included in the reading analyses are balanced
literacy instruction, reading preparation, reading instruction preparation, literacy
knowledge, and technology preparation related to reading/writing instruction. According
the National Reading Panel, it is critically important to know what teacher characteristics
influence successful reading instruction (National Reading Panel, 2000). The Panel
targeted a combination of reading comprehension techniques.

Balanced literacy orientation reflects a moderate orientation toward reading
instruction which includes emphasis on the mechanics of reading and writing as well as
on comprehension. Reading preparation combines scores for reading/English major,
passed courses in subject matter course work, and holding a license in

reading/English/language arts.

36

Reading/writing instruction preparation measures how well teachers are prepared
to teach vocabulary, oral and silent reading, comprehension activities, etc. Knowledge of
language, literacy development, and writing processes as well as knowledge of literature
is included in Literacy knowledge. Technology preparation, a scale encompassing
teachers’ preparation to use computers and software for reading/writing instruction, is
included in the NAEP analyses.

Professional Development. Professional development and continuing education
allow the teacher to hone current skills and knowledge, as well as keep abreast of new
knowledge, theories, and methods (Borko & Putnam, 1996). Professional development
has the ongoing potential to support and improve the existing teaching force. The
National Reading Panel (National Reading Panel, 2000) found that in-service
professional development produced significantly higher student achievement. For many
in-service teachers, professional development was the first and only interface with
computer technology in the classroom, one of the most important instructional
innovations of the last decade (Prawat & Worthington, 1998; Rich, Sun, Velu, &
Wallace, 1998).

Professional development is defined as a combination of mentoring experiences
that teachers received and professional development they attended since completing
formal teacher preparation. Scales for professional development impact, professional
development frequency, and mentoring worth were developed as part of the original
BTPS study. Of the three, only professional development frequency was available in
NAEP Math and Reading across study years.

Mentoring scales from BTPS are highly correlated with each other r2 =.936,

37

p<.001), a strong indication that the scales are measuring the same construct. Only
mentoring frequency was chosen for the analyses since it had the fewest missing cases.
Professional development impact includes ratings of how helpful teachers found various
activities commonly covered in professional development, such as using a wider range of
strategies to help students learn or managing the classroom more effectively. How much
time teachers spent in professional development activities is measured by the professional
development frequency scale. This scale is available across all data sets.

Teacher Practices. Teacher practices in mathematics and reading are assumed to
be the most important influence on student learning. It is in the classroom where
pedagogical content knowledge, the combination of subject matter knowledge and
pedagogical skills, come into play. According to Shulman (1987), pedagogical content
knowledge is more elusive than subject matter knowledge but equally important. It is the
knowledge base, that allows a teacher to synthesize disciplinary knowledge and
pedagogical knowledge into a cohesive lesson that students can understand. Pedagogical
content knowledge also enables the teacher to elicit and/or circumvent common
misconceptions surrounding the subject matter (Borko & Putnam, 1996).

Mathematics Only. As mentioned earlier, the 2000 NCTM Standards recommend
a combination of strategies (both conceptual and computational) as important to student
mathematics achievement. BTPS scales for teacher mathematics practices covered three
important types of activities which include both conceptual and computational elements:
mathematics group and project activities, mathematics drill and lecture activities, and
mathematics sense-making activities. Frequencies for items measuring each activity

were added together to form the three teacher practice scales.

38

Mathematics group and project activities combine items such as working in small
groups to solve math problems and working on mathematical investigations that span
several days. Items like lecturing or presenting information and drilling students on
computational skills are included in mathematics drill and lecture activities. The
mathematics sense making activities scale reflects the frequency of conceptual activities
that teachers employ in the classroom to help students make sense of mathematics (i.e.,
having students make conjectures, leading discussions on ways students solve particular
problems, applying math to real world problems, modeling different learning strategies,
etc.)

The scale for drill and lecture activities is not available in NAEP Mathematics
1996. Since SEM requires three indicator variables, emphasis on group and project
assessment was chosen since the other possible variables measuring teacher practices had
already been incorporated into scales.

Reading Only. Reading practices measure the frequency that teachers use reform
activities, traditional activities, computer activities, and literacy materials in there
classrooms. Reform activities include comprehension activities, metacognitive routines
for independent reading, oral and written responses to literature, etc. Traditional
activities encompass activities such as oral and silent reading and reading vocabulary.
Most teachers use both types of activities in the classroom. The scale for literacy
materials is only available in BTPS and shows the range of textbook use from “none” to
“a great amount.” Computer activities in the NAEP analyses provides a connection from
teacher technology preparation to student achievement.

Student Achievement. Data sets in this study have two different outcome

39

measures for student mathematics and reading, gain scores in BTPS and multiple
plausible values in NAEP. While student gain scores have been considered somewhat
suspect due to the many influences besides teaching that can affect student scores over the
course of a single year, gains over three years have been successfully employed as a
barometer of growth in Tennessee (Sanders & Horn, 1998). Each student essentially
serves as his own control group eliminating many family and societal factors external to
the classroom. The Connecticut State Department of Education (CSDE) supplied two
scores on the Connecticut Mastery Test per student, one from the beginning and one from
the end of the 1999-2000 school year for all teachers involved in the BTPS. Gains scores
were then computed subtracting pre-test results from post-test results.

The outcome variable in NAEP is a measure of achievement status, a one time
score. NAEP is unusual in that no student completes the entire assessment. Using a
complex system of weighting and matrix sampling, five plausible values representing a
range of possible scores are created for each student based on like student patterns found
in the data. For the SEM analyses, the five plausible values (MMPCMl-S for
mathematics or RRPCMl-S for reading) are included as indicator variables for the
outcome variable, student achievement. HLM handles the five plausible values

simultaneously by running five separate analyses and then averaging the results.

40

6. METHODS & PROCEDURES

Overview. All analyses were based on the conceptual model (Figure 1) and
repeated separately for reading and mathematics across six data sets. Two primary types
of data analysis software were used to confirm constructs, measure relative relationships
to student achievement, determine differential effects by student demographics groups,
and compare results for mathematics and reading across policy contexts: structural
equation modeling (SEM) using AMOS 4.01 (Arbuckle, 1994-1999) and hierarchical
linear modeling (HLM) using HLM 6.01h (Raudenbush, Bryk, & Congdon, 2005).

SEM was especially useful for analyses conducted at the teacher level with
aggregated mean student gains/achievement standardized by grade as the outcome. It
allowed estimation of the relationships among the latent variables and their indicators in
order to validate constructs. SEM also provided standardized direct effects for teacher
preparation, teacher characteristics, and professional development on teacher practices
and indirect effects on student gains/achievement simultaneously. This answered
questions regarding the relative strength of the relationships among the latent variables.
The results from the SEM analyses were then compared across data sets.

Education data often has two- or sometimes three-level structures. In this study,
students were nested within teachers who were then nested within teacher preparation
programs. HLM was particularly appropriate because it solved the unit of analysis
problem: each level in the structure was formally represented by its own submodel
(Raudenbush & Bryk, 2002). HLM did not require that student data be aggregated to the

teacher level, as in SEM, or teacher data to be replicated for each student at level one, as

41

found in NAEP data. HLM also allowed the selection of the level of generalization for
the analyses. This was particularly important for NAEP since the students were randomly
sampled at level one, not the teachers at level two. Results can then be interpreted for a
random sample of students who have teachers with certain characteristics.

Given the complexities of the educational system and the many intervening
circumstances that impact teachers and students inside and outside the classroom, effect
sizes are expected to be small. The primary criterion for judging effect size in education
research, however, is its potential value for informing or benefitting education practice
(Camilli et al., 2003; McCartney & Dearing, 2002; McCartney & Rosenthal, 2000).
BTPS researchers anticipated effects sizes of no more than .05 over the course of one year
and this is a potentially significant finding (Reckase, 2002; Valli et al., 2003).

Due to limitations in variable availability, it was not possible to run every model
with every data set. Great care was taken to make scales across data sets as similar as
possible. Table 4 shows the analyses that were performed and Table 5 indicates the
variables available in each data set. Correlation matrices for all SEM models may be

found in Appendix A.

42

 

 

 

 

 

 

Table 6.1 Analysis Matrix
CT TN NAEP NAEP NAEP NAEP
1996 1998 2000 2000
Analysis Unit of Math Read Math Read Math Read Math Read
analysis
SEM Teacher x x x x x x x x
HLM Lv|1 x x NA NA x x x x
Student
Lv12 x x NA NA x x x x
Teacher
Lvl 3 x x NA NA NA NA NA NA
Teacher
Preparatio
n Program

 

 

 

 

 

 

 

 

 

 

 

Correlation matrices served as input for all SEM analyses since the student

achievement data from Tennessee was only available in this form. HLM analyses were

not conducted with Tennessee data since there were no individual student scores that

could be nested within teachers. Only the data from Connecticut was suitable for the

three level analysis.

43

 

Table 6.2 Variable Availability Across Data Sets

SLcLaIeLsLNariLabLes ALvaiLlabILe by DLataLsetL

 

Math

CT TN_ N_AEP4t_h NAEP4th CT TN NAEP4th NAEP4Lth

___... _ __ .‘_.—._

 

 

 

 

 

 

 

 

 

 

 

 

 

2000 1996 2000 1998 ’
{HE Teacher Prgparation L L LLLL LLlL ,LLLL I L‘ , j _j
__FacuLIty QhaIECSEristLlssL L L L .x L 2:, N: L NA L x xLNALL NAL
LPrograLm QuaIityL LL x x NA L _NA L xL xL LNAL L NLAL L
Field Experience L LLL LxL xLL NAL L NAL L L x x NA NA L
ELLQEI.,9_I!aéct3Lfi§Lti¢6L L L L - I T I T If 7:
LEATLHEMAI’CE _ L L L LLL LLL- L LL LLL L L L
TLLraLditioLnal Orientation x x L LN/A _N/A 1 l .
L_RLefoer—L orientation x .Z‘ LN/AL 2‘. LL L L L_LL LL .1
Math LPrLepLaLratiLonL x x x x ,L I
jathlnstruthionLPrepL L L L x x L x 3; L L L -
MLLaLth knowledgeL LLL x Lx_ LLN/AL .XL ; T
LNLCTM LsLtLandLardLs kflowILedgLe _..* I . x L* L LLLLLLLLL LLLLLI
State sLtandards LkLnoLvaledLgeL L L: * _x _ _L‘L' L
.LLQQTLHLL L L L L L L LL_____.-____ 4L __ LL'L l
, encacy LLLL fix- £3116 _.NL/£~ .xL xLNA. L N_A_ L L
LDLivLersityL PrLep L Lx _XL NIA .XL _xLL_xL_x L L ‘xL L L
L_LAsLseLssnLIefltPLrLepgratIonNariety .X LL x LN/ALL NIA x _LxLLLx x L
LEEADLNE L _ _ L ’ ‘ ‘ L 1 L LLLLJ
L_BLaILancedL LLiteLracLy OriLentationL L I I" LxL LAL L L L L L
Reading/Writing F’LrLepaLratioLn L E i L 5x LNALL _L Lx L L
Reading/Writing Instruction LPrepL L_L L ,. L'xL _xLx L L_xL L L
:LL LitLLeLraLcy LknowiLedge L L L , LL IL_ L_ L_jx ~NL L NA L L
Technology preparLLatioLLn L , ; I * L__*LLL)L<LL LL x L LL
Eofessional Development , i __ 1 I , r 9
LLIILIILeLntorjngL worthL L LL LLLLLL L_x LxL ,NA NA x x LNA L NAL L
ProfessionaLlL deveIgmeent freq LLL Lx LL 3‘ LL Lx xL x L L LL xLL L L
Professnonai development impact x .X NA NA x x LNA L NA
[Teacher Practice I l I l b I L
LEATHEMAI’Q- L i ‘ L L L L L
LMLaLth DriILl and LecyreL activities x x x NIA
:Math group and LProLjLectLL activitieLsL x x LL Lx _x
Math sLeLnsLeLmLakinLgLaLtivitiLes LxLL x x _x L
Emphasize Group& Project N/A NIA N/A x
A5§§S§mL§nL L L L. L__ . 5"
LEELADINGLL L L, i __ .‘
LLLRELOUT'LaEHY'tieLSLL , L L ‘ IX .X XL .XL L -
TLLLrLadLitIorLIaIAthiwIiesL L i Ax L)_(L Lx L’SLL L L
COTBUIEI ACIiL/‘tiEE' L ,L I “___L' ,ILX _ -Z‘LL L
Literacy materials LL LLL LLL _ I..- L L LIX Lx xL x L
LGainsIAchievement L_LL _ L.-. L j I a: T
GLain LLoLnL State TeL§t ,_ x L LxLL NLALL LNA LxL xLL NA__ NA L
PILLausibIe VaIUes 1-5 NA NA x x NA NA x x

 

* Computer activities# were available but were not included for CT and TN A

 

 

 

 

 

 

 

 

44

 

 

Methodological issues unique to NAEP data analyses. Methodological issues
that need to be addressed when dealing with NAEP data are the appropriate uses of
design weights and multiple plausible values. NAEP oversamples students from various
categories to ensure sufficient sample size for within group analyses. Without proper
weighting, these groups will be over-represented. Unfortunately, AMOS does not adjust
for weighting. Using origwt2 as the design weight, weighted correlation matrices were
generated in AM Software (J. Cohen, 2002), a statistical package created specifically for
use with NCES data sets.3 These weighted correlation matrices were then suitable for
input into SEM as excel spreadsheets.

In NAEP, students do not take all portions of the test but their probable total
scores are imputed and captured by five multiple plausible values. Each of the five
plausible values became an indicator variable for the latent variable for student
achievement in the AMOS analyses. HLM 6 includes special estimation settings for
design weights and multiple plausible values.4 The program runs the same equation five
times using each multiple plausible value as an outcome and then averages the results.

Methodological issues unique to the Connecticut analyses. Variation between
schools in many areas (socioeconomic status, funding, urbanicity, teacher quality, etc.) is

known to account for large portions of variation in student scores (D. K. Cohen & Hill,

 

2This is the weight for the national sample using psamp, the public school sample,
no testing accommodations.

3The software and documentation for this program may be downloaded at
http://am.air.org/.

4For descriptions of these processes, please see “Three additional features of
HLM” at http://www.ssicentral.com/hlm/example6-2.html.

45

2001; Raudenbush, Kasim, Eamsukkawat, & Miyazaki, 1996). BTPS collected
information on student race, grade, and gender but no measure of student SES.
Consequently, district percent free and reduced lunch from the 1999—2000 school year
was added to the teacher data as a measure of district socioeconomic status. The

information came from the Connecticut Department of Education website.5

 

5Data originally comes from the Common Core of Data, NCES.

46

7. STRUCTURAL EQUATION MODELING RESULTS

The SEM analysis is not only intended to address the strengths and weaknesses of
the conceptual model across data sets, but also the research questions about relationships
among teacher preparation, teacher characteristics, and professional development and
their relative contributions to teacher practice and subsequent student achievement in
mathematics and reading.

Teachers are the unit of analysis for SEM analyses. Student gains for Connecticut
and Tennessee have been aggregated to the teacher level and standardized by grade to
control for grade level effects. The five plausible values in NAEP have been aggregated
by teacher for these analyses. No standardization by grade was necessary since all
students are in the 4th grade. The five aggregated plausible values serve as indicator
variables for a latent variable labeled student achievement.

Using a recursive hybrid structural equation model, it is possible to judge the
adequacy of the conceptual model as a base for SEM analysis, determine the size of the
relationships among the latent and indicator variables, and assess their indirect and direct
effects on student achievement/gains across two curriculum contexts, mathematics and
reading. The hybrid model includes measurement models for each of the latent variables
and their indicators as well as the overall structural model based on the conceptual model
(Figure 1). To assess the adequacy of the measurement models, variance explained (r2)
indicates the extent to which each indicator variable measures its underlying construct
(Byme, 1998). Generally, r2 >.35 is considered large, r2>.15 is moderate, and r2>.02 is

small in structural equation modeling (J. Cohen, 1988). Overall model fit for the

47

structural model is determined using cmin (x2)/df <3, GFI (Joreskog)>.90, Pratio
(parsimony index>.60), and RMSEA<.05.

Results are reported as standardized effects to make it easier to compare results
across models. AMOS uses Monte Carlo estimation when data is input as a correlation
matrix. The procedure also calculates bias corrected significance levels and confidence
intervals. While the measurement and structural models are expected to retain high
goodness-of-fit properties across data sets, it is also expected that the models will show
sensitivity to policy and curriculum contexts at the state and national levels. Alternative
models were analyzed in Connecticut, but only the best-fitting model is presented below.
The identical model was used for the Tennessee analyses.

Connecticut Base Model for Mathematics

There are four latent exogenous variables indicated by ovals, each with at least
three observed endogenous variables (indicated by rectangles). Disturbance terms (ex., e
GPA for group and project activities) are included for all endogenous variables. One
coefficient for each latent variable is constrained to 1 as required by SEM. In the case of
reform and traditional orientation, both were set to 1 since they are believed to exert equal
influence on teacher characteristics. Teacher preparation, teacher characteristics, and
professional development are correlated with each other as indicated by the curved

arrows.

48

   

Emma.
Eman_w>mo ll .0... m

 

\ 3:03.33... F
E “aisle/gov 2.26 m... \ .l\o
_a :o_on80.m\\V\ E95293 . /,

_mcoamfleri _2w

\.\ ~
@wfl \ \x 552.
. / \ x \ 9:82.22
F \ , x), o
ll \ / . Salaam... .\ /
m

7/5
I
l>
?
2
\‘/)\£

 

 

    

 

 

 

 

  
 

 

       

I . ,. co_.ml.cm_.o . .l -
mcwflgwwwxom \\ 5.8mm “F0 .x. EmEmmomm< FA2w< .
fins. l l H: U xxx.-.)
..i ll .. 5.6535 0 /
\\ F \\ 3.995 --_...l moo.
. .1. ., \ / \llbxx... co. Elmaomm o/l.
F I)-.- x/V, X / 5:02.35 \Wd
. 32.8.. , .8..m_.eo~.m_.o\.. algaelgmz F is.
.323... . l .933... ./ m .o/\\.\)/
. . K, \.._// //, o 8.265. N _ .
. . /{ it ,3 N i x: 0
. wwzflfifll FF? .68 o
a”... ,1......_..._.._...._.,. m...
t l. l. 6.0.85 \l/
H 95 95.0 .1 .- / / cozm.lmlao.n_ o ltbxas. ow
.x.Fl ,, F O._. Vm moszmfims. lFl /,l....
x. 0 \ l / _ . \\l.//.O
r50 V . // .o/l... 3:90 On. my
_O/ A<m0F wv Elm/69m F _
\ ’wll -/ ,/l\ \j
. //\ ,/.’\ .
o . :ozuEQo/E/ worlo...mlaxul Em“. ow
.283... 22”. F /¢\
/W-l\\ o... m 0
3238“. F6

commences can 52.8500 5 83582.32 8.. 3on cozascm 3.302% _S 23E

49

Table 7.1 presents the standardized total effects for the Connecticut mathematics

model. Variance explained is indicated by r2. Tennessee results follow in Table 7.3.

Table 7.1 Connecticut Standardized Total Effects for Mathematics

 

 

 

 

 

 

   
  
  
 

 

 

 

 

 
   
  

 

 

 

 

Teacher Teacher Professional Teacher
r2 Preparation Cha_r§cteristics Development Practices
eglfiflrgarationi - ffffffffffff - - _ ‘ ‘ ’ g #_ .__ g .
Faculty Characteristics 0.314 0.560 *
Program Quality 0.592 0.769 *
Field Experience 0.200 0.447 **
eacher Cthacteristics
raditional Orientation 0.004 0.063 *
Reform Orientation 0.009 0.095 *
Math Preparation 0.013 0.113
Math Instruction Preparation 0.630 0.794 *
Diversity Preparation 0.313 0.559 **
sessment Preparation 0.578 0.760 **
Math Knowledge 0.155 0.393 *
Efficacy 0.115 0.339 *
Professiona_|[_)evelopment
Mentoring worth 0.372 0.610 *
Professional Development 0.303 0.551 *
Frequency
Professional Development 0.776 0.881 **
Impact - _
Teacher Practices 0.3@ -0.660 * 1.017 ** 0.166
Math Drill 8 Lecture 0.114 -0.223 * 0.344 ** 0.056 0.338 ** ~
Activities
Math Group & Project 0.691 -0.549 " 0.845 ** 0.138 0.831 *
Activities
Mathsensemaking 0.371 -0.402 * 0.619 ** 0.101 0.609 **
SandergiaeiMéjwflS- .0002. £028.- -_-, - .9053. -._-QQQL... 9.042
Fit Indices cmin/df NFI RMSEA PRATIO
2908*" 0.984 0.107 0.760 '

 

 

p<.01“, p<.001**, r2>.35 is bold

Discussion of Connecticut mathematics results:

Measurement model. Given the complexity of the education system, the theory

and research undergirding the conceptual model and constructs, and the rather small

sample for Connecticut, the four measurement models were not respecified based on r2

values since overall model fit was high. The construct for teacher preparation accounts

for 59.2% of the variance in program quality, 31.4% of the variance in faculty

50

characteristics, and 20% of the variation in field experiences. Of the eight teacher
characteristics indicators, the largest r2 are reported for mathematics instruction
preparation (pedagogy) which explains 63% of the variation and preparation for
assessment which explains 57.8% of the variation in teacher characteristics in
Connecticut.

Math preparation, the indicator of subject matter preparation, does not explain any
of the variance in Connecticut teacher characteristics (r2 =.Ol3)‘ Similarly low r2 using
Tennessee data was noted in Valli, Reckase, & Raths (Valli et al., 2003). Reform
orientation and traditional orientation are candidates for deletion due to low r 2. These
three variables, however, are too conceptually important to discard and they will also be
included in the later HLM analyses. Consequently, they are kept for comparison
purposes.

The construct for professional development explains 77% of the variation in
professional development impact, 37.2% of mentoring worth, and 30% of the variation in
professional development frequency. The construct for teacher practice explains 69.1%
of the variation in a teachers use of group and project activities and 37.1% of the
variance in math sensemaking activities (r2=.371). The percent of variance explained for
drill and lecture activities is small (11.4%).

On the whole, the latent variables account for moderate to large amounts of
variance in this model and the literature supports the connections.

Structural model. The model for Connecticut could be improved. For the
purposes of this research, however, it was deemed more important to rely on theory to

develop a model that would work well across multiple data sets, rather than tailor it too

51

specifically for a single state. The overall fit of the Connecticut mathematics model is
good (cmin(x2)/df <2.908, p<.000), GFI>.984, Pratio>.760). RMSEA is a little high
(.107). Additional relationships among the indicator variables for professional
development and the indicator variables for the other latent variables are the most likely
source of misfit. The correlation between teacher preparation and teacher characteristics
is high (r=.87) suggesting that these may not be separate constructs for this data.

Effects. Generally effect sizes above .10 are considered small, above .30 are
considered moderate, and above .50 are considered large (Kline, p. 98). This is a relative
rather than absolute rule of thumb. As mentioned earlier, an effect of 5% is deemed
potentially significant over the course of one academic year (Sanders & Rivers, 1996;
Valli, Raths, & Rennert-Ariev, 2001; Valli et al., 2003).

Teacher preparation and teacher characteristics have large effects on teacher
practice. The link between teacher preparation and classroom practice, however, was
negative (0=-.66), corroborating the findings in (Valli et al., 2003). They suggest that
elementary teachers, the largest proportion of teachers in the BTPS sample, are not as
likely to have majored in mathematics at the undergraduate or graduate level since it was
not a requirement for teaching elementary mathematics in 1999. The effect of teacher
characteristics on teacher practices is 5:1.017. Professional development produces a
small, positive effect on teacher practice (5:.166). Teacher practice has a small, positive
direct mathematics effect on gains (5:.042). The negative direct effects of teacher
preparation on teacher practice translated into a small, negative indirect effect on student
achievement (0=—.028).

The hybrid structural equation model has good measurement and structural fit.

52

Effect sizes are reasonable given theory and prior research. The model was adapted to
analyze reading in Connecticut.
Connecticut base model for reading

Once it was clear that the structural equation model (Figure 2) adequately fit the
mathematics data, the same structural model was fit to Connecticut reading data.
Elements of the education system, represented by the latent variables, are presumed to be
related in similar ways across subject matters in a strongly centralized state such as
Connecticut. The measurement model is slightly different since it was necessary to

substitute indicator variables that were pertinent to reading.

53

\ll// F
000500.050, \\.\\ m5.020.).
A35. 0V 300.3228.
.o.lwr \.\~Vflu/lll\
\ .._
L l - \ ..
\

. l \
l/ 8.8.8.8.). \\. \ \

 

socmllslami rfll/
\\ l l .cmE00.0>00 ..o
.mclo.mmmllF0.ln. F

fl£0203me—

 

T

 

 

5n. 0

>099... :Emmmm
G00 0 . . .l l l“
0.8.; 0 .x. co_l.m.000. /
ml .../.,,V A / ..x.-- 2.8.8208 . l. .V.8/
l / l l. . l- an. .
F ’x\\ /’\ \\...\\ l. \ l. Hlll F x x. .\
0:.00 \. 000.000.... V 033.800.25 ..0..0:..mcl_ xx , 0 l
00.003. . l . L/ .0000“; l l J. l A 3:000... V l l l 0500M”. A _m 0 V
ll. ,. . . ~ . l F
. \ ////lll. llll\ /,/ lll. ,,/l . l
0 \ .0 // 0000.305. l .0 . /
wmflzfifllV F ltllx-..’ .- P l Dal >092... Fll v... 0 V
>092... A .l- l . 00:00.6... ,. / lll ll.l // \._
5.00.200; ., mm...>_.0< >092... ,, 0 l
: wFl l 00.0.... 0.5.3. 000lcm.mm >000am. l
.l l 1H ll lll F “Em V
F./ l/ .lllll l\ _.-.l
8. 9m. WV .._.880 _o
7.0 F V ..H A Em.mo.n. l. On. 0
. llllll// \\ \\\\ ll. F \\
./l0 (Em 0V {rhuafiaoi /_0\/..l\ .l ll 0..
,l... 3:000... l l l l 00:0..00xw \
.0 , MI. ., 0.0.“. F A mu. 0
/./l l\\ H . .(\ \
0 F 0.622090 0“. m
3.000“. F
i .0

88880000... 000 300005.00 a. wficaom .00 .0002 00.80%. Egogm N... 0.595

54

Results of the SEM analysis for reading in Connecticut are presented in Table 7.2.
Results for Tennessee follow in Table 7.3.

Table 7.2. Connecticut Standardized Total Effects for Reading

 

 

 

 

 

 

 

 

 

 

 

 

Teacher Teacher Professional Teacher
r2 Preparation Characteristics Development Practices
Teacher Preparation
Faculty Characteristics 0.368 0.607 *
Program Quality 0.461 0.679 *
gig! Experience 0.231 0.481 *
T__eacher Chgfleristics
Balanced Literacy 0.025 0.157 *
Orientation
.Reading Instruction 0.832 0.912 **
. Preparation
Diversity Preparation 0.350 0.591 **
Assessment Preparation 0.697 0.835 *
Literacy Knowledge 0.324 0.569 *
Efficacy - 0.067 0.259 *
Professional Development
Mentoring Worth 0.301 0.549
Professional Development 0094 0.307 *
Frequency
‘Professional Development 0.795 0.891 **
__Irmact ¥ _
Iggcher Practices 0.144 0.274 0.010 0.163
Reform Literacy Activities 0.438 0.181 0.006 0.108 0.662 *
Traditional Literacy 0.359 0.164 0.006 0.098 0.599 *
Activities .
_Literacy Materials 0.006 0.021 0.001 0.012 0075;
Reading Gains 0.081 0.078 0.003 0.047 0.285 * .
‘Fit Indices cmin/df NF I RMSEA PRATIO
3264*" 0.985 0.116 0.721

p<.01*, p<.001**, r2>.35 is bold

Connecticut Reading Results

Measurement Model. Teacher preparation explains less of the variance in
program quality (46.1%) but more of the variance in faculty characteristics (36.8%) and
field experience (23.1%) than in the CT mathematics analysis. Teacher characteristics
explain 83.2% of the variance in reading instruction preparation and 69.7% of the
variance in assessment preparation, both larger than the r2 for corresponding mathematics

variables. Professional development impact accounts for the largest amount of variance

55

(79.5%) in CT reading. Teacher practice explains 43.8% of the variance in reform
reading activities and 35.9% of the variance in traditional reading activities. No
variance is explained for literacy materials.

Structural model. The overall fit of the CT reading model is good (cmin(x2)/df
<3.264, p<.000), GFI>.985, Pratio>.721) although the RMSEA is high (.116). A review
of the residual covariance matrix shows additional unanalyzed relationships among
indicators for each of the latent variables. The correlation between teacher preparation
and teacher charactersitics is high (r=.83) indicating these may not be separate constructs
for this data. In spite of this, the Connecticut models for reading and mathematics have
many good properties and appear to respond similarly in the analyses.

Effects. When looking at the pattern of effect sizes in the structural model for
reading, teacher preparation, teacher characteristics, and professional development have
only moderate effects on teacher practice. The indirect effects of teacher practice on
student reading gains are small but closer to the predicted level of 5:.05 than for
mathematics. One notable difference by subject matter is the direct effect of teacher
practice on reading gains (5:.285) compared to mathematics (5:.042).

The hybrid structural equation model fits both mathematics and reading data well
indicating that the latent variables are reasonably well-measured given theory and the
structural model is performing as anticipated across the Connecticut education system.
Effects appear sensitive to subject matter context. The next step is to confirm the model
in the remaining data sets. Results for Tennessee mathematics and reading come first

followed by the results for NAEP.

56

Tennessee Mathematics and Reading Results:

Tennessee models for mathematics and reading are identical to the Connecticut
models presented in Figures 7.1 and 7.2. Table 7.3 includes r2 and standardized effects
for mathematics and reading for Tennessee and NAEP. Tables 7.4 and 7.5 which present
results for all data sets, include the estimates of direct and indirect effects for
mathematics and reading in Tennessee.

Measurement Model. In general, there is more variation explained across
constructs in Tennessee than in Connecticut. Teacher preparation program explains more
variance in faculty characteristics (mathematics=75%, reading=76.3%), program quality
(mathematics=49.5%, reading=48.8%), and field experience (mathematics=27.8%,
reading=27.3%) in mathematics and reading than in CT. Teacher characteristics explain
the large amounts of variation in the variables for preparation to teach a particular subject
(mathematics=66.7%, reading=71%), diversity preparation (mathematics=45%,
reading=42.4%), and assessment preparation (mathematics=63.7%, reading=68.8%), all
components of teachers’ educational background. Professional development explains the
large amounts of variation in professional development impact (mathematics=86.2%,
reading=73.9%), followed by mentoring worth (mathematics=26.8%, reading=29.3%),
and professional development frequency (mathematics=24%, reading=28.6%)

Structural Model. The overall fit of the TN reading model was good
(cmin(x2)/df <4.962, p<.000), GFI>.974, Pratio>.76) and the RMSEA is lower (.095)
than CT although still above the desired level of .05. As in CT, a review of the residual
covariance matrix shows additional unanalyzed relationships among indicators for each

of the latent variables. The correlation between teacher preparation and teacher

57

characteristics is again high (mathematics r = .87; reading r = .85) but similar to the
results for Connecticut. The Tennessee models for reading and mathematics have many
good properties and responded similarly to the Connecticut analyses.

Effects. The patterns of coefficient sign and size in the structural model for
Tennessee begin to show differences from the Connecticut mathematics and reading
models. For example, the effect of teacher mathematics preparation on teacher practice is
considerably smaller although still negative (CT 5=-.660, p<.05; TN 5=-.111). A more
complete discussion of these differences will follow in the data analysis results across
data sets.

The same hybrid structural equation model fit mathematics and reading data well
in both Connecticut and Tennessee indicating that the latent variables were reasonably
well-measured given theory and the structural model was performing as anticipated
across the states. Effects appeared sensitive to subject matter context. The next step was
to run similar models using NAEP data.

NAEP Mathematics and Reading Models.

Figures 5 to 8 present the models for NAEP 4th grade mathematics and reading.
NAEP questionnaires changed between administrations and subject matter from 1996 to
2000. Consequently it was not possible to use the same variables across all NAEP
models. Reasonable substitutions were made when possible but sometimes different
variables all together were incorporated to fulfill specifications necessary to run SEM.6

Comparisons of NAEP results with those of Connecticut and Tennessee follow in

Tables 7.3-7.6. Results are discussed as part of the cross data analysis.

 

6Each latent variable needs to have at least three indicator variables.

58

 

«cacao—goo
Eggnog—2....

 

       

 

 

 

  

 

   

mo£>=o<
EoEmwmmvé.
60.85 a 9.90. ccwfimmwmw
ii 8:325:
Eosa>2=o< \ 8268.. \ 8.
14:13.82.“
3322...: .22.: ./._.w._h_fimzo ll 5....

 

\ _ \V i
z .. i L

i c3832.”. mum 0v
acimsowcmm 8308235. . F. .
- lllll o
m0€>z0< 2 FwE/wV it ill. maggocv. x 2 m
896$ \ 8385.5 # 8=mEo£mE _.

   

 

 

 

 

 

 

 

l l- -l -wcm 95.0 .o /\ Enos—om .O
~>n_
ll 1 P
r>a
F Om m
S. m l l I l 382 cesium _eaesm 3. 50552 035 av mm<z 32 ms 9...»:—

59

 

 

 

EmEasgoo

 
 

 

 

 

 

 

     

 

 

 

   

 

 

 

 

A m>n_ m V Q3 my __.—333.05
.0. .0
_. I
ll rm .1 \l / - l . mEmemcmw ...\
\
A m>a :94 my y x
. \ , \\..\
_. .\. . 5.8.39...
.\ .. mo=mEo£m2 A, \l/
. .\ ‘ /
\ ....\ ll/V \\|.M \ n=2 0 V
A, 3.35232 5. A .333 VA A
../ -. , .50—Baa... . 359.9
\imjfilifi .\- - ., ./|\l/\/ 23m 30:!
l l N. ,_ 1! l. t A n... m V .l ill ti o / r/ .il. ,.A.. mm o
9.. m2. -. $22.2 ,,
_. .O a . 896.... a 96.0 / mEmucSm A..//\\V
l l 1 1 l l 1 . .252 c
we F ! / >65. \1/
.r/ 1:- l A Thug“... o F - F#:5sz
. , \ ___. . ,.
A m>a m V _.>a m V A (m0 e V cozmfiaoi l\.\.
. /!l --i... ,l. lll: A N>Q \ ,.,,/,.il\\. CO_«Ong_ .o
o A» . ..... l .o 8:325: f<\l//
\Xx.‘ rI//, o F x
. As:
A 5.. m V , x
\ ./l\.
.o o

382 8.33m 3.33% 8.3605“: 2.80 5.. .m<z 8cm 3 2am...—

60

.\ //

EoEe>eEo<
,.. a553—

1.!

\

‘x

the 1..

N>n_

- HM;

A~>¢mV

«>Q

m1. -
/,l-\ .1 A5. mV

 

 

.A/UI.IF \\\\ \ \. J/ .././
A093. m \V
mm§>=o<
A 5.5.50
N.‘
A... 33an
V9155~oh V9 ,- - -

3.3.2

.2363

me>zo<
c551

.. .- P.
A. 5. 0J fig m V
..//llt\\ \ /Il\...

o .o

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

   

 

 

   

 

 

 

 

   

2.950550 A >< m V
E56355 ...., ./ x
F .oll
, 55555
/ .coEwmommé. \ \l/
.. an m V
\. F w/i\\
\ \. o
. 55.52.”.
.\ 21.520 -.
\\ \\ /
_ .._”. m
35:23.20 F V
50:0flwh 1' 55555 _. .0 li\
,, x... 5.8:me
,,/ ..... \Axx 9.65m \\,//
.o lll \ / /
,. F , “23 «V
/ / /
/ w / .\
/ 55555 .0
30.053.
\\/
55.505 mm o
.3993 .
8 .l\.
u 55m cozmiaoi P .o
F mcfimom
\
r...
. \\
o

.822 8.235 _eaofim 9.68m 82o 5.. $22 .32 m h 9...»...—

61

 

A m>m 0 V ...-l/
/ .. A x
/ v>m 0 V
o
\.. / ., ll . l o
A m>mm V Afr-nmzommm R-
.m, % m v20mlmmA
?
. a -l ... /.
VilmsammmA VA). AomihmV
4m, .. / A
,- - - --A A, .o 3
,. . ~=0E0>0Eu<
V 55.... V1, 9:50”. V9i
A \V . .
.m-ll. .._...o-l .- lV-
A. A
l R. .l-
A 3.2.2
A F @505. i052};
Vl-w l l_ l . +l
\F A WEommlmA _lrr
\.-..l-. \ l - l .._
.. <8.» 0V
A ~>mm V W A
/, .L/ l.
.m--\ A Saw V o
/. \\
p°llll

 

 

..\ll//
A950 0V
.0 \. 2.950.050 ..ll/
_mcofionE /
V .- l . -l l- A 9.5 V
V- Myers... - .... 85805 - fl...
A- .0508loolA l.W:melm/M F .O
l\ ..... ... cozmaawil l \\ of...
A. .. . . . 35.020 —. A 10V
\lklllr. :l 1 l -ll.- ,./- l\...
V -. ..-/ , co_.0l.0al0.n_ _O
mug—02n— ,/ .. mo_um_._a~om._ap_0 .. 5:03:05 ..ll /,, ,
.0508. .V9 A .0500... V Wc-fimmml —./,,.,..A. _m 0 V
Arc-l All-l. _ l lll.- —. -5l_.0.0la-0Ma ../,-l
V w / O xlmgoczooh , _O
/ ll - , l/
ll 1 , . A. , . \ ,
J k l ,
$5.2 a: 0 V
. 5am.lmnl0.u A .
A 5...... V .A .._...
l / . lll
a - l .
F\l/l/ F x,/// \.\l l/,
.\.~.\ // /A mm m V
Cam 0 V , .
/Il.l\\
..I//l\ .o
o

.282 8:50 __.safim ”scam 2.80 5.. .32 88 .3. 2.0....

62

 

 

 

 

 

 

 

 

 

 

 

 

 

 

000.0 000.0 0>5 V0000 000.0 0?.
000.0 30.0 0?. .000 000.0 «>5
0.00 30.0 0>5 V0000 .000 0>5
30.0 000.0 0>5 .000 $00 0?.
0.00 0000 $5 0000 0000 S5
3.00 000.0 000.0 000.0 8.000852%"! 3.8.5 000.0 000.0 000.0 0000 080.20.838.00 50:
0006 @006 03:05:. >099: 000.0 EmEmmmmmm (a0 mN_mchEm

0:0 00.0 80302850500 A0000 000.0 3.00 :00 000308 00.80.0080 505.
.0 00.0 000.0 000.0 803005000060: 000.0 000.0 0.00 .000 803088.05 08 890 50.2
0.0 000.0 3.0.0 09.0 000308. 5.0.00 000.0 000.0 «:0 80308 0.0.03 08 ___.0 50.2
000.0 0.0.0 3.0.0 8085.880» 000.0 000.0 000.0 000.0 8085.888
. 00...... 000.0 .009550855530560055 «00.0 02.6 50030 50602053050029...
owmd vmod 00.25.5205“.502000.95 ovwd mend 00.300.00.205“.5020005...

000.0 80.0 5.0; 0000.82 000.0 0.00 5.0; 0000.82
50.52050 50.20005... 50802050 50200055..

vwmd mam—305— mgmvcmum 93m

0000 003305. 00808.0 .252

000.0 .000 800.0 0000 0:0 .825

30.0 000.0 008.305. .88.: 0.0.0 0.00 00.0 0002305. 505.

000.0 000.0 000.0 .000 85 .8888... 000.0 0.0.0 85 20508004.
000.0 000.0 000.0 000.0 85 00.020 .0000 000.0 20.0 85 0.0820
000.0 000.0 2.0 000.0 85 .00. 0000505805 000.0 000.0 .000 000.0 85 0008005502
E0 0.0.0 85 306008.. .0000 .000 000.0 0.00 0000.085 502
20.0 0.00 0000885 0000020585 «000 000.0 0000 0000.800 5.0.5
02.0 30.0 0000 800.800 .88.: 80800 000.0 30.0 0000.800 080.00:
00_.0_.050.05 .0500 .. 005550.05 .0500...

0.0.0 .000 88880 205 0.00 000.0 8889.0 205

000.0 .000 £000 8.85 000.0 000.0 0.000 8.005

000.0 0000 8008.885 .038. 000.0 300 8008.885 .038.

0.2.0.055 .0500 .. m... 02.0.05... .0500 h m...—

000. 0000 0702.050 000. 0000 0.00.25.01.52
5022 822 z. .0 8.88> 5022 ..m<z z. B 028.8
.0. .00 03. 00. 0 .0080. «00 000. 03 00. 0 .0080

 

 

 

 

 

0.0m 0.00 000.30. 0.00.005U $500M 0:0 02.080502 5.. 0. .5 505950 ms 030“.

63

 

 

 

 

 

 

 

 

 

 

 

 

 

 

:3 :93... x x :«83 :33 x x x x x x 3.
23 :33 x x :«8... :33 x x x x x x c2...
:3 :33 x x :Nuo... :3... x x x x x x 9..
:3 :93... x x :33 :3... x x x x x x 9..
:3 :33 x x :33 :33 x x x x x x .2.
:23 :83 .83 83 :33 :83 S3 93 x x 23. .33. «co—55:33:30 5a:
.33 .33 x x 53383 <3 38:35
:33 .83 L... 3 63 .933 :33 :83 :23 x x 8.? .83- 838 aciaeaca £22
.33 :93 .83 $3 .53 :33 :53 :33 x x $3.? .333- .238 .86... 2a 963 522
.93. .83... $3 .23- :93 :33 x x 3.9 R3- 8:33 228.. as. ___5 £22
:23... :83 .23 83 .«33 :33 .33 :23 x x :3. .83. 8.88.. .283
.33 .. S3 .89... 638.9% 38.3.2...
:33... . $3 388.. £38.98 .2233...
:23 .23 £82 9.8.82
2.25.2260 33.829...
.383 $8.38. 3322... 29m
33 38.255. ”285.... .252
:83 .38... .685
.83 :23 .83 835. 522
:83 :83 am... 33332
:83 ..E3 :38... no... 2.29.0
EN... .33 :3 .33 am... 8.32.»... 5.22
.53 83 :23 2 3 5.5.32; 5a...
.83 : 53 .83 8.5.5.5 5.23
:33 .83 8.6.5.5 .2863
0030:3023? 52.085.
x x L3... :53 8:383 23.“.
x x .83 .393 3.30 semen.
x x 3.... .83 8.3.6.835 3.8....
8.8.8.2.. 358.. m:.
82 88 z» 5 82 88 z» 5 82 88 2h 5 33...;
5v $22 5.. am<z 5.. am<z 5v am<z 5v am<z 5v am<z
3m .3. «3. 8. 3m .3. «3. 8. v8 .3. «3. 8. c .288.
2.8.5293 33.392... 8.5.8.855 3:08.? 3:832... .283 w...

 

 

 

80» 8.0 30¢ 832.35.! .3 8085 .SOh ton—209.8” v.5 039—.

64

awed owed 39o Sta (mmzm
mafia 59° echo echo one.“
Bad 086 2.3. ‘86 £2
.59... 2.. and Ewomé ..zwoad $3.5
83 83 z... ._.o 03.32.“.
3?.

595.:

Eva:

no.1...

3:85.56

 

.536
toned
:avod
tamed
zoned

2.09.0
2.09.0
..oopd
2%ch
{med

XXXXX

o>n_
v>m
m>a
N>a
w>a

 

wood

13...:

x
x
x
x
x

.o

.2. v

N36

253233355 5...:

 

.0556
.850
.966

cmcmd
c.0006
CWNOAUI

{Numd
.owmd
:Nmmd

{wood
. and
2.986

208.333 <10 mummcaEw
3.3.6..” mcEmEomcmm 522
8:33 6&9.“ can 955 £32
852.8 2303 new ___5 5m:

 

3.895 .053...

 

63.5 E95355“. 55383..“
3:352. EoEno_o>oo _mcoamobea
583 9:25:

 

30:52:2— 35.389...

 

[$8.35 $35.». 29m
$8.265 «Ea—ES» 3.02
685m

38?ch 522

aoi .563me

85 £295

85 5.8:me 5w:
cozfimnEQ £22
5:855 E23.
8:2:ch _mcoaug.

 

8.538830 .553...

 

mocctmaxw 29...
2:30 E885
83:98.20 5.33

 

5:832.— .Euuo... mt.

 

 

8a..

#8

Doom

5v am<z Ev Qm<Z

0N3

z._.

N:

8.895 553%

._.0

wow

mflnwtm>

: .653»

 

 

 

43:8. in 93..»

65

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

.833 383 x x :23. :83? x x x x x x 3?.
.833 833 x x :83. :833. x x x x x x 3..
.833 383 x x :83. :833. x x x x x x 9..
.833 383 x x :23 :833. x x x x x x «>3
.833 833 x x :33? .883. x x x x x x .2.
.833 383 833. 5.3. :83. 833. «83 833 x x 833. 2.33 8.5362133 3538!
:383 ~83. 83- 533 x x 383 .83 288.35 892:
.383- :33 .83 83 x x 8338585388
.833- :23 .83 833 .53 .883 233- 833 x x 23 83 88858283:
.833. :83 :83 83 :83 :83 833- 833 x x 83 .33 8332. £383
-33. 833 .23 83 .333 :53 83. 333 x x 83 2.3 8.823358»
.83 :83 88.5 358.28 888.2“.
.956 Road 33.0360: «35:50.93. 395.390....
.83 .883 58; 38282
2036.260 .3232...
.383 .83 38$
.83 383 88.255. .622:
.. 5.3 :38... bar; 8383.?
:383 :83 32.. 585832
.83 .83 :83 :83 32.... 3.3.25
:83 .83 :383 :«33 gen. 88.52. 3:383”.
.23 .83 8i 8.253
.83 .83 3. 3 8.5. 83282“. 8.3323538".
:23 .P .3 .8 3 833380 399: 8:98
3.8.3.0235 .203...
x x .883 .83 8:888 22”.
x x :383 .383 3:80 5932..
x x .83 .883 83820280 .383...
3382.9... 32.30.? m...
83. 88 82 88 83. 88 8.88>
r... $3.53 $27 23 8 5.. am<z 53 8:2 2.. B 5.. 855.. has, 2» 8
8.. 83 «3. 8. E 83 «3 8. Bu ..8 «3. 8. c 8583
~§o_m>ou 3:28.291 8.5.3.0955 .Ozuao... Sagan—en. gawk wI.

 

 

 

88 88 823 33.38. .2 828m .82. 328:8» 3. .33...

66

mmcd
mmhd
mgd
t.wmm.m
can

mood Rod 05.0 (mmzm
v2.6 ..th Fund 038a
32. «wad m8... _m2
235°... ..zmhod :acmm...” 52.80
32. 2... #0 03.05:“.
3V...

33.3...

3.3..

no.3.

8:35.35

 

.

55d.
$3.0.
6:6.
.236.
.336.

:od
:od
:od
:od
:06

XXXX

x

XXXX

m>m
1?...
m>n_
N>n.
F>n_

 

394. 9.0.

«36

2.06:

8.58.5323 8.38..

 

:28...
{hand
:98...

..o 3.0
:and
.586

.ovmd

Land
Land

9.0.0

.3890
..N86

235.9: .682...
$3.28... 853500
8.3.83. 8:33.39...

mo...>..o< E83”.

 

3.88.. 3:03...

 

883E. €083.93 33.8335
.6538. E33293 83.882...
58.... 3.5.5.2

 

2053.260 8:22.82...

 

428$

88.255. 322...

.338) 23.833

30... 89:88.?

32a 2.995

32.. 5.62.8. 9:33:63”.
gen. 8.2.53

8...... 8.539.... 9.35:3”.
5.8.5.5 .899... 8.8.8

 

8.3.3.0223 3:03..

 

838% 2....
>530 .550...
8888920 2.88

 

5.8.2.2.. 3:03... m1.

 

 

89

he.

coom

no»

5% am<z 5v aw<2 2..

N3

3.88... 853..

._.0
8..

 

wflnaca>

c .383

 

 

A938. ufi 03....

67

Comparisons Across All Data Sets:

Connecticut and Tennessee results are based on correlation matrices developed
from identical variables and scales from the BTPS Questionnaire. Both use identical
models presented for mathematics (Figure 2) and reading (Figure 3). NAEP models, also
based on correlation matrices, vary from the Connecticut and Tennessee models as well
as each other and are shown in Figures 4-8.

Tables 7.3 through 7.5 compare r2 and direct and indirect effects from all SEM
analyses. Effects are standardized for comparison purposes although it is important to
note that variables/scales and their standard errors are not always the same across data
sets (see Tables 4.2-4.7). A discussion of the model confirmation, goodness of fit, and
standardized direct and indirect effects across all six data sets is included in this section.
Model Confirmation across data sets

Measurement model. As anticipated, model-fit improves with increased sample
size. The fit indices indicate that all models have good fit across data sets. In general, the
measurement models show that latent variables explain large amounts of variance (r2) in
indicator variables in Connecticut and Tennessee compared to moderate variance in
NAEP. The latent variable for teacher practice in mathematics explains comparable
variance across data sets (r2=.237 to .397). For reading, the results for teacher practice
are more varied (r2 =.144 to .937).

Structural model. The four latent variables performed reasonably well in the
structural portion of the model for in Connecticut and Tennessee and have already been
discussed in the preceding sections. The overall fit for the mathematics analyses based on

the conceptual model is good across data sets (cmin(x2)/df<1.463 to 4.962, p<.00;

68

GFI>.974 to .997; Pratio>.681 to .760). While RMSEA is slightly high for Connecticut
and Tennessee mathematics (.095 to .107), it is well within acceptable range for NAEP
mathematics (.028 to .048). It is possible that possible the latent variables for teacher
preparation and teacher characteristics are not measuring distinctly different constructs in
Connecticut and Tennessee. Teacher preparation programs are not measured in NAEP
and professional development is measured by a single exogenous variable rather than a
latent variable with three indicators.

The reading analyses had similar goodness of fit statistics (cmin(x2)/df <3.264 to
4.057, p<.000; GFI>.984 to .994; Pratio>.714 to .725) None of the reading analyses had
RMSEAs under .05 although NAEP data sets are close (.059). It is again possible that the
latent variables for teacher preparation and teacher characteristics for reading in
Connecticut and Tennessee do not measure separate constructs. Teacher preparation
programs are not measured in NAEP reading data and professional development is
measured by a single exogenous variable as in NAEP mathematics.

Direct Effects. As discussed earlier, the direct effects of teacher preparation on
teacher mathematics practice is very large and negative in Connecticut (5=-.660). In
Tennessee the effect is smaller but still negative (5=-.111). There is no variable for
teacher preparation in NAEP. The direct effects of teacher characteristics on teacher
mathematics practices are large and positive, exceeding 5=.421 for all data sets. The
direct effects of professional development on teacher mathematics practices are positive
and small to moderate (5=.118 to .339) across data sets. NAEP 1996 is the only data set
showing professional development effects on teacher mathematics practices in the

moderate range (5=.339).

69

The direct effect of teacher preparation on teacher reading practice is small and
positive in Connecticut (5:.274). In Tennessee the direct effect is moderate and positive
(5=.3 67). The direct effects of teacher characteristics on teacher practice in reading is
inconsistent across data sets. The effects are negligible to large as well as both negative
and positive depending on the data set. In Connecticut, the direct effect is negligible
(5=.010); in Tennessee, the direct effect is small and negative (5=-127); in NAEP 1998,
the effect is nearly perfect and positive (5:991); and in NAEP 2000, the effect is positive
and very large (5=.727).

The direct effect of professional development on teacher reading practice is also
mixed. At the state level, the effect is small and positive for both Connecticut (5:. 163)
and Tennessee (5=.118). At the national level using NAEP reading, the effects of
professional development are less than 5:05. The effect is negative in NAEP 1998 and
positive in NAEP 2000.

The direct effects of teacher practice on student mathematics achievement are
positive ranging from 5:.042 to .170. For reading, the direct effects of teacher practice
on achievement are mixed and range from 5=-.125 to .285.

Indirect effects. The indirect effects of teacher preparation, teacher
characteristics, and professional development on average student achievement in
mathematics and reading across data sets are small (5=-.123 to .087) but often significant
at 5>|.05|.

Summary of SEM Results
In addition to the stability of the hybrid SEM model across policy environments

and curriculum contexts, effect sizes are sufficiently varied to provide a venue for

70

comparing results across data sets.

1. The conceptual model (Figure 1) served well as a base for the SEM analyses.

2. It is possible to trace the pathways from teacher preparation, teacher
characteristics, and teacher professional development to teacher practice and subsequent
student achievement in mathematics and reading across multiple data sets. Many of the
direct and indirect effects on student achievement were significant at the predetermined
threshold of 5>.05.

3. It is possible to assess the relative contributions of teacher preparation,
teacher characteristics, and professional development to teacher practice and subsequent
student achievement. Of the variables measured in this data, teacher characteristics have
the largest indirect effects on teacher practice followed by professional development.
Teacher preparation played a larger role than professional development in Connecticut
and Tennessee but it is not possible to extend this analysis to NAEP. The only variable
modeled to have direct effects on student achievement was teacher practice.

a. Teacher preparation at the university level has large negative direct
effects on teacher mathematics practice in Connecticut and small negative direct
effects on teacher mathematics practice in Tennessee. The direct effects of teacher
preparation on teacher reading practice are positive and small in Connecticut and
positive and moderate in Tennessee.

The indirect effects of teacher preparation on student mathematics
achievement are very small and negative for both Connecticut and Tennessee.
The indirect effects of teacher preparation on average reading achievement in

Connecticut are positive and significant at 5>.05. In Tennessee the effect of

71

teacher preparation on average reading achievement is negligible.

Connections among teacher preparation, teacher practice, and student
achievement are similar for mathematics in both states. The negative relationships
among the latent variables in Connecticut are larger than those in Tennessee but
both ultimately have negative effects on student mathematics achievement. The
effects of the latent variables for reading vary by state but the consistently positive
connections in Connecticut are worth noting.

b. Teacher characteristics have mostly large, positive direct effects on
teacher practices employed in the classroom for both mathematics and reading
across data sets. The indirect effects of teacher characteristics on mathematics are
small although positive across data sets. The indirect effects of teacher
characteristics on student reading achievement are inconsistent and negligible
(5<.01) in both Connecticut and Tennessee. For NAEP reading 1998 and 2000,
the indirect effects of teacher characteristics on average achievement are negative
and exceed 5>.05. The links among teacher characteristics, teacher practices, and
student achievement work similarly for mathematics across state and national
contexts (the same relative size and direction) but vary across those same contexts
for reading.

c. Professional development has small to moderate, positive direct effects
on teacher mathematics practices across all data. Professional development also
has small, positive direct effects on reading practices in Connecticut and
Tennessee. These effects are smaller and, in one case, negative in NAEP. In

general, professional development has positive but very little indirect effect on

72

average student achievement. The strongest effects of professional development

on teacher practices and student achievement occur in reading in Connecticut

(5=.047).

d. Teacher practice has positive, direct effects on average student
mathematics achievement across all data. The results for reading are mixed.
Teacher reading practices in Connecticut have a positive, direct effect (5=.285)
while teacher reading practices in NAEP 1998 have a negative direct effect (5:-
.125) on average student reading achievement.

4. Curriculum context makes a difference. Mathematics, for the most part, has a
connected structure where each element is consistently related throughout the system and
across data sets. With the exception of teacher preparation, the effects are positive and
related to increases in average student achievement. When it comes to reading, the
structure does not work in the same way across data sets. Connecticut is the only data set
with consistent positive effects throughout the system related to increases in average
student reading gains.

5. Policy contexts are difficult to discuss in more than speculative fashion.
Changes in policy may or may not be related to differences in results across policy

environments. Variables to test such hypotheses do not exist in this data.

73

8. RESULTS FROM THE HLM ANALYSIS OF
CONNECTICUT 4th-sth GRADE MATHEMATICS GAINS

The HLM analysis is intended to address the research questions about the

relationships among student achievement and

0 Teacher characteristics and practices

0 Characteristics of professional development

' Characteristics of teacher preparation
The analysis also is intended to address the ways in which these relationships vary by
student demographic group, including gender, ethnicity, and grade. Tennessee is not
included in the HLM analyses because there is no individual measure of student
achievement available in the BTPS data, only correlation coefficients based on
aggregation of student scores within teacher.

To set the context for addressing these questions, results are presented first which
describe the differences in student achievement (actually achievement gains in CT,
achievement status in NAEP data7) across student demographic groups. The analyses
yield results in terms of regression coefficients. To aid in understanding what the
coefficients mean, predicted achievement (gain over one year) is computed for selected
combinations of student characteristics.

The district percent free and reduced lunch8 was assigned to each teacher as a

 

7 Nye, Konstantopoulos, & Hedges {Nye, 2004 #402} use “gains” to indicate the
difference between pre- and post-test scores and “achievement status” to indicate a one-
time administration score such as NAEP (p. 17).

8 1999-2000 district data from Connecticut Department of Education website.

74

measure of classroom socioeconomic status since there was no individual student
indicator available in the BTPS data.9

The full set of mathematics variables for CT is given in Table 8.1.10 These are the
same variables used in the earlier structural equation models.

Table 8.1 Description of Variables Used in the Connecticut Mathematics Analyses.
Type Variable Variable Description
Abbreviation 7

Level 1 - students

Continuous mdiff Mathematics gain, difference between pre- and post—test
.S9‘it..°°_m"_) _ __. _ _ _ _ _ ic‘lies.('1‘_e?1‘=i"3_°: 3919-73) ___________
Dichotomous dblack Black (Black=1, else=0; mean=.14, sd=.34)
dhispanic Hispanic (Hispanic=1, else=0; mean=.09, sd=.29)
dallother All other ethnicities (all other=1, else=0; mean=.09, sd=.29)
dfem Female (female=1, else=0; mean=.52, sd=.50)
dgrade4 Grade 4 (grade 4:1, else=0; mean=2.30, sd=1.98)
. ______ dress- _grassreszsassflnsa:228252-142.___
Effects coded egenxbl Effect of Black female; interaction of eBlack x egender

where Black =1, White=-1, otherwise=0 multiplied by
female=1, male=-1 (mean=.01, sd=.90)

egenxhisp Effect of Hispanic female; interaction of eHispanic x
egender where Hispanic=1, White=-1, otherwise=0
multiplied by female=1, male=-1 (mean=.01, sd=.88)

egenxoth Effect of All other female; interaction of eAllother x egender
where All other =1, White=-1, otherwise=0 multiplied by
female=1, male=-1 (mean=.01, sd=.88)

 

9 This causes some disaggregation bias since the schools in which teachers teach
are likely to vary within districts on this variable and students are likely to vary on SES
within teachers. Bias is less problematic, however, than leaving out an indicator of SES
for these analyses.

10All variables used in the SEM analysis are listed here and all were tested.
Results for teacher level variables that did not explain variance in average student gains
across teachers are not reported in all sections.

75

Table 8.1 (cont'd).

Level 2 - teachers

Continuous Z-scores Sum of interview responses rescaled to have mean=0, sd=1
Teacher zmthgpa Frequency of group and project activities
Practices
zmthdla Frequency of drill and lecture activities
zmthsmkg Frequency of sensemaking activities
Teacher zmoref Teacher's reform orientation
Characteristics
zmthknow Self report of mathematics knowledge
ztrad Teacher’s traditional orientation
zmathprep Major/coursework/license in mathematics, mathematics
educafion
zmathinst Preparation for mathematics instruction, pedagogical
coursework
zdivprep Preparation for diversity
zefficacy Teacher’s sense of efficacy, that a teacher can make a
difference in students’ learning
zassessprep Preparation for assessment
Professional zpdi professional development impact (self report)
Development
zpdf professional development frequency
zmentfrq mentoring frequency
Continuous pctfrlu Percent district free and reduced lunch (mean=.21,sd=.16)

1. Setting the Stage: Variation in Mathematics Achievement Gains by Student
Demographic Groups and Grade Level (Results at the Student Level)

In this section, sets of background variables are added to look at changes in
mathematics gains in Connecticut. Model 1 predicts mean student gains across the
sample and partitions the variance between students and teachers where students are
nested within teachers. Models 2-5 predict differences in mean gains across gender,

ethnicity, and grade.

76

Mean Gains. To set the context for answering the questions about connections
between student achievement and teacher variables, results are presented first for
differences in student achievement gains across student demographic groups. The level
one and level two equations for the model are given followed by their estimated
regression coefficients in Tables 8.2.

Model I. Mean Gains
Level 1 - Students
Yij:fl0j+eij (8-1)
Level 2 — Teachers
:60j = Too T r0j
where Y ij is the mathematics gain for student i in teacher j ’s classroom, 1603' is the

intercept or mean students’ mathematics gain within teacher j’s classroom, e is the level

ij

1 error, 700 is the mean value of classroom gains for all teachers, and ’Oj is the level 2

error.

Table 8.2 Connecticut’s Average Mathematics Gain 1999-2000

I Mode|1 *
”.‘mérée‘pinooy * i '4'.85(.40)*** *
”(VET/Tana}; (02) if F 35.8? 8
NE E Qgrignéehoo) 8 8.42m

“* p<.001, ** p<.01, * p<.05, ~ p<.10

The predicted average gain score for 4th-8th grade mathematics students for this sample of
students in Connecticut is 4.85 points (y00=4.85, sd=.40) when no other variables are
included in the model.

2

O and 1:00 are used to compare variation in the errors at the teacher level with -

77

variation at the student level using the formulas:

0'2 2
. r
——————- = level one variance —T—()L

2 = level two variance (8.2)

91% of the variance in gains for Model 1 is within teachers (between students) leaving
9% of the variance in gains between teachers. That is to say, there is ten time more
variation within teachers than between teachers.

Mean Gains by Demographic Group. Next, demographic variables are added to
form Models 2, 3 and 4. Model 5 introduces the control variables for grade level. The
equations for Model 2, ethnic group, are presented first (Eq.8.3) To interpret the
coefficients, the estimated gain for ethnicity can be computed by using the appropriate set
of ls and 05 for the dummy variables. As an example, Equations 8.4 and 8.5 demonstrate
the computation for obtaining Black student gains. Estimated gains by ethnicity are
presented in Table 8.3.11
Model 2. Ethnic Groups
Level 1- Students
Y ij = IBOj + flljdblackij+ fl2jdhispaniclj+ ,33jdallotherij eI-j;

Level 2 - Teachers (8.3)

1601: 700+ ’Oj

 

11 Note that the category for All other includes Asian and those students who did
not report ethnicity. Often these smaller groups are folded into the intercept since they
are not significantly different from the intercept. In this case, the coefficient for all other
was nearly significant at p<.053. Additionally, it was important to maintain the meaning
of the intercept as a specific group, in this case white students, before analyzing the
differential effects of teacher variables.

78

161} = 710
162} = 720
163} = 730
The regression coefficients for predicting student achievement gains from ethnic
background are found in Table 8.3.

Table 8.3 Regression Coefficients for Connecticut Mathematics Gain 1999-2000 by

Ethnicity

ff” % * * mam *’

Intercept (v00) 4.05(.42)***

WHEE’E'I‘JCXVozl , 2890-04)”
All other (Vo3)_ , 7 , W W 1.53(.80)~ 7

IVOVérianceloZ) * 8 85.32

VLVL2Variance (TOOL ,, s, ”7.64“?

*** p<.001, ** p<.01, " p<.05, ~ p<.10

To make it easier to interpret the results from the gains by ethnicity group
analysis, these coefficients can be used to compute the gain for a student with a given set
of demographic characteristics. Because the demographic characteristics are coded as 1 or
0, the computation involves summing the terms which are coded l for a particular
student.12 For example, to determine the predicted gains for a black student who would
have ones for the intercept and black the resulting equation would be:

Level 1 - Students

vi]. = p], 1+1}, jdblackij (8.4)

12 A table showing ones and zeros for all dummy variables is provided in the
Appendix, p. 295.

79

Level 2 - Teachers . .
[301' = 700 = 4-05

hj=no=263

When the betas from Equation 8.4, are substituted in the level one equation (Eq. 8.4), the
result for black gain is produced (Eq. 8.5).

Ii). =(4.05)+(2.63)*1 (8.5)

= 6.68

Predicted mathematics gains for all demographic groups are presented in Table 8.4. This
table shows that the model predicts that all non-white student groups gain more in
mathematics than white students. The coefficients for White, Black, and Hispanic
students are significantly different from zero. The coefficient for all other is nearly
significant at p<.053.

Table 8.4 Predicted Mathematics Gains by Student Demographic Group

 

 

Ethnic Group Gains
White 4.05
Black 6.68
Hispanic 6.94
All other 5.58

 

In the next model, gender predicts average mathematics gains.

Model 3. Gender

Level 1- Students
Yij = 1601”” '641 dfemij + eij;

Level 2 - Teachers (8.6)
1601' = Too T rOj

.641“ = 710

80

The regression coefficients for predicting student achievement gains from gender
are found in Table 8.5.

Table 8.5 Regression Coefficients for Connecticut 4th Grade Mathematics Gain 1999-

2000 by Gender

F G G Mode|3

Intercept (V00) , , 44.84(.51)***
Female (v04) .03(.52)

EVE (Variafncveuiz) if i 8 85.93

LVL2Variance (To ) 7 8.42“” 7

p<.001, ** p<.01. * p<-05. ~ P<-10

These coefficients indicate that male mean gains in mathematics are predicted to be 4.84
points compared to 4.87 for females. The coefficient for female is not significantly
different from zero.
Model 4. Combined Gender and Ethnicity

Model 4 tests whether or not there is an interaction between gender and ethnicity.
Equation 8.7 computes coefficients for combinations of gender and ethnicity in CT.
Effects coding for gender x ethnicity produces the same fixed effects as dummy coding.13
Level 1- Students

Y ij = flOj + flljdblackiJ-Jr flzjdhispaniclj+ fl3jdallotherij+ ,64j dfeml-j

+ flsjegenxblij+ fl6jegenxhispij+ ,B7jegenxothij + e (8.7)

if

Level 2 - Teachers

:60} = 700 + rOj

 

13 When interactions are effects—coded, it facilitates statistical testing of the
differences by gender by ethnicity variables found in Section 2. (Anderson, p.22, web
notes)

81

.51} = 710

:62; = 720
.631' = 730
I64} = V40
:35} = 750
fig = 760
.67} = 770

The regression coefficients for predicting student achievement gains from ethnic
background and gender are found in Table 8.6.

Table 8.6 Regression Coefficients for Connecticut Mathematics Gain 1999-2000 by
Gender and Ethnicity

hllodel4 *
TntetrEeEtlyoo) 7 7 ' antes)“ 7'
Three} (yo—1)" _ ' ' “263—(1083*
”Rigging? if i if 3.76(1:09): *
”flAflitBturierfiwog) .1.49(.84)~
”’an’a‘ie 064; i i * 166(89) if

Effect Black x female(v05) -.35(.63)

Effect Hispanicxfemale (V06) .93(.71) ,
‘ Effect All other xA female (v07) -.02(.70)

LVL 1 Variance (02) 585.35

LVL 2 Varian~ce(T00) f 7.70““

*“ p<.001, ** p<.01, * p<.05, ~ p<.10

To make it easier to interpret the results from the gains by demographic group
analysis, these coefficients can be used to compute the gain for a student with a given set
of demographic characteristics. Because the demographic characteristics are coded as 1 or

0, the computation involves summing the terms which are coded l for a particular

82

student. 14 For example, to determine the predicted gains for a black female student who
would have ones for the intercept, black, female and black female, the resulting equation
would be:

Level 1 - Students

Ylj = flOj + BljdblaCkU- + B4jdfemij + B5jeblackxefemij

Level 2 - Teachers (8.8)

do j = )700 = 3-72

A,- =Ao = 2.63
1341' =f40 = ~66
1851' =f50 =-°35

When the level two betas from Equation 8.8, are substituted in the level one equation (Eq.
8.8), the result for black female gain is produced (Eq. 8.9).
17,]- = (3.72) + (2.63)*1 + (0.66)*l + (-o.35)*1 (8.9)
= 6.66
Predicted mathematics gains for all subsets of students by demographic groups are
presented in Table 8.6. The table shows that the model predicts all female students are

predicted to gain more than their male counterparts.

 

14 A table showing ones and zeros for all dummy variables is provided in the
Appendix.

83

Table 8.7 Predicted Mathematics Gains by Student Demographic Group with Differences

 

 

by Gender
Males Females Difference in Gains
by Gender
White 3.72 4.38 + .66
Black 6.35 6.66 + .31
Hispanic 6.48 8.07 +2.52
All other 5.21 5.85 + .62

 

Score differences for males and females are predicted to be less than 1 point for White,
Black, and All other students. Hispanic females are estimated to gain 2.52 points more
than Hispanic males. Coefficients for gender x ethnicity effects are not significantly
different from zero over the course of one year.
Gender, Ethnicity, and Grade Level

Another known source of variation in student achievement is grade level. In the
earlier SEM analyses, the outcome variable for gains was standardized by grade level. In
the upcoming NAEP analyses, only one grade is represented in each analyses.
Consequently, dummy variables for grade level are introduced at level one in the
Connecticut analyses to control for grade level. The reference category for Model 5 is
therefore white male students in 8th grade.
Mode15. Gender, Ethnicity, and Grade Level
Level 1- Students

Y ij = 1601' + flljdblackij+ flzjdhispanicljwL fl3jdallotherij+ fl4j dfemij (8.10)

+ flsjegenxblij+ fléjegenxhispij+ ,B7jegenxothij

84

Level 2 - Teachers

:60} = 7’00 + "Oj

:61} = 710
.52]: = 720
:63} = 730
:64} = 740
:65} = 750
’36)“ = 760
571' = 770
.58} = 770
:69) = 7’70

The regression coefficients for predicting student achievement gains from ethnic

background, gender, and grade level are found in Table 8.8.

85

Table 8.8 Regression Coefficients for Connecticut Mathematics Gain 1999-2000 by
Gender, Ethnicity, Grade

. a v 7 7 7 _ - Miode|4
Intercept (yoo) 0.6(. 98)
Black (V01) 2 36(1. 09)*
Hispanic (v02) 2.29(1.07)*
All other (y03) 1.44(.80)~
Female (v04) 8 if i f f" 64(.88)

Effect Black x female (v05) - .33(. 62)

Effect Hispanic x female (V06) 1. 00(. 70)

Effect All other x female (v07) - .10(. 69)

.fitth'a‘ieWoal: - e- e- - 9929*: -
_,,9E*I9t999<vtt9> - e 9919?"
LVL~1VaTiahce(02) . 7 8528
-Lyezyarianee “00’ e 911*“

p<.001, ** p<.01, * p<.05, ~ p<.10

As demonstrated in prior sections, the computational example to determine the
predicted gains for a black female student in the 6th grade who would have ones for the
intercept, black, female, black female, and grade 6, would be:

Level 1 - Students

v” = [30]. + Bljdblackij + B4jdfem,-j + BSjegenxblij + ngdgrade6

Level 2 - Teachers (8.11)
.50 j =7A00 =06
[Bi] =730 - 2 36
5:1] 3’40 - 64
:33, =7}0 - — 33
:59, =J’A90 - 94

When the betas from Level 2, Equation 8.11, are substituted in the Level 1 equation, the
result for black female gain is produced (Eq. 8.12).
9,, = (.06) + (2.36)*1 + (O.64)*1 + (-o.33)*1 + (94*1) (8.12)
= 3.67

Predicted mathematics gains for all subsets of students by demographic group and
grade are presented in Table 8.9. The estimated gains associated with the 4th and 6th
grade years average just under one point each and are the largest, significant predictor of
gains for all students. Note that estimated gains during the eighth grade year are
considerably smaller than gains made at Grades 4 and 6 for all ethnicities.

Table 8.9 Predicted Mathematics Gains by Student Demographic Group and Grade

 

Grade 4 (+92) Grade 6 (+94) Grade 8

 

Males White .98 1 .00 .06
Black 3.34 3.36 2.42
Hispanic 3.27 3.29 2.35
All other 2.42 2.44 1.50
Females White 1 .62 1.06 .70
Black 3.65 3.67 2.73
Hispanic 4.91 4.92 3.99
All other 2.96 2.98 2.04

 

The table also indicates that the model predicts that all non-white student groups gain
more in mathematics than white students once ethnicity, gender, and grade level are taken
into consideration. Hispanic students show the largest differences by gender with girls
predicted to gain 1.64 points more than boys; however, coefficients for dfemale and the
interaction terms (gender x ethnicity) are non-significant.

firmmarv of Section 1. Developing the level one equation for students. As the
level one base equation for mathematics in Connecticut developed, average yearly gains

were first predicted with a simple unconditional model. This model indicated that gain

87

variance was ten times larger within teachers (between students) than across teachers at
level 2. Variables known to affect mean gains (ethnicity, gender, and grade) were added
to the level one equation revealing that while differences in gains exist within teachers,
only the coefficients for ethnicity and grade level are significant predictors of yearly
mathematics gains when gender is controlled. For the Connecticut mathematics data, all
groups in all grades are expected to have higher average gains than White students.

11. Effects of Teacher Practices, Teacher Characteristics, and Professional
Development on Student Achievement Gains (Results at the teacher level)

This section addresses the questions:

1. Does mathematics achievement gains for demographic groups vary across
teachers?
2. How much variation in student mathematics gains do variables about

teacher practices, teacher characteristics, and professional development
explain?

3. Is the amount of variance explained by teacher characteristics and

practices different for different student demographic groups?

The analyses in this section first test if the achievement gains for the various
demographic groups vary across teachers as the unit of analysis shifts from students to
teachers. There is a brief discussion of those demographic group gains which vary across
teachers followed by changes to the results when district SES is controlled. This
establishes the level two context for a series of analyses which look at the effects of
teacher practice, teacher characteristics, and professional development on the variance of

achievement gains by demographic groups.

88

Variance in Connecticut mathematics achievement across teachers

In order to determine if there is significant variance in mathematics achievement
by demographic groups across teachers, it is important to understand how significant
variance is indicated and its impact on the level two equation. When main effects
variables are dummy coded, not only does [30 represent the mean for white male
achievement, but the variance of [30, known as 1700, is the variance of white male
achievement (see Raudenbush, p.34). Nonsignificant error terms for variance (r00, r10,
...rx0) as determined by X2 indicate that group mean achievement does not vary across
teachers so the error terms for those demographic groups may be dropped from the
model. Although the nonsignificant error terms are dropped, the fixed level two equations
for those demographic groups remain in the analysis.

Analyses indicated that average gains for female and Black varied significantly
across teachers (Ha: 100990). Black females, Hispanic males and females, as well as All
other males and females, did not vary significantly from the intercept (H: 17_.0=0) so their
error terms have been eliminated. Their variance is then captured as part of the error term
for the intercept (r00). In other words, the variance is no different from that of White.

As teacher variables are added at Level 2, different proportions of variance are
explained depending on the student demographic group whose mean achievement is
being predicted. Each group with significant variance (Female, Black) in its gains across
teachers is addressed below in a series of separate analyses which include teacher
practices, teacher characteristics, and teacher professional development. The results from
the analysis of student gains by gender across teachers are followed by the results from

the analysis of gains by White and Black students across teachers.

89

Effects of teacher level variables on mean gains by gender

While the fixed effects model for gender, Model 3, predicted average gain scores,
a random slopes and intercepts model, Model 6, is necessary to test whether or not the
effects of male and female vary significantly across teachers. If they vary, do teacher
practices, characteristics, or professional development have differential effects on the
average mathematics gains for one gender as opposed to the other? If variance is
significant across teachers, it is then possible to determine how much of the variation can
be explained by a specific level two variables.
Model 6. Unconditional Model for Gender
Level 1- Students

Y ij = 160j + flljdblackij+ flzjdhispaniclj+ fl3jdallotherij+ ,B4J- dfeml-j (8.13)

+flgjdgrade4ij +fl9jdgrade6ij + el-j;

Level 2 - Teachers

:60) = 700 + "0j (8-14)
:61} = 710
:32) = 720
1531' = 730

54):: 740 + "4j

:68j = Y70
:39)‘ = 1’70
where
YOOj = intercept, average student gains across teachers
roj = the error term or unique teacher effects

90

710’ = intercept, average Black student gains across teachers
J

YZOj = intercept, average Hispanic student gains across teachers
Y30j = intercept, average All other student gains across teachers
Y40j = intercept, average female student gains across teachers
r4]. = the error term or unique teacher effects

Ygoj : intercept, average grade 4 gains across teachers

Y90j = intercept, average grade 6 gains across teachers

The estimated level two regression coefficients for Model 6 are given in Table

8.10.
Table 8.10 Regression Coefficients for Connecticut Mathematics Gain 1999-2000 by
Gender

FT 8 i F i iModel3 W
Hint'erciepmbo) * A if f f if '.50(1.04)

Black (v10) 2.46(1.06)*
mitéiiaiicwgo) * V i i i * 263(99)‘:

All other (v30) 1.50(.76)*
flréfiaiemof i f i if ’-.08(.54) I
“‘“CFaBeTwsvm * i * 188(29):"
1199939 Eva: * 9.9-<99“ .
LVL1Vafiath(CZ) * i 8 84.04
'Lvrfzflifitémcétiddo) 8.33m

Female (T40) 509*

p<.001, ** p<.01, * p<.05. ~ p<.10

The effect of female varies significantly across teachers (140:5.09, p<.05) as does
the effect of male (1:00=9.33, p<.001). Coefficients from this analysis serve as the base for
the variance decompositions for gender found in Tables 8.13-8.16. Variance

decompositions systematically present the amount of variance explained by each

91

additional variable included at level two.

Model 7. SES and Gender

Level 1- Students - Same as Equation 8.13

Level 2 - Teachers

where

Y00j

You

roj

Y41j

I'4j

.30]- : 700 + Y0] pctfrlu + r0]. (8.16)
.61): = I’m
.82) = 720
1331' = 730

:B4j = 740 + Y41P6’frlu + r4}
.68) = 770
59' = 770

= intercept, average student gains across teachers

= the change in average gains associated with a one unit increase in district
free and reduced lunch

= the error term or unique teacher effects

= the change in average female student gains associated with one unit
increase in average district free and reduced lunch

= the error term or unique teacher effects

The estimated level two regression coefficients and variance components for

Model 6 are given in Table 8.11.

92

Table 8.11 Regression Coefficients for Average Gains by Gender Predicted by Average

District SES
T if 77 Gender
880.3000) 8 i * i.33(1.06) *
Free and reduced lunch (V01) -6.44(3.94)
81a§<(-y1gf A i '8‘2.59(1.07)*
"f fistianicwzo) * f i * I2.77(1.06)*
_I'Alldthgrhgof) * i * i1.46(.77)~
“_Ee—rnahug) ' I h i if 8 903(54) 8
_ ” ‘Ffeeand reduced lunch(v41) * '7.‘90(4.25)~
fléfacélwgof T A F I V.90(.29)**
'“GTadeéwgw ' .94(.20)***
LVL? VafianCe(02) l 884.02
LVL2 VErianSe (T00) * i i 8.76“” *
___ “1740) -, , ,, - 391* ,

*** p<.001, ** p<.01, * p<.05, ~ p<.10

Table 8.11 shows that being in a district with average SES (21% free and reduced
lunch) has a negative effect on boys’ average mathematics gains across teachers (Y02 =
-6.44, sd=3.94) and a positive effect on girls’ average gains across teachers (y 12 = 7.90,
sd=4.25,p<.066). There are two districts in this data that have over 50% (+2 standard
deviations) free and reduced lunch. Boys would be expected to lose -12.5 points (Yij='33
+2 x (-6.44)) while girls would be expected to gain 3.6 points (Yij=-12.5 +.33 + (-.03) +
2*(7.9) = 16.1) in those same districts over the course of the year.

How much do the average gains for males and females vary across average SES
districts? To answer this question, the following equation calculates the amount of

variance explained by average district SES across teachers. 15 For example, to find the

 

15 A three-level analysis, students nested within teachers nested within districts,
was conducted for the ten Connecticut districts that were represented by 5 or more

93

variance in female gains explained by district SES, substitute the estimates for 1:40 from

Table 8.12 into Equation 8.18.

 

T400199991 6) ‘ T40(m"de' 7) = 199419}. = .232 = 23.2% (8.17)
140 (model 6) 5.09

This means that 23.2% of the variance in the average female gains is explained by district
SES across teachers. Using the same process for 1:00, the results are presented at the
bottom of Tables 8.12.-8.15.

Table 8.12 Variance Decomposition for Gender by District SES

 

 

 

 

 

 

SES
Variables (y01 & v11) Uncond F ree/Red
Lunch

Intercept (v00) .50(1.04) O.33(1.06)
SES(v02) —6.44(3.94)
Female (v10) -.08(.54) -.03(.54)
SES(v12) 7.90(4.25)~
Intercept Variance (T00) 9.33“" 876*“
Dfem Variance (T40) 509* 3.911
(02) 84.04 84.02
% variance reduction male 6.1%
% variance reduction female 23.2%

 

 

 

 

“* p<.001, ** p<.01, * p<.05, ~ p<.10

One standard deviation increase in district free and reduced lunch accounts for a -
6.44 point drop in mathematics gains for boys. In the same district, a one standard
deviation increase in district free and reduced lunch signals an increase of +7.90 points
for girls. The effect of being in an average SES district is the opposite for boys and girls

across teachers. SES, therefore, has a differential effect on boys and girls. Although

 

teachers. There was no significant variance across districts for this sample so it was
determined that the two-level model controlling for SES was the most appropriate.

94

there is more variance in average achievement gains for boys across teachers, SES
accounts for only a small percentage of the variance in boys gains (6.1%) compared to
girls average gains (23.2%) across teachers.
Gender and Teacher Effects

Now that the context is set at level two, it is time to determine the effects of
teacher variables on the variance of average gains for boys and girls across teachers. The
next analyses track variance explained by teacher level variables controlling for SES and
also determine whether or not particular teacher variables explain different proportions of
variance in the average achievement gains for males and females. One teacher variable at
a time is added to the level two equation, EQ 8.16, forming an equation template (Eq.
8.18). The term “variable x” is exchanged for a single teacher variable for Mathematics
Practice (math sensemaking, drill and lecture, or group and project activities), Teacher
Characteristic (mathematics preparation, preparation for mathematics instruction,
mathematics knowledge, reform and traditional orientation, assessment preparation, or
efficacy), or Professional Development (professional development impact, mentoring
frequency, or professional development frequency).
Model 8. Template for Level 2 equations

Level 2 - Teachers

fig]. = You + 701.0009,“ + ya?” variable x + r0]- (8.18)
:62} = 720
1621' = 720
331' = 7’30

fl4j = 740 + n, pctfrlu + )142” variable x + r41-

95

1651' = 1’50

fl6j = 1’60
.37)- : 1’70
.68)‘ = 1’70
.59)“ = 1’70

where
7024' = change in average in male student gains associated with a one unit
change in “variable x” holding SES constant at the district average
and
Y42..j = change in average female student gains associated with one unit change
in “variable x” holding SES constant at the district average
Gender a_nd Teac_her Pmctices. The variance decomposition for teacher
practices is located in Table 8.14. Teacher practices explain more of the variation across

teachers in average gains for girls than boys in average SES districts.

96

Table 8.13 Variance Decomposition for Gender Controlling SES - Teacher Practices

 

 

 

 

 

 

 

 

 

 

SES Practice

Variables (v01 8 VII) Uncond Free/Red Sensemaking iDrill8LectureiGroup&Project

Lunch l I
Intercept (v00) .50(104) O.33(1.06) 0 40(1. 07)I 0. 33(1. 07)i 0.34(1.06)
(v01) 0. 70( 45)I -0. 30(. 48h 0.52(.49)
SES(v02) —6.44(3.94) -6. 53(3 94): -6. 57(3. .96)I -6.18(3.98)
Dfem (v10) -.08(.54) -.03(.54) 0. 00(. 53): -0. 04(. .53)I 0.01(.53)
(v11) -0.98(.48)*I 0. 60(. .53)I -1.06(.59)~
SES(v12) 7.90(4.25)~ 8.O4(4.26)~I 8.16(4.26)~I 7.58(4.39)~
Intercept Variance (T00) 933*“ 876*“ 8.54***I 8.80***I 8.53“
Dfem Variance (T40) 5.09* 3.91' 360* 3.37‘I 3.18~
(02) 84.04 84.02 83.93, 84.15I 84.10
% variance reduction male 6.1% 8.5%' 5.7%I 8.6"/
% variance reduction female 23.2% 29.3%I 33.8%I 37.59]

 

“* p<.001, ** p<.01, * p<.05, ~ p<.10

Variance in boys’ gains explained by math practice variables across teachers

controlling for average SES ranges from 5.7% to 8.6%. Although variation in teacher

practices across teachers explains up to 8.6% of the variance in gains for boys, it explains

23%-38% of the variance in girls’ gains controlling for SES. Variance explained by

mathematics sensemaking activities across teachers adds a small amount to the variance

already explained by SES in the average achievement gains for boys (a little over 2%

more) while it explains an additional 6% of the variation in girls’ gains across teachers.

The change attributable to one standard deviation increase in frequency of a particular

mathematics practice is the opposite for boys and girls in this data. Math sensemaking

and group and project activities decrease female gains by approximately one point (Y1 I:-

.98 and 71 1: -1.06, respectively) while male gains increase by about one half of a point

(YOI=.70 and y01=.53, respectively). Drill and lecture activities increase female gains

('y1 1=.60) while decreasing average gains for males (701: -.30). Teacher mathematics

97

practices have differential effects on gains by gender.

Gender and Teacher Characteristics. The variance decomposition for teacher
characteristics (Table 8.14) answers questions regarding how much of the variance in
average gains for boys and girls can be explained by specific teacher characteristics
controlling for SES across teachers. While there are small percentages of variance
explained by teacher characteristics across teachers for boys, up to 8 times as much
variance is explained for girls by the same teacher characteristics. With the exceptions of
math preparation, which is negatively related to average gains, and mathematics
knowledge, which is near zero for both genders, the effect of teacher characteristics on
the average gains of boys and girls across teachers is differential (in the opposite
direction) once SES is controlled.

Table 8.14 Variance Decomposition by Gender for Teacher Characteristics Controlling

 

 

 

 

 

 

 

 

 

 

 

for SES
SES Teacher Characteristics

Variables (V01 8 VII) Uncond Free/Red Lu MthPrep pl/lthlnst lMthKnow I
Intercept (v00) 50(1.04) 0.33(1.06) 1.51(1.10)I ..30(1 08)I 0. 05(1. .11)I
(v01) -O.86(.46)~I .43(. 45)I -0. O1(. .47)I
SES(v02) -6.44(3.94) -7.12(3.86) -5. 98(3. 99)I -6. 36(4. 02‘
lDfem (on) -.08(.54) -.03(.54) 0.03(.54)I -.02(.54)I -0. 03(. .53)I
(v11) -O.18(.45)I -.43(.71)I 0.89(.44)*I
SES(v12) 7.90(4.25)~ 7.60(4.21)~I7.49(4.45)~I8.36(4.22)]
Intercept Var (T00) 9.33‘“ 876*" 8.83***I 8.84***I 9.07***I
Dfem Variance (T40) 509* 3.91" 4.40* 4.32* 397*
(02) 84.04 84.02 84.00, 83.96I 83.88I
% variance reduction male 6.1% 5.4%I 5.3%I 2.8%I
% variance reduction female 23.2% 13.6%I 15.1%I 22.00%I

 

 

98

Table 8.14 (cont'd).

 

 

 

 

 

 

 

 

 

SES Teacher Characteristics

Variables (v01 8 VII) Uncond Free/Red Lu Reform Or Trad Or Assess Efficacy
Ilntercept (v00) 50(1.04) 0.33(1.06) 0.33(1.07I' .32(1.07) 0.30(1.06)I 0.38(1.07)
(v01) 0.58(.44)I .25(.48) 0.29(.38)I 0.56(.43)
SES(v02) -6.44(3.94) -6.17(3.83I -6.70(3.92)~ -6.22(3.92)I -5.85(3.91)
IDfem (v10) -.08(.54) -.03(.54) -0.02(.53)I -.02(.54) -.01(.53)I -0.02(.52)
(v11) -0.70(.47I -.29(.59) -0.73(.51)I -1.11(.54)*
SES(v12) 7.90(4.25)~ 7.66(4.17)~I 8.19(4.13)* 7.27(4.32)~, 6.78(4.22)
Intercept Var (T00) 933*“ 876*" 8.73“, 8.97“" 8.71“" 8.59“”
Dfem Variance (T40) 509* 3.91“ 3.89“ 4.47‘ 356* 2.97~
(02) 84.04 84.02 83.96, 83.95 84.12, 84.15
% variance reduction male 6.1% 6.4‘7r 4.0% 6.6%' 7.9%
% variance reduction female 23.2% 23.6%, 12.1% 30.1%, 41.7%

 

 

 

 

 

 

**" p<.001, ** p<.01, * p<.05, ~ p<.10

For every one standard deviation increase in mathematics preparation, both boys

(-.86) and girls (-.18) lose a partial point .The negative relationship between teacher
preparation and student achievement is puzzling but inline with results from the earlier
SEM analyses in this dissertation and with Valli and Reckase (Valli et al., 2003), and
Rowan, Correnti, and Miller (B. Rowan, Correnti, & Miller, 2002). For teachers math
knowledge, girls average gains increase by +.89 points for every standard deviation
increase while boys gains stay nearly level. One standard deviation of teacher efficacy,
however, decreases average gains for girls by -1.11 points.

Orientation, mathematics preparation, math instruction preparation, and
mathematics knowledge variables controlling for SES do not explain more variance in the
average gains for boys and girls across teachers than does SES alone. Teacher efficacy
and preparation for assessment are the only two teacher characteristics that explain

variation across teachers for boys (6.6-7.9%). These teacher variables explain 30.1%-

99

41.7% of the variation in gains for girls across teachers controlling for SES.

Gender a_nd Teacher Professional Development. This analysis focuses on the
differential effects of teacher professional development as measured by professional
development impact, mentoring frequency, and professional development frequency.
Professional development impact (self report) explains very little variance in average
male or female gains across teachers. It predicts less well than SES alone. Mentoring
frequency explains no more variation in boys gains across teachers than SES alone but it
does explain 27.8% of the variance in girls gains. Professional development frequency
explains 15-17% of the variance for both boys and girls gains across teachers controlling
for SES. The variance decomposition for professional development follows in Table
8.15.

Table 8.15 Variance Decomposition for Gains by Gender Controlling for SES - Teacher
Professional Development

 

 

 

 

 

 

 

 

 

SES Professional Dev

Variables (v01 8 VII) Uncond ree/Red Lu PDlmpact :MentFrq lPDFreq
Intercept (v00) 50(1.04) 0.33(1.06) -0.03(1.04II 0.33(1.03) .12(.99)
(v01) -0.34(.46I -0.09(.50) -.12(.05)*
SES(v02) -6.44(3.94) -6.79(3.94)~I -6.53(3.99) —6.13(3.69)~
Dfem (v10) -.08(.54) -.03(.54) -.02(.55I 0.06(.52) -.03(.53)
(v11) -0.48(.51I -1.41(.60)* .02(.07)
SES(v12) 7.90(4.25)~ 7.35(4.22)~I 6.35(3.98) 7.79(4.22)~
Intercept Var (Too) 9. 33*“ 8.76*** 886*“, 9. 16*" 768*“
Dfem Variance (T40) 5.09” 391* 440*, 3.53~ 4.53”
(02) 84.04 84.02 83.92, 83.53 83.94
% variance reduction male 6.1% 509/ 1.8% 17.6%
% variance reduction female 23.2% 13.6%, 27.8% 15.5%

 

 

 

 

... p<.001, ** p<.01, * p<.05, ~ p<.10

Mentoring frequency is negatively related to average achievement for girls across

100

teachers. For every one standard deviation increase in mentoring frequency across
teachers, girls lose -1.41 points compared to boys who remain nearly level. Mentoring
frequency explains no more additional variation in average achievement for boys across
teachers but it explains 27.8% of the variance in average gains for girls across teachers
controlling for SES.

Professional development frequency has a negative relationship to boys
mathematics gains across teachers controlling for SES. For every standard deviation
increase in professional development frequency, boys lose -. 12 points. Girls gains remain
nearly level. Professional development frequency explains 11.5% of the variance in
mathematics gains for boys across teachers while it adds nothing to the variance
explained for average gains by girls across teachers.

Summagy of differential gender effects - The effects of teachers on the average
gains for male and females controlling for SES varies for certain teacher practices,
charactersitics, and professional development. Teacher practice, characteristics, and
professional development variables generally explain more of the variation that exists in
average mathematics gains for girls across teachers in Connecticut once SES is
controlled. SES explains 23.2% of the variation in female gains and 6.1% of the variation
in male gains. The loss in gains for boys attributable to a one standard deviation increase
in district free and reduced lunch (proxy for lower SES) is over 6 points while girls gain
+7.9 points for a one standard deviation increase in free and reduced lunch.

The models for teacher practice explain 24-30% of the variation in average female
gains across teachers when SES is controlled; the models for teacher characteristics

explain 15-37% of the variation in female gains across teachers; and mentoring frequency

101

explains 28% of the variation in female gains across teachers. The same models explain
less than 7% of the variation in average achievement for males across teachers. One
exception is the variable for professional development frequency which explains 15-17%
of the variation in gains across both genders and is consistently, negatively related to
gains controlling for SES.

Teacher variables that have differential effects on gains by gender (negative for
girls and positive for boys) include two practices, math sensemaking activities and group
and project activities, and one teacher characteristic, teacher efficacy. The teacher
characteristic, math knowledge, is positive for girls and level for boys. In this data, the
effects of all professional development variables are negative for both genders.

Three variables, mathematics sensemaking, group and project activities, and
teacher efficacy signaled predicted losses of -.95 to -1.07 points for girls across teachers
while predicting gains of +52 to +.70 for boys. Teachers’ professional development
frequency signaled a decrease in gains for boys at -.12 points across teachers with girls
remaining level. The only variable exhibiting a positive effect for girls was math
knowledge which predicted an increase in girls’ average gain of +89 points for every
standard deviation increase across teachers. Boys’ average gain remains nearly level.
Effects of Teacher Level Variables on Mean Gains by Black and White

While the fixed effects model for ethnicity, Model 2, predicts that average gains
vary significantly by ethnicity, a random slopes and intercepts model, Model 9, tests
whether or not the average achievement gains for white and black students vary
significantly across teachers. Other demographic groups did not vary significantly from

Whites. In other words, do teacher practices, characteristics, or professional development

102

variables have differential effects on the average mathematics gains of Black and/or

White students across teachers? If variance is significant across teachers, it is then

possible to determine how much of the variation in each groups’ achievement can be

explained by single level two teacher variable.

Model 9. Unconditional model for Black/ White

Level 1- Students - Same as Equation 8.13

Level 2 - Teachers

where

Y00j

roj

YIOj

r4j

:30) = 1’00 + "0j (8-19)

1611': 710+r4j

1621' = 1’20
:63) = 1’30
[B8j = 1’70
.59) = 1’70

= intercept, average in student gains/achievement in teacher j’s classroom
holding teacher practice constant

= the error term or unique teacher effects

= intercept, black student gains/achievement in teacher j’s classroom all
other predictors held constant

= the error term or unique teacher effects

The estimated level two regression coefficients for Model 9 are given in Table

8.16.

103

Table 8.16 Regression Coefficients for Gains by Ethnicity 1999-2000

Model9
"lichen“, ’ f ’ .43(.94) ’
Black (V10) 1 .92(1.04)~
__ 11185580,, ”2.29(1—.02);
All other (v30) 1.38(.76)~
"Grfladfe4(v80) i f f i I 188(25):" ’
: Gravde'6(v90) ’.98(.19)***
NI} \7ariance(62) * l 83.95
iii/(2 InQSept (100) 3.13“
Black (710) 18.16“

p<.001, ** p<.01, * 9905, ~ P<-10

The effect of Black varies significantly across teachers (T,o=18.30, p<.01) as does
the effect of White (Too=3. 13, p<.01). Selected coefficients from this analysis serve as the
base for the variance decompositions found in Tables 8.18-8.21. Next district SES is
added to the model to determine its effect on the average achievement gains for blacks
and whites across teachers.

Model 10. Black, White achievement gains and SES
Level 1- Students - Same as Equation 8.13

Level 2 - Teachers

floj = yOO ‘1" Yo] pCtfi'lu ‘1' r0]- (8.21)

1611' = 1’10 + 711190091“ + ’4}

162) = 1’20
163)" = 1’30
768} = 1’70
169} = 1’70

104

where
YOOj = intercept, average in student gains/achievement in teacher j’s classroom

holding teacher practice constant

= the change in average gains/achievement associated with mean district

Y01j
free and reduced lunch, all other predictors held constant
r0, = the error term or unique teacher effects
Y 10' = intercept, black student gains/achievement in teacher j’s classroom all
J
other predictors held constant
Y] 1. = the change in average black student gains/achievement associated with
J
one unit change in average district free and reduced lunch
r, j = the error term or unique teacher effects

The estimated level two regression coefficients and variance components for
Model 6 are given in Table 8.17.

Table 8.17 Regression Coefficients for Black/White Gains by Average District SES

Model 10

Intefircveti (ft/00)? .50(85) F
8 T ‘ ‘Ffet; aitdireduced lunch (v01).1.39(2.26) ?
“BLEQITIETT T if #1388117); F
' W’Frgefan'd rveduvcedvlunch (v11) '39.1d6.49)4*
’ "ligation” " A '1.92(.95)~

All other (v30) 1.32(.83)~
Wei—aldefiygo)? i i if I :.88(.24)** *

Grade 6 (v90) .98(.17)***
tVffilaiiancE (02) if 84.13
fiftfizVadanc; (100)— 1 (3.05—*9
WM ”13:105." - 979" -

99* p<,oo1, ** p<.01, * p<.05, ~ P<910

105

How much do teachers vary around the average mathematics gains for White and
Black students in average SES districts? To answer this question the following equation
computes the variance in black gains explained by average SES across teachers. 16 The
estimated 1:,0 values from Tables 8.16 and 8.17 would be substituted in the following
equation:

r,0(model 6)— 7,0(model 7) _ 18.16-9.7
1,0(model6) 18.16

 

= .465 = 46.5% (8.22)

This means that 46.5% of the variance in the predicted average achievement gains for
Black students is explained by variance in district SES across teachers. Using the same
process for the remaining taus, the results are presented at the bottom of Tables 8.18.-
8.21.

Table 8.18 Variance Decomposition for Black/White Gains by SES

 

 

 

 

 

 

 

 

 

SES
Variables (v01 8 VII) Uncond Free/Red
Lunch

Ilntercept (v00) .43(.94) 0.50(.85)
SES(v02) 1.39(2.26)
IBlack (v10) 1.92(1.04)~ 3.86(1 .17)“
SES(v12) -19.14(6.49)*1
Intercept Variance (T00) 6.13" 3.05“
Black Variance (710) 18.16" 970*
Level 1 variance (02) 83.95 84.13
% variance reduction White 2.2%
% variance reduction Black 45.9%,

 

 

*** p<.001, ** p<.01, * p<.05, ~ p<.10

 

16 A three-level analysis, students nested within teachers nested within districts,
was conducted for the ten Connecticut districts that were represented by 5 or more
teachers. There was no significant variance across districts for this sample so it was
determined that the two-level model controlling for SES was the most appropriate.

106

The table indicates that being in a district with average SES (21% free and
reduced lunch) has a large, negative effect on Black mathematics gains ('y ,2 = —19.14,
sd=6.49). One standard deviation increase in district percentage of free and reduced
lunch accounts for a -19.14 point drop in average mathematics gains for black students
across teachers and explains a small increase of +1 .39 points in White students average
gain. SES, therefore, has a substantial differential effect on Black and White student
gains. There are two districts in the Connecticut data that have over 50% free and
reduced lunch (over +2 sd). This analysis predicts that Blacks would be expected to lose
-28.28 points over the course of one year while White students are predicted to gain +2.78
points for a two standard deviation increase in free and reduced lunch.

SES explains almost 46% of the variance in average black gains across teachers
while it accounts for only 2% of the variance in average white (non-black) gains across
teachers.

Ethnicity and Teacher Effects

Now that the context is set for the analyses at level two, it is possible to track the
variance explained by each teacher level variable controlling for SES and to determine its
effect on average mathematics gains for Black and White across teachers. One variable at
a time is added to the level two equation, EQ 8.23. The term “variable x” is exchanged
for a single Mathematics Practice (math sensemaking, drill and lecture, or group and
project activities), Teacher Characteristic (mathematics preparation, preparation for
mathematics instruction, mathematics knowledge, reform and traditional orientation,
assessment prep, or efficacy), or Professional Development (professional development

impact, mentoring frequency, or professional development frequency).

107

Model 1]. Template for Level 2 equations
Level 1 remains constant (Eq. 8.13)

Level 2 - Teachers

fioj = yOO '1' yo] pCtfi'lu ‘1' yozu variable x ‘1" 7'0} (8.23)

fllj = 710 + yllpctfrlu + 1’12. variable x+ r4]-

flZj = 1’20
fl3j = 1’30
fl8j = 1’70
flQj = 1’70

where

Y 02” j = change in average in average student gains associated with a one unit
change in “variable x” holding SES constant
and

Y 12 , = the change in average black student gains associated with one unit
..j

change in “variable x” holding SES constant
Ethnicity and Tegher Practices. A variance decomposition for teacher
practices can be found in Table 8.19. The table indicates that very little additional
variance in predicted gains for White or Black students is explained by teacher practices

across teachers once SES is controlled.

108

Table 8.19 Variance Decomposition for White/Black Controlling SES - Teacher

 

 

 

 

 

 

 

Practices
SES Practice

Variables (v01 8 VII) Uncond Free/Red Lu Sensemaking :Drill8Lecture iGroup8Project
Ilntercept (v00) .43(.94) 0.50(.85) .62(.95)I .56(.89)I 49(93)
(v01) .18(.32)I -.27(.31I .04(35)
I SES(v02) 1.39(2.26) 1.34(2.11)I 1.45(2.11I 1.41(2.08)
Black (V10) 1.92(1.04)~ 3.86(1.17)** 3.96(1.14)**I 3.77(1.15)**I 3.85(1.16)**
(v11) 1.25(.95)I .74(.89I .21(1.13)
SES(v12) -19.14(6.49)** —20.58(6.50)**I -18.63(6.91)**I -19.19(6.60)**
Ilntercept Var (T00) 3.13“ 3.05“ 3.12“ 3.02**I 3.17“
Black Var (T40) 18.16” 970* 954*, 1054* 11.21“
(02) 83.95 84.13 84.13I 84.14 84.091
% variance reduction White 2.2% .3%i 3.5%I 0%
% variance reduction Black 45.9% 47.5%, 42.0%, 38.3%,

 

 

 

 

 

p<.001, ** p<.01, * p<.05. ~ P<-10

Drill and lecture activities in average SES districts explains 3.5% of the variance

in White achievement gains across teachers, slightly more than SES alone. It explains no

additional variance for Black gains across teachers. One standard deviation increase in

drill and lecture activities decreases white gains by -.27 points while simultaneously

increasing black gains by +.74 points, an indicator of differential effects. For Black

students, only mathsensmaking activities across teachers explain additional variance in

average SES districts. The predicted change attributable to one standard deviation

increase in frequency of sensemaking activities is +.16 point for Whites and +1.28 points

for Blacks. Group and project activities do not explain variance in average gains across

teachers for either group.

Ethnicig and Teacher Characteristics. The variance decomposition for teacher

characteristics follows in Table 8.20. There are only two variables which explain more

109

variance in the gains across teachers than SES alone - mathematics preparation and

reform orientation. Mathematics preparation explains additional variation in gains for

both Black and White Students across teachers while reform orientation only explains

additional variance in gains for Black students.

Table 8.20 Variance Decomposition for Black/White Controlling for SES - Teacher

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Characteristics
SES Teacher Characteristics
Variables (yo, 8 VII) Uncond Free/Red Lu IMthPrep thhlnst FillthKnow j
Ilntercept (V00) .43(.94) 0.50(.85) 1.62(.94)I .47(.96)I .28(.97)),
(v01) -.92(.39)*, .26(.31)I .41(.33fl
SES(v02) 1.39(2.26) .61 (1 .92‘ 1.69(2.07I 1.72(2.12)I
IBlack (v10) 1.92(1.04)~ 3.86(1.17)** 3.15(1.21)*, 3.83(1.18)**, 3.97(1.17)**,
(V11) -1.77(1.09I .01 (95” .42(.68
SES(v12) -19.14(6.49)** -19.20(6.34)**, -19.22(6.52)**I -19.46(6.53)**,
Intercept Var (T00) 3.13“ 3.05" 2.33“, 3.08", 3.08“,
iBlack Variance (1'40) 18.16" 970* 7.64* 1080* 1121*,
Level 1 Variance (02) 3.95 84.13 84.32, 84.13, 84.0fl
% variance reduction White 2.2% 2569/ 1.69/' 1.3%I
% 99199199999 Black - - - i- 499% e- e- 3 @991 L 4998 -._ “97-991
SES Teacher Characteristics
Variables (v01 8 Uncond lFree/Red Lu Reform Or ITrad Or IAssess I
“1’ J. l I
Ilntercept (v00) .43(.94) 0.50(.85) .50(.92)I .50(.93), .50(.92I
(V01) -.07(.27), .07(.34I -.15(.26I
SES(v02) 1.39(2.26) 1 .49(2. 12), 1.32(2.21)I 1.23(2.08)I
lBlack (yIO) 1.92(1.04)~ 3.86(1.17)** 3.73(1.14)**, 3.84(1.24)**I 3.86(1.17)**,
(v11) 1.33(.86)I -.03(1.06)I .69(.80)I
SES(v,2) -19.14(6.49)** -17.34(6.02)**, -19.04(6.44)**I -18.73(6.82)**I
Intercept Var(roo) 3.13” 3.05" 3.15" 3.17", 3.19",
IBlack Var (T40) 18.16” 9.70* 8.09*I 10.81" 997*
Level 1 Var (02) 83.95 84.13 84.161 84.11J 84.11J
% variance reduction White 2.2% 0%I 0%I 0‘77,
% variance reduction Black 45.9% 55.5%, 40.4%, 45.1%,

 

llO

Table 8.20 (cont'd).

 

 

 

 

 

 

 

 

 

 

 

SES Teacher

Charactersitics

Variables (v0, 8 VII) Uncond Free/Red Lu Efficacy
Intercept (v00) .43(.94) 0.50(.85), .50(.93)
(V01) I .00(.30)
SES(v02) 1.39(2.26), 1.39(204)
IBIack (v10) 1.92(1.04)~ 3.86(1.17)**, 3.80(1.23)**
(Y111 I -44(-931
SES(v12) -19.14(6.49)**I -18.86(6.36)**
Intercept Var (TOO) 3.13“ 3.05“, 3.19"
Black Variance (T40) 18.16“ 970*, 1079‘
Level 1 Variance (02) 83.95 84.13l 84.11
% variance reduction White 2.2%i 0%
% variance reduction Black 45.9%, 40.6%

 

p<.001, ** p<.01, * p<.05, ~ p<.10

Note that for every standard deviation increase in mathematics preparation, both Black
and White students are predicted to lose points and the loss is doubled for Black students
(Black= -l .78, White =-.90). The negative relationship between teacher preparation and
student achievement is puzzling but inline with results from the earlier SEM analyses in
this dissertation and with Valli and Reckase (2000), and Rowan, Correnti, and Miller
(2004)

Reform orientation controlling for SES across teachers explains 55% of the
variation in Black gains over the course of one year while explaining no variance in the
gains for White students. For every one standard deviation increase in teachers’ reform
orientation, Black students are expected to gain an additional +1.35 points over the course
of one year. White students are expected to remain nearly level (-.08, sd=.27), another
differential effect.

Although teachers’ math knowledge explains significant variance in gains by

111

gender, this is not true for Black and White gains across teachers in Connecticut. Both

groups are predicted to increase gains by the same amount, about +.41 points, for every

one standard deviation increase in mathematics knowledge across teachers.

E_t_l_rnicitv a_md Teacher Professional Development. This analysis focuses on the

differential effects of teacher professional development as measured by professional

development impact, mentoring frequency, and professional development frequency. All

professional development variables explain additional variance in mathematics

achievement gains across teachers for White after SES is controlled. Only professional

development frequency explains additional variance in Black gains across teachers once

SES is controlled. The variance decomposition for professional development follows in

Table 8.21.

Table 8.21 Variance Decomposition for Black/White controlling for SES - Teacher
Professional Development

 

 

 

 

 

 

 

 

 

 

SES Professional Dev

Variables (v01 8 VII) Uncond Free/Red Lu PDlmpact i MentFrq IPD Frq
Intercept (v00) l43('94) 0.50(.85) -.19(.92)I .48(.86) .24(.86)
(v01) -.53(.27)~I -.77(.32)* -1.08(.24)***
SES(v02) 1.39(2.26) .74(2.06), .56(2.22) 1.29(2.03)
Black (v10) 1.92(1.04)~ 3.86(1.17)** 3.78(1.22)**I 3.62(1.18)** 3.96(1.23)**
(v11) -.03(.98h -1.27(.94) .90(.88)
SES(v12) -19.14(6.49)** -18.78(6.52)**, -19.82(6.59)** -19.77(6.94)**
Intercept Variance(T00) 3.13" 3.05“ 2.83", 2.58 1.74**
iBlack Variance (T10) 18.16“ 970* 1022* 10.20 863*
(02) 83.95 84.13 84.16, 84.02 84.31
% variance reduction White 2.2% 9.9%I 17.5% 44.4%
% variance reduction Black 45.9% 43.0%, 43.7% 52.5%

 

 

 

”a p<_001’ ** p<.01. * p<.05, ~ p<.10

For every standard deviation increase in professional development impact, Black students

112

are predicted to lose nothing compared to White students who lose about half a point.
Professional development impact in average SES districts explains 9.9% of the variation
in White gains across teachers, an increase of 7.7% over SES alone.

For every one standard deviation increase in mentoring frequency in average SES
districts, Black students lose -1.27 points. White students lose -.77 points for every
standard deviation increase in mentoring frequency. Mentoring frequency explains 44.4%
of the variation in average White gains across teachers controlling for SES and no
additional variation over SES alone for Black gains across teachers.

Once SES is controlled, professional development frequency explains the most
variance in the average gains for both Black and White students across teachers.
Professional development frequency explains 44% of the variation in White gains across
teachers and 53% of the variation in average Black gains. For every one standard
deviation increase in professional development frequency, White gains are expected to
drop by -1.08 points. Black gains are predicted to increase by +.90.

Summagy of differential effect; bv ethnicity - In Connecticut, teachers vary
significantly around the average mathematics gains for two groups, Black and White
students. SES explains 45.9% of the variation in Black gains but only 2.2% of the
variation in White gains. For every one standard deviation increase in free and reduced
lunch, Black students lose -19.1 points.

Variables that explain additional variance in Black gains across teachers once
SES is controlled are math sensemaking activities (an additional 1.5% of the variance
explained), mathematics preparation (an additional 10.9%), reform orientation (an

additional 9.1%), and professional development frequency (an additional 6.8%). Every

113

one standard deviation increase in the teacher math sensmaking activities increases Black
gains by +1.28 points over the course of one year; mathematics preparation decreases in
Black gains by -1.78 points; reform orientation increases in Black gains of +1 .35 points;
and professional development frequency increases average Black gains by +.10 points.

Professional development variables explain the largest amounts of variance in
White gains across teachers, an additional 7.7-42.7% once SES is controlled.
Mathematics preparation explains an additional 23% of the variance in White gains
across teachers after controlling for SES. The only teacher practice variable that adds
slightly to variance explained for White student gains across teachers is drill and lecture
activities (an additional 1.3%).

For every one standard deviation increase in the teacher variable for professional
development impact, mentoring frequency, or professional development frequency,
average White gains are expected to decrease from -.54 to -.75 points across teachers.
For every standard deviation increase math drill and lecture activities, average White
gains across teachers are predicted to decrease by -.29 points. For every standard
deviation increase in mathematics preparation , White gains are also expected to decrease
-.90 points.

There are indications that certain teacher variables, reform orientation and
professional development frequency, have differential effects on average gains for Black
and White students across teachers when SES is controlled. Generally, however, teacher
variables predict losses for both groups or gains for both groups. Most of the variation
explained in Black scores is explained by SES and much of the variation in White scores

is explained by professional development variables.

114

III. IHE Teacher Preparation, Teachers, and Student Achievement

This section addresses questions about university teacher preparation programs
and their relationship to teacher practices, characteristics, professional development, and
subsequent student achievement. That is to say, do Connecticut teacher preparation
programs vary in the average levels of 4th-8th grade student gains, teacher practice,
characteristics, or professional development they predict The analyses in this section use
a smaller sample of teachers who indicated that they attended specific teacher preparation

programs in Connecticut.

Effects of teacher preparation programs on the mathematics gains of 4th-8th

 

gr_ade stadents. The equation for the three level mathematics analysis with students
nested within teachers nested within teacher preparation institutions is shown by Equation
8.23. There are 1098 students nested within 61 teachers nested within 13 identified

teacher preparation institutions in Connecticut. 1 7.

Y-- =7r0jk+e

1} ijk’

”Ojk = 500k + rOjk’ (3.24)
500k = 7000 + ”00k

Less than 1% of the variance in student mathematics gains in the unconditional three
level model was located at level three, teacher preparation institution,. In other words,
teacher preparation program effects are nearly identical when predicting 4th-8th grade
student gains nested within their teacher graduates. Lack of variability at level three

suggests that a simpler two level model is the better choice for further analysis

 

‘7 Fixed effects are unbaised although standard errors are likely too small in
small samples (Raudenbush, p. 281).

115

(Raudenbush et al., 2000, p237). No further investigation into variance explained in
student gains across teachers by teacher preparation institution was possible for this data.

Effects of teacher preparation on levels of teacher practice, characteristics,
and professional development. HLM analyses from Section 2 indicated that ten
variables (math sensemaking, drill and lecture, group and project activities, mathematics
preparation, assessment preparation, reform orientation, teacher efficacy, professional
development impact and frequency, and mentoring frequency) controlling for SES
explained variation in student gains by gender or ethnicity across teachers. A two level
model with Connecticut teacher preparation institutions at level two predicting one
teacher variable at a time at level one was constructed. The model included 61 teachers
nested within 13 teacher preparation institutions.

Of the ten variables found to explain variance in average student gains by gender
and ethnicity across teachers, only one, mentoring frequency, varies across teacher
preparation programs. Although it did not explain variation in student gains across
teachers, teachers’ mathematics knowledge also varied across institutions so it is included
below.

The set of new variables used in the Connecticut teacher preparation analysis

follow in Table 8.22.

116

Table 8.22 Description of Teacher Preparation Variables

 

 

Type Variable Variable Description (Aggregated from teacher
Abbreviation level)
Preparation Program facchar Faculty characteristics (mean=24.37, sd=1.82)
Characteristics
progqual Program quality (mean=18.71, sd=1.93)
fieldexp field experience (mean=16.52, sd=1.45)
rank Program rank by pass rates on Praxis basic math

(mean=6.28, sd=3.51)

Effects of teacher preparation institutions on teachers’ mathematics knowledge

and mentoring frequency. In the next set of analyses, characteristics of teacher
preparation programs predict levels of average teacher mathematics knowledge and
frequency of mentoring.

Unconditional models predicting average mathematics knowledge and mentoring
frequency are presented first followed by the results for each of four teacher preparation
program variables, faculty characteristics, program quality, field experiences, and
program rank. Program rank is based on teacher candidate pass rates on the ETS Praxis
Basic Skills Exam, CBT: Mathematics, from 1998 as reported in the Initial Report of the
Secretary on the Quality of Teacher Preparation (Riley, 2000), otherwise known as the
State Report Card.

Model 12. Unconditional model for teacher variables across teacher preparation
institutions

Level 1 - Teacher

variable xy- 2 30 + eij (8.25)

117

Level 2 - IHE teacher preparation program
:60} = y00+ r0]. (8.26)
The coefficients for mathematics knowledge are presented in Table 8.22 followed
by the coefficients for mentoring frequency in Table 8.23. Mathematics knowledge

Table 8.23 Connecticut’s Average Mathematics Knowledge by Teacher Preparation
Institutions

Modei12a
Ihtefcepf(voo) if f i f * -26.46(.66)***
LVL 1 Variance (02) 11.89
18938997999906) I I 321-347“

*** p<.001, ** p<.01, * p<.05, ~ p<.10

Predicted average mathematics knowledge reported by teachers varies
significantly across teacher preparation institutions in Connecticut. 83.6% of the variance
in mathematics knowledge for Model 12 is within programs (between teachers) leaving
16.4% of the variance between programs. That is to say, there is five times more variation
in average mathematics knowledge within programs than between programs. The average
level of mathematics knowledge is 26.46 (sd=.63, p<.001) when no other variables are
included in the model. This average scale score roughly corresponds to “good.”

Results from the unconditional model for mentoring frequency are presented in

Table 8.23. For this analysis, mentoring frequency is substituted for variable x in Eq.

8.24.

118

Table 8.24 Connecticut’s Average Mentoring Frequency by Teacher Preparation

Institutions

* F — ' ' Model 125
Intercept (v00) 33.38(2.31)***
LVL 1 Variance (02) 143.16
,_LYL.9)/9rianae<Too.> 9591*

p<.001, ** p<.01, * p<.05, ~ p<.10

Predicted average mentoring frequency varies significantly across teacher preparation
institutions in Connecticut. 80.1% of the variance in mentoring frequency for Model 12b
is within programs (between teachers) leaving 19.9% of the variance between programs.
That is to say, there is four times more variation within programs than between programs.
The average level of mentoring frequency is 33.38 (sd=2.31, p<.001) when no other
variables are included in the model. This roughly corresponds to a scale frequency of “a
few times.”

There are three rating scales for teacher preparation programs measured at the
teacher level: faculty characteristics, program quality, and field experiences. These were
aggregated from the teacher level to the program level to test if they explained variance in
mathematics knowledge and mentoring frequency. The fourth variable, program rank,
reflects teacher candidate pass rates on the Connecticut basic skill mathematics test from
1998.

Model 13. Base model to predict teacher variables by teacher preparation program
characteristics

Level 1 - Teacher

variable xij = BO + eij (8.27)

119

Level 2 - IHE teacher preparation program
1601‘ = 700 + 1’01 program characteristic y + roj (8.28)
where the program characteristic y is faculty characteristics, program quality, field
experiences, or program rank.
The coefficients for mathematics knowledge are presented in Table 8.25 followed
by the coefficients for mentoring frequency in Table 8.26.

Table 8.25 Average Mathematics Knowledge Predicted by Program Characteristics

7 Unconditional 7 Faculty ’7 Program Field? Program Rank

Characteristics Quality Experiences on CBT: Math

Interceptwoo) 26.46(.66)*** 26.55(.69)*** 2650(70)": 2649(69)"; 2646(70}; _

_PiogChar(v01) '.28(.48) .29(53) ’ 112(57) _ i.o4(.20)— 4'
LVL—1Variance(02) 11.89 11.89 3 311.78 * 11.88 ’8 11.86
LVL {(15.11.8600) 8234*" i ”2.67? 7 7 295* 284*“ i 291*" 7
’v'ar‘egp'bypiog Char. 0% 0% 10% i i 0% i ’

9W2 - - - , -
*** p<.001, ** p<.01, * p<.05, ~ p<.10

None of the teacher preparation program characteristics in this data explain variance in

average mathematics knowledge reported by teachers across preparation programs.

Table 8.26 Average Mentoring Frequency Predicted by Program Characteristics

Unconditional AFacuIty H Program 7 Field RahkofiCB—T:
Characteristics Quality Experiences Math
InterEept (v00) 3328(242)“ 82.99(2.14)*** ’33.45(2.30)*** 3355(23’2)“ 3355(248);
Prog Char (v01) -1.11(1.14) 202(179) 2.09(1.91) -.53(.71)
LVL—1_Varg(o2)_n 143.16 3 143.47 I 145:72 _ 145.57 _, 143.44
tvt’zvarnoo, 3561* 3879* ” “28.739 "297419 ‘ 37:96; ' ”
Var exp by Frog 0% 19.3% 17.4% 0%

Char at Lvl 2 .
***r><1001.**b<-01; * p<.05, ~ p<.10

Of the four measures for teacher preparation programs, two explain variance in average

120

teacher mentoring frequency across teacher preparation institutions. Program quality
explains 19.3% of the variance and field experiences explain 17.4% of the variation in
teacher mentoring frequency across teacher preparation institutions. Both program quality
and field experiences as rated by graduates of the program predict a 2 unit increase in
mentoring frequency for every one standard deviation increase in program quality or field
experiences. The mean of mentoring frequency roughly corresponds to “a few times.”
Ranking teacher preparation institutions on levels mathematics k_nowledga
and mentoring freguency. For the next analysis, 13 teacher preparation institutions are
added to the model as dummy variables at level two. The intercept is allowed to vary.
Model 13 answers the question, “Are different levels of specific teacher variables
associated with particular teacher preparation institutions in Connecticut?” For these
analyses, one of Connecticut’s largest teacher preparation programs serves as the
reference category and three small teacher preparation programs are combined into one
dummy variable for “all other universities.”
Model 13.
Level 1 - Teacher

variable xi]- = Bo + e,-j (829)

Level 2 - IHE teacher preparation program

BOO = 700 + yo, Univl + 702 [/71in + 703 Univ3 + y04Univ4
+y05Univ5 + y06Univ6 + y07Univ7 + yogUniv8 (8.30)
+ 709Univ9 + yIOAIIOthUniv

where

variable xi]. = one variable of two teacher variables (mathematics knowledge or

121

mentoring frequency)
r, = the error term or unique teacher preparation program effects
Coefficients for mentoring frequency by teacher preparation institutions are
presented in Tables 8.27.

Table 8.27 Average Mathematics Knowledge Distributed Across Teacher Preparation
Institutions

Unconditional ’ Mathematics 7’ ..
Knowledge

Interwcéptfvoo) ’26.46(.63)*** 26.64.70}: A
__ Urfiversity1 (v01) * i f ’ 6.80(2.87)~ *
"UnigersityZ (v02) * i ’ 525(298) *
“07688165809 f f i f if i i if H487('2.8“0)T if ’
”magnum ’ f i f if ' i ’ 300(263) *
WUhiverTsity 50705) J ' ’ 633(280)
--Jnicefsifyavbg) ’ f if ‘ 400(314)
’“LI—niirewrsityflgoi ’ _ ' ' 1.68(2.87)
’ bhiilefsity 8(vI)8) 555(272)
Uhiversity 901091 i N f ’ 8.00(3.44)~’ ’
”Thacher Uhiversities (v30) 1 i f ’ '5.00(2T98)A if
vafvérigncefo’z) ’ 11.89 i i F 11.90 i ’
"LVTEVaiiahcé (100) i f I 8 2.34m ’ 1.45 *
Variance explained at Lvl 2 38.0%

p<.001, ** p<.01. * p<.05. ~ P<-10

The addition of dummy variables for 13 teacher preparation institutions explains
38% of the variance in average teachers’ mathematics knowledge across programs. 100 is
no longer significantly different from zero. Fixed effects for University 1 may be

computed using the following formula:

122

" " 8.31
Ymthknow : flO ( )
'30 = 1300 +1*f01
= 2664+ 1*(6.80)
= 33.44

Table 8.28 presents the fixed effects for mathematics knowledge by teacher
preparation program sorted from lowest to highest.

Table 8.28 Distribution of Predicted Mathematics Knowledge across Teacher Preparation
Programs (low to high)

1 Average I Approximate Scale
Mathematics Correspondence

Knowledge
'7 Refégnbehniversiw 7 7 7 7 726.63 7 A ’Goon “ A M
University 7 0,07) 28.31 Good
tiniest—y“ 47v“) " l h ,. 29.63 _ ' coed "I
FUnivérsfiitvfielvasf * 80.83 Excelleht 7
'Uhi‘ve‘rs‘ty‘aiyog)“ * 3 31.30 Excellent
1 wuhiveisityE (Yea) I i M 31.58 Excellent 7
Three Other Universities (v30) 3164p -- , Excellent“
University 8 (v08) 32.19 Excellent
M ’UhiT/e‘rsityAS (v65) 7’ 7 7 832.96 3 Excellent
Uni/grain} (v01) A 33.44 ‘eiceuent
118195819<yo>1 “ * 94.9 * .681th ‘*

There are small differences in expected teacher mathematics knowledge across teacher
preparation programs but none of the coefficients are significantly different from zero.
All programs are predicted to produce teachers, who on average, report good to excellent
knowledge of mathematics.

Coefficients for mentoring frequency by teacher preparation institutions are

presented in Tables 8.29.

123

Table 8.29 Average Mentoring Frequency Distributed Across Teacher Preparation
Institutions (low to high)

7 Unconditional Mentoring
Frequency
.n‘tegeawoo) ’ * * * * ’ 3333,23in 33338.11)?" *
University1(y01) ’ ' ’:16.0’0(8.05) if
” 6611,3131}; 21,702) * ’ * ’ * i i " ” 17488647) "
University 3 (v03) -22. 83(7. 75)
"“0388;th (v04) * i ” 333,634)
W University5(vo5) _ . 7 h 1’ 7 l -9. 99(7:75)
11161365786 ($08)” 7 7 i 7 7 7 _-1333(9.14)
University 7 (V07) -3 40(8 05)
HUnIverSIty8(v08) ’ 7 7 7 7 7 i * -7.4Q7.—36)V ' l
’ University9(y09) ’ ’8 -, i f i i i 5 1070080335)
1 Three Other Univergltles (v30) ’ * -8.64(9.12)
LVL1Variance (02) 83 143.16 ” ’8 2 141.45 T ’
LVL2Var1ance(TOO) ’ 3561* .53 I T
Variance explained at Lvl 2_ 98.5%

p<.001, ** p<.01, * p<.05, ~ p<.10

The addition of dummy variables for 13 teacher preparation institutions explains
nearly all of the variance in mentoring frequency across programs. T00 is no longer
significantly different from zero. Fixed effects for University 1 may be computed using

the following formula:

Ymentfi'q 160
.30 =1700 +1"“1901 (8.32)
= 33.56 + l*(— 16.00)
= 17.56

Table 8.25 presents the fixed effects for mentoring frequency by teacher

preparation program.

124

Table 8.30 Distribution of Mentoring Frequency Across Teacher Preparation Programs

(low to high)
G fffff ’ Average 1’ Approximate
Mentoring Scale
Frequency Correspondence
Univefsitfiwos) ’ ’ ”10.73 ’ Neva '
University 1 (v01) 17.56 Once
mUnivevrsity6 (V06) 20.23 3 Once ’
F'JrTverQQ’sms)“ ' f M ”23.57— “ ”113.311.333.333
"‘"Thiee—oihe—r Univer—sitiieslv30) 24.92 ’ ”More-than 3333 ’
" Drfivérsityewoa) * G i i ’ 426.07 ’ TMbrethgn 6nd!
" TUfiivhrsTityAZ (v02) 5 i f if 3 26.08 ’ ’Morethah once
"6303,55,, T6071 ,, i i 3 30.16 I ’Mbrethavn ohcé
“tinder—sit; 41,764) 7 33.23 ’Afew times 1
“Reference university 33.56 ’A few time;
111783311ngon _ _ ' ’ M43356 _. “monthly _ _

There are differences in the expected frequency of teacher mentoring predicted by teacher
preparation program but none of the coefficients are significantly different from zero.
Summary of the HLM results for mathematics gains in Connecticut.

Once the fixed effects for gains by ethnic and gender groups controlling for grade
level were estimated at level one, it is possible to answer the main research questions
regarding how those relationships changed when teacher practices, teacher characteristics,
and professional development variables were added to the model. Fixed effects for
students are presented first followed by the random effects of teachers on student gains.
Last, the results for the analyses of student outcomes and teacher variables by teacher
preparation institution in Connecticut are presented. The findings for mathematics gains

in Connecticut can be summarized as follows:

125

Results Summary

Section I Results Summary. Contextual Effects of Gender, Ethnicity, and Diistrict
SES on mathematics gains in Connecticut

1. Fixed effects at the student level show significant gains by gender, ethnic
groups, and grade level. Girls gained more in mathematics than boys. Black, Hispanic,
and All other students gained more in mathematics than White students over the course of
one year. Coefficients for all groups are significantly different from zero with the
exception of All other students which is nearly significant at p<.053. There are no gender
by ethnicity interactions.

2. After grade level is introduced to the model, the patterns for gains by gender
and ethnicity remain the same. Grade level, however, controlling for gender and ethnicity
is a large predictor of mathematics gains and is highly significant at p<.001. 4th grade
and 6th grade each account for nearly 1 additional point in mathematics gains over the
course of the year. Curiously, being in Grade 8 controlling for gender and ethnicity does
not increase gains over the year.

3. Most of the variation in Connecticut mathematics classroom is between
students (91 %). About 9% of the variation is located at the teacher level.

4. District SES varied significantly across teachers when predicting student
mathematics gains by gender and ethnicity. Being in an average SES district (21% free
and reduced lunch) was associated with a -6.44 loss in gains for boys when compared to
girls and a -19.14 loss in gains for Black students when compared to White students.
There are two Connecticut districts in this data that have more that 42% of it’s students

receiving free and reduced lunch. In these districts, the losses for boys and Black students

126

are predicted to double.

Average district SES explains 23.2% of the variation in female gains and 6.1% of
the variation in males gains across teachers. It affects male and female gains
differentially — boys lose -6.44 points while girls gain +7.9 points for a one standard
deviation increase in average district free and reduced lunch.

Average district SES explains 46.5% of the variation in Black gains across
teachers but only 2.2% of the variation in White gains. Average district SES affects
Black and White gains differentially — Black students are predicted to lose over -19 points
in an average SES district while White students gain +1.4 points across teachers.

Section II Results Summary. Teacher practice, characteristics, and professional
development effects on student mathematics gains in Connecticut

4. Average mathematics gains for boys and girls controlling for SES varies across
teachers. There is twice as much variation in average male gains compared to female
gains across teachers but teacher variables generally explain more of the variation that
exists in average gains for girls across teachers once SES is controlled

a. Teacher practice variables controlling for SES explain 5.7% to

8.6% of the variation in boys’ average mathematics gains. The same

practice variables explain 23% to 38% of the variance in girls’ gains

across teachers. Mathematics sensemaking, drill and lecture, and group

and project activities have differential effects on average gains for boys

and girls across teachers. Boys, on average, gain more than half a point for

a one standard deviation increase in mathematics sensemaking and group

and project activities while girls lose about one point over the course of

127

the year. The opposite is true for mathematics drill and lecture activities.
Girls gain about half a point and boys lose about one/third of a point for a
one standard deviation increase in drill and lecture activities controlling
for SES.

b. Teacher characteristics of reform orientation, preparation for
assessment, and efficacy explain up to 41 .7% of the variance in girls’
gains across teachers controlling for SES, more variation than SES alone.
The same variables only explain up to 7.9% of the variance in boys’ gains
across teachers. All three have differential effects on boys and girls across
teachers controlling for SES. In general, girls lose up to 1 point over the
course of the year while boys gain up to half a point for every standard
deviation increase in these three teacher characteristics.

Mathematics preparation is negatively related to gains for both
boys and girls. A one standard deviation increase in mathematics
preparation results in a -.86 point loss for boys and a -.18 point loss for
girls across teachers controlling for district SES. Similar results are
reported by Valli and Reckase (2000) and Correnti, Rowan, and Miller
(2002)

c. One teacher professional development variable, mentoring
frequency, explains more variation in girls gains across teachers than SES
alone. For a one standard deviation increase in mentoring frequency, boys
remain about level (-.08) and girls lose -1 .41 points. This explains about

28% of the variation in girls’ average gains across teachers.

128

5. Average gains for Black and White students controlling for SES varied across
teachers. There is 3 times as much variance in Black gains compared to White gains
across teachers once SES is controlled. Average gains for Hispanic and all other students
did not vary significantly from White students.

a. Teacher practices explain little about the variation in average
mathematics achievement across teachers for Black and White students. The
teacher practice variable for mathematics sensemaking controlling for SES
explains 47.5% of the variation in Black average mathematics gains across
teachers, approximately 2% more than SES alone. The same variable explains no
additional variation in White gains. For every one standard deviation increase in
sensemaking activities, Blacks gain +1.25 points and Whites gain +.18 points.
Drill and lecture activities is the only variable which explains more variation in
White gains than SES alone, about 2%. For every one standard deviation increase
in drill and lecture activities, white students lose -.27 points and Blacks gain +.74
points, a differential effect.

b. Teacher characteristics for math preparation and reform orientation
controlling for SES explain up to 57% of the variation in Black gains and up to
25.6% of White gains across teachers. Math preparation explains the largest
amount of variation for both groups across teachers. It accounts for an additional
23% of the variation in average White gains and 12% of the additional variation in
average Black gains over SES alone. For every one standard deviation increase in
mathematics preparation, White students lose nearly a point and Black students

lose nearly -2 points over the course of the year.

129

Reform orientation explains about 10% more variance in Black gains
across teachers than SES alone. For every standard deviation increase in reform
orientation, Black students gain +1.33 points across teachers. White students’
average gains remain nearly level and it explains no additional variation.

c. Of the professional development variables, only professional
development frequency explains additional variance in Black gains when SES is
controlled, about 6% more. The effect of professional development frequency is
differential. For every standard deviation increase in professional development
frequency, Black students gain nearly a point while White students lose a point.
Professional development impact and mentoring frequency explain additional
variation in White gains across teachers, 7% to 40% more than SES alone. For
every one standard deviation increase in professional development impact, White
students lose half a point while Black student gains remain nearly level across
teachers. For every standard deviation increase in mentoring frequency, White

students lose -.77 points and Black students lose nearly -1.25 points.

Section III Results Summary. University teacher preparation program effects on
student mathematics gains and teacher variables.

6. Average 4th-8th grade student mathematics gains produced by teachers who

graduated from a specific program did not vary by teacher preparation programs.

7. Only one of the teacher variables that explained variance in average student

scores across teachers also varied across teacher preparation programs, mentoring

frequency. Both program quality (19.3%) and field experiences (17.4%) explained

variance in average teacher mentoring frequency.

130

Although it did not explain variance in average student scores across teachers,
mathematics knowledge varied across teacher preparation institutions. None of the
measured university level teacher preparation variables (faculty characteristics, program
quality, field experience, and program rank from the initial state report card) explained
variance in average teacher knowledge across teacher preparation programs.

It is interesting to note that program rank based on pass rates reported in the Initial
State Report Card did not explain variance in average mentoring frequency or
mathematics knowledge.

8. It is possible to rank teacher preparation programs in Connecticut based on the
fixed effects produced by a two level HLM model. Level two dummy variables for
teacher preparation institutions predict levels of mathematics knowledge and mentoring
frequency at level one. Table 8.31 presents program rankings on the average mathematics
knowledge and mentoring frequency of its graduates.

Table 8.31 Teacher Preparation Programs Ranked on Teacher Graduates’ Mathematics

Knowledge and Mentoring Frequency

’ _ O y I _ I _ ‘d _‘ M—athe—matgs Mentoring
knowledge Frequency

universityo ' V 1 10
I Univgrsity 1 f if ’ 10 f f 2
“Ur—ifvgrgityfiz * 7 i 6 7’ 6 7
Iii-’“Uhivefirsfiity 3V 5 if 1
“01111763184 3 9
w"ti—riitlefsiyfis'm __ _ 9 4 '
’ Uri—veféit—y 6 '4 3__
University 7 2 8
: 03193131193“ 7 8 7
$0,133,363 * 11 11
iThTewefGthfer'U diversities ' 7 5

131

It is important to interpret this table with caution since institutions did not vary widely on
mathematics knowledge which ranged from 26.63 (good) to 34.64 points (excellent).

Mentoring frequency had a larger range from 10.73 (never) to 43.56 points (monthly).

9. RESULTS FROM THE HLM ANALYSIS
OF CONNECTICUT 4th-8‘“ READING GAINS

This section, like the previous section for mathematics, addresses the research

questions about the relationships among student reading gains, and

0 Teacher characteristics and practices
9 Characteristics Of professional development
0 Characteristics of teacher preparation

The analysis also focuses on the ways in which these relationships vary by student
demographic group, including gender, ethnicity, and grade. As before, Tennessee is not
included in the HLM analysis because there is no individual measure of student
achievement available in the BTPS data, only correlation coefficients based on the
aggregation of student scores within teacher.

TO set the context for addressing the questions, results are presented first which
describe the differences in student reading gains in Connecticut across student
demographic groups. The analyses yield results in terms of regression coefficients. To
aid in understanding what the coefficients mean, predicted gains are computed for
selected combinations of student characteristics. As was the case for mathematics,
district percent free and reduced lunch is assigned to each teacher as a measure of
classroom socioeconomic status since there was no individual student indicator available
in the BTPS data.

The full set Of reading variables for Connecticut used in these analyses can be

found in Table 9.1.

133

Table 9.1 Description of Variables Used in the Connecticut Reading Analyses.

Continuous
(Outcome)

Dichotomous

Effects
coded

Continuous

Z-scores

,, Abbreviation

Variable

readdiff

dblack
dhispanic
dallother
dfem
dgrade4
dgrade6

egenxbl
egenxhisp

egenxoth

ztradlit
zreform
zlitmat
zcomp
zdivprep
zblito
zEngprep
zlitknow

zreadinst

zassessprep

zefficacy

Variable Description

Level 1 - Students
Reading gain, difference between pre- and post-test scores
(mean = 3.40, sd = 7.71)

Black (Black=1, else=0; mean=16.0, sd=.37)

Hispanic (Hispanic=1, else=0; mean=.09, sd=.29)

All other ethnicities (all other=1, else=0; mean=.08, sd=.27)
Female (female=1, else=0; mean=.53, sd=.50)

Grade 4 (grade 4=1, else=0; mean=.43, sd=.50)

Grade 6 (grade 6=1, else=0; mean=.32, sd=.47)

Effect of Black female; interaction of eBlack x egender (Black
=1, White=-1, otherwise=0 multiplied by female=1, male=-1;
mean=.01, sd=.91)

Effect of Hispanic female; interaction of eHispanic x egender
(Hispanic=1, White=-1, otherwise=0 multiplied by female=1,
male=-1; mean=.01, sd=.87).

Effect of All other female; interaction of eAIIother x egender (All
other =1, White=-1, otherwise=0 multiplied by female=1, male=-
1; mean=01, sd=.87)

Level 2 - Teachers
Traditional literacy activities (mean=0, sd=1)

Reform literacy activities (mean=0, sd=1)

Literacy materials usage (mean=0, sd=1)

Computer activities (mean=0, sd=1)

Preparation for diversity (mean=0, sd=1)

Balanced literacy orientation (mean=0, sd=1)

English preparation (mean=0, sd=1)

Self report of literacy knowledge (mean=0, sd=1)
Reading instruction, reading pedagogy (mean=0, sd=1)
Preparation for assessment (mean=0, sd=1)

Teacher sense of efficacy, that a teacher can make a
difference in students' learning (mean=0, sd=1)

134

Table 9.1 (cont'd)
zreadprep Reading preparation (mean=0, sd=1)

zpdimpact Professional development impact, teacher’s sense of the
impact that professional development had on teaching
(mean=0, sd=1)

zpdfreq Professional development frequency (mean=0, sd=1)
______ 29993- _ Matanafrswrsvtmsfaswi _ _ _ _ _ _ _ _ _
999999999 _, 9999.9 _, , .- 3999999919999 9991999999Neither-13239917)- ,-
Level 3 - Teacher preparation programs

Continuous facchar Faculty characteristics (mean=23.99, sd=1.22)

fieldexp Field experiences (mean=16.32, sd=1.30)

progqual Program quality (mean=18.35, sd=1.50)

rank Rank on the State Report Card (2000), The Initial Report of the

Secretary of Education on the Quality of Teacher Preparation
based on the number of teachers passing the state certification
examination (basic) (mean=6.67, sd=4.27)

1. Setting the Stage: Variation in Reading Achievement Gains by Student
Demographic Groups (Results at the Student Level)

In this section, sets of background variables are added to look at the changes in
reading gains in Connecticut. Model 1 predicts mean student gains in reading across the
sample and partitions the variance between students and teachers where students are
nested within teachers. Models 2-5 predict differences in mean gains across gender and
ethnicity.

Mean Gains. To set the context for answering the questions about the connections
between student achievement and teacher variables, results are presented first for
differences in student achievement gains across demographic groups. The level one and
level two equations for the model are given followed by their estimated regression

coefficients in Table 9.2.

135

Model 1. Mean Reading Gains

Level 1 - Students

Y0=fl01+eij (9.1)

Level 2 - Teachers

:60): = 700+ ’0
where Y ij is the reading gain for student i in teacher j’s classroom, flOj is the intercept or a

student’s mean reading gain within teacher j’s classroom, e is the level 1 error, 700 is

1.].

the mean value of classroom gains for all teachers, and raj is the level 2 error.

Table 9.2 Connecticut’s Average Reading Gain 1999-2000

A ‘ F i ‘ Model 1

113136331800) ’ 7 ’3.4Z(.31)m ”
LVL 1 Variance (02) 53.94
LVL 2‘Variance (100) f 576*“

*** p<.001,“ p<.01,* p<.05,~<.10

_8th

The predicted average gain for 4th grade reading students fOr this sample of students

in Connecticut is 3.44 points (YOO=3.44, sd=.31) when no other variables are included in
the model.

2

o and 1300 are used to compare variation in the errors at the teacher level with

variation at the student level using the formulas:

0’2 ij _ . 700
— level one variance

2 = level two variance (9,2)
0' ij + Too 0' ’7 + TOO

90.4% of the variance in gains for Model 1 is within teachers (between students)

leaving 9.6% of the variance in gains between teachers. This means that there is nine

136

times more variation within teachers than between teachers.

Mean Gains by Demographic Group. Student demographic variables are added to
form Model 2, 3, and 4. Model 5 introduces the control variables for grade level. The
equations for Model 2, ethnic group, are presented first (Eq. 9.3). To interpret the
coefficients, the estimated gain for each combination of gender and ethnicity can be
computed by using the appropriate set of Is and OS for the dummy variables. As an
example, Equations 9.5 and 9.6 demonstrate the process for Obtaining Black student
gains. The estimated gains for ethnicity are presented in Table 9.3.

Model 2. Ethnic groups

Level 1- Students

Y ,1- = flOj + flljdblackij+ flzjdhispanicij+ ,33jdallotherI, + e ij; (9.3)
Level 2 - Teachers

.50; = 1’00 + ’0): (9.4)

.31} = 1’10

:52] = 1’20

:53} = 1’30

The regression coefficients for predicting student achievement gains from ethnic

background are found in Table 9.3.

137

Table 9.3 Regression Coefficients for Connecticut Reading Gain 1999-2000 by Ethnicity

_ _ModeI2
Interc‘ephyoo) * '3.15(.32)***
TEIaCkTVOIf ’ 906(64)"
’in‘ispEnic (Ty/072) 1 .54(.66)* .
All other (v03) 1.67(.65)*
LVL 1Wv3riance102 )3 53.62
‘AL’VEZTVTaArianCe—(TOQ) ’ i i 601*" i i i

p<.001 p<.01,* p<.05,~<.10

TO make it easier to interpret the results from the gains by demographic group analysis,
the coefficients can be used to compute the gain for a student with a given set Of
demographic characteristics. Because the demographic characteristics are coded as 1 or 0,
the computation involves summing the terms which are coded 1 for a particular student.18
For example, to determine the predicted gains for a Black student who would have ones
for the intercept and Black, the resulting equation would be:

Level 1 - Students
" = A " (9.5)

Level 2 - Teachers

.30} :1700 =3~15

.. . (9.6)
131} =1’I0 =06

When the betas from Equation 9.6 are substituted in the level one equation (Equation

 

18 A table showing ones and zeros for all dummy variables is provided in the
Appendix.

138

9.5), the result for Black gain is produced (Equation 9.7).

A

Yi, = (3.15) + (.06)*1 (9.7)
= 3.21

Predicted reading gains for all subsets Of students by ethnicity are presented in Table 9.4.

This table shows that the model predicts White students gain less in reading than the

other groups. Coefficients for all groups but Black are significantly different from zero.

Table 9.4 Predicted Reading Gains by Student Demographic Group

 

 

Ethnic Group

White 3.15
Black 3 .21
Hispanic 4.69
All other 4.82

 

In the next model, gender predicts reading gains.

Model 3. Gender

Level 1- Students

Yij = floj + ,Bljdfem + eI-j; (9.8)
Level 2 - Teachers

1501' = 1’00 + r0j (9-9)

1611' = 1’10

The regression coefficients for predicting student achievement by gender are found in

Table 9.5.

139

Table 9.5 Regression Coefficients for Reading Gains by Gender (CT)

- _. _ 113331‘3‘

Intercept (v00) 3.18(.40)***
Female (V01) .42(.34)

[VET-Variance (02) 53.91

LVL 2 Variance (T00) 5.77***

**T' rT<2001j*‘p<.01,* p<.05,~<.10
These coefficients indicate that males are predicted to gain 3.18 points in reading on
average over the course of the year and females are predicted to gain 3.60 points. The
coefficient for female is not significantly different from zero.
Model 4. Combined gender and ethnicity
Model 4 tests whether or not there in an interaction between gender and ethnicity when
predicting reading gains. Equation 9.10 computes coefficients for combinations of
gender and ethnicity.
Level 1- Students

Y ij = flOj + flljdblackij+ flzjdhispanicij+ fl3jdallotherij+ fl4j dfemij

+ flsjegenbeIj+ ,Béjegenxhispij+ )67jegenxothij + e (9.10)

ij;

Level 2 - Teachers

160) = 1’00 + ’0):

fly = 1’10
.32)- = 1’20
:63j = 1’30
:64) = 1’40
1651' = 1’50

140

An = 1’60
1671' = 1’70
The regression coefficients for predicting student achievement gains from ethnic

background and gender are found in Table 9.6.

Table 9.6 Regression coefficients for Connecticut Reading Gain 1999-2000 by Gender

and Ethnicity
Model 4 ’
”ina'cépirvoo) ’ i 7 7 7 '2.77I(.47)*1* *
Black (v01) -.14(.66)
7112531113er P i i i ’ "1.361.669 ’
All other (v03) 1 .70(.61)**
Female (v04) .81 (.53)
Effect Black x female (v05) .33(.44)

Effect Hispanic x female (V06) .54(.48)

. Effect All other x female (v07) -.49(.47)

LVL 1 Variance (o2 ) 53.58

LVL 2 Variance (T00) 603*“

*** p<.001,** p<.01,* p<.05,~<.10

To aid in interpreting these coefficients, an example using Black female students is

presented next. The process is then repeated for all other groups using the coefficients

from Table 9.6.
Level 1
17,-]- : 8‘0,- + Ajdblackij + [314]. dfemI-j + fisjdblackfemij (9.11)
Level 2
00,. = 2.77
31]: = -.14
.. (9.12)
fl‘ij = .81
flSj = 33

141

The estimated beta values are calculated first and then substituted into the level one

equation (9.9). The sample computation for Black female is demonstrated in Equation

9.11.
1’.)- = 3.15 + (-.86)*1 + (.04)*1 + (1.43)*1 (9.13)
= 3.76
Using this approach for each subgroup, expected reading gains by demographic groups
over 1 year are presented in Table 9.7. The model predicts that girls gain more in reading
than their male counterparts. Black males gain the least of all groups. The coefficient for
Black is not significantly different from zero which implies that Black predicted gains are

not statistically different from the intercept or White predicted gains for this sample.

Table 9.7 Predicted Reading Gain by Student Demographic Groups with Differences by

 

 

Gender
Males Females Difference in ,
Gains by Gender
White Students 2.77 3.65 + .88
Black Students 2.63 3.84 + 1.21
Hispanic Students 4.13 5.48 + 1.35
All other Students 4.47 4.79 + .32

 

Hispanic and All other students, groups which include the students most likely to have
English as a second language, have higher average reading gains than their White and
Black classmates by at least one point over the course of one year. Coefficients for the
intercept, Hispanic, and All other are Significantly different from zero.
Gender, Ethnicity, and Grade Level

Another known source of variation in student achievement is grade level. In the
earlier SEM analyses, the outcome for gains was standardized by grade level. In the

upcoming NAEP analyses, only one grade is represented in each analysis. Consequently,

142

dummy variables for grade are introduced at level one in the Connecticut analyses to
control for grade level. The reference category for Model 5 is therefore white, male
students in the 8th grade.
Model 5. Gender, ethnicity, and grade level
Level 1- Students
Y ij = IBOj + flljdblackij+ ,szdhi.spanicij+ fi3jdallotherij+ ,5,“- dfeml-j (9.14)
+ ,55jegenxbl,j+ fl6jegenxhispij+ ,B7jegenxothIJ-
+flgjdgrade4ij+ flgjdgrade6ij+ e ij;

Level 2 - Teachers

160) = 1’00 1" ’0):

.31} = 1’10
.52): = 1’20
:53) = 1’30
.34) = 1’40
:65) = 1’50
fl6j = 1’60
fly = 1’70
168} = 1’70
.39} = 1’70

The regression coefficients for predicting student achievement gains from ethnic

background, gender, and grade level are found in Table 9.8.

143

Table 9.8 Regression Coefficients for Connecticut Reading Gain 1999-2000 by Gender,
Ethnicity, and Grade

Model 4
Intercept (v00) 3 ’-.01 (.65)
u Black (v01) ’ -.35(.66) ’
"fin—ispflanic (1702) ‘ 1.27(.64)*
’ _Allother(_y03) 1.67(.61)“
F Female (v04) i.81(.53)

EffectBlack x female (v05) .39(.45)
Effect Hispanic x female (V06) .50(.48)

—Effect All other x female (v07) -.51 (.47)
4th grade (V08) 4.28(.72)***
6th grade (v09) 2.91(.66)***

LVL 1 Variance (02) 53.61

LVL 2 Variance (TOO) 3.25m

*7" b<.001,**p<.o1,* p<.05,~<.10

Predicted reading gains for all subsets Of students by demographic group and
grade are presented in Table 9.9.19 There is a large, statistically significant difference in
gains by grade with students in the 4th grade gaining the most, followed by those in the

8th

6th grade, and with the smallest reading gain made by students during grade.

Table 9.9 Predicted Mathematics Gains by Student Demographic Group and Grade

 

Grade 4 (+4.28) Grade 6 (+2.91) Grade 8

Males White 4.27 2.90 - .01
Black 3.92 2.55 - .36
Hispanic 5.54 4.17 1.26
All other 6.23 4.57 1.66
Females White 5.08 3.71 .80
Black 5.12 3.75 .84
Hispanic 6.85 5.48 2.57
All other 6.24 4.87 1.96

 

 

'9 To see a sample Of the calculations performed to create Table 9.8, please refer
to p. 35, Eq. 9.9-9.11.

144

Hispanic and All other students, those most likely to have English as a second language,
are still expected to gain more in reading over the course of a year than White and Black
students once grade is taken into account. Note that the coefficients for Hispanic and All
other remain significantly different from zero for this analysis. Girls gain more than boys
within each group.
Sammarv of Section 1. Developing the level one equation for students. As the level one
one base model for reading in Connecticut developed, average yearly gains were first
estimated with a simple unconditional model. The model revealed that average gain
within teachers (between students) was nine times larger than across teachers at level two.
Variables know to affect mean gains (ethnicity, gender and grade) were added to the level
one equation indicating that while differences in gains exist, only the coefficients for
ethnicity and grade level are significant predictors of reading gains when gender is
controlled. By far, the most significant predictor of reading gains was grade level with
scores declining as students progressed by grade. There are no significant interactions
between gender and ethnicity. Additionally, the highest gains were predicted for students
from ethnic categories that likely contained the most ESL students.
II. Differential Effects of Teacher Practices, Teacher Characteristics, and
Professional Development on Student Reading Achievement Gains (Results at the
teacher level)

This section addresses the following questions:

1. DO reading achievement gains for demographic groups vary across

teachers?

145

2. How much variation in student reading gains do variables about teacher

practices, teacher characteristics, and professional development explain?

3. Is the amount of variance explained by teacher practices, teacher

characteristics, and professional development different for different
student demographic groups?

Analyses in this section first test if the achievement gains for the various
demographic groups vary across teachers. In other words, is the variance significantly
different fi'om zero: There is a brief discussion Of variance across teachers followed by
changes to the results when district SES is controlled. This establishes the level two
context for a series of analyses which look at the effects of teacher practice, teacher
characteristics, and professional development on the variance of reading gains.

Variance in Connecticut reading achievement across teachers

Unlike mathematics, the X2 test that HLM provides indicated that average reading
gains for ethnicity and gender did not vary across teachers (100:0) but X2 test for the
effect Of Black was based only 56 Of 90 available teachers who had sufficient numbers Of
Black students in their classrooms. An alternate test to compute significance using
change in deviance over degrees of freedom indicated p<.01.20 In other words, average
black gains do vary across teachers. A subsequent series of analyses for variance
explained in Black average gains across teachers revealed that no variance in average
reading gains for black students was explained by the selected teacher variables so the

results are not included in this dissertation.

 

20 Deviance: 12l74-12l64=|10|, degrees of freedom: 2-4=|—2|

146

Consequently, the next series of analyses focus on question two, how much
variance is explained by teacher level variables on the average reading gains for all
students in the sample.

Effects of teacher level variables on mean reading gains in Connecticut

The level one equation for ethnicity and grade is presented as Model 6. Note that
the non-significant interaction and error terms for gender by ethnicity have been dropped.
Only the intercept is allowed to vary. The dummy variable for female remains in the
equation as a control.

Model 6. Unconditional model for average reading gains
Level 1- Students

Y ij = flOj + flljdblackij+ flzjdhispanicij+ fl3jdall0therij +fl4j dfeml-j (9.15)

+fl8jdgrade4ij+ flgjdgrade6ij+ el-j;

Level 2 - Teachers

:60)" = 1’00 T ’0): (9-16)
:51} = 1’10
:52) = 1’20
:33) = 1’30
I64j = 1’40
.68) = 1’70
:59) = 1’70
where
YOOj = intercept, average student gains for all teachers
to, = the error term or unique teacher effects

147

The estimated level two regression coefficients for Model 6 are given in Table
9.10.

Table 9.10 Regression Coefficients for Average Connecticut Reading Gain 1999-2000
Across Teachers

8 Model 6

lhtercepiwoo) ’ .18(.65f
-BIack (v10) 2 i f i i 3 927(64)
-_ 'HiSphnic (112—0)" ' __ _ ' i _ _ 1.40(.63)9

All other ethnicities (v30) 1.59(.64)*
'ffemalewm) .43(.34) 1

4th grade (V80) 4.26(.72)***

6th grade (v90) 2.89(.66)***
LVLA1 Variance (02) 3 153.65
LVL 2 Variance Intercept (T00) ’ 325*“

p<.001," p<.01,* p<.05,~<.10

Selected coefficients from this analysis serve as the base for the variance decompositions
found in Tables 9.12 and 9.13. Next, Model 7 expands Model 6 to include district SES
(average free and reduced lunch).
Model 7. Average reading achievement gains and SES
Level 1 - Students (same as Eq. 9.1 1)
Level 2 - Teachers
1501' = 700 + yOIpctfrlu + roj (9.17)
:51)" = 1’10
1621' = 1’20
:33): = 1’30
fl4j = 1’40

148

fl8j = 1’70

169' : 1’70
where
YOOj = intercept, average student gains for all teachers
you = change in average gains associated with mean district free and reduced

lunch, all other predictors held constant
r0, = the error term or unique teacher effects
The estimated level two regression coefficients and variance components for
HLM Model 7 are given in Table 9.11.

Table 9.11 Regression Coefficients for Average Reading Gain by Average District SES

Model 7 1
16333353650) 8 if i h I if ’ 8’.16(.66)
.. fl r83 andreduced lUnch (V06) 189—(1.78)
Black (v10) -.18(.66)
lll-lispa’nicwzo) ’1.52(.69)*
I All otherethnicities (v30) ’ 164(64)::
l_l‘=einaglv4o)’ 3 1441.347 T
“4’5 grade (V801 3 ’ 4.24(.73)***
V 393.333 (v50) F ’ 2.88(.68)*** *
LVL 1Variénce162) * 53.64 ’
LVLTZAV‘arTanvcelnterpotUOO) if 332*“ ’8

*** p<.001,“ p<.01,* p<.05,~<.10

Table 9.12 presents the variance decomposition for SES. Unlike mathematics, average

district SES (22% free and reduced lunch”) does not explain variation in average student

 

2' This sample includes different teachers and districts than the mathematics
sample which changes the district free and reduced lunch percentage by 1%.

149

reading gains across teachers.

Table 9.12 Variance in Average Reading Gains Explained by SES

-A_._..’ __ A _ A 7 i A A # i i A,, 7.

Outcome: Read diff Model 6 1 Model 7
unconditional SES

lHtcheAptivoof * * 118(65) i * ..16(.66) *

TSTESA (v01) 8 A h * i-.89(1.78)

Iv]; {variénéelom “58.65 l “53.64

TVL 5 variance (TDD) 8325*" '3.32***

eavariancé explained. * A * .00/o if "

*** p<.001," p<.01,* p<.05,~<.10

One standard deviation in increase in free and reduced lunch reduces average student
reading gains across teachers by nearly one point over the course of one year. Although it
does not vary across teachers, SES is retained in the teacher effects analysis as a control.
Teacher Effects on Reading Achievement

Using the same equations as presented for mathematics (EQ 8.18), variable x is
replaced with teacher level variables for teacher practices, charactersitics, and
professional development one at a time. This is the same process outlined for the
mathematics decompositions in Chapter 8. One classroom practice (computer activities),
one teacher characteristic (English preparation), and one professional development
variable (professional development frequency) explained variance in average student

scores across teachers. Results are presented in Table 9.13.

150

Table 9.13 Variance Decomposition Table for Average Reading Gains Across Teachers
in Connecticut

Outcome: Read diff Unconditional 7 SES Computer -- 7 English f Professional
Activities Preparation development
frequency
Interdepuvoo) .18(.65) 116(66) * 1.13(.66) A A fl.58(.63)A F 104643" -_.
TTSETSWIM) * i ’ f ’ ’-.39(1.73) ’-2.47(1.91) ”439(155)?"1.558.673 "
T FT ’(voz) T i M H H i i i l A I .-.7#0(.2€i)*_ h b-.80(.2—8.)u — .-.5‘3(.38)v H
[vi 1—V_ari—ance(o2_) ”536—5— ‘ ' 53.64" ' ”63.252— “ ' 33.66 _ " "”5365" ‘
'LVL’EVéEaaECé (Tovo)fl8j2g;** * i ” $.52?" * i 3.93""? i i "370??” T 3717*? i
% variance explained 0% 8.3% 7.3% 4.3%

***“ p<.001," p<.01,*p<.05,~<.10

Computer activities explain 8.3% of the variance in student gains across teachers
controlling for SES. A one standard deviation increase in computer activities decreases
student gains by -.70 points. English preparation, like mathematics preparation, is
negatively related to student gains. For every one standard deviation increase in teacher
English preparation, -.60 points in reading gains are lost across teachers over the course
of one year. English preparation explains 7.3% of the variation in average student gains
across teachers. Professional development frequency explains 4.3% of the variation in
student gains across teachers. About half a point is lost for every one standard deviation
increase in professional development frequency.

Simmary of the effects of teacher variables on reading gains in Connecticut. It is
striking that the results from the reading analysis are so different than the mathematics
results. Average gains by gender and ethnicity do not vary across teachers. SES does not
explain any of the variance in average student reading gains across teachers. Computer
activities, English Preparation and, professional development frequency account for

variance across teachers in Connecticut but they are negatively related to average student

151

gains across teachers. One standard deviation increase in them decreases student gains by

half to three quarters of a point over the course of one year.

III. IHE Teacher Preparation, Teachers, and Student Achievement.

Section III section addresses questions about university teacher preparation
programs and their connections to teacher practices, characteristics, professional
development, and subsequent student achievement in reading. In other words, do
Connecticut teacher preparation programs vary in the average levels of teacher practice,
characteristics, and professional development produced by their programs and do teacher
preparation programs predict the teacher variables associated with average student
achievement across teachers?

Effects of teacher preparation program on student reading achievement. The
analysis in this section uses a smaller sample of teachers who indicated that they attended
a teacher preparation program in Connecticut. There are 1285 students nested within 65
reading teachers within 13 identified university teacher preparation programs in
Connecticut.

The unconditional three level analysis shows no significant variation in average
student reading gains within teachers across teacher preparation programs. No further
investigation into variance explained in student gains across teachers by teacher
preparation programs was possible for this data.

Effects of teacher preparation institutions on teachers. This section addresses
questions about Connecticut teacher preparation programs and their connections to

teacher practice, characteristics, professional development. Afier finding relationships

152

among student outcomes and certain teacher practices (computer activities),
characteristics (English preparation), and professional development (professional
development frequency), the next logical step is to explore the connections between
teacher preparation institutions and those particular teacher variables.

A two level model (Model 8) with Connecticut teacher preparation programs at
level two predicts one of three teacher variables at a time on level one (computer
activities, English preparation, and professional development frequency). Results for the
variance explained in average teacher variables across teacher preparation programs are
presented in Table 9.22. This analysis included 67 reading/literacy teachers nested within
13 teacher preparation programs.22
Model 8. Base model for teacher variables across teacher preparation institutions.

Level 1 - Teacher

variable xi]- = 30 + eij (9.17)

Level 2 - IHE teacher preparation program
:60} = 1’00+ ’01 (9-18)
The coefficients for the intercepts and variances for computer activities, English

preparation, and professional development frequency can be found in Table 9.14.

 

22 Fixed effects are unbaised although standard errors are likely too small in
small samples (Raudenbush, p. 281).

153

Table 9.14 Unconditional Models for Levels of Teacher Variables Across Teacher
Preparation Institutions

Unconditional Models Computer 9 - English Professional
(outcomes) Activities Preparation Development
7 Frequency
Intercept (v00) 8.04(.32)*** 3.75(.37)*** 18.57(.35)***
LVL1Variance (02) 6.94 3.93 4.35
LVL2Variance(T00) .01 ,, g f .71“ .53

p<.001,“ p<.01,* p<.05,~ p<.10

Of the teacher variables that were related to variation in student reading gains across
teachers, only average English preparation varies significantly (p<.05) across teacher
preparation institutions. 15% of the variance is located at the teacher preparation level
and 85% of the variance is between teachers.

In Model 9, qualities of teacher preparation programs predict teacher English
preparation. Variables available at the university level are faculty characteristics, field
experiences, and program quality as reported by program graduates as well as rank on the
Initial State Report Card based on teacher certification examinations.

Model 9. Base model to predict English preparation by teacher preparation program
characteristics
Level 1 - Teacher
EngPrepl-j 2 BO + eij (919)
Level 2 - IHE teacher preparation program
IBOj = 700 + 701 program characteristic y + rOj (9.20)
where the program characteristic y is faculty characteristics, program quality, field

experiences, or program rank.

154

Table 9.15 Connecticut’s Average English Preparation by Teacher Preparation
Institutions

'English preparation unconditional- field 7 faculty , Program Rank on
experiences characteristics Quality State Report

Card

lntercept(v00) 3.75(.37)*** 3.70(.35)*** 3.73(.35)*** 3.86(.38)*** 3.80(.40)*f*

- SV1o) -- g g g .4403?» - 440027) - raw-3.5) --05<-09>

LVL1Variance (02) 3.93 f 3.93 3.88 3.88 3.89

LVL 2Variance('r00) .71* .59* .79* .80* .91*

Variance explained 16.9% 0% 0% 0%

p<.001 p<.01,* p<.05,~<.10

Field experience at the university level explains 16.9% of the variance in English
preparation across teacher preparation institutions. For every one standard deviation
increase in field experiences, teacher English preparation increases by .44. This is
roughly equivalent to half of a major in English, English education, or half a course in
English, writing, or literature.
Effects of specific teacher preparation institutions on reported teacher practices.

A two level analysis (Model 9) uses dummies for teacher preparation institutions on level
two to predict English preparation by program.
Model 9. 13 teacher preparation institutions predicting English preparation
Level 1- Teachers

teacher variable xij 2 30 + ea- (921)

Level 2- University teacher preparation programs

300 = 700 + 701 Univ] + 702 Univ2 + 703 Univ3+ y04Univ4
+705Univ5 + 706Univ6 + y07Univ7 + yogUniv8 (9.22)
+ yogUniv9 + leAIIOIhUniv

The university teacher preparation coefficients predicting English preparation are

155

presented in Table 9.16 followed by program rankings in Table 9.17.

Table 9.16 Regression Coefficients for Teacher English Preparation by Teacher

Preparation Institutions

Intercept (V00)
University 1 (v01)

8 Drfivérsityzwoz) ’ i ’

”"DBiv2r;Itya‘(Qog) 7’ i i

University4 (V94) 3

' University 5 (v05)

" ’uBivergityé(yoé) ’5
University 7 (y07)

if University8 (V93)
University 9 (v09)

Three Other Universities (v30)

LVL 1 Variance (02)

LVL 2 variance (T00)
Variance explained at Lvl 2

3.75(.37)***

3.93

*** p<.001, ** p<.01, * p<.05, ~ p<.10

Unconditional

English .. 3
preparation

3 99(. 85) --
- 63(3. 99)
-2 23(3. 98)

2133 97)

.A i i 7 A

-1 43(3. 94)
..95(4 02)

-3 30(4 15) if

.20(4 01)
1.301395) *
120(437)

52.6961) &

3 79
7. 26*
0%

Table 9.17 Teacher English Preparation by 13 Connecticut Teacher Preparation

Programs (low to high)

University 6

University 2
University 3

3 University 8

University 4

University 1

Three Other UnIverSItIes
University 0

University 7

University 5

University 9

0.69

176

1.81
2.19
2.56
3.36

3.73
3.99
449
i 4.94

5.19

156

The maximum score for this scale was 12. Interpretation is complicated. An average
program score of less that one point means roughly that reading teachers from this
program have degrees in majors other than English or English education with no
coursework in English, writing, and literature. A score of 1-2 means roughly that
universities whose graduates score in this range average up to an undergraduate minor in
English or English education or up to 5 courses in English, writing, and literatures or an
equivalent combination. 3-4 points is roughly equivalent to an undergraduate major with
additional coursework up to a master’s degree or an equivalent amount of coursework.
Three universities scored over 4 points indicates at least Masters degree in English or
English Education with up to two additional courses in English writing or literature or an
undergraduate major with at least 3-5 additional courses.

Although it is possible to rank programs on the amount of English preparation
their teacher graduates have had (fixed effects), the addition of 13 programs to the model
does not explain variance across teacher preparation institutions and coefficient
significance could not be calculated by HLM.

Summary of the HLM results for reading gains in Connecticut.

Once the contextual effects for gains by ethnic and gender groups controlling for
grade level were estimated at level one, it is possible to answer the main research
questions regarding how those relationships changed when teacher practices,
characteristics, and professional development were added. Then the results for student
outcomes and teacher variables predicted by teacher preparation program in Connecticut
are presented. The findings for reading gains in Connecticut can be summarized as

follows:

157

Results Summary

Section I Results Summary. Contextual effects of gender, ethnicity, and district SES
on reading gains in Connecticut

1. Fixed effects at the student level show significant gains by ethnicity and grade
level. Hispanic, and All other students are predicted to gain more than White and Black
students over the course of one year and this gain is significantly different from zero. The
categories for Hispanic and All other are also the ones most likely to contain English as
second language students. Gender does not play a significant part in gains although fixed
effects predict that girls tend to score slightly higher in reading than boys. There are no
significant interactions between ethnicity and gender.

2. After grade level is introduced to the model, the estimated gains by
demographic group pattern remains the same with Hispanic and All other students
outpacing their White and Black counterparts. Grade level, however, is the largest
predictor of reading gains in Connecticut and highly significant (p<.000). All students
are predicted to gain 4 points from the beginning to the end of the 4th grade year. During
6th grade, students add nearly 3 points to their reading gains. Black and White students
gain less than one point in reading during 8th grade. Hispanic and All other students gain

an additional +1.5 points in reading during 8th

grade.
3. Most variation in Connecticut reading classrooms is between students (90%).
Only about 10% of the variation is located across teachers.

4. Average district SES measured at the teacher level did not vary significantly

across teachers when gender, ethnicity, and grade level were controlled.

158

Section II Results Summary. Teacher practice, characteristics, and professional
development effects on reading gains in Connecticut

5. Average reading gains did not vary significantly by demographic groups or
gender across teachers in Connecticut when gender, ethnicity, and grade level were
controlled. Consequently, there are no differential effects to report.

a. Of the teacher practices variables introduced at level two, only the
practice of computer activities explained average reading gain variance across
teachers (8.3%). For every one standard deviation increase in the use of computer
activities, students are expected to lose about three quarters of a point over the
year.

b. Of the teacher characteristics variables introduced at level 2, only
English preparation accounted for average reading gain variance across teachers.
English preparation explained only 7.3% of the variance in average reading gains
across teachers. For every one standard deviation increase in teacher efficacy,
average student gains dropped by -.60 points over the course of the year.

c. Of the teacher professional development variables, only professional
development frequency explained average student reading gain variance across
teachers (4.3%). For every one standard deviation increase in professional
development frequency, average student gains dropped by half a point.

Section III Results Summary. University teacher preparation program effects on
Student reading gains and teacher variables

9. Average 4th-8th grade student reading gains produced by graduates of specific
teacher preparation programs did not vary across Connecticut teacher preparation

Programs in a three level model.

159

10. Only one teacher preparation variable that explained variance in average
student scores across teachers also varies across teacher preparation institutions, English
preparation. 15% of the variance in teacher English preparation is attributable to teacher
preparation programs while 85%of the variance is between teachers. Increases in English
preparation are related to decreases in student scores.

11. Of the institutional variables available in the BTPS data, faculty
characteristics, program quality, and field experiences, only field experience accounted
for significant variation in English preparation (16.9%). Note that the program ranking
from the Initial State Report Card did not predict levels of English preparation, the only
variable shown to vary across Connecticut institutions.

12. It is possible to produce fixed effects rankings of teacher preparation
programs based on levels of teacher English preparation (see Table 9.17); however,
knowing specific teacher preparation programs does not account for the variance in

average English preparation across institutions.

160

10. RESULTS OF THE HLM ANALYSIS
FOR 1996 NAEP 4TH GRADE MATHEMATICS

The HLM analysis using NAEP data is intended to address the research questions
about the relationships among student achievement and

0 Teacher characteristics and practices

. Characteristics of professional development
The analysis also is intended to address the ways in which these relationships vary by
student demographic group, including gender and ethnicity, across the United States and
to see if the national mathematics results reflect trends similar to those found in the
Connecticut BTPS data, a more detailed data set. Analyses concerning the relationships
between teacher preparation institutions and teachers/students are not possible for this
data since the names of teacher preparation programs are not collected in NAEP. The
HLM analyses are weighted (origwt) and use five plausible values (mrpcm1-5) for the
student outcome. There are 4,409 students and 584 teachers in the 1996 mathematics
sample.23

To set the context for addressing the questions, results are presented first which

describe the differences in student achievement (achievement status in NAEP data) across
student demographic groups. The analyses yield results in terms of regression
coefficients. To aid in understanding what the coefficients mean, predicted achievement

(achievement status score) is computed for selected combinations of student

 

23 The n varies from that in the structural equation models due to the number of
variables included in maximum likelihood imputation vs. listwise deletion when creating
the .mdm file in HLM.

161

characteristics.

The full set of mathematics variables for NAEP 1996 is given in Table 10.1. The
outcomes and most teacher variables are the same as those used for the earlier SEM
models.

Table 10.1 Description of Variables Used in the NAEP 1996 4th Grade Mathematics

Analyses
Type Variable Variable Description
Abbreviation
Level 1 - students
Continuous mrpcm1 15t of 5 multiple plausible values for the achievement of
(Outcomes) each student (mean=222.38, sd=30.53)
mrpcm2 2nd of 5 multiple plausible values for the achievement of
each student (mean=222.51, sd=30.56)
mrpcm3 3rd of 5 multiple plausible values for the achievement of
each student (mean=222.78, sd=30.33)
mrpcm4 4th of 5 multiple plausible values for the achievement of
each student (mean=222.18, sd=30.67)
mrpcm5 5th of 5 multiple plausible values for the achievement of
each student (mean=222.64, sd=30.06)

Dichotomous dblack Black (Black=1, else=0; mean=.18, sd=.38)

dhispanic Hispanic (Hispanic=1, else=0; mean=.16, sd=.36)

dallother All other ethnicities (all other=1, else=0; mean=.07, sd=.25)

dfem Female (female=1, else=0; mean=.51, sd=.50)

SES Average free and reduced lunch (eligible=1, else=0 mean=,
sd=)

Effects coded egenxbl Effect of Black female; interaction of eBlack x egender
where Black =1, White=-1, otherwise=0 multiplied by
female=1, male=-1 (mean=.01, sd=.88)

egenxhisp Effect of Hispanic female; interaction of eHispanic x
egender where Hispanic=1, White=-1, otherwise=0
multiplied by female=1, male=-1 (mean=-.02, sd=.87)

egenxoth Effect of All other female; interaction of eAIIother x egender
where All other =1, White=-1, othenNise=0 multiplied by
female=1, male=-1 (mean=-.01, sd=.81)
_ Continuous Origwt Original weight (correction for oversampling)

162

Table 10.1 (cont'd).

Continuous

Teacher
Practices

993

gpaasses

sensemaking

Level 2 - teachers
Grand mean centered

Frequency of group and project activities (mean=11.97,
sd=2.66)

Emphasize group and project assessment (mean=7.56,
sd=2.41)

Frequency of sensemaking activities (mean=10.47, sd=2.71)

Teacher

Characteristics

mathprep

NCTM
mathknow

mathinst

divprep

Teacher’s reform orientation (mean=3.02,sd=1.60)

Major/coursework/license in mathematics, mathematics
education (mean=.59, sd=1.00)

Knowledge of NCTM standards (mean=1.88, sd=.90)
Math knowledge (mean=7.46, sd=.91)

Preparation for mathematics instruction, pedagogical
coursework (mean=3.34, sd=1.01)

Preparation for diversity (mean=1.65, sd=1.11)

Professional

Development

Continuous

Average percent free and reduced lunch per teacher/

_vclavssrogm aggregated from level 1 (mean=.39,sd=.49)

1. Setting the Stage: Variation in Mathematics Achievement Gains by Student

Demographic Groups and SES (Results at the Student Level)

In this section, sets of background variables are added to look at changes in

mathematics gains in the United States. Model 1 predicts mean student gains across the

sample and partitions the variance between students and teachers where students are

nested within teachers. Models 2-5 predict mean gains across gender, ethnicity, and SES

(eligibility for free and reduced lunch).

Mean Gains. To set the context for answering the questions about connections

between student achievement and teacher variables, results are presented first for

163

differences in student achievement across student demographic groups. The level one
and level two equations for the model are given followed by their estimated regression
coefficients in Tables 10.2.
Model I. Mean achievement
Level 1 - Students

Yfiflofieg (10.1)
Level 2 - Teachers

1501' = 3’00 + rOj
where Y ij is the mathematics achievement for student i in teacher j’s classroom, IBOj is the
intercept or mean students’ mathematics achievement within teacher j’s classroom, eij is
the level 1 error, 700 is the mean value of classroom achievement for all teachers, and r0]-

is the level 2 error.

Table 10.2 NAEP 1996 Average Mathematics Achievement

Model 1*
Intéréepuvoo) * A * ‘223.13(1.04)***
LVL 1 Variance (02) 574.32
WiVALEYaienstd - _ 3 98134“

*** p<.001,“ p<.01,* p<.05

The predicted average achievement score for 4‘h grade mathematics students for this
sample of US students is 223.13 points (Y00=223.13, sd=1.04) when no other variables
are included in the model.

02 and 1700 are used to compare variation in the errors at the teacher level with

variation at the student level using the formulas:

164

2 2
0’
_ = level one variance __7_T__09__.

2 = level two variance (10.2)
0- + TOO O'- '1‘ T00

60% of the variance in achievement for Model 1 is within teachers (between students)
leaving 40% of the variance in achievement between teachers. That is to say, there is 1.5
times more variation within teachers than between teachers.

Mean Achievement by Demoggaphic Group. Next, demographic variables are
added to form Models 2, 3 and 4. Model 5 introduces the control variable for SES. The
equations for Model 2, ethnic group, are presented first (Eq.10.3) To interpret the
coefficients, the estimated score for ethnicity can be computed using the appropriate set
of Is and Os for the dummy variables. As an example, Equations 10.4 and 10.5
demonstrate the computation for obtaining Black student achievement. Estimated
achievement by ethnicity are presented in Table 10.3.24
Model 2. Ethnic groups
Level 1- Students

Y ij = IBOj + flljdblackij+ flzjdhispaniclj+ ,33jdallotherij + eI-j;
Level 2 - Teachers (10.3)
:30} = 700 + rOj

.61} = 710

 

24 Note that the category for All other includes Asian and those students who did
not report ethnicity. Often these smaller groups are folded into the intercept since they
are not significantly different from the intercept. In this case, the coefficient for all other
was significant at p<.05. Additionally, it was important to maintain the meaning of the
intercept as a specific group, in this case white students, before analyzing the differential
effects of teacher variables.

165

7621' = 1’20
[331' = Y30
The regression coefficients for predicting student achievement from ethnic background

are found in Table 10.3.

Table 10.3 Regression Coefficients for NAEP 4th Grade Mathematics 1996 by Ethnicity

i Model2
Interest); (v00) 229.23(.98)***
Black (v01) -22.14(1.77)***
ffiigéhic (v02) * i-18.30(1.63)***
jAllotherTwOé) I if if 3 * 454(192y *
LVL1mVariance (02 ) 547.99
*Lv'Lyzi/QrISrISenOO) f“ i 3 254.233"

p<.001,“ p<.01,* p<.05

To make it easier to interpret the results from the achievement by ethnicity group
analysis, these coefficients can be used to compute the achievement for a student with a
given set of demographic characteristics. Because the demographic characteristics are
coded as 1 or 0, the computation involves summing the terms which are coded 1 for a
particular student.25 For example, to determine the predicted achievement for a black
student who would have ones for the intercept and black the resulting equation would be:

Level 1 - Students

A A

vi]. = fl0j+l31jdblackij (10.4)

 

25 A table showing ones and zeros for all dummy variables is provided in the
Appendix.

166

Level 2 - Teachers

flOj = foo = 229.23

flu =7“, = -—22.14

When the slope values, the betas from Equation 10.4, are substituted in the level one
equation (Eq.10.4), the result for black achievement is produced (Eq. 10.5).
9,. = (229.23) + (-22.14)*1 (10.5)
= 207.09

Predicted mathematics achievement for all demographic groups are presented in
Table 10.4. This table shows that the model predicts that White students score higher in
mathematics than the other ethnic groups. The coefficients for ethnicity are significantly
different from zero.

Table 10.4 Predicted Mathematics Achievement Student Demographic Group

 

 

Ethnic Group Achievement
White 229.23
Black 207.09
Hispanic 210.93
All other 224.59

 

In the next model, gender predicts average mathematics achievement.

Model 3. Gender

Level 1- Students
Yij = :50? ”41 dfemij + e”;

Level 2 - Teachers (10.6)
301' = 1’00 + r0j
:34)“ = 1’10

The regression coefficients for predicting student achievement from gender are found in

167

Table 8.5.

Table 10.5 Regression Coefficients for NAEP Mathematics Achievement 1996 by

Gender

, M ,M‘oaeia
Intercept (V00) 224.76(1.22)***
Female (v04) , f f 7 , '3~27(1'02)" - ,

LVL 1Variance(oz) * 3 571-75

LVL 2 Variance (T00) 4381-73”

*** p<.001,“ p<.01,* p<.05

These coefficients indicate that male mean achievement in mathematics is predicted to be
224.76 points compared to 221.49 points for females. The coefficient for female is
significantly different from zero (p<.01).
Model 4. Combined gender and ethnicity

Model 4 tests whether or not there is an interaction between gender and ethnicity
in the 1996 NAEP 4th grade mathematics data. Equation 10.7 computes coefficients for
combinations of gender and ethnicity.
Level 1- Students

Y ij = pOj + flljdblackij+ ,szdhispaniclfr fl3jdallotherij+ '64j dfeml-j

+ ,stegenxblij+ ,B6jegenxhispij+ ,B7jegenxothij + eij; (10.7)

Level 2 - Teachers

7601' = 700 + ’0j

:31} = 710
:52)- = 1’20
:631 = 1’30

168

:541' = 740

:35): = Y50
fléj = V60
7871' = 1’70

The regression coefficients for predicting student achievement from ethnic background
and gender are found in Table 10.6.

Table 10.6 Regression Coefficients for NAEP 4th Grade Mathematics Achievement 1996
by Gender and Ethnicity

* M5334
Intercepiwoo) if f i 231.02(1.18)***
5391:7001) * f * 3-22.07(1.77)***
W‘Hiispahicwoz) ' _ 4-013.’33(1._52):**_
All other (v03) -4.59(1.90)*
Female (y04) -3.61 (1 .27)“

Effect Blackxfemale (v05) 1.33(123)

f Effect Hispanicxfemale (v06) :.71(1.19)

Effect All other x female (v07) -.82(1.59)

LVL 1 Variance (02) if * 545.33

LVL 2 Variance (T00) 255.02***

p<.001,” p<.01,* p<.05

To make it easier to interpret the results from the achievement by demographic
group analysis, these coefficients can be used to compute the achievement for a student
with a given set of demographic characteristics. Because the demographic characteristics
are coded as 1 or 0, the computation involves summing the terms which are coded 1 for a

particular student.26 For example, to determine the predicted achievement for a black

 

26 A table showing ones and zeros for all dummy variables is provided in the
Appendix.

169

female student who would have ones for the intercept, black, female and black female,
the resulting equation would be:

Level 1 - Students

A

Yij = ,30 j + Bljdblackl-j + B4 jdfeml-j + B5 jeblackxefemij

Level 2 - Teachers (10.8)

[3’0]. = 7‘00 = 231.02

0,]. =y‘m = —22.07

fi4j =f40 = ‘ 3-61

.BASJ- = 1:50 = 1-33
When the level two betas from Equation 10.8 are substituted in the level one equation
(Eq. 10.8), the result for black female achievement is produced (Eq. 10.9).

)7, = (231.02) + (-22.07)*1 + (-3.51)*1 + (1 .33)*1 (10.9)
= 206.67
Predicted mathematics achievement for all subsets of students by demographic

groups are presented in Table 10.6. The table shows that the model predicts female
students from all ethnic groups have lower scores than their male counterparts.

Table 10.7 Predicted Mathematics Achievement by Student Demographic Group with
Differences by Gender

 

 

Males Females Difference in Achievement
by Gender
White 231.02 227.41 - 3.61
Black 208.95 206.67 - 2.28
Hispanic 212.64 208.32 - 4.32
All other 226.42 222.00 - 4.42

 

170

Score differences between males and females of the same ethnicity are predicted to range
from 2.28 to 4.42 points. Coefficients for gender times ethnicity effects are not
significantly different from zero over the course of one year.

Another known source of variation in student achievement is SES as measured by
eligibility for free and reduced lunch. In NAEP, this variable is measured at the student
level. Model 5 predicts mathematics achievement by gender, ethnicity, and SES.

Model 5. Gender, Ethnicity, and SES
Level 1- Students
Y i!- = flOj + flljdblackij+ flzjdhispaniclfr fl3jdallotherij+ fl4j dfemI-j (10.10)
+ fl5jegenxblij+ ,B6jegenxhispij+ ,B7jegenxothij
+fl8jSESiJ- + eij;
Level 2 - Teachers

:60} -—- 700 + rOj

.61} = 7’10
:52} = 1’20
:63} = 730
7541' = 740
1651' = 1’50
fl6j = 1’60
.371' = 1’70
.681' = 770

The regression coefficients for predicting student achievement from ethnic

background, gender, and SES are found in Table 10.8.

171

Table 10.8 Regression Coefficients for NAEP 4th Mathematics Achievement 1996 by

Gender, Ethnicity, and SES

Black (v01)
7 Hispanitgwoé) 7
All other (v03)
rem... 8,4) ’
Effect Black x female (v05)

Effect Hispanic x female (v06)

Effect All other x female (v07)

LVL 1 Variance (02)

LVL 2 Variance (Tool + I _

*** p<.001," p<.01,*

533.28

Model 5

1234.18(1.22)***

-18.46(1.74)***

’ W-16.48(1.57)***
-2.87(1.87)

325(123)" 1

"112(122)

:.52(1_.1_8) _ *
-.59(1.55)

_-12.55(1.34)***

211.36***

As demonstrated earlier, the computational example to determine the predicted

achievement for a black female student eligible for free and reduced lunch who would

have ones for the intercept, black, female, black female, and free and reduced lunch,

would be:

Level 1 - Students

Yij = A” + Bljdblackij + B4jdfemij + stegenxblij + ngSESij

Level 2 - Teachers

(10.11)
00]. :70, =234.18
Ajzfmz —18.46
fi4j=1940=‘ 3-26
fisj=1350= 1-12

flgj =l’;80 =—12.65

172

When the betas from Level 2, Equation 8.11, are substituted in the Level 1 equation, the
result for black female achievement is produced (Eq. 8.12).
Vij = (234.18) + (-l8.46)*1 + (-3.26)*1 + (1.12)*1 + (-12.65)*K10.12)
= 200.93
Predicted mathematics achievement for all subsets of students by demographic
group and grade are presented in Table 10.9. The estimated point loss associated with

free and reduced lunch eligibility is -12.65 points.

Table 10.9 Predicted Mathematics Achievement by Student Demographic Group and

 

 

SES
Males Females Difference in Achievement
by Gender
White 221.53 218.27 - 3.26
Black 203.07 200.93 - 2.14
Hispanic 205.05 201.17 - 3.88
All other 218.66 214.81 - 4.85

 

The table also indicates that the model predicts that all non-white student groups have
lower mathematics scores than white students once ethnicity, gender, and SES are taken
into consideration. Girls are predicted to score lower than boys. Hispanic students show
the largest differences by gender with girls predicted to lose -3.88 points. The point gap
between males and females in Table 10.9 is somewhat less than the point gap in Table
10.7 which did not include SES.

filmmarv of Section 1. Developing the level one equation for students. As the
level one base equation for 1996 NAEP 4th grade mathematics developed, average yearly
achievement was first predicted with a simple unconditional model. This model indicated
that achievement variance was 1.5 times larger within teachers (between students) than

across teachers at level 2. Variables known to affect mean achievement (ethnicity,

173

gender, ethnicity x gender, and SES) were added to the level one equation revealing that
while differences in mathematics achievement by ethnicity and gender exist within
teachers, the interaction terms for gender times ethnicity were not significant predictors of

student achievement.

II. Differential Effects of Teacher Characteristics, and Teacher Practices on Student
Achievement (Results at the teacher level)

This section addresses the questions:

1. Does mathematics achievement for demographic groups vary across
teachers?
2. How much variation in student mathematics achievement do variables

about teacher practices, characteristics, and professional development
explain?

3. Is the amount of variance explained by teacher practices, characteristics,
and professional development different for different student demographic
groups?

The analyses in this section first test if average achievement for the various
demographic groups varies across teachers. The unit of analysis shifts to teachers.
Nonsignificant error terms for variance (r00, r10, ...rx0) indicate that group mean
achievement does not vary across teachers so the error terms predicting mean
achievement for those demographic groups are dropped from the model. Although the
error terms are dropped, the fixed level two equations for non-significant variation by

demographic groups are retained in the analysis. Analyses indicated that average

174

achievement for Hispanic and All other males and females across teachers did not vary
significantly from the intercept (110:0) so the error terms have been eliminated. Their
variance is then captured as part of the error term for the intercept (r00).

Mathematics achievement varies across teachers for three demographic groups,
females, Blacks, and Black females. The x2 tests indicate that the variance of the
intercept and the average mathematics achievement of females, Black, and Black female
are significant (1:..0 150). Average scores for each group are addressed in a series of
analyses which include teacher practices, teacher characteristics, and teacher professional
development.27 The results for average achievement by gender across teachers is
presented first followed by average achievement by Black students across teachers.
Finally, results for average achievement by Black female students across teachers
conclude the chapter.

Effects of teacher level variables on mean gains by gender

While the fixed effects model for gender, Model 3, predicted that average
achievement varied significantly by gender, a random slopes and intercepts model, Model
6, is necessary to test whether or not the effects of teachers vary when predicting the
average achievement for males and females. If they vary, do teacher practices,
characteristics, or professional development have differential effects on the average
mathematics achievement for one gender as opposed to the other? If variance is
significant across teachers, it is then possible to determine how much of the variation can

be explained by a specific level two variables.

 

27 Average student SES is nearly significant (p<.08).

175

Model 6. Unconditional model for Teacher effects on Male/Female average achievement
Level 1- Students
YU- = flOj + flljdblaCkij'l‘ fl2jdhlSPQNlCI-j'l' fl3jdallotherij+ fl4j dfema-
+fl5JSESn + e.)- (10.13)

Level 2 - Teachers

floj = 700 + r0,- (10.14)
.Blj = 1’10
7621' = 1’20
fl3j = 1’30

:341' = 1’40 + ’4)“
pay = 1’50
where
YOOj = intercept, average male achievement across teachers all other predictors
held constant
raj = the error term or unique teacher effects
Y40j = intercept, average female achievement across teachers all other
predictors held constant

r 4]. = the error term or unique teacher effects

176

Table 10.10 Regression Coefficients for NAEP Mathematics Achievement 1996 by

Gender
Model6
Intercept (v00) 1 234.12(1.20)***
Black (v10) -18.41 (1 .75)***
Hispanic (v20) -16.42(1.56)***
All other (v30) —2.73(1.88)
Dfem (v40) -3.09(1.00)**
N 7353 (1750) i f I 9 * '-12.67(1.33)*** ”
'LVL1VaFIance(oz) 518.07
1912151385.. (T00)~ if f 1 253.12*** 0
I M 6.51.1384,» 1 - i * 51.84“

*** p<.001,“ p<.01,* p<.05

The effect of female varies significantly across teachers (T40=61.84, p<.01) as does the
effect of male (T00=253.12, p<.001). Selected coefficients from this model serve as the
base for the variance decomposition presented in Tables 10.13-10.16. Variance
decompositions systematically demonstrate the amount of variance explained by each
teacher variable included at level two.
Model 7. SES and Gender
Level 1- Students

Y ij = flOj + flljdblackij+ ,BZJ-dhispaniclf fl3jdallotherij+ '64j dfemij

+IB5JSESy. + eij (10.15)

Level 2 - Teachers

:601' = 1’00 + 1’01 “V555 T ’0} (10-16)
.51} = 1’10
flZj = 1’20

177

53): = 1’30
fly = 1’40 T 1’41 ”SES T ’4j
flSj = 1’50
where
YOOj = intercept, average male achievement across teachers all other predictors
held constant
"0j = the error term or unique teacher effects
Y40j = intercept, average female achievement across teachers all other
predictors held constant
r 4]. = the error term or unique teacher effects

Table 10.11 Regression Coefficients for NAEP Mathematics Achievement 1996 by
Gender and Average Classroom SES

* Model7 *
Intercept (V09) 233.35(1.22)1**
3163357370161) 3 if h ' —5.87(2.23)**
Beck (7)1107- 13 I i *1 I 7 9178211751“?
919.551.11.820) _. " L15.15(1.56)*1*
Knotnerwgo) 1-2.653(1.89) *
Dfem (v40) -3.12(.97)**
3 ‘A'vSES (v41) 1 151(215)
515331756) A i if i 3-11.51(1.3E)***
LVL1Variance(02) 515.45 3
LVszfilnfteerept (100)?~ if 3 .252.63***
I 1 W 6.5.3.1140» * * 363.57;

p<.001,“ p<.01,* p<.05

How much variation does average classroom SES explain in the average mathematics

achievement of males and females across teachers? To answer this question, the

178

following equation computes the variance in female student achievement explained by

average SES.

T40 (model 6) — T40 (model 7) _ 61.84 - 63.57
140 (model 6) 61 .84

 

 

= —.28 = 0% (10.17)

This means that 0%28 of the variance in predicted average achievement for female
students is explained by average SES across teachers (classrooms). Using the same
process for the remaining tau, less than 1% of the variance in male average achievement
is attributable to SES across teachers. SES does not explain variation in average
achievement by gender.
Gender and Teacher variables

Now that the context is set for the analyses at Level Two, it is possible to track the
variance explained by each teacher level variable controlling for classroom SES and to
determine teacher effects on average mathematics achievement for male and female
students across teachers. One variable at a time is added to the level two equation, EQ
10.14. The term “variable x” is exchanged for a single Mathematics Practice (math
sensemaking activities, group and project activities, or group and project assessment

activities), Teacher Characteristic (mathematics preparation, preparation for mathematics

 

28 Negative estimates for variance are puzzling. Sometimes variability in a data
set is large enough to produce a negative estimate, even though the true value of the
variance component is positive. NAEP average classroom SES data is aggregated from
student level records so it is not an independent measure - students may not only share
teachers but schools and districts. It is common practice to treat negative variance
components as if they are zero (SAS Institute Inc., 1999).

179

instruction, mathematics knowledge, reform orientation, diversity preparation), or
Professional Development (professional development frequency in the last year).
Model 8. Template for Level 2 equations

Level 1 remains constant (Eq. 10.10)

Level 2 - Teachers

1801' ___ 1’00 + 1’01 avSES + 702” variable x + r0j (10.18)
1311' = 1’10
:62)“ = 1’20
:33} = 1’30

,B4j : 1’40 + y41avSES + y42 variable x+ r4j

1391' = 1’70
where
7024' = change in average in average male student achievement associated with
a one unit change in “variable x” holding SES constant at the classroom
(teacher) average
and

Y42 , = change in average female student achievement associated with one unit
..j

change in “variable x” holding SES constant at the classroom (teacher)
average
Gender and Tgpher Pr1c_ti_c_e§. A variance decomposition for teacher practices
can be found in Table 10.13. The table indicates that almost no additional variance in
average achievement for female students is explained by teacher practices across teachers

when SES is controlled. Small amounts are explained by teacher practices for male

180

students.

Table 10.12 Variance Decomposition for Male/Female Controlling SES - Teacher

 

 

 

 

 

 

 

 

 

Practices
avSES Practice
Variables Uncond Free/Red Lu Sensemaking EPA t ErpuiparProject
sessmen c M Ies

Intercept (v00) 234.12(1.20)*** 233.35(1.22)*** 233.43(1.22)***i233.44(1.21)***| 233.52(1.20)m
(v01) .96(.43)*| .73(.46 1.35(.36)""‘I
avSES(v02) -5.87(2.23)** -5.48(2.21)*I -5.97(2.23)**| -5.95(2.21)‘“l
Female (v10) -3.09(1.00)** -3.12(.97)** -3.18(.99)**| -3.22(.98)***| -3.13(.98)*"
(v11) -.54(.42)| -.79(.41) .42(.36)
avSES(v12) -.51 (2.16) -.77(2.12)l -.40(2.16)| -.48(2. 16)
Intrcpt Var(‘roo) 253.12*** 252.63*** 243.66*** 248.96*** 239.40“
Fem Var (T40) 61 .84" 63.57“ 61 .27* 61 .32" 61.93“
(02) 518.07 516.45 516.921 516.41 516.60
% variance reduction intercept <1 % 3.7%i 3.5%i 5.4V]
% variance reduction fem slope 0% <1%l <1%1 0‘7

 

 

 

 

For every one standard deviation increase in mathematics sensemaking and group

and project assessment activities, male scores are predicted to increase by up to .96 points

while female scores decrease by up to -.79 points. This is an indication of differential

teacher practice effects in 1996 NAEP 4th grade mathematics data. For every one

standard deviation increase in group and project assessment activities, male scores

increase by .73 points while female scores decrease by -.79 points, another indicator of

differential effects. A one standard deviation increase in group and project activities

increases average achievement for both boys and girls.

Gender. Teacher Characteri_stics, and Professional Development. The

 

variance decomposition for reform orientation follows in Table 10.14. Most of the teacher

characteristic and professional development variables explain less than one percent of the

variance in the scores of both genders. The one variable which explains variance across

181

teachers for NAEP 1996 mathematics is reform orientation when SES is controlled.

Table 10.13 Variance Decomposition for Male/Female Controlling for SES - Teacher

 

 

 

 

 

 

 

Characteristics
SES Teacher
Characteristics

Variables (v01 & v11) Unconditional Free/Red Lunch Reform Orientation
Intercept (v00) 234.12(1.20)*** 233.35(1.22)*** 233.551 .21)W
(v01) 1.65(.44)**"‘
avSES(v02) -5.87(2.23)** -5.47(2.22)1
IFemale (v10) -3.09(1.00)** -3.12(.97)** 315(98):"l
(v1 1) -.65(.45)
avSES(v12) -.51(2.16) -.69(2.15)
Intercept Var(T00) 253.12*** 252.63*** 238.14***
Female Var (T40) 6184“ 63.57" 60.57‘I
(02) 518.07 516.45 517.15

% variance reduction intercept <1% 5.9°/
% variance reduction female slope 0% 2.1V]

 

 

 

 

A one standard deviation increase in teacher reform orientation differentially increases
the average male score by 1.65 points while decreasing the average female score by -.65
points.

Summary. Although the average scores for males and females vary across
teachers, the effects of teacher practices, characteristics and professional development
measured in this data do not explain much of the variance in scores for boys and girls.
With the exception of group and project activities, it appears that these small effects are
differential (i.e., increases in the teacher variables increase scores for boys while
decreasing scores for girls on average). Increases in group and project activities increases
average scores for both groups.

Effects of teacher level variables on mean achievement by Black and White

The fixed effects model for ethnicity, Model 2, predicts that average achievement

182

varies significantly by ethnicity but a random slopes and intercepts model, Model 9, tests
whether or not the average achievement for white and black students varies significantly
across teachers. In other words, do teacher practices, characteristics, or professional
development variables have differential effects on the average mathematics achievement
of Black and/or White 4th grade students in the US? If variance is significant across
teachers, it is then possible to determine how much of the variation in each groups’
achievement can be explained by single level two teacher variable and if that effect is
different for each group.

The unconditional Model 9 establishes the maximum amount of variance that can
be explained by teacher level variables for this data.
Model 9. Unconditional model for 4th grade Black/White achievement
Level 1- Students
Y i]- = IBOj + flljdblackij+ flzjdhispanicl-jJr fl3jdallotherij+ ,64j dfem].j +fl5JSESI-j + eij(10°19)
Level 2 - Teachers

floj = 700 + r0,- (10.20)

.51}: 710T’4j

1521' = 1’20
163' = 1’30
164} = 1’70
1651' = 1’70
where
Y 001- = intercept, average in student achievement/achievement in teacher j’s

classroom all other predictors held constant

183

YOI' = the change in average achievement associated with mean free and
.1

reduced lunch, all other predictors held constant

r0j = the error term or unique teacher effects

YIO‘ = intercept, average black student achievement across teachers all other
J

predictors held constant
Y] 1' = the change in average black achievement associated with one unit
J
change in average free and reduced lunch
rllj = the error term or unique teacher effects

The estimated level two regression coefficients for Model 6 are given in Table

10.15.
Table 10.14 Regression Coefficients for 1996 NAEP 4th Mathematics Achievement by
Ethnicity
» g e - g M6de19 ,-
Intercept (yoo) 234.16(1.20)***_ _
..o 91'3ko 01 ,, - - 719-090-781”
_fi'ipél‘Eflzol- - - ,- - '19-47(1;55)”’
., fl'fl‘EUVso) e :2-79Q-89) z,
#13me (V40) fffff f p 396098)" f
- 51.53.0150) , -‘12-6‘31-331'"
191985809 _ *1 __ 530-45; "
LVLZlntercept (Too) 7 2154?"

v v ank (T10) W , 7 57.10*

***p<.001,** p<.01,* p<.05

The effect of Black varies significantly across teachers (T,0=57.10, p<.05) as does
the effect of White (T00=215.41, p<.000). Selected coefficients from this analysis serve as

the base for the variance decompositions found in Tables 10.12-10.15. Next district SES

184

is added to the model to determine its effect on the average achievement for Black and

White students across teachers. The variable for student free and reduced lunch has been

aggregated to the teacher level as a control for classroom SES.

Model 10. 4th grade Black and White achievement controlling for SES

Level 1- Students - Same as unconditional Equation 10.10

Level 2 - Teachers

where

1100)“

You

YIOj

YIIj

150} = 1’00 T 1’010VSES T raj (10.21)

1511' = 1’10 T 1’11 ”SES T ’4j

.sz = 1’20
1531 = 1’30
fl8j = 1’70
.59} = 1’70

= intercept, average in student achievement in teacher j’s classroom

holding all other predictors constant

= the change in average achievement associated with mean free and

reduced lunch, all other predictors held constant

= the error term or unique teacher effects

= intercept, black student achievement in teacher j’s classroom all other

predictors held constant

= the change in average black student achievement associated with one

unit change in average free and reduced lunch

185

r1 j = the error term or unique teacher effects

The estimated level two regression coefficients and variance components for

Model 10 are given in Table 10.15.

Table 10.15 Regression Coefficients for Black/White Achievement Predicted by

Classroom SES
Model10 3
wIntéiEpfwoo) ’ 1 i f 233.52(1.23)f** i
avSES (v01) -5.16(1.97)**
Black (v10) -17.85(1.80)***
Ifiimfiavseswh) 15.223937) 3
__ 1315311116 (Iv—20) _ l16.33(1.55)f** H
“Aistriartyggf ’ 1-2.80(1.89) ’ '
Dfem (V80) -3.05(.98)**
”615’s (ng0) T '1 A 3 -11.69—(1.38)"’;‘i
”LVETv‘érenZeioz’)" 529.49 _
LVl.2\lariaInce (100) 1 215.04*** 3 i
'3 f k ”3 A (110) ”48.48?

***p<.001,** p<.01,* p<.05

How much variation does average SES explain in the average mathematics

achievement for White and Black students across teachers? To answer this question the
following equation computes the variance in black achievement explained by average
SES. The estimated 1:10 values from Tables 10.10 and 10.11 would be substituted in the
following equation:

r10(model 9)— 710(mode110) _ 57.10-48.48
" 57.10

=.151=15.1%

 

 

2'10 (model 9) (1022)

This means that 15.1% of the variance in predicted average achievement for black

students is explained by average SES across teachers (classrooms). Using the same

186

process for the remaining tau, less than 1% of the variance in White average achievement
is attributable to SES across teachers. SES is a strong predictor of Black student
achievement.

Table 10.16 Variance Decomposition for Black/White Achievement by SES

 

 

 

 

 

 

 

 

 

SES
Variables Unconditional Free/Red Lunch
Intercept (v00) 234.16(1.20)*** 233.52(1.23)***
avSES(y02) -5.16(1.97)**
lBlack (v10) -19.09(1.78)*** -17.85(1.80)***
avSES(v12) -5.22(3.37)
Intercept Variance (T00) 215.41*** 215.04***
Black Variance (T10) 5710* 4848*
Level 1 variance (02) 30.45 529.49
% variance reduction intercept <1%.
% variance reduction black ' 11.7%

 

 

*** p<.001,** p<.01,* p<.05

A one standard deviation increase in average classroom free and reduced lunch
across teachers reduces average Black achievement by -5.12 points and decreases average
White student achievement by -4.83 points. The variance decomposition from the table
indicates that average classroom SES explains 11.7% of the variance in average Black
achievement across teachers and almost none of the variance in average White
achievement. SES, therefore, has a differential effect on Black and White student (non-
Black) achievement.

The Effects of Black, SES, and Teacher Variables
Now that the context is set for the analyses at Level Two, it is possible to track the
variance explained by each teacher level variable controlling for SES and to determine its

effect on average mathematics achievement for Black and White students across teachers.

187

One variable at a time is added to the level two equation, EQ 10.14. The term “variable
x” is exchanged for a single Mathematics Practice (math sensemaking activities, group
and project activities, or group and project assessment activities), Teacher Characteristic
(mathematics preparation, preparation for mathematics instruction, mathematics
knowledge, reform orientation, diversity preparation, knowledge of NCTM standards), or
Professional Development (professional development frequency in the last year).
Model 8. Template for Level 2 equations
Level 1 remains constant (Eq. 10.10)
Level 2 - Teachers

160} = 1’00 + 1’01 avSES + yo“). variable x + r0]- (10.23)

fllj = 1’10 T 1’11 “VSES T 712” variable x+ r41.

.52} = 1’20
.63} = 1’30
.68} = 1’70
.39} = 1’70

where
7024 = change in average in average student achievement associated with a one
unit change in “variable x” holding SES constant at the classroom
(teacher) average

and

Y 12 , = the change in average black student achievement associated with one
..j

unit change in “variable x” holding SES constant at the classroom

(teacher) average

188

Ethnicig and Teacher Practices. A variance decomposition for teacher

practices can be found in Table 10.17. The table indicates that little additional variance in

average achievement for Black students is explained by teacher practices across teachers

once classroom SES is controlled. For instance, average SES explains 15.1% of the

variance in average Black achievement compared to average SES plus sensemaking

which explains only 10.1% of the variance. Average SES by itself does a better job of

explaining variation in Black achievement. Small amounts of additional variance are

explained for White students.

Table 10.17 Teacher Practices - Variance Decomposition for Average White/Black
Achievement Across Teachers Controlling SES

 

 

 

 

 

 

 

 

 

 

avSES Practice
Variables Unconditional IFree/Red Lun SensemakingfilGPA iGroupatProject
lassessment [Activities

Ilntercept (v00) 234.16(1.20)*** 233.52(1.23)*** 233.60(1.23)***i233.62(1.22)***i 233.71 (1 .21 )**"
(v01) .72(.35)*l .43(.39h 1.14(.33)“
avSES(v02) -5.16(1.97)** -4.84(1.98)*| -5.14(1.97)*| -5.19(1.97)*"
(Black (v10) ~19.09(1.78)*** -17.85(1.80)*** -17.97(1.80)***| -17.91(1.80)***| -18.17(1.84)*“
(v11) -.11(.49)| -.44(.59)I .02(.56)
avSES(v12) -5.22(3.37) .‘5'409'38ll --5'1-5—(3f§1r f 3343.35)
InterceptVar(T00) 215.41*** 215.04*** 210.71*** 213.49*** 20586“
Black Var (140) 710* 4848* 5131* 5035* 5355*I
(02) 30.45 529.49 529731 529641 529.40

% variance reduction intercept <1 % 2.2% <1% 4.4°/
% variance reduction black 15.1% 10.1%; 11.8%‘l 6.2°/

 

 

 

 

For every one standard deviation increase in sensemaking and group and project

assessment activities, White scores increase by up to .72 points while Black scores

decrease by up to -.44 points. This is an indication that there are differential teacher

practice effects present in 1996 NAEP 4th grade mathematics data. For every one

189

standard deviation increase in group and project activities, White scores increase by 1.14
points while Black scores remain nearly level. None of the practice variables explain
more variation in average Black student scores across teachers than SES alone; however,
the variables do explain small amounts of variance in White average achievement across
teachers (up to 4.4%) after controlling for SES.

The Effects of Teacher Characteristics on Average Black Achievement. The
variance decomposition for teacher characteristics follows in Table 10.19. There are four
variables which explain more variance in average achievement across teachers than
classroom SES alone - mathematics preparation, knowledge of NCTM standards, reform
orientation, and mathematics knowledge. Mathematics preparation explains additional
variation in achievement for both Black and White Students across teachers while
mathematics knowledge explains additional variance in achievement for Black students
only. Reform orientation explains a little additional variation in White average
achievement controlling for SES as does knowledge of NCTM standards. A more

detailed explanation follows the table.

190

Table 10.18 Variance Decomposition for Black/White Controlling for SES - Teacher

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Characteristics
SES Teacher Characteristics
Variables (v01 & v11) Uncond Free/Red Lu MthPrep NCTM
llntercept (v00) 234.16(1.20)*** 233.52(1.23)*** 233.48(1.23)***| 233.56(1.21)***|
(v01) 1 .48(1.00)| 2.35(1.05)*
avSES(v02) -5.16(1.97)** -4.92(1.96)*| -4.96(1.95)*
iBlack (v10) -19.09(1.78)*** -17.85(1.80)*** -17.77(1.77)***| .17.91(1.80)***|
(v11) -1.38(1.49 -2.17(1.67
avSES(v12) -5.22(3.37) '5'33(3°36)l -5.25(3.36)|
Intercept Var (T00) 215.41 *** 215.04*** 212.77***l 210.26***l
Black Variance (T40) 57.10* 4848* 47.01* 50.55‘
Lave. 1 Variance (02) 30.45 529.49 529.6 I 529.5
% variance reduction intercept <1% 1.3°/T 2.4%i
% variance reduction black 15.1% 17.7%] 11.4%]
SES Teacher Characteristics
Variables (v01 & v11) Uncond Free/Red Lu Reform Or ' thKnow

 

 

 

 

 

 

 

 

Ilntercept (v00) 234.16(1.20)*** 233.52(1.23)*** 233.72(1.22)***| 233.54(1.23)***|
(v01) 1.29(.41)**l .27(.92)|
avSES(v02) -5.16(1.97)** -4.82(1.96)*| -5.10(1.97)*|
Black (v10) -19.09(1.78)*** -17.85(1.80)*** -18.19(1.78)***l -18.34(1.85)***|
(v11) .28(.65)| 2.96(1944fi
avSES(v12) -5.22(3.37) -5.67(3.38)I -5.11(3.34h
Intercept Var (TOO) 215.41*** 215.04*** 206. 38***' 215.58”*i
Black Variance (T40) 57.10' 4848* 53.17*| 4694*
Leve. 1 Variance (02) 530.45 529.49 529 731 529.44
% variance reduction intercept <1 % 4.2%i 0%i
% variance reduction black slope 15.1 % 6. 9% 17.8%]

 

For every standard deviation increase in mathematics preparation (degrees/licenses),

average White scores increase by 1.48 points and Black scores drop by -1.38 points.

Mathematics preparation explains 17.7% of the variation in average Black achievement

across teachers controlling for classroom SES but less than 1% for average White

achievement.

191

Reform orientation controlling for SES across teachers explains 4.2% of the
variation in white average achievement over the course of one year while explaining no
additional variance in the average achievement for Black students across teachers
controlling for SES. For every one standard deviation increase in teachers’ reform
orientation, black students are expected to gain .28 points over the course of one year.
White students are expected to gain 1.29 points.

Teachers’ math knowledge controlling for SES explains 17.8% of the variation in
average Black achievement across teachers but nothing for average White achievement.
A one standard deviation increase in mathematics knowledge increases Black
achievement by 2.96 points while increasing White achievement by .27 points.

SES is again the most important single predictor of Black achievement across
teachers but SES explains little about White achievement across teachers. Two teacher
characteristic variables, mathematics preparation and mathematics knowledge, explained
more variance in average Black achievement across teachers once SES was controlled
than SES alone. The effect of mathematics preparation was differential. Increases in
mathematics knowledge and reform orientation increased the average scores for both
student groups across teachers.

Ethnicity and Teacher Profession_al Development. Professional development
frequency does not explain variance for either group of students across teachers.

Effects of teacher level variables on the mean achievement of Black Females

Using the same process and equations demonstrated earlier for female and Black

average achievement, the resulting variance decomposition for Black female is presented

below. A more detailed explanation follows the table.

192

Table 10.19 Variance Decomposition for Black Female Controlling for SES - Teacher
Practice, Characteristics, and Professional Development

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

SES Practice
Variables (v01 & v11) Uncond Free/Red Lu GPA GPA Assess
Intercept (v00) 234.02(1.23)*** 233.17(1.26)*** 233.40(1.24)***| 233.34(1.25)***i
(v01) 1.14(.31)**| .33(.37
avSES(v02) -6.18(1.83)** -6.29(1.82)**| -6.24(1.83)**|
lBlack Fem (v10) 1.42(1.22 1.56(1.23) 1.48(1.211 1.36(1.22
(v11) .43(.20)*| .58(.23)*|
avSES(v12) —1.41(1.15) '1'39(1'14)1 -1.37(1.14)J
Intercept Var (T00) 21 1 .04*** 209.04*** 199.34”*| 207.28***
Black Fem Var (T40) 2025* 1999* 18.86* 1781*
Level 1 Variance (02) 517.45 516.28 516.2 I 516.84
% variance reduction intercept <1 % 5.5%; 1.8%;
% variance reduction Black Female 1.3% 6.9% 12.0%
Practice Teacher Characteristics
Variables (v01 & v11) Uncond Sensemaking thPrep eform Or
Intercept (v00) 234.16(1.20)*** 233.26(1.25)***i 233.13(1.26)*** 233.45(1.25)***
(v01) .69(.33)*| 1.31 (.92) 1.32(.37)**
avSES(v02) -5.95(1.83)**| -5.99(1.82)** ~5.89(1.81)**
Black Fem (v10) -19.09(1.78)*** 1.56(1.23)| 1.55(1.23) 1.44(1.23)
(v11) .30(.24)| .24(.47) .62(.25)*
avSES(v12) -1.26(1.13)l -1.39(1.15) -1.19(1.15)
Intercept Var (T00) 21 1 .04*** 204.36***| 207.62*** 199.48***
FBlack Fern Var (T40) 2025* 1954* 2033* 1931*
Level 1 Variance (02) 517.45 516.54 529.69 516.09
% variance reduction intercept 3.20/I 1.6% 4.2%
% variance reduction black slope 3.5%! 0% 6.9%

 

 

 

193

Table 10.19 (cont'd).

 

 

 

 

 

 

 

 

Characteristics Professional Dev

Variables (v01 8 V“) Uncond NCTM knowledge PDF
Ilntercept (v00) 234.02(1.23)*** 233.23(1.25)*** 233.17(1.25)***
(v01) 2.07(.99)* -.32(.68)
avSES(v02) -5.93(1.82)** -6.19(1.83)*'1
Black Fem (v10) 1.42(1.22 1.54(1.23) 1.55(1.23)
(v11) .84(.62) .61(.45)
avSES(v12) -1.32(1.14) -1.36(1.15)
Intercept Var (T00) 21 1 .04*** 205.58*** 209.39”
Black Fem Var (140) 2025* 1963* 19.63'1
Level 1 Variance (02) 517.45 516.29 516.30
% variance reduction intercept 2.6% <1°/
% variance reduction black female 2.8% 3.19]

 

 

 

 

The effect of the included teacher variables on Black female average achievement
across teachers is positive. Group and project activities and group and project assessment
activities explain the most variance29 in black female achievement across teachers afier
SES is controlled. Note that SES accounts for little variation in this analysis, similar to
the results from the male/female variance decomposition. Group and project activities
explains nearly 7% of the variance in black female scores across teachers and
group/project assessment activities explain 12% of the variance. A one standard
deviation increase in these two teacher variables results in an increase of about half a
point for Black females.

The only differential effect present is that for professional development frequency

which is very small. For every one standard deviation increase in professional

 

29 Variance explained in Black female achievement across teachers should be
interpreted with caution since it is the effect of level two teacher variables on the level
one interaction of black and female.

194

development frequency, Black female average achievement increases by .61 points while

average achievement for white male students decreases by -.32 points.

195

11. RESULTS OF THE HLM ANALYSIS
FOR 2000 NAEP 4TH GRADE MATHEMATICS

The HLM analysis using NAEP data is intended to address the research questions
about the relationships among student achievement and

0 Teacher characteristics and practices

‘ Characteristics of professional development
The analysis also is intended to address the ways in which these relationships vary by
student demographic group, including gender, ethnicity, and SES across the United States
and to see if the national mathematics results reflect trends similar to those found in the
Connecticut BTPS data, a more detailed data set. Analyses concerning the relationships
between teacher preparation institutions and teachers/students are not possible for this
data since the names of teacher preparation programs are not collected in NAEP. The
HLM analyses are weighted (origwt) and use five plausible values (mrpcm1-5) for the
student outcome. There are 6,205 students and 1,245 teachers in the 2000 mathematics
sample.

To set the context for addressing the questions, results are presented first which
describe the differences in student achievement (achievement status in NAEP data) across
student demographic groups. The analyses yield results in terms of regression
coefficients. To aid in understanding what the coefficients mean, predicted achievement
(achievement status score) is computed for selected combinations of student
characteristics.

The full set of mathematics variables for NAEP is given in Table 11.1. The outcomes

and most teacher variables are the same as those used for the earlier SEM models. In

196

certain cases, variables that were only available in this NAEP data set were included as

well.

Table 11.1 Description of Variables Used in the 2000 NAEP 4th Grade Mathematics

Analyses
Type Variable Variable Description
Abbreviation
Level 1 - students
Continuous avepv The average of the 5 plausible values for mathematics
(Outcomes) achievement (mean 224.16, 50:29.75)
mrpcm1 1St of 5 multiple plausible values for the achievement of
each student (mean=224.01, sd=31.18)
mrpcm2 2"d of 5 multiple plausible values for the achievement of
each student (mean=224.35, sd=30.77)
mrpcm3 3rd of 5 multiple plausible values for the achievement of
each student (mean=224.09, sd=31.28)
mrpcm4 4th of 5 multiple plausible values for the achievement of
each student (mean=224.17, sd=31.00)
mrpcm5 5th of 5 multiple plausible values for the achievement of
each student (mean=224.16, sd=30.90)
Dichotomous dblack Black (Black=1, else=0)
dhispanic Hispanic (Hispanic=1, else=0)
dallother All other ethnicities (all other=1, else=0)
dfem Female (female=1, else=0)
SES Average free and reduced lunch (eligible=1, else=0)
Effects coded egenxbl Effect of Black female; interaction of eBlack x egender
where Black =1, White=-1, otherwise=0 multiplied by
female=1, male=-1.
egenxhisp Effect of Hispanic female; interaction of eHispanic x
egender where Hispanic=1, White=-1, othenNise=0
multiplied by female=1, male=-1.
egenxoth Effect of All other female; interaction of eAIIother x egender
where All other =1, White=-1, otherwise=0 multiplied by
female=1 , male=-1.
Continuous Origwt Original weight (correction for oversampling)

197

Table 11.1 (cont'd).

Level 2 - teachers

Continuous Grand mean centered
Teacher gpa Frequency of group and project activities (mean=12.52,
Practices sd=2.61 )

dla Frequency of drill and lecture activities (mean=8.62, sd=.80)

sensemaking Frequency of sensemaking activities (mean=11.20, sd=2.61)

Teacher NCTMstds Knowledge of NCTM standards (mean=1.96,sd=.89)
Characteristics
statestds Knowledge of state standards (mean=3.38,sd=.60)
mathknow Knowledge of mathematics (mean=27.93, sd=3.31)

assessprep Preparation for assessment (mean=11.36, sd=2.24)

mathinst Preparation for mathematics instruction, pedagogical
coursework (mean=27.95, sd=3.39)
motrad Traditional orientation, emphasize subject (mean=8.42,
sd=1.63)
moref Reform orientation, emphasize reasoning (mean=7.64,
sd=1.30)
Professional pdf Professional development in the last year (2.84, sd=1.22)
Development
Continuous avSES Average percent free and reduced lunch per teacher/

classroom aggregatedfrom level 1 (mean=.39. §d=.49_) _ _

1. Setting the Stage: Variation in Mathematics Achievement by Student
Demographic Groups and SES (Results at the Student Level)

In this section, sets of background variables are added to look at changes in
mathematics achievement in the United States. Model 1 predicts mean student
achievement across the sample and partitions the variance between students and teachers
where students are nested within teachers. Models 2-5 predict differences in mean
achievement across gender, ethnicity, and SES (eligibility for free and reduced lunch).

Mean Achievement. To set the context for answering the questions about connections

198

between student achievement and teacher variables, results are presented first for
differences in student achievement across student demographic groups. The level one
and level two equations for the model are given followed by their estimated regression
coefficients in Tables 8.2.
Model 1. Mean achievement
Level 1 - Students

Yij=fi0j+ e1]- (11.1)
Level 2 - Teachers

floj‘ = 700 + rOj
where Y ij is the mathematics achievement for student i in teacher j’s classroom, flOj is the
intercept or mean students’ mathematics achievement within teacher j’s classroom, e ij is
the level 1 error, 700 is the mean value of classroom achievement for all teachers, and ’0}

is the level 2 error.

Table 11.2 2000 NAEP 4th Grade Average Mathematics Achievement

‘Model1
"Interceptwoof :225.57(.873)***
LVL 1 Variance (02) 681.44
LVI. 2 Variance (T00) 362. 09*“

“766001," 15<.01,* p<.05
The predicted average achievement score for 4th grade mathematics students for this
sample of US students is 225.57 points (y00=225.57, sd=.80) when no other variables are
included in the model.

02 and 1:00 are used to compare variation in the errors at the teacher level with

variation at the student level using the formulas:

199

2 7

o' . I" .
-——2———— : level one variance Toi— = level two variance (11.2)
0' + 2'00 0'- + TOO

65% of the variance in achievement for Model 1 is within teachers (between students)
leaving 35% of the variance in achievement between teachers. That is to say, there is
almost 2 times more variation within teachers (at the student level) than between teachers.
Mean Achievement by Demograghic Group. Next, demographic variables are
added to form Models 2, 3 and 4. Model 5 introduces the level one control variable for
SES. The equations for Model 2, ethnic group, are presented first (Eq.1 1.3) To interpret
the coefficients, the estimated score for ethnicity can be computed using the appropriate
set of ls and Os for the dummy variables. As an example, Equations 11.4 and 11.5
demonstrate the computation for obtaining Black student achievement. Estimated
achievement by ethnicity are presented in Table 11.3.30
Model 2. Ethnic groups
Level 1- Students
Y ij = IBOj + flljdblackij+ fl2jdhispaniclj+ [33jdallotherij + eij;
Level 2 - Teachers (11.3)
1301' = 1’00 + rOj

.31} = 710

 

30 Note that the category for All other includes Asian and those students who did
not report ethnicity. Often these smaller groups are folded into the intercept since they
are not significantly different from the intercept. In this case, the coefficient for all other
was significant at p<.05. Additionally, it was important to maintain the meaning of the
intercept as a specific group, in this case white students, before analyzing the differential
effects of teacher variables.

200

I82j = 720
1531' = 730
The regression coefficients for predicting student achievement from ethnic background

are found in Table 11.3.

Table 11.3 Regression Coefficients for 2000 NAEP 4th Grade Mathematics by Ethnicity

Model2 I
iniercegt 0100) 233.43(.64)***
Black (v01) -25.86(1.39)***
Hispanic (v02) -20.42(1.29)***
FjAllotheriwm) * i i * ”:5.59(2.53)* i
LVLA1Vafiance (o2 ) f f C 3644.18
EVIéVananvce(Too) * if if 232.97*** -,

*** p<.001,“ p<.01,* p<.05

To make it easier to interpret the results from the achievement by ethnicity group
analysis, these coefficients can be used to compute the achievement for a student with a
given set of demographic characteristics. Because the demographic characteristics are
coded as 1 or 0, the computation involves summing the terms which are coded 1 for a
particular student.31 For example, to determine the predicted achievement for a black
student who would have ones for the intercept and black the resulting equation would be:

Level 1 - Students

{(1.}. = no]. + Eljdblackl-j (ll-4)
Level 2 - Teachers '30]. = 7‘00 = 233.43

fllj 3f”) = ‘25.86

 

3] A table showing ones and zeros for all dummy variables is provided in the
Appendix.

201

When the slope values, the betas from Equation 8.4, are substituted in the level one
equation (Eq. 8.4), the result for black achievement is produced (Eq. 8.5).
91:933-43) + (-25.86)*1 (11.5)
= 207.57
Predicted mathematics achievement for all demographic groups are presented in
Table 11.4. This table shows that the model predicts that White students score higher in
mathematics than the other ethnic groups. The coefficients for ethnicity are significantly
different from zero.

Table 11.4 Predicted Mathematics Achievement Student Demographic Group

 

 

Ethnic Group Achievement
White 233.43
Black 207.57
Hispanic 213.01
All other 227.84

 

In the next model, gender predicts average mathematics achievement.

Model 3. Gender

Level 1- Students
Yij = 301'“ .341 dfemij + eij;

Level 2 - Teachers (11.6)
:60} = 700 + rOj
r34j = 710

The regression coefficients for predicting student achievement from gender are found in

Table 11.5.

202

Table 11.5 Regression Coefficients for 2000 NAEP Mathematics Achievement by

Gender

if i I -, Model3 *
Intercept (v00) 7 7 7 226.55(.91)***
Female(v04) -2.00(.84)*
LVI:1AVariance(02) * 5 680.53

LVL 2 Variance (T00) 362.03***

*** p<.001,“ p<.01,* p<.05

These coefficients indicate that male mean achievement in mathematics is predicted to be
226.55 points compared to 224.56 points for females. The coefficient for female is
significantly different from zero.
Model 4. Combined gender and ethnicity
Model 4 tests whether or not there is an interaction between gender and ethnicity in the
NAEP 4th grade mathematics data. Equation 11.7 computes coefficients for combinations
of gender and ethnicity.
Level 1- Students

Y1.j = IBOj + flljdblackij+ flzjdhispanicij+ fl3jdallotherij+ ,B4J- dfeml-j

+ flsjegenxblij+ ,36jegenxhispij+ ,37jegenxothij + c (11.7)

if;

Level 2 - Teachers

:50} = 700 + "0j

.BIj = 710
flZj = 720
331' = 730
1641' = 740

203

p51 = Y50

:66j = 1’60

1571' = 770
The regression coefficients for predicting student achievement from ethnic background
and gender are found in Table 11.6.

Table 11.6 Regression Coefficients for 2000 NAEP 4th Grade Mathematics Achievement
by Gender and Ethnicity

7 Model 4

'“te'CepUVool , , 233'5ll-95)”

Black (.1/01) -25.75(1.39)***

Hispanic (1102) g -20.22(1.29)***
_LA'L°E"eL(V03) '5'589-52)’

female<v04) '-.24(1.18)
Effect Black x female (v05) 2.19(.82)**

Effect Hispanic x female (1106) -.91 (.82)

Effect All other x female (v07) .54(1.38)

L_vLi variance (02) ’5 641.50

LVL2 Vafiance (T00) (232.27“

***p<.001,** p<.01,* p<.05

To make it easier to interpret the results from the achievement by demographic
group analysis, these coefficients can be used to compute the achievement for a student
with a given set of demographic characteristics. Because the demographic characteristics
are coded as 1 or 0, the computation involves summing the terms which are coded 1 for a
particular student.32 For example, to determine the predicted achievement for a black

female student who would have ones for the intercept, black, female and black female,

 

3’2 A table showing ones and zeros for all dummy variables is provided in the
Appendix.

204

the resulting equation would be:

Level 1 - Students

A

Yij = 1301' + Bljdblackl—j + B4jdfemij + 35 jeblackxefemij

Level 2 - Teachers (11.8)

[3‘0]. .—. 7‘00 = 233.51

5”,]. =y‘10 = —25.75

1341' =};40 = — ~24

195) = 750 = 2-19
When the slope values, the level two betas from Equation 11.8, are substituted in the
level one equation (Eq. 11.8), the result for Black female achievement is produced (Eq.
11.9).

1i}: (233.51) + (-25.75)*1 + (-.24)*1 + (2.19)*1 (11.9)
= 209.71

Predicted mathematics achievement for all subsets of students by demographic groups are
presented in Table 11.7. The table shows that the model predicts that White males are
predicted to have the highest scores followed closely by white females.

Table 11.7 Predicted Mathematics Achievement by Student Demographic Group with
Differences by Gender

 

 

Males Females Difference in Achievement
by Gender
White 233.51 233.27 - .24
Black 207.76 209.71 +1.95
Hispanic 213.29 212.14 -1.15
All other 227.93 228.23 + .30

 

Score differences between males and females of the same ethnicity are predicted to be

205

less than 2 points. Both Black and Other females are expected to do slightly better than
their male counterparts. The coefficient for the effect of female x Black is significantly

different from zero (p<.05).

Gender, Ethnicity, and SES

Another known source of variation in student achievement is SES as measured by
eligibility for free and reduced lunch. In NAEP, this variable is measured at the student
level.
Model 5. Gender, Ethnicity, and SES
Level 1- Students

Y1.j : :60j + flljdblackij+ flzjdhispanicij+ fl3jdallotherij+ ,B4j dfem)!- (11.10)

i + ,stegenxblij+ fl6jegenxhispij+ fl7jegenxothij
+fl8jSESiJ- + eij;

Level 2 - Teachers

1801' = 700 + ’01

flu = 710
.52)- = Y20
[331' = 730
fly = 740
:65} = 750
:66) = 760
1371' = 770
r68j = 770

206

The regression coefficients for predicting student achievement from ethnic background,
gender, and SES are found in Table 11.8.

Table 11.8 Regression Coefficients for 2000 NAEP 4th Mathematics Achievement by
Gender, Ethnicity, and SES

5 Models
Intercepf (V00) * 236.94(1.00)***
Black (v01) -20.42(1.43)***
Hispanic (v02) -16.40(1.27)***
All other (v03) -3.64(2.46)
Female (v04) .09(1.15)

Effect Blackxfemale (1,05) - 2.08(.80)*
Effect Hispanic x female (V06) -1.13(.81)

Effect All other x female (v07) 1.04(1.37) *

SES (v08) -14.25(1.17)***
“LVL 1Variance(02) * 628.74 *
LVL 2 Variance (T00) - _185.41***

“7 p<.001 ,*‘ p<.01,* _p<.05

As demonstrated in prior sections, the computational example to determine the
predicted achievement for a black female student eligible for free and reduced lunch who
would have ones for the intercept, black, female, black female, and free and reduced
lunch, would be:

Level 1 - Students

A

Yij = A” + Bljdblackl-j + B4jdfemij + stegenxblij + ng-SESU-

207

Level 2 - Teachers (11.11)

When the slope values, the betas from Level 2, Equation 8.11, are substituted in the Level
1 equation, the result for black female achievement is produced (Eq. 8.12).
ft], = (236.94) + (-20.42)*1 + (.09)*1 + (2.08)*1 + (-14.25* 1) (11.12)
= 204.44
Predicted mathematics achievement for all subsets of students by demographic group and
grade are presented in Table 11.9. The estimated point loss associated with free and

reduced lunch eligibility is -14.25 points and its coefficient is significantly different from

 

 

zero (p<.000).
Table 11.9 Predicted Mathematics Achievement by Student Demographic Group and
SES

Males Females Difference in Achievement

by Gender

White 222.69 222.78 + .09
Black 202.27 204.44 + 2.17
Hispanic 206.29 205.25 - 1.04
All other 219.05 220.18 + 1.13

 

This table indicates that the model predicts that all non—white student groups achieve
lower mathematics scores than white students once ethnicity, gender, and SES are taken
into consideration. Black students show the largest score difference by gender with girls

predicted to score 2.17 points higher than boys. Only Hispanic girls score lower than

208

their male counterparts (-1.04 points).

Summary of Section 1. Developing the level one equation for students. As the
level one base equation for 2000 NAEP 4th grade mathematics developed, average yearly
achievement was first predicted with a simple unconditional model. This model indicated
that achievement variance was nearly two times larger within teachers (between students)
than between teachers at level 2. Variables known to affect mean achievement
(ethnicity, gender, ethnicity x gender, and SES) were added to the level one equation
revealing that the coefficients for the intercept, Black, Hispanic, Black female, and SES
were significantly different from zero. Being eligible for free and reduced lunch
predicted a large scores drop for all students and White, Black, and All other girls have
higher scores than their male counterparts when SES is controlled. In other words, the

scores of boys from these groups drop farther than girls when SES is controlled.

II. Differential Effects of Teacher Characteristics, and Teacher Practices on Student
Achievement (Results at the teacher level)

This section addresses the questions:

1. Does mathematics achievement for demographic groups vary across
teachers?
2. How much variation in student mathematics achievement do variables

about teacher practices, characteristics, and professional development
explain?
3. Is the amount of variance explained by teacher practices, characteristics,

and professional development different for different student demographic

209

groups?

The analyses in this section first test if average achievement for the various
demographic groups varies across teachers. Nonsignificant error terms for variance (r00,
r10, ...rx0) indicate that group mean achievement does not vary across teachers so the error
terms predicting mean achievement for those demographic groups are dropped from the
model. Although the error terms are dropped, the fixed level two equations for non-
significant achievement by demographic groups are retained in the analysis. Analyses
indicated that average achievement for female, the female times ethnicity interactions,
Black, and All other did not vary significantly across teachers (17”O=0) so their error term
have been eliminated. Their variance is then captured as part of the error term for the

intercept (r00). The X2 tests for the intercept33

and the average mathematics achievement
for Hispanic students across teachers are significant (1:..0¢0).

2000 NAEP mathematics achievement varies across teachers for White and
Hispanic. The X2 tests indicate that the variances of the intercept and the average
mathematics achievement of Hispanics across teachers are significant (THO-i=0). The
effects teacher practices, teacher characteristics, and teacher professional development on
mean achievement for White and Hispanic is addressed below.34

Effects of teacher level variables on mean Hispanic achievement

While the fixed effects model for ethnicity, Model 2, predicts that average

 

33 When main effects variables are dummy coded, not only does [30 represent the
mean for white male achievement, but the variance of [30, 1:00, is the variance of white
male achievement (see Raudenbush, p.34).

34 Average free and reduced lunch varies significantly across teachers
(classrooms). The level two control variable for average SES is introduced in Model 7.

210

achievement varies significantly by ethnicity, a random slopes and intercepts model,
Model 6, tests whether or not the average achievement for White and Black students
varies significantly across teachers. In other words, do teacher practices, characteristics,
or professional development variables have differential effects on the average
mathematics achievement of Hispanic and/or White 4th grade students in the US? If
variance is significant across teachers, it is then possible to determine how much of the
variation in each groups’ achievement can be explained by single level two teacher
variable and if that effect is different for each group.
The unconditional Model 6 establishes the maximum amount of variance that can
be explained by teacher level variables for this data.
Model 6. Unconditional model for 4th grade Hispanic/ White achievement
Level 1- Students
Y1.j = IBOj + flljdblackij+ flzjdhispanicij+ fl3jdallotherij+ ,B4J- dfeml-j +,B5JSES,-j + eff-(11.10)
Level 2 - Teachers
.30): 700+r0) (11.11)
161} = 710

162) = 720 + ’2j

163; = 730
164} = 770
1351' = 770
where
YOOj = intercept, average in student achievement/achievement across teachers

all other predictors held constant

211

rOj = the error term or unique teacher effects
Y 10' = intercept, average Hispanic student achievement/achievement across
J
teachers all other predictors held constant
rnj = the error term or unique teacher effects

The estimated level two regression coefficients for Model 6 are given in Table

11.10.
Table 11.10 Regression Coefficients for 2000 NAEP 4th Mathematics Achievement by
Ethnicity
_ - "I; L ,- , - _ --M0d—9'5 -
’Interceq (v00) g - g .238.07(.94)***
“3593010)- -, , , .‘20-5711-44)”
Hispanicwzo) -16.21(1.29)***
f)! 0303(Y301 . '3-86945) --
_. Pme£V401 .- -, -, s - @8133)"
_ $30150) , . 442511-13)”
EVE—13751159073 (05> - g- ”623771
LVL 2 Intercept (T00)~ a182.54***
_ Hispanicszol” .. , 5940*

***p<.001,** p<.01,* p<.05

The effect of Hispanic varies significantly across teachers (T20=59.40, p<.05) as
does the effect of White (Too=182.54, p<.000). Selected coefficients from this analysis
serve as the base for the variance decompositions found in Tables 11.12-11.15. Next
classroom SES is added to the model to determine its effect on the average achievement
for Hispanic and White students across teachers. The variable for student free and

reduced lunch has been aggregated to the teacher level as control for classroom SES.

212

Model 7. 4th grade Hispanic, White achievement and SES

Level 1- Students - Same as unconditional Equation 11.10

Level 2 - Teachers

where

Y00j

You

roj

Y20j

Yzlj

rlj

:30) = 1’00 + Y0] “V555 + ’0) (1 1-12)
:31) = 1’10

,sz z 1’20 + y21avSES + r21.

:63} = 730
1681' = 770
.39j = 1’70

= intercept, average in student achievement/achievement across teachers
all other predictors held constant

= the change in average achievement/achievement associated with one unit
change in average free and reduced lunch

= the error term or unique teacher effects

= intercept, Hispanic achievement across teachers all other predictors held

constant

= the change in average Hispanic student achievement associated with one

unit change in average free and reduced lunch

= the error term or unique teacher effects

The estimated level two regression coefficients and variance components for

Model 7 are given in Table 11.11.

213

Table 11.11 Regression Coefficients for Black/White Achievement Predicted by Average

Classroom SES
7 h i I Model7
intercept W00) 236.43(.97)***
avSES (v01) -7.29(1.47)***
Black (v10) -19.39(1.44)***
Hispanic (v20) -15.60(1.31)***
”fiafivsés (v21) i3.59(2.59) *—
3; All other ((130) *3 i i l i * -3.45(2.45) ?
Dfem (1180) -2.05(.82)*
Message)” ‘-12.56(1.25)*** w
LVL~1 variance (62) ._ i f M 620734
Lit/C2 Varianze (T00) * 5180.04***
Hispanic (T20) 5590*

*** p<.001,“ p<.01,* p<.05

How much variation does average SES explain in the average mathematics
achievement for White and Black students across teachers? To answer this question the
following equation computes the variance in black achievement explained by average
SES. The estimated 1:10 values from Tables 11.10 and 11.11 would be substituted in the

following equation:

720(model 6) — 720(model 7) _ 59.40 — 55.90
720 (model 6) 59.40

 

 

= .059 = 5.9% (11.13)

This means that 5.9% of the variance in predicted average achievement for Hispanic
students is explained by average SES across classrooms (teachers). Using the same
process for the remaining tau, 1.2% of the variance in White average achievement is

attributable to SES across teachers.

214

Table 11.12 Variance Decomposition for Hispanic/White Achievement by SES

 

 

 

 

 

 

 

SES

Variables Uncond Free/Red Lunch
Intercept (v00) 238.07(.94)*** 236.43(.97)***

avSES(v02) -7.29(1.47)***
Hispanic (v10) -16.21(1.29)*** -15.60(1.31)***

avSES(v12) 3.59(2.59)
Intercept Variance (T00) 182.54*** 180.04***
Hispanic Variance (T40) 5940* 5590"
Level 1 variance (02) 622.71 620.341
% variance reduction intercept 5.9°/
% variance reduction Hispanic 1.2°/:I

 

 

 

SES explains variance in the average mathematics achievement of White and
Hispanic students across teachers in this data. For every one standard deviation increase
in free and reduced lunch, White students (which includes all non-Hispanic students) lose
-7.29 points while Hispanic students gain 3.59 points. SES explains 5.9% of the variance
in average White scores but only 1.2% of the variance in average Hispanic scores across
teachers.

Ethnicity and Teacher variables

Now that the context is set for the analyses at Level Two, it is possible to track the
variance explained by each teacher level variable and to determine its effect on average
mathematics achievement for Hispanic and White students across teachers. One variable
at a time is added to the level two equation, EQ 11.14. The term “variable x” is added to
“avSES” in Eq. 11.12 and represents a single Mathematics Practice (math sensemaking
activities, group and project activities, or group and project assessment activities),
Teacher Characteristic (mathematics preparation, preparation for mathematics instruction,

mathematics knowledge, reform orientation, diversity preparation), or Professional

215

Development (professional development frequency in the last year).

Wig/3nd Tmher Practices. A variance decomposition for teacher
practices can be found in Table 11.13. The table indicates that only Group and project
activities explains additional variance in average achievement for White students across
teachers once SES is controlled (2.5%).

Table 11.13 Variance Decomposition for Hispanic/White Controlling SES — Teacher

 

 

 

 

 

 

 

 

 

 

Practices
SES Practices
Nariables Uncond lFree/flied Sensemakinagrgup8ProjectiDLA
unc l c1v11es I
llntercept (v00) 238.07(.94)*** 236.43(.97)*** 236.45(.97)***i 236.45(.97)***i 236.45(.97)***
(v01) .29(.27" .68(.25)**| -1.12(1.01)
avSES(v02) -7.29(1.47)*** -7.19(1.47)***| -7.29(1.46)***| -7.32(1.47)***
Hispanic (v10) -16.21 (1 .29)*** -15.60(1.31)*** -15.71(1.32)***|-20.69(1.43)***I-15.59(1.32)***
(v11) -.39(.47)| .06(.49)| -.42(1.79)
avSES(v12) 3.59(2.59) 3.55(2.59)l 3.52(2.59)l 3.60(2.60)
Intercept Var (T00) 182.54*** 180.23*** 180.23***| 178.17*"*| 179.15***
Hispanic Var (T40) 5940* 55.90“ 56.72*| 57.94*| 57.06“
Level1Vari (02) 622.71 620.341 620.121 619.831 620.491
% variance reduction intercept 1.3% 1.3°/I 2.5"/I 1.9°/j
% variance reduction Hispanic 5.9% 4.5%! 2.5%1 3.9"/

 

 

 

For every one standard deviation increase in sensemaking activities, White scores
increase by up .29 points while Hispanic scores decrease by up to -.39 points. For every
one standard deviation increase in group and project activities, White scores increase by
.68 points while Hispanic scores remain nearly level. Both Hispanic and White scores
decrease withe a one standard deviation increase in Drill and lecture activities. Teacher
practices explain very little about the variance of Hispanic and White mathematics
achievement across teachers.

The results for teacher characteristics and professional development variables are

216

similar to the results for teacher practices. They explain little variance in average scores

of Hispanics and Whites across teachers once SES is taken into account.

217

12. RESULTS OF THE HLM ANALYSIS
FOR 1998 NAEP 4TH GRADE READING

The HLM analysis using NAEP data is intended to address the research questions
about the relationships between student achievement and

0 Teacher characteristics and practices

0 Characteristics of professional development
The analysis also is intended to address the ways in which these relationships vary by
student demographic group, including gender, ethnicity, and SES across the United States
and to see if the national reading results reflect trends similar to those found in the
Connecticut BTPS data, a more detailed data set. Analyses concerning the relationships
between teacher preparation institutions and teachers/students are not possible for this
data since the names of teacher preparation programs are not collected in NAEP. The
HLM analyses are weighted (origwt) and use five plausible values (rrpcml-S) for the
student outcome. There are 4,303 students and 828 teachers in the 1998 reading sample.

To set the context for addressing the questions, results are presented first which
describe the differences in student achievement (achievement status in NAEP data) across
student demographic groups. The analyses yield results in terms of regression
coefficients. To aid in understanding what the coefficients mean, predicted achievement
(achievement status score) is computed for selected combinations of student
characteristics.
The full set of reading variables for NAEP is given in Table 12. 1. The outcomes and

most teacher variables are the same as those used for the earlier SEM models. In certain

cases, variables that were only available in this NAEP data set were included as well.

218

Table 12.1 Description of Variables Used in the 1998 NAEP 4th Grade Reading Analyses

Continuous
(Outcomes)

Dichotomous

Continuous

Teacher
Practices

Variable
Abbreviation

rrpcm 1
rrpcm2
rrpcm3

rrpcm4

dblack
dhispanic
dallother

dfem

egenxbl
egenxhisp

egenxoth

reformact

tradact

compact

Variable Description
Level 1 - students

1St plausible value for reading achievement (mean 213.36,
sd=37.50)

2nd plausible value for reading achievement (mean 212.96,
30:37.78)

3rd plausible value for reading achievement (mean 212.64,
sd=38.05)

4rth plausible value for reading achievement (mean 213.04,
sd=37.55)

5th plausible value for reading achievement (mean 213.32,
sd=37.43)

Black (Black=1, else=0)

Hispanic (Hispanic=1, else=0)

All other ethnicities (all other=1, else=0)
Female (female=1, else=0)

Average free and reduced lunch (eligible=1, else=0)

Effects coded

Effect of Black female; interaction of eBlack x egender
where Black =1, White=-1, othenivise=0 multiplied by
female=1 , male=-1 .

Effect of Hispanic female; interaction of eHispanic x
egender where Hispanic=1, White=-1, othenivise=0
multiplied by female=1, male=-1.

Effect of All other female; interaction of eAIIother x egender
where All other =1, White=-1, othenivise=0 multiplied by
female=1, male=-1 .

Continuous

Original weight (correction for oversampling)
Level 2 - teachers
Grand mean centered

Frequency of reform activities (mean=38.18, sd=6.86)

Frequency of traditional activities (mean=37.69, sd=3.91)

Frequency of computer activities (mean=6.00, sd=2.54)

219

Table 12.1 (cont'd).

Teacher readprep Major/coursework/Iicense in reading/Englishlwriting,

Characteristics reading/English education (mean=15.49, sd=7.48)
blito Balanced literacy orientation (mean=14.93, sd=3.27)
techprep Technology preparation (mean=5.67,sd=159)
divprep Preparation for diversity(mean=3.95, sd=1.24)

assessprep Preparation for assessment (mean=28.95, sd=5.48)

readinst Preparation for reading instruction, pedagogical coursework
(mean=23.42, sd=3.32)
Professional pdf Professional development in the last year (mean=3.15,
Development sd=1.15)
Continuous avSES Average percent free and reduced lunch per teacher/

,, classroom aggregated from level 1 (mean=.46,sd=.50_)

 

1. Setting the Stage: Variation in Reading Achievement Gains by Student
Demographic Groups and SES (Results at the Student Level)

In this section, sets of background variables are added to look at changes in reading
gains in the United States. Model 1 predicts mean student gains across the sample and
partitions the variance between students and teachers where students are nested within
teachers. Models 2-5 predict differences in mean gains across gender, ethnicity, and SES
(eligibility for free and reduced lunch).

Mean Gains. To set the context for answering the questions about connections
between student achievement and teacher variables, results are presented first for
differences in student achievement across student demographic groups. The level one
and level two equations for the model are given followed by their estimated regression

coefficients in Tables 8.2.

220

Model I. Mean achievement
Level 1 - Students
Yrj=floj+ er)- (12.1)
Level 2 - Teachers
:50) = Yon + ’0)
where Y ij is the reading achievement for student i in teacher j’s classroom, flOj is the

intercept or mean students’ reading achievement within teacher j’s classroom, e is the

1.].

level 1 error, 700 is the mean value of classroom achievement for all teachers, and raj is
the level 2 error.

Table 12.2 1998 NAEP 4th Grade Average Reading Achievement

Model 1
intercept (v00) 3 216120.08)”
HLvL 1 variance (02) l f 3 1025.62 A T
Mavanasceooo) I C 451.13*** *

*** p<.001,** p<.01,* p<.05
The predicted average achievement score for 4th grade reading students for this sample of
US students is 223.41 points (YOO=214.66, sd=1 .07) when no other variables are included
in the model.

02 and 1:00 are used to compare variation in the errors at the teacher level with

variation at the student level using the formulas:

2 2
0'
—-—-—- = level one variance ___T_OO__

2 2 = level two variance (12.2)
0' + TOO 0' + TOO

70% of the variance in achievement for Model 1 is within teachers (between students)

221

leaving 30% of the variance in achievement between teachers. That is to say, there is 2.3
times more variation within teachers than between teachers.

Mean Achievement by Demographic Group. Next, demographic variables are
added to form Models 2, 3 and 4. Model 5 introduces the control variables for SES. The
equations for Model 2, ethnic group, are presented first (Eq.12.3) To interpret the
coefficients, the estimated score for ethnicity can be computed using the appropriate set
of ls and OS for the dummy variables. As an example, Equations 12.4 and 12.5
demonstrate the computation for obtaining Black student achievement. Estimated
achievement by ethnicity are presented in Table 12.3.35
Model 2. Ethnic groups
Level 1- Students

Y ij = ,BOJ- + flljdblackiji- flzjdhispaniclj+ ,B3jdallotherij + eij;

Level 2 - Teachers (12.3)

160) = 700 + "0j

:60 = 710
flzj = 720
:631' = 730

The regression coefficients for predicting student achievement from ethnic background

are found in Table 12.3.

 

35 Note that the category for All other includes Asian and those students who did
not report ethnicity. Often these smaller groups are folded into the intercept since they
are not significantly different from the intercept. In this case, the coefficient for all other
was significant at p<.05. Additionally, it was important to maintain the meaning of the
intercept as a specific group, in this case white students, before analyzing the differential
effects of teacher variables.

222

Table 12.3 Regression Coefficients for 1998 NAEP 4th Grade Reading by Ethnicity

* 3 Model 2
Interceptwoo) * 224.40(1.04)***
Black (v01) -28.29(2.00)***
Hispanic (v02) -25.24(2.08)***
"' All other (1103) 6978(320)“
LVL? variance (62 ) 1988.20
LVL 2 Variance(T00) ’ C 5276.13***

***p<.001,** p<.01,* p<.05

To make it easier to interpret the results from the achievement by ethnicity group
analysis, these coefficients can be used to compute the achievement for a student with a
given set of demographic characteristics. Because the demographic characteristics are
coded as 1 or 0, the computation involves summing the terms which are coded 1 for a
particular student.36 For example, to determine the predicted achievement for a black
student who would have ones for the intercept and black the resulting equation would be:

Level 1 - Students

A

Yij = fl0j+1§1jdblacklj (124)

Level 2 - Teachers
flOj = foo = 224.40
,6]!- =}710 = -28.29
When the slope values, the betas from Equation 8.4, are substituted in the level one

equation (Eq. 8.4), the result for black achievement is produced (Eq. 8.5).

 

36 A table showing ones and zeros for all dummy variables is provided in the
Appendix.

223

r, = (224.40) + (-28.29)*l (12.5)
= 196.11
Predicted reading achievement for all demographic groups are presented in Table 12.4.
This table shows that the model predicts that White students score higher in reading than
the other ethnic groups. The coefficients for ethnicity are significantly different from

zero.

Table 12.4 Predicted Reading Achievement Student Demographic Group

 

 

Ethnic Group Achievement
White 224.42
Black 1 96. 1 1
Hispanic 1 99. 1 6
All other 214.62

 

In the next model, gender predicts average reading achievement.

Model 3. Gender

Level 1- Students
4- = 41+ 4.- dfemi- +

Level 2 - Teachers (12.6)
:50) = 700 + '0}
164) = 710

The regression coefficients for predicting student achievement from gender are found in

Table 12.5.

224

Table 12.5 Regression Coefficients for 1998 NAEP Reading Achievement by Gender

Mode|3
Intercept (v00) 213.69(1 .35)***
Female (v04) 4.80(1.37)***
Lvti Variance(02) 9101961 f g
LVI: 2Variance (Too) 7 A451.34***

*** p<.001,“ p<.01,* p<.05
These coefficients indicate that male mean achievement in reading is predicted to be
213.69 points compared to 218.49 points for females. On average, girls are expected to

score nearly 5 points higher than boys on NAEP 4th

Grade Reading. The coefficient for
female is significantly different from zero.
Model 4. Combined gender and ethnicity

Model 4 tests whether or not there is an interaction between gender and ethnicity
in the NAEP 4th grade reading data. Equation 12.7 computes coefficients for
combinations of gender and ethnicity.
Level 1- Students

Y i]- = IBOj + flljdblackij+ flzjdhispaniclj+ fl3jdallotherij+ ,3,” dfem,-j

+ flsjegenxblij+ fl6jegenxhtspij+ ,67jegenxothij + c (12.7)

if;

Level 2 - Teachers

:60) = 700 + r0j

.51} = 710
.62} = 720
:63) = 730
1341' = 740

225

165} = 750
floj = 760
fl7j = 770
The regression coefficients for predicting student achievement from ethnic background

and gender are found in Table 12.6.

Table 12.6 Regression Coefficients for 1998 NAEP 4th Grade Reading Achievement by
Gender and Ethnicity

M0d6l 4’ g *
Intercept(y00) 220.94(1.41)***
V Black (Y01) 7 '28.62(2.04)*** 7
* H’SF’aniClVOZ) , , -25.15(2.08)***
A" gtner1vo3) ~10.13(3.14)**
Fema'e (V04) g 6.94(2.01)**

‘ 'EvffectiBlack x female (v05) (437(131)
Effect Hispanic x female (V06) -.83(1.40)

_ Effect All other x female (1107) 2.30(2.07)

’LvL1 variance (02) 980.99 ’
LVL 2 Variance (T00) 276.24***

*** p<.001,“ p<.01,* p<.05

To make it easier to interpret the results from the achievement by demographic
group analysis, these coefficients can be used to compute the achievement for a student
with a given set of demographic characteristics. Because the demographic characteristics
are coded as l or 0, the computation involves summing the terms which are coded 1 for a

particular student.3'7 For example, to determine the predicted achievement for a black

 

37 A table showing ones and zeros for all dummy variables is provided in the
Appendix.

226

female student who would have ones for the intercept, black, female and black female,
the resulting equation would be:

Level 1 - Students

A

Yij = 1301' + Bljdblackij + B4jdfemij + 85 jeblackxefemij

Level 2 - Teachers A A (12.8)
flOj = 700 = 220.94

p1]. =y"10 = —28.62
1641' =}?40 = 694
135,- =i750 = ‘37

When the slope values, the level two betas from Equation 12.8, are substituted in the
level one equation (Eq. 12.8), the result for Black female achievement is produced (Eq.
12.9).

in, = (220.94) + (-28.62)*1 + (6.94)*1 + (-.37)*1 (12.9)

= 198.89

Predicted reading achievement for all subsets of students by demographic groups are
presented in Table 12.6. The table shows that the model predicts all female students are
predicted to have higher scores than their male counterparts by at least six points.
Coefficients for gender times ethnicity effects are not significantly different from zero.

Table 12.7 Predicted Reading Achievement by Student Demographic Group with
Differences by Gender

 

 

Males Females Difference in Achievement
by Gender
White 220.94 227.88 + 6.94
Black 192.32 198.89 + 6.57
Hispanic 195.79 201.90 + 6.11
All other 210.81 220.05 + 9.24

 

Black students are predicted to score lower on 4th Grade NAEP Reading than the other

227

demographic groups, including those students who likely have English as a second
language.
Gender, Ethnicity, and SES
Another known source of variation in student achievement is SES as measured by
eligibility for free and reduced lunch. In NAEP, this variable is measured at the student
level.
Model 5. Gender, Ethnicity, and Grade Level
Level 1- Students
Y ij = IBOj + flljdblackij+ ,szdhispaniclfr fl3jdallotherij+ ,6,” dfemI-j (12.10)
+ ,stegenxblij+ ,B6jegenxhispij+ fl7jegenxothij
+fl8jSESij + e if;
Level 2 - Teachers

:60) = 700 + rOj

:51} : 710
:62)‘ = 720
:33) = 730
1641' = 740
:35) = Y50
166] = 7’60
.671' = 770
fl8j = 770

The regression coefficients for predicting student achievement from ethnic background,

gender, and SES are found in Table 12.8.

228

Table 12.8 Regression Coefficients for 1998 NAEP 4th Reading Achievement by
Gender, Ethnicity, and SES

Model 5
|ntercept (yoo) 225.10(1.40)***
Black (v01) -22.65(2.13)***
7 Hispanic (Yoz) , , -20..92(203)“*
All other (v03) -8.37(3.09)**
Female (v04) 7 7.23(1.98)**

Effect Blackxfemale (v05) l -.38(1.30)

Effect Hispanic x female (v06) -.69(1.30)

.r fmav. A _

Effect All other x female (v07) 239(139)

; SE§<voo> e . ee - 45940-64)“
'vai valiancemz) - 962.97-
LVL 2Variance (T00) # f 22026“ g

***p<.001,** p<.01,* p<.05

As demonstrated in prior sections, the computational example to determine the
predicted achievement for a black female student eligible for free and reduced lunch who
would have ones for the intercept, black, female, black female, and free and reduced
lunch, would be:

Level 1 - Students

A

Yij = 1301' + Bljdblackl-j + B4jdfemij + B5J-egenxblij + ngSESij

Level 2 - Teachers (12.11)
30)- : too = 225.10
[31]. =y‘10 = ——22.65
.34.) =i¢40 = 7-23
1&5) = 7350 = — 38

A

I88] :f80 2—1594

229

When the slope values, the betas from Level 2, Equation 12.11, are substituted in the
Level 1 equation, the result for black female achievement is produced (Eq. 12.12).
'1?”- = (225.10) + (-22.65)*1 + (7.23)*l + (-.38)*1 + (-15.94*1) (12.12)
= 193.36
Predicted reading achievement for all subsets of students by demographic group
and grade are presented in Table 12.9. The estimated point loss associated with free and
reduced lunch eligibility is -l2.31 points.

Table 12.9 Predicted Reading Achievement by Student Demographic Group and SES

 

 

Males Females Difference in Achievement
by Gender
White 209.16 216.39 + 7.23
Black 186.51 193.36 + 6.85
Hispanic 188.24 194.78 + 6.54
All other 200.79 210.41 + 9.62

 

This table also indicates that the model predicts that all non-white student groups achieve
lower reading scores than white students once ethnicity, gender, and SES are taken into
consideration. Hispanic students show the largest differences by gender with girls
predicted to lose -4.23 points. The point gap between males and females in Table 12.9 is
nearly the same as the point gap in Table 12.7 which did not include SES.

Summagy of Section 1. Developing the level one equation for students. As the
level one base equation for 1998 NAEP 4th grade reading developed, average yearly
achievement was first predicted with a simple unconditional model. This model indicated
that achievement variance was 1.5 times larger within teachers (between students) than
across teachers at level 2. Variables known to affect mean achievement (ethnicity,

gender, ethnicity x gender, and SES) were added to the level one equation revealing that

230

while differences in achievement exist within teachers, the interaction terms for gender

time ethnicity were not significant predictors of student achievement.

II. Differential Effects of Teacher Characteristics, and Teacher Practices on Student
Achievement (Results at the teacher level)
This section addresses the questions:
1. Does reading achievement for demographic groups vary across teachers?
2. How much variation in student reading achievement do variables about
teacher practices, characteristics, and professional development explain?
3. Is the amount of variance explained by teacher practices, characteristics,
and professional development different for different student demographic
groups?

The analyses in this section first test if average achievement for the various
demographic groups varies across teachers. Nonsignificant error terms for variance (r00,
r10, ...rx0) indicate that group mean achievement does not vary across teachers so the error
terms predicting mean achievement for those demographic groups are dropped from the
model. Although the error terms are dropped, the fixed level two equations for non-
significant achievement by demographic groups are retained in the analysis. Analyses
indicated that average achievement for female, Hispanic and All other across teachers did
not vary significantly from the intercept (1:..0=0) so the error terms have been eliminated.

Their variance is then captured as part of the error term for the intercept (r00). The X2

231

tests for the intercept, average reading achievement of White students,38 and for the
average reading achievement of Black students across teachers are significant (“CO-0950) so
they are addressed in a series of analyses which include teacher practices, teacher
characteristics, and teacher professional development.
Effects of teacher level variables on mean achievement by Black and White

While the fixed effects model for ethnicity, Model 2, predicts that average
achievement varies significantly by ethnicity, a random slopes and intercepts model,
Model 6, tests whether or not the average achievement for white and black students varies
significantly across teachers. In other words, do teacher practices, characteristics, or
professional development variables have differential effects on the average reading
achievement of Black and/or White 4th grade students in the US? If variance is
significant across teachers, it is then possible to determine how much of the variation in
each groups’ achievement can be explained by single level two teacher variable and if
that effect is different for each group.

The unconditional Model 6 establishes the maximum amount of variance that can
be explained by teacher level variables for this data.
Model 6. Unconditional model for 4th grade Black/White achievement
Level 1- Students

Y tj = :BOj + flljdblackij+ flzjdhispaniclj+ fl3jdallotherij+ fl4j dfem!)-

+.55,SES,-j + er,- (12.10)

 

38 When main effects variables are dummy coded, not only does [30 represent the
mean for white male achievement, but the variance of [30, 1:00, is the variance of white
male achievement (see Raudenbush, p.34).

232

Level 2 - Teachers

where

12.10.

Y00j

1’0 J

Yloj

I'4j

:50) = 1’00 + r0)- (12.11)

1611': 710+r4j

1521‘ = 720
:63) = 730
fly = 770
’55) = 770

= intercept, average in student achievement/achievement in teacher j’s
classroom all other predictors held constant

= the error term or unique teacher effects

= intercept, average black student achievement/achievement across
teachers all other predictors held constant

= the error term or unique teacher effects

The estimated level two regression coefficients for Model 6 are given in Table

233

Table 12.10 Regression Coefficients for 1998 NAEP 4th Reading Achievement by

Ethnicity

I I Model6

’lniercepftwoo) ’ ’ 226.19(1.28)*** ’

F Black (v10) 3 f ’ r-22.50(2.09)***

l—lispahicfyzo)~ * i i l f if if :21.20(2.02)***
All other (v30) -8.34(3.13)

infofemivwfl if I i 1' 536(129) ’

" sap/50) i if i i i l-16.05(1.62)***
LVLF1\lariance(02) * f i if 3 956225 f
LVL 2 Intercept (700)" C 201.52“ *

W Black (T10)~ ~124.27*

*** p<.001,** p<.01,* p<.05
~ X tests from analysis using average of mrpcm1-5, no weights

The effect of Black varies significantly across teachers (T,o=63.36, p<.05) as does
the effect of White (T00=213.02, p<.000). Selected coefficients from this analysis serve as
the base for the variance decompositions found in Tables 12.12-12.15. Next district SES
is added to the model to determine its effect on the average achievement for Black and
White students across teachers. The variable for student free and reduced lunch has been
aggregated to the teacher level as control for classroom SES.

Model 7. 4th grade Black, White achievement and SES
Level 1- Students - Same as unconditional Equation 12.10

Level 2 - Teachers

fir),- = 700 + 701 avSES + raj (12.12)
fl]! = 710 + y“ avSES + r4]-

flZj = 720

1631 = 730

234

where

Y00j

You

Yloj

Yllj

TU

'58} = Y70

169} = 1’70

= intercept, average in student achievement/achievement in teacher j’s
classroom holding all other predictors constant

= slope, the change in average achievement/achievement associated with
mean free and reduced lunch, all other predictors held constant

= the error term or unique teacher effects

= intercept, black student achievement/achievement in teacher j’s

classroom all other predictors held constant

= slope, the change in average black student achievement/achievement

associated with one unit change in average free and reduced lunch

= the error term or unique teacher effects

The estimated level two regression coefficients and variance components for

Model 7 are given in Table 12.1].

235

 

 

 

 

Table 12.11 Regression Coefficients for Black/White Achievement Predicted by Average
District SES
Model7

Inferceptwoo) * i f f i * '224.41(1.32)***
, “asts (v01) ’ '-7.93(2.06)***
.3 _Blafzkivm) A 7 7 7 7 if 7 0-21.67(2.08)*** .
F T avSES (v11) 8 f i i " '-2.09(3.42) ’3 i
I Hispanic (v20) 3 i f f * i-20.06(2.05)***
_, Moment/35) if i i l * l-7.81(3.14)* if
_ Dfem (yéé) V -3.11(.98)***
TsEs (V90) * i ,, i if * -14.31(1.74)***
”LvLi variance (62) 7 952.93
LVL 2Variance(roo) * __ ”198.921"

(T10) 116.26*

***p<.001,** p<.01,* p<.05

How much variation does average SES explain in the average reading
achievement for White and Black students across teachers? To answer this question the
following equation computes the variance in black achievement explained by average

SES. The estimated 1310 values from Tables 12.10 and 12.11 would be substituted in the

following equation: T:

 

 

110(model 6)— 110(model 7) = 63.36-55.93 = .117 =11.7% (12.13) M

110(model 6) 63.36

This means that 11.7% of the variance in predicted average achievement for black
students is explained by average SES across teachers (classrooms). Using the same
process for the remaining tau, less than 1% of the variance in White average achievement

is attributable to SES across teachers.

236

Table 12.12 Variance Decomposition for Black/White Achievement by SES

 

 

 

 

 

 

 

 

 

SES
Variables Uncond Free/Red Lunch
Intercept (v00) 26.19(1.28)*** 224.41 (1 .32)***
avSES(v02) -7.93(2.06)***
Black (v10) -22.50(2.09)*** -21.67(2.08)***
avSES(v12) —2.09(3.42)
Intercept Variance (T00) 201.52*** 198.92***
Black Variance (T10) 124.27* 116.26*
Level 1 variance (02) 956.25 952.93
% variance reduction intercept 1.6%
% variance reduction black slope 64%|

 

A one standard deviation increase in average free and reduced lunch across
teachers reduces average Black achievement by -5.12 points and decreases average White
student achievement by -4.83 points. The variance decomposition from the table
indicates that average SES explains 11.7% of the variance in average Black achievement
across teachers and almost none of the variance in average White achievement. SES,
therefore, has a differential effect on Black and White student (non-Black) achievement.
Ethnicity, SES, and Teacher variables

Now that the context is set for the analyses at Level Two, it is possible to track the
variance explained by each teacher level variable controlling for SES and to determine its
effect on average reading achievement for Black and White students across teachers. One
variable at a time is added to the level two equation, EQ 12.14. The term “variable x” is
exchanged for a single Reading Practice (read sensemaking activities, group and project
activities, or group and project assessment activities), Teacher Characteristic (reading
preparation, preparation for reading instruction, reading knowledge, reform orientation,

diversity preparation), or Professional Development (professional development frequency

237

 

 

in the last year).
Model 8. Template for Level 2 equations
Level 1 remains constant (Eq. 12.10)
Level 2 - Teachers
1601' = 700 + yo, avSES + 1’02" variable x + roj (12.14)

:61)“ = y“) + y” avSES + 712.. variable x+ r4]-

flZj = 1’20
:33j = 730
flay = 770
.59j = 770
where
702.3 = change in average in average student achievement associated with a one
unit change in “variable x” holding SES constant at the classroom
(teacher) average
and

Y1? , = the change in average black student achievement associated with one
-..j

unit change in “variable x” holding SES constant at the classroom
(teacher) average
Ethnicity and Teacher Practices. A variance decomposition for teacher
practices can be found in Table 12.13. The table indicates that no additional variance in
average achievement for Black students is explained by teacher practices across teachers

when SES is controlled. Small amounts are explained for White students.

238

 

Table 12.13 Variance Decomposition for White/Black Controlling SES - Teacher

 

 

 

 

 

 

 

Practices
avSES Practice
Variables iJncond Free/Red Lu Reform iTraditional iComputer
lactivities lActivities IActivities
Intercept (v00) 226.19(1.28)*** 224.41(1.32)*** 224.39(1.34)***i233.87(1.21)***i233.94(1.21)***
(v01) .02(.14)I .46(.39)| 1.11(.33)**
avSES(voz) -7.93(2.06)*** -7.99(2.07)***| -4.82(1.96)*| -4.93(1.97)*
Black (v10) ~22.50(2.09)*** -21.67(2.08)*** -21.75(2.08)***| -18.42(1.86)***| -18.68(1.90)**‘l
(v11) .40(.28)l -.48(.59 .03(.56)
avSES(v ) -2.09(3.42) 1.56(3.40) -5.04(3.53 -5.21(3.44)**
12 I .
Intrcpt Var(T00) 201.52*** 198.92*** 19952*** 21 1 .15*** 204.45***
Black Var (T40) 124.27* 116.26* 112.15* 5761* 60.71
(02) 56.25 952.93 952.771 530.081 529.
% variance reduction intercept 1.6% >1°/ol 0°/ r 0°/
% var reduction black slope 6.4% 9-8%L 53.6%] 499°/

 

 

 

 

For every one standard deviation increase in sensemaking and group and project

assessment activities, White scores increase by up to .74 points while Black scores

decrease by up to -.48 points. This is an indication that there may be differential teacher

practice effects present in 1998 NAEP 4th grade reading data. For every one standard

deviation increase in group and project activities, White scores increase by 1.11 points

while Black scores remain nearly level. None of the variables explain more variation in

average Black student scores across teachers than SES alone; however, the variables do

explain small amounts of variance in White average achievement across teachers (up to

4%) after controlling for SES.

Ethnicity and Teacher Characteristics. The variance decomposition for teacher

 

characteristics follows in Table 12.14. There are only two variables which explain more

variance in achievement across teachers than SES alone - reading preparation and reform

orientation. Reading preparation explains additional variation in achievement for both

239

 

 

Black and White Students across teachers while reform orientation only explains

additional variance in achievement for Black students. There is twice as much variation in

the average achievement for Black students across teachers when compared to White

students. There is 5 times as much variance for black achievement across teachers

explained by reform orientation. A more detailed explanation follows the table.

Table 12.14 Variance Decomposition for Black/White Controlling for SES - Teacher

 

 

 

 

 

 

 

 

Characteristics
SES Teacher Characteristics

Variables Uncond Free/Red Lu IReading Prep iAssess Prep 1
Intercept (v00) 226.19(1.28)*** 224.41 (1 .32)*** 224.35(1.32)***| 224.19(1.32)***|

(v01) -.21(.12 -.28(.17
avSES(vOZ) -7.93(2.06)*** 7.76(2.06)**| 7.84(2.06)**|
Black (v10) 922.50(2.09)*** -21.67(2.08)*** 21 .65(2.08)***| 21 .46(2.08)***|
(V11) .14(.24)| .37(.30)|
avSES(v12) -2.09(3.42) 1'89(3'43)l 2.01 (3%
Intercept Var(r00) 201.52*** 198.92*** 197.99***| 197.83***|
Black Var (T40) 124.27* 116.26* 114.98*| 114.29*
(02) 956.25 952.93 952.7q 952.71,
°/o variance reduction intercept 1.6% 1.8%i 1.8%i
% variance reduction black slope 6.4% 7.5%1 8.0%]

 

 

 

 

Note that for every standard deviation increase in reading preparation (degrees/

licenses), White students scores increase by 1.49 points and Black scores drop by 1.59

points. The negative relationship between student achievement and teacher preparation

persists for Black students and it accounts for 16% of the variation in average Black

achievement across teachers in this data. White students gain points but teacher reading

preparation only account for a little more than 1% of the variance in average White

achievement across teacher - a differential effect. Across the US, low SES Black students

are likely to have teachers with less reading preparation than low SES White students. It

240

 

at

is puzzling that this is not the case in Connecticut where the explanation for the negative
relationship between reading preparation and all student achievement controlling for SES
appears related to the types of degrees that early childhood educators often receive.
Reform orientation controlling for SES across teachers explains 18.4% of the

variation in Black average achievement over the course of one year while explaining little
variance in the average achievement for White students. For every one standard deviation
increase in teachers’ reform orientation, black students are expected to lose an additional
-1.81 points over the course of one year. White students are expected to gain 1.02 points
but it explains little of the variation in average White student scores across teachers.

Teachers’ read knowledge explains 13.8% of the variation in average Black
achievement across teachers but nothing for average White achievement when SES is
controlled. A one standard deviation increase in reading knowledge increases Black
achievement by 2.81 points while leaving White achievement almost level.

SES continues to be the most important single predictor of Black achievement
across teachers but SES explains little about white achievement. Three teacher
characteristic variables increased the variance explained in average Black achievement
across teachers once SES was controlled, read preparation, reform orientation, and
reading knowledge. The effect of these variables was differential since they only
explained variance in average Black achievement. Reform orientation and reading
preparation reduces Black achievement while increasing White achievement. One
exception is teachers’ reading knowledge which increases average Black achievement
while leaving average White achievement virtually unchanged.

Ethnicity and Teacher Professiopgl Development. This analysis focuses on the

241

 

 

 

differential effects of teacher professional development as measured by professional
development impact, mentoring frequency, and professional development frequency. All
professional development variables explain additional variance in reading achievement
achievement across teachers for non-blacks after SES is controlled. Only professional
development frequency explains additional variance for black achievement across
teachers once SES is controlled. The variance decomposition for professional

development follows in Table 8.16.

Table 12.15 Variance Decomposition for Black/White Controlling for SES - Teacher
Professional Development

 

 

 

 

 

 

 

 

ariables Uncond SES PD Frq

Intercept (v00) 226.19(1.28)*** 224.41(1.32)*** 224.27(1.34)***
(v01) -.68(.86)
avSES(voz) ~7.93(2.06)*** 7.93(2.07)***
Black (v10) r22.50(2.09)*** L21.67(2.08)*** -21.54(2.08)***
(v11) 3.38(1.58)*
avSES(v12) -2.09(3.42) -1.72(3.35)
Intercept Var(T00) 201 .52*** 198.92*** 200.72***
Black Var (T40) 124.27* 116.26* 105.09*
(62) [956.25 1952.93 951.69

% variance reduction intercept 1.6% >1%
% variance reduction black slope 6.4% 15.4%

 

 

 

For every standard deviation increase in professional development frequency, black
students are predicted to gain about .17 points compared to White students who lose

about half a point. Professional development frequrncy controlling for SES districts

nothing more than SES alone for either group.

Summary of differential effects by ethnicig in 1998 NAEP 4th mde reading
- Average achievement for Black and White students varies across teachers for 1998

NAEP 4th grade reading data. SES, group and project assessment activities (teacher

242

practice), reading preparation, reform orientation and reading knowledge (teacher
characteristics) explain up to 18.4% of the variance in average Black achievement across
teachers. Reading sensemaking activities, group and project activities, and group and
project assessment activities (teacher practices), reading preparation and reform
orientation (teacher charactersitics) explain very small amounts of the variation in
average White student achievement (up to 4%). Professional development frequency
does not explain variation for either group.

Of the variables, predominantly teacher practices, which account for variation in
average White scores when SES is controlled, 8 one standard deviation increase in those
variables increases White scores while simultaneously decreasing Black scores. They are
examples of differential effects. Of the variables which account for variance in average
Black achievement controlling for SES, predominantly teacher characteristics, a one
standard deviation increase teachers read knowledge increases Black scores (2.81 points)
while leaving White achievement level. The rest of the variables differentially decrease

Black scores while increasing White scores by up to 1.8 points.

243

13. RESULTS OF THE HLM ANALYSIS
FOR 2000 NAEP 4TH GRADE READING
The HLM analysis using 2000 NAEP 4th grade reading data is intended to address
the research questions about the relationships between student achievement and
0 Teacher characteristics and practices
0 Characteristics of professional development
The analysis also is intended to address the ways in which these relationships vary by
student demographic group, including gender, ethnicity, and SES across the United States
and to see if the national reading results reflect trends similar to those found in the
Connecticut BTPS data, a more detailed data set. Analyses concerning the relationships
between teacher preparation institutions and teachers/students are not possible since the
names of teacher preparation programs are not collected in NAEP. The HLM analyses are
weighted (origwt) and use five plausible values (rrpcml-S) for the student outcome.There
are 4,548 students and 918 teachers in the 1998 reading sample.

To set the context for addressing the questions, results are presented first which
describe the differences in student reading achievement (achievement status in NAEP
data) across student demographic groups. The analyses yield results in terms of
regression coefficients. To aid in understanding what the coefficients mean, predicted
achievement (achievement status score) is computed for selected combinations of student
characteristics.

The full set of reading variables for the NAEP analysis is given in Table 13.1.The
outcomes and most teacher variables are the same as those used for the earlier SEM

models. In certain cases, variables that were only available in this NAEP data set were

244

included as well.

Table 13.1 Description of Variables Used in the 2000 NAEP 4th Grade Reading Analyses

Type Variable Variable Description
Abbreviation

Level 1 - students

Continuous rrpcm1 1St plausible value for reading achievement (mean 212.98,
(Outcomes) sd=39.61)
rrpcm2 2nd plausible value for reading achievement (mean 213.45,
sd=3919)
rrpcm3 3rd plausible value for reading achievement (mean 213.45,
sd=3911)
rrpcm4 1rth plausible value for reading achievement (mean 213.16,
sd=3931)
rrpcm5 5th plausible value for reading achievement (mean 213.53,
sd=3937)
Dichotomous dblack Black (Black=1, else=0)
dhispanic Hispanic (Hispanic=1, else=0)
dallother All other ethnicities (all other=1, else=0)
dfem Female (female=1, else=0)
SES Average free and reduced lunch (eligible=1, else=0)
Effects coded egenxbl Effect of Black female; interaction of eBlack x egender

where Black =1, White=-1, otherwise=0 multiplied by
female=1, male=-1.

egenxhisp Effect of Hispanic female; interaction of eHispanic x
egender where Hispanic=1, Whit =-1, otherwise=0
multiplied by female=1, male=-1.

egenxoth Effect of All other female; interaction of eAIIother x egender
where All other =1, White=-1, otherwise=0 multiplied by
female=1, male=-1.

Continuous Origwt Original weight (correction for oversampling)

Level 2 - teachers

Continuous Grand mean centered
Teacher refrmact Frequency of reform activities (mean=15.03, sd=2.79)
Practices
tradact Frequency of traditional activities (mean=24.36, sd=2.32)
compact Frequency of computer activities (mean=7.31, sd=2.94)

245

Table 13.1 (cont'd).

Teacher readprep Major/coursework/license in reading/Englishlwriting,

Characteristics reading/English education (mean=.62, sd=1.27)
techprep Technology preparation (mean=7.85,sd=2.13)
divprep Preparation for diversity(mean=6.27, sd=1.68)

assessprep Preparation for assessment (mean=16.56, sd=3.20)

readinst Preparation for reading instruction, pedagogical coursework
(mean=21.12, sd=3.01)
Professional pdf Professional development in the last year (mean=, sd=)
Development
Continuous avSES Average percent free and reduced lunch per teacher!

classroom aggregated from level 1 (mean=.45,_ sd=.38) -

1. Setting the Stage: Variation in NAEP 2000 Reading Achievement Status by
Student DemographicGroups and SES (Results at the Student Level)

In this section, sets of background variables are added to look at changes in reading
gains in the United States. Model 1 predicts mean student gains across the sample and
partitions the variance between students and teachers where students are nested within
teachers. Models 2-5 predict differences in mean gains across gender, ethnicity, and SES
(eligibility for free and reduced lunch).

Mean Gains. To set the context for answering the questions about connections
between student achievement and teacher variables, results are presented first for
differences in student achievement across student demographic groups. The level one
and level two equations for the model are given followed by their estimated regression
coefficients in Tables 8.2.

Model 1. Mean achievement

Level 1 - Students

szp0j+eij (13.1)

246

Level 2 - Teachers

.30)- : 1’00 + ’0}
where Y ij is the reading achievement for student i in teacher j’s classroom, t60j is the
intercept or mean students’ reading achievement within teacher j’s classroom, e,j is the
level 1 error, 700 is the mean value of classroom achievement for all teachers, and r0j is
the level 2 error.

Table 13.2 2000 NAEP 4th Grade Average Reading Achievement

Model 1
Intercepfwoo) * 214.66(1.07)***
LVL 1 Variance (02) 1183.40
LVL 2Variance (TOO) 479.40***

*** p<.001,** p<.01,* p<.05

The predicted average achievement score for 4th grade reading students for this sample of
US students is 214.66 points (y00=214.66, sd=1.07) when no other variables are included
in the model.

02

and 1:00 are used to compare variation in the errors at the teacher level with
variation at the student level using the formulas:

0' 2 *[2
—— = level one variance —09—

2 7 2 level two variance (13.2)
0' + TOO O" '1' TOO

A little over 70% of the variance in 2000 reading achievement for Model 1 is within
teachers (between students) leaving a little less than 30% of the variance in achievement
between teachers. That is to say, there is 2.5 times more variation within teachers than

between teachers, very similar to proportion found in 1998 reading.

247

Mean Achievement by Demographic Group. Next, demographic variables are
added to form Models 2, 3 and 4. Model 5 introduces the control variable for SES. The
equations for Model 2, ethnic group, are presented first (Eq.13.3) To interpret the
coefficients, the estimated score for ethnicity can be computed using the appropriate set
of Is and Os for the dummy variables. As an example, Equations 13.4 and 13.5
demonstrate the computation for obtaining Black student achievement. Estimated
achievement by ethnicity are presented in Table 13.3.39
Model 2. Ethnic groups
Level 1- Students

Y ij = IBOj + flljdblackij+ fiZjdhispanicij+ fl3J-dallotherij etj;

Level 2 - Teachers (13.3)

:60) = Y00 + ’0}

1611' = 710
.sz = 720
:63) = 7’30

The regression coefficients for predicting student achievement from ethnic

background are found in Table 13.3.

 

39 Note that the category for All other includes Asian and those students who did
not report ethnicity. Often these smaller groups are folded into the intercept since they
are not significantly different from the intercept. Additionally, it was important to
maintain the meaning of the intercept as a specific group, in this case white students,
before analyzing the differential effects of teacher variables.

248

 

Table 13.3 Regression Coefficients for 2000 NAEP 4th Grade Reading by Ethnicity

Model2
IntercepwaO) ’ l f f * 222.77(1.07)***
mElaEfiVM) if if f i f ’ i-25.01(2.20):*‘* I.
*Higpahic(v0;) * ’ v-23.06(1.81):** ’
F’Allgotherivgé)” M l ‘ H-5.66(3T647) _.
LVVL1Vafianvcei(02) I if if 1163.26
[VL‘ZVariahcgtrgof * I 311.99; 7

***Tp<T001,**p<.01 ,* p<.05
To make it easier to interpret the results from the achievement by ethnicity group

analysis, these coefficients can be used to compute the achievement for a student with a
given set of demographic characteristics. Because the demographic characteristics are
coded as l or 0, the computation involves summing the terms which are coded 1 for a
particular student.40 For example, to determine the predicted achievement for a black
student who would have ones for the intercept and black the resulting equation would be:

Level 1 - Students

A

vi]. : p0 j+1§1 jdblackij (134)

Level 2 - Teachers

IBOj = foo = 222.77

fllj =}’;10 = -25.01

When the slope values, the betas from Equation 8.4, are substituted in the level one
equation (Eq. 8.4), the result for black achievement is produced (Eq. 8.5).

1ft,- = (222.77) + (-25.01)*1 (13.5)

 

40 A table showing ones and zeros for all dummy variables is provided in the
Appendix.

249

..J

= 197.76
Predicted reading achievement for all demographic groups are presented in Table 13.4.
This table shows that the model predicts that White students score higher in reading than
the other ethnic groups. The coefficients for the intercept, Black, and Hispanic are
significantly different from zero. Hispanic and All other students, groups most likely to
contain English as a second language students, are expected to outperform Black students

on NAEP 2000 reading.

Table 13.4 Predicted Reading Achievement Student Demographic Group

 

 

Ethnic Group Achievement
White 222.77
Black 1 97.76
Hispanic 199.71
All other 217.1 1

 

In the next model, gender predicts average reading achievement.

Model 3. Gender

Level 1- Students
Yij = fl0j+ ,6,” dfem,j + eij;

Level 2 - Teachers (13.6)
130)- = 1’00 + ’0j
1341' = 1’10

The regression coefficients for predicting student achievement from gender are found in

Table 13.5.

250

Table 13.5 Regression Coefficients for 2000 NAEP Reading Achievement by Gender

. i L , 7 ,, ,, 7 77 , , Model3
Intercept (v00) 209.38(1.37)***

Female(vo4) f f 10.34(1.62)***
'L‘VLTvanahcetoZ) * * 1155.36 *
LVL 2 Variance (T00) - 479.40***

*** p<.001,** p<.01,* p<.05

These coefficients indicate that male mean achievement in reading is predicted to be
209.38 points compared to 219.72 points for females. The coefficient for female is
significantly different from zero. On average, girls are expected to outscore boys by more
than 10 points.
Model 4. Combined gender and ethnicity
Model 4 tests whether or not there is an interaction between gender and ethnicity in the
NAEP 4th grade reading data. Equation 13.7 computes coefficients for combinations of
gender and ethnicity.
Level 1- Students

Y i]- = IBOj + ,Bljdblackiji- flzjdhispaniclfr fl3jdallotherij+ fl4j dfemI-j

+ ,stegenxblij+ ,B6jegenxhispij+ ,B7jegenxothij + e (13.7)

ij;

Level 2 - Teachers

.30)“: 1’00 T rOj

’31} = 1’10
.32} = 1’20
163) = 1’30
r34j = 1’40

251

 

.35} = 1’50
:56) = 1’60
1571' = 1’70
The regression coefficients for predicting student achievement from ethnic background

and gender are found in Table 13.6.

Table 13.6 Regression Coefficients for 2000 NAEP 4th Grade Reading Achievement by
Gender and Ethnicity

_Model4
interceptwoo) ’ .21—6.29(1.41)‘*** I '
6lackA (V01) 8 i if F-25.39(2.19)***

"7 HispanicTVOZ) ’ '922.82(1.80)3***
All other (v03) -5.08(3.64)
Female (v04) 12.71(1.86)***
Effect Black x female (v05) ‘.74(1.58)

Effect Hispanic x female (V06) -1.05(1.31)

Effect All other x female (v07) 2.09(234)

LVL 1 Variance (02) 111346.42

tyL} VariancufooL p 31092;"
*** p<.001,** p<.01,* p<.05

To make it easier to interpret the results from the achievement by demographic
group analysis, these coefficients can be used to compute the achievement for a student
with a given set of demographic characteristics. Because the demographic characteristics
are coded as 1 or 0, the computation involves summing the terms which are coded l for a
particular student.“ For example, to determine the predicted achievement for a black

female student who would have ones for the intercept, black, female and black female,

 

4] A table showing ones and zeros for all dummy variables is provided in the
Appendix.

252

 

the resulting equation would be:

Level 1 - Students

A

Yij = ,60 j + Eljdblackl-j + 134 jdfeml-j + B5 jeblackxefemij

Level 2 - Teachers (13.8)

flOj 2}];00 =3 216.29
4,- =ro = —2539

When the slope values, the level two betas from Equation 13.8, are substituted in the
level one equation (Eq. 13.8), the result for black female achievement is produced (Eq.
13.9).

vi]. = (216.29) + (-25.39)*1 + (12.71)*1 + (.75)*1 (13.9)

= 204.36

Predicted reading achievement for all subsets of students by demographic groups are
presented in Table 13.7. The table shows that the model predicts female students from all
groups are predicted to have higher average achievement than their male counterparts.

Table 13.7 Predicted Reading Achievement by Student Demographic Group with
Differences by Gender

 

 

Males Females Difference in Achievement
by Gender
White 216.29 229.00 + 12.71
Black 190.90 204.36 + 13.46
Hispanic 193.47 205.13 + 11.66
All other 211.21 226.01 + 14.80

 

253

Score differences between males and females of the same ethnicity are predicted to range
from 11.66 to 13.46 points. Coefficients for gender times ethnicity effects are not
significantly different from zero over the course of one year. Black average achievement
is less than the average reading achievement for other groups for both boys and girls.
Gender, Ethnicity, and SES
Another known source of variation in student achievement is SES as measured by
eligibility for free and reduced lunch. In NAEP, this variable is measured at the student
leveL
Mode15. Gender, Ethnicity, and Grade Level
Level 1- Students
Y ij = IBOj + flljdblackij+ flzjdhispanicij+ ,B3J-dallotherij-l- fl4j dfem,j (13.10)
+ flsjegenxblij+ fl6jegenxhispij+ ,67jegenxothij
+r68jSESij + etj;
Level 2 - Teachers

I60j= 700+ rOj

fllj = 1’10
.sz = 1’20
:63} = 1’30
’34)‘ = 1’40
1551' = 1’50
.56)“ = 1’60
1671' = 1’70
.38j = 1’70

254

The regression coefficients for predicting student achievement from ethnic background,

gender, and SES are found in Table 13.8.

Table 13.8 Regression Coefficients for 2000 NAEP 4th Grade Reading Achievement by

Gender, Ethnicity, and SES

lhtefceht (V001 *

C Black (Y01)

W Hispanic (V02) *
All other (v03)

.2 Ema'éwml . _

Effect Black x female (V05) 7‘ ,

* Model 5

221.35(1.41)*;*
-17.50(1.80)***

-17.47(1.57)*** if ’

1272(355)

13.48(1.79)***

.-99<1-58>

Effect Hispanic x female (v06) -1.43(1.27)
Effect All other x female (v07) 2.17(2.29)

- SF§ (1’08)

LVL1 Variance (02)

LVL 2 Variance (T00) 77

***p<.001,** p<.01,* p<.05

-20.43(1.66)***

238.76"

”#109954?

As demonstrated in prior sections, the computational example to determine the

predicted achievement for a black female student eligible for free and reduced lunch who

would have ones for the intercept, black, female, black female, and free and reduced

lunch, would be:

Level 1 - Students

Yij=

flOj + BljdblaCkU- + B4jdfemy- + stegenxbly- + ngSESU

255

 

Level 2 - Teachers (13.11)

00,. =y‘00 = 22135
011.430: —17.50
fi4j=f40= ”-48
fiszf50= ~99

A

I68j 2' igo = —20.43

When the slope values, the betas from Level 2, Equation 8.11, are substituted in the Level
1 equation, the result for black female achievement is produced (Eq. 8.12).
?ij = (221.35) + (-17.50)*1 + (13.48)*l + (.99)*1 + (-20.43*1) (13.12)
= 197.89
Predicted reading achievement for all subsets of students by demographic group and
grade are presented in Table 13.9. The estimated point loss associated with free and

reduced lunch eligibility is -20.43 points.

Table 13.9 Predicted Reading Achievement by Student Demographic Group and SES

 

 

Males Females Difference in Achievement
by Gender
White 200.92 214.40 + 13.48
Black 183.42 197.89 + 14.47
Hispanic 183.45 195.50 + 12.05
All other 198.20 213.85 + 15.65

 

This table indicates that females score higher than their male counterparts by more than
12 points. Average Black and Hispanic reading achievement controlling for SES is
expected to be similar, around 183 points for males and about 197 points for females.
The same is true for average White and All other achievement, around 200 points for

males and 214 points for females.

256

Summary of Section 1. Developing the level one equation for students. As the

level one base equation for 2000 NAEP 4th grade reading developed, average yearly

achievement was first predicted with a simple unconditional model. This model indicated

that achievement variance was 2.5 times larger within teachers (between students) than

across teachers at level two. Variables known to affect mean achievement (ethnicity,

gender, ethnicity x gender, and SES) were added to the level one equation revealing that

while differences in achievement by gender and ethnicity exist within teachers, the

interaction terms for gender time ethnicity were not significant predictors of student

achievement.

II. Differential Effects of Teacher Characteristics, and Teacher Practices on Student
Achievement (Results at the teacher level)

This section addresses the questions:

1.

2.

Does reading achievement for demographic groups vary across teachers?
How much variation in student reading achievement do variables about
teacher practices, characteristics, and professional development explain?
Is the amount of variance explained by teacher practices, characteristics,
and professional development different for different student demographic

groups?

The analyses in this section first test if average achievement for the various

demographic groups varies across teachers. Nonsignificant error terms for variance (r00,

r10, ...er) indicate that group mean achievement does not vary across teachers so the error

terms predicting mean achievement for those demographic groups are dropped from the

257

model. Although the error terms are dropped, the fixed level two equations for non-
significant achievement by demographic groups are retained in the analysis.

Average achievement did not vary across teachers for any of the demographic
groups in NAEP 2000 4th grade reading. Consequently, there are no differential effects of
teacher practices evident in this data. It is possible, however, to look at the amounts of
variation that teacher practice variables explain for average achievement for all students
across teachers.

Effects of teacher level variables on mean achievement

The unconditional Model 6 establishes the maximum amount of variance that can
be explained by teacher level variables for this data. Note that the interaction terms for
gender times ethnicity have been dropped as they were in preceding sections about the
effects of teacher practices, characteristics, and professional development.

Model 6. Unconditional model for 4”“ grade achievement
Level 1- Students
Y ij = flOj + flljdblackij+ flzjdhispaniclj+ fl3jdallotherij+ 164j dfeml-j +fl5fS‘ESI-j + eij-(l3.10)

Level 2 - Teachers

00].: yoo + raj - (13.11)
1511' = 1’10
.32) = 1’20
1531' = 1’30
164} = 1’70
165)“ = 1’70

where

258

YOOj = intercept, average in student achievement/achievement in teacher j’s
classroom all other predictors held constant
roj = the error term or unique teacher effects

The estimated level two regression coefficients for Model 6 are given in Table

13.10.
Table 13.10 Regression Coefficients for 2000 NAEP 4th Grade Reading Achievement by
Ethnicity
-- _ Model 6
Intercept (v00) 7 V222.45(1.28)***
23'3“ (V10) -, a - , '17-3513-301'"
Hispanic (v20) -17.46(1.85)***
- A1 other (v30) ,, h-2.94(3.58)
*Dfem (v40) f 11.25(1.54)***
7 SES (V50) -20.40(1.67)*"’"
LVLA1_Variance(62) - ' i ' '1 100.32
LVL_2 Intercept('r00)~ 238.75***

“p<.001,** p<.01,* p<.05

Selected coefficients from this analysis serve as the base for the variance
decompositions found in Tables 13.12-13.15. Next district SES is added to the model to
determine its effect on the average achievement for Black and White students across
teachers. The variable for student free and reduced lunch has been aggregated to the
teacher level as control for classroom SES.
Model 7. 4th grade average achievement and SES
Level 1- Students - Same as unconditional Equation 13.10

Level 2 - Teachers

flo- = 1’00 + r01 avSES + r0,- (13.12)

259

1511' = 1’10

1321' = 1’20
:63} = 1’30
1581' = 1’70
169' = 1’70
where
y 001- = intercept, average student achievement across teachers holding all other

predictors constant
you = change in average achievement associated with a one standard deviation
increase in mean free and reduced lunch all other predictors held constant
rOJ- = the error term or unique teacher effects
The estimated level two regression coefficients and variance components for
Model 7 are given in Table 13.11.

Table 13.11 Regression Coefficients Mean Achievement Predicted by Average District

 

 

 

 

SES
_ _ _ _ _ _ "—Modeli
lntErc—eptfvao) * f? -2- 2E44<146)*** '
avSES (v01) -18.25(3.29)***
'fiaGckam) if EI4.85(2.¢TO)***
Hispanic (1120) -15.96(1.88)***
tangerine; g :76 7
Dfem (1180) 1 1 .08(1.53)***
SES (ygo) -15.21(1.95)***
_L‘vfiwviafiiaoz)“ 11693.81 1

FY}? Variance (100)” 225.51***
*** p<.001“ p<.01,* p<.05

How much variation does average SES explain in the average reading

260

achievement across teachers? To answer this question the following equation computes
the variance in mean achievement explained by average SES. The estimated 1:00 values
from Tables 13.10 and 13.11 would be substituted in the following equation:

1'00 (model 6) — 100 (model 7) _ 238.76 — 225.51
r00 (model 6) 238,75

 

 

= .055 = 5.5% (13.13)

This means that 5.5% of the variance in predicted average achievement is explained by
average SES across teachers (classrooms).

Table 13.12 Variance Decomposition for Mean Achievement by SES

 

 

 

 

 

 

 

 

 

SES
Variables Uncond Free/Red Lunch
llntercept (v00) 222.45(1.28)*** 218.44(1.46)***
avSES(voz) -18.25(3.29)***
Intercept Variance (T00) 238.75*** 225. 51 ***
Level 1 variance (02) 1100.32 1093.81
% variance reduction intercept 5.5%

 

A one standard deviation increase in average free and reduced lunch across
teachers reduces average achievement across teachers by -18.25 points.
Teacher Effects on Average Achievement

Now that the context is set for the analyses at Level Two, it is possible to track the
variance explained by each teacher level variable controlling for SES and to determine its
effect on average reading achievement across teachers. One variable at a time is added to
the level two equation, EQ 13.14. The term “variable x” is exchanged for a single
Reading Practice (reform activities, traditional activities, and computer activities),
Teacher Characteristic (reading preparation, preparation for reading instruction,

technology preparation, assessment preparation, and diversity preparation), or

261

Professional Development (professional development frequency in the last year).
Model 8. Template for Level 2 equations
Level 1 remains constant (Eq. 13.10)

Level 2 - Teachers

flOj = 700 + yo, avSES + 1’02” variablex + r0j (13.14)
fllj = 1’10
:52; = 1’20
:53)‘ = 1’30
58)“ = 1’70
:69) = 1’70

where
1’02.) = change in average in average student achievement associated with a one
unit change in “variable x” holding SES constant at the classroom
(teacher) average
The analysis process followed the same pattern for all variables as that presented
in prior sections. One teacher practice variable, computer activities, and one teacher
characteristic, preparation for reading instruction, explained variation across teachers in

excess of that explained by SES. Results are presented in Table 13.13.

262

Table 13.13 Teacher Effects on Mean Achievement Controlling SES

 

 

 

 

 

 

 

 

 

avSES Practice 6| Characteristic 1

Variables Uncond Free/Red Lu Computer iReading Inst |
activities I l

Ilntercept(v00) 222.45(1.28)*** 218.44(1.46)*** 218.39(1.46)***i 218.49(1.45)***i
(v01) -.38(.28)| .74(27)“I
avSES(voz) -18.25(3.29)*** -18.03(3.29)***| -18.01(3.29)***|
Intercept Var(‘roo) 238.75*** 225.51*** 224.54***i 224.07***i
(02) 1100.32 1093.81 1093.901 1092.271

% variance reduction intercept 5.5% 5.9%; 6.1%;

 

For every one standard deviation increase in computer activities, average
achievement is expected to decrease by -.38 points. Computer activities explain 5.9% of
the variance in average achievement controlling for SES. Preparation for reading
instruction explains 6.1% of the variation average achievement across teachers. A one
standard deviation increase in preparation for reading instruction increase average

achievement by .74 points.

263

14. COMPARISON OF HLM RESULTS
FOR MATHEMATICS AND READING ACROSS DATA SETS

The HLM analyses addressed questions about the relationships among student

achievement and

0 Teacher characteristics and practices
0 Characteristics of professional development
0 Characteristics of teacher preparation

The clearest way to summarize the HLM results across data sets is to use a series of
tables. Table 14.1 summarizes the demographic distribution of students by gender,
ethnicity, SES, student and teacher 11, as well as the variation present at both student and
teacher levels for all data sets.

Table 14.1 Comparison of Variation Across Data Sets

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Mathematics Reading
Demographics CT NAEP NAEP CT NAEP NAEP
_ _ _ _ 1996 ___ 2000 _ _ 199: __ 2000#fi
White 68% 59% 54% 67% 55% 54%
Black 14% 18% 18% 16% 19% 19%
Hispanic 9% 16% 21% 9% 21% 22%
Other 9% 7% 7% 8% 5% 6%
Male 48% 49% 50% 47% 50% 51%
Female 52% 51% 50% 53% 50% 49%
Free/Red Lunch 20% 36% 43% 22% 43% 42%
Student n 1550 4409 6205 1776 4303 4548
Teacher 11 81 584 1245 90 828 918
Variation at Lvl 1 91 % 60% 65% 90% 70% 70%
Variation at Lvl 2 9% 40% 35% 10% 30% 30%

 

264

 

Connecticut has fewer minorities and fewer free and reduced lunch eligible students
than the national sample. A large proportion of the variation in student achievement in
Connecticut is between students within classrooms - about 90%. This percentage captures
variation specific to each student in a particular classroom such as individual variation
due to parental education, home environment, peer influence, etc. At level two, about
10% of the variation in student scores is attributable to teachers. It is this variation that
can be impacted systematically by state policy, teacher education, and professional
development.

In NAEP, there are more minority students than in the Connecticut data. It is
interesting to note that by data collection in 2000 the national Hispanic student population
had grown larger than the Black population, which remained relatively stable across
administrations. Less variation is located at the student level and more initial variation
across teachers, over 20% more, than found in Connecticut.

SES, as measured by free and reduced lunch, plays a large role in student achievement
by demographic group. Although SES was not a focus of this study, it is important to
acknowledge its relationship to student achievement and control for it. NAEP samples
have higher proportions of free and reduced lunch students than the Connecticut samples.

Student gain scores in mathematics and reading in Connecticut show a different
picture of achievement than do achievement status scores in NAEP. In Connecticut,
minority and female students generally gain more over the course of the year than do
White and male students. As in NAEP, White and male students tend to score higher
overall than minority and female students. This is to say that minorities and females have

not yet achieved the same level of academic attainment as their White and male

265

 

counterparts although they gain more over the course of a single year.

To tackle questions about the relationships between student achievement and teacher

practices, characteristics, and professional development, it is first necessary to determine

which group means vary across teachers. If group means do not vary across teachers, it is

only possible to talk about the relationships of average achievement to teacher practices,

characteristics, and professional development and the variation attributable to teachers for

all students. If group means vary by gender and/or ethnicity across teachers, then

differential relationships are present depending on group membership. Table 14.2

indicates the group means which varied significantly across teachers (marked with an x).

Table 14.2 Group Means Which Varied Across Teachers (x) for All Data Sets

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Mathematics Reading
Demographics CT NAEP NAEP CT NAEP NAEP
_________ __ -1996- _2002 _ _ 42912399
White (Intercept) x x x x x x
Black x x x x
Hispanic X
Other
Male (Intercept) x x
Female x x
Black female42 x
Student n 1550 4409 6205 1776 4303 4548
Teacher n 81 584 1245 90 828 91 8

 

 

 

 

Analyses using average gain scores by gender and ethnicity in Connecticut were more

sensitive to variation across teachers than achievement status in NAEP. Teacher

 

42 It’s difficult to interpret three way interactions among black x female x teacher

variables.

266

mathematics variables were more likely to be associated with mathematics achievement
than teacher reading variables were likely to be associated with reading achievement
across data sets. Group means for at least one group varied across teachers in all data sets
except NAEP 2000 Reading.

Black mean achievement varied across teachers in four of the six data sets, two for
mathematics (CT mathematics, NAEP 1996) and two for reading (CT reading, NAEP
1998). Group means for Black (CT Reading) varied across teachers but teacher variables
did not account for any of the variation. Female mean mathematics achievement varied
across teachers in two data sets. Hispanic mean mathematics (NAEP 2000) and Black
female mean mathematics achievement (NAEP 1996) varied across teachers in one data
set each.

Table 14.3 summarizes the variance explained by teacher variables on mean
achievement by ethnicity and Table 14.4 presents the variance explained by teacher
variables on Male/Female achievement. Only one teacher variable at a time controlling
for SES was entered into an analysis.

Ethnicity

Table 14.3 Variance in Mean Achievement by Ethnicity Explained by Teacher Variables
Across All Data Sets

 

 

 

 

 

 

 

 

 

 

 

 

Mathematlcs Reading
Demographics CT NAEP NAEP CT NAEP NAEP
7 i 7 . 1996 2000 7 - 1998 A? 51004
Intercept White White White All“ ? White All
SES 2.2% <1% 1.3% 0% 1.6% 5.5%
Practice-Sensemaking 0%" 2.2% 1.3%* x x x
GPA x 4.4% 2.5% x x x

 

267

 

Table 14.3 (cont'd).

 

Reform activities

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

x x x x <1%* x
Traditional activities x x x x 0%* x
Computer activities x x x 8.3% 0%* 5.9%
Characteristics-Math prep 25.6% 1.3% x 7.3% X x
Math knowledge x 0% x x x x
Reform orientation 0%" 4.2% x x x x
Read Prep x x x x 1.8% 6.1%
Assess Prep x x x x 1.8% x
Professional Dev- PD Impact 9.9% x x x x x
Ment Frq 17.5% x x x x x
PD frq 44.4% x x 4.3% <1%* x
Group Black Black Hispanic All Black All
SES 45.9% 15.1% 5.9% x 6.4% X
Practice-Sensemaking 47.5% 10.1%* 4.5%" x x x
GPA x 6.2%* 2.5%* x x xi"
Reform activities 7 x i 7 x , Xi 7x77 9;8%7 7 I 7
Traditional activities x x x x 53.6% x
Computer activities fix x ’L x 49.0% 7 x ,
Characteristics-Math prep 57.9% 17.7% x x x 7x
Math knowledge x 17.8% x x x x
Reform orientation 55.5% 6.9%” 7 x i x l ,, x ,, W x
Read Prep 7 x x x x , 7.5% 5.
Assess Prep "x x x x 8.0% x
Professional Dev- PD Impact 43%” x x x x x
Ment Frq 43.7%" x x x xi , x~
PD frq 52.5% x x x 15.4% x
* Less than SES alone. Likely due to mult:collinearity.

 

 

 

 

 

** Black mean achievement varied but selected teacher variables did not explain variation. The
variance decomposition is presented for the intercept only (all students).

Different proportions of variance in average White/Black achievement are explained

across teachers in four data sets. Since teacher variables did not explain variation in Black

268

 

achievement across teachers in Connecticut Reading, the results are not included here
Results for the HLM analyses across data sets are summarized below.

Controlling for SES. SES explains a large proportion of the variation in average
Black mathematics achievement (15%-45%) across teachers in CT and NAEP 1996 but
only a small proportion of the variance in average White achievement (1%-2%). Low
SES has a large, negative effect on average Black achievement in Connecticut (-19 points
in average SES districts). NAEP 1996 shows a loss of about -5 points for both groups in
average SES classrooms.

While SES plays a part in NAEP 2000 White/Hispanic mathematics achievement and
NAEP1998 Black/White reading achievement, it only explains 1-2% of the variance in
White scores and approximately 6% of the variance in Black and Hispanic scores across
teachers. Average White scores decrease as a function of increased free and reduced
lunch but Hispanic students increase average mathematic achievement by +3.5 points in
NAEP 2000. When the effect of teachers on Hispanic achievement is compared to the
effect on White achievement, it is important to realize that the intercept also includes all
non-Hispanic groups. No teacher level variables explain more variation in average ,
Hispanic mathematics achievement across teachers than SES alone.

Relationships between teacher practice and Black/White achievement. Positive
effects of teacher classroom practices on average White/Black student achievement
controlling for SES are evident when teachers use math sensemaking activities in CT.
Teacher use of math sensemaking activities explains none of the variance in White
average achievement across teachers in CT but it explains 47.5% of the variation in Black

average achievement. For every one standard deviation increase in sensemaking

269

activities, Black achievement increases by +1.25 points. White achievement increases by
+.18 points. In NAEP 1996, less variance is explained across teachers but average White
achievement increases by +.72 points while Black average achievement decreases by -.11
points after controlling for SES.

Teacher practice variables explain small amounts of variance in average White
reading achievement across teachers in NAEP 1998. Average Black reading achievement
varies from 10%-50% across teachers controlling for SES depending on the specific
teacher practice. A one standard deviation increase in reform activities leaves average
achievement for White students virtually unchanged but increases Black achievement by
+.4 points. A one standard deviation increase in traditional activities increases average
White achievement by about half a point while simultaneously decreasing Black average
achievement by half a point. No similar effects were detected in CT reading or NAEP
Reading 2000. A one standard deviation increase in computer activities increases average
White reading achievement by over one point while leaving Black average achievement
unchanged.

Relationships between teacher characteristics and Black/White achievement.
Teacher preparation in mathematics, i.e. major/credentials, explains 25.6% of the
variation in average White gains and 57.9% of the variation in average Black gains
controlling for SES in CT. A one standard deviation increase in teacher mathematics
preparation signals a decrease in average White gains of -1 point per year and a decrease
in average Black gains of -2 points per year. A slightly different phenomenon occurs in
NAEP mathematics 1996. Teacher preparation in mathematics explains 1.3% of the

variation in White average achievement and 17.7% of the variation in average Black

270

achievement controlling for SES. For every one standard deviation increase in teacher
preparation, White scores increase by +1.5 points while Black average achievement
decreases by -1 .4 points. The counterintuitive relationship between mathematics
achievement and teacher preparation has been noted by Valli and Reckase (2000) and
Rowan, Correnti, and Miller (2004). Possible explanations include that elementary
mathematics teachers often major and get licenses in subjects other than mathematics.

Reform orientation in mathematics explains less than 4% of the variation in White
achievement across teachers controlling for SES while it explains 55% of the variation in
average Black achievement in CT and 7% of the variation in NAEP mathematics 1996.
A one standard deviation increase in teacher reform orientation raises Black gains in CT
by +1.33 points and raises achievement in NAEP 1998 by +.28 points. White
achievement stays nearly level in CT and increases by +1.3 points in NAEP 1996.

Teacher reading preparation and assessment preparation explain approximately 8% of
the variation in Black average reading achievement in NAEP 1998 and less than 2% for
White average achievement. Increases in reading preparation and assessment preparation
increase Black average achievement by nearly +.3 points while decreasing White average
achievement by similar amounts.

Relationship between teacher professional development and Black/White
achievement. Teacher professional development variables do not explain variation in
Black/White average mathematics achievement at the national level but they do explain
variance in CT. The variable that is most consistent across data sets and that has the
largest effect on Black/White achievement is professional development frequency. It

explains 44.4% of the variance in average White gains and 52.5% of the variance in

271

average Black gains across teachers once SES is controlled. A one standard deviation
increase in professional development frequency has a differential effect on students.
Average White achievement decreases by -1 point while average Black mathematics
achievement increases by nearly +1 point.

Teacher professional development frequency explains 15.4% of the variance in
average Black reading achievement across teachers in NAEP 1998. A one standard
deviation increase in professional development frequency increases Black average
achievement by +3.38 points while it decreases White achievement by -.68 points.
Gender

The variance explained results for Connecticut and NAEP 1996 mathematics for male
and female are presented in Table 14.4. Teacher effects did not vary by gender for
reading achievement or NAEP Mathematics 2000. Average achievement for Black
female in NAEP 1996 mathematics varies across teachers but the three way interactions
among black, female, and teacher variables are difficult to interpret since the main groups
for Black and female also vary across teachers. Results for Black female are not

discussed again here.

272

Table 14.4 Variance in Mean Achievement by Gender Explained by Teacher Variables

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Mathematics
”Demographics, 7 i 7 CT , NAEP 1996
Intercept Male Female Male Female
SES 6.1% 23.2% <1% 0%
Practice-Sensemaking 8.5% 29.3% 3.7% <1%
GPA 6.6% g 37.57% _ 5._4% % 3%: W
GPA assessment x A X; 3.5% :1 %1 _
DLA 5.7%* 33.8% x x“ i
Characteristics-Math prep 54%" 116%: x w x -__
Math knowledge 2.8%* 32%? x .J‘W.
Reform orientation 6.4% 23.6% 5.9% ; 2.1% #
Assess Prep 6.6% 30.1% x 77x
Efficacy 7.9% 41.1% x * x __
Professional Dev- PD Impact 50%" g 15.5%? x x»
Ment Frq 1.8%* a 27.8% x x__
PD frq 17.6% 15.5% x (X

 

 

* Less than SES alone. Likely due to multicollinearity.

Controlling for SES. SES explains 23.2% of the variation in average female
mathematics achievement across teachers in CT but none of the variation in NAEP

mathematics. For every one standard deviation increase in district fi'ee and reduced lunch

 

in CT, boys lose more than -6 points. Girls gain them back (+8 points).

Teacher variables explain variance in average mathematics achievement for males and
females across teachers in both data sets. Variance explained in female average
achievement is very large when compared to the variance explained in male average
achievement across teachers in CT. In NAEP 1996, more variance is explained in the

achievement of males than females.

273

Relationships between teacher practice and male/female achievement. One
teacher practice variable, math sensemaking, explains variance in male/female
achievement across data sets. A one standard deviation increase in sensemaking activities
increases male average achievement across teachers by .70 points in CT and .96 points in
NAEP 1996. Sensemaking decreases female achievement by -.54 in CT and -.98 in
NAEP 1996. Reform orientation has a similar pattern. A one standard deviation increase
in teacher reform orientation increase male achievement by +.58 points in CT and +1.65
points in NAEP 1996. Female achievement decreases by -.70 and -.65 respectively.

The one teacher practice that appears to increase female mathematics gains
controlling for SES in CT is drill and lecture activities. It explains 33.8% of the variance
in female achievement but only 5.7% of the variance in male gains across teachers. A
one standard deviation increase in drill and lecture activities increases female
achievement by +.6O points while male gains decrease by -.30 points. This effect was
not present in the national sample.

Relationships between teacher characteristics and male/female achievement. One
teacher characteristic, reform orientation, explains variances in male/female achievement
across both data sets. In CT, reform orientation explains 6.4% of the variance in male
achievement and 23.6% of the variation in female achievement controlling for SES. A
one standard deviation increase in teacher reform orientation increases male scores by .58
points and decreases female achievement by -.70 points. In NAEP 1996, a one standard
deviation increase in reform orientation, increases male mean achievement by 1.65 points

and decreases female achievement by -.65 points.

274

Relationship between teacher professional development and male/female
achievement. While professional development frequency does not explain variance in
NAEP 1996 mathematics achievement for male/female, it does explain male/female
achievement variance across teachers in CT. For average male achievement, it accounts
for 17.6% of the variance and for 15.5% of the variance in female achievement. A one
standard deviation increase in professional development frequency decreases male gains
by -.12 points while female gains remain level.

Teacher practices, characteristics, and professional development do not have the
same effects on achievement for all students across subject matter and data sets.
Generally, teacher variables explain more variance in female and Black average
achievement across teachers than for male and White achievement. Relationships among
achievement and teacher variables are stronger for mathematics than for reading. Teacher
variables for mathematics sensemaking, reform orientation, and professional development
frequency (for mathematics and reading) have the strongest relationships with student
achievement across multiple data sets but the results showing who ultimately benefits are
mixed. In general, gain scores offer more insight into which students are benefitting from
teacher practice, characteristics, and professional development.

Relationship between teacher preparation and student achievement. Connecticut
student data could be nested within teachers and teachers nested within teacher
preparation programs. Average student gains produced by teacher graduates of specific
programs did not vary by teacher preparation program in CT for either reading or
mathematics.

Two level analyses were possible with teacher practices and characteristics as the

275

outcomes. 38% of the variance in teacher mathematics knowledge was explained by
teacher preparation program and 15% of the variation in teacher English preparation was
explained by university teacher preparation. In the two level student/teacher analyses for
CT, a one standard deviation in mathematics knowledge was related to an increase in
female gains of .89 points and +.41 point increase in Black/White gains. Male scores
remained level. This implied relationship should be interpreted with caution since there
was not enough variation in teacher preparation institutions to support a three-level model

using student gains over one year as the outcome.

276

15. DISCUSSION AND IMPLICATIONS

Using results from the analysis of six state and national data sets, this study
provides evidence on the strength of the relationships and relative contributions of four
important elements in the education system to student achievement in mathematics and
reading: teacher preparation, teacher characteristics, professional development, and
teacher practice. Results indicate that teacher practices, teacher characteristics, and
professional development across teachers generally have moderate to large direct effects
on teacher practice and small but significant indirect effects on student achievement in
mathematics and reading. Sometimes those effects differentially impact student
achievement by ethnic and gender groups. Findings vary based on curriculum and policy
context. While there are many interesting results related to specific data sets throughout
the analyses, this discussion will concentrate on the larger picture by identifying common
threads across data sets and analyses.

The discussion begins with a review of the conceptual model which also addresses
design and methodological strengths and weaknesses, Next, the discussion moves to the
relative importance and contributions of teacher preparation, teacher characteristics, and
professional development to teacher practice and student achievement. The discussion of
differential effects on student achievement by gender and ethnicity follows. Implications
for the present and future conclude.

The Conceptual Model.
For the purposes of framing the discussion about the strength of relationships and

relative contributions of variables to student achievement, a quick review of the

277

conceptual model is in order.

Figure 14.1. Conceptual Model (repeated)

m
Teacher
Preparation

Government ~. . >

 
  
  

 

   
 

          

Teacher
Practices

Teacher
Characteristics

A Professional
Development

The model graphically illustrates study research questions concerning the

Student Gains!
Achievement

 

 

Policy

 

 

relationships among variables. In the model, state and/or federal government education
policy is directed at teacher preparation, teacher characteristics, and professional
development which, in turn, are expected to influence teacher practices and subsequently
increase student achievement. Prior research results supported the connections but no
single study looked at all the connections simultaneously. This study used SEM to look
at the relationships among variables simultaneously and to estimate standardized direct
and indirect effects. HLM analyzed the effects of individual teacher variables on student
achievement as well as differential teacher effects on achievement by gender and
ethnicity. Additionally, the study presented effects of teacher preparation programs on
teacher characteristics.

Methodological choices which may have affected the interpretations for both
SEM and HLM analyses include large variations in sample size and variations in the
specific variables available across data sets. Sample size varied from 168 teachers and

1550 students in Connecticut to over 1000 teachers and 6000 students in NAEP. Larger

278

 

sample sizes have more stability and accuracy than smaller ones. To improve
interpretation across differing sample sizes, analysis proceeded with the smallest data set
serving as the base analysis with results confirmed in successively larger data sets.
Variation in available variables across data sets was unavoidable. Reasonable care was
taken to ensure similarities of scales and variable contributions to constructs.

SEM Analysis. The conceptual model served well as the structural model for
SEM analysis across all data sets. The unit of analysis for SEM was teachers with average
classroom achievement standardized by grade level as the outcome. Each latent variable
(other than government policy) had a series of descriptor variables that formed separate
measurement models. The measurement models successfully defined the latent variables
for teacher preparation, teacher characteristics, professional development, and teacher
practice. When the measurement models were combined with the structural model, the
final hybrid model had good fit (NF I>.984, p<.001, RMSEA=.05 to .116) and worked in
similar ways across data sets. Variations that occurred appeared related to subject matter
and government policy context.

Methodological choices which may have affected the interpretation of SEM

results include the use of correlation matrices as input for the SEM and potential naming

 

fallacy. Although covariance matrices are the input of choice for SEM, the use of
correlation matrices was not possible since student data from Tennessee was only
available as part of a correlation matrix. In order to treat all data sets in similar fashion,
weighted correlation matrices were constructed for NAEP data to ensure appropriate
standard errors. There is always a chance that the name chosen for a construct is not

appropriate (otherwise called “naming fallacy”). Correlations suggest that teacher

279

preparation and teacher characteristics may not be distinctly different constructs and
consequently one or both may be misnamed.

HLM Analysis. HLM analysis was based on the conceptual model although
nesting requirements necessitated slight modifications. The model took the following
form:

Figure 14.2. Three Level HLM model

 

,,/’ ‘\\\ ,,//‘—‘\\\ ///——\
/ // \\ \\ /I// \\\ // \x
r ' ’l '
Fri-agggfion . Teachers H Students ,)
v/I/ \\\\_,,/// V

Students were nested within teachers who were nested within teacher preparation
programs. The three level models did not have enough variation present at level three for
meaningful analysis across teacher preparation programs. Therefore, two level models
with students nested within teachers or teachers nested within teacher preparation
programs were employed to look at the effects of teachers on students and the effects of
teacher preparation programs on teachers. In the first case, students were the unit of
analysis. In the second case, teachers were the unit of analysis. There was evidence of
differential effects of teacher characteristics, professional development, and practices on
achievement by gender and ethnic group. It was also possible to rank teacher preparation
programs based on the levels of certain characteristics found in beginning teachers who
attended that program. While interesting, results were not significantly different from
zero.

Methodological issues which may affect the interpretation of HLM results include

280

 

 

a non-random sample of teachers, variations in the measure of SES included across data
sets and the use of gains for state analyses versus achievement status in NAEP. Students
are randomly sampled in NAEP and teacher variables are included as a part of the student
record. It is not a random sample of teachers. District SES was added to the Connecticut
data as part of the teacher data since no measure of SES was available in BTPS data.

SES is measured at the student level in NAEP and aggregated to the teacher level.

Gains Versus Achievement Status. In general, gains are more informative of
progress than achievement status. They more accurately portray the variance associated
with teachers by reducing variation attributable to factors outside of the classroom. When
achievement status is the outcome, white students have higher overall scores than other
demographic groups and boys have higher average scores than girls. In the HLM
analyses, gain scores pointed to a narrowing of the achievement gap over the course of
the year with minorities and girls gaining the most points. It is very important not to
confuse narrowing the achievement gap with attaining equal levels of achievement.
When either pre- or post-test scores in Connecticut were analyzed separately, results
revealed the same patterns as achievement status -- White and male students had the

highest overall scores.

 

State Versus National Data Sets. State data used gain scores as the student
outcome while the national data used achievement status. In general, state samples where
students varied in similar ways based on the education policy environment were more
informative than the national samples which tended to average out variance found at the
state level. In other words, a random sample of students from all states averages out

indicators that might be important at the state level, masking important trends. High

281

achievement in one state is balanced by low achievement in another. Specifically, a
highly centralized state might have strong positive connections from policy to student
achievement while a decentralized state might have strong negative connections. At the
national level this would show up as no connection.

Relationships Across Models

What do the SEM analyses say about the structure of the education system?
When looking at the absolute size of effects, there are strong similarities in structural
equation modeling results across data sets. Analyses based on the conceptual model
indicate relatively stable direct and indirect effect sizes for teacher preparation, teacher
characteristics, professional development, and teacher practice across policy contexts.
Most surprising is the persistent, small indirect effect (5% |.05|) for teacher preparation,
teacher characteristics, and professional development through teacher practice to student
outcomes regardless of whether the outcome is gains or achievement status in
mathematics or reading.

While some might argue that an effect of this size is too small to be worth
pursuing, a small effect can be very important (Camilli et al., 2003; Reckase, 2002; Valli
et al., 2003; Wayne & Youngs, 2003). Sanders et a1. (1999) have shown that teacher
effects are additive across years and can be of great importance to future student
achievement. If policy makers plan to evaluate the education system based on student
achievement outcomes, knowing the size of the effect that can be reasonably traced back
to teacher preparation, teacher characteristics, and professional development is key.

Teacher preparation, teacher characteristics, and professional development

produce measurable direct effects on teacher practice. Generally, teacher preparation has

282

 

medium direct effects on teacher practice (5 2 L301)“. On average, teacher characteristics
have large direct effects on teacher practice (5>|.40|). The direct effects of professional
development on teacher practice are mostly small (5z|.10| to|.20|). Teacher practice has
small direct effects on student achievement although the effects are larger than the other
constructs (53|.01|to|.30]).

In other words, teacher characteristics have the largest effects on teacher practice
followed by teacher preparation and then professional development. The size of indirect
effects on student achievement are in the range of 5'»: |.05| for all three. The direct effects
of teacher practice on student achievement, as might be expected, are larger than any of
the indirect effects of teacher preparation program, teacher characteristics, or professional
development. Predictably, 68-85% of the variance in teacher mathematics practice is yet
to be explained by the SEM models. Generally, 50-75% of the variation in teacher
reading practices is left to be explained.44 The unexplained variance represents
unmeasured variables that may be related to a teachers’ personal backgrounds, individual
styles, peer influences, school climate, and other factors not included in the models but
which appear to have great influence on the practices teachers use.

Contexts
This section discusses the potential roles that policy and subject matter context
play in differences found across data sets. While effect size is important to understanding

the strength of relationships among variables, the sign of the effect indicates differences

 

43 Data is only available for Connecticut and Tennessee.

44Atypically, NAEP 1998 model only has 18% of the variation in reading
practices left to be explained.

283

 

 

 

by subject matter and policy context.

Government Policy. Since there were no empirical measures of government
policy present in the data, this study could only review policies at the national and state
levels that were believed to impact teacher preparation, teacher characteristics, and
professional development. Prior to 1998, most federal policy did not intervene in the
rights of states to regulate education except in the area of equality, a fundamental and
federally protected right for all citizens. Since 1998, two major pieces of legislation, the
Amendments to the Elementary and Secondary School Act (ESEA, Title I) of 1965 and
HEA (Higher Education Act, Title II) have had far-reaching effects into state regulation
of education. These acts include No Child Left Behind (N CLB) and the Title II State
Report Card on Teacher Preparation which tie student achievement and teacher
preparation to federal funding. Neither was in force during the data collection years for
this study. Connecticut BTPS data was augmented with results from the Initial State
Report Card on Teacher Preparation (1998) but there were no detectable effects of IHE
teacher preparation program certification pass rates on teacher practices or student
achievement.

A review of education policies from Connecticut and Tennessee indicated that the
states took different approaches to regulating education. Two ways in which the state
policy environments varied during 1999-2000 were in the choice of method for policy
implementation (top-down centralized-CT vs. bottom-up decentralized-TN) and testing
purposes (low-stakes-CT vs. high-stakes-TN). Given the similarities of analysis results by
state and the lack of within-state policy variation by data set, any conclusions about the

influence of policy would be circumstantial at best.

284

 

 

 

Subject matter context. Positive values for teacher characteristics and
professional development translate into positive values for teacher practice and student
achievement across all data sets. In the two data sets that have a measure of teacher
preparation program (Connecticut and Tennessee), teacher preparation program effects
are negative on both teacher mathematics practice and student mathematics achievement.
Valli et a1. (2003) suggest that this negative relationship may be related to the level of
subject matter preparation required for teaching elementary mathematics. There were
also indications that teacher preparation program as measured in this data was not a
distinctly different construct from teacher characteristics.

With the exception of Connecticut, effects for reading are mixed across data sets.
Positive effects at one level of the system sometimes translate into negative effects at the
next level and vice versa. This suggests that education system is not aligned to support
reading achievement in the same way it is to support mathematics achievement. It also
suggests that Connecticut is doing something right when it comes to reading achievement.
Lessons from Connecticut could be valuable as other states move toward comprehensive
reading policy in order to reach NCLB reading achievement requirements.

Differential Effects

The preceding discussion describes a model of education that is connected in
reasonable ways. The system is linked to mathematics achievement and less well linked
to reading achievement. Teacher preparation programs, teacher characteristics, and
professional development have strong effects on teacher practices that in turn have small
but significant effects on average student achievement. What does this mean for various

subgroups of students who come from a variety of ethnic and gender groups?

285

 

 

 

A large portion of this study was devoted to the potential differential impact of
teacher characteristics and practices on student achievement in mathematics and reading.
Several earlier studies indicated that subjects such as reading are culturally embedded and

may be more effectively addressed with explicit rather than implicit instructional

 

strategies (Delpit, 1988; Hirsch, 1996; Heath, 1983). Mathematics also embodies both
computational and conceptual components which can be approached in both explicit and
implicit ways (N CTM 2000, Ma 1999). Does a particular approach disadvantage some
students while advantaging others? HLM was the appropriate method for addressing
questions regarding the differential effects of teacher variables on student achievement by
gender and ethnicity. Tennessee was excluded from the analyses since it did not have
individual student scores nested within teachers.

Effects of teachers on student achievement. Achievement by gender and
ethnicity varied significantly across teachers in some but not all data sets. The following
is a list of groups whose achievement varied and the data sets in which they are found.

1. Black mathematics and reading achievement varied across teachers in four data

sets: Connecticut and NAEP 1996 mathematics, Connecticut and NAEP

1998 reading.

 

2. Hispanic mathematics achievement varied across teachers in NAEP 2000.

3. Female mathematics achievement varied across teachers in two data sets:
Connecticut and NAEP 1996.

4. Black female mathematics achievement varied across teachers in NAEP1996.

For the most part, teacher effects which explained variance in average

achievement by demographic groups were not consistent across data sets. Results for

286

 

average Black achievement across teachers are discussed below followed by results for
average female achievement. Although average Hispanic achievement and average Black
female achievement varied across teachers in certain data sets, teacher level variables
explained little variance in their average achievement so they are not discussed further.

Teacher effects on average Black mathematics and reading achievement.
Teacher mathematics preparation, an indicator of teacher quality, explains variance in
achievement for both Black and White students across teachers in Connecticut and NAEP
1996. Controlling for SES, variance explained for average Black achievement (57.9%) is
more than twice that for White (25.6%). This suggests that while teacher mathematics
preparation varies considerably for all students, Black students are more than twice as
likely to have mathematics teachers with less disciplinary course work.

Teachers’ frequency of sensemaking activities, which are facilitated by subject
matter preparation, account for nearly 48% of the variance in Black average gains and
none of the variance in White gains across teachers in Connecticut. If teachers increased
the frequency of sensemaking activities by 1 SD, Black average gains would increase by
an +1.25 points over the year. White students would also increase their scores slightly by
about +.18. In NAEP 1996, sensemaking activities accounted for 10% of the variation in
Black average achievement and only 2% of the variation in White achievement. Every
one SD increase of sensemaking activities explained a -.11 difference in overall Black
achievement status compared to White achievement status. In other words, the frequency
of sensemaking activities was less for Black students than for white students and resulted
in a -.11 drop in average achievement across teachers.

By NAEP 2000, Black mathematics achievement did not vary across teachers at

287

 

 

 

 

the national level. It is worth noting that the Connecticut data was also collected in 2000.
The lack in variance in NAEP is likely due to the balancing out state level effects in the
national sample.

Other differential teacher effects that explain variation in Black average
mathematics achievement across teachers in Connecticut include reform orientation and
professional development frequency but these effects are not present in multiple data sets.
These results are discussed in the HLM section but not repeated here.

Average Black reading achievement varied in two data sets but only the teacher
variables in NAEP 1998 explained variance. NAEP variables related to teacher quality -
teacher reading preparation and assessment preparation - explained 7-8% of the variance
in Black achievement across teachers and none of the variance in White achievement
controlling for SES. All three teacher reading practices explained variance in Black
achievement (IO-50%) across teachers. None of the practice variables explained variance
in White achievement. As in mathematics, the effect disappeared at the national level in
NAEP 2000. The pattern of differential teacher effects on Black reading and mathematics
achievement across subject matter and policy contexts is worth further exploration.

Teacher effects on average female mathematics achievement. Differential
effects of teachers on average female mathematics achievement were also present in
Connecticut and NAEP 1996. Differential effects attributable to teachers were very large
in Connecticut and very small in NAEP 2000. In general, teacher variables explained
three to five times more variation in girls’ average mathematics gains compared to boys.
District SES also explained four times more variance in average gains for girls across

teachers. Interpretation, however, is key. Boys are disproportionately disadvantaged by

288

 

 

 

lower SES than girls in this sample. For every one standard deviation increase in free and
reduced lunch, boys lose more than -6 points in mathematics gains over the year while
girls gain +8 points. For each standard deviation increase in teacher variables, boys tend
to gain less than a point while girls tend to lose less than a point.

In NAEP 1996, a similar pattern emerges where a one standard deviation increase
in free and reduced lunch (SES) signals an achievement status drop for boys of -6 to -7
points while girls lose less than one point. As in Connecticut, each standard deviation
increase in teacher variables signals a gain in average achievement status for boys of less
than a point while girls tend to lose less than a point. One exception is the effect of
teacher reform orientation where boys increase average achievement status by +1.65
points across teachers while girls decrease by -.65 points.
Implications for the Present and Future

The education system in the United States is connected and can be measured from
policy to student achievement. Teacher preparation programs, teacher characteristics, and
professional development have reasonably strong effects on teacher practices that in turn
have small but significant effects on average student achievement in mathematics and
reading. As a whole, the system supports the development of mathematics teachers in
more predictable ways than it supports the development of reading teachers but the
ultimate effect on student achievement for both subjects is in the range of 5% .05. There
is room for tightening the system and increasing the size of those effects.

It is good news for government that it is possible to track the contributions of
teacher preparation, teacher characteristics, and professional development, to teacher

practice and student achievement. This makes it possible to measure improvements in

289

 

 

 

the connections among the variables as well as predict future student achievement. Using
data from this study, one could predict that a Black student in Connecticut with gains at
the average on the state mathematics achievement test would grow from +6.68 points to
+334 points from 2000 to 2005. Adding a one standard deviation increase in teacher
sensemaking activities (+1.25 points) would result in a further increase from 33.4 points
to almost 40 points over that same time period.

The bad news is that this effect may not be enough to attain the NCLB goal of
proficiency for all students by the 2014-15 school year. For the sake of argument, take a
White student who has a mathematics CAPT score of 199 (below basic in CT) and
project this score until 2005 using the +4.05 average increase until 2005. The anticipated
outcome score would be just under 220 by 2005, just short of the proficient level (223-
260). By 2010, the projection would reach 240; by 2015, the student would reach the
Connecticut Goal of 261, just above proficient, the level required for AYP. Of course,
this presumes a perfect world with no intervening circumstances and steady progress at
the predicted average. In spite of average increases over time, 5:: .05 may not be enough
to make AYP requirements in Connecticut or anywhere else. Increases in certain teacher
practices, such as mathematics sensemaking, could add additional point increments;
however, in the case of differential effects, one disaggregated group may improve at the
expense of another.

Improvement in academic achievement does not exist in a vacuum. Students
move in and out of districts and states frequently. Immigration patterns guarantee that
many schools start over with new students every year, new students who will struggle to

acclimate to their new school, curriculum, and testing. Although it is possible to improve

290

 

 

the achievement of students currently in the system, the constant flow of new students
means that the system will always have an unending supply of students who will need to
be brought up to speed.

It is good news that minority groups and women in Connecticut are closing
achievement gaps in mathematics and reading when gain scores are the outcome. This
bodes well for reporting improvement in subgroup scores for NCLB. It is bad news,
however, that achievement status from NAEP still indicates that minorities have yet to

equal or outpace the overall scores of their majority peers.

 

The biggest hurdle for all students is still low SES. The effect of one standard
deviation increase in free and reduced lunch at the district level lowers the average gains
of Black students by around -19 points for mathematics in Connecticut. In NAEP 2000
Reading, the effect of one standard deviation increase in free and reduced lunch at the
classroom level lowers student scores by -18 points. While it is clear that the effects of
certain teachers variables on average achievement by ethnicity and gender is differential,
no teacher level variables can overcome the overwhelming effects associated with SES.
As long as socioeconomic inequality exists in the extremes, adequate yearly progress may

be impossible.

 

As always, more research needs to be done with data that spans government
policy to student achievement which means appropriate databases and/or ways to bridge
data sets need to be developed. Stronger connections need to be made throughout the
education system, especially in reading, so that all elements of the system are working in

concert. More evidence of teacher practices that improve student achievement for all

291

students needs to be gathered. And in all this, it can’t be forgotten that human beings are

not perfectly predictable so education systems will never be.

292

APPENDICES

 

293

APPENDIX A: Correlation Tables

 

 

294

 

vnod

Fmomv

Nomd
mmmd
FFNd
ooNd
moFd
ooNd
mFod
oovd
ooo.F

NF

ooFd
NNmmF
Fvod
mood-
mood-
oood-
vmod
oood
mood
FNod-
omod
ooo.F

FF

oomd
oNnNF
oood
vad
Nvod-
NFFd
Food-
NFod
FoFd-
mood
ooFd
vFNd-
ooo.F

oF

oFod

mvomm

wood-
wood-
Food
oood
n FNd
om F .o
oood
oood-
omod-
omod
vvod
ooo.F

 

Novd

moNNN

mood-
omod-
omFd
vad
mF Fd
mFFd
FoFd
moFd
omFd
vad
NFod
Nvmd
ooo.F

omod

NmFmN

oFod-
vmod
mood
FF Fd
omod
mood
Food-
vvod
moFd
oood-
oFod
FoFd
oomd
ooo.F

mNNd

Nomdo

oood-
mood-
mood
mFNd
Fde
noFd
mood
vood-
mood
NoFd
oFod-
omod
nomd
man
ooo.F

ommd

ooNdv

mood
oood
mood
mmwd
ooFd
ooNd
vFNd
owod
owod
vwod
mFod
oFNd
ooFd
oFod
mood
ooo.F

Aom _HS momHmEofimE Foo-5v doom 50:02:50 —.< 93.5.

NoFd
oomN
oood
oan
omod
vood-
mood
omod-
nmod
m Fod-
mood
omod-
mF Fd
omod
NmFd
Fnod
om F .o-
oood-
oooF

oood

NooNv

omod
mood
mood
omod
Numd
FNvd
oood
ovod-
mmFd
god
Food
o de
mmwd
moFd
oFNd
moFd
omFd
ooo.F

 

mom .o

noodo

Food
mood-
mmvd
mood
oovd
wood
wood
vood
ooFd
Nood-
oood
mnod
oood
onod
ooNd
ooFd
omod
onmd
oooF

N

mood
No F .o
ovwd-
howd-
voFd-
oFod-
Nood-
omFd
nood-
mood-
moFd-
omod
vnod
Nood
mood
nuod
NF Fd
Food-
nood-
mood-
mudd-
oooF
F

>moEm

15.13...sz oN
Omuhzmfi oF
m<IOO<u oF
Adamo-E...— NF
..(DOOOmn. mF
ameZO m F
>O<0Euw VF
“guanin— oF
Swamma NF
ammo—2 FF
mambo—2 oF
>>Zv=...:>_ o
(Q0723. o
(401.;— 5
EthS. m
..(DOIow m
Quad-mks). v
th_I._.<_>. o
._.wZ_mw< N
mow—25. F

oEmtm>

cmmE

295

 

 

Bond
vmoNN
ooo.F

do

moo.F
moF.Fo
mnod
ooo.F

oF

 

omod
oomoN
ooFd
mNFd
oooF

oF

vad
Foo.mF
Fan
vad
FmNd
ooo.F

NF

moNd
oomoF
Food
omod
vmvd
moNd
ooo. F

mF

 

vad
mvv. FN
FNFd
FoFd
omod
mmNd
ond
oooF

mF

mood
oFo.oN
omFd
Fmod
noFd
NF Fd
vad
moFd
ooo.F

VF

 

mood
onNoF
mood
Fmod
Fde
mood
Fmod-
mmNd
oood-
oooF
oF

>mu2m

cme
IFm>>._.2w_2 om
Smut-292 oF
m<IOO<m oF
axmnfimi NF
4<DOOO¢Q mF
ammaZD mF
>O<0Euw vF

u>w0uma oF
o_nm:m>

92-5 85823: 52:. 88 5285.8 8858on _.< 2...;

296

oood

oFo.Fo

mood
omod
omod-
voFd-
oood-
otd
oood-

oF

mood

mmm .mv

oovd
mmod
oood
oood
oFd

wood
oood
Food

FF

ommd

ovodv

FFod
oFod
omod
mod
ooFd
ovod
Food
oood-
omod
F

3

omvd

ooooo

ooFd
moF.o
voFd
Fvod
mvod
Food
moF.o
moF.o
ooFd
ood
F

 

Fmd

ooodo

ooFd
omFd
omvd
oood
mmvd
movd
mood
omod
vad
voFd
novd
F

Food

ooF.mo

Food
ode
Fovd
mowd
vovd
momd
oood
oFod
oFod
Food
ovod
Fomd
F

oFod

moF.oo

moF.o
oood
om Fd
vood
vad
wood
omod
mood
moF.o
oood
ovod
de
vood
F

oood
R. Fo
oood-
ood-
oood
mmod
oo F .o
voFd
Fo F .o
oovd-
om F .o
FoFd
moF.o
Fmod
FoFd
oood

oood
moF.o
oood
vFod
ovod
ode
oFod
ooFd
moF.o
oood
omod
ofiod
oFd

oF Fd
F Fd

omod
mood

.3.-5 323. 3-5.. 88 3385.8 3. 2.3

 

oood
oood
m Fod
omod
wood
oood
mood
m F Fd
vF Fd
mood
Food
FmFd-
m Fod-
Food
ooFd
omF .o
oood
oood-

oood
oooK
oood
Fod-
oood-
ooFd
oood
moFd
Fmod-
moF.o
m Fod
FumFd-
Food-
oood
oood
mFod-
mood
Fod-
Food

Fvod

omoév

Food
voFd
vovd
oood
oFmd
oomd
oFod
mood-
Fmod
oood
oomd
mmod
med
FmFd
moF.o
vad
oood
oFod

>mUEm
:mmE
I._.~....>>._.Zm=>. oF
O._.:m oF
m<IOO<u NF
axwo-EE mF
4450001.”. mF
men.>.o VF
>O<O.n.n.m oF
“Sup-En. oF
_>m.Dn.ma FF
4<Dczow oF
>>OZv.._.3 o
..wz.wm< o
amehOE N.
Emma-F... m
o<m._:_..-. m
2200...... v
Eamon. o
h<§._._-. o

..wz.o<mm F
$3355

297

oood
vmo. Fo

oF

vmod
oFo.mo
FumFd
F

oF

Fmod
mom.oo
Food
m Fod
F

F

cos-5 380m 52:. 88 323258 885603 ~.< as;

.Vmod
moo.mF
oFd
Fvod
mmod

mF

mood
oom.oF
VFFd
ode
omvd
oood

.VF

 

oood
mvv. Fo
ooFd
Food-
oood
mmod
mvvd

F

vF

oood
oo.oo
mmod
FoFd
o F Fd
oood
mood
omFd
F

oF

>255

CNOE
Ihm>>hzm§ oF
0......m oF
m<Ioo<m NF
wanfiwi mF
4<DOOO~E mF
mmmaZo .VF

>O<0_--.n.w oF
mofimcm>

298

ooF.o
omF.mF
ooF.o-
Food
omFd
ooF.o
moF.o
ooF.o
oood
omod
woFd
omFd
oood
mood
ooo.F

mF

oFo.oF
mow.mw
ooo.o-
omod-
mmod
Fwod
Fmod
oood
oood
wood
mde
o Fod
oood
ode
oood
ooo.F

wF

on. FF
oooww
ooF.o-
moo.o-
oomd
oFmd
ommd
ooF.o
woFd
ond
ooF.o
omod
Fmod
ooF.o
oood
oood
ooo.F

oF

oom.w ooo.o
oFo.oo moF.mo
oood oood
oo F .o ooo.o-
omFd momd
ooF.o F Fod
FmFd oowd
woFd ooFd
oo F .o omod
omod wowd
oFo.o- moF.o
Food wood
mood owod
ooF.o mmFd
oF Fd moo.o
woFd oood
oo F .o oowd
ood. F moF.o

ooo.F
oF FF

moo.o
ooo.oF
moo.o
wF Fd-
moF.o
wmod
ooF.o
Food
moo.o
ooo.o-
moF.o
wood
oFod
oood
ooo.o-
oF Fd
moo.o-
wmod-
oFod
ooo.F

oF

moo.w
mom.oF
Food-
ooF.o
Food
omod
oood
ooF.o
oood
oood
oood
ooF.o
o Fod
ooF.o
mmFd
oomd
oood
oood
Food
Fmod
ooo.F

axis 80:22.8 2.8: Ba 8.35232 .32... 88 casssF 3. 2.5

oom.o
moo.oF
wood
moFd
owod
oood
mFod
moo.o
omod
moF.o
omod
ooF.o
oood
moF.o
omFd
wmwd
FoFd
moo.o
omFd
omod
wood
ooo.F

o

FFo.oF
Fww. Fo
omod-
ooo.o-
oFod
oood
omod
omFd
moo.o
oood
FmF .o
oood
omod-
ooF.o
oood
ded
mwod
oood
moo.o
wood
wood
moo.o
ooo.F

o

owm.o
ooo.oo
ooF.o-
Food-
wwod
ode
m Fod
ooF.o
Food
oood
ooF.o
wm F .o
moo.o-
oFod
moF.o
oowd
oood
omod
oood
wo F .o
ooF.o
moo.o
mmod
ooo.F

m

ooo.w
ooo.oF
wmod-
mood-
momd
oowd
womd
omod
owod
mwod
oood
omFd
ode
omFd
ode
moo.o
owmd
ooF.o
omwd
mmod
oowd
oo F .o
oood
Food
ooo.F

m

on.F
woo.o
ooo.o-
ooo.o-
ooo.o-
ooo.o-
moo.o-
oood
moo.o
wad
ooo.o-
oood
Food
oFod
oood
wmod-
oood
oFo.o-
oood
omod-
ooo.o-
moo.o
o Fod-
o Fod
oFo.o-
ooo.F

w

ooo.w owo.o
Foo.oo mFo.wF
wood- Foo.o-
oood oood
omFd wood
moF.o wood
ooF.o mmod
oFFd Fmod
moF.o FoFd
oFod ooF.o
ode oood
ode owod
oood ooF.o
oFo.o- mmFd
moF.o moo.o
mmod mFod
moo.o owod
oFo.o- wood
oood Food
mmod- omod
oood Fwod
ooo.o- omFd
oood oood
FFod wood
moo.o ond
ooF.o wmod-
ooo.F oood

ooo.F
o o

mFo.w

mFo.Fo

oFod
oood
oowd
mowd
wowd
ooF.o
ooF.o
oood
ooF.o
wood
F F Fd
omFd
oood
oood
wwwd
moo.o
omod
o Fod-
o Fod
ode
ooF.o
moF.o
mFmd
moo.o-
mFod
oowd
ooo.F
F

52%
:mmE
0256”. R
:25. mm
5255: mm
ammnSa «N
5259.. mm
9.55 mm
anmFJFN
2.0.ze om
05m 2
Sets. 8
<¢0xo2~+
2.25 S
ammo: m.
_>moamavF
5255.2 2
55:5 N.
265:: :
SES: 2
o>moammm
u>moamdm
9.“:sz o
15.5sz o
d<nooomam
ammaxbsz v
32:15 m
Ema-mm N

m<zoo<a.
285$

299

oF F.w oow.o
mowd- wowd-
ooo.F oood

oo

ooo.F

mo

ooo.o

oom.ow

omo.o-
omFd-
ooo. F

mo

Aowwncv 30:22.50 oEomoM 93 8.38053). .oo-5w oooo oommoscoh 6035253 o.< 033—.

ooo.m owo.o
omo.o F om F .oo
oood moF.o-
odod omo.o-
momd mFod
ooo.F wmmd

ooo.F
wo oo

 

ooo.o

mom. Fo

wmod
oFod
oood
Food
ooF.o
ooo.F

oo

woo.m

owo.oo

mwod-
ooFd
ooF.o
oood
ooF.o
wwwd
ooo.F

Fo

Foo.o

owo.oo

moo.o
omFd
moo.o
omod
wwod
Fwod
oomd
ooo.F

oo

 

Foo.o

Fow.wo

omo.o-
omo.o-
omFd
ooF.o
mde
moo.o
oood
oF Fd
ooo.F

oF

owo.o

ooo.oo

wFod-
wood-
FoFd
omod
ooF.o
mowd
omFd
oood
moF.o
ooo.F

oF

woo.m

ooF.oF

wwod-
ooFd
ooF.o
omod
omFd
Foo.o
oomd
womd
oFod
wood
ooo.F

oF

 

mmo.o

FwF.FF

wmod
oood
ooF.o
moF.o
omFd
ooF.o
mF Fd
med
oood
omod
moo.o
ooo.F
mF

5.86
59:

ocfimo. oo
Ems mo
55.62 mo
amazu wo
.mEmmm oo
22:: oo
.59.: Fo
89:8 oo
0:3 3
99...: oF
39:5 t
as; 2
o_nm:m>

300

oFo.F oFF.F mooF

momd oom.F ooo.o
ooo.F ooFd med

wF

ooo.F oowd
ooo.F
oF oF

oom.F
oFo.o
ooF.o
wood

omod
ooo. F

FF

mow.o
mow.o
ooF.o
oFo.o

oFo.o

ooo.o
ooo.F

oF

mFod
oww.o
ooF.o
ooF.o

ooF.o
mmo.o
moo.o
ooo.F

moo.o
Fow.oF
moo.o
ooo.o

ooo.o
FoFd
oom.o
moF.o
ooo.F

oom.o
ooo. FF
omFd
ooo.o

moo.o
FFod
oow.o
moF.o
oom.o
ooo.F

oFo.F
woo.o
ooF.o
wFod

moF.o
ode
woo.o
omod
wmod
moo.o
ooo.F

m

.2282 sans 2832.8 8:323: $22 :55 5.. 2.2 3.. 2.3

ooo.oo

ooo.ooo

wood
moo.o-

woo.o-
wmod
moo.o-
omod
ooF.o
oF Fd
omo.o-
ooo. F

m

Fow.oo omw.oo oFm.oo woo.oo
oow.ooo owo.ooo ooF.woo ooo.ooo

wo F .o
ooo.o-

ooo.o-
moo.o
ooo.o-
ooo.o
Foo.o
moF.o
omo.o-
mmo.o
ooo.F

w

9 Fd
moo.o-

moo.o
woo.o
ooo.o-
omod
ooo.o
mF Fd
Fmod-
omod
mmo.o
ooo.F

o

Foo.o
ooo.o-

moo.o
ooo.o
wwod-
Fmod
ooo.o
moo.o
wmod-
mmo.o
omod
omod
ooo.F

o

ooo.o

moo.o-

ooo.o
omod

mwod-

omod
ooo.o
wo F .o

mmo.o-

Fmod
omod
omod
mmo.o
ooo.F
F

am
58.2

5.5835 Ems. ..F
3652.. .o. cozmamaoi 9

5.83.2.

£9: 5. cozmamaoan. oF
co_.m.co_.o ESE”. FF
.coEmmommm (do 3
ooo.-“.335. moszoEms. o
mm_._>_.om ocimEomcoo o
822.2 8.9a a 9.90 A
c9“. >00 $568.91 m

m>n.-ms.oam.2 m
w>n.-w.20n.m.2 w
o>n.-o.20n.m_2 o
o>n_-o.20&m_>_ o
F>n_-F.20n.m.2 F

301

oFo.o ooo.o wmod omo.o oww.o oFmd oood mmF.F moo.oo ooo.wo ooo.wo ooo.oo FmF.wo 509m
www.o oFo.oF omo.o owo.o omo.oo omo.o omo.F oow.o oFo.moo omo.moo omo.moo on.ooo omF.ooo cmoE

ooo.F mow.o moo.o wwod owo.o moF.o omFd woFd ooF.o ooo.o moo.o ooo.o ooo.o (n.0oF
ooo.F ooo.o ooo.o- ooo.o FoFd Food ooo.o Fmod Fmod omo.o omo.o ooo.o mm_._>_.0m ocimEmmcmw oF
wozmsmcums
ooo.F oFo.o- omFd owo.o ooo.o ooo.o mFod moo.o- ooo.o- oFo.o- 05.0- .0. cozmfiami FF
ooo.F woo.o mmo.o ooo.o- ooF.o- mwod- ooo.o- ooo.o- mood- wwod- wm.._>_.0mw3m oF
0..

8323:. 8.30.33... o
33 933 .o mommies. o

ooo.F ooo.o oFo.o omFd wmod mwod woo.o moo.o moo.o
ooo.F ooo.o omFd owo.o omo.o wood omo.o ooo.o

302

ooo.F ooo.o ooo.o ooo.o ooFd oF Fd moF.o £03 5....02 .0 mo0m_>>0cv. o
. ooo.F Food oFod ooo.o wood- oFod 9.. >00 3.5.3905 m
ooo.F Fmod omo.o owo.o mwod m>a m
ooo.F omo.o Fwod owo.o w>n. w
ooo.F mwod mwod o>a o
ooo.F Fmod o>n. o
ooo.F F>a F
oF oF FF oF o o o m m w o o F mBmtm>

Goofs. m5.3.2.80 8.3.82.3... a... oooo om<z m.< o.._-.._-

oowd
omo.o
nwF d
oow.o
ooo. F

mF

Nmnd
mwmsm
Fmod
wad
womd
ooo.F

NF

moo.o ooF.o oom.F
oFon FFowF Fund

ooF.o
momd
oww.o
woo.o
ooo.F

FF

ode
oom.o
ooF.o
momd
oom.o
ooo.F

oF

ooo.o
FoFd
ooo.o
momd
oFo.o
moF.o
ooo.F

oom.F
mmo.o
moo.o
otd
oFo.o
oww.o
wmmd
ooF.o
oow.o
ooo.F

woo.m
oomdw
wwod
mow.o
med
womd
moo.o
moF.o
omo.o
Fmod
ooo.F

mwFF
oFo.o
ooo.o
moF.o
oFFd
moF.o
wmmd
omo.o
mow.o
NF F.o
oww.o

ooo.F

oNoNN FFo.oN
moo.o—.N oom.oFN

unod-
oNF.o-
ooo.o-
oo F .o-
mmo.o-
woFd-
mo F d-
oood-
ooo.o-

med-
ooo.F

ooF.o-
FNFd-
woo.o-
FoFd-
mmo.o-
mF Fd-
mde-
woo.o-
mwod

moF.o-
woo.o
ooo.F

onus Fund 5003
FmF.mF mow.ow camp.
83 Rod Sigma 9.68m 2
ooo. F g 30:9 EoEmmmmm< 3
3 3 03955

finned wagon woo. $22 825303 e.< 93;

>025
cme

wFoNN omo.om moFNN
moo.mFN wmmdFN FNndFN

mmo.o- omo.o- Bod- Sigma 968m 8
auto- 53- 83. 90553553323
Foo.o- «Ed- mmo.o- 8226323582
mmo.o- 53. :8- mmzszomacoaug 9
god. Fmod- Sod- 8:38 Eemm 2
KS. mmo.o- 23- 852.2592:889mm?
ES- 83- BS. 8:989“. £233
:3. mmo.o- mmo.o- Sigma 32858;
08.0 Foo.o- 30.0- 92a cofiafismsummwn

50

onto- 08.0. 83- am. 5 use; a @589 - on. o
mow.o v8.0 33 3:
Sad mow.o :3 «>3.
83 «5.0 oom.o min
83 oom.o Nam

83 $3

m N F 29%.;
Anmhucv

wcamom M32 mm<z eé 9::

303

wmmdm NNFdN omod
mmowFN oNNdFN moms

mmo.o
Nod
mmo.o
F

NF

Fod
wFod
wood
wood
F

FF

Foo.o-
ooo.o-
woo.o-
ood-
mmo.o-
F

9

oomd
NNNwN
owo.o-
owo.o-
oFo.o-
owo.o-
owo.o-
med
F

mFo.w
moo.mF
moo.o-
Food
oFo.o
wood
wood
oom.o
mmo.o
F

ome
wmod
wood
wood
mood
wood
wood
mwod
Nuod
Foo.o

Foo.o
omo.FN
mmFd
mNFd
ooF.o
oFFd
ooF.o
moF.o
moF.o
oFo.o
moF.o

wooF
NF F.o
whod-
oFo.o-
oFo.o-
muod-
omo.o-
oNFd
oww.o
oom.o
omo.o
oww.o

NNFN
9055
wood
wood
wood
wood
oFo.o
mwd

med
wwwd
woo.o
hood
Fmod

oFm.oN
Fow.wFN

F

mF

gonna 88 380m “:22 625::on 2 «35

no F.o
wadF
nod-
Food-
mmo.o-
ood-
mwod-
woFd
mow.o
owd
owo.o
tho
FmNd
womd
F

«068 3de 502m
www.mFN 3‘me I 59:
mFmd Fmod Flmzommm mF
F moo.o Fuzzoamm 3
F F mzoamm mF
wF mF m_nm_.m>

woo.o
mFo.w
NoFd
woo.o
NF F.o
FoFd
wood
wood
Rod
mwod
wwod
oom.o
mowd
F Fwd
moo.o
F

N

gang ooom macaw $22 F4 2...;

NwF.F
NF F.o
omo.o-
omod-
wood.
wood.
Fmod-
moo.o
man
mde
mmo.o
oow.o
Nd
FmFd
FNd
woo.o
F

F

>89»
:09:
FHmzoamm 2
£285 3
F «28% 9
Fmgoamm NF
F onamm FF
wm_F_>_Fo< $5.300 oF
wwE>fi0< 9»: a
wm_:>:o< 58mm m
ammn. 90m F
mama 52. o
mum“. >5 m
ammn. :8: v
mwmwm< m
mmma >5 N
503mm. 5 on. F
m_nm_._m>

304

APPENDIX B: Miscellaneous Tables

305

F o o F F o o F 0.95: 550
o F o F o F o F 232 2:82:
o o F F o o F F 29:00 xoflm
o o o F o o o F 0.080“.
c o o o F o o F 0.9: 550
o o o o o F o F was 2&2:
o o o o o o F F was xogm
o o o o o o o F was 2E>>
0.2.3 068$ 29:3
.050 2335 xogm 0580”. 550 2:33. xom_m 3089:.

9:000 NEESD - momsafiwofioa EonEm —.m 935—.

306

REFERENCES

Amrein, A. L., & Berliner, D. C. (2002). High stakes testing, uncertainty and student
learning. Education Policy Analysis Archives, 10(18).

Angus, D. L. (2001). Professionalism and the public good: A brief history of teacher
certification. In J. Mire] (Ed.). Washington, CD: Thomas B. Fordham Foundation.

Arbuckle, J. L. (1994-1999). AMOS (Version 4.01 (Build 344)). Chicago, IL:
SmallWaters, Corp.

Archer, J. (1999). Sanders 101. Education Week.

Ball, D. L. (1997). Developing mathematics reform: What we don't know about teacher
learning -- but would make a good working hypothesis. In S. N. Friel & G. W.
Bright (Eds), Reflecting on our work: NSF Teacher enhancement in K-6
mathematics (pp. 77-111). Latham, MD: University Press of America.

Barton, R., & Oja, S. (1999). Predicting beginning teachers' feelings of efficacy.
Educational Reseach Quarterly.

Borko, H., & Putnam, R. (1996). Learning to teach. In R. C. Calfee & D. C. Berliner
(Eds), Handbook of Educational Psychology (pp. 673-708). New York:
Macmillan.

Byme, B. M. (1998). Structural equation modeling with LISREL, PRELIS, and SIMPLIS:
Basic concepts, applications, and programming. Mahwah, NJ: Lawrence Erlbaum
Associates.

Camilli, G., Vargas, S., & Yurecko, M. (2003). Teaching children to read: The fragile
link between science and federal education policy. Education Policy Analysis
Archives, 11(15).

Cohen, D. K. (1990). A revolution in one classroom: The case of Mrs. Oublier.
Educational Evaluation and Policy Analysis, 12(3), 327-345.

Cohen, D. K., & Hill, H. C. (2001). Learning policy: When state education reform works.
New Haven: Yale University Press.

Cohen, D. K., & Spillane, J. (1993). Policy and practice: The relations between

governance and instruction. In Designing coherent educational instruction (pp.
35-95). San Franciso: Jossey-Bass.

307

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New
York: Academic Press.

Cohen, J. (2002). AM statistical software (Version 0.06.01): American Institutes for
Research.

Cusick, P. A. (1992). The educational system: its nature and logic. New York: McGraw-
Hill.

Darling-Hammond, L. (2000). Teacher quality and student achievement: A review of
state policy evidence. Education Policy Analysis Archives, 8(1), 1-56.

Darling-Hammond, L., & Youngs, P. (2002). Defining "highly qualified teachers": What
does "scientifically-based research" actually tell us? Educational Reseacher,
31(9), 13-25.

Delpit, L. D. (1988). The silenced dialogue: Power and pedagogy in educating other
people's children. Harvard Educational Review, 58, 280-298.

Eberts, R. W., & Stone, J. A. (1984). Unions and public schools: The effect of collective
bargaining on American education. Lexington, MA: Heath.

Education Counts Research Center. (2002). Education Counts, 2004, from
http://www.edweek.org/rc/edcounts/

Ferguson, R. (1991). Paying for public education: New evidence on how and why money
matters. Harvard Journal of Legislation, 28(Summer 1991), 465-498.

Floden, R. E., & Buchmann, M. (1993). Detachment and concern: Conversations in the
philosophy of teaching and teacher education. New York: Teachers College
Press.

Foster, M., & Peele, T. (1999). Teaching and learning in the contexts of Afi‘ican
American English culture. Education and Urban Society, 31(2), 177-189.

Fuhrman, S. H. (2001). Introduction. In S. H. F uhrman (Ed.), From the capitol to the
classroom: Standards-based reform in the states (pp. 81-98). Chicago: The
Unviersity of Chicago Press.

Goertz, M. E., Duffy, M. C., & with Le F loch, K. C. (2001). Assessment and
accountability systems in the 50 states: 1 999-2000 (N o. RR-O46). Philadelphia,
PA: Consortium for Policy Research in Education, University of Pennsylvania
Graduate School of Education.

308

Greenwald, R., Hedges, L. V., & Laine, R. D. (1996). The effect of school resources on
student achievement. Review of Education Research, 66(Fall 1966), 361-396.

Grossman, P. L. (1989). Learning to teach without teacher education.
Teachers College Record, 91(2), 192-208.

Heath, S. B. (1983). Ways with words: language, life, and work in communities and
classrooms. Cambridge [Cambridgeshire] ; New York: Cambridge University
Press.

Hirsch, E. D. (1996). The schools we need and why we don 't have them (lst ed.). New
York: Doubleday.

Kennedy, M. M. (1998). The effects of inservice programs on student achievement. Paper
presented at the Annual Meeting of the American Educational Research
Association, San Diego.

Kennedy, M. M. (1999). The problem of evidence in teacher education. In R. Roth (Ed.),
The Role of the University in the Preparation of Teachers (pp. 87-107). London:
Falmer Press.

Kingdon, J. (1995). Agendas, alternatives, and public policies. Boston: Little, Brown and
Co.

Labaree, D. F. (1997). Public goods, private goods: The American struggle over
educational goals.

Linn, R. L. (2001). The design and evaluation of educational assessment and
accountability systems (N o. CSE Technical Report 539). Los Angeles: Center for
the Study of Evaluation.

Linn, R. L., & Baker, E. L. (1998). School quality: Some missing pieces. The CRESST
Line: Newsletter of the National C enter for Research on Evaluation, Standards,
and Student Testing, 2-3(7).

Linn, R. L., Baker, E. L., & Betebenner, D., W. (2002). Accountability systems,
Implications of requirements of the No Child Left Behind Act. Educational '
Researcher, 31(6), 3-16.

Lipsky, M. (1980). Street-level bureaucracy : dilemmas of the individual in public
services. New York: Russell Sage Foundation.

Lortie, D. C. (1975). Schoolteacher: a sociological study. Chicago,: University of
Chicago Press.

309

McCartney, K., & Dearing, E. (2002). Evaluating effect sizes in the policy arena. The
Evaluation Exchange Newsletter, 8(1), 4 & 7.

McCartney, K., & Rosenthal, R. (2000). Effect size, practical importance, and social
policy for children. Child Development, 71, 173-180.

McDonald, L. M., & Elmore, R. F. (1987). Getting the job done: Alternative policy
instruments. Educational Evaluation and Policy Analysis, 9(2), 133-152.

Muncey, D. E., & McQuillan, P. J. (1996). Reform and resistance in schools and
classrooms. New Haven: Yale University Press.

National Center for Research on Teaching and Learning. (1993). Findings on learning to
teach. East Lansing: Michigan State Univerisity.

National Council of Teachers of Mathematics. (2000). Principles and standards for
school mathematics. Reston, VA.

National Reading Panel. (2000). Teaching Children to Read: An evidence-based
assessment of the scientific research literature on reading and its implications for
reading instruction. Washington, DC: NICHD.

No Child Left Behind Act of 2001, (2002).

Nye, B., Konstantopoulos, S., & Hedges, L. V. (2004). How large are teacher effects?
Education Policy Analysis Archives, 26(3), 217.

Paige, R. (2002). Meeting the highly qualified teachers challenge: The Secretary 's annual
report on teacher quality. Washington, DC: U. S. Department of Education.

Paige, R. (2003). Meeting the highly qualified teachers challenge: The Secretary's annual
report on teacher quality. Washington, DC: U. S. Department of Education.

Paige, R. (2004). Meeting the highly qualified teachers challenge: The Secretary 's annual
report on teacher quality. Washington, DC: U. S. Department of Education.

Pearson, P. D. (1996). Reclaiming the center. In The First R: Every Child's Right to Read:
Teachers College Press.

Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and
data analysis methods (Second edition ed. Vol. 1). Thousand Oaks, CA: Sage
Publications.

Raudenbush, S. W., Bryk, A. S., & Congdon, R. (2005). HLM for Windows (Version
6.01h). Chicago, IL: Scientific Sotware International.

310

 

Raudenbush, S. W., Kasim, R., M., Eamsukkawat, S., & Miyazaki, Y. (1996). Social
origins, schooling, adult literacy, and employment: Results from the National
Adult Literacy Survey.

Ravitch, D. (2002, March 2002). A brief history of teacher professionalism. Paper
presented at the White House Conference on Preparing Tomorrow's Teachers,
Washington, DC.

Reckase, M. (2002, May 2002). What factors in the preparation of teachers influence
how much students learn? Paper presented at the First International Education
Conference, F amagusta, North Cyprus.

Riley, R. W. (2000). The initial report of the Secretary on the quality of teacher
preparation. Washington, DC: US. Department of Education.

Rowan, B., Chiang, F.-S., & Miller, R. J. (1997). Using research on employees'
performance to study the effects of teachers on students' achievement. Sociology
of Education, 70(October), 256—284.

Rowan, B., Correnti, R., & Miller, R. J. (2002). What large-scale, survey research tells us
about teacher effects on student achievement: Insights from the Prospects Study of
elementary schools. Teachers College Record, 104(8), 1525-1567.

Sanders, W., & Horn, S. (1998). The Tennessee Value-Added Assessment System
(TVAAS): Mixed model methodology in educational assessment. In Models for
Teacher Evaluation.

Sanders, W., & Rivers, J. C. (1996). Cumulative and residual eflects of teachers on future
student academic achievement. Knoxville: University of Tennessee Value-Added
Research and Assessment Center.

Sheff v O'Neill (Connecticut Supreme Court 1996).

Shulman, L. S. (1987). Knowledge and teaching: Foundations of the new reform.

Spellings, M. (2005). Meeting the highly qualified teachers challenge: The Secretary 's
annual report on teacher quality. Washington, DC: U. S. Department of

Education.

Spillane, J. (2005). Standards deviation: How schools musunderstand education policy.
CPRE Policy Briefs, RB-43(June 2005).

Supovitz, J. (2001). Translating teaching practice into improved student achievement. In
S. H. Fuhrman (Ed.), From the capitol to the classroom: Standards-based reform
in the states (pp. 81-98). Chicago: The Unviersity of Chicago Press.

311

 

Tennessee Small School Systems v. McWherter (Supreme Court of Tennessee 1993).
Traub, J. (2002, April 7, 2002). The test mess. The New York Times on the Web.

Tyack, D. B., & Cuban, L. (1995). T inkering toward utopia : a century of public school
reform. Cambridge, Mass: Harvard University Press.

Amendments to the Higher Education Act, (1998a).
Reauthorization of the Elementary and Secondary Education Act, (1998b).

Valli, L. (2000). Beginning Teacher Survey. In National Partnership for Education and
Accountability in Teaching (Ed.). Columbia, PA: National Computer Systems,
Inc.

Valli, L., Raths, J ., Reckase, M., & Rennert-Ariev, P. (2001). Beginning Teacher Survey
Study: Progress report & preliminary analysis of findings. College Park, MD:
University of Maryland.

Valli, L., Raths, J ., & Rennert-Ariev, P. (2001). The Beginning Teacher Survey Study:
Paper presented at the American Educational Research Association, Seattle, WA.

Valli, L., Reckase, M., & Raths, J. (2003). Teacher education, program outcomes,
teaching practice, and pupil achievement. Paper presented at the Annual meeting
of the American Educational Research Association, Chicago, IL.

Valli, L., Reckase, M., Raths, J ., & Rennert-Ariev, P. (2001). Beginning Teacher Survey
Study final report. College Park, MD: University of Maryland.

Wayne, A. J ., & Youngs, P. (2003). Teacher characteristics and student achievement
gains: A review. Review of Educational Research, 73(1), 89-122.

What Works Clearinghouse. (2002). Overview.
http://www.whatworks.ed.gov/whatwedo/overview.html.

Wilson, 8., Darling-Hammond, L., & Berry, B. (2000). Teaching policy: Connecticut's
long-term eflorts to improve teaching and learning.

Wilson, S. M., Floden, R. E., & F errini-Mundy. (2001). Teacher preparation research:
Current knowledge, gaps, and recommendations: Prepared for US Department of
Education by the Center for the Study of Teaching and Policy in collaboration
with Michigan State University.

Youngs, R, & King, M. B. (2002). Principal leadership for professional development to
build school capacity. Educational Administration Quarterly, 38(5), 643-670.

312

I- Ill. |A\

 

1111111111:1