I. .‘9,t>.a...:<h ..:. ..wu.l..,. .. . . A .. . . . . . ‘ . . ‘ ‘ 3%.. an . ml. (w. 3 ”.5
. , . - , . . . ... . . . . , ~‘. .

a! i
v

.
.

1..
.

V... .v

I
V
.1

="4.; .
“5"; .F
‘3 4L 3;.

".VII'
' J
' h
‘1
{R

mu
'4
.5?”
50*.

 

 

1

_i1

 

l.‘
N

 

 

 

 

 

‘iigl

A \IJ\
I .r .v" .
".u..l
L val

' P

 

new? .. -. v4

 

. £711.51;an
V 1.11.1..."1 :1...
V .. W‘w‘

L Yyll.b .I ‘... II. .
; null-'I

 

 

THESIS

NISVER SITY LIBRARIE

IIIIIIIIIIIIIIIIIIIIIIIIIIIIII III I III IIIIIIIIIII

3 1293 01688 471

 

 

 

 

 

 

This is to certify that the

dissertation entitled

The Impact of Test Consequences and
Response Format on Performance

presented by

Christine E. DeMars

has been accepted towards fulfillment
of the requirements for

Ph.D. degreein Measurement and
Quantitative Methods

/

Major professor

Date £‘g’j3

MSU is an Affirmative! Action/Equal Opportunity Institun'on 0— 12771

 

 

LIBRARY

Michigan State
; Unlverslty

 

 

 

PLACE IN RETURN BOX
to remove this checkout from your record.
TO AVOID FINES return on or before date due.

 

DATE DUE I DATE DUE DATE DUE

 

 

 

Wk .2 I did 985 80670 g

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

1m WM“

THE IMPACT OF TEST CONSEQUENCES AND RESPONSE FORMAT ON
PERFORMANCE

By

Christine DeMars

A DISSERTATION
Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of

DOCTOR OF PHILOSOPHY

Measurement and Quantitative Methods

1998

Copyright by
CHRISTINE ELIZABETH DEMARS
199s

ABSTRACT

THE IMPACT OF TEST CONSEQUENCES AND RESPONSE FORMAT ON
PERFORMANCE

By

Christine DeMars

Students generally perform higher on tests that have some consequences. This
study examined whether the performance difi‘erence between high- and low-stakes tests
remained constant across response formats, genders, and ethnic groups. Also, this
research explored whether any of these factors were associated with non-response, and
compared the fit of item estimates based on student responses under low-stakes to the
response patterns under high-stakes. Data were obtained from pilot and operational
administrations of the science and mathematics sections of the Michigan High School
Proficiency Test (HSPT).

Results showed that students were more likely to respond to constructed response
items, and to score higher on the overall test, when the test had consequences. Whites
were more likely to respond than Blacks, and Whites scored higher on the test. Girls were
more likely to respond than boys, but boys scored higher overall on the science test and
there were no gender difi‘erences in overall scores on the math test. Increased test stakes

increased performance on constructed response items more than on multiple choice items,

and the gender diflerence in performance changed with the response format (boys scored
higher on the multiple choice section, while girls scored higher on the constructed
response section). The ethnic by format interaction depended on the subject matter and
test form, but tended to be small. In math, pilot item estimates fit the operational data
better when omitted items were treated as not-administered, rather than incorrect, during
the item estimation. They also fit girls better than boys, and multiple choice items better
than constructed response items. In science, where the fit statistics were better and

response rates were higher, difi‘erences in fit were small.

ACKNOWLEDGMENTS

Writing these acknowledgments is among the last of my tasks in completing this
dissertation, but those who have helped me along the way certainly are not last in my
thoughts.

I would like to thank my committee members for all their assistance. Dr. Betsy
Becker gave me especially detailed feedback on my proposal, and fi'om her course I
learned the skills needed for the IRT portion of this project. Dr. Susan Phillips kindly
stepped-in at the last moment to join the committee as Dr. Becker prepared to leave for a
sabbatical. Dr. Steve Raudenbush gave me guidance in defining my models and
interpreting my results. Dr. Neal Schmitt provided a flesh perspective from
industrial/organizational psychology. Dr. Chris Schram, as an alumnus of my program and
current stafl‘ member of MEAP, was a helpfiil link to the "real-world" context of my study.
All my committee members provided useful feedback, and I appreciated their positive,
helpful attitudes at committee meetings.

Most of all, I am grateful for the assistance of my advisor and dissertation director,
Dr. William Mehrens. He guided me throughout the dissertation process, as well as earlier
”milestones” in my doctoral program. He always found time to discuss my progress and
encourage me. I knew I could count on him for prompt, helpfiil (and tactful) feedback on

my work.

TABLE OF CONTENTS

LIST OF TABLES ....................................................................................................... viii
LIST OF FIGURES ..................................................... ix
CHAPTER 1
INTRODUCTION ........................................................................................................... 1
Purpose .................................................................................................................... 1
Need ........................................................................................................................ l
The Study ................................................................................................................ 2
CHAPTER 2
REVIEW OF THE LITERATURE ................................................................................ 10
Efi‘ects of Consequences on Test Scores ................................................................. 10
Effects of Consequences on Motivation .................................................................. 12
Efi‘ects of Motivation on Performance .................................................................... 15
Response Format ................................................................................................... 19
Effects of Response Format on Group Differences ............................................. 19
Effects of Response Format on Motivation and Performance .............................. 25
Efi‘ects of Anxiety on Performance ......................................................................... 31
Summary of the Literature ..................................................................................... 35
CHAPTER 3
METHOD ..................................................................................................................... 4O
Participants ............................................................................................................ 4O
Instrument ............................................................................................................. 42
Analysis ................................................................................................................. 46
Response/Non-Response. ................................................................................... 46
Test Scores ........................................................................................................ 50
Item Parameter Estimates ................................................................................... 54

CHAPTER 4

RESULTS ..................................................................................................................... 58
Response Rate ....................................................................................................... 58
Ability Scores ........................................................................................................ 73

Relationship Between Constructed Response and Multiple Choice Scores .......... 73
Factors Afi‘ecting Scores .................................................................................... 76
Fit of the High-Stakes Responses to Low-Stakes Item Estimates ............................ 93
Person-Fit .......................................................................................................... 93
Item-Fit .............................................................................................................. 95

CHAPTER 5

DISCUS SIGN/SUMMARY, IMPLICATIONS, AND CONCLUSIONS ....................... 99
Discussion/ Summary .............................................................................................. 99

Response Rate: Science ...................................................................................... 99
Response Rate: Math ....................................................................................... 100
Ability Scores: Science ..................................................................................... 101
Ability Scores: Math ........................................................................................ 103
Comment on the Ethnicity Variables ................................................................. 104
Pilot Item Fit to Operational Responses ............................................................ 105
Limitations ........................................................................................................... 106
Implications ......................................................................................................... 108
Implications for Test-Development ................................................................... 108
Broader Implications for Educators and Policy-Makers .................................... 110
Directions for Further Research ............................................................................ 112

APPENDD(

RESULTS FROM THE SECOND FORM OF THE SCIENCE TEST ......................... 114

LIST OF REFERENCES ............................................................................................. 122

LIST OF TABLES

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 1 - Student-Level Predictors for Log-Odds of Resppnse, Science ......................... 62
Table 2 - Student-Level Predictors for LogOdds of Response, Mat_h ............................. 63
Table 3 - _L_og;Odds of Response on Science CR Items. The Full Model ......................... 65
Table 4 - Lpg-Odds of Response on Mpth CR ItemLThe Full Model ............................ 66
Table 5 - _L_o,g-Odds of Response on Science CR Items. the Final Model ........................ 68
Table 6 - Log-Odds of Response on Math CR Items the Final Model ............................ 68
Table 7 - Predicted Log-Odds, Probabilig (Rpte), and Odds of Respog, Sg'ehpe; ......... 69
Table 8 - Predicted Log-Odds, Probability (ate), and Odds of Response, Math ............ 69
Table 9 - Predictions for Schools with High and Low Minority Enrollment, Scienpe; ....... 71
Table 10 - Prediptipns for Schools with High and Low Minorig Empllment, Math ........ 72
Table 11 - Student-Level Predictors for Ability Scores, Science ..................................... 80
Table 12 - Student-Level Predictors for Ability Scores, Math ......................................... 81
Table 13 - Predicted Scores in Science. Controlling School Minority Compositipn ......... 84
Table 14 - Predicted Scores in Math, Controlling Schpol Minorihg Composition ............ 85
Table 15 - The Final Model for Science Ability Scores ................................................... 86
Table 16 - The Ema]; Model for Math Ability Scores ....................................................... 87
Table 17 - Average Predicted Scores in Science ............................................................. 89
Table 18 - Standardized Within-School Efi‘gts in Science .............................................. 89
Table 19 - Average Predicted Scores in Ma_th ................................................................. 90
Table 20 - Standardized Within—School Efi‘ects in mth ................................................. 90
Table 21 - Variance Components for Science Ability ...................................................... 92
Table 22 - ym'ance Components fpr Mam Abilig ......................................................... 93
Table 23 - Appropriateness Fit Index ............................................................................. 94
Table 24 - Avemge OUTFIT ......................................................................................... 96
Table 25 - ANOVA Sum Table for OUTFIT, Science ............................................ 96
Table 26 - ANOVA Summm Table for OUfIflIf, Math ................................................ 97
Table A1 - Participants ................................................................................................ 114
Table A2 - Student-Level Prfiictors for Qg-Qdds of Rpsppnse, Science Ygt 2 ......... 114
Table A3 - Log-Odds of Response on Science CR Items, Full Model Year 2 ................ 115
Table A4 - Qg—Odds of Response on Science CR Items, Final Model Yga_r 2 .............. 116
Table A5 - Prfiicted Log-Odds, Probability (Rate), and Odds of Response, Yg 2 ..... 116
Table A6- Schpols with High and Low Minorig Enrollment, Science Year 2 ............... 117
Table A7 - Smdent-Level Predictors for Abilig Scores, Science Year 2 ....................... 118
Table A8 - Prflicted Scores, Controlling School Minorig Comppsitiph, Year 2 .......... 119
Table A9 - The Final Model for Science Ability Scores, Year 2 .................................... 120
Table A10 - Average Predicted Scores in Science, Year 2 ............................................ 121

Table A11 - Standgdized “fithin-School Effects in Science, Year 2 ............................. 121

LIST OF FIGURES

Figure 1: Information Function, Science ......................................................................... 75
Figure 2: Information Function, Math ............................................................................ 76

CHAPTER 1

INTRODUCTION

Pumas:

The purpose of this study was to examine how responses to items on the science
and math sections of the Michigan High School Proficiency Test (HSPT) changed as the
stakes of the test changed, and how (or if) these changes were associated with the
response format of the items and the examinees’ gender and ethnicity. The test was
initially administered in the 1994-95 academic year under low-stakes (essentially no-
stakcs) conditions, as a pilot test (final field test). In the spring of 1996 and 1997, the test
was taken by the next cohorts of students under high-stakes conditions-students who did
not score satisfactorily would not be eligible for a state-endorsed diploma. Both
constructed response and multiple choice items were on the test. Two types of changes in
response were studied: changes in the quantity of responses (response rate) and changes in
the quality/correctness of responses (item scores). Also, the fit of the operational data to
the parameter estimates of the pilot data was examined, as a measure of the accuracy of
the pilot estimates.

Need

Test developers and researchers want estimated item dificulties fiom pilot tests to
reflect the relative dificulties the items will have under operational conditions. If one type
of item changes more than another, the test developers will not get the mix of item

1

2
dificulties they intended. If the performance of one demographic group changes more
than another, estimates of group difi‘erences will not be accurate, which makes it difficult
to assess the impact of cut scores or the educational needs of the groups. Further, if there
is an item by group by testing condition interaction, estimates of difi‘erential item
functioning will not be the same.

Also, pilot tests share similarities with other low-stakes tests, such as the National
Assessment of Educational Progress (N AEP) and various exams conducted by the
International Education Association (IEA). On these examinations, students are informed
that their scores will be anonymous and they receive no individual feedback. These tests
are often even more visible than the tests which have stakes on an individual level.
Findings about pilot tests could generalize to these other important low-stakes situations.
Low-stakes tests may underestimate student performance, and they may not result in
accurate estimates of group differences or response format differences. Accurate
estimates fi'om these tests are important because results from these tests may be used in
making policy decisions.

The Study

The influences of response format, gender, and ethnicity on changes in
performance from low to high stakes testing conditions were the focus of the present
study. The math and science sections of the Michigan High School Proficiency Test
(HSPT) were investigated here. This test was administered to a sample of 11th graders as
a pilot test (low stakes) during the 1994-95 school year. This exam was later administered
to all 11th graders in the state during the following years, leading to state high school

diploma endorsements (high stakes). The test responses of students in the schools which

3

participated in the piloting of the 1996 science form and 1997 math form were studied
here (these were the first years, respectively, that most students needed this test for the
diploma endorsement-4n 1996, many of the tested students had earned math
endorsements from a previous test). The science test form had 8 written constructed
response items and 34 multiple choice items (there were 8 additional multiple choice items
which were difi‘erent on the operational testnthese served as linking items across forms).
The math form had 6 constructed response items and 32 multiple choice items (again,
there were 8 additional items which were difi‘erent on the pilot and operational forms).
Items in both formats were intended to assess some higher level cognitive skills as well as
basic content knowledge. Under both low and high stakes conditions, students were
allowed as much time as they needed to complete the test (according to the administration
instructions). Blacksl were the only ethnic minority members who participated in large
numbers in the pilot study, so this was the only ethnic minority group studied here.
Existing literature suggests motivation and performance should be higher on high
stakes exams. Wolf, Smith, and Bimbaum (1995) found students who could be placed in
remedial classes based on test scores had greater motivation and omitted fewer items than
students facing no consequences. Wolf and Smith (1995) and Wolf, Smith, and DiPaolo
(1996) recorded considerably higher motivation scores (a difference of about 1.5 standard
deviations) in consequential than non-consequential test conditions. Arvey, Strickland,
Drauden, and Martin (1990) measured higher motivation in job applicants taking a

screening test than in current employees.

 

‘ The label "Black" was used here because that was the option listed on the student identification sheet.
Some students who marked this option may prefer the term "African American”.

4

While motivation increases on high stakes tests, anxiety may increase as well. This
could decrease test performance, because highly anxious students tend to score lower on
tests (Crooks, 1988; Hembree, 1988; Wolf& Smith, 1995; Wolf, Smith, & DiPaolo,
1996). On average, though, higher consequences lead to higher test scores (Burke, 1991;
Jennings, 1953; Rothe, 1947; Taylor & White, 1981; Wolf& Smith, 1995; Wolf, Smith, &
DiPaolo, 1996). For the majority of students, then, it appears that the motivating effects
of consequences are stronger than the anxiety-provoking efi‘ects.

This study difl‘ered from previous research in that it focused on how characteristics
of the items and of the test-takers impacted the degree of the effect of motivating
consequences. This study assessed whether test consequences had a greater (or lessor)
efi‘ect on performance on constructed response items than on performance on multiple
choice items. This study also explored whether difl‘erences in test consequences affected
males and females to the same extent, and whether the degree of change was similar for
Blacks and Whites.

Given Freund and Rock’s (1992) findings that males and Blacks appeared to be
less motivated on a low-stakes test, and Karmos and Karmos’ (1984) findings that the
correlation between attitude and test scores was higher for males, a tentative hypothesis
was that the performance of males and Blacks would increase more as the stakes
increased. This would also be supported by Kiplinger and Linn’s (1992) evidence that the
scores of Blacks increased slightly more than the scores of Whites as consequences
increased, though the difi‘erence was small. These findings also suggested that scoring
omitted items as zero rather than treating them as “not administered” would have a larger

impact on ability estimates for Blacks and males.

‘1

5

Regarding response format, nonresponse has been high for low-stakes constructed
response items (Badger, 1989; Freund & Rock, 1992). Motivation has been found to be
lower on constructed response items compared to multiple choice items under low stakes
(Sundre, 1996; Wainer, 1993). Ifthis “motivation gap” narrowed with the external
motivation of high stakes, performance difi’erences would likely decrease as well. Findings
that performance on “mentally taxing” items increased more than performance on less
taxing items under higher stakes (Wolf, Smith, & Birnbaum, 1995) also suggest that
response format efi‘ects (on both response rate and test scores) would decrease under
higher stakes. Findings on ethnic-group difi‘erences on constructed response items
compared to multiple choice items are mixed (Badger, 1995; Bond, 1995; Feinberg, 1990;
Klein, J ovanovic, Stecher, McCafi'ey, Shavelson, Haertel, Solano-Flores, and Comfort,
1997; Linn, Baker, & Dunbar, 1991). If some of the ethnic-group differences were due to
one group having especially low interest or motivation on low stakes tests, the difi‘erences
would decrease under higher stakes, but differences due to curricular/mstructional
difi‘erences would not be affected by the stakes. Specifically, the following questions were
addressed:

(1) What effects do test consequences, response format, gender, and ethnicity have on
response rate?

(2) What effects do test consequences, response format, gender, and ethnicity have on
test scores?

(3) How well do its: parameter estimates based on pilot data fit the operational data, and
how do pilot estimates based on treating omitted items as not-administered compare

to estimates based on treating omitted items as incorrect?

6

To analyze non-response, students’ responses (coded 0 for non-response and 1 for
response) on the constructed response items (response rates were universally high on the
multiple choice items, so they were not included) were analyzed in a hierarchical
generalized linear model. The responses were assumed to be binomially distributed; a
transformation of the probability of responding (the log-odds of response) was used as the
outcome variable, because the error variance of the dichotomous outcome would be
neither normally distributed nor independent of the underlying probability of response. At
the lowest level, within students, the probability of response was modeled as a function of
the students’ general tendency to respond. At the between-student level, several factors
potentially influenced the student’s average response tendency: the test stakes for that
student, the student’s gender, and the student’s ethnic group. At a higher level, school
factors impacted the student factors as well as the average response tendency within the
school. One of these factors was the proportion of students in the school who identified
themselves as non-White (which, among other things, serves as a proxy variable for SES).
Other school factors were not modeled here, but the variance of random school effects
was assessed, so that the variance at the other levels could be more accurately estimated.

For the analysis of the item scores, neither the log-odds model nor the usual linear
model was appropriate, because some of the items were dichotomous while others were
on a multi-point scale. Instead, a one-parameter item response model (partial-credit
model for the constructed response items) was used to estimate two composite ability
scores for each student, one based on the constructed response items and one based on the
multiple choice items. The one parameter model was used because it is the model used to

equate and to score the HSPT. The item parameters were estimated from the operational

7
(high stakes) administration, because effort was more likely to be stable across items in
this group (thus efi’ort was less likely to be a second dimension underlying performance
than it might be if both samples were combined to estimate item parameters). These
parameter estimates were used to estimate the two abilities for each student in both
samples. The parameters for all items were calibrated simultaneously for both sets of
items to place them on the same metric, so for the calibration students the two ability
estimates were the same, on average. However, if the relative dificulty of the items in the
two scores was not constant across subgroups of students, the ability estimate for one
format would be consistently higher in the afi‘ected subgroup.

These scores were the dependent variables in a hierarchical linear model. At the
first level, response formats (two) were nested within students. The error variance was
based on the standard errors of measurement estimated from the IRT analysis, which were
difi‘erent for each scale for each student (the error variance was not homogeneous). At
the next level, the student factors were those used in the non-response analysis: the test
stakes for that student, the student’s gender, and the student’s ethnic group. A unique
term for uncontrolled variance between students was also in the model. At the school
level, school-efl‘ects again were modeled to depend on the proportion of students who
were minority members. The variance of the school means (conditional on the means
predicted from the student ethnic composition) was estimated to allow for accurate
estimation of error variance at the other levels.

At the lowest-level, the average format coefiicient showed the average
performance difference between the two formats. At the student level, the average

coeficients estimated the average performance difference within schools due to the stakes

8

of the test, ethnic group, and gender as well as the impact of each of these factors on
format difi’erences. At the school level, the association between school percent-minority
and each of the factors at the lower levels (including the school intercept) was estimated.

The analysis described above used item estimates based on the high-stakes data. In
a further analysis, item parameters were re-estimated from the low-stakes data, to see how
well these estimates will fit the response patterns of the high-stakes students. Estimates
were made twice under low-stakes conditions, once treating omitted items as incorrect
and again treating omitted items as not—administered. Parameters were estimated
separately for males and females.

These item estimates from the pilot data were then used to re-estimate the abilities
of the high-stakes students. An estimate of ability was obtained for each student, using
gender-specific item estimates, once using the item estimates based on scoring omits as
zero, and again using the item estimates based on treating omits as not-administered. The
ability estimates were based on all items, with omitted items scored as wrong (regardless
of how omitted items were treated in the item estimation). Based on this ability estimate,
the fit of each student’s responses to the item parameters was calculated, using the
Drasgow, Levine, and Williams (1985) standardized appropriateness index for
polychotomous data. This index was used to flag students who had particularly poor fit.
The number of misfitting students was compared across item-estimation conditions and
genders. If there were more boys than girls who had poor fit, for example, it would mean
that the pilot item estimates were less accurate for boys. The information about which ‘
way of treating omits produced the most accurate estimates can be used by those who

make decisions of how to treat omits for item estimation for low-stakes tests.

9

The appropriateness index measures “person-fit”. “Item-fit” was also of interest,
because it is the item parameters which are used in equating and in some types of
Difi’erential Item Functioning (DIF) estimates. Generally, if person-fit is poor then item-fit
will also be poor, but it is possible for a few items to misfit without highly afi’ecting the
overall person-fit. To measure item-fit, the OUTFIT statistic (Wright & Masters, 1982)
was calculated. For this measure, the standardized difference between each person’s
observed and expected score (based on ability and item difiiculty) was found for each
item, and these standardized difi’erences were summed across peOple. Item difiiculty
estimates, again, were based on the low-stakes responses while the responses used in the
test of fit were the responses of the high-stakes students.

The average OUTFIT was calculated for each gender group, estimation condition
(method of treating omits), and response format (constructed response or multiple
choice). These means were compared to see if gender, way of treating omits, or response
format afl’ected the fit. Especially large values for individual items (1.5 or more-50%
greater than average) were also noted.

If there were a stakes by format performance interaction, the measures of fit would
tend to be worse for the items which showed the greater performance difi‘erence between
the two test conditions. If one gender group showed greater performance differences
when the stakes of the test changed, the fit could be worse for this group. If there were a
gender by stakes interaction in tendency to omit items, the fit would be particularly poor
when omitted items were treated as incorrect for the group which increased its response

rate more.

CHAPTER 2

REVIEW OF THE LITERATURE

How do the consequences of a test afi’ect student performance, and do the
consequences afl’ect students of difi’erent genders or ethnic groups difl‘erently? Do the
consequences have a greater impact on performance on constructed response or multiple
choice items? Previous research has touched on some of these issues.

Effects pf Consequengps pn Test Scores

Examinees generally score higher under high stakes conditions. An early example
of this phenomena was described by Rothe (1947). On each of four tests (two each for
laundry workers and machine-shop workers), the scores of job applicants (high stakes)
were higher than scores of current workers (low stakes). The differences ranged from
about 0.20 standard deviations to greater than one standard deviation. Rothe (1947) ruled
out several alternative explanations: the applicants were younger (so they would be
expected to score higher on certain tests, especially speeded tests), but two of the four
tests were unrelated to age; the laundry plant applicants were female so they were not test-
wise fi'om the military (as male job applicants were likely to be); the new applicants were
applying for the same types of jobs as the current workers (there was not an imbalance of
clerical v. shop workers); and there was not a recent drop in applications, which could '
indicate self-selection. Rothe concluded the applicants did better because they were more
motivated. Similarly, Jennings (1953) found that supervisors who thought their test scores

10

1 1
would be used for deciding promotions scored more than one standard deviation higher
than those who were told the testing was for research purposes.

Other examples are more recent and more relevant to educational testing. Wolf
and Smith (1995) administered two counterbalanced forms of a class exam to
undergraduates who were aware which test would be graded. On average, students
scored 0.26 standard deviations better (64% compared to 61%) on the consequential

exam, though some individuals (about 1/3) scored better on the nonconsequential form.
Similarly, Wolf, Smith, and DiPaulo (1996) found undergraduates scored an average of
0.44 standard deviations better under consequential conditions. Taylor and White (1981)
increased the consequences for Title I second-graders taldng the Stanford Achievement
Test by ofl‘ering the students money for improving their scores. This reinforcement led to
an increase of more than one standard deviation in reading scores.

Similar efi’ects have been found when stakes have increased in non-experimental
conditions. Burke (1991) demonstrated that scores on the NAEP in Louisiana and
Maryland improved when these states began using the test for high school graduation
(Burke also suggested increased curricular alignment due to this use of the test could play
a large role). However, when NAEP mathematics items were added to Georgia’s state
tests, scores did not improve on one of two forms and increased by 0.18 standard
deviations on the other, a difi’erence similar in magnitude to difi’erences found on other
NAEP tests due to context changes (Kiplinger & Linn, 1992). Though these tests had
some consequences at the school level, they had no consequences at the individual student
level so smaller changes would be expected (in fact, a focus group of 12th graders

recommended increasing motivation by not informing students the test had no individual

12

consequences). Even in studies where the group mean increased as the consequences
increase, not all individual student scores increased. In Jennings’ (1953) sample, the rank
order of supervisors tested under consequence and no-consequence conditions was
unstable; some of those with high scores under one condition had low scores under the
other condition. In Wolf and Smith's (1995) sample, about 1/3 of students scored higher
under no consequences.

Efi’ects of Consequengs pn Motivation

Why is performance higher on high-stakes tests? One explanation is that test
consequences increase motivation, which in turn increases performance. In the present
study, the measure of motivation is response rate, a somewhat muddy indicator because
non-response may be due to lack of knowledge rather than lack of motivation. Previous
findings about the effects of consequences on motivation are described in this section, and
the link between motivation and performance is the t0pic of the next section.

Motivation seems to be low on many low-stakes exams. . About 36% of a group of
8th graders who participated in the NAEP thought it was “not very important” or only
“somewhat important” to do well on the test (Kiplinger & Linn, 1992). Further, 28%
reported they had tried "not at all hard" or only "somewhat hard" on the test. In
preliminary studies, external consequences increased the scores of eighth graders, and the
information that schools or students would be compared had a greater efi‘ect than small
amounts of money. Twelfth-graders, in a focus group, also reported very low levels of
motivation (Kiplinger & Linn, 1992). Given these low levels of motivation, it might be
expected that some students would just fill-in bubbles on the answer sheet to complete the

test quickly and without efi’ort. F reund and Rock (1992) developed an algorithm to

 

13

identify students who were suspected of using some type of pattern to mark their answer
sheet (patterns would tend to lead to similar differences between the locations of any two
responses). Eighth and 12th graders who scored more than 1.5 standard deviations below
the eighth-grade mean for this measure were identified as “pattern markers” on the NAEP.
More males than females, and a greater proportion of Hispanics and Blacks than Whites,
met this criterion. F reund and Rock (1992) suggested it might be better to delete the
responses of pattern-markers before item estimation, because these responses would add
“unnecessary” variance.

Motivation has been found to be low on other tests as well. A group of 45 college
students whose scores decreased between their fieshmen and sophomore years on a low-
stakes test (intended to assess university goals) participated in interviews concerning the
exam (Olsen & Wilson, 1991). The students said they were not very serious about the test
and did not approach it like a “r ” classroom test. When asked how much various things
would motivate them, the students rated personal stakes (fi'ee electives for high scores,
additional courses if low scores, and even personal feedback on performance) as more
motivating than consequences for the university (government funding). This might
suggest that high school and middle school students (such as those in Kiplinger & Linn,
1992) would be unlikely to be motivated by school or district level stakes.

On the Michigan Educational Assessment Program (MEAP) reading tests, which
previously had few individual stakes, motivation decreased with grade level (Paris,
Lawton, & Turner, 1992; Paris, Turner, Lawton, & Roth, 1991). Students were asked if
they had read all the reading passages on the test and if they had checked their answers.

High school students were less likely than younger students to have done these things.

14

Greater proportions of high school students said they had just filled-in some of the
answers without trying to choose the correct answer. The survey also asked students how
much they cared about their scores, and how much they thought their parents and teachers
cared about their scores; high school students responded more negatively than younger
students. These findings were similar to the results from two broader surveys of students
in four states: Older students had more negative attitudes and engaged in poorer test-
taking behaviors.

These results show motivation is low on low-stakes tests, but they do not
necessarily imply that motivation is higher on high-stakes tests. Other studies, though,
have compared motivation under high and low (no) consequence conditions and have
found higher motivation when the test has individual consequences. Wolf, Smith, and
Birnbaum (1995) studied a test given in New Jersey to 10th graders to assess need for
individual remediation and to 11th graders for high school graduation. At the first
administration, though, few 11th graders were taking the test for graduation because they
had passed a previous graduation exam as 9th graders. The 10th graders rated their efi‘ort
on the test significantly higher than the 11th graders did, and 10th graders omitted fewer
items (though omit rates were low for both groups). Similarly, Wolf and Smith (1995)
and Wolf, Smith, and DiPaulo (1996) collected motivation scores for college students who
completed two exams, only one of which contributed to their course grades. In both
studies, the motivation scores were about 1.5 (1.45 and 1.58) standard deviations greater
for the test which was included in the course grade. Parallel results have been found on
personnel tests. Arvey, Strickland, Drauden, and Martin (1990) measured test motivation

for a job screening test for highway maintenance workers. Job applicants had higher

l 5
motivation scores (greater than one standard deviation) than current workers. In another
sample of applicants for a county financial position, they found Whites had higher
motivation scores than Blacks on the application test, and controlling for test attitudes
lowered the association between ethnic group and test performance.
Efi‘ects of Motivation on Performance

Motivation is relevant to testing issues because increased motivation tends to lead
to improved test performance. There have been at least two meta-analyses summarizing
this relationship (Multon, Brown, & Lent, 1991; Uguroglu & Walberg, 1979). Combining
findings fiom 232 samples, Uguroglo and Walberg (1979) found an average correlation
between motivation (including measures of self-concept and locus of control) and
achievement (both test scores and course grades) of .34. This association tended to be
higher for older students and for class grades compared to achievement tests (and lowest
for general ability tests), but it did not vary significantly by gender. Multon, Brown, and
Lent (1991) also found the correlation between feelings of self-efiicacy (their model
proposed that self-eficacy increased motivation) and test performance increased with
grade level and was higher on classroom and basic skills tests than on standardized
achievement tests. Their aVerage unbiased correlation, for 38 samples, was .38. Multon,
Brown, and Lent also synthesized 18 samples from studies of the relationship between
self-eficacy and the number of items answered on a test, finding an average correlation of
.48. In many of these studies, though, it is diflicult to know whether the more motivated
students did better because of high motivation/confidence, or if the higher
motivation/confidence was due to the students' knowledge of their own high abilities.

Also, Multon, Brown, and Lent did not show how highly related self-efiicacy and

 

l6

motivation were in the samples they synthesized; the link between self-eflicacy and
motivation was based on theory and results from other studies.

The relationship between performance and motivation may vary by gender.
Though the meta-analysis of Uguroglo and Walberg (1979) showed no average gender
efi‘ects on the correlation, Karmos and Karmos (1984) found the correlation was higher
for males than for females. Females, however, had more positive attitudes toward tests,
and this gender difi’erence was more apparent at higher grades (students from grades 6-9
participated in the study). The attitudes of males had a greater variance. However,
Brown and Walberg (1993) found no interaction between gender and motivation when
motivation was operationalized as an experimentally-manipulated variable rather than a
characteristic of students, and their sample included middle school students (as Karmos &
Karmos did) as well as elementary students. For both males and females, motivating
instructions (informing students their school would be compared to other schools and their
scores would be used in evaluating their teachers) increased achievement test scores an
average of 0.30 standard deviations.

If external motivation is high enough test-takers may report only small difi‘erences
in motivation, depressing the correlation between test scores and motivation scores.
Arvey, Strickland, Drauden, & Martin (1990) measured the test attitudes (including
motivation) of job applicants and current employees (highway workers) taking several job-
related tests. The variance in motivation scores was over five times as high in the
employee group compared to the applicant group. The correlations between motivation
scores and test scores were small but significant and consistent in the employee group (r =

.24, .25., and .23 on three tests). In the applicant group, where motivation scores were

_.——_-

17

uniformly high (though test scores were more variant), the correlations were not
significantly different from zero (-.01, .04, .10). In a sample of applicants in another
occupation (county financial workers), correlations of test scores with motivation scores
were small; the largest correlation, between a simulated work sample and motivation, was
.20. There were no comparisons with current employees with this classification, nor was
the variance on the motivation scale reported.

Some tasks may be more motivating because test-takers find them more
interesting. For example, computer adaptive tests are more interesting than paper-and-
pencil versions for many examinees. The computer adaptive version of the Armed
Services Vocational Aptitude Battery (ASVAB) was described as less boring, more
challenging, and more interesting by a sample who took both the computer and standard
versions (Arvey, Strickland, Drauden, & Martin, 1990). These test-takers also reported
working harder on the computer version (in efl‘ect, they were more motivated). A large
sample of recruits took both computer-adaptive and paper-pencil versions of the ASVAB,
as well as the Test Attitude Survey (TAS) after each version; these recruits had higher
average attitude scores associated with the computer version (Arvey, Strickland, Drauden,
& Martin, 1990). Another study of randomly equivalent groups of recruits taking the test
under no-stakes conditions showed that the equating of the computer and paper versions
of the tests did not generalize to operational conditions (Segall, 1997). Under no-stakcs
conditions, the computer adaptive test appeared easier, relative to the paper test, than it
did under operational conditions. Essentially, scores on the paper version increased more

as the stakes increased, changing the relationship between the formats.

18

Children tend to score higher on reading comprehension when the topics of the
reading passage interest them (Asher, 1979; Asher, 1980; Bernstein, 1955; Stevens,

1979). These findings hold whether the students’ interest is measured on the specific
passages after reading (Bernstein, 1955) or on the general topics of the passages in a
seemingly unrelated session at some point before the test (Asher, 1979; Asher, 1980;
Bernstein, 1955; Stevens, 1979). Findings regarding the interaction of gender and interest
on performance are rrrixed. Asher (1980) and Bernstein (195 5) found that testing students
on topics they were interested in increased the scores of boys more than girls. However,
there was no interaction between gender and interest in the studies of Asher (1979) and
Stevens (1979). Only one of these studies looked for a possible interaction between
ethnicity and interest; Asher (1979) showed that the scores of Blacks increased to the
same degree as the scores of Whites when students’ comprehension scores were compared
on topics they rated as high and low on interest.

An interesting finding in this area is that the efi’ects of interest are less when there
are some consequences for students. Asher (1980) found that scores on high interest
topics were not significantly higher than scores on low interest topics when students were
offered external incentives. This suggests that interest is more motivating in the absence
of other motivation and may not play a large role on high stakes tests where students are
already motivated to earn high scores. Scores on low-interest tasks or items, then, may
increase more from pilot to operational test than scores on high-interest tasks. This is

relevant to the present study if interest is associated with response format on the HSPT.

19

Response Format

Constructed response items are often viewed as performance tests, or as being
closer to authentic performance tests than multiple choice items. In writing, for example,
writing an essay is a sample of the desired performance (writing) to be measured. In
mathematics, the target performance might be problem-solving in ill-structured situations,
which could be operationalized as a written task where students explained their reasoning
in interpreting and solving the problem. In science, the desired performance might be
designing and conducting appropriate experiments; written responses in which students
explained how they would do this or interpreted results supplied in the test could be
considered a less expensive simulation. More authentic performance tests in science have
also been proposed for large-scale use: Gao, Shavelson, & Baxter (1994) and Shavelson,
Baxter, & Pine (1992) reported research on field tests of science assessments where
students designed and carried out scientific investigations.

Efi'ggts pf My; F amt pn Grppp Differences

Some proponents of constructed response tests suggest that ethnic and/or gender
difi’erences will be lower on constructed response tests. Messick (1994) explained that
pr0ponents of this position contend that there is “less construct under-representation and
construct-irrelevant method variance” in performance tests. If there were no group
differences on the construct, then there should be fewer differences on such tests (if the
claims for their validity were assumed). On a test for skilled metal-workers, for example,
there were no ethnic group difi‘erences in the scores on the actual products, but there were
fairly large difi’erences (over one standard deviation on the total score) on a paper-and-

pencil test covering the same machines (Schmidt, Greenthal, Bemer, Hunter, & Seaton,

20

1977). On a test of situational judgment in the workplace for blue-collar workers,
Black/White performance difi‘erences were smaller when the scenarios (prompts) were
displayed on videotape rather than described in writing (Chan & Schmitt, 1997).

However, academic performance tests, especially large-scale tests, often involve a
written product (even if there is some manipulation of materials), which might add
irrelevant variance if written communication were not part of the target construct. In the
Chan and Schmitt study (1997), controlling for the reading skills by testing method
interaction (level of reading skills had a larger impact on test scores for the written prompt
method) decreased the race by test method interaction. In most large-scale tests in
education, though, reading/writing skills are likely to be at least as influential in
“performance” tests as in multiple choice tests. Also, there might be larger group
difl‘erences (due perhaps to educational difi‘erences) on the types of constructs (such as
science or math knowledge) measured in educationnremoving “construct irrelevant”
variance would not change this.

One series of tests with school-level stakes, the Massachusetts Educational
Assessment Program, which included both multiple choice and written constructed
response items, did find smaller ethnic group difi’erences on constructed response items
(Badger, 1995). In eighth grade mathematics, the scaled scores of Hispanic and Black
students were higher on the constructed response items than the multiple choice items,
bringing them closer to the scores of White and Asian students, and patterns were
reportedly similar in other grades and subject areas. Many educators have emphasized

features which need attention if alternative assessments are to be equitable (Darling-

21

Hammond, 1995; Roeber, 1995; Winfield, 1995), but with no concrete evidence yet that
the resulting tests are more equitable than multiple choice tests.

However, other studies have found ethnic group difi‘erences on constructed
response tests are likely to be at least as large as group difi‘erences on multiple choice
tests. On science items field tested for the California Learning Assessment System
(CLAS), ethnic difi’erences (White/Black, Anglo/Hispanic) were relatively similar on
multiple-choice items and written constructed response items following hands-on
activities, especially among fifth and sixth graders (Klein, J ovanovic, Stecher, McCaffrey,
Shavelson, Haertel, Solano-Flores, and Comfort, 1997). In ninth grade, the White-Black
difi‘erence was 0.51 standard deviations on the constructed response section and 0.85 on
the multiple choice section; the White-Hispanic difl‘erence was more similar across formats
(0.61 and 0.73). On the California Bar Exam, a written section added in 1984 did not
decrease the difi’erences between the average scores of whites and minorities, and the rank
order of students was similar across the essay, performance, and multiple choice sections
(F einberg, 1990). Difl‘erences between Blacks and Whites on the 1988 NAEP writing
tests (which were essay tests) were of about the same size, in standard deviation units, as
difi‘erences on the reading tests (primarily, though not entirely, multiple choice) (Linn,
Baker, & Dunbar, 1991). Also on the NAEP, Bond (1995) reported that differences
between the scores of Blacks and Whites were greater on extended response items than on
multiple choice items. On performance tests developed in Great Britain, gender and ethnic
group differences increased on the constructed response items (Nuttall & Goldstein, 1990,

cited in Shepard, 1993).

 

22

Gender differences on the Advanced Placement exams (high stakes) depend on
response format. In most science areas (Biology, Chemistry, Physics B, Physics C-E&M,
Physics C-Mech) the gender difi‘erence (favoring males) is smaller on the constructed
response items (Bridgeman, 1989; Bridgeman & Lewis, 1994; Schmitt, Mazzeo, &
Bleistein, 1991). This pattern of gender differences was found in all ethnic groups tested
in large numbers in biology (White, Asian American, and Black) and chemistry (White and
Asian American). Bolger and Kellaghan (1990) found the same pattern with Irish high
school students. Klein, Jovanovic, Stecher, McCafi’ey, Shavelson, Haertel, Solano-
Flores, and Comfort (1997) found fifth and sixth grade girls scored significantly higher
than boys on written constructed responses to hands-on science activities, while gender
difi’erences were nearly zero on a multiple choice science test (ITBS). By ninth grade,
boys scored significantly higher on a multiple choice science test, while girls scored
somewhat higher on the hands-on constructed-response test. On the pilot (low stakes)
administration of the Michigan High School Proficiency Test, DeMars (in press) found a
similar pattern in mathematics and science (gender difi‘erences favoring males were greater
on the multiple choice section), but gender difi‘erences in most of the ability range were
small. Also, gender difi’erences were relatively stable across format in the calculus
(Calculus AB and Calculus BC) and computer science AP Exams (Schmitt, Mazzeo, &
Bleistein, 1991).

The NAEP is a low-stakes test, and Freund and Rock (1992) found higher rates of
suspected “pattem-mar ' g” by Blacks and Hispanics on the multiple choice section. This
suggests that minorities have especially low motivation on low stakes tests, and if this

afi‘ects constructed response items more than multiple choice items, the scores of

23
minorities may increase relatively more on constructed response items than multiple choice
items under high stakes conditions where students are motivated on both types of items.
Because constructed response items take longer to complete than multiple choice

items, fewer tasks are sampled. Linn, Baker, and Dunbar (1991) raised the possibility that
this would “make it harder to achieve balance with regard to group differences in prior
knowledge” (p. 18). When many items are utilized, group difl‘erences are more likely to
cancel out (assuming the item-specific differences are incidental to the primary construct). r
Feinberg (1990) explained: “Compared to multiple-choice tests of similar length, written I:
exams more arbitrarily emphasize one topic or another with which a student may (or may
not) be familiar” (p. 30). Similarly, Messick (1994) explained that “contextualizin ” items
(a purported advantage of using fewer, longer items, including constructed response
items) will afi’ect individual students differently, depending on their familiarity with the
context. He noted:

We should not take it for granted that a richly contextualized assessment task is

uniformly good for all students . . . contextual features that engage and motivate one

student and facilitate his or her efl‘ective task performance may alienate and confirse
another student and bias or distort task performance. (p. 19)

While constructed response items do not necessarily involve greater contextualization,
given the greater time devoted to an individual constructed response item compared to a
multiple choice item, there is greater opportunity for contextualization and more
“authentic” tasks.

Shepard (1993) also pointed out that assessments more closely aligned with
recommended curriculum may result in greater ethnic differences than more general

assessments, if minority groups have less access to the assessed curriculum. Johnson

24

(1995) voiced similar concerns. Messick (1994) recommended studying the possibility of
such curricular and instructional difi‘erences. Variance in time allocated to each content
area can be large, even within a sample of low-SES schools (Winfield, 1995), so it is
diflicult to generalize about opportunity to learn for groups of students. Minority students
are over-represented in the lower tracks of school, where some say they are more likely to
“experience instruction geared only to multiple-choice tests” (Darling-Hammond, 1995, p.
96) presumably, fi'om the additional context, meaning multiple choice tests which measure

lower-level skills. Dreeban and Garnoran (1986) showed that controlling for time spent

 

on reading instruction and individual aptitude reduced (essentially to zero) the effects of
race and SES on first grade reading achievement, concluding that students in primarily
Black schools (most of the schools studied were racially homogeneous) received inferior
instruction. In Massachusetts, teachers in low-advantaged schools were less likely to
think their students were prepared to answer items involving judgments, inferences,
scientific procedures or mathematical reasoning (which were emphasized in constructed
response items on the state tests), though their confidence in their students’ preparation
for factual or computational tasks was equivalent to the teachers’ confidence in high
advantaged schools (Badger, 1995). Similarly, students in low-advantaged schools were
more likely to agree that “learning is mainly memorizing”. The low-advantaged schools
spent less money on science materials, and the math teachers in these schools were less
likely to feel they had adequate equipment (calculators, computers, manipulatives). These
results suggest curricular difi‘erences favoring the advantaged students. Therefore, it is '
surprising that minority and disadvantaged students did relatively better on the

constructed response items, which purportedly measured more higher-level skills than the

25

multiple choice items (the scores of minority students were lower on both types of items,
but the gap was somewhat smaller on the constructed response items).
Effects pf Resmnse Format on Motivation and Performance

One manifestation of low motivation is nonresponse. Conceptually, nonresponse
seems to be more likely on constructed response items than on multiple choice items. It
takes very little efi‘ort to mark an answer on a bubble grid; it is not even necessary to read
the item. However, writing an answer (even one unrelated to the question) takes some
thought and time. Nonresponse, though, could also signify the student knows nothing
about the question. Ifa student has a very low level of knowledge, the student could
guess randomly on the multiple choice items but not the constructed response items. The
response pattern would be similar to that of an unmotivated student: random responses on
multiple choice items and nonresponse on constructed response items. This pattern, then,
while suggesting low motivation, is not a sure indication of lack of motivation.

Nonresponse or irrelevant response is common on low-stakes constructed
response tests. The Massachusetts Educational Assessment Program, for example, has no
student-level stakes. In one sample of students taking the mathematics portion, where
students were asked to generate solutions to relatively ill-structured problems, many of the
8th and 12th graders simply left the items blank (Badger, 1989). The highest nonresponse
rates were for an item which asked students to explain how adding a constant to each
number in a set would affect the average (students were given concrete numbers to work
with), and for an item which required students to explain how they would estimate the
product of two numbers (a specific pair of numbers was supplied for students). Over 20%

of the students left each of these items completely blank, not even supplying a numerical

26

answer (numerical answers with no explanations or irrelevant explanations were also
common). On other constructed response items, non-response rates were about 8-15%.

On science constructed response items (completed by another sample of students
at the same time), nonresponse rates (including irrelevant responses) for 8th and 12th
graders ranged from 5% to 30% (Badger & Thomas, 1989b). A physics content item
(“explain how a fuse works in a circuit”) was most frequently left blank (30%). This
might suggest students were omitting items because they lacked the specific content
knowledge needed, but two process-type items also had nonresponse rates of nearly 20%.

In social studies (Thomas, 1989), response rates varied widely, with from 4% to
73% of 8th and 12th graders omitting items or supplying irrelevant answers. In reading
(Badger & Thomas, 1989a), percentages of students giving no answer were reported
separately from those giving irrelevant answers. On three 8th grade items, 6-10% of the
students gave no answer, and another 6%-25% gave irrelevant answers. On one 12th
grade item, 6% left the item blank and 12% gave irrelevant answers; on another item, 3%
left it blank and 7% supplied an irrelevant answer. As reading generally demands less
specific content knowledge than other subjects, such responses are more logically related
to low motivation.

Nonresponse rates are also higher on constructed response items than on multiple
choice items on the NAEP (Freund & Rock, 1992). Wainer (1993) citing Bock (1991)
discussed how students taking the California Assessment appeared to be more motivated
on the multiple choice items than on the constructed response items.

Response rates were also fairly low in a sample of students who participated in the

second follow-up of the National Education Longitudinal Study of 1988 (NELS-88)

27
(Gerber, 1996). In the open-ended mathematics section, students left an average of 3 .06
of 17 subitems blank. This research also examined the effects of gender and ethnicity. On
some items, there was almost no gender difference in likelihood of responding to all parts
of the item; on other items girls were about 2/3 as likely to complete all parts. Afiican-
American and Hispanic students were less than half as likely to complete all parts of an
item as nonminority students were. However, when students’ scores on the multiple
choice section and their perceptions of the difiiculty of previous items were controlled,
there were no gender or ethnicity effects. The perceived difficulty of previous items (a
motivational factor) had a greater influence on minorities than nonminorities.

Again, in all these situations, it is difiicult to know whether students know so little
about the task they can give no related response or if they simply are unmotivated,
especially if there is no external measure of ability. For example, Gerber (1996) found that
students who scored high on the multiple choice section were more likely to complete all
constructed response items. Part of this could be due to the increased sense of self-
eficacy/motivation in high-achieving students, but at least some of it is surely due to
increased knowledge about the answer.

Nonresponse is only one measure of motivation. Student responses on a test
motivation or attitude scale might be less confounded with performance (though
performance can obviously influence these measures as well). In a sample of college
students taking a required low-stakes test, those who took a traditional multiple choice
test had higher scores on the Student Motivation Questionnaire (Sundre, 1996) than
students who took an essay test. The students taking the traditional test had motivation

T-scores of 50.6 compared to 46.5 (a difference of about 0.40 standard deviations).

28

However, the results were confounded by differences in subject areas tested as well as the
accompanying tests (the students who wrote the essay wrote in a language arts context
and also responded to a multimedia Fine Arts test, while the students who took the
traditional multiple choice test responded in the context of natural and social sciences).
Observational reports of task engagement provide another measure of motivation.
Constructed response tests involving active manipulation of materials may be more
motivating than constructed response tests where all tasks are completed on paper. The
Massachusetts Educational Assessment Program used some active mathematics and
science tasks in 1989 (Badger, Thomas, & McCormack, 1990). For these tasks, students
worked in pairs to use materials to solve problems. Observers reported about 80-95%
(depending on the task) of the eighth graders were engaged throughout the task. About
70-85% of the students seemed enthusiastic; almost all others were rated neutral, not low,
on enthusiasm. Enthusiasm was lowest for the least-structured tasks. On one task,
observers also recorded how carefully the students worked and how concerned they were
with the accuracy of their response. Though most students were actively engaged, 23%
appeared to have no concern for accuracy. Interesting tasks may motivate students to
participate in a test, but this is not always the same as motivating them to do well.
Test-takers’ perceptions of validity are likely to be related to motivation; students
may try harder when the task seems meaningful. Chan and Schmitt (1997) reported that
Black students rated the face validity of a videotaped situational judgment test to be higher
than a written version, while White students saw little difference between the face
validities of the two formats. Adding face validity to the model for test performance

decreased the race by method interaction (without controlling face validity or the reading

29

skills by test method interaction, the race differences were much greater under the written
format).

Attributions and expectancies are associated with motivation and performance
(Curran & Harich, 1993; Gerber, 1996; Multon, Brown, & Lent, 1991; Uguroglu &
Walberg, 1979; Wolf, Smith, & DiPaolo, 1996). Attributions may be difi'erentially
associated with performance depending on response format. Chandler and Spies (1981)
found that expectancies of success were more highly correlated with attributions of ability
on objective tests than they were on essay tests (students with higher expectancies were
more likely than students with low expectancies to think their performance was due to
their own abilities on multiple choice tests). Correlations of expectancies with attributions
of mood, luck, and help from others were higher on essay tests than objective tests
(though they were still lower than correlations of expectancies with ability on both
formats). Apparently, students who expected to do well on multiple choice tests were
more likely than students who did not expect to do well to think that their performance
was due to their own abilities, especially on multiple choice tests. On essay tests, the
relationship between expectancy and ability was lowered and the relationship between
expectancy and mood, luck, and external help was increased. Students who are confident
in their abilities, then, may be more motivated on multiple choice tests, where they believe
there is a clearer relationship between performance and ability.

One aspect difi‘erentiating multiple choice and constructed response items may be
the cognitive level of the items. Though both response formats can address all cognitive
levels (from basic factual knowledge to complex synthesis and evaluation), proponents of

constructed response items tend to claim they are more appropriate for testing higher-level

30

cognitive skills. In studies of constructed response and multiple choice items written to
assess the same cognitive level, Crooks’ (1988) review of the literature found few
performance differences. Further, in studies where students were told the response format
of the exam but were not given concrete examples of items, there were only small
difi‘erences in test performance and study behavior. It was only when the students were
provided with concrete examples of the types of items they would encounter that there
were difierences in performance. This summary suggests that it is the students’ study
behavior in anticipation for the cognitive level of the items, rather than the items
themselves, that influences performance differences by response format. However,
studying implies high-stakes tests of limited content, primarily classroom tests.

Another difference characterizing constructed response items is that they may be
perceived by students as more “work” than multiple choice items. Even if they require the
same thought processes, constructed response items require students to physically write
the answer (instead of filling-in a bubble) and they often require students to choose words
to explain their reasoning. \Vrthin the multiple-choice format, Wolf, Smith, and Birnbaum
(1995) showed that motivation is afi‘ected by the degree to which an item is perceived as
"mentally taxing". Educators rated items based on how much “mental energy” they would
require fi'om students apart fi'om the “difiiculty” (in terms of how many students might get
the answer correctly). For example, a long division problem would not be diflicult for
high school students, but it would be mentally taxing. One group of students taking this
test faced consequences (10th graders who would be targeted for remedial work if their
scores were too low) and another group did not (11th graders who were taking the test

essentially as a pilot/norming test because it would be used as a graduation exam for

31

future cohorts). A DIF index (comparing the 10th and 11th graders) was computed for
each item. This index was correlated with difficulty and taxation; the more taxing or less
difficult (meaning the item had a higher p value) the item, the greater the DIF favoring
10th graders (the correlations were -.39 with dimculty and -.55 with taxation, once a
single outlier was omitted). This suggests that students in high stakes conditions are more
likely to work carefirlly on easy items (where students in low stakes conditions could be
bored and careless) and on mentally taxing items (where low stakes students might not
want to extend the necessary efi’ort).

If constructed response items are perceived as more taxing (and even if they
require no more efi‘ort to solve the problem, they would seem to require more effort to
write the answer and explanation), the situation may be similar for constructed response
compared to multiple choice items. Under low stakes conditions students may not try as
hard on the constructed response items, making them appear differentially harder under
low stakes.

Efi‘fis pf Anxim pn Perfprmgpe

Ifmotivation is the primary difl‘erence between performance on high stakes and
low stakes test, it would seem that scores would improve on high stakes tests. However,
some have suggested that the relationship between arousal and performance is an
‘inverted-u’; performance is highest at moderate level of arousal. Arousal may be
motivating up to a point, where it becomes anxiety-inducing. The Yerkes-Dodson Law
describes this relationship: “There exists an optimal level of arousal for performance of I
any given task. Levels of arousal above and below this optimal level will be associated

with relatively lower performance” (Smith & Smoll, 1990, p. 43 7). However, others

32

contend this relationship may be due to measures of arousal which mix the constructs of
arousal and stress. Using instruments designed to measure these constructs separately,
Stanley, King, and Glass (1989) found a small negative correlation between stress and
mathematics test performance and a small positive correlation between arousal and test
performance, with no evidence of a quadratic trend between arousal and performance.

Many have focused not on the more general construct of arousal but on the
specific construct of anxiety (the component of arousal hypothesized to decrease
performance). In a meta-analysis, Hembree (1988) concluded that anxiety scores were
higher in high-stress, ego-involving testing situations, which would include high stakes
exams. The correlation between anxiety and test scores tends to be greater (more
negative) in studies of standardized tests than in studies of classroom tests (Crooks,
1988); possibly other factors (motivation, preparation) play a bigger role and override
some of the efi‘ects of anxiety on classroom tests.

In generaL the test scores of students who have higher anxiety scores are lower
(Crooks, 1988; Hembree, 1988; Wolf& Smith, 1995; Wolf, Smith, & DiPaolo, 1996).
This could be because anxiety interferes with cognitive processing during the test or
because anxiety interferes with learning (then the low test scores would be an accurate
reflection of how much the students learned). Another possibility is that anxiety is caused
by the students’ awareness of lack of ability. Tobias (1985), after reviewing the literature,
concluded at least some part of the relationship between anxiety and test scores was due
to the effects of anxiety during the test, because the relationship between anxiety and

performance is stronger under “ego-involving” testing situations.

33

More evidence that anxiety interferes with test-taking is shown in the work of Hill
(1980) and Plass and Hill (1986). In these studies, conditions were manipulated to reduce
test anxiety When time limits for completing a standardized test were relaxed, the test
scores of middle school students who had moderate or high levels of anxiety (with anxiety
measured as a trait, not specific to a given test administration) increased compared to the
standard timed condition (Hill, 1980). When students were warned that some items would
be dificult and they should not worry about missing some items, scores did not improve
for any group and the scores of the low anxious children decreased. These instructions
might have decreased motivation as much as they decreased anxiety. The most highly
anxious group, however, did have higher scores under a combined more time, lower
expectation condition. Plass and Hill (1986) included gender in the design of a similar
study of the effects of timing/not timing third and fourth graders on a standardized
mathematics test. The highly and moderately anxious boys did better when they were not
timed; their scores in this condition were not significantly difi’erent fi'om the scores of the
low-anxious boys. Low anxious boys and high anxious girls, however, did somewhat
better when they were timed. The authors speculated that the highly anxious girls may
have disliked taking mathematics tests so much that a test of unlimited duration was even
more aversive than a test they were unable to complete in the allotted time.

Hembree (1988) concluded in a meta-analysis that Hispanic high school students
had higher anxiety levels than Black or White high school students. Results varied among
younger students; by upper elementary school Hispanic students had higher levels of
anxiety, and at some grade levels Blacks were more anxious than Whites. In Hill’s (1980)

sample of fourth through eighth graders, Hispanic students had higher levels of test

34

anxiety than Black students, who had higher levels than White students. Not only did the
levels of anxiety vary by ethnicity, but the correlation between anxiety and performance
difi‘ered somewhat as well. This correlation was -.36 for Whites, -.41 for Blacks, and -.45
and -.51 for Hispanic students in two types of bilingual or ESL classes (the correlation
was even more negative, -.93, for Hispanic students completely in mainstream classes, but
there were only eight students in this group).

Studies which have manipulated the consequences of the exam have shown no
evidence of a moderating effect of consequences on the relationship between anxiety and
performance. In Wolf and Smith’s (1995) study of students who took one form of a test
which would affect their course grade and another form which would not, anxiety was
equally associated with test scores regardless of the consequences (correlations of -.28
and -.29). In this study, anxiety was treated as a stable characteristic, measured five days
prior to the tests, rather than an efi'ect of the test consequences. Wolf, Smith, and DiPaulo
(1996) found that students were more anxious in the consequential condition (the groups
differed by 0.46 standard deviations), but anxiety had a significant efl‘ect on performance
under both conditions.

Though anxiety has been found to be associated with lower test scores, and higher
consequences would seem to provoke some anxiety, performance has generally been
shown to increase with higher consequences, on average (see “Efi‘ect of Consequences on
Test Scores”, this chapter). For most students, the motivation of higher stakes seems to
override any increased anxiety. Anxiety is not measured in the present study; the literature
on anxiety was briefly described here to acknowledge the possible negative efi‘ects of

anxiety (logically linked to test consequences) on the scores of some students.

35

Summag of the Literature
Average scores are generally higher on high stakes tests (Burke, 1991; Taylor &

White, 1981; Wolf& Smith, 1995; Wolf, Smith, & DiPaulo, 1996). One reason for this
could be motivation. Motivation tends to be low on low stakes tests such as the NAEP
and standardized tests with few individual stakes (Kiplinger & Linn, 1992; Paris, Lawton,
& Turner, 1992; Paris, Turner, Lawton, & Roth, 1991), and it is lower under non-
consequential conditions than consequential conditions (Arvey, Strickland, Drauden, &
Martin, 1990; Wolf& Smith, 1995; Wolf, Smith, & Birnbaum, 1995; Wolf, Smith, &
DiPaulo, 1996). If“pattem-marking” (marldng bubbles in a set pattern) is indicative of
very low motivation, extremely low motivation is more common among boys than girls
and among minorities than non-minorities on low stakes tests (F reund & Rock, 1992).

In turn, motivation is positively associated with performance (Multon, Brown, &
Lent, 1991; Uguroglu & Walberg, 1979), though one study of adult job applicants found
this association only for one of three tests (Arvey, Strickland, Drauden, & Martin, 1990).
Some have found the relationship to be higher for boys than for girls (Karmos & Karmos,
1984), but Brown and Walberg (1993) found no interaction between motivation and
gender, and across many studies Uguroglo and Walberg (1979) found no average
difi‘erences in the correlation by gender.

Interest may be a factor in motivation (Arvey, Strickland, Drauden, & Martin,
1990), and interest increases reading comprehension scores (Asher, 197 9; Asher, 1980;
Bernstein, 1955; Stevens, 1979). There is no evidence that the association between
interest and test scores varies by ethnicity (Asher, 1980). The findings of the efl‘ects of

gender on the correlation are mixed, with some studies showing a stronger relationship for

36

boys (Asher, 1980; Bernstein, 1955) and others showing no gender differences (Asher,
1979; Stevens, 1979).

The response format of items may influence the degree of group differences on a
test. Results are mixed, with most studies finding that adverse impact related to minority
group status seems to be as large or larger on constructed response items as on multiple
choice items (Bond, 1995; Feinberg, 1990; Linn, Baker, & Dunbar, 1991), though on one
series of tests (Badger, 1995) the relative position of minorities increased on the
constructed response items (minority students scored lower on both item types, but the
difi‘erence was smaller on constructed response items). One reason for group differences
on constructed response items is that the smaller number of independent tasks on
constructed response items, due to the longer time involved and the often greater
contextualization provided, allow for a smaller variety of topics (F einberg, 1990; Linn,
Baker, & Dunbar, 1991; Messick, 1994). Group difl'erences in interests or experiences
have less chance to balance out when there are fewer topics.

If some of the adverse impact on low stakes tests is due to especially low
motivation of minority students (suggested by Freund and Rock’s (1992) findings of
greater “pattern-mar ' g” by minorities), and if motivation on high stakes tests is more
uniform across ethnic groups and response formats, it would be expected that
performance would increase more for groups (and items) for which motivation was lower
on the low consequence test (the smaller motivation gap under high stakes would lead to a
smaller performance gap). If a topic held less interest for one group (but the students had
adequate background knowledge to respond appropriately when motivated), that group

would be expected to show greater increases in performance when the stakes were

37

increased. On the other hand, if most of the adverse impact were due to lower topic
knowledge, greater motivation would not have much impact on the scores of the group
with low topic knowledge. The curricula delivered in disadvantaged schools tends to be
of lower quality and quantity (Badger, 1995; Darling-Hammond, 1995; Dreeben &
Gamoran, 1986), so disadvantaged students may have lower topic knowledge and
experience.

Motivation and performance may be influenced by item response format.
Nonresponse is one indication of low motivation, and nonresponse tends to be high on low
stakes constructed response items (Badger, 1989; Freund & Rock, 1992; Gerber, 1996;
Wainer, 1993). In one sample, nonresponse on constructed response items was higher
among minority students, though there was no ethnicity effect when performance on
multiple choice items was controlled (Gerber, 1996). Students taking a low stakes essay
test had lower motivation than students taldng a multiple choice test (Sundre, 1996).
Constructed response items may be perceived as more “work”, and Wolf, Smith, and
Birnbaum (1995) showed that scores on “mentally taxing items” were afi‘ected more by
the consequences of the test.

Anxiety is negatively correlated with test scores (Crooks, 1988; Hembree, 1988;
Wolf & Smith, 1995; Wolf, Smith, & DiPaolo, 1996), and high stakes tests increase
anxiety (Hembree, 1988; Wolf, Smith, & DiPaulo, 1996). Highly anxious students
generally perform better under less stressful conditions (Hill, 1980; Plass & Hill, 1986).
Hispanic students have greater test anxiety (Hembree, 1988; Hill, 1980), and the
relationship between anxiety and performance is at least as strong (somewhat stronger) for

minorities (Hill, 1980). The consequences of the test, though, do not seem to change the

38
relationship between anxiety and test scores (Wolf & Smith, 1995; Wolf, Smith, &
DiPaulo, 1996). Though anxiety levels are higher on high stakes tests, given the findings
described earlier on test consequences and performance, anxiety apparently does not
increase enough to ofl‘set the positive efi‘ects of increased consequences.

As noted, the present study did not measure motivation or anxiety. Rather, these
constructs were described as a possible explanation of why test performance increases on
tests which have greater consequences, and why students might be less likely to leave
items blank on a test with higher consequences. The literature summarized above did not
specifically study how item response format affects the degree of change in performance.
Most of the work on test consequences used only multiple choice items (Burke, 1991;
Freund & Rock, 1992; Taylor & White, 1981, Wolf & Smith, 1995; Wolf; Smith, &
Birnbaum, 1995; Wolf, Smith, & DiPaulo, 1996). Some researchers observed that
motivation was particularly low on constructed response items without presenting
evidence that this had a larger performance impact for constructed response items than for
multiple choice items (Badger, 1989; Badger & Thomas, 1989a; Badger & Thomas,
1989b; Freund & Rock, 1992; Thomas, 1989). While F reund and Rock (1992) showed
that Blacks and males were particularly unmotivated on a low-stakes test, they did not test
students under more consequential conditions to see if motivation (or performance)
changed differently for Blacks and males compared to Whites and females.

In the present study, the following questions were addressed:

(1) What efi‘ects do test consequences, gender, and ethnicity have on response rate?
Response rate is one indication of motivation, and results related to this question

contributed information about whether or not the listed factors were associated with

39

motivation. Ifthe results were similar to the results related to test scores (see question 2),

then motivation would be a likely explanation for the findings about test performance.

(2) What efi‘ects do test consequences, response format, gender, and ethnicity have on
test scores?

As noted, previous literature showed that higher consequences increased test scores.

Results from this question added information about possible interactions between

consequences and the other factors.

(3) How well do item parameter estimates based on pilot data fit the operational data, and
how do pilot estimates based on treating omitted items as not-administered compare
to estimates based on treating omitted items as incorrect?

Significant interactions found for the first two questions would lead to difl'erences in item

fit. Results fi'om this question were used to assess how much of an impact differences in

response rate and test performance had on item fit.

CHAPTER 3

METHOD

Participants

Students participated in either the spring-semester 1996 or spring-semester 1997
Michigan High School Proficiency Test (HSPT), or the piloting of the test forms later
administered at these testing times. For science, both the 1996 and 1997 tests counted
towards diploma endorsements; the first year (1996) is reported in detail, and the second
year (1997) is described in the Appendix. All information in Chapters 3 and 4 concerns
the 1996 form unless reference is specifically made to the second science cohort. In math,
the 1996 test did not affect the diploma endorsement for most students, so only the 1997
form was analyzed.

In science, in the non-consequential (low stakes) test condition there were 512
White females, 89 Black females, 487 White males, and 59 Black males. In math, there
were 579 White females, 69 Black females, 548 White males, and 62 Black males. Blacks
were the only ethnic group used in these analyses because there were insuficient numbers
of other minority groups. The students in the science sample attended 28 difi‘erent
schools; approximately 1/3 of the 11th grade students in each of these schools took this
form of the pilot test (a 29th school was originally in the pilot sample, but data from this
school was discarded because none of the high-stakes students in this school indicated
their ethnicity). In math, the students attended 32 schools. Schools were randomly

40

41

selected to participate (from all population/urbanicity strata within the state), and students
within schools were randomly assigned to test forms (two additional forms, to be used in
other testing periods, were administered to the other 11th graders in these schools).

Participants in the consequential (high stakes) test condition were students who
were tested during the regular spring-semester administration of the HSPT. Only the
results from the schools which participated in the pilot-testing of this test form were used
in this analysis. In these schools, all students who indicated their ethnic/racial group was
either “White” or “Black” were selected for analysis. In science there were 198 Black
females, 1272 White females, 149 Black males, and 1255 White males. In math there
were 206 Black females, 935 White females, 148 Black males, and 954 White males.

The students in the selected schools were similar to the state population of 11th
graders, in terms of their average scores on the high stakes test and their ethnic
background. In the population of 11th graders tested at the 1996 administration (science),
about 22% of those who indicated their ethnicity identified themselves as non-white,
compared to 21% in the selected schools (though 26% of the students did not respond to
this question). In the population of 11th graders tested at the 1997 administration (math),
about 25% of those who indicated their ethnicity identified themselves as non-white,
compared to 22% in the selected schools. However, 41% of the students in the selected
schools did not identify their ethnicity, compared to 30% of all tested students. In schools
where less than 75% of the students indicated their ethnic group, the school proportion
minority was based on the pilot data, where there was greater information.

Limiting the sample to Blacks and Whites who indicated their gender, the mean

science scale score was 383.7 (SD = 37.9) in the larger group and 385.9 (SD = 37.5) in

42

the selected schools. In math, the mean scale score was 401.00 (SD = 50.3) in the larger
group and 404.04 (SD = 51.4) in the selected schools. Including students of all ethnic
groups, students who did not indicate their ethnicity scored slightly lower than students
who did, but the difference was similar across gender. In math, the girls who supplied
ethnic information averaged 2.5 points higher than those who did not; the difi‘erence was
3 .2 for boys. In science, the difference was 6.2 for girls and 8.9 for boys.

In both groups, students who responded to less than half the multiple choice items
were excluded fi'om the analyses. In science, five low-stakes students and one high-stakes
student were excluded for this reason and were not included in the numbers above. These
students answered several items at the beginning, but then appeared to quit early in the
test. In math, 10 low-stakes students and 16 high-stakes students were excluded for this
reason. Under high-stakes, all these students responded to items on only one of the two
days of testing; under low-stakes, some of these students responded to only a few items.

Instrumpn_t

The Michigan High School Proficiency Test (HSPT) had four components during
the years studied here: mathematics, science, reading, and writing. The tests were not
designed as minimum competency exams; they were intended to reflect high school level
(through the end of the sophomore year) skills. Beginning with the graduating class of
1997, students who scored in the proficient category received endorsement seals on their
diplomas at graduation. Separate endorsements could be earned in mathematics, science,
and communication arts. Students initially took the tests in their junior year of high
school, and students who did not earn endorsement seals had opportunities for retaking

the tests.

43

The HSPT was first administered as an operational test (leading to diploma
endorsements) in the spring of 1996. The science section administered during this testing
period counted for diploma endorsement. The mathematics section administered in 1996
was not high-stakes for most students because this cohort had had an opportunity to earn
diploma endorsements in mathematics (and reading) the previous year based on a
discontinued statewide examination of 10th graders. The mathematics form analyzed here
was the form administered in the spring of 1997, the first year the mathematics test was
widely used for diploma endorsement.

Each form of the science section of the HSPT consists of 42 multiple choice items
and eight constructed response items. Only 34 of the multiple choice items were analyzed
here because eight of the items were different on the pilot test (for equating purposes).
The maximum point value of the constructed response items varies across forms (a
difi‘erent form is developed for each administration), but on this test form, each
constructed response item was worth two points, for a total of 16 points.

Two of the constructed response items are related to a scientific investigation
described in the test, and two are responses to a text passage related to science; the other
four items involve applications of science knowledge or methods. These items could be
considered intermediate on the “performance” continuum; the students do not actually
carry out the investigations they design or expand, and they interpret or explain supplied
results or described phenomena rather than those they have generated or observed (or the
students may be asked to create a pattern of results or a scenario that would support a
hypothesis). Some items require more content knowledge and others require more

procedural knowledge, as is true for the multiple choice items as well.

44
The two investigation items were highly related, both conceptually and statistically.
The correlation between these two items was .60, compared to correlations of .19-.33 for
all other pairs of constructed response items. Therefore, these items were treated as a
single item worth four points (nine categories when 1/2 points were considered).
Efi’ectively, then, there were seven constructed response items. The two constructed
response items based on the same text passage, as well as several clusters of multiple
choice items (each cluster was written to relate to a common scenario or graphic) might
also be logically viewed as more interdependent than items on the test as a whole. To
check for this possibility, Yen’s Q3 (Yen, 1993), was calculated for all item pairs, using
operational data. For this statistic, an expected score is calculated for each student on
each item, based on an item response function (Yen developed the statistic for the three-
parameter model, but here the one-parameter model was used, as explained in the analysis
section). The residual between each student’s observed item score and his/her predicted
item score is then found, and these residuals are correlated for pairs of items. Yen (1993)
suggested a cutofi‘ of .20 in deciding whether the assumption of local independence was
violated to an extent which would make a practical difierence. After combining the two
investigation items, only two pairs of items met this criteria for the HSPT science test.
One pair of multiple choice items had a residual correlation of .23; these items were
treated as a set (scored 0-2) in item and ability estimation. Another pair, with one multiple
choice and one constructed response item (adjacent and on the same theme), had a
residual correlation of .21. These items were left separate because later analyses required
separate scores on the multiple choice and constructed response sections. Two other pairs

had residual correlations between .10 and .20. One was another mixed-format pair, which

45

was left as two separate items, and the other was a multiple choice pair, which was
combined into a single item (scored 0-2). All other correlations were less than .10.

The math section of the HSPT consists of 40 multiple choice items and 6
constructed response items (worth 2-5 points each). Only 32 of the multiple choice items
were analyzed here because 8 items were not consistent from pilot to operational test. As
in science, the constructed response items involve written responses, which may include a
drawing or figure and usually require words to be answered completely. In these items,
students are expected to explain their solutions and reasoning processes.

There was no reason to expect to find sizable residual correlations on the math
section, but for consistency with the science analyses the local item independence
assumption was checked with Q3. Again, the operational data were used for the item
calibration, and all items were estimated together. One pair of constructed response items
was on the edge of Yen's (1993) suggested limit of .20, but I did not combine them
because there were already so few constructed response iterrrs. There were four multiple
choice items with intercorrelated residuals: five of the correlations were greater than .20,
and the sixth was greater than .10. These items covered three difi‘erent content areas, and
only two were adjacent; there was no conceptual reason to expect them to be related. I
summed these four items and treated them as one item with five ordered score categories.

For the pilot test, the students’ answers to the constructed response items were
read by two raters, and if the scores were more than one point apart an additional rater
scored the response. The item score was the average of these two (or three) ratings (if '
three raters were used, the score was rounded to the nearest .5 for the parameter

estimation analyses in this study; rounding was necessary for less than 1% of the pilot

46
students). In the operational test, if a third rater was needed, a more highly skilled rater
was used and the rating of this “expert” replaced the ratings of the other two raters.
Amuse
Rgpohse/an-Response.

For the analysis of response rate, each item was coded one or zero for each
student. If the student wrote any answer related to a constructed response item prompt,
this response was coded “1”. Blanks and completely irrelevant responses (i.e. “this is a
stupid question”) or “don’t knovrf’ were coded “0”. The combined items were coded “1”
if the student responded to either part. These dichotomous codings are problematic for
common linear models. The errors are not likely to be normally distributed or
homogeneous. Additionally, errors are unlikely to be independent within students, and the
errors may be correlated within schools as well. Ifthe responses follow a binomial
distribution, a hierarchical generalized linear model which uses the estimated log-odds of
response as the outcome variable, as operationalized in the software package HLM 4
(Bryk, Raudenbush, & Congdon, 1996), can deal with these complications.

Response rates for the multiple choice items were very high (over 99%), so only
the constructed response items were analyzed. The proposed model had three levels. At
the lowest level, student j’s (nested within school k) response to item i was a function of
the student’s average log-odds of response. At the next level (level-2), each student’s
response propensity was modeled as a function of the school mean and the efi‘ects of
gender, ethnicity, and stakes (and their interactions) within that school, with an error term

for individual student difi‘erences. At the highest level, each of the level-2 coeficients was

47
a function of the grand mean, the proportion of minority students in the school, and
random variance.

At the first level, the outcome variable in the linear model was not the
dichotomous response but a transformation of the underlying probability of response. The
transformation used was the log of the odds of response because the resulting error
(unexplained variance) should be homogeneous and normally distributed with this
transformation, if the responses follow a binomial distribution within students. Also, this
outcome variable has no lower or upper bounds, with a value of zero when the probability

ofresponse is 0.5.

13(an =1)
1-P(yijk =1

 

Level 1: log [ )j = “051:. (1)

where 1:05.‘ is the log of the odds of student j (in school k) responding to any item i. The
first subscript (0) signifies this is the intercept-additional within-student coeficients
would be labeled rtljk, 1:2,}, etc.
At the second level, each student’s log-odds of response was then predicted by

student characteristics:

nor: = Book + l301k(xljk) + 1302::(in) + Bosdxsjk) 'I' BWIrxzrl) + BoskO‘rnxsjk) +

Bosdxzrtxsrk) "' [3071.(X15rX25rX35r) + for: , (2)

where X1; = 0.5 for high stakes and -0.5 for low stakes, X231, = 0 for Whites and 1 for
Blacks, centered around the proportion of Blacks in the total population (i.e., the code for
White = 0 - proportion of Blacks) , ngk = 0 for females, 1 for males, centered around the
proportion of males in the high stakes sample. These codings resulted in an intercept

(13001:) which was the predicted mean (of the log-odds of response) for school k, with half

48

the students taking a pilot test and half taking an operational test, adjusted for ethnicity
and gender (based on the high-stakes condition, because students were not sampled within
schools under this condition, which should lead to more stable estimates). Ifthe gender
and ethnic codes were not centered, the intercept would be the predicted score for White
females (the group with X; = 0 and X3 = 0). While there were not equal numbers of
students taking the pilot and operational tests, the test-stakes codes were not intended to
make the intercept reflect the average test conditions (the number of students in each
condition was based on practical reasons, not on some average mix of conditions in the
population of tests). The -0.5 and 0.5 codings allowed for an easy interpretation of the
coefficients.

The first subscript of each X identifies which factor it is associated with (1 for
stakes, 2 for ethnic group, 3 for gender), and the j,k connect it with student j in school k.
The first subscript of each [3 links it to a particular it with the same initial subscript, the
second subscript identifies which X it is associated with, and the third subscript (k) stands
for school k.

Effects were later removed if they were not significant, except that main effects
were not removed if related interactions were significant.

At the third level, school efl’ects were modeled as a function of the proportion of
students in the school who identified themselves as non-White. This proportion was based
on all students, including those who were not used in the other analyses because they
belonged to ethnic groups other than Black or White. In some schools, a large proportion
of high-stakes students did not indicate an ethnic group. In the math (1997) sample, non-

response to this question was greater than 50% in 13 of the 32 schools. In these schools,

49

the proportion minority variable was instead based on the pilot data (where, in every
school, over 80% of the students indicated their ethnicity). This procedure was also used
for six schools on the science test. All other schools had response rates of at least 80% to
the ethnicity question.

Theoretically, the model would also have a random efi‘ects component for each B;
schools vary in how large the eflects of stakes and student characteristics are. However,
as noted, it was difficult to estimate the variance of these random efi‘ects, especially with
only 28/32 schools, so a random efl‘ects term was initially included only for the school
mean. The model was:

Level-3: (3)
BOO]: = 7000 'I' 7001 (Wk) + Door.
3011: = ‘Yoro 'I' You (Wk)
3021: = ‘Yozo 'I’ 7021 (W Ir)
Boar: = ‘Yoso + 7031 Mk)
I304: = 7040 ‘I' Yo41(Wk)
305k = ‘Yoso ‘I' You (W k)
Bock = Yoso + 7061(Wk)
[3071: = 7070 + You (Wk),

where the 1’s are means, W. is the proportion of students who are minorities, centered
around the weighted mean of the schools in the sample, and the u’s are random school
efi‘ects. The first two subscripts on the 7 link it with a particular B, and the third subscript
identifies the order in the sequence of 7's predicting a B; 0 is the third subscript on the
intercept, l is the third subscript on the coefiicient for the first predictor, 2 would be the
subscript for the next predictor if there were one (and the WS would be correspondingly

numbered).

50

“low through 7070 (the averages of Bart through B071.) represented the effects of
gender, stakes, ethnicity, and their interactions. The variance of um is the variance in
school means, alter controlling the proportion of students who were minorities.
Test Sppres

The item scores were more problematic than the response/non-response codings.
Most of the items were dichotomous right/wrong (the multiple choice items), but some
had many score points. Instead of working with individual item scores, two composite
scores were predicted for each student, one based on the multiple choice items and one
based on the constructed response items. These ability estimates were based on item
response theory. The one parameter model was used because that is the model actually
used to equate and score the tests. The constructed response items were estimated with
the one-parameter partial-credit model, simultaneously with the multiple choice items.
Thus, in the calibration sample, the average scores (estimated abilities) were equal for both
types of items (though subgroups of students could have systematic difi‘erences in the two
scores). The high-stakes responses were used to calibrate the items, under the assumption
that efi‘ort would be more constant (within students, across items) on the operational test,
leading to better fit between the model and the data. On the no-stakes test, if some
students and some items were more influenced than others by lack of efl‘ort, poor fit could
result (if a student’s low efi‘ort uniformly depressed his performance on all items, or a
certain set of items tended to elicit little efi‘ort fi'om all students, this would not be a
problem for fit; it is the student by item interactions which lead to poor fit).
For the actual HSPT, item estimates and person estimates are obtained by joint

maximum likelihood. For this study, item estimates were instead obtained by marginal

5 1
maximum likelihood, using PARSCALE 3.3 (Muraki, E., & Bock, RD., 1997), with the
ability distribution empirically estimated. For long tests (such as the HSPT), correlations
between estimates under joint maximum likelihood and marginal maximum likelihood tend
to be high and close to the true parameters (Abdel-fattah, 1994; Lord, 1986; Mislevy &
Stocking, 1989, Stocking, 1989; Yen, 1987).

After the items were estimated, Bayesian Expected A-Posterior (EAP) scores were
estimated for each person on the two subscales, with a normal prior distribution. The
scores were also estimated under maximum likelihood (MLE). While the correlation
between the MLE and EAP scores was .99, the EAP scores resulted in ability distributions
which were more similar for the two subscales than the distributions estimated under
maximum likelihood. In the calibration sample, for science the standard deviation using
BAP scores was 0.76 for the multiple choice abilities and 0.84 for the constructed
response abilities. Using maximum likelihood scores, the standard deviations were 0.91
and 1.27. In math, the standard deviations were 1.06 (multiple choice) and 0.88
(constructed response) using BAP scores, compared to 1.36 and 1.16 using MLE scores.
If the underlying ability for the two types of items is the same, the distributions of abilities
should be similar. Also, for shorter tests (such as the constructed response scale here),
Bayesian scores tend to be closer to the true abilities than maximum likelihood scores,
though they are somewhat biased (Abdel-fattah, 1994; Lord, 1986; Yen, 1987). Bayesian
estimates tend to be reasonably accurate even when the true distribution departs from
normality and the prior distribution is specified as normal (Reise & Yu, 1990; Yen, 1987).
Though reported scores on the HSPT are estimated by joint maximum likelihood, scores

are reported only for the total set of items, not for the subscales used in this study. These

52

shorter scales introduced difi’erent problems (less information to estimate abilities, and a
greater proportion of students with 0 or perfect scores) which made EAP estimates more
appropriate.

Two ability scores, then, were nested within each student, and students were

nested within schools. The level one model was:

A

9 in =“or: + “Martyr can, (4)
where éijk is the predicted ability score under format i for student j in school k, “or: is the
average score of studentj (in school k), am = -0.5 for multiple choice and 0.5 for
constructed response, so nljk is the difference between the two scores (the format effect).
The first subscript for it identifies whether it is the intercept (0) or the coefficient for the
first predictor (Do-additional within-student coemcients would be labeled nzjk, 1C3jk, etc.

The measurement error is eijk, and its variance difi‘ered for each ability estimate for each

student. This variance and the 83,} were estimated in the IRT software (PARSCALE 3.3).
The level-two model used the same predictors as the level-two model for
response/non-response, and the same predictors were used for both the intercept (rrojk) and

the format efi’ect (rt 13k).

nojr = BOOk + Bork(xljk) + 802091101“ 3031:0911) + Bou(lekX23k) + Bosdxljkxsrt)
+ [3061:(sznxsjk) + Bonmrnxzuxsjk) + r05k. (5)

flux = Bron: ‘I' Bru(erk) ‘I' 3121:0910 1‘ Bl3k(xljkx2jk) + Bremlnxm) ‘I' Bramljkxak)
+13 rsk(X2r:X3r:) + Bmmaxaxa) + Ilik: (6)

where lek = 0.5 for high stakes and -0.5 for low stakes, X251, =0 for Whites and 1 for
Blacks, centered around the proportion Black in the high stakes sample, X351, =0 for

females, 1 for males, centered around the proportion of males in the high stakes sample.

53

The first subscript of each X identifies which factor it is associated with (1 for stakes, 2
for ethnic group, 3 for gender), and the j,k connect it with studentj in school k. The first
subscript of each B links it to a particular 1: with the same initial subscript, the second
subscript identifies which X it is associated with, and the third subscript (k) stands for
school k.

At the third level, school efi‘ects were modeled as functions of the proportion of
students who were non-White. A random term was included in the model for the school
mean ability and school mean format effect (300k and 3101:). The variance of these terms
would show the conditional variances (controlling percent minority) in school means.

Level-3: (7)
3001: = 7000 + 7001 (Wk) + 11001:, .3101: = 7100 'I' 7101 (wlk) + 11101:,

3011: = 7010 + 7011 (Wk), 3111: = 7110 'I' 7111 (Wk),

3021: = 7020 ‘I' 7021 (W 1:), 3121: = 7120 + 7121 (W 1:).

3031: = 7030 + 7031 (W 1:), 3131: = 7130 'I' 7131 (W 1:),

3041: = 7040 + 7041(Wk), 3141: = 7140 'I' 7141 Wk),

3051: = 7050 ‘I' 7051 (Wk), 3151: = 7150 'I’ 7151 (Wk),

3061: = 7060 + 7061 (Wk), 3161: = 7160 ‘I' 7161 (Wk),

3071: = 7070 'I' 7071 (W 1:). 3171: = 7170 + 7171 (W 1:),
where the 7’s are means, Wk is the proportion of students who are minorities, centered
around the weighted mean of the schools in the sample, and the u’s are random school
efi‘ects. The first two subscripts on the 7 link it with a particular B, and the third subscript
identifies the order in the sequence of 1‘s predicting a B; 0 is the third subscript on the
intercept, 1 is the third subscript on the coeficient for the first predictor, 2 would be the

subscript for the next predictor if there were one (and the W's would be correspondingly

numbered).

54

Level-2 effects, and their level-3 counterparts, were removed if they were not
significant. Ifan effect was significant for the format effect, it was also left in the model
for the mean to keep the interpretation clearer.
Item Parameter Estimates

To see how well the pilot parameter estimates fit the operational data, item
parameters were estimated from the pilot data, again using the one-parameter model
(partial-credit model for the polytomous items). Initially, omitted items were treated as
incorrect, because students had adequate time to attempt all items if they wished.
Parameters were estimated separately for each gender. Based on these item estimates, the
abilities of the operational students were re-estimated. Then a measure of fit, the
standardized appropriateness index of Drasgow, Levine, and Williams (1985) was

calculated for each person, using the formula:

21, = [10,, - E,(é)] + 6,,(0), (8)
where
1.. =;;6,(v.)loga,(é). (9)
are = iiaiénogmé). <10)
and F1 F1
021(6) = g:1;P..(é)P..<é)logtrénoga’ré)/P...(é». (11)

The total number of items is n, A is the number of categories (including 0 in this case),

6j(v,) =1 if the student scored in category j on item i and 0 otherwise, and Pij(é ) (the

probability of category j on item i, given 0) is calculated fi'om the item parameters.

55

This index is approximately normally distributed when item parameters are known.
Empirical distributions, based on item estimates from samples in which most examinees fit
the model, have been found to approximate the normal distribution fairly well in a practical
sense (Drasgow, Levine, & erliams, 1985).

For my purposes, the multiple choice items were treated as having only two
options, correct and incorrect, because I was not interested in patterns of choice of
distractors. Non-response was counted as zero, not as another option, because it was
scored that way on the operational test. Students were flagged as misfitting if they had
standardized indices less than -2.58, which would be expected for only 0.5% of the sample
if these students’ response patterns fit the item estimates.

These analyses were then repeated using item parameter estimates obtained when
treating omitted items as if they were not-administered rather than incorrect. For the test
of fit, the non-responses of the high-stakes were scored as zero; it was only in the item
estimation (using the pilot data) that omitted items were treated as not-presented. If non-
response was due to difl‘erent factors under low-stakes than under high-stakes (such as
low motivation under low-stakes), not scoring omitted items from the pilot test could
produce item estimates which fit the response patterns of the high stakes students better.

These analyses were not computed by ethnic group because, with less than 200
minority students (both genders combined) on the pilot forms, the parameter estimates
were unlikely to be stable, especially for the polytomous items.

The standardized appropriateness fit index flags students whose responses do not
fit the pattern expected from the item dificulties. Another way of looking at the fit of

items and persons is to flag items on which many people seem to have unexpected

56

responses, given ability and item difiiculty. To do this, the OUTFIT statistic (Wright &
Masters, 1982) was calculated based on the differences between observed and expected

scores. For item 1,
OUTFIT,=222,./N, . (12)

where N is the total number of persons,

x.-—E.

n1 nr

7111 = T—a
"Z(k’Em)2“n§

xd is the observed score for person 11 on item i,
k is the score point (range O-m),

m is the probability of person 11 scoring k on item i, and

15.; (the expected score for person 11 on item i) = Z k7:njk .

1:=o

OUTFIT is more sensitive to unexpected outliers then a related measure, INFIT, in
which each residual is weighted by its variance. Someone who has a high probability of a
high score on an item but earns a low score instead (or someone who has a low probability
of a high score but nevertheless obtains one) will have a larger impact on the OUTFIT
statistic than on the INFIT statistic.

The expected value for OUTFIT, if the responses fit the model, is 1. Values less
than 1 indicate the responses fit better than expected (less random variance than usual),
and values greater than 1 indicate the response patterns do not fit the model as well as
expected.

The average OUTFIT was calculated for each gender group by estimation

condition (method of treating omits) by response format (constructed response or multiple

57
choice), for a total of eight averages. These means were compared with a repeated-
measures ANOVA, with item as the unit of analysis. In addition to these means,
individual items were examined for especially large OUTFIT values (1 . 5 or more-5 0%

greater than average).

CHAPTER 4

RESULTS

As noted at the beginning of Chapter 3, the science results here pertain to the first
year (1996) of the HSPT, and results for the second year are in the Appendix. The math
results are provided only for the second year, because the math test did not affect diploma
endorsement in the first year.

Resp_onse Rate

Response rates for the multiple choice items were very high, averaging over 99%
in science (99.6% under low stakes, 99.8% under high stakes) and over 97% in math
(97.8% under low stakes, 99.7% under high stakes). Therefore, response/non-response
was analyzed only for the constructed response items. Items were nested within students
who were nested within schools. Initially, a hierarchical linear model with no predictors
was run to partition the variance between-schools compared to between-students within-
schools. The outcome variable was the predicted log-odds of response, with the error
variance for each student based on the binomial model, given the student’s predicted
probability of response.

In science, school means could be estimated more reliably than student means
(reliability = .935 for schools and .599 for individuals). The estimated average log-odds of

response was 2.61, which is equivalent to a probability of 93% (log (93/7) = 2.61). About

58

59

24% of the between-student variance was due to between-school difi‘erences, leaving 76%
due to difl‘erences within schools.

In math, difl‘erences were also more reliable for school means than for individual
students (reliability = .893 for schools and .595 for individuals). The average student had
an 88% probability of responding (log-odds = log(88/ 12) = 1.95). About 18% of the
between-student variance was due to between-school differences, and 82% was due to
differences within schools.

Next, predictors were added to the model. At the student level, test stakes,
gender, ethnicity, and the interactions of these variables were viewed as potential
predictors. Stakes were coded -0.5 for low stakes and 0.5 for high stakes. Ethnicity and
gender were first coded 0/1 (0 for White, 1 for Black, 0 for female, 1 for male) and then
centered around their means in the high-stakes schools, so that the intercepts in the model
would be the predicted results for a school with an average number of males and Blacks
(because the average codes for gender and ethnicity would be 0 in such a school). In the
high-stakes group, 49% of the students were male, so after centering, males were coded
0.51 and females were coded -0.49. In science, 12.1% of the students were Black so
ethnicity was coded 0.879 for Black and -0.121 for Whites. In math (a new cohort of
students), 15.8% of the students were Black, so ethnicity was coded 0.842 for Blacks and
—0. 158 for Whites. The only effect which was allowed to vary across schools was the
school mean; other efi‘ects were held constant across schools for ease in estimation. The
results appear in Tables 1 and 2.

The model estimated was:

6O

Level-1 Model

P0131: = 1) _
108 - 11011:,
1" P691: = I)

where Yijk is the observed response of student j (in school k) to item i (1 if an active,

 

relevant response is made, 0 otherwise). The first subscript (0) on 7:051, signifies this is the
intercept-additional within-student coefficients would be labeled 7:1,}, 7C2jk, etc.
Level-2 Model

“011: = 3001: + Borrmrnismkes) + BoflzriEMCity) + Bo3k(X3jk:Gender) +
Bmmljkxzik) + Bosaxmxa) + Bosk(xzrtxsrt) + Boumraxzjkxm) + f 01k.

The first subscript of each X identifies which factor it is associated with (l for
stakes, 2 for ethnic group, 3 for gender), and the j,k connect it with student j in school k.
The first subscript of each B links it to a particular it with the same initial subscript, the
second subscript identifies which X it is associated with, and the third subscript (k) stands
for school k.

Level-3 Model

3001: = 7000 4‘ Book.
3011: = 7010
3021: = 7020
3031: = 7030
3041: = 7040
3051: = 7050
3061: = 7060
3071: = 7070

The first two subscripts on the 7 link it with a particular B, and the third subscript
identifies the order in the sequence of 7's predicting a B; 0 is the third subscript on the
intercept, 1 would be the third subscript on the coeflicient for the first predictor (if there

were one).

61

In the tables below and throughout this chapter, the level-3 coefiicients are listed
below the level-2 coeficients with which they are associated, and the level-2 coeficients
are listed below the level-1 coefficients with which they are associated. Only the level-3
coeficients are actually estimated; they represent the average effects across students and
schools. In this model, all the level-2 efi‘ects are predictors of each student's average
(intercept), and each school's level-2 efl’ect is predicted by the mean (intercept) across
schools for that effect. The random school efi‘ects for the grand mean (um) are not
actually estimated, though their variance is. The coeficients reported in the tables can be
substituted for the 7's in the model, which can be used to estimate the average B's, and the
B's can then be multiplied by appropriate X‘s (the X values given in the description of the
codes for stakes, ethnicity, and gender) to obtain estimates for a particular ethnic or
gender group taking the test under high (or low) stakes. These youp estimates are
reported later after a final model is chosen.

Controlling student characteristics did not change the within-school variance in
science, reducing it only 4%. In math, however, these student characteristics accounted
for about 13% of the within-school variance. Tables 1 and 2 show that gender, test
stakes, and ethnicity were all significantly associated with the log-odds of response (p <
.01 for all three efl’ects, 1010, 7020, and yoga, in both subject areas). On average, girls
responded more often than boys (7030 was negative and X3 was positive for males),
students in high stakes conditions were more likely to respond than students in low-stakes
conditions (you; was positive and X: was positive for high stakes), and Whites were more
likely to respond than Blacks (yoga was negative and X; was positive for Blacks). In

science, there was an interaction between stakes and ethnicity (7040 = 0.510, p = .008);

62
high stakes produced more of a change in the response rate of Black students. In math,
there was an interaction between stakes and gender (7050 =-0.335, p = .001); high stakes
resulted in greater changes in the log-odds of response for girls. However, because of the
non-linear relationship between odds and probabilities, the probability of response
increased (with high stakes) slightly more for boys. These probabilities are shown later in

Tables 7 and 8, for the final model.

Table 1 - Student-Level Predictors for Log-Odds of Response, Science

 

 

Standard Approx.
Coemcient Error T-ratio df P-value
For INTERCEPTI, “Oil:

For INTERCEPTZ, 3001:

INTERCEPT3, 1000 2.711094 0.154924 17.499 27 0.000
FOI' STAKES (S), 3011:

INTERCEPT3, 7010 0.652446 0.142266 4.586 3992 0.000
FOI’ ETHNICITY (E), 302k

INTERCEPT3, 7020 -0.746545 0.194068 -3.847 3992 0.000
For GENDER (G), 303k

INTERCEPT3, 7030 -0.523382 0.083474 -6.270 3992 0.000
For S X E, B041t

INTERCEPT 3, 7040 0.510278 0.191364 2.667 3992 0.008
For S X G, 305k

INTERCEPT3, 7050 -0.092514 0.180280 -0.513 3992 0.607
For E X G, Boa,

INTERCEPT3, 7060 0.231921 0.164135 1.413 3992 0.158
ForSXGXE, Bm

INTERCEPT3, 7070 0.175609 0.240172 0.731 3992 0.465

 

63

Table 2 - Student-Level Predictors for Log-Odds pf Response, Math

 

For INTERCEPTI, 1:05;,
For INTERCEPTZ, 300k
INTERCEPT3, yooo
For STAKES (S), 3011:
INTERCEPT3, 7010
For ETHNICITY (E), B021,
INTERCEPT3, yoga
For GENDER (G), B031t
INTERCEPT3, 7030
For S X E, Boa
INTERCEPT3, 7040
For S X G, 3051:
INTERCEPT3, 7050
FOI‘ E X G, Boa;
INTERCEPT3, 7060
For S X G X E, BM
INTERCEPT3, 7070

 

Standard Approx.
Coefficient Error T-ratio d.f. P-value
2.008863 0.073248 27.426 31 0.000
1.335928 0.154946 8.622 3468 0.000
-0.811124 0.257083 -3. 155 3468 0.002
-0.468083 0.060657 -7.717 3468 0.000
-0. 139183 0.353218 -0.394 3468 0.693
-0.335268 0.093038 -3.604 3468 0.001
-0.059533 0.132146 -0.451 3468 0.652

0.171011 0.274219 0.624 3468 0.533

 

For the next models, a school-level predictor was added. The proportion of

students who identified themselves as non-White was used as a predictor of both the

intercept (school mean) and the efl‘ects of the student characteristics. This proportion was

centered around the grand mean. With this centering, school efi‘ects would be the

predicted effects for a school with an average balance of ethnic groups (after centering,

proportion-minority would be zero), with halfthe students taking a low-stakes test and

halftaking a high-stakes test.

The first model estimated with this school-level predictor used no student-level

predictors, to see how much of the between-school variance in mean response rate could

be accounted for by the linear efl‘ect of the proportion of minority students. With this term

added, the between-school variance was reduced by 44% in science and 45% in math.

64

For the next model, all the student level factors were included, and proportion-
minority (Wk) was added to predict each level-2 effect.
The model estimated was:

Level-1 Model

lo [ P(yij1: :1) J =1”
8 1-P(y,. =1) I“

 

Level-2 Model
“011: = 3001: 1’ B01k(X15k:Stakes) + 303(ertiEthni6it7) + 3031:(X3jk3Gender ) +
[3041:(X151X2a) + Bosr(X1,-1:ijr) + BO6k(X2ij3jk) + Bo7k(erkX21kX31k) + fork.

Level-3 Model
3001: = 7000 + 7001 (szMinority) + u001:.
3011: = 7010 “I" 7011 (Wk)
3021: = 7020 'I' 7021 (W 1:)
3031: = 7030 + 7031 (IN 1:)
3041: = 7040 + 7041 (Wk)
3051: = 7050 'l' 7051 (Wk)
3061: = 7060 ‘I' 7061 (Wk)
3071: = 7070 ‘I’ 7071 (Wk)

If there were additional school-level predictors, the W's would be numbered W11, W23, etc.,
to correspond with the third subscript of the associated y. Vlfrth this many school
predictors and so few schools, the estimates are not very precise; non-significant effects
were eliminated so more precise estimates could be obtained for the remaining efl’ects.

The estimated coefficients for this model are displayed in Tables 3 and 4. Again,
the 1's are listed under the B's they predict, but now the predicted B depends on the
school's ethnic composition (Wk). Because the proportion-minority is centered around the
mean (so an average school has a W = O), the 7's which represent the intercepts are still
the averages of their respective B's; a test of one of these intercepts is a test of the average

of the efi‘ect it represents. For example, the test of 7010 is a test of whether the average

65

stakes efl‘ect is significantly difi‘erent fiom zero. you represents the main effect of school

ethnic composition (the efi'ect of proportion-minority on the intercept for school k, 3001:).

The other y's which are coeficients to W's (third subscript = 1: 7011, 7021, 7031, etc.)

efi‘ectively serve as interaction efi‘ects because they show how school ethnic composition

moderates the stakes efi‘ect (7011), the ethnicity effect (you), etc.

Table 3 - Log-Odds pf Response on Science CR Items, The Full Model

 

For INTERCEPTI, 70),}
For INTERCEPT 2, 3001:
INTERCEPT3, you,
NIINORITY, 7001
For STAKES (S) , 301k
INTERCEPT3, 7010
MINORITY, 7011
For E'I'HNICITY (E), B02.E
INTERCEPT3, 7020
NIINORITY, 7021
For GENDER (G), 303k
INTERCEPT3, 7030
MINORITY, 7031
For S x E, B04.t
INTERCEPT3, 7040
MINORITY, 7041
For S X G, B051t
INTERCEP T3, 7050
MINORITY, 7051
For E X G, Bock
INTERCEPT3, 7060
wow, you
For 3 X GX E, 007.,
INTERCEPT3, 1m
MINORITY, 7071

 

Standard Approx.
Coeficient Error T-ratio d.f P-value
2.749266 0. 144220 19.063 26 0.000
-1.885434 0.744067 -2.534 26 0.018
0.875167 0.161280 5.426 3991 0.000
2.965402 1.0683 95 2.776 3991 0.006
-0.600929 0.311481 -1.929 3991 0.053
0.591835 0.775899 0.763 3991 0.446
-0.473463 0.108083 -4.381 3991 0.000
1.218082 0.648573 1.878 3991 0.060
-0.223209 0.571132 -0.391 3991 0.696
-2.535735 1.291677 -1.963 3991 0.049
-0.399672 0.276334 -1 .446 3991 0.148
-2.462978 1.172343 -2.101 3991 0.035
-0. 159644 0.403909 -0.395 3991 0.692
-0.852668 0.849911 -1.003 3991 0.316
-0.202866 1.030249 -0.197 3991 0.844
3.962828 2.174412 1.822 3991 0.068

 

66

Table 4 - Log-Odds of Response on Math CR Items; The Full Model

 

Standard Approx.

 

Coeficient Error T-ratio d.f P-value
For INTERCEPTl, not

For INTERCEPTZ, 300k

INTERCEPT3, yooo 2.006821 0.102156 19.645 30 0.000

MINORITY, 7001 -1.350982 0.705900 -1.914 30 0.065
For STAKES (S), 3011:

INTERCEPT3, 7010 1.408136 0.095899 14.683 3467 0.000

MINORITY, 7011 0.576021 1.592548 0.362 3467 0.717
For ETHNICITY (E), 3021:

DITERCEPT3, yozo -0.384300 0.220630 -1.742 3467 0.081

MINORITY, you 0.206182 0.828233 0.249 3467 0.803
For GENDER (G), 3031:

INTERCEPT3, 7030 -0.399634 0.102136 -3.913 3467 0.000

MINORITY, 7031 1.171521 0.899497 1.302 3467 0.193
For S x E, 3041:

INTERCEPT3, 7040 -0.243580 0.347431 -0.701 3467 0.483

MINORITY, yo." -0.521962 1.700159 -0.307 3467 0.759
FOI' S X G, 3051:

INTERCEPT3, 7050 -0.554819 0.148069 -3.747 3467 0.000

MINORITY, yon -1.966347 1.032644 -1.904 3467 0.056
For E X G, 3051:

INTERCEPT3, yo“ -0.533490 0.387374 -l.377 3467 0.169

MINORITY, 7061 -.893796 0.974605 -0.917 3467 0.359
For S X GX E, 3071:

INTERCEPT3, 7070 0.514745 0.841908 0.611 3467 0.541

MINORITY, you 2.235412 1.478214 1.512 3467 0.130

 

In science, the within-school stakes by ethnicity interaction was no longer

significant when the proportion of minority students was included in the model (7040 = -

0.223, p=.696). In high-minority schools, the students were more influenced by the test

stakes (you = 2.97, p = .006), but when this school efl‘ect was controlled there was not an
interaction between stakes and ethnicity (though within high-minority schools, high stakes

did not increase the responses of Blacks as much as Whites (yo41)). The interactions

67

among the other student-level factors were not statistically significant (as before) and I
removed them from the next model. In both this group and the second cohort (see
Appendix), the effect of school proportion-minority on the stakes by gender by ethnicity
interaction (yo-,1) was nearly statistically significant (because school proportion-minority
modified the stakes by gender by ethnicity effect, there was essentially a four-way
interaction), but the efi’ect was different for the two test forms. In the first, in high-
minority schools, Black boys had an especially large increase in response under high
stakes. In the second cohort, this efi’ect was observed in the low-minority schools. To be
able to make some more general interpretations, the main effects model was estimated.
The proportion of minority students in the school had a statistically significant effect on
the school mean and on the stakes and gender effects; I left the proportion-minority term
in the model for these terms, but removed it as a predictor of the ethnicity efi’ect.

In math (see Table 4), the proportion of minority students had a "borderline"
significant efi’ect on the intercept (yool = -1.35, p = .065) and on the gender by stakes
interaction (yon = -1.97, p = .056). Schools with a large proportion of minority students
had lower intercepts and a greater (more negative) gender by stakes interaction. I
removed the three-way interaction and the non-significant two-way interactions fi'om the
model. I also deleted the proportion-minority predictor for all effects except the intercept
and gender by stakes interaction. In this model (not shown), the proportion-minority no
longer significantly influenced the gender by stakes interaction (p = .785), so I removed it
fi'om the final model summary in Table 6.

These more parsimonious models are shown in Tables 5 and 6.

68

Table 5 - ng-Qdds of Response on Science CR Items. the Final Model

 

 

Standard Approx.

 

 

 

 

Coefficient Error T-ratio d.f P-value
For INTERCEPTI, irojk
For INTERCEPTZ, 3001:
INTERCEPT3, yooo 2.781400 0.1 15269 24.130 26 0.000
MINORITY, yool -1.633169 0.609339 -2.680 26 0.013
FOI' STAKES, 301k
INTERCEPT3, yolo 0.633131 0.146497 4.322 3991 0.000
MINORITY, you 0.962471 0.267452 3.599 3991 0.001
For ETHNICITY, 3021:
INTERCEPT3, 7020 -0.503269 0.216514 -2.324 3992 0.020
FOI' GENDER, 303k
INTERCEPT3, 7030 -0.555828 0.076539 -7.262 3991 0.000
MINORITY, yon 0.474410 0.134265 3.533 3991 0.001
Table 6 - Qg-Qdds of Response on Math CR Items, the Final Mods]
Standard Approx.
Coeficient Error T-ratio d.f P-value
For INTERCEPTI, nojk
For INTERCEPTZ, 3001:
INTERCEPT3, yooo 2.022953 0.063159 32.029 30 0.000
MINORITY, yool -1.624456 0.176036 -9.228 30 0.000
For STAKES (S), 3011:
INTERCEPT3, yolo 1.353810 0.162707 8.321 3468 0.000
For GENDER (G), 303:
INTERCEPT3, 7020 -0.485008 0.061482 -7.889 3468 0.000
FOI' S X G, 3031:
INTERCEPT3, yo3o -0.335005 0.096537 -3.470 3468 0.001

 

The coemcients in Tables 5 and 6 are in the log-odds metric. Tables 7 and 8 show
the predicted log-odds, odds, and probabilities by gender, stakes, and ethnicity. These
would be estimates for a school with an average proportion of non-White students
(because school proportion-minority was centered around the mean). When ethnicity was

not a factor in the tables, the estimate was for a group of students whose ethnic

69

background was proportional to the high-stakes sample. The log-odds were found by
substituting the coemcients from Tables 5 and 6 into the hierarchical models (using the
codings for stakes, gender, and ethnicity detailed earlier), and then the log-odds were
mathematically transformed to odds and probabilities. The probability-ratios and odds-
ratios shown in the tables are the ratio of the first group to the second group listed for that

factor (for example, the gender ratio is the ratio of girls:boys because girls are listed first).

Table 7 - Predicted Log-Odds, Probability (fire), and Odds of Response, Science

 

 

 

probability
log-odds probability ratio odds odds ratio
population average 2.78 0.94 16.14
gender 1.03 1.88
girls 3.05 0.95 21.18
boys 2.50 0.92 12.15
stakes 1.02 1.61
high stakes 3.10 0.96 22.15
low stakes 2.46 0.92 11.76
ethnicity 0.97 0.60
Black 2.34 0.91 10.37

White 2.84 0.94 17.16

 

Table 8 - Prfl'ctsd Log-Odds, Prphahiliht (fite ), ar_1d Qdds of Response, Math

 

 

probability
log-odds probability ratio odds odds ratio

population average 2.02 0.88 7.56
Stakes 1.14 3.87

High Stakes 2.70 0.94 14.88
Girls 3 .02 0.95 1.03 20.48 1.92

Boys 2.37 0.91 10.67

Low Stakes 1.35 0.79 3.84
Girls 1.50 0.82 1.05 4.49 1.37

Boys 1.18 0.77 3.27

 

70

In science, girls were 0.03 times more likely (in the probability metric, because
95/92 = 1.03) to respond to each item than boys were (odds ratio of 21 .18/ 12.15 = 1.88).
Students taking the test under high stakes conditions were 0.02 times more likely
(probability ratio:96/94 = 1.02) to respond than students tested under low stakes
conditions (odds ratio of 1.61). Blacks were 0.97 times as likely as Whites to respond to a
constructed response item (odds ratio of 0.60).

In math, girls were 0.05 times more likely (main efl’ect not shown in table,
probability ratio = 93/88 = 1.05) to respond than boys were (odds ratio of 1.62). Under
high stakes, students were 0.14 times more likely to respond than they were under low
stakes (odds ratio of 3.87). In the odds metric, the ratio of girls' to boys' response
tendencies was greater under high stakes than under low stakes, but in the probability
metric this ratio was slightly greater under low stakes. This shows the non-linear
relationship between the two metrics.

To illustrate how these effects difi‘ered with the proportion of minority students,
predicted results for a school with 5% minority students are compared with the predicted
results for a school with 90% minority students in Tables 9 and 10. Because there were
only a small number of schools, the estimates of the effect of the school-level proportion
minority are less precise then the estimates of the student-level efi’ects, but they give an

approximation of the possible between-school difi’erences.

71

Table 9 - Prflictions for Sshools with High and Low Minority Enrollment, Scienpp

 

 

 

5% Minority probability
log-odds probabth ratio odds odds ratio
population average 3.05 0.95 21.07
gender 1.03 1.88
girls 3.36 0.97 28.71
boys 2.72 0.94 15.24
stakes 1.02 1.61
high stakes 3.29 0.96 26.73
low stakes 2.81 0.94 16.60
ethnicity 0.97 0.60
Black 2.61 0.93 13.54
White 3.11 0.96 22.39
90% Minority probability
log-odds probability ratio odds odds ratio
population average 1.66 0.84 5.26
gender 1.04 1.26
girls 1.77 0.85 5.88
boys 1.54 0.82 4.67
stakes 1.24 3.65
high stakes 2.31 0.91 10.04
low stakes 1.01 0.73 2.75
ethnicity 0.91 0.60
Black 1.22 0.77 3.38

White 1.72 0.85 5.59

 

72

Table 10 - Predictions fpr Schools with High and Low Minority Enrollment, Math

 

 

 

5% Minority probability
log-odds ratio piobability odds odds ratio
population average 2.29 0.91 9.85
Stakes 1.14 3.87
High Stakes 2.96 0.95 19.39
Girls 3.28 0.96 1.03 26.70 1.92
Boys 2.63 0.93 13.90
Low Stakes 1.61 0.83 5.01
Girls 1.77 0.85 1.05 5.85 1.37
Boys 1.45 0.81 4.26
90% Minority probability
log-odds ratio probability odds odds ratio
population average 0.92 0.71 2.51
Stakes 1.48 3.87
High Stakes 1.60 0.83 4.93
Girls 1.92 0.87 1.12 6.79 1.92
Boys 1.26 0.78 3.54
Low Stakes 0.24 0.56 1.27
Girls 0.40 0.60 1.15 1.49 1.37
Boys 0.08 0.52 1.08

 

In science, for the efi’ects of test stakes, the odds ratio (high stakes: low stakes)

was much larger in the high-minority school (10.04/2.75 = 3.65) than in the low-minority

school (26.73/16.60 = 1.61). To a lessor extent, the opposite was true for the gender

efi‘ect. However, because of the non-linear relationship between the odds and the

probability, the probability ratio (girls: boys) was actually slightly larger in the high-

minority school though the odds ratio was larger in the low-minority school. [In the

second cohort, both the odds ratio and the probability ratio showed greater gender

difl’erences in high-minority schools—see Appendix] While the odds ratio remained

unchanged for the ethnic efi’ect (because school proportion-minority did not have a

73

significant effect on the ethnic efi‘ect in the log-odds units), the probability ratio changed
when the school-average log-odds changed.

In math, the average log-odds changed significantly with proportion-minority; the
log-odds decreased as the proportion-minority increased. The odds-ratios for the other
efi‘ects did not change significantly with school proportion-minority. However, in the
probability metric, stakes made a bigger difi‘erence in the high-minority schools. In the
low-minority schools, students were 14% more likely to respond under high stakes, while
in the high-minority schools, students were 48% more likely to respond under high stakes
(while the odds-ratio was 3.87 for both comparisons).

Further attempts were made to add a random school efi‘ect to the model for the
efi‘ects of stakes, gender, and ethnicity (random efi’ects were added to each of these terms
individually). In science, the between-school variance in gender efl’ects was not
statistically different from zero. With only 28 schools and one random term already in the
model (the school mean), the variance could not be estimated for the stakes or ethnicity
effects. The problem appeared to be collinearity with the random school intercept. In
math, the variance in the school stakes efi‘ect was about 44% the size of the variance in
school means, and the two efiects had a correlation of .36. The variance in gender efl’ects

and ethnicity effects could not be estimated while the random school mean efi’ect was in

the model.
Ability Ssptes
Relsp'phship Boweg Qppsttpcted Rmnse spd Mphiplp thice Scores

All items were calibrated together, using the responses of the high stakes group.

This procedure implicitly involves the assumption that both response formats measure one

74

predominant factor. One way of checking this is Yen’s Q3, which was described in the
Method section to support the treatment of item clusters as independent items. Ifthe
constructed response items measured something consistently different than the multiple
choice items, there would be highly correlated residuals among the constructed response
items. However, in science all the correlations among the constructed response items
were no larger than .04 (slight negative correlations are expected because each item
contributes to the ability score), after the two “investigation” items were combined. The
constructed response items, then, were not measuring a common factor apart from the
predominant factor measured by the test as a whole. In math, on the other hand, 3 of the
15 correlations among the constructed response items were greater than .10, and the other
12 were greater than 0. While none of the correlations were large enough to cause a
practical problem for item estimation, these items did seem to measure (to a small degree)
something difl‘erent fi'om the test as a whole, which could indicate either the items were
measuring another aspect of math or they were measuring a construct-irrelevant, but
shared, source of variance.

A simple way of checking the relationship between the formats is to look at the
correlation between the raw scores. In the high stakes group, this correlation was .78 in
science and .80 in math. Another way to estimate the correlation is to run a random-
efl’ects hierarchical linear model with two outcomes for each student, multiple choice
score and constructed response score (these were estimated on the same scale, using a
one-parameter item response theory model). With no student-level factors in the model,
the correlation between the residuals for these two scores is the estimated within-school

correlation between the two "true" scores. The measurement error is taken into

75
consideration by using the standard error of the ability estimate (fi'om the [RT estimation)
in the within-student variance. Using this model, the correlation between the scores was
estimated to be about .98 in science, and .92 in math.
The constructed response items did add to the precision of the measurement scale.
The information functions for scores based on the multiple choice items alone, compared

to scores based on all items, are plotted in Figures 1 and 2.

 

.a
N

 
  

A
O
1

SCUM-flawed:—
O

 

    

 

0 .
4.00 -3.17 -2.34 -1.52 -0.69 0.14 0.97 1.79 2.62 3.45

Ability

1LL1141111111111111111111111
IIfiIIIIIIIIITIIIIIIIIITIIIII

+ Multiple Choice Items —cr-AII Items

Figure 1: Information Function, Science

76

 

:o-o-ona-uo-ca—

 

 

 

 

-o-— Multiple Choice Items —1:1—All Items

Figure 2: Information Function, Math

Faptpts Afi’ecting Spores

From these item estimates based on the high-stakes sample, two scores were
estimated for each student, one based on the multiple choice items and the other based on
the constructed response items. Because both types of items were estimated together, the
average score was defined to be 0 on both subscales, in the high-stakes group.

After scores were estimated for all students, the variance due to schools and to
students within schools was estimated with a random-efl‘ects hierarchical linear model,
with the within-student variance set to the square of the estimated standard error of
measurement (based on the IRT score estimation—the inverse of the information function

at the student’s estimated ability). About 25% of the between-student variance in science,

77

and 23% in math, was due to between-school difi’erences. Next, the proportion of
students who identified themselves as non-White was entered into the model to predict
each school mean. This characteristic accounted 'for about 68% of the between-school
variance in scores in science, and 48% in math.

Next, the format of the items was added to the model. Format was coded +0.5 for
the constructed response score and -0. 5 for the multiple choice score, so the intercept
would be the mean of the two scores and the coeficient for format would be the
difi'erence between the scores. In science, the within-school variance in format effects
could not be estimated. At an intermediate estimation step, the estimated variance was
not significantly greater than zero (x2=3697.57, df=3993). In firrther models, the format
efi’ect was held constant (or constant conditional on student factors) within schools, but
allowed to vary across schools. Given that the within-school variance in the format efi‘ect
seemed to be nearly zero, it might not seem reasonable to attempt to find factors which
explained difi‘erences in the format effect. However, as there were theoretical reasons to
include specific factors (test stakes, gender, and ethnicity) which might be associated with
the size of the format efi‘ect, these factors were included in later models (and the format
effect was held constant within schools, conditional on these factors). [In the second
science cohort, there was small but significant variance in format efi‘ects, until student-
level characteristics were added to the model.]

In science, schools’ overall means could be estimated more reliability than their
format efl‘ects; the reliability was .954 for the means and .552 for the format efi‘ects. The

reliability was .793 for the student-level means. The correlation between the school mean

78

and the school format efi‘ect was .807. In schools with low means, the constructed
response items decreased scores more.

In math, the within-school variance due to format could be estimated, but the
reliability of the format efi’ect for individual students was low (.221). The reliability of the
school format efi‘ect was much higher (.943 ), as was the reliability for student means
(.858) and school means (.917). There was a negative correlation between format effects
and means of -.29 at the student level and -.35 at the school level. Those who scored high
overall were likely to have a greater decrease due to the constructed response format, the
opposite of the finding in science. Across subject areas, the format of the items did not
consistently have a greater impact on students of one achievement level.

For the next model, I added the student characteristics. At level-1 (within-
students), the format of the items in the subscale was a factor (held constant within
schools for science). At level-2, the factors were test stakes, student gender, and
ethnicity. They were coded the same as in the models for response rate: high stakes = 0.5,
low-stakes = -0.5, male = 0.51, female = —0.49, Black = 0.879 in science and 0.842 in
math, White = -O.121 in science and -0.158 in math. These codings allowed the level-3
intercepts to be the average efiects for schools with average proportions of genders and
ethnicities (because in these schools the average gender and ethnic group would equal
zero). The school mean (intercept) and format effect were allowed to vary randomly
across schools (proportion of minority was added back later), but the stakes, gender, and

ethnicity efi'ects were held constant across schools.

79

The model was:

A

9111: =1tojr: + 1tm,(format)+ 6151:»

where 9 ijk is the estimated ability under format 1 for student j in school k, 7toj-k is the
average of the two formats, 1:111: is the difl‘erence between the formats, and eijk is random

error (the variance of which was estimated through IRT).

11105]; = 3001: + Bo1k(stakes:X1,-k) + BozflCthIliCltYIijk) + Bo3k(gender:X3,-k) + 3041:(X1j1:X2jk) +
Bosk(erkX31k) + Bo61:(ijkX31k)+ 3071:(X1ij2ij3jl:) + You.

111,1: = 3101: ‘I' 3111:(er1:) + 3121:(ij1:) 'I' 313k(X151:X251:) 1‘ 3141:(Xr,1:X2j1:) + 3151:(X1j1:X3jr:)
+3161:(X2jkx3jk) + 3111((X1ij2ij3jk) + 1‘1 jk, (no 1' ljk for 80181108)
where 3001: and 3101: are the intercepts for school k's mean and format, efi‘ect, respectively,
the other B's are the coeflicients for the factors (X‘s) as defined the first time each factor is
listed, and the variances of rajk and r131, are random within-school variances.

3001: = 7000 ‘I' 11001:, 3101: = 7100 + urok,

3011: = 7010 , 3111: = 7110 ,
302: = 7020 , 3121: = 7120 3
3031: = 7030 , 3131: = 7130 ,
304k=7040, 3r41:=7r40,
3051: = 7050 , 3151: = 7150 ,
3061:=7060, 316k=71605
3071: = 7070 , 3171: = 7170 3

where the y's are intercepts (averages across schools) for their respective B's and the
variances of uoo.K and ulok are random between-school variances.

The estimates for this model are shown in Tables 11 and 12. Again, level-3
coeficients are shown under the level-2 effects they predict, which in turn are listed under
the associated level-1 efi’ects. Thus, all predictors of student j's (in school k) intercept

(nojk) are listed before the predictors of the student's format efl‘ect (it 151:). The efi‘ects listed

80

under £151: essentially represent interactions with format because they modify the format

efl‘ect (except the intercept, yloo, which is the main efi‘ect of format).

Table 11 - Student-Level Predictors for Abilig Scores, Science

 

 

Standard Approx.
Coefficient Error T-ratio d.f. P—value
For INTERCEPTl, 1ro,1c

For INTERCEPTZ, 3001:

INTERCEPT3, yooo -0.219714 0.046325 -4.743 27 0.000
For STAKES (S), 3011:

INTERCEPT3, 7010 0.266526 0.054286 4.910 3964 0.000
For ETHNICITY (E), 3021:

INTERCEPT3, yooo -0.441193 0.091912 -4.800 3964 0.000
For GENDER (G), 3031:

INTERCEPT3, 7030 0.050724 0.018751 2.705 3964 0.007
For S X E, 3041:

INTERCEPT3, yo4o -0.037600 0.079854 -0.471 3964 0.637
For S X G, 3051:

INTERCEPT3, 7050 0.046886 0.047226 0.993 3964 0.321
For E X G, 3061:

INTERCEPT3, 7060 -0.034031 0.046008 -0.740 3964 0.459
For S X G X E, 3071:

INTERCEPT3, 7070 -0.129336 0.180275 -0.717 3964 0.473

For FORMAT slope, K151:

For INTERCEPTZ, 3101:

INTERCEPT3, 7100 -0.097831 0.016483 -5.935 27 0.000
For STAKES, 3111:

INTERCEPT3, 7110 0.187509 0.022731 8.249 4020 0.000
For ETHNICITY, Bm

INTERCEPT3, ym -0.041197 0.037648 -1.094 4020 0.274
For GENDER, 313k

INTERCEPT 3, 7130 0182170 0.018896 -9.641 4020 0.000
For S X E, 3141:

INTERCEPT3, yooo 0.038476 0.074645 0.515 4020 0.606
For S X G, B151,

INTERCEPT3, 7150 0.017272 0.039639 0.436 4020 0.663
For E X G, B151,

INTERCEPT3, 716o 0.036570 0.083583 0.438 4020 0.661
ForS XGXE, Bm

INTERCEPT3, yno -0.036484 0.071973 -0.507 4020 0.612

 

81

Table 12 - Student-Level Predictors for Ability Scores Math

 

 

Standard Approx.

 

Coefficient Error T-ratio d.f. P-value
For INTERCEPTI, noo,

For INTERCEPTZ, 3001:

INTERCEPT 3, yooo -.292614 0.041978 -6.971 31 0.000
For STAKES (S), 3011:

INTERCEPT3, 7010 0.476044 0.082057 5.801 3436 0.000
For ETHNICITY (E), 302k

INTERCEPT3, yozo -0.699293 0.090220 -7 .751 3436 0.000
For GENDER (G), 803k

INTERCEPT3, 7030 -0.031611 0.038610 -0.819 3436 0.413
For S X E, 3041:

INTERCEPT3, 7040 0.079787 0.100327 0.795 3436 0.427
For S X G, 3051:

INTERCEPT3, 7050 0.086615 0.048598 1.782 3436 0.074
For E X G, 3061:

INTERCEPT3, 7060 -0.097234 0.102314 -0.951 3436 0.342
ForS XGXE, [307k

INTERCEPT3, 7070 0.041274 0.120575 0.342 3436 0.732

For FORMAT slope, “111:

For INTERCEPTZ, 3101:

INTERCEPT3, 7100 -0.239405 0.050330 -5.830 31 0.000
FOI' STAKES, 3111:

INTERCEPT3, 7110 0.477970 0.100358 4.763 3436 0.000
For ETHNICITY, 3121:

INTERCEPT3, ym 0.130640 0.076187 1.715 3436 0.086
For GENDER, Bm

INTERCEPT3, y13o -0.182232 0.026402 -6.902 3436 0.000
For S X E, Bug

INTERCEPT3, 7140 0.212012 0.134392 1.578 3436 0.114
For S X G, 3151:

INTERCEPT3, 7150 -0.017074 0.052135 -0.328 3436 0.743
For E X G, 3161:

INTERCEPT3, 7160 -0.034828 0.060627 -0.574 3436 0.565
ForSXGXE, Bm

INTERCEPT3,y1-,o -0.025654 0.136738 -0.188 3436 0.851

 

82

None of the interactions among the student-level characteristics were significant at
the .05 level (though in math the stakes by gender efi’ect was close (yoso = .087, p = .074)),
but each of the student-level characteristics had a significant main efi‘ect (except gender in
math, yo3o) and stakes and gender influenced (interacted with) the response format efi‘ect
(yuo and mo). When students took the test under high stakes, their subscores were
defined to be approximately equal on average. Under low-stakes, however, the multiple
choice score was higher than the constructed response score. In science, for boys the
average constructed response score was lower than the average multiple choice score,
while for girls the two scores were, on average, about the same. In math, the average
constructed response score was lower for both genders, but the difference was larger
(more negative) for boys. VVrthin schools, the average score for Blacks was lower than
the average score for Whites.

Next, the interactions among the student-level factors (stakes, ethnicity, and
gender) were removed fi'om the model for science. Removing all these efi‘ects
simultaneously decreased the number of parameters estimated by eight, and increased the
deviance. of the model by about 4.209. The deviance is a measure of fit (or misfit); it is
distributed as chi-square with degrees of freedom equal to the number of parameters, so
the change in deviance is approximately distributed as chi-square with degrees of fieedom
equal to the change in number of parameters (Bryk & Raudenbush, 1992, Chapter 3). The
fit of the model, then, did not significantly change with the removal of these eight
interaction terms (12(3) = 4.209, p = .838).

For math, all the interactions among the student level factors, except stakes by

gender, were removed. The stakes by gender efi’ect became less significant when these

83
other effects were removed (and remained so in later attempts at models including the
school proportion-minority), so it was removed also. The change in deviance between the
models with all student-level interactions and the model with none of these interactions
was 15.377, again with 8 degrees of freedom. This was almost significant at the .05 level,
so removing the set of interactions as a whole did lead to somewhat worse fit. However,
none of the individual interactions could be pinpointed as being particularly useful.

For the next model, the proportion of students who identified themselves as non-
White was used as a predictor at the school level. This predictor was centered at the
grand mean, so the intercepts would be the values for a school with an average proportion
of non-White students (about 21-22%). The estimated coeficients of this model are
displayed in Tables 13 and 14.

In science, proportion of minority students was a significant predictor only of the
school intercept (yool = -0.60, p = .025). [In the second cohort, proportion of minority
students was a significant predictor only of the gender and gender by format effects]. For
the final model (Table 15), then, proportion of minority students was removed as a
predictor except for this one efl‘ect. Notice this change produced a change in the ethnicity
efi’ect on the format effect (the ethnicity by format interaction, yuo). Vlrrth proportion of
minority students as a school level predictor of all efi‘ects, the ethnicity by format effect
was slightly positive and not significantly difi‘erent fi'om zero (see Table 13). With
proportion of minority students as a predictor only of the mean, the ethnicity by format
effect was significantly negative and its standard error decreased by half (see Table 15).

When proportion of minority students was removed (except for as a predictor of the

84

mean), seven fewer parameters were estimated and the deviance of the model changed by

only 4.03 (not a statistically significant change).

Table 13 - Prgicted Scores in Science, Controlling School Minorig Composition

 

 

Standard Approx.
Coefiicient Error T-ratio d.f. P-value
For INTERCEPTI, Itojk

For INTERCEPTZ, B001:

INTERCEPT3, yooo -0.224317 0.051854 -4.326 26 0.000

MINORITY, 7001 -0.604706 0.254938 -2.372 26 0.025
For STAKES, Bo“,

INTERCEPT3, 7010 0.267159 0.052223 5.116 3963 0.000

MINORITY, you 0.040849 0.101976 0.401 3963 0.688
For ETHNICITY, B02k

INTERCEPT3, 7020 -0.391861 0.127932 -3.063 3963 0.003

MINORITY, 7021 0.116587 0.278516 0.419 3963 0.675
For GENDER, 3031:

INTERCEPT3, 7030 0.060345 0.016665 3.621 3963 0.001

MINORITY, 7031 -0.028359 0.037500 -0.756 3963 0.450

For FORMAT slope, 7:1,],

For INTERCEPTZ, B101:

INTERCEPT3, 7100 -0.090846 0.024023 -3.782 26 0.001

MINORITY, 7101 -0. 120986 0.119589 -1.012 26 0.321
For STAKES, 3111:

INTERCEPT3, 7110 0.189326 0.023235 8.148 4019 0.000

MINORITY, 7111 0.051586 0.093655 0.551 4019 0.581
For ETHNICITY, Bm

INTERCEPT3, yuo 0.019063 0.064370 0.296 4019 0.767

MINORITY, 7121 -0.075475 0.144707 -0.522 4019 0.602
For GENDER, B13:t

INTERCEPT3, 7130 -0. 179031 0.020228 -8.851 4019 0.000

MINORITY, ym -0.007193 0.123969 -0.058 4019 0.954

 

85

Table 14 - Predicted Scores in Math, Conttolling School Minorig Composition

 

 

Standard Approx.
Coefficient Error T-ratio d.f. P-value
For INTERCEPTI, nojk

For INTERCEPTZ, 3001:

INTERCEPT3, yooo -0.274292 0.069874 -3.925 . 30 0.001

MINORITY, 7001 0.005295 0.417567 0.013 30 0.990
POI STAKES (S), 3011:

INTERCEPT3, 7010 0.476844 0.081774 5.831 3435 0.000

MINORITY, 7011 0.140810 0.138483 1.017 3435 0.310
For ETHNICITY, BozjE

INTERCEPT3, 7020 -0.643642 0.112995 -5.696 3435 0.000

MINORITY, 7021 -0. 188647 0.494516 -0.381 3435 0.702
For GENDER (G), 3031:

INTERCEPT3, 7030 -0.026184 0.038278 -0.684 3435 0.494

MINORITY, you 0.003311 0.118061 0.028 3435 0.978
For S x G, 3041:

INTERCEPT3, 7040 0.074895 0.047568 1.574 3435 0.115

MINORITY, yo“ -0. 145633 0.161804 -0.900 3435 0.368

For FORMAT slope, am

For INTERCEPTZ, 3101:

INTERCEPT3, yloo -0.328920 0.051302 -6.41 1 30 0.000

MINORITY, 7101 -0.331858 0.186887 -l.776 30 0.085
For STAKES, 3111:

INTERCEPT3, 7110 0.477706 0.101277 4.717 3435 0.000

MINORITY, 7111 0.265831 0. 149640 1.776 3435 0.075
For ETHNICITY, 3121:

INTERCEPT3, 7120 0.156460 0.099890 1.566 3435 0.1 17

MINORITY, ym 0.346326 0.276510 1.252 3435 0.211
For GENDER, 313k

INTERCEPT3, 7130 -0.184764 0.027467 -6.727 3435 0.000

MINORITY, 7131 -0.052860 0.098589 -0.536 3435 0.591

 

86

In math, school proportion-minority was a borderline significant predictor of the

format effect (ylol) and the stakes by format interaction (7111). After removing it as a

predictor for all other effects, and removing the stakes by gender effect (as described

above), eight fewer parameters were estimated and the deviance changed by 6.363 (non-

significant).

Table 15 - The PM Model for Science Ability Scores

 

ForINTERCEPTl, nojk
For INTERCEPT2, 3001:
INTERCEPT3, 7000
MINORITY, 7001
FOI' STAKES, 301k
INTERCEPT3, 7010
For ETHNICITY, 3021:
INTERCEPT3, 7020
FOI' GENDER, B03k
INTERCEPT3, 7030
For FORMAT slope, 7:1,}
For INTERCEPT2, 3101:
INTERCEPT3, 7100
For STAKES, 3111:
INTERCEPT3, 7110
For ETHNICITY, Bm
NTERCEPT3, 7120
For GENDER, 3131:
INTERCEPT 3, 7130

 

Standard Approx.

Coefiicient Error T-ratio d.f. P-value
-0.214849 0.037118 -5.788 26 0.000
-0.473096 0.149135 -3.172 26 0.004

0.267049 0.053331 5.007 3964 0.000
-0.3 80583 0.100030 -3.805 3964 0.000

0.060361 0.017315 3.486 3964 0.001
-0.095642 0.015553 -6.149 27 0.000

0.188382 0.023123 8.147 4020 0.000
-0.081150 0.032538 -2.494 4020 0.013
-0. 177916 0.019134 -9.299 4020 0.000

 

87

Table 16 - The Final Model for Math Ability Scores

 

ForIN'I‘ERCEPTl, 1:0,:
For INTERCEPT2, 3001:
INTERCEPT3, 7000
For STAKES, Bo“,
INTERCEPT3, 7010
For ETHNICITY, [3021:
INTERCEPT3, yozo
For GENDER, B03k
INTERCEPT3, 7030
For FORMAT slope, am
For INTERCEPT2, B101:
INTERCEPT3, y1oo
MNORITY, 7101
For STAKES, B111
INTERCEPT3, 7110
MINORITY, 7111
For ETHNICITY, pm
INTERCEPT3, 7120
FOI' GENDER, B131;
INTERCEPT3, 7130

 

Standard Approx.
Coeflicient Error T-ratio d.f. P-value
-0.289806 0.042248 -6.860 3 1 0.000
0.473559 0.084870 5.580 3436 0.000
-0.673611 0.089103 -7 .560 3436 0.000
-0.016969 0.037698 -0.450 3436 0.652
-0.293539 0.050224 -5.845 30 0.000
-0. 149719 0.073394 -2.040 30 0.050
0.477662 0.101225 4.719 3435 0.000
0.281601 0.145959 1.929 3435 0.053
0.171526 0.088514 1.938 3436 0.052
-0. 183307 0.027994 -6.548 3436 0.000

 

The final models are shown in Tables 15 and 16. In both math and science,

students scored higher under high stakes (yolo—the estimated difference was greater for

math). White students scored higher than Black students (yooo), but in science some of this

difi’erence was due to between-school differences in school proportion-minority (yool).

There was a small but significant gender difi’erence in science (yooo), favoring males. In

both subject areas gender moderated the response format efi‘ect (7130), with males scoring

higher on multiple choice and females scoring relatively higher on constructed response.

Within schools, the ethnicity by format interaction (yuo) was in opposite directions in math

and the first science cohort, and in math, the school percent minority (yool) had a different

88
effect on the format effect than the individual student's ethnicity did (ylzo). Students fi'om
high minority schools had a larger (more negative) format effect, but within schools the
average format efi’ect was larger for Whites. [The efi’ect in the second science cohort was
similar to that in math].

Using these coemcients (and the codings for format, stakes, ethnicity, and gender
described above), mean scores can be estimated within a typical school (one with about
21-22% minority students). These mean scores are shown in Tables 17 and 19. Tables 18
and 20 provide standardized differences for the significant effects. For these tables, the
difi‘erences were divided by the standard deviation in the entire sample (0.745 in science
and 0.891 in math). This allowed for comparison of effects within this study; if the gender
groups differed by the same non-standardized value as the ethnic groups difl’ered by, for
example, the standardized difi’erence would be the same for both gender and ethnicity.
When comparing results from this study with other research, readers might wish to
calculate an efi‘ect size based on the pooled within-group standard deviation. The
standard deviations provided in Tables 17 and 19, with the group sizes reported in
Chapter‘3, can be used for this purpose. The standard deviations in Tables 17 and 19
were calculated from the variance of the entire sample; they are not within-school
variances. Also, they were based on the reliability-weighted scores (scores weighted by
the inverse of the square of the standard error of measurement for that score), as the

group means were.

89

Table 17 - Averoge Predicted Scoros in Science

 

 

 

 

 

multiple choice constructed response mean
we 3.4 mean sci man sci
average -0.17 0.72 -0.26 0.78 -0.21 0.68
high stakes -0.08 0.71 -0.08 0.77 -0.08 0.67
low stakes -0.25 0.73 -0.44 0.77 -0.35 0.68
Blacks -0.47 0.59 -0.63 0.74 -0.55 0.58
Whites -0.13 0.69 -0.21 0.74 -0.17 0.65
Boys -0.09 0.75 -0.28 0.80 -0. 18 0.71
Girls -0.24 0.68 -0.25 0.77 -0.24 0.66
Table 18 - Sgpdardized Within-School Efl‘ects in Soieng
standardized difference
_u_n__L0__m lti le h ice W W

stakes 0.23 0.48 0.36
ethnicity -0.46 -0.57 -0.51
gender 0.20 -0.04 0.08

 

90

Table 19 - Average Predicted Scores in Math

 

 

 

multiple choice constructed response mean
mean sd mm .891. mean .811
average -0.14 0.94 -0.44 0.83 -0.29 0.78
high stakes -0.03 0.97 -0.08 0.76 , -0.05 0.77
low stakes -0.26 0.88 -0.79 0.80 -0.53 0.75
Blacks -0.78 0.66 -0.93 0.85 -0.86 0.66
Whites -0.02 0.94 -0.34 ‘ 0.80 -0.18 0.76
Boys -0.10 0.99 -0.49 0.85 -0.30 0.81
Girls -0.18 0.90 -0.38 0.80 -0.28 0.76
W 90% minority
multiple constructed mean multiple constructed mean
choice response choice response

average -O.16 -0.42 -0.29 -0.09 -0.49 -0.29
high stakes -0.03 -0.08 -0.05 -0.02 -0.08 —0.05
low stakes -0.29 -0.77 -0.53 -0.16 -0.89 -0.53
Blacks -0.80 -0.92 -0.86 -0.73 -0.98 -0.86
Whites -0.04 -0.33 -0.18 0.03 -0.39 -0.18
Boys -0.12 -0.48 -0.30 -0.05 -0.54 -0.30
Girls -0.19 -0.37 -0.28 -0.13 -0.43 -0.28

 

Table 20 - Stgdgdggl’ Mthm' -Sohool Effects in Math

 

 

standardized difi’erence
Mm ' l h ice W was;
stakes 0.26 0.80 0.53
ethnicity -0.85 -0.66 -0.76
gender 0.08 -0. 12 -0.02

 

In a school which had a 5% minority composition, the mean in science would be
predicted to increase by 0.054. In a school with a minority enrollment of 90%, the mean

would be predicted to decrease by 0.325. In math, predicted scores are shown separately

91

for a school with 5% minority enrollment and a school with 90% minority enrollment,
because this factor was associated with (interacted with) changes in other factors. Note
that means are adjusted for student-level ethnicity and gender (except when each is the
target variable), so the mean score for the 90% minority school is an estimate of the mean
if the school effect related to proportion-minority stayed the same while the ratio of Black
to White students was adjusted to reflect the proportions in the total sample. This makes
more sense if one regards the school proportion-minority as merely a proxy variable for
school and community resources.

The lower means of science scores in high-minority schools applied to both Black
and White students within the schools, but the mean of the Black students in the total
sample was more influenced by the low school means in the high-minority schools. The
within-school ethnic difference was smaller than the difference between the total sample
means of the two ethnic groups. Using the reliability-weighted means for the groups in
the total sample, the standardized ethnic difference was -0.87 on the multiple choice score,
-0.95 on the constructed response score, and -0.89 on the average score. To a lesser
degree, because the school proportion minority influenced the format effect in math, the
format by ethnicity interaction in math was slightly smaller in the total sample than within-
schools. The standardized ethnic difi‘erence was -0.83 on the multiple choice score, -0.67
on the constructed response score, and -0.75 on the average score. Again, the
standardized difi‘erences were computed with the total-sample standard deviation, not the
pooled within-group standard deviation (the standard deviations in Tables 17 and 19 can

be used to calculate pooled within-group standard deviations).

92

Next, each school was allowed to have a random efi’ect on the test stakes efi'ect.
In science, the variance of these random terms was slightly higher than the variance of the
school means, and both were almost four times as high as the variance in school format
efi‘ects (see Table 21). When random school efi‘ects for the ethnicity and gender effects
were added to the science model (individually), the estimation algorithm failed to converge
in 100 iterations. At that point, estimates suggested that the random effects variance was
not significantly greater than zero. In math, the variance in the school stakes effects was
higher than the variance in school means but lower than the variance in school format
efi’ects (Table 22). The variance in school gender efi‘ects was smaller than the other
variances. A model with a random term for the school ethnicity effect (including random
means and format efi'ects) failed to converge, and intermediate steps indicated the variance

was not significantly greater than zero.

Table 21 - Vao'ance Commnents for Science Abtlhy' '

 

Standard Variance '

 

Effect Deviation Component df Chi-square P-value
IN'I'ERCEPTl/INTERCEP'N, U0 0.18020 0.03247 26 258.5801 0.000
INTERCEPTII STAKES, U1 0.19291 0.03722 27 100.2475 0.000

FORMAT/INTERCEPT2, U2 0.04986 0.00249 27 54.5586 0.002

 

Table 22 - Variance Components for Math Abilig

 

 

 

Standard Variance
Effect Deviation Component df Chi-square P-value
MCEFTIWTERCEPIY. U0 0.22282 0.04965 30 333.6813 0.000
WTERCEPT 1/ GENDER U1 0.29751 0.08851 30 166.6610 . 0.000
FORMAT/INTERCEPT2, U2 0.32318 0.10445 30 1505.8560 0.000
WTERCEPTl/H‘JTERCEP'U. U0 0.22059 0.04866 30 392.8767 0.000
lNTERCEPTl/ STAKES. U1 0.13379 0.01790 30 59.2044 0.001
FORMAT/IN'I'ERCEP'I'Z, U2 0.32407 0.10502 30 1510.5385 0.000
F i f the Hi - Res onses w-Stak 5 Item Estimates

Potson-Fit

Item parameters were re-estimated from the low-stakes-students’ responses. All

items were calibrated together, and separate estimations were made for boys and girls.

Estimates were made twice, once treating omitted items as incorrect and again treating

omitted items as not-administered. Each high-stakes-student’s ability was re-estimated

based on these item estimates (using only the gender appropriate item estimates, and

always scoring omitted items incorrect regardless of how omits were scored for the item

estimates). The high stakes-students’ fit to these item estimates was measured by the

standardized appropriateness index of Levine, Drasgow, and erliarns (1985), a composite

index based on how probable the student’s response to each polychotomous item is, given

the student’s estimated ability and the item’s estimated parameters. This index has been

shown to be roughly normally distributed in actual use (Levine, Drasgow, & Williams,

94

1985). Values less than zero indicate lower than expected fit, and values less than -2.58
would be rare, occurring for only 0.5% of the students in a sample where the model fit.
Table 23 shows the mean standardized appropriateness fit, as well as the percent of

students with indices less than -2.58, for each condition.

Table 23 - Appropriateness Fit Index

 

 

Gila he
m mean % <-2.58 mean % <-2.58
omitted items scored as incorrect -0.28 1.77% -0.11 0.43%
omitted items treated as not-administered -0.25 1.43% -0.05 0.36%
M
omitted items scored as incorrect -0.43 8.06% -0.53 9.07%
omitted items treated as not-administered -0.24 7.09% -0.25 7.26%

 

In science, the low-stakes item estimates fit better for the boys than for the girls.
Fewer students were judged as misfitting when the item estimates were obtained by
treating the omits as not-administered in the low-stakes test (though they were treated as
incorrect on the high-stakes test). In math, the low-stakes item estimates fit better for
girls than for boys, especially when omitted items were scored as incorrect. For both
genders, item estimates fit better when omits were treated as not-administered. However,
all sets had very large numbers of misfitting persons.

The appropriateness index was also calculated using the operational item estimates
(ignoring gender), to serve as a baseline. The distributions were not precisely normal: In

science, the standard deviation was 0.90, the skew was -0.50, and the kurtosis was 0.48.

95

In math, the standard deviation was 0.99 (essentially 1), the skew was -0.61, and the
kurtosis was 0.46. The mean was 0.01 in science and 0.04 in math, and only 0.7% of the
students in science and 1.60% in math had values less than -2.58 (somewhat more than the
0.5% expected if normally distributed).

lam-lit.

The appropriateness index targets students whose responses poorly fit the
expected pattern. Another index, OUTFIT (Wright & Masters, 1982), targets items which
fit the model poorly. The expected value of this index is one. Values greater than one
indicate many responses are unexpected, and values less than one indicate responses are
“too predictable”-there is not as much unexplained variance as expected.

The four sets of item estimates (gender by method of treating omits) based on the
pilot data and responses from the operational data were used to generate four OUTFIT
indices for each item. For every set, one science item (a constructed response item) had
OUTFIT greater than 1.5. OUTFIT for this item was larger for girls than boys, and it was
larger when omits were treated as missing (not wrong) in the pilot data (recall that omits
were always treated as wrong in the operational data). In math, two constructed response
items had OUTFIT greater than 1.5 for all four sets, a third constructed response item had
OUTFIT of 1.46 for one group and above 1.5 for the other three, and the multiple choice
item composed of four individual items had OUTFITS between 1.28 and 1.52. Averages

for each of the four sets, by item type, are displayed in Table 24.

Table 24 - mom

96

 

m

Multiple Choice

ConstructedResponse

Mal-h

Multiple Choice

ConstructedResponse

 

 

Bars §'_3
omitsasincorrect omitsasmissing omitsasincorrect omitsasmissing
mean sd mean sd mean _ sd mean sd
0.9827 0.1894 0.9884 0.1972 1.0296 0.1306 1.0371 0.1391
0.9998 0.2971 1.0631 0.2375,, 1.0407 0.2922 1.0984 0.2457
1.0445 0.1637 1.0051 0.1400 1.0646 0.1566 1.0387 0.1296
1.6634 0.8989 1.4258 0.5221 1.3589 0.5250 1.2876 0.4455

 

A repeated-measures AN OVA, with item as the unit of analysis, was conducted on

the OUTFIT measures. There were two repeated measures, gender and method of

treating omits, and one between factor, item type.

 

 

Table 25 - ANQyA Sm Tghlo for OUTFIT, Scienoo

Source SS MS df F prob. F
Item Type (IT) 0.058 0.058 1 0.46 0.4999
error (between subjects) 4.592 0.124 37

Not administered/Incorrect (NA) 0.018 0.018 1 17.03 0.0002
NA X IT 0.025 0.025 1 24.00 <0.0001
error - 0.038 0.001 37

Gender (G) 0.044 0.044 1 2.96 0.0935
G X If 0.008 0.008 1 0.05 0.8194
error 0.555 0.015 3.7

NA X G 0.000 0.000 1 0.08 0.7772
NA X G X IT 0.000 0.000 1 1.32 0.2573
error 0.003 0.000

 

Table 26 - ANOVA Sm Table for OUTFIT, Math

 

 

Source SS MS df F prob. F
Item Type (IT) 3.114 3.114 1 12.12 0.0014
error (between subjects) 8.475 0.257 33
Not administered/Incorrect (NA) 0.174 0.174 1 9.38 0.0043
NA X Tl‘ 0.074 0.074 1 . 3.97 0.0546
error 0.612 0.019 33
Gender (G) 0.188 0.188 1 6.09 0.0190
G X IT 0.306 0.306 1 9.92 0.0035
error 1.019 0.031 33
NA X G 0.040 0.040 1 10.70 0.0025
NA X G X IT 0.029 0.029 1 7.73 0.0089
error 0.124 0.004 33

 

In science, the method of treating omits efl‘ect and the method by item format
interaction were statistically significant. Misfit was higher when omits were treated as
not-administered (opposite the finding for the appropriateness index), and the gap was
larger on constructed response items than on multiple choice items. Gender did not have a
significant main efi‘ect on fit, nor did it interact with method of treating omits or item type.
The trend was for OUTFIT to be greater for girls, but this efl’ect was not statistically
significant.

In math, the three-way interaction among way of treating omits, gender, and item
type was statistically significant. For both genders, there was an interaction between
treatment of omits and response format; treating omits as not-administered improved fit
more for the constructed response items than for the multiple choice items. This
interaction was more extreme for boys than for girls.

Both the person-fit and the item-fit were extremely poor in math. One possible I
reason is because the equal-slopes model was more problematic in math than it was in

science. Half-points were possible for the constructed-response items, and in order to

98

utilize all information fi'om these half-points and yet avoid overweighting the constructed-
response items, the slope of each constructed-response item was set to halfthe slope of
each multiple choice item for this study (the slope determines the weight of the item, and
each category of the constructed response items was to be weighted a half point). To
check the appropriateness of this ratio, a two-parameter model was run, using the high-
stakes data. When the slopes were free to vary, the ratio of the average constructed
response slope to the average multiple choice slope was about 0.6 in science, fairly close
to the imposed 0.5 ratio. In math, however, this ratio was approximately 0.3. This
probably contributed to the poor fit. The average OUTFIT for the constructed response
items based on the operational item estimates was 1.265, which was better than the
OUTFIT based on the pilot item estimates but still quite high for an average. However,
the person-fit of the operational data using operational item estimates, as described above,
was considerably better. Much of the poor fit, therefore, seems to be due to difi‘erences

between pilot and operational responses.

 

CHAPTERS

DISCUSSION/SUMMARY, IMPLICATIONS, AND CONCLUSIONS

Discussion/Sum

On the constructed response items, the stakes of the test, student ethnicity, and
student gender were associated with response rate, with the school mean response rate
controlled. Students were more likely to respond under high stakes, girls were more likely
to respond than boys, and Whites were more likely to respond than Blacks. These factors
did not interact with each other. The higher response rate under high stakes suggests that
some of the non-response under low-stakes was due to lack of motivation and not solely
to lack of knowledge. The lack of interaction between stakes and ethnicity or gender
suggests that, within a school, Blacks and Whites, and boys and girls, are equally affected
by test stakes and the ethnic and gender difi‘erences do not change with the test stakes.

The ethnic composition of the school was significantly associated with the effects
of test stakes and gender, but not with the efl‘ects of ethnicity. Schools with more
minority students had a larger stakes efi‘ect. In schools with high minority enrollments,
students appear to be particularly unmotivated (as evidenced by response rate) on low-
stakes tests. When the minority enrollment was not included as a factor, and the stakes
effect was held constant at the average across schools, there was an interaction between
ethnic group and test stakes. This was apparently due to between-school difi‘erences, not

99

1 00
to ethnic difl‘erences within schools, because it disappeared when school minority
enrollment was added to the model. [Though in the second cohort there appeared to be
some stakes by ethnicity interaction within schools as well]. Looking at the means for
each condition, the response rate of Blacks on the high stakes test would appear to be
underestimated (relative to the scores of Whites) by the low stakes test when all schools
are pooled together.
mm

In math, students were more likely to leave constructed response items blank than
they were in science. This suggests students might have had more dificulty thinking of a
relevant solution to the math constructed response items, and were not just poorly
motivated. Or perhaps many students are more familiar with making written verbal
responses in science than in math. In math, their experiences with constructed response
items might be limited to showing their computations, while the HSPT requires students to
explain their reasoning and solutions.

The stakes of the test, gender, and the school's ethnic composition were all related
to tendency to respond to the constructed response items. Students were more likely to
respond under high stakes, and girls were more likely to respond than boys. Students in
schools with many non-White students were less likely to respond than students in mostly-
White schools. Once this efi‘ect was controlled, ethnicity was not associated with
response rate; the sizable difi’erences between the means for Blacks and Whites could be
attributed to between-school difi‘erences. This is in contrast to the findings in science,
where there were still some within-school ethnic group difi‘erences. However, any

conclusions about ethnic efi‘ects at the student level should be tentative because of the

1 0 1
large number of students who did not provide their ethnicity on the high stakes test and
thus were not included, especially in the schools selected for the math analysis.

The results in math also difi‘ered fiom the results in science in that the school
proportion of minority students did not interact with stakes or gender when the outcome
was in the log-odds metric (there appeared to be a stakes by proportion minority
interaction using the probability metric, but, as noted, this was not tested for statistical
significance). Schools with high non-White enrollment did not consistently difi‘er fi'om
mostly-White schools in the degree of the stakes or gender efl’ect (when using the log-
odds metric), only in the school mean efi‘ect (intercept).

ili r : 'en

The stakes of the test, student ethnicity, and student gender were also associated
with test scores. Students scored higher under high stakes, Whites scored higher than
Blacks, and boys scored slightly higher than girls. These effects, though, all significantly
interacted with response format. Because the residual variance of the format efi’ect within
schools seemed to be very small (not significantly different from zero), these student-level
characteristics would not be expected to influence the format efi‘ect. Ifthe formats were
measuring the same thing, as suggested by the lack of variance in format difl‘erences, there
would be no reason to think gender, ethnicity, or stakes would interact with format. The
evidence that the format effect does depend on stakes, gender, and ethnicity suggests that
the two formats are not measuring exactly the same thing.

High stakes tended to increase scores for both response formats, but the efi’ect was
larger for constructed response items. This was expected based on findings fiom low-

stakes tests that motivation is lower on constructed response items (Sundre, 1996;

102

Wainer, 1993). Ifmotivation was more equal across item format on the high stakes tests,
the format efi‘ect would be expected to be smaller on the high stakes test, as was found in
this study. This is also consistent with Wolf, Smith, and Birnbaum’s (1995) finding that
“mentally taxing” items were less dificult (relatively) under high stakes.

Whites scored higher on both formats, but the difi’erence was slightly larger on the
constructed response scale in the first cohort and slightly smaller on the constructed
response scale in the second cohort.

The gender gap changed direction depending on the response format; boys did
better on the multiple choice subscale and girls did slightly better (about the same) on the
constructed response subscale. Using only the pilot data, and ignoring the school efl’ect,
DeMars (in press) found a trend in this direction (significant on one test form but
significant only for certain ability groups on another), which is apparently confirmed with
the operational data. This interaction has been consistently found on Advanced Placement
exams in the sciences (Bridgeman, 1989; Bridgeman & Lewis, 1994; Schmitt, Mazzeo, &
Bleistein, 1991).

Schools with high minority enrollment had lower mean scores (in the second
cohort, there was a larger ethnic difi‘erence within-schools, but not a significant difference
between schools when student characteristics were controlled). However, minority
enrollment did not significantly influence the other effects (except gender and gender by
format in the second cohort). Because the ethnic composition of the school had an effect
on differences in response rate due to test stakes, it might also be expected to influence
difi‘erences in test scores due to test stakes, at least for the constructed response items.

While schools with high minority enrollments had a larger increase in response rate under

103

high stakes than schools with lower minority enrollments, the increase in test scores did
not depend on minority enrollment. Under high stakes, test scores increased about equally
for schools with many or few minority students.

The stakes of the test and student ethnicity were associated with test scores. On
average, students did better under high stakes, and Whites did better than Blacks. On the
total test score, there was essentially no gender difi’erence, but there was an interaction
between gender and response format. Girls scored higher on constructed response items
and boys scored higher on multiple choice items. This same interaction was found in
science. It is interesting that math (calculus) is the one area on the AP exams where this
type of interaction is not found (Schmitt, Mazzeo, & Bleistein, 1991). Also, no
interaction was found between response format and gender on several math tests of the
General Certificate Exam (GCE) in England (Murphy, 1982). However, the AP calculus
exams generally do not require much verbal reasoning/explanation on the constructed
response items, in contrast to the HSPT. The interaction found here on the HSPT was
similar to that found with a sample of Irish students (Bolger & Kellaghan, 1990).

Stakes and ethnicity also interacted with response format (though ethnicity was not
quite statistically significant at the .05 level, and many students provided no ethnic
information, so this effect should be interpreted cautiously until it is replicated with other
samples). As in science, under low-stakes students did more poorly on the constructed
response items than the multiple choice items (in the high stakes sample, the difi‘erences
were set to approximately zero on the ability scale). The ethnic gap within schools slightly

decreased on the constructed response items, compared to a slight increase in science (first

 

104

cohort). A decrease in ethnic group differences would be consistent with Badger‘s (1995)
findings in math, but in that study the interaction was larger. On the HSPT, I would
conclude that ethnic group differences remained largely stable across response format.
Also, the efi‘ect of ethnicity at the individual level should be interpreted in the context of
the efi’ect of school minority composition in math. This efi‘ect, while small, was in the
opposite direction; as the proportion of minority students increased, the format efi‘ect
increased (became more negative).

There was also a significant interaction between school minority composition, test
stakes, and response format. In schools with many non-White students, students taking
the constructed response items under low-stakes scored particularly low. In these schools,
constructed response performance increased more under high-stakes conditions than
would be expected in an average school. This efi‘ect was not found in science. In science,
the only efi’ect of school ethnic composition was on mean scores or gender efl‘ects,
depending on the cohort. The ethnic group differences in math ability scores seemed to be
mainly within schools, not between schools (though, again, lack of ethnic information at
the student level may have led to bias in estimates of ethnic efi’ects if the remaining
students were not representative of their groups). This is particularly interesting given
that the ethnic group differences in response tendency in math seemed to be between
- schools, not within schools.

Commeot on tho Etmorty' Varighlos

The largest discrepancies between math and science, and between the two science

forms, involved the efi’ects of student ethnicity and school proportion-minority. Some of

these difi‘erences may have been due to the omission of students who did not identify their

l 05

ethnic groups; the type of student not indicating ethnicity may have varied in the different
samples. For example, the test score difi’erences between students who did and did not
indicate their ethnicity were slightly greater in math than in science. At the school level,
ethnic composition was most likely fairly accurate, as it was based on the pilot data (where
ethnic identification was much higher) for schools which had low response to the ethnicity
item on the operational test. However, only a relatively small number of schools (about
30) were used for each test form, which would tend to lead to less stable results than a
larger number of schools. Furthermore, student-level ethnicity and school ethnic
composition are correlated, so it is dificult to separate the efi’ects of each.
Pilot Itm Fit to Mional Responses

Because boys were less likely to respond to constructed response items, I expected
that, for boys, ignoring omitted items on the pilot test when estimating item parameters
would lead to item estimates which fit the operational data better than parameters
estimated when omitted items were scored zero on the pilot test. In science, ignoring
omitted items (treating them as if they had not been administered), improved person-fit
(the appropriateness fit index) only slightly, and it led to somewhat poorer item-fit for the
constructed-response items. These difi‘erences were small, and probably have little
practical meaning. Fit (both item-fit and person-fit) was worse for girls than boys, which
was an unexpected finding. Because boys seemed to be less motivated on the pilot test (as
evidenced by lower response rates on the constructed response items), I had expected
them to have more idiosyncratic responses which would lead to poorer item estimates.

In math, my expectations held. Item fit and person fit both were better when

omitted items were treated as not—administered, especially for boys and constructed

 

106

response items. In math, the fit was generally much worse than it was in science, so there
was more opportunity for differences. Also, in math non-response was higher, so method
of treating omits would be expected to make more of a difi‘erence. The fit of the persons
and of the constructed response items, though, regardless of how omits were treated, was .
remarkably poor. About 13 to 17 times more people than expected had very poor fit (z-
scores less than -2.58), and this was mostly due to the poor fit of the constructed response
items. The average outfit of the six constructed response items was 1.29 - 1.66
(depending on the group and conditions). For an individual item this would be 30%
higher than expected if there were good fit, and for an average across multiple items it is
quite high. This poor fit was partly due to the constraint of equal slopesuwhen the slopes
were he to vary, the slopes of the constructed response items were not as steep as those
of the multiple choice items.

One limitation to this study is that there was no measure of motivation other than
tendency to omit constructed response items. As noted earlier, students may omit items
for other reasons, such as little or no knowledge of the correct answer. It would be useful
to have data on some other measure of motivation to separate these reasons for omission.
Also, students likely varied in how important they felt the diploma endorsements were,
depending on their interpretations of the messages they received fi'om the schools, parents,
other students, and the media about the diploma endorsements. These student beliefs, in
turn, had an impact on how motivated the students were.

Another issue was the problem of disentangling student-level and school-level

factors. Within schools, ethnic difi‘erences in response rate in science were relatively small

 

1 07
(though significant), comparable in size to gender differences, and in math the within-
school ethnic difl‘erences were not significantly different fiom zero. However, the
differences between high-minority schools and low-minority schools were larger, and a
large percentage of the Black students attended high-minority schools. The school is
essentially a proxy variable for a composite of school, neighborhood, and family efi‘ects.
Students can not be randomly assigned to schools, so there is no easy way of assessing
what the within-school ethnic effect would be if non-school background variables were
controlled.

Determining the proportion of minority students in a school was in itself a
problem. On the operational test, 26% of the tested students in the first cohort and 30%
of the students in the second cohort did not identify their ethnic group (even more in the
selected schools). In some schools, nearly all students responded, while in others none of
the students responded. In the pilot test, at least 80% of the students in each school
supplied their ethnicity, so these data could be used for schools which had low-response
on the operational test. However, this solution was available only because this study
design included only the schools which participated in the pilot test. This option would
not be available in most situations. Also, this does nothing to help the problem of missing
data at the student level. Though score differences between students who answered the
ethnicity item and students who did not were not that great, this information was probably
not randomly missing. Teachers' tendencies to include this item as they led students
through the process of filling out background information could be associated with school
factors, and students' tendencies to complete this item when their classmates did not could

be associated with other student factors.

108

The generalization of these findings should be limited to relatively structured tests.
In particular, the findings concerning the constructed response items should not be
extended to all forms of performance assessments without firrther research. The
constructed response items were intended to elicit scientific/mathematical reasoning and
communication skills. However, all situations and materials were presented on paper only,
and all responses were in written form. Somewhat different skills and processes may be
involved when students interact with real materials, record their own observations, work
with others, and communicate verbally.

The content level of the tests also limits the generalizability of the findings. The
level is above the “basic-skills” level, but it includes only concepts to which all students
should have been exposed by the end of the 10th grade. Results might be difi‘erent for
advanced content or high-achieving students (such as students taking Advanced Placement
exams) and different again for low-level, nrinirnal competency exams. This research
focused on a cross-section of students and “typical” required high school curricula.

mm
Impligtions for T ost-Devolopmcnt

The implications for test-development follow directly fiom the results. When the
dependent variable was the test score (rather than response rate), gender did not interact
with test stakes. Ethnicity did not interact with test stakes in science, and the interaction
in math (school proportion-minority, format, and stakes interacted) was quite small. This
suggests that gender and ethnic difi'erences in test scores can be accurately estimated on-
low stakes tests; all groups increased similar amounts when the stakes of the test

increased. Iftest-developers were trying to predict how a test cut-score would affect

1 09
these groups (Blacks and Whites, boys and girls), they would only need to have some idea
of how the overall mean would shift. [Of course, if the cut-score were far from the mean
this would not be accurate because it is very possible that group difi‘erences new the mean
are not the same near the tails of the distribution] The response rates of students in high-
minority schools increased more as the test stakes increased in science, but this did not
lead to a disproportionate increase in test scores.

All these efi‘ects were moderated by the format of the items. If scores were
estimated separately for each response type, increasing the stakes of the test would have a
bigger efl‘ect on the constructed response scores than on the multiple choice scores. The
estimated dificulties of constructed response items relative to multiple choice items on
low stakes tests, then, will be ofi‘ somewhat compared to the relative dificulties estimated
under high stakes. Therefore, the item dificulties fi'om pilot forms could lead to
inaccuracies if used to equate test forms to be administered under higher stakes. Ideally,
information fi'om the high stakes administration should be used in the final equating (as is
done for the HSPT).

Ethnic differences seemed about the same on both formats, when school ethnic
composition and student-level ethnicity were considered simultaneously; in math, the
ethnic difi’erence was somewhat larger on the multiple choice items, but in science the
difi’erence was larger on the constructed response items. Gender differences varied with
the response format. Using only constructed response items, gender difl‘erences would
appear to slightly favor girls instead. Adding constructed response items of this type

would boost the relative position of females.

110

Broodor Impligtions for Edocatots go Polig-Mahors
In addition to the implications summarized above, there are some broader

implications which are supported by the results, yet move a step further. These are the
issues which warrant attention on a policy level. One finding which should be of interest
to educators and policy-makers is that some of the ethnic differences could be attributed
to school-level difi‘erences rather than differences between Blacks and Whites in the same
schools. If some of the related causal factors in the school or community can be
identified, efi’orts can be directed toward changing these factors on a schoolwide basis.

The use of multiple response formats is another topic highlighted by these findings.
One important issue is the question of what each format measures. The high correlation
between the formats suggests they measure something similar, and the fact that the items
are estimated on one scale and a single score is typically estimated (the subscales were
created only for this study) suggests that the educators involved in the test development
believe a single predominant factor is being measured. The constructed response items,
though, are not completely redundant and do add information in estimating this single
score. Both the multiple choice and constructed response items are designed to measure
concepts in the Michigan Essential Goals and Objectives, and each section contributes
something slightly different towards that purpose. If both formats measure slightly
different, but relevant and intended constructs, then both are useful. Ifthe construct-
irrelevant sources of variance measured by the formats essentially "balance-out", then that
would also be beneficial.

Some educators have advocated using performance-type assessments to drive

instruction (Pomplun, 1997), though this departs from the traditional purpose of tests.

111

Though both response formats may measure very similar constructs, teachers may
perceive constructed response items as measuring more reasoning and problem-solving,
and adjust their instruction and assignments accordingly (F rederiksen, 1984). F rederiksen
also suggested that parents, who have influence on what is taught in schools, are much
more likely to be aware of the need for teaching the types of problem-solving used in
responding to open-ended items if those items are part of the testing system.

Bennett, Rock, and Wang (1991), in the context of the AP computer science
exam, also raised the issue of instructional focus in response to testing. While they found
the multiple choice and constructed response portions of the test to be highly correlated,
they suggested that one justification for keeping the constructed response items was to
keep the emphasis in the class on computer programming (the constructed response items
assessed students’ abilities to write programs).

Stecher and Hamilton (1994) examined changes in instruction in Vermont after the
state mandated portfolio assessments for fourth and eighth grade mathematics. Teachers
reported they had students spend more time on problem-solving, class discussions, and
explaining and writing about their solutions. Teachers also assigned more problems with
ill-defined outcomes, used more hands—on materials, and had the students work together
more often. Changes were larger for fourth grade teachers than for eighth grade teachers.

One important difi‘erence between portfolio assessment and the constructed
response items on the HSPT is that, for portfolio assessment, classroom activities are not
necessarily preparation for the assessment; they may actually be part of the assessment. At
the extreme, if a teacher previously had not assigned any activities which would be

appropriate for the portfolios, he or she would have to add at least enough activities to

112

build a minimal portfolio. In the case of the HSPT, such an extreme teacher would not
have to add any activities beyond the two-hour test itself.

The Kansas Mathematics Assessment is more comparable in format to the HSPT.
In Kansas, teachers reported increases in emphasis similar to the Vermont teachers (for
example, problem-solving, communication, reasoning, estimation, multiple solutions) in
response either to the testing program or to the new state curriculum which went with it
(Pomplun, 1997). Change was greatest among elementary school teachers and lowest
among high school teachers. Teacher attitude toward the test, and district and building-
level responses influenced the amount of change.

Cost is also an important issue for statewide testing. Wainer and Lukhele (1997)
estimated that a 30-minute essay (in student time) costs about $7 to score. Stecher
(1995), using costs from standardized tests, estimated that constructed response tests
(with no hands-on activities) cost about 15 times as much as multiple choice tests of
comparable length (in testing time). Further, if hands-on activities and trained test
administrators are used (the HSPT uses neither), the cost is about 100 times the cost of
multiple choice testing. Given such estimates, the constructed response items must give
additional, meaningful information if their cost is to be justified.

Dirthigns for Further Rflch

Further research into the constructs measured by each format, and their possible
effects on teaching and learning, would be usefirl to educators weighing the costs and
benefits of the different formats. As noted, the constructs measured by the constructed
response and multiple choice items are highly correlated. Though the constructed

response items do add information to the scores, additional multiple choice items might

113

add more information at a cheaper cost. Ifthe constructed response items are intended to
reinforce certain instructional methods, it would be reasonable to expect research, perhaps
a survey, to explore this relationship. Ifthe constructed response section is intended to
boost teacher/student/community perceptions and attitudes toward the test, it would make
sense to study perceptions and attitudes. Such findings could help in determining the
direction of future test development.

A large part of the ethnic difi‘erence in performance in science was due to between-
school differences. More school-level factors could be entered into the model to examine
some factors which might explain these school difi‘erences. School climate, curriculum,
and community factors are likely associated with both ethnicity and school performance.
After measures of these factors were included, the school's ethnic composition would
likely account for little additional variation. School climate and curriculum are also
potentially changeable, so lmowing more about how these factors relate to student
achievement would be usefirl to policy-makers.

Further research using other tests and student populations will help to reveal to

which situations the results of this study generalize.

APPENDIX

RESULTS FROM THE SECOND YEAR OF THE SCIENCE TEST

RESULTS FROM THE SECOND FORM or THE SCIENCE TEST

Table A1 - Mam

APPENDD(

 

 

White Females Black Females White Males Black Males
High Stakes 1454 241 1311 211
Low Stakes 519 68 547 57

 

Average scale score in tested population (Blacks and Whites only): 387.06 (sd = 42.67)

Average scale score in selected schools (Blacks and Whites only): 389.58 (sd = 42.01)

Table A2 - Swim-Mel Prfl'gtggs for Log-Odds of Respong, Science Ygfl 2

 

For INTERCEPT], «0,1
For INTERCEPT2, Book
INTERCEPT3, 7000
FOI’ STAKES (S), 301k
INTERCEPT3, ‘Yom
For ETHNICITY (E), 502..
INTERCEPT3, 7020
For GENDER (G), Boar:
INTERCEPT3, ‘Yo3o
For S X E, [304.I
INTERCEPT3, 7040
For S X G, Bog:
INTERCEPT3, 7050
For E x G, Boa
INTERCEPT3, ‘Yoso
FOl' S X G X E, B011:
INTERCEPT3, 7070

 

Standard Approx.
Coeficient Error T-ratio d.f P-value
2.800651 0.095298 29.388 37 0.000
0.933603 0.074453 12.540 43 69 0.000
-0.810397 0.197113 -4.111 4369 0.000
-0.533 162 0.069176 -7.707 4369 0.000
0.794302 0.221220 3.591 4369 0.001
-0.010617 0.138337 -0.077 4369 0.939
-0.295909 0.198696 -1.489 4369 0.136
0.001481 0.396929 0.004 4369 0.997

 

114

115

Table A3 - Log-Odds of Response on Science CR Items. Full Model Yefl

 

Standard Approx.

Coeficient Error T-ratio d.f P-value
For INTERCEPTI, nojk

For INTERCEPT 2, Book

INTERCEPT3, 7000 2.891245 0.072705 39.767 36 0.000

MINORITY, 1001 -1.066216 0.307136 -3.471 36 0.002
For STAKES (S) , 501k

lNTERCEPT3, ‘Yoro 0.716872 0.100991 7.098 4368 0.000

MINORITY, You 0.988755 0.683999 1.446 4368 0.148
For ETHNICITY (E), 6021:

INTERCEPT3, yam -0.267060 0.209546 -1.274 4368 0.203

MINORITY, you -0.628945 0.407656 -1.543 4368 0.123
For GENDER (G), 303k

INTERCEPT3, 7030 -0.608106 0.097247 -6.253 4368 0.000

MINORITY, 1031 -l.090945 0.537663 -2.029 4368 0.042
For S x E, Boa:

INTERCEPT3, 7040 -0.999135 0.363708 -2.747 4368 0.006

MINORITY, 7041 1.521008 0.917019 1.659 4368 0.097
For S X G, BOSk

INTERCEPT3, 7050 0.335921 0.161174 2.084 4368 0.037

MINORITY, 7051 -0.210093 1.443877 -0.146 4368 0.885
For E X G, BOG]:

INTERCEPT3, yooo -0.410597 0.323266 -1.270 4368 0.204

MINORITY, you 1.591441 0.851798 1.868 4368 0.061
For S.X GX E, Bonn

INTERCEPT3, 7070 2.444570 0.575440 4.248 4368 0.000

MINORITY, 7071 -3.264864 1.735777 -1.881 4368 0.060

 

 

 

116

TableA4- - s fR ns n cien RItems FinalM elYearZ

 

Standard Approx.
Coemcient Error T-ratio d.f P-value

For INTERCEPTI, nojk

 

 

 

For INTERCEPT2, Boot
INTERCEPT3, yooo 2.844423 0.065017 43.749 36 0.000
MINORITY, yooo -1.264149 0.264041 -4.788 36 0.000
For STAKES, Bork
INTERCEPT3, 7010 0.856606 0.092626 9.248 4368 0.000
MINORITY, 7011 1.052756 0.155070 6.789 4368 0.000
For ETHNICITY, [302k
INTERCEPT3, 7020 -0.489279 0.144558 -3.385 4369 0.001
FOI' GENDER 303k
INTERCEPT3, you -0.500459 0.078295 -6.392 4368 0.000
MINORITY, you -0.374395 0.124985 -2.996 4368 0.003
Table A5 - Frayed Log-Odds, Prghflilig (Qte), and Odds of Respong, YE 2
probability
log-odds probability ratio odds odds ratio
population average
gender 1.03 1.65
girls 3.08 0.96 21.78
boys 2.58 0.93 13.21
stakes 1.05 2.36
high stakes 3.27 0.96 26.38
low stakes 2.42 0.92 11.20
ethnicity 0.97 0.61
Black 2.42 0.92 1 1.29

White 2.91 0.95 18.42

117

Table A6- Schools with High Ed Low Minority Enrollment, Science Year 2

 

 

 

5% Minority probability
lg—odds probability ratio odds odds ratio
population average 3.12 0.96 22.65
gender 1.02 1.55
girls 3.33 0.97 27.85
boys 2.89 0.95 18.00
stakes 1.03 1.97
high stakes 3.46 0.97 ' 31.79
low stakes 2.78 0.94 16.14
ethnicity 0.98 0.61
Black 2.70 0.94 14.88
White 3.19 0.96 24.27
90% Minority probability
log-odds probability ratio odds odds ratio
population average 1.74 0.85 5.70
gender 1.12 2.13
girls 2.10 0.89 8.15
boys 1.34 0.79 3.83
stakes 1.28 4.82
high stakes 2.53 0.93 12.52
low stakes 0.95 0.72 2.60
ethnicity 0.92 0.61
Black 1.32 0.79 3.75
White 1.81 0.86 6.11

 

118

Table A7 - Student-Level Predictors for Abiligg Scores, Science Year 2

 

Standard Approx.

 

Coefficient Error T-ratio d.f P-value
For INTERCEPTI, nojk

For INTERCEPT2, Book

INTERCEPT3, yooo -0.368311 0.032772 -11.239 37 0.000
For STAKES (S), [ionK

INTERCEPT3, 7010 0.628454 0.037958 16.557 4331 0.000
For ETHNICITY (E), Bozr:

INTERCEPT3, 7020 -0.792693 0.104973 -7.551 4331 0.000
For GENDER (G), Boar:

INTERCEPT3, yo3o 0.090958 0.028244 3.220 4331 0.002
For S X E, [3041:

INTERCEPT3, yooo 0.015578 0.128680 0.121 4331 0.904
For S X G, [305k

INTERCEPT 3, yooo 0.074922 0.054425 1.377 4331 0.169
For E X G, Book

INTERCEPT 3, yooo -0.148572 0.064037 -2.320 4331 0.020
ForS XGXE, [3m

INTERCEPT3, yooo 0.025874 0.129166 0.200 4331 0.841

For FORMAT slope, 1:15.;

For INTERCEPT2, Bros

INTERCEPT3, yooo -0.303127 0.015753 -19.242 37 0.000
FOI' STAICES, Bug

INTERCEPT3, ‘Yuo 0.590857 0.024124 24.493 4407 0.000
For ETHNICITY, [3m

INTERCEPT3, 7120 0.114266 0.037900 3.015 4407 0.003
For GENDER, 613k

INTERCEPT3, y13o -0.248776 0.023390 -10.636 4407 0.000
For S X E, Bur:

INTERCEPT3, ‘Yuo -0.040230 0.071488 -0.563 4407 0.573
For S X G, 3151:

INTERCEPT3, yooo -0.047652 0.041683 -1.143 4407 0.253
FOI’ E X G, 8161:

INTERCEPT 3, 7160 0.123967 0.076168 1.628 4407 0.103
ForS XGXE, [3m

INTERCEPT3, 7110 -0.050973 0.094102 -0.542 4407 0.588

 

119

Table A8 - Predictfi Smres, Controlling School Minority Composition, Year 2

 

 

Standard Approx.
Coemcient Error T-ratio d.f P-value
For INTERCEPTl, nojk

For INTERCEPT2, Bock

INTERCEPT3, yooo -0.370712 0.040003 -9.267 36 0.000

MINORITY, 7001 -0.295758 0.206037 -1.435 36 0.160
FOI' STAKES, B011:

INTERCEPT3, ‘Yoro 0.630912 0.037307 16.911 4330 0.000

MINORITY, 7011 -0.021979 0.176277 -0. 125 4330 0.901
For ETHNICITY, Bonn

INTERCEPT3, yooo -0.755193 0.170286 -4.435 4330 0.000

MINORITY, 7021 0.110763 0.266260 0.416 4330 0.677
FOI' GENDER, 303k

INTERCEPT3, 7030 0.124936 0.035782 3.492 4330 0.001

MINORITY, 7031 0.042350 0.267658 0.158 4330 0.875
For E x G, Boa:

INTERCEPT3, yooo -0.044665 0.130526 -0.342 4330 0.732

MINORITY, 7041 -0.204481 0.403585 -0.507 4330 0.612

For FORMAT slope, 1cm

For INTERCEPT2, Bros

INTERCEPT3, yooo -0.301751 0.015070 -20.023 36 0.000

MINORITY, 7101 -0.159088 0.090054 -l.767 36 0.085
For STAKES, Bllk

INTERCEPT3, ‘Yrro 0.592530 0.024506 . 24.178 4406 0.000

MINORITY, 7111 0.010234 0.060442 0.169 4406 0.866
For ETHNICITY, I312:

INTERCEPT3, 7120 0.156582 0.067906 2.306 4406 0.021

MINORITY, 7121 0.062494 0.134940 0.463 4406 0.643
FOI' GENDER, Bug

INTERCEPT3, 7130 -0.267795 0.020918 -12.802 4406 0.000

MINORITY, 7131 0.120736 0.053582 2.253 4406 0.024

 

 

 

120
Table A9 - The PM Model for Sg'ence Ability Scores, Year 2
Standard Approx.
Coeficient Error T-ratio d.f P-value
ForlNTERCEPTl, nofii
For INTERCEPT2, Bock
INTERCEPT3, yooo -0.368950 0.032495 -11.354 37 0.000
For STAKES, BOlk
INTERCEPT3, 7010 0.629250 0.037422 16.815 4331 0.000
For E'I‘HNICITY, Bozk
INTERCEPT3, yooo -0.796182 0.103219 -7.714 4331 0.000
For GENDER, Bo3k
INTERCEPT3, Yoso 0.118441 0.028818 4.110 4330 0.000
MINORITY, 7031 -0.158953 0.048068 -3.307 4330 0.001
For FORMAT slope, nm
For INTERCEPT2, BlOk
INTERCEPT3, 7100 -0.298535 0.015854 -18.830 36 0.000
MINORITY, 7101 -0.107970 0.069178 -1.561 36 0.127
For STAKES, Bllk
INTERCEPT3, 7110 0.592874 0.023961 24.743 4407 0.000
For ETHNICITY, Bm
INTERCEPT3, 7120 0.163372 0.055858 2.925 4407 0.004
For GENDER, Brat
INTERCEPT3, 7130 -0.267749 0.020878 -12.825 4406 0.000
MINORITY, 7131 0.120501 0.053250 2.263 4406 0.024

 

 

121

Table A10 - Ave_rege Preg'gfi Scores in Science, Year 2

 

 

 

multiple choice constructed response mean
meg sd M Q man fl
average -0.22 0.85 -0.52 0.89 -0.37 0.80
high stakes -0.05 0.82 -0.06 0.84 -0.05 0.77
low stakes -0.39 0.91 -0.98 0.76 -0.68 0.77
Blacks -0.97 0.70 -1 . 13 0.74 -1.05 0.64
Whites -0.10 0.81 -0.42 0.87 -0.26 0.76
Boys -0.09 0.91 -0.53 0.92 -0.31 0.84
Girls -0.34 0.79 -0.51 0.86 -0.42 0.75
W 90% minorig

MC CR mean MC CR mm
average -0.23 -0.51 -0.37 -0. 18 -0.55 -0.37
high stakes -0.06 -0.05 -0.05 -0.02 -0.09 -0.05
low stakes -0.40 -0.97 -0.68 -0.35 -1.02 -0.68
Blacks -0.98 -1.12 -1.05 -0.94 -1.17 -1.05
Whites -0.11 -0.41 -0.26 -0.06 -0.45 -0.26
Boys -0.08 -0.51 -0.29 -0. 13 -0.60 -0.36
Girls -0.37 -0.51 -0.44 -0.23 -0.52 -0.37

 

 

The standard deviation of ability scores in the total sample was 0.87; this standard

deviation was used to calculate the standardized difl‘erences in Table All.

Table A11 - Steedgm Wim -§eheel Efi‘ects in Sg'eng, Year; 2

 

 

standardized difl‘erence
multiple eheiee genstructg respense 829188;
stakes 0.38 1.06 0.72
ethnicity -1.01 -0.82 -0.91
gender 0.29 -0.02 0.14

 

The standardized ethnic group differences across the total sample were -1.03 for multiple

choice, -.85 for constructed response, and -.95 for the average.

LIST OF REFERENCES

LIST OF REFERENCES

Abdel-fattah, A (1994, April). Comparing BILOG and LOGIST estimates for
normal, truncated normal, and beta ability distributions. Paper presented at the annual
meeting of the American Educational Research Association, New Orleans. (ERIC
Document Reproduction Service No. ED 374158)

Arvey, R D., Strickland, W., Drauden, G., & Martin, C. (1990). Motivational
components of test taking. Peesennel Psychelegy, fl, 695-716.

Asher, S. R. (1979). Influence of topic interest on black children's and white
children's reading comprehension. Chile Develepmegt, i0, 686-690.

Asher, S. R (1980). Topic interest and children's reading comprehension. In R J.
Spiro, B. C. Bruce, & W. F. Brewer (Eds), Th r ' ' ° '
(pp. 525-534). Hillsdale, NJ: Lawrence Erlbaum Associates.

Badger. E. (1989). Wand;
Quincy, MA: Massachusetts State Department of Education (ERIC Document
Reproduction Service No. ED 317573)

Badger, E. (1995). The efl‘ects of expectations on achieving equity in state-wide
testing: Lessons from Massachusetts. In M. T. Nettles & A L. Nettles (Eds), mm
. _ - - (pp. 289-308). Boston: Kluwer

 

Badger. 15.. & Thomas, B. (1989a). Wm
W. Quincy, MA: Massachusetts State Department of Education.
(ERIC Document Reproduction Service No. ED 317573)

Badger E» & Thomas B (198%) MM
W. Quincy, MA: Massachusetts State Department of Education.
(ERIC Document Reproduction Service No. ED 317573)

Badger, E., Thomas, B., & McCormack, E. (1990). W.
Boston: Massachusetts Educational Assessment Program, Massachusetts State
Department of Education (ERIC Document Reproduction Service No. ED 336400)

Bennett, RE, Rock, DA, & Wang, M. (1991). Equivalence of free-response
and multiple choice items. MW 28., 77-92.

Bernstein, M. R (195 5). Relationship between interest and reading
comprehension. WM 4.9.. 283-288.

122

123

Bolger, N., & Kellaghan, T. (1990). Method of measurement and gender
difl‘erences in scholastic achievement. Journal of Educational Measurement. 21, 165-174.

Bond, L. (1995). Unintended consequences of performance assessment: Issues of
bias and fairness. E u ti M rement: Issue and Practice 14 (4), 21-24.
Bridgeman, B. (1989). Comparative validig of m_ultiple choice and free-response

new on the advanced placement examination in biology (College Board Report No. 89-
2). (ERIC Document Reproduction Service No. ED 308 228)

Bridgeman, B., & Lewis, C. (1994). The relationship of essay and multiple-choice
scores with grades in college courses. Journal of Educational Measurement, 3_1, 37-50.

Brown, S. M., & Walberg, H. J. (1993). Motivational efi‘ects of test scores of
elementary students. 191ml of Educational Research, &, 133-136.

Burke, P. (1991). Y 1 adolescen te ut on 't make em
PM (Contract No. 0TA-H3-6110.0). Washington, DC: Oflice of Technology
Assessment. (ERIC Document Reproduction Service No. ED 378221)

Bryk, A, & Raudenbush, S. (1992). Hierarchical linear models. Newbury Park,
CA: Sage Publications.

Bryk, A, Raudenbush, S., Congdon, R (1996). I-ILM4: Hierarchical linear &
nonlinear modeling [Computer software]. Chicago: Scientific Software International.

Chandler, T. A, & Spies, C. J. (1981). Attribution as predictor of expectancy in
three component exams. T ' fP holo _8_, 174-175.

Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of
assessment in situational judgment tests: Subgroup differences in test performance and

facevalidity perceptions. li P h l 82 143-159.

Crooks, T. J. (1988). The impact of classroom evaluation practices on students.
BMW... 58 43 8-481

Curran, M. T., & Harich, K. R (1993). Performance attributions: Effects of
mood and involvement. f ' nal P h l 8_5_, 605-609.

Darling-Hammond, L. (1995). Equity issues in performance-based assessment. In

M T- Nettles & A L- Nettles (Ede). W
W (pp. 89-114). Boston: Kluwer Academic.

DeMars, C. (in press). Gender differences in mathematics and science on a high
school proficiency exam. Applig Mmrement in ELISE on.

Drasgow, R, Levine, M. V. ,erliams, E. A (1985). Appropriateness measurement
with polychotomous item response models and standardized indices.B mm
mm 38 67-86

Dreeban, R, & Gamoran, A (1986). Race, instruction, and learning. m
Melony—Rm. 51. 660-669-

124

F einberg, L. (1990). Multiple-choice and its critics. The College Beer'd Revim,
(157), 12-17, 30-31.

F rederiksen, N. (1984). The real test bias: Influences of testing on teaching and
learning. Amm'can ngchelegist, 32, 193-202.
Freund, D. S., & Rock, D. A (1992, April). A re ' ' investi ion 1‘

pagfl-mldng in 1990 NAEP gate. Paper presented at the annual meeting of the
American Educational Research Association, San Francisco. (ERIC Document

Reproduction Service No. ED 347189)

Gao, X., Shavelson, R J ., & Baxter, G. P. (1994). Generalizability of large-scale
performance assessments in science: Promises and problems. Applied Mmremept in

M 1, 323-342.

Gerber, S. (1996, April). Self-eflicag, itm dificulg, Ed pereistenge in
W- Paper presented at the annual meeting of the National Council
on Measurement in Education, New York.

Hembree, R (1988). Correlates, causes, efl‘ects, and treatment of test anxiety.
WM... 58 47-77

I-IilLK.T..(1980)E°'" . - _. F'
Cetebg 1, 1976 Mu gl_r Mgch 31, 1980. Urbana, IL: Illinois University, Institute for

Child Behavior and Development. (ERIC Document Reproduction Service No. ED
196936)

Jennings, E. E. (1953). The motivation factorintesting supervisors. W
1i P ch l 31, 168-169.

Johnson, S. T. (1995). Visions of equrtymnatronal assessment. InM. T. N,ettles
&AL..,Nettles(Eds) a - .- ... z :
343-366). Boston. Kluwer Academic.

Karmos, A H., & Karmos, J. S. (1984). Attitudes toward standardized
achievement tests and their relation to achievement test performance. Metivatien ape
'ninC ' andD 1 mn fi,56-66.

Kiplinger, V. L., & Linn, R L. (1992, April). Raierp' g the stakfi efxese
° ati n: The irn n stu erforman n NAEP. Paper presented at the
annual meeting of the American Educational Research Association, Los Angeles. (ERIC
Document Reproduction Service No. ED 378221)

Klein, S. P. ,Jovanovic, J., Stecher, B. M. ,McCafi’ey, D. ,Shavelson, RJ. ,Haertel,
E., Solano-Flores, G., and Comfort, K. (1997). Gender and racial/ethnic difi‘erences on

Performance assessments in science Warns. l9 83-
97.

 

 

Linn, R L., Baker, E. L., & Dunbar, S. B. (1991). Complex, performance-based
assessment: Expectations and validation criteria. WM, e9 (8), 15-21.

Lord, F. (1986). Maximum likelihood and Bayesian parameter estimation in item
response theory. 191m efEdpcatiefl Measuement, 23, 157-162.

125

Messick, S. (1994). The interplay of evidence and consequences in the validation
of performance assessments. Edpgfienal Remeher, 2 (2), 13-23.

Mislevy, R J ., & Stocking, M. L. (1989). A consumer’s guide to LOGIST and
BILOG. 1i P ch 1 ' M em _1_3, 57-75.

Multon, K. D., Brown, S. D., & Lent, R W. (1991). Relation of self-eficacy
beliefs to academic outcomes: A meta-analytic investigation. W
Barnum 38, 30-38.

Muraki, E., & Bock, RD. (1997). PARSCALE 3.3 [Computer software].
Chicago: Scientific Software International.

Murphy, RJ.L., 1982. Sex difi'erences in GCE examination entry statistics and
success rates. Educational Studies, 6, 169-17 8.

Olsen, S. A, & Wilson, K. (1991). A follew—up ef gem eephemeres’ egree
en the CQMP teg. Kirksville, MO: Northeast Missouri State University. (ERIC

Document Reproduction Service No. ED 339735)

Paris, S. G., Lawton, T. A, & Turner, J. C. (1992). Reforming achievement
testing to promote students' learning. In C. Collins & J. M. Mangieri (Eds) Tgehing

Wm en f 21 (pp. 223 -241). Hillsdale, NJ: Lawrence Erlbaum

Paris, S. G., Turner, J. C., Lawton,T. A, &Roth,J. L. (1991). Adevelopmental
perspective on standardized achievement testing. WA 20 12-20.

Plass, J. A, & Hill, K T. (1986). Children' s achievement strategies and test
performance: The role of time pressure, evaluation anxiety, and sex. Wm

Ember... 22 31-36
Pomplun, M. (1997). State assessment and instructional change: A path model
analysis WA 10 217-234

Reise, S. P., & Yu, J. (1990). Parameter recovery in the graded response model
usingMULTILOG. . z. . ~ - 21,133-144.

Roeber, E. D. (1995). Using new forms of assessment to assist in achieving
student equity: Experiences of the CCSSO State Collaborative on Assessment and Student

Standards InM T Nettlcs&AL Nettle8(EdS)E&innLand_ex_<allmern_edmona1
W (pp 265-288). Boston: Kluwer Academic.

Rothe, H. F. (1947). Distribution of test scores in industrial employees and
applicants. f li P ch 1 3_1_, 480-483.

Schmidt, F. L., Greenthal, A L., Bemer, J. G., Hunter, J. E., & Seaton, F. W.
(1977). Job sample vs. paper-and-pencil trades and technical tests: Adverse impact and
examine attitudes. P an IP h l 3_0, 187-197.

Schmitt, A P., Mazzeo, J., & Bleistein, C. (1991, April). Are gender drfi' erf mm
mm -.-. Ad 2 qularr‘u‘ tm tilechoi ‘ t d or: edr'm‘m ; must

 

 

126

flmion ef multiple ehoice DIF? Paper presented at the annual meeting of the National
Council of Measurement in Education, Chicago.

Segall, D. O. (1997, March). The efi‘ects of metivation en equating adaptive and
W. Paper presented at the annual meeting of the National Council on

Measurement m Education, Chicago.

Shavelson, R J ., Baxter, G. P., & Pine, J. (1992). Performance assessments:
Political rhetoric and measurement reality. Educational Researcher 21_ (4), 22-27.

Shepard, L. (1993). Evaluating test validity. Review of Research in Educatien,
12, 405-450.

Smith, R E., & Smoll, F. L. (1990). Sport performance anxiety. In H. Leitenberg

(Ed) WWW 417-454) New York: Plenum
Press.

Stanley, G., King, M., & Glass, J. (1989). Stress arousal and performance. In J.
P. Forgas & J. M. Innes (Eds), nt advan s in ci ch 1 ' An int '
My (pp. 229-234). Amsterdam: Elsevier Science Publishers.

Stecher, B. (1995, April). The cost of performance assessment in science: The
RAND perspective. Paper presented at the annual meeting of the National Council on
Measurement in Education, San Francisco.

Stecher, B., & Hamilton, E. G. (1994, April). Portfolio assessment in Vermont,
1992-93: The teachers’ perspective on implementation and impact. Paper presented at the
annual meeting of the National Council on Measurement in Education, New Orleans

Stevens, K (1979). The efi‘ect of topic interest on the reading comprehension of
higher ability students. WM, “.73. 365-368.

Stocking, M L. (1989). Empirical estimation errors in item response theory as a
firnction of test properties. NJ: ETS. (ERIC Document Reproduction Service No. ED
395027)

Sundte D L (1996) W

W Paper presented at the annual meeting of the National Council
on Measurement m Education, New York.

Taylor. 0.. & White. K. R (1981. APE“)- Wen
Tifle 1 students' greup standarflfl' tee; xrformang. Paper presented at the annual
meeting of the American Educational Research Association, Los Angeles. (ERIC

Document Reproduction Service No. ED 206655)

Thomas, B. (1989). W
series. Quincy, MA: Massachusetts State Department of Education. (ERIC Document
Reproduction Service No. ED 317573)

Tobias, S. (1985). Test anxiety: Interference, defective skills, and cognitive
capacitY- Wists .22. 135-142.

 

 

127

Uguroglu, M. E., & Walberg, H. J. (1979). Motivational achievement: A
quantitative synthesis. Amerieep Eggcational Reseegch J ourng, lé, 375-3 90.

Wainer, H. (1993). Measurement problems. Journal of Educational
mm. fl, 1-21.

Wainer, H., & Lukhele, R (1997). Managing the influence of DIF fiom big items:
The 1988 Advanced Placement History Test as an example. Applied Measurement in

Education. 1.2 201-215.

Winfield, L. F. (1995). Perforrnance-based assessments: Contributor or detractor
to equity? In M. T. Nettles & A L. Nettles (Ed. ), Eguig and excellence m educational
W (pp. 221-241). Boston: Kluwer Academic.

Wolf, L. F., & Smith, J. K. (1995). The consequence of consequence:
Motivation, anxiety, and test performance. Applied Measurement in Educatiep, 8, 227-
242.

Wolf, L. F ., Smith, J. K., & Birnbaum, M. E. (1995). Consequence of
performance, test motivation, and mentally taxing items. Applieg Mmemem in
M., 8 341-351.

Wolf; L. F. ,Smith, J. K., & DiPaulo, T. (1996, April). The efl‘ecte efgest Mo
W Paper presented at the annual meeting of the

National Council on Measurement m Education, New York.

Wright, B. D., & Masters, G. N. (1982). W. Chicago: MESA
Press.

Yen, W. M. (1993). Scaling performance assessments: Strategies for managing
local item dependence Wrath—palm t30 187-213

Yen, W.M. (1987). A comparison of the efliciency and accuracy of BILOG and
LOGIST. mm 52, 275-291.

        

nrcnran sin-E UN

"11111111111911"lrlriifllwiifliglrs

6