DOCTORAL DISSERTATION SERIES

PUBLICATION: 5922
AUTHOR: Gerald Lloyd Kincaid, Ed. D ,, 1953
Michigan State College
TITLE: SOME FACTORS AFFECTING
VARIATIONS IN THE QUALITY OF
STUDENTS' WRITING

University Microfilms, Ann Arbor, Michigan

SOME FACTORS AFFECTING VARIATIONS IN THE
QUALITY OF STUDENTS* WRITING

By
G erald L . K incaid

A THESIS

S ubm itted to th e School o f G raduate S tu d ie s o f M ichigan
S ta te C o lleg e o f A g ric u ltu r e and A p p lied S cience
in p a r t i a l f u l f i l l m e n t o f th e req u ire m e n ts
f o r th e degree o f
DOCTOR OF EDUCATION
School of E ducation

1953

80KB VICTOR* AfrVTZVO VUOifXHtt XV TUI
q u iik

ov 8 r m h < v m n v

*7

Rmp814 I . K lM ftid

A* AMIRAOT

|* H L th > 4 %# %lw M m I § f Qm Rn MMi K i H i i § f

Ita te O t llif i « f ip lM lla r t « a i AfpUid lsl«n e*
la pnrt& lX WXTXASm m A o f tlMi V9RlriUNMKt8
tm t tv * dagTM t f
BOOyqi

IM
Tu t

A w ro w i

X0UCATX0V

# f M m ttm
I ff)

ff

u

f

! I

I
fI

f S ff if . f
; f ( m

4 11 | *

{
!
lh

1

I !ill

I

i

*

I

r
i

i i

f S«

i

H tflS tt* ftm

taaflytia i f ito i w i | i w i t t a i y i y f w iiii by g w p

af fM is to ! i t «m fanad teat* m ID h t tte w Wi i
dlffarant taplaa aaadfMd,

i n ta t t i

in tte afflalaM y a f i t a M

ftMi

day ta day an tte m m taplaa, n r ik t payate la g la tl praaaara t i M M l
by tte fla a l i t e t e l t e H t e i t e ted agy d p l f t i i r t a ffw t mi t e
tvigttM aaallbr a fM itiM bar a

bnmb

af

tdM tr

ar m m ite te to .

twm tte i H l |i l i i f tte w i t t n pmrttmmmm by t t e v f t t e l atndaata
in cMfe p t e i lb «M f«M i t t e t i
1* aaataal f l i t e i bad wftftUia> la tffte is tr a r t rtw i itd lV
MapaMlbla fir algnlflaant M ifttin i la tte faiiigf at anting
by U7 if tte 60 atndaata isvAvad la tte atndfrrj dad ttete arl*
atlaaa low n d nite rtpiHaaHy g te te te p n ty fir tte
ibM f atndaata ttea far tte Mik ntedaBtn.
ft, far tte atraag itatetfp dlaalallar taplaa mm r a p f lt t i far
m mm

aianlflaaat MrlntlaM ttea

mm

tea aminr laalaa.

far tte walk atndanta, teanaar, dlaaiaflar taplaa did im ta an*
adt la a Mpif flaatitiy yaa te frataaway af tarlntlaaa la tte
Mfeiltar af nritlna ttea did ib d lir tanlaa.
3.

tte teOratelagtnal pwaa r a a f tea aMafnatfan did M t ite d t la
a alg alflaastty graatar fragaaray af gttaa aad/ar laaaaa f a r
tadHrldml atadaa ta tean t e a te a t praaaart ana ateaat.
t e a tte raaalta a f te la i t e f * I t aaaaa raaaanabla ta aa*»

a&nda te a t fa r tte fte y aii af atelnatiag atndant aahlaraMnt a t ajy tftna,
ar fa r tte pnr,?aaa « f atelratln g fnyrnraaant af indM daal atadaata la a
a n tin g aaaraa# atnplaa af a n tin g abated te abtaiaad an d iffaraat t apl aa
an tte t e a day aad an tte aate tap laa an di f faraat teya.

ACKNOWLEDGMENTS

The a u th o r w ishes to e x p re ss h is s in c e r e a p p r e c ia tio n
to h is a d v i s e r , D r. M ilosh M untyan, f o r h is encouragem ent
and c r i t i c a l a s s is ta n c e th ro u g h o u t th e developm ent o f t h i s
stu d y .
He i s a ls o in d e b te d to D r. H arry W. Sundwall f o r h is
h e lp fu l su g g e stio n s in p la n n in g th e stu d y and i n a n a ly z in g
th e r e s u l t s .
The w r i te r d eep ly a p p r e c ia te s th e c o o p e ra tio n o f
Mr. P aul D. B agw ell, D r. Clyde W. Dow, and Mr. Judson M.
P erk in s i n making a v a ila b le th e s tu d e n ts f o r t h i s s tu d y .
G ra te fu l acknowledgment i s a ls o due D rs . John Schmid
and W illa rd G. W arrin g to n f o r t h e i r su g g e stio n s and ad v ice
r e l a t i v e to th e s t a t i s t i c a l tre a tm e n t o f th e d a t a .

*

■—

VITA
G erald Lloyd K incaid
c a n d id a te f o r th e d eg ree o f
D octor o f E d u catio n

F in a l e x a m in a tio n , May 2 0 , 9*00 A .M ., 202A M o r r i ll H a ll.
D is s e rta tio n *

Some F a c to rs A ff e c tin g V a r ia tio n s i n th e Q u a lity o f
S tu d en ts 1 W ritin g

O u tlin e o f S tu d ie s
M ajor f i e l d s o f em phasis:
Cognate f i e l d :

E d u c a tio n a l P sy ch o lo g y , Guidance
and C o u n selin g

Communication S k i l l s

B io g ra p h ic a l Item s
B orn, March 1 , 191^, P a l e s t i n e , I l l i n o i s
U ndergraduate S tu d ie s , E a s te rn I l l i n o i s S ta te T eachers
C o lle g e , 1935-39
G raduate S tu d ie s , U n iv e rs ity o f I l l i n o i s , 19U0-1±1, M ichigan
S ta te C o lle g e , 19^8-53
E x p erien ce:

Speech te a c h e r and d eb a te co a ch , P ek in Community High School
P e k in , I l l i n o i s , 19U5-H6, I n s t r u c t o r in Communication S k i l l s
M ichigan S ta te C o lle g e , 1 9 ^6 -5 0 , E d u c a tio n a l R e sea rc h ,
Board o f E xam iners, M ichigan S ta te C o lle g e , 1950-53

TABLE OF CONTENTS

CHAPTER
I
II
III

IV

V

IV

PACE
STATEMENT OF THE PROBLEM AND ITS IMPORTANCE..............................

1

RESEARCH ON METHODS FOR EVALUATING WRITING ABILITY—REVIEW
OF THE LITERATURE...................................................................................

8

DESIGN AND PROCEDURE FOR CONDUCTING THE STUDY.........................

30

A ssignm ents and D esig n ..............................................................
C o n tro ls P ro v id e d ...................................................................................

30
36

METHOD FOR EVALUATING THE QUALITY OF STUDENTWRITING

bh

S e le c tin g th e R a tin g M eth o d . ................................
O b ta in in g R e lia b le R a tin g s ................................................................
C o n tro l of F a c to rs A ffe c tin g R a te r s ............................................
C r i t e r i a f o r R a tin g Themes...........................................
Summary.........................................................................................................

UU
53
56
56
60

ANALYSIS OF THE DATA...............................................................................

61

Background A n a ly s is ...............................................................................
E f f e c t o f Combined F a c t o r s ................................................................
C o m p a ra b ility o f C ontent F a c to rs and V a r ia tio n s i n
E f f ic ie n c y ...................
C o m p a ra b ility o f S im ila r and D is s im ila r T o p ic s ....................
E f f e c t o f th e P re s s u re F a c t o r .........................................................

62
79

SUMMARY, CONCLUSIONS, AND IMPLICATIONS.............................

83
86
86
92

BIBLIOGRAPHY........................................................................................................................ 101
APPENDICES............................................................................................................................ 105

CHAPTER I

STATEMENT OF THE PROBLEM AND ITS IMPORTANCE

S tatem ent o f th e p ro b lem .

T each ers o f E n g lish co m p o sitio n c o u rse s have

been fa c e d w ith two m ajor problem s i n e v a lu a tin g s tu d e n t ach iev em en t.
F i r s t has been th e problem o f o b ta in in g r e l i a b l e and v a lid r a t i n g s f o r
each sample o f w r i tin g .

Second has been th e problem o f d e c id in g w hat

sam ple, or sam p les, o f a s t u d e n t 's w r itin g sh o u ld be used f o r a p a r t i c u ­
l a r e v a lu a tio n p u rp o s e ,
When th e t e a c h e r 's p u rp o se i s to a s s ig n a grade b e s t r e p r e s e n tin g
achievem ent in a w r itin g c o u r s e , th e sam p le, o r sa m p les, o f a s t u d e n t 's
w r itin g should be o b ta in e d a t th e end o f th e cou rse .

But when th e te a c h e r

wants to e v a lu a te improvement i n th e q u a l ity o f w r i t i n g , a sam p le, o r
sam p les, should be o b ta in ed a t th e b eg in n in g and a t th e end o f th e c o u r s e .
To accom plish th e s e p u rp o s e s , te a c h e rs f r e q u e n tly use o n ly one sample
of w r itin g i n each case .

The re a s o n f o r t h i s i s th e amount o f tim e and

e f f o r t n e c e ssa ry to o b ta in r e l i a b l e and v a lid r a tin g s o f E n g lish com posi­
tio n s .

The q u e s tio n has been ask ed w hether a s in g le p ap er p ro v id e s a r e p ­

r e s e n ta tiv e sample o f w r itin g a b i l i t y .

Does th e q u a lity o f a s t u d e n t 's

w r itin g v ary c o n s id e ra b ly from to p ic to to p ic and from tim e to tim e?

Thus

f a r , no s tu d ie s have been re p o r te d wliich p ro v id e a d e f i n i t i v e answer to
th a t q u e s tio n .
The purpose o f t h i s t h e s i s was to determ in e w hether a s in g l e paper
w r itte n by a s tu d e n t on a g iv e n to p ic a t a p a r t i c u l a r tim e can b»

2

co n sid e re d a s a r e p r e s e n ta tiv e sam ple o f iiis w r itin g a b i l i t y —and th u s
p ro v id e a v a l i d b a s is f o r e v a lu a tin g a b i l i t y a t any tim e i n a w r itin g
c o u rs e .

The use o f a s in g le p ap e r f o r e v a lu a tin g a s tu d e n t* s achievem ent

under th e s e c o n d itio n s in v o lv e s th e fo llo w in g b a s ic a ssu m p tio n s:
1) t h a t , f o r p r a c t i c a l p u rp o s e s , any given to p ic p ro v id e s th e same
stim u lu s a s any o th e r t o p i c ,
2) t li a t such a stim u lu s w i l l e l i c i t c o n s ta n t re sp o n se s a t d i f f e r e n t
tim e s ,
3) t h a t , s in c e ouch a p ap er would be a f i n a l ex am in atio n f o r th e
c o u r s e , th e p s y c h o lo g ic a l p r e s s u re in tro d u c e d by th e f i n a l ex am in ation
s i t u a t i o n would have no a d v e rs e e f f e c t on th e q u a l ity o f w r i t i n g , and
ii) t l i a t th e q u a l ity o f w r itin g i s s ta b le from to p ic to to p ic and
from tim e to tim e , w ith o r w ith o u t th e p sy c h o lo g ic a l p r e s s u r e o f a f i n a l
e x a m in a tio n , f o r a l l s tu d e n ts r e g a r d le s s o f In d iv id u a l v a r ia tio n s i n
w r itin g a b i l i t y .
T h e r e f o re , i n o rd e r to d eterm in e whether a s in g le p ap e r w r i tte n on a
g iven to p ic a t a p a r t i c u l a r tim e p ro v id e s a v a l id b a s is f o r e v a lu a tin g
s tu d e n t achievem ent i n a w r itin g c o u r s e , i t was n e c e ss a ry to t e s t th e fo u r
assum ptions s t a t e d above .

T his t h e s i s i s a r e p o r t o f a s tu d y p lan n ed and

conducted f o r th e p u rpose o f t e s t i n g th o se assu m p tio n s.
Im portance o f th e p ro b lem .

P erhaps th e im portance of o b ta in in g a r e l i a b l e

sample o f s tu d e n t w r itin g f o r e v a lu a tio n p urposes can be b e t t e r u n d ersto o d
by exam ining 1) th e r e l a t i o n o f t r a i n i n g in language s k i l l s to o th e r
e d u c a tio n a l o b je c ti v e s , 2) th e r e l a t i o n o f t r a i n i n g in w r itte n e x p re s sio n

3

to th e t r a i n i n g i n th e language s k i l l s , and 3 ) th e i n a b i l i t y o f com posi­
tio n te a c h e rs to a d e q u a te ly answ er th e c r i t i c i s m t h a t to o many s tu d e n ts
do n o t develop s a t i s f a c t o r y w r itin g s k i l l s .
T ra in in g in th e u se o f th e E n g lis h lan g u ag e has been ac ce p ted a s one
o f th e m ost im p o rta n t o b je c tiv e s i n Am erican s c h o o ls —a s d em onstrated by
th e u n iv e r s a l re q u ire m e n t t h a t a l l s t u d e n t s , from th e elem en tary to th e
c o lle g e l e v e l , s u c c e s s f u lly com plete c o u rse s w hich in v o lv e such t r a i n i n g .
The r e p o r t o f th e P r e s i d e n t s Commission on H ig h er E d u catio n i s t y p i c a l
of th e em phasis cbmmonly p la c e d on th e im portance o f t r a i n i n g i n language
e x p re s s io n .

A ccording to t h a t r e p o r t , "Few o f th e a b i l i t i e s men p o sse ss

a re o f g r e a t e r human s ig n if ic a n c e th a n t h e i r power to o rd e r id e a s c l e a r l y
and to s e t th e s e b e fo re t h e i r fe llo w s by tongue o r pen."'*'
A lthough t r a i n i n g in b o th o r a l and w r itte n e x p re s s io n has b een u n i­
v e r s a lly re c o g n iz e d a s an im p o rta n t e d u c a tio n a l o b je c ti v e , w r i t t e n ex­
p r e s s io n has re c e iv e d th e g r e a t e s t amount o f a t t e n t i o n .

T h is i s demon­

s t r a t e d by th e u n iv e r s a l em phasis on w r i tin g i n Am erican p u b lic s c h o o ls ,
and by a c o n tin u o u s stream o f c r i t i c i s m to th e e f f e c t t h a t to o many young
people h av e n ’t been ta u g h t to w r i t e .

The r e s u l t s o f t r a i n i n g in t h i s

a re a seem t o have been s u b je c te d to more c r i t i c i s m th a n th e r e s u l t s o f
t r a i n i n g i n any o th e r a r e a .
Greene has summarized th e c r i t i c i s m s o f t r a i n i n g in w r itte n e x p re s­
s io n .

He s t a t e s t h a t
High sch o o l and c o lle g e g ra d u a te s a r e s a id to be u n ab le to
w r ite l e g i b l y , s p e l l a c c u r a te ly , o r compose an a c c e p ta b le

1 . P r e s id e n t’s Commission on H igher E d u c a tio n , " E s ta b lis h in g th e G o als",
v o l. 1 , H igher E d u catio n f o r Am erican Democracy. U. S . Government
P r in t in g O f f ic e , W ashington, D. C . , 19 ^7 , p p . 5 2 -5 3 .

...

h

l e t t e r . P a r t i c u l a r l y a r e th e y c r i t i c i z e d as b e in g unable to
th in k c l e a r l y and l o g i c a l l y anu to ex p re ss t h e i r th o u g h ts in
w e ll-c h o se n , p r o p e r ly e n u n c ia te d words a rra n g e d i n i n t e r e s t ­
in g and c le a n - c u t s e n te n c e s .^
Many te a c h e rs o f E n g lish co m p o sitio n have come to a c c e p t such c r i t i ­
cism a s an in e v ita b le co n co m itan t to t h e i r p r o f e s s io n .

From th e w rite r * s

o b s e rv a tio n , how ever, co m p o sitio n te a c h e r s r e s e n t th e im p lic a tio n t h a t
t h e i r te a c h in g i s i n f e r i o r to t h a t i n o th e r a r e a s .

They m a in ta in t h a t

in c o r r e c t language h a b i ts a re d i f f i c u l t to change—t h a t p ro g re s s in
changing such h a b its i s n e c e s s a r ily slo w .
t h i s c la im .

S e v e ra l s tu d ie s te n d to su p p o rt

Lyman s t a t e s t h a t " I t i s now g e n e ra lly a c c e p te d t h a t ab o u t

.5 of a s te p on a te n - p o in t s c a le i s th e norm al co m p o sitio n Improvement
in one y e a r ." ^

H uddelson r e p o r te d an av erag e improvement p e r y e a r o f

about fo u r p e r cen t.^4 A nderson and T ra x le r r e p o r te d an av erag e im prove­
ment p e r y e a r f o r h ig h sch o o l s tu d e n ts o f 3 .3 p o in ts when th e p ap ers were
£
r a te d on a s i x t y - p o i n t s c a le
An u n p u b lish ed stu d y by th e w r ite r showed
an average improvement o f fo u r p o i n t s , a t th e end o f a y e a r* s t r a i n i n g o f
c o lle g e freshm en, when th e p a p e rs were r a te d on a f i f t y - p o i n t s c a l e .
These a v e ra g e s a re im p o rta n t o n ly to th e e x te n t t h a t th e y in d ic a te
sm a ll in crem en ts i n w r itin g im provem ent.

N a tu r a lly , some s tu d e n ts make

2 . H. A. G reene, " E n g lis h -L a n g u a g e , Grammar, and C o m p o sitio n ", E ncyclo­
p ed ia o f E d u c a tio n a l R e se a rc h , M acm illan, New Y ork, 19f?0, p . 3 8 5 .
3 . R. L . Lyman, " I n v e s tig a tio n s i n th e F ie ld o f W ritte n C om position",
Summary o f I n v e s tig a tio n s R e la tin g to Grammar. Language , and C om position.
Supplem entary E d u c a tio n a l Monographs No. 3 6 , The U n iv e rs ity o f C hicago,
1929, p . 196.
h. E a rl H uddelson, "The E f f e c t o f O b je c tiv e S tan d ard s Upon C om position
Teachers* Judgm ents", J o u rn a l o f E d u c a tio n a l R e se a rc h , v o l. 12 (December,
19 2 5 ), p p . 329-lR).
5 . H. A . A nderson, and A. E. T r a x le r , "The R e l i a b i l i t y o f th e Reading o f
an E n g lish E ssay T e s t," a second s tu d y , School R eview , v o l. i±8, 19^0, ^
p p . 521-30.
/'

5

more th a n averag e improvement* b u t , a t th e same tim e , a s im ila r number
of s tu d e n ts make l e s s th a n av erag e improvement d u rin g any g iv en t r a i n i n g
p e r io d .

I t i s a l s o tr u e t h a t f o r any group o f s tu d e n ts e n te r in g a w r itin g

c o u rs e , u s u a lly a number o f them w ill be c o n s id e ra b ly below av erag e in
w r itin g a b i l i t y .

U nless th o s e s tu d e n ts make av erag e o r b e t t e r th a n a v e r­

age im provem ent, th e q u a lity o f t h e i r w r itin g i s l i k e l y t o be f a i r l y low
even a t th e end o f a y e a r* s t r a i n i n g .
Lack o f c o o p e ra tio n by te a c h e rs in o th e r a r e a s i s a l s o g iv en a s a
re a so n f o r s tu d e n t f a i l u r e to make th e d e s ire d improvement i n w r itin g .
Lange found t h a t s tu d e n ts were n o t h e ld r e s p o n s ib le f o r th e q u a lity o f
com position i n o th e r c o u r s e s .

He found t h a t when s tu d e n ts were ask ed to

p ro o f-re a d t h e i r p a p e rs w r i t t e n f o r co u rse s o th e r th a n E n g lis h , th e y were
a b le to c o r r e c t o n e - th ir d o f t h e i r s p e llin g e r r o r s and o n e - h a lf o f t h e i r
p u n c tu a tio n e r r o r s .

He a ls o found t h a t assig n m e n ts were a t f a u l t , t h a t

some to p ic s were to o com plex f o r c l e a r and ad eq u ate answ ers i n th e tim e
and space p ro v id e d , t h a t sk e tch y and ambiguous q u e s tio n s in v i t e d c a r e le s s
an sw ers.^

T hus, th e s tu d e n ts * e x p e rie n c e s i n o th e r c o u rse s ten d ed to

c o u n te ra c t th e e f f o r t s o f th e co m p o sitio n t e a c h e r s .
F i n a l l y , th e r e i s th e q u e s tio n o f ad eq u ate te a c h in g methods and p ro ­
cedures f o r b r in g in g ab o u t th e d e s ir e d improvement in s tu d e n t w r i t i n g .
C om position te a c h e rs have made numerous claim s f o r c e r t a i n methods and
p ro c e d u re s , b u t have been u n ab le to p r e s e n t s u f f i c i e n t evidence to make
t h e i r claim s c o n v in c in g , even to t h e i r f e llo w te a c h e r s .

In h is summary

6 . P h il C. L ange, **A Sam pling o f C om position E rro rs o f C o lleg e Freshmen
i n a Course O th er Than English**, J o u rn a l o f E d u c a tio n a l R e s e a rc h , v o l.
1*2 (November 191*8), p p . 191-200.

6

o f " I n v e s tig a tio n s i n Methods o f T eaching" (E n g lis h c o m p o s itio n ), Lyman
s t a t e s t h a t m ost o f th e s tu d ie s were d e f i n i t e l y la c k in g i n c e r t a i n
re s p e c ts *

" l ) th e y e x e rc is e d in ad eq u ate c o n tr o l o v e r te a c h in g c o n d itio n s ;

2) th e y f a i l e d to i s o l a t e s in g le v a r ia b le s f o r m easurem ent; and 3) many
o f them employed u n s a tis f a c to r y c r i t e r i a f o r im provem ent.

The e x p e rim en te rs

them selves f r a n k ly a d m itte d th e im p e rfe c tio n s th u s en u m erated ."^
A lthough c a r e f u l p la n n in g o f a r e s e a r c h s tu d y on m ethods o f te a c h in g
com position may p ro v id e f o r ad e q u ate c o n tro l o v e r te a c h in g c o n d itio n s and
f o r th e i s o l a t i o n o f s in g le v a r i a b l e s , such a stu d y c a n n o t be s u c c e s s f u l
u n le s s a s a t i s f a c t o r y c r i t e r i o n f o r improvement i n w r itin g i s employed.
And th e employment o f a s a t i s f a c t o r y c r i t e r i o n i s d ep en d en t upon v a lid and
r e l i a b l e e v a lu a tio n s o f th e q u a lity o f s tu d e n t w r i t i n g .
T y le r o f f e r s two assum ptions b a s ic to th e e d u c a tio n a l p ro c e s s and
im p o rta n t to th e p r e s e n t argument*

l ) E d u catio n i s a p ro c e s s which seeks

to change b e h a v io r p a t te r n s o f human b e in g s , and 2) e v a lu a tio n o f th e edu­
c a tio n a l program i s a p ro c e ss f o r f in d in g o u t to what d eg ree th e s e changes
8
a re ta k in g p la c e .
C e r t a in l y , th e co m p o sitio n te a c h e r s e e k s to change th e
language p a t te r n s o f human b e in g s .

Even though t h i s may be d i f f i c u l t ,

some methods may be b e t t e r th a n o th e rs f o r b rin g in g ab o u t th e d e s ire d
ch a n g es.

Yet r e s e a r c h on such methods can n o t be s u c c e s s f u l u n le s s i t i s

p o s s ib le to f in d o u t to what d eg ree th o se changes have ta k e n p la c e , i . e . ,
u n le s s s tu d e n t improvement in w r itin g can be e v a lu a te d r e l i a b l y and v a l i d l y .

7 . Lyman, 0 £ . c i t . , p . 2£3.
8 . R alph W. T y le r , "P urposes and P ro ced u res o f th e E v a lu a tio n S t a f f " ,
A p p ra isin g and R ecording S tu d en t P ro g r e s s , H arp er and B r o th e r s , New
Y ork, 19U2, p p . 1 1 -1 2 .

7

Even though th e la c k o f c o o p e ra tio n by te a c h e rs o f o th e r c o u rse s may
te n d to c o u n te ra c t th e e f f o r t s o f th e co m p o sitio n t e a c h e r , r e s e a r c h
s tu d ie s to d eterm in e th e e f f e c t s o f o b ta in in g such c o o p e ra tio n c a n n o t be
s u c c e s s fu l u n le s s s tu d e n t improvement can be e v a lu a te d r e l i a b l y and v a l i d l y .
T hus, r e l i a b l e and v a l i d e v a lu a tio n o f s tu d e n t improvement i n w r itin g
seems to be th e key to s u c c e s s f u l r e s e a r c h on b e t t e r te a c h in g m ethods.
C o n sid e ra b le r e s e a r c h has been conducted to improve p ro ced u res and
te c h n iq u e s f o r e v a lu a tin g w r itin g a b i l i t y .

Such r e s e a r c h has been on two

ty p e s o f e v a lu a tio n —d i r e c t and i n d i r e c t .

D ir e c t e v a lu a tio n has c o n s is te d

o f d e te rm in in g th e q u a l ity o f a p a r t i c u l a r co m p o sitio n and a s s ig n in g a
v alue to t h a t q u a l i t y .

I n d ir e c t e v a lu a tio n has c o n s is te d o f c o n s tru c tin g

and a d m in is te rin g an o b je c tiv e - ty p e t e s t composed o f t e s t item s co n c ern in g
v a rio u s elem e n ts o f th e w r itin g p r o c e s s .

The p e r t i n e n t r e s e a r c h l i t e r a t u r e

p e r ta in in g t o th e s e two ty p e s o f e v a lu a tio n i s review ed i n th e fo llo w in g
c h a p te r .

CHAPTER I I

RESEARCH ON METHODS FOR EVALUATING WRITING ABILITY—
REVIEW OF THE LITERATURE

D uring th e p a s t f i f t y y e a r s , c o n s id e ra b le r e s e a r c h has been conducted
in an attem p t to improve th e r e l i a b i l i t y o f e v a lu a tin g both th e w ritin g
a b i l i t y of s tu d e n ts and th e q u a l ity o f s p e c if ic p a p e rs w r itte n by s tu d e n ts .
A ttem pts have been made to d evelop o b je c tiv e - ty p e t e s t s f o r e v a lu a tin g
stu d e n ts* w r itin g a b i l i t y and to d ev elo p more o b je c tiv e and r e l i a b l e
methods f o r e v a lu a tin g th e q u a l ity o f th e s p e c if ic p a p e rs w r i t t e n .
The o b je c tiv e E n g lish t e s t p ro v id e s o n ly an i n d i r e c t e v a lu a tio n o f a
s tu d e n t's w r itin g a b i l i t y .
m a te r ia l.

Tliat i s , th e s tu d e n t i s p re s e n te d some w r i tte n

H is a c t i v i t y c o n s is ts o f s e le c tin g th e b e s t o f s e v e ra l su g g e sted

a l t e r a t i o n s o f t h a t m a te r ia l and th e n i n d ic a t in g t h a t s e l e c t i o n by making
a mark i n th e a p p r o p r ia te p l a c e .

He does no a c tu a l w r i t i n g .

r e l i a b i l i t i e s have b een o b ta in e d f o r such t e s t s .

F a i r l y hig^i

H u d d lesto n r e p o r ts r e ­

l i a b i l i t i e s ra n g in g from .88 to ,9l± f o r a group o f o b je c tiv e E n g lish t e s t s . ^
Such r e l i a b i l i t i e s in d ic a te t h a t s tu d e n t perform ance on such t e s t s does
n o t vary g r e a tly from tim e to tim e .

However, some v a r i a t i o n does o c c u r.

And i f such a t e s t i s t o be s u c c e s s f u l in m easuring s tu d e n t improvem ent,
th e n th e improvement by each in d iv id u a l m ust be g r e a te r th a n th e s t u d e n t 's

1 . E d ith M. H u d d lesto n , Measurement, o f W ritin g A b ilit y a t th e C o lle g e E ntrance L e v e lt O b je c tiv e v s . S u b je c tiv e Techniques (P h . D. t h e s i s ) ,
New York U n iv e r s ity , 1952; p u b lis h e d a s R esearch B u l l e t i n RB-52-7,
E d u c a tio n a l T e s tin g S e r v ic e , P rin c e to n , N. J . , 1952, p p . 9 -1 0 .

9

v a r ia tio n i n perform ance on th e t e s t , o th e rw ise no r e a l improvement w i l l
be shown by th e p r e - t e s t and p o s t - t e s t s c o r e s .
The v a l i d i t y o f an o b je c tiv e E n g lish t e s t i s a d i f f e r e n t m a tte r .

The

v a l i d i t y o f a t e s t i s d eterm in ed by com paring th e perform ance o f p e rso n s
on th e t e s t w ith some o u ts id e c r i t e r i o n .

As A dkins s t a t e s , "One can r e s o r t

to a l l s o r t s o f s t a t i s t i c a l m aneuverings w ith sc o re s on a t e s t and nev er
f u l l y e s t a b l i s h i t s v a l i d i t y i f a n in d ep en d en t c r i t e r i o n i s la c k in g ."
I f an o b je c tiv e E n g lis h t e s t i s supposed to e v a lu a te a s tu d e n t* s w r i t ­
in g a b i l i t y , th e n i t s v a l i d i t y f o r t h a t purpose m u st, i n th e f i n a l a n a l y s i s ,
be based on some o th e r k in d o f e v a lu a tio n o f th e s tu d e n t* s a c t u a l w r itin g
p erfo rm an ce.

On t l i i s p o in t Adkins s t a t e s th a t

. . . th e b e s t and most l o g i c a l way to determ in e what a s e l e c t i o n
t e s t p r e d ic ts i s to make a com parison betw een th e perform ance o f
p e rso n s on th e t e s t and t h e i r perform ance on th e jo b o r on a c t u a l
work a s s ig n m e n ts . I f th o s e who sc o re h ig h on th e t e s t a re a ls o
th e most s u c c e s s f u l on th e jo b , th e t e s t i s s a id t o be v a lid f o r
th e purpose o f p r e d ic tin g perform ance on t h a t p a r t i c u l a r j o b .3
Few a tte m p ts have b een made, how ever, to e s t a b l i s h th e v a l i d i t y o f
o b je c tiv e E n g lis h t e s t s f o r p r e d ic tin g th e a c tu a l w ritin g perform ance o f
s tu d e n ts .

I n most v a l i d i t y s t u d i e s , E n g lish g rad es have b een used a s th e

independent c r i t e r i o n .

H uddleston r e p o r ts v a l i d i t y c o e f f i c i e n t s ra n g in g

from .11 to .7 3 , when E n g lish g rad es were used a s th e c r i t e r i o n i n each
case

Edm iston and G in g erich o b ta in e d a v a l i d i t y o f .55 f o r th e " E n g lish

2 . D orothy C . .A dkins, C o n s tru c tio n and A n a ly sis o f A chievem ent T e s ts , U. S .
Government P r in t in g O f f ic e , W ashington, D. C ., 19U7, p p . 1&L.
3. Ib id .
It. H u ddleston , 0£ . c i t . , p p . 1 0 -1 1 .

10

Usage T e s t o f th e Ohio S ta te Every P u p il T e s ts " , when th e s c o re s on a
com position t e s t were used a s th e in d ep en d en t c r ite r i o n .'*
Lockwood found t h a t v a l i d i t y c o e f f i c i e n t s o f i n t e l l i g e n c e t e s t s w ith
com position s c o re s a r e a b o u t th e same a s th o s e o f o b je c tiv e E n g lish t e s t s .
He found th e c o r r e la tio n between g e n e ra l i n t e l l i g e n c e (u s in g th e O tis t e s t )
to be ,67 f o r 57 b o y s, and .7 6 f o r th e U3 g i r l s i n th e s tu d y .

A t th e same

tim e lie found a c o r r e l a t i o n o f .77 betw een co m p o sitio n sc o re s and sem ester
6
g ra d e s , w hich i s s l i g h t l y h ig h e r th a n th e h ig h e s t v a l i d i t y c o e f f i c i e n t
re p o rte d by H uddleston f o r an o b je c tiv e E n g lis h t e s t w ith a s im ila r
c rite rio n .
A p a r t o f th e d i f f i c u l t y i n o b ta in in g e s tim a te s o f o b je c tiv e E n g lish
t e s t v a l i d i t i e s can be a t t r i b u t e d to th e la c k o f a r e l i a b l e c r i t e r i o n .
Both co u rse g rad es and r a tin g s o f in d iv id u a l co m p o sitio n s have been g ro s s ly
u n r e lia b le in th e p a s t .

However, th e v a l i d i t y o f o b je c tiv e E n g lish t e s t s

has been q u estio n ed on l o g i c a l g ro u n d s.

Greene co n clu d es t h a t "m ost ob­

j e c t i v e t e s t s m easure only a few o f th e more obvious and m echanical s k i l l s .
Im p o rtan t elem ents o f s t y l e and q u a lity u n d o u b ted ly l i e b e n e a th th e s u rfa c e
o f th o se m echanical f a c t o r s ."

7

Greene a l s o s t a t e s t h a t th e e x p e rim en ta l

5 . R. W. E dm iston, and C. N. G in g e ric h , "The R e la tio n o f F a c to rs o f E n g lish
Usage to C o m p o sitio n ", Jo u rn a l o f E d u c a tio n a l R e se a rc h , v o l. 3 6 , 19U2,
pp. 269-71.
6 . H. R. Lockwood, C o r r e la tio n o f th e M ental M a tu rity o f One Hundred C o lleg e
Freshmen and T h e ir A b ilit y to W rite E n g lish C om position ( M a s te r s T h e sis)
U n iv e r s ity o f C h icag o , 1925. (Seen in a b s t r a c t only) R . L . Lyman, " I n ­
v e s tig a tio n s in th e F ie ld o f W ritte n C o m p o sitio n ", Summary o f I n v e s tig a ­
tio n s R e la tin g to Grammar. Language, and C o m p o sitio n , Supplem entary
E d u c a tio n a l Monographs No. 3 6 , The U n iv e rs ity o f C h icag o , 1929, p . 175.
7 . H. A . Greene , " E n g lish —Language Grammar , and C om position", E ncyclopedia
o f E d u c a tio n a l R e se a rc h , M acm illan, New Y ork, 1950, p . 39U.

11

evidence r e v e a ls a la c k o f r e l a t i o n s h i p betw een knowledge o f th o se
m echanical f a c t o r s and a b e t t e r u t i l i z a t i o n o f them when w r itin g com posi­
tio n s .

F u rth e rm o re , he r e p o r ts t h a t i n 1936, " th e C urriculum Commission

of th e T each ers o f E n g lish recommended t h a t a l l te a c h in g o f grammar
se p a ra te from th e m a n ip u la tio n o f se n te n c e s be d is c o n tin u e d , sin c e e v e ry
s c i e n t i f i c a tte m p t to prove tli a t knowledge o f grammar i s u s e f u l has f a i l e d . "
W ritin g r e q u ir e s more th a n a r e c o g n itio n o f c o r r e c t m ech an ical f a c t o r s ;
i t r e q u ir e s a p ro p e r use o f them .

L ikew ise w r itin g in v o lv e s more th a n a

re c o g n itio n , o f v a lid i n t e r p r e t a t i o n s o f d a ta ; i t in v o lv e s th e e x p re s sio n
of o r ig in a l i n t e r p r e t a t i o n s by the w r i t e r .

The d if f e r e n c e betw een making

o r ig in a l i n t e r p r e t a t i o n s o f d a ta and th e r e c o g n itio n o f such i n t e r p r e t a ­
tio n s was i l l u s t r a t e d i n a stu d y r e p o r te d by H artu n g , and o t h e r s .

T h e ir

stu d y was conducted to d eterm in e a v a l i d method f o r m easuring stu d e n ts*
a b i l i t y to i n t e r p r e t d a ta c o r r e c t l y .
f o r t h a t p u rp o s e .

An o b je c tiv e t e s t was c o n s tru c te d

The s tu d e n t was p re s e n te d a c o l l e c t i o n o f d a t a , to g e th e r

w ith a l i s t o f i n t e r p r e t a t i o n s .
which th e d a ta j u s t i f i e d .

He was to s e l e c t th o s e in t e r p r e t a t i o n s

To p ro v id e a c r i t e r i o n f o r th e o b je c tiv e t e s t ,

th e s tu d e n ts had been p ro v id ed th e same d a ta a t an e a r l i e r tim e and had
been asked to w r ite f r e e e ssa y re sp o n se s fo llo w in g such g e n e ra l d ir e c tio n s
a s : •»*Write f iv e sta te m e n ts t h a t you a r e su re a re t r u e ac c o rd in g to th e
f a c t s given in th e s e d a ta ,* and ‘W rite th r e e s ta te m e n ts based on th e d a ta
9
w hich you a re n o t q u ite s u re a re tr u e a c c o rd in g to th e s e data*.**

8 . I b i d . . p . 392.
9 . M. L. H artu n g , L . Weisman, H. G. McMullen, and H. C. T rim b le , "A spects
o f T liin k in g ", A p p ra isin g and R ecording S tu d en t P r o g r e s s , H arper and
B r o th e rs , New Y ork, 19k2t p p . 6^ - 6 7 .

c

12

A com parison o f th e re sp o n se s f o r th e same in d iv id u a ls on th e two
t e s t form s produced th e fo llo w in g p a t t e r n s ;
a . The s tu d e n t r e a c t s s im ila r ly on co rresp o n d in g ite m s o f th e
two fo rm s,
b . The s tu d e n t i s o v e rc a u tio u s on an item in ju d g in g i n t e r p r e t a ­
t i o n s made by o th e rs b u t goes beyond th e d a ta on th e c o r r e s ­
ponding item i n making h is own i n t e r p r e t a t i o n s . The re v e rs e
p a t t e r n a ls o a p p e a r s .
c . The s tu d e n t i s e i t h e r very c a u tio u s o r goes beyond th e d a ta
in ju d g in g i n t e r p r e t a t i o n s made by o th e rs b u t i s a c c u ra te
when making liis ova i n t e r p r e t a t i o n s . Here a l s o th e r e v e r s e
p a tte rn a p p e a rs. ^
The o n ly claim made by th e a u th o rs f o r th e o b je c tiv e t e s t was t h a t
i t can be used as an in d e x o f th e g e n e ra l a c cu rac y w ith w hich a group can
make o r i g i n a l i n t e r p r e t a t i o n s .

On th e o th e r hand, th e a u th o rs adm it t h a t

i t s v a l i d i t y a s an in d e x f o r p r e d ic tin g th e ac cu rac y o f o r i g i n a l i n t e r ­
p r e ta tio n s by in d iv id u a ls i s n o t h ig h .^ *

The d i f f e r e n t p a t t e r n s o f

b e h a v io r in d e a lin g w ith id e a s on th e two t e s t form s used i n t h i s stu d y
in d ic a te r a t h e r c l e a r l y t h a t a s t u d e n t ’s tre a tm e n t o f id e a s p re s e n te d in
an o b je c tiv e t e s t p ro v id e s a v ery poor b a s is f o r p r e d ic tin g how he w ill
t r e a t th o se same id e a s i n a w r itte n co m p o sitio n .
In summary, i t seems t h a t th e s t u d e n t 's knowledge o r r e c o g n itio n o f
m echanical and gram m atical f a c t o r s in v o lv e d in w r itin g has a low r e l a t i o n ­
sh ip w ith h is a c t u a l use o f them in h i s own w r itin g .

A lso , i t seems t h a t

th e s t u d e n t 's r e c o g n itio n o f c o r r e c t i n t e r p r e t a t i o n o f d a ta made by o th e rs
has a low r e l a t i o n s h i p w ith h is own i n t e r p r e t a t i o n o f th e same d a t a .

10. I b i d . , p . 73.
1 1 . I b i d . , p . 72.

13

T h e re fo re , on l o g i c a l g ro u n d s, i t seems re a s o n a b le to q u e s tio n th e v a l id ­
i t y o f th e s c o re s on an o b je c tiv e E n g lish t e s t a s an in d e x e i t h e r o f
s p e c if ic a s p e c ts o f a s t u d e n t ’s w r itin g a b i l i t y , o r o f h is o v e r - a l l w r i t ­
in g a b i l i t y a t a p a r t i c u l a r tim e .

A t b e s t , i t seem s, th e o b je c tiv e E n g lish

t e s t sc o re s can be used w ith some v a l i d i t y o n ly f o r p r e d ic tin g th e l i k l i hood o f a s t u d e n t s su c c e s s o r f a i l u r e in a w r itin g c o u rs e .

F or t h a t

p u rp o s e , i t h as ab o u t th e same p r e d ic tin g e f f ic ie n c y as a g e n e ra l i n t e l l i ­
gence t e s t .

I t would seem t h a t th e o b je c tiv e E n g lis h t e s t would p rovide

l i t t l e o r no b a s is f o r e v a lu a tin g s tu d e n t achievem ent i n a w r itin g c o u rse ,
which could be u s e d , i n t u r n , as a b a s is f o r e v a lu a tin g th e e f f e c tiv e n e s s
o f te a c h in g methods f o r in d iv id u a l s tu d e n ts .
On th e o th e r h an d , th e t r a d i t i o n a l ( d i r e c t ) method o f e v a lu a tin g a
s tu d e n t’s w r itin g has been s u b je c te d to more se v ere c r i t i c i s m th a n has
th e o b je c tiv e t e s t .

I n f a c t , th e o b je c tiv e t e s t had i t s o r ig in in an

a tte m p t to overcome th e g ro ss u n r e l i a b i l i t y o f s c o rin g th e e ssa y t e s t .

12

S ta ln a k e r s t a t e s t h a t ” . . . th e most r e c u r r e n t c r i t i c i s m o f th e e s sa y
t e s t , and th e one ab o u t which m ost has b een w r i t t e n , co n cern s th e un­
r e l i a b i l i t y o f e v a lu a tin g e s sa y answ ers . . . .

th e t y p i c a l e s sa y t e s t as

t y p i c a l l y h a n d le d , w hether by th e classroom te a c h e r o r by ‘ex p e rts*

. . .

i s n o t r e l i a o l y graded a n d , t h e r e f o r e , can n o t sta n d a lo n e as a good
m easuring in stru m e n t .**13

Huddelson r e p o r ts a stu d y in w hich th e av erag e

d e v ia tio n o f th e s c o re s a s sig n e d c e r t a i n p a p e rs by te n te a c h e rs was more

12. A d k in s, o £ . c i t . t p p . 6 -7 .
13. John N. S ta ln a k e r , ”The E ssay Type E xam ination” , E d u c a tio n a l Measure
m ent, Am erican C o u n cil on E d u c a tio n , W ashington, D. C ., 1951, p p .
Ii9B^501.

th a n two y e a rs o f norm al p u p il grow th.

Of e ig h t th em es,

te a c h e r Awould

have f a i l e d s i x w liile te a c h e r B would have p assed a l l o f th e m .^ 4
S ta ln a k e r* s s ta te m e n t r e l a t i v e to " e x p e rts " f a i l u r e to grade e ssa y s
r e l i a b l y i s p a r t i a l l y su p p o rted by a r e c e n t stu d y by H uddleston on
•'measurement o f W ritin g A b i l i t y a t th e C o lleg e -E n tran c e L e v e l."

The

stu d y was conducted a t th e r e q u e s t and under th e p a r t i a l s u p e rv is io n of
th e C ollege E n tran ce E xam inations B o ard .

Y et H uddleston re p o r te d a re a d e r

r e l i a b i l i t y o f o nly .62 f o r th e e s s a y s used i n th e stu d y
On th e o th e r h an d , S ta ln a k e r s t a t e s t h a t " R e lia b le re a d in g s o f e ssa y
pap ers i s p o s s i b l e , how ever, where th e q u e s tio n s a re c a r e f u l ly fram ed
w ith th e problem o f e v a lu a tio n i n mind and where th e re a d e rs a re tr a in e d
in th e te c h n iq u e s o f c o n s is te n t r e a d i n g . L y m a n a ls o in d ic a te s t h a t
r e l i a b l e re a d in g s o f e s sa y s can be b ro u g h t about by e x te n siv e t r a i n i n g
in th e use o f o b je c tiv e te c lin iq u e s . 17
R e lia b le r e a d in g s , a lth o u g h n o t t y p i c a l , have been re p o rte d o c c a sio n ­
a lly .

T ra x le r and A nderson r e p o r te d one stu d y w ith a re a d e r r e l i a b i l i t y

o f ,9b on pap ers w r i tte n on T opic A , and o f .85 on p ap e rs w r itte n on
lh
Topic B.
In a l a t e r s tu d y , th e same a u th o rs r e p o rte d r e l i a b i l i t i e s by
th e same re a d e r on d i f f e r e n t days o f .8 9 and ,9b f o r th e same two t o p ic s ,

1R. E a r l H uddelson, "The E f f e c t o f O b je c tiv e S tandards Upon C om position
T e a c h e rs' Judgm ents", Jo u rn a l o f E d u c a tio n a l R e se a rc h , v o l. 12 (December
1 9 2 5 ), p p . 329-UO.
15. H u d d lesto n , 0£ . c i t . , p . 73 .
16. S ta ln a k e r , ojd. c i t . , p . 503.
17. Lyman, 0£ . c i t . , p . 196.
18. A rth u r E. T ra x le r and H arold A. A nderson, "The R e l i a b i l i t y o f an Essay
T e s t i n E n g lis h ," School R eview , v o l. U3 (Septem ber 1 9 3 5 ), PP. 53R-39. ^

r e s p e c tiv e ly .

They a ls o re p o r te d r e l i a b i l i t i e s , f o r two r e a d e r s , o f

.86 and .89 on th e same two t o p i c s , r e s p e c t i v e l y .1^

Of e ig h t re a d in g

p e rio d s re p o r te d by H uddleston f o r th e C o lleg e E ntrance E xam inations
B oard, fo u r p e rio d s had re a d in g r e l i a b i l i t i e s ra n g in g from .82 to .8 9 .20
In a r e l i a b i l i t y stu d y made by th e w r i t e r , o f te n p a i r s o f r e a d e r s , two
p a ir s had r e l i a b i l i t i e s above .9 0 .
Even when r e l i a b l e re a d in g s o f E n g lish co m p o sitio n s have been ob­
t a in e d , th e problem o f r e l i a b l e e v a lu a tio n o f s tu d e n t w r itin g a b i l i t y has
n o t been s o lv e d .

In a d d i t i o n , i t i s n e c e ssa ry to c o n s id e r th e r e l i a b i l i t y

o f th e s tu d e n t’s perform ance from p ap er to p a p e r.

D ie d e ric h s t a t e s t h a t

a c o r r e la tio n o f o n ly .55 can be ex p ected between sc o re s o f p a p e rs by th e
same s tu d e n ts on s im ila r to p ic s on d i f f e r e n t days—even when r e l i a b l e
re a d in g s have been o b ta in e d . 21

T ra x le r and A nderson re p o rte d a c o r r e l a t i o n

o f .60 between s c o re s o f p ap e rs by th e same s tu d e n ts on d i f f e r e n t t o p i c s ,
when th e re a d e r r e l i a b i l i t i e s were ,9U on Topic A and .85 on T opic B.

pp

Thus i t ap p ears t h a t th e q u a l ity o f a s tu d e n t’s w r itin g may v a ry
n o tic e a b ly e i t h e r from to p ic to to p ic o r from tim e to tim e , o r b o th .

No

s tu d ie s a r e r e p o r te d , how ever, i n which an a tte m p t has been made to
d i f f e r e n t i a t e betw een th e s e two so u rce s and t h e i r p o s s ib le e f f e c t s on th e
q u a lity o f a s tu d e n t’ s w r i tin g .

19. H. A. A nderson, and A . E . T r a x le r , ’’The R e l i a b i l i t y o f th e R eading o f
an E n g lish E ssay T e s t” , School Review , v o l. 1*8, 19u0, p p . 521-30.
20. H uddlesto n , 0£ . c i t . , p . 7 .
21. P au l B. D ie d e ric h , ’’The Measurement o f S k i l l i n W r i t i n g School Review
v o l. 55 (A p ril 19U8), p p . 375-8U.
22. A. E. T ra x le r and H. A. A nderson, op . c i t .

16

S ince th e r e l i a b i l i t y o f an o b je c tiv e - ty p e t e s t u s u a lly i s n o t
in flu e n c e d by d i f f i c u l t i e s i n o b ta in in g a c c u r a te s c o rin g (w hich i s a
m a tte r o f c l e r i c a l a c c u ra c y ), c o n s id e ra b le a t t e n t i o n has been d ir e c te d
tow ard th e so u rc e s o f u n r e l i a b i l i t y f o r t h a t ty p e o f t e s t .

F or th e most

p a r t , th o se so u rce s a r e d i r e c t l y r e l a t e d to v a r ia tio n s i n s tu d e n t perfo rm ­
ance on th e t e s t .

Adkins s t a t e s t h a t

The t e s t i t s e l f o f te n f a l l s f u r t h e r s h o r t o f p e r f e c tio n th a n i s
u su a l w ith p h y s ic a l in s tr u m e n ts . But th e v a r ia tio n s in th e
p erso n s m easured—v a r ia tio n s due to such in f lu e n c e s as f a t i g u e ,
p re v io u s t e s t i n g w ith th e same o r a s im ila r t e s t , and la c k o f
i n t e r e s t o r e f f o r t —and v a r ia tio n s i n c o n d itio n s and methods o f
t e s t a d m in is tr a tio n from tim e to tim e o r from exam iner to
examiner o f te n c o n trib u te most to th e u n r e l i a b i l i t y of t e s t r e ­
s u l t s . These l a t t e r f a c t o r s , w hich o r ig in a t e o u ts id e th e t e s t
i t s e l f , can be c o n tr o lle d to a c o n s id e ra b le e x te n t by c a r e f u l
a t t e n t i o n to p ro ced u res o f t e s t a d m in is tr a tio n .23
A d k in s• d e s c r ip tio n o f " v a r ia tio n s i n th e p e rso n s measured" in d ic a te s
t h a t a s t u d e n t 's perform ance may v ary from tim e to tim e a s a r e s u l t o f
p sy c h o lo g ic a l v a r ia tio n s i n b eh av io r which a r e tem porary in c h a r a c t e r .
R. L . T horndike in d ic a te s a s im ila r so u rce o f v a r i a t i o n i n t e s t p erform ance.
Under "tem porary f a c t o r s a f f e c t i n g perform ance on many o r a l l t e s t s a t a
p a r t i c u l a r tim e ," he l i s t s h e a l t h , f a t i g u e , m o tiv a tio n , em otional s t r a i n ,
e x te r n a l c o n d itio n s , momentary s e t , f l u c t u a t i o n s and id io s y n c r a s ie s o f
2ii
human memory, e t c .
T h o rn d ik e, how ever, c o n s id e rs a n o th e r sou rce o f
v a r ia tio n which i s more l a s t i n g o r perm anent.

He c a l l s t h i s so u rce " th e

chance elem ent d e te rm in in g w hether th e in d iv id u a l does o r does n o t know

23. A d k in s, ojd. c i t . , p . 11*9.
2li. R . L . T h o rn d ik e, " R e l i a b i l i t y " , E d u c a tio n a l M easurem ent. American
C o u n cil on E d u c a tio n , W ashington, D. C . , 195^1, p . £68.

17
o<
a p a r t i c u l a r f a c t . ” ^ A ccording to T h o rn d ik e,
There w i l l be a c e r t a i n amount o f v a r i a t i o n i n s p e c if ic b i t s
o f knowledge or s k i l l , so t h a t even th e in d iv id u a l who has a
h ig h o v e r - a l l a b i l i t y in th e a r e a in q u e s tio n w i l l la c k
s p e c if ic item s o f knowledge o r s k i l l and th e in d iv id u a l low
in g e n e ra l perform ance w i l l su cceed on i s o l a t e d ite m s n o t
known by h is g e n e ra lly more p r o f i c i e n t f e l l o w .^6
The so u rce s o f t e s t v a r i a t i o n d e s c rib e d by A dkins as ’’v a r ia tio n s i n
th e p erso n s" and by T horndike a s "tem porary f a c t o r s a f f e c t i n g perform ance"
a re d e s c rib e d by L oevinger a s " t r a n s i t o r y v a r ia tio n s in e f f ic ie n c y
On th e o th e r hand, T h o rn d ik e 's more perm anent so u rce o f v a r ia tio n
co n cern in g a c c id e n ta l knowledge ( o r la c k o f i t ) i s in c lu d e d and extended
somewhat by L oevinger u n d er what she c a l l s " a c c id e n ta l c o n te n t f a c to r s "
To i l l u s t r a t e h e r e x te n s io n o f t h i s sou rce o f v a r i a t i o n , L oevinger s ta te s *
We may c o n s id e r a t e s t c o n ta in in g a. number o f problem s in v e rb a l
d e d u c tiv e re a s o n in g . The problem s would d i f f e r i n s u b je c t
m a tte r , and a p a r t from th e d i f f i c u l t y o f th e r e la tio n s h i p s i n ­
v o lv e d , some to p ic s would be e a s ie r f o r a p a r t i c u l a r i n d iv i d u a l,
r e l a t i v e to o th e r i n d i v i d u a l s , th a n o th e r t o p i c s , w hether because
o f f a m i l i a r i t y , c o n g e n ia lity , o r s p e c i f i c em o tio n al f a c t o r s . °
(For co n v e n ien c e, th e s e two so u rces o f v a r i a t i o n i n s tu d e n t perform ance
w i l l be r e f e r r e d t o , h e n c e fo rth , as " c o n te n t f a c t o r s " and " v a r ia tio n s in
e f f i c i e n c y ." )

25. I b id .
26. I b id .
27 . Jane L o e v in g e r, "A S y stem atic Approach to th e C o n s tru c tio n and Evalu­
a t io n o f T e s ts o f A b i l i t y " , P sy c h o lo g ic a l M onographs. v o l. 6 1 , No. ij.,
19147, p . 5 .
28. Ib id .
29. I b id .

18

The e ffe c t, o f 'th ese "two so u rc e s on th e r e l i a b i l i t y o f ■bests i s con­
s id e re d by C ronbach.
ty p e s :

He c l a s s i f i e s r e l i a b i l i t y c o e f f ic ie n ts under th r e e

th e c o e f f i c i e n t o f e q u iv a le n c e , th e c o e f f i c i e n t o f s t a b i l i t y , and

th e c o e f f i c i e n t o f s t a b i l i t y and e q u iv a le n c e .

"The c o e f f i c i e n t o f e q u i­

valence in d ic a te s how p r e c i s e ly th e t e s t m easures th e p e rs o n ’ s perform ance
a t a p a rtic u la r tim e ." ^

I t i s o b ta in e d by c o r r e la tin g th e s c o re s on two

t e s t s d e sig n e d to m easure th e same a b i l i t i e s , a d m in iste re d t o th e same
s tu d e n ts d u rin g th e same p e r io d , o r by what i s known as th e " s p l i t - h a l f ”
m ethod.

T h is method u s u a lly c o n s is ts o f c o r r e la tin g th e s c o re s on th e odd

item s w ith th o s e on th e even ite m s .

The c o e f f i c i e n t o f e q u iv a le n c e , th e n ,

i s concerned w ith v a r i a t i o n s in s tu d e n t perform ance when th e s u b je c tm a tte r ( c o n te n t f a c t o r s ) has been changed b u t w h ile th e " e f f ic i e n c y le v e l"
rem ains f a i r l y c o n s ta n t.
"The c o e f f i c i e n t o f s t a b i l i t y shows th e e x te n t t o which s c o re s on
th e p a r t i c u l a r t e s t ite m s a r e s ta b le o ver a p e rio d o f tim e .

I t in d ic a te s

w hether a sample o f b eh av io r ta k e n a t one tim e i s t y p i c a l o f b eh a v io r a t
o th e r tim es

T h is c o e f f i c i e n t i s o b ta in e d by c o r r e la tin g s c o re s on

one t e s t a d m in is te re d to th e same s tu d e n ts a t d i f f e r e n t tim e s .

Thus th e

c o e f f i c i e n t o f s t a b i l i t y i s concerned w ith v a r ia tio n s i n s tu d e n t perform ­
ance due to v a r ia tio n s i n e f f ic ie n c y when th e c o n te n t f a c t o r s have been
h e ld c o n s ta n t.
The c o e f f i c i e n t o f s t a b i l i t y and eq u iv a le n c e i s o b ta in e d by c o r r e l a t ­
in g th e s c o re s on two com parable t e s t s a d m in iste re d to th e same s tu d e n ts

3 0 . Lee J . C ronbach, E s s e n tia ls o f P sy c h o lo g ic a l T e s tin g . H arper and
B r o th e rs , New Y ork, 19U9, p p .
o9.
31. I b id .

19

a t d i f f e r e n t tij n e s .

C o n seq u en tly , b o th c o n te n t f a c t o r s and v a r ia tio n s

in e f f ic ie n c y a f f e c t th e r e s u l t s .

There i s no way o f knowing, how ever,

which o f th e f a c t o r s may be r e s p o n s ib le f o r such v a r ia tio n s as o ccu r in
th e s c o re s o f any in d iv i d u a l.
L oevinger d is c u s s e s th e s e two so u rc e s more s p e c i f i c a l l y i n r e l a t i o n
to b o th th e c o e f f i c i e n t o f e q u iv a le n c e and th e c o e f f i c i e n t o f s t a b i l i t y .
She s t a t e s t h a t when th e r e l i a b i l i t y c o e f f i c i e n t ( o f e q u iv a le n c e ) i s
computed by th e s p l i t - h a l f m ethod, u s in g th e s c o re s on th e odd and even
item s to make up th e two h a l v e s , th e c o n te n t f a c t o r s a c t to low er th e
c o e f f ic ie n t o b ta in e d , w h ile th e e f f ic ie n c y le v e l ( e n te r in g b o th sc o re s in
th e same way) a c ts to r a i s e th e c o e f f i c i e n t o b ta in e d .^ 2

In o th e r w ords,

th e c o n te n t f a c t o r s a re p r im a r ily r e s p o n s ib le f o r th e v a r ia tio n s which
occur betw een th e s tu d e n ts * sc o re s on th e two h a lv e s o f th e t e s t .

S ince

th e e f f ic ie n c y le v e l of th e s tu d e n t e n te r s in to b o th s e t s o f s c o re s in
th e same way, th e r e i s no in d ic a tio n o f how much v a r ia tio n m ight occur i f
th e t e s t were a d m in is te re d a t a n o th e r tim e .
B ut when th e t e s t - r e t e s t i s used to o b ta in th e c o e f f i c i e n t o f s t a b i l i t y ,
L oevinger in d ic a t e s t h a t v a r i a t i o n s i n e f f ic ie n c y l e v e l a c t to lo w er th e
c o e f f i c i e n t , w h ile th e c o n te n t f a c to r s (b e in g re p e a te d ) may a c t to r a i s e
th e c o r r e l a t i o n . 33

I n o th e r w ords, v a r ia tio n s i n e f f ic ie n c y seem p r im a r ily

re s p o n s ib le f o r th e v a r ia tio n s which occur betw een th e s tu d e n ts 1 sc o re s
on th e two a d m in is tr a tio n s o f th e same t e s t a t s h o r t i n t e r v a l s .

32. L o ev in g er, o £ . c i t . , p p . 5 -6 .
33. I b id .

S ince th e

20

c o n te n t f a c t o r s rem ain th e sam e, th e r e i s no in d ic a tio n o f how much
v a r ia tio n would o ccu r i f two com parable t e s t s ( d i f f e r i n g o n ly in c o n te n t)
were a d m in is te re d d u rin g th e same p e r io d .
T hus, to d eterm in e th e r e l a t i v e e f f e c t s o f th e s e two so u rc e s on
in d iv id u a l s t u d e n t s , i t would be n e c e s s a ry to a d m in is te r two com parable
t e s t s d u rin g tiie same p e r io d and th e n to r e p e a t b o th t e s t s a t an o th e r
tim e .

V a r ia tio n s in perform ance on th e two t e s t s d u rin g th e same p e rio d s

would p ro v id e an e s tim a te o f th e e f f e c t o f c o n te n t f a c t o r s ; w h ile th e
v a r ia tio n s i n perform ance on th e same t e s t s a t d i f f e r e n t tim es would p ro ­
v id e an e s tim a te o f th e e f f e c t o f v a r i a t i o n s i n e f f ic ie n c y .
S tu d ie s o f in te l l i g e n c e t e s t s p ro v id e some c o n c re te evidence s u p p o rt­
in g th e s e c o n te n tio n s ab o u t th e e f f e c t s o f c o n te n t f a c t o r s and v a r ia tio n s
in e ffic ie n c y .

Cronbach r e p o r ts a s p l i t - h a l f c o e f f i c i e n t o f .91 from an

a d m in is tr a tio n o f th e " D e tr o it B eginning F irs t-G ra d e I n te ll ig e n c e T e s t" ,
A d m in is tra tio n o f th e same t e s t fo u r months l a t e r produced a t e s t - r e t e s t
c o e f f i c i e n t o f s t a b i l i t y o f .7 6 .* ^

The p e r io d betw een th e two a d m in is tra ­

t io n s o f th e t e s t d i d , o f c o u rs e , p ro v id e an o p p o rtu n ity f o r th o se who
were u n f a m ilia r w ith th e c o n te n t o f th e ex am in atio n a t th e tim e o f i t s
f i r s t a d m in is tr a tio n to g a in t h a t f a m i l i a r i t y b e fo re th e second adm inis­
tra tio n .

In t h i s case th e e f f e c t o f th e c o n te n t f a c t o r s may have been as

g r e a t a s though a d i f f e r e n t t e s t had been a d m in is te re d .

In o th e r w ords,

th e s p l i t - h a l f c o e f f i c i e n t o f .91 p ro v id e d a n e s tim a te o f v a r ia tio n due to
c o n te n t f a c t o r s a t t h a t p a r t i c u l a r tim e .

3U. C ronbach, oj>. c i t . , p . 69.

The t e s t - r e t e s t c o e f f i c i e n t o f

21

.76 prov id ed an e s tim a te o f th e e f f e c t n o t o n ly o f v a r ia tio n s i n effi**
c ie n c y b u t a ls o o f c o n te n t f a c t o r s a s w e ll.
S tro u d r e p o r ts t h a t even when i n t e l l i g e n c e t e s t s a r e a d m in iste re d
o nly two o r th r e e days a p a r t , " th e median v a r i a b i l i t y i s no rm ally expected
to be ab o u t 5 IQ p o i n t s .

T h at i s , o n e -h a lf o f th e p u p ils change 5 IQ

p o in ts o r more; a few change m ark ed ly .
Not o n ly lias a v a r i a b i l i t y i n s tu d e n t perform ance on in te lli g e n c e
t e s t s been n o ted from day to d ay , b u t a d if f e r e n c e in th e amount o f v a r i ­
a t io n has been noted f o r s tu d e n ts w ith d i f f e r e n t a b i l i t i e s .

S tro u d r e ­

p o r ts tlia t th e p ro b ab le e r r o r o f an IQ sco re ra n g e s from l . i i 9 , f o r sc o re s
below 70, to 3.5U , f o r s c o re s above 1 3 0 .^ ^

M e r r i l l r e p o r ts a s im ila r

range f o r th e sta n d a rd e r r o r o f IQ s c o re s —from 2 .2 , f o r s c o re s below 70,
to 5 .2 , f o r sc o re s above 1 3 0 .-^

S in ce such t e s t s a re c a r e f u l l y a d m in iste re d

and sin c e th e v a r ia tio n s a re b o th p lu s and m in u s, th e s e f in d in g s would
su g g e st t h a t , in s o f a r a s such t e s t s denote a b i l i t y , th e d eg ree of v a r i ­
a tio n s i n t e s t perform ance i s i n d i r e c t p r o p o rtio n to th e a b i l i t y o f th e
i n d iv i d u a ls .
An u n p u b lish ed stu d y by th e w r ite r su g g e s ts t h a t v a r ia tio n s i n th e
q u a l ity o f s tu d e n t w r itin g may c lo s e ly p a r a l l e l th e v a r ia tio n s a c co rd in g
to a b i l i t y found i n s tu d e n t perform ance on i n te lli g e n c e t e s t s .

T h is stu d y

concerned e v a lu a tin g improvement in th e q u a l ity o f s tu d e n t w r itin g over a

35. James B. S tro u d , P sychology i n E d u c a tio n , Longmans, G reene, and
Company, New Y ork, 19U6, p . 300.
36. I b i d . , p . 301.
37. Maud A. M e r r i l l , MThe S ig n ific a n c e o f IQ ’ s on th e R evised S ta n fo rd B in e t S c a le s ” , J o u rn a l o f E d u c a tio n a l P sy ch o lo g y , v o l. 29 (December
1938) p p . 6U1-651.

22

p e rio d o f a y e a r 's t r a i n i n g i n co m p o sitio n .

Each s tu d e n t w rote a pap er

a t th e b e g in n in g and a n o th e r a t th e end o f th e y e a r .
on th e same t o p i c .

B oth p a p e rs were

T w en ty -six and e i g h t - t e n t h s p e r c e n t o f th e 198 s t u ­

d e n ts in v o lv ed r e c e iv e d low er sc o re s on th e f i n a l p ap er th a n on th e i n i t i a l
p ap e r.

E ig h ty p e r c e n t o f th e s e s tu d e n ts who r e c e iv e d low er sc o re s on th e

f i n a l p a p e r , had re c e iv e d s c o re s on t h e i r i n i t i a l p a p e rs w hich were above
th e mean f o r t h a t group o f p a p e r s .

I n s o fa r a s th e s c o re on th e i n i t i a l

paper in d ic a te d th e s t u d e n t 's w r itin g a b i l i t y , th e f a c t t h a t t h i s e ig h ty
p e r c e n t were above th e i n i t i a l mean su g g e s ts t h a t s tu d e n ts above average
in w r itin g a b i l i t y may v ary more in th e q u a lity o f t h e i r w ritin g th a n
th o se w ith l e s s th a n average a b i l i t y .
A lthough much o f our e d u c a tio n a l measurement has ten d ed to ig n o re th e
v a r ia tio n s i n s tu d e n t perform ance n o ted th u s f a r , such v a r ia tio n s i n human
b eh a v io r a r e c o n s is te n t w ith c u r r e n t p s y c h o lo g ic a l th e o ry and f i n d i n g s .
However, th e use o f a s t u d e n t 's sc o re on a s in g le t e s t o r on a s in g le
paper a s in d ic a t iv e o f h i s achievem ent a t tli a t p a r t i c u l a r tim e i s to
o p e ra te on th e b a s is o f th e m e ch an istic ( r e f l a x - a r c ) psychology i n vogue
a t th e tu r n o f th e c e n tu ry and s h o r tly t h e r e a f t e r .

As Mowrer s t a t e s , t h i s

psychology was b ased on two a b s t r a c t i o n s , S (s tim u lu s ) and R ( re s p o n s e ) ,
og
"w ith on ly a t h i n , e q u a lly a b s t r a c t , arrow c o n n e c tin g th e m ."J
In answer
to F e a r in g 's r e p e a te d re f e r e n c e " to th e q u a l ity o f 'i n v a r i a b i l i t y * which
r e f le x e s a r e a lle g e d to p o s s e s s ," Mowrer w r ite s :
. . . from th e o u ts e t t h i s p ro p e rty was an em barrassm ent to
p s y c h o lo g is ts who were a tte m p tin g to make th e r e f l e x a r c the

38. 0 . H. w ow rer, "L earn in g T h eo ry ", Review o f E d u c a tio n a l R e s e a rc h , v o l.
22, No. S> (December 1 9 5 2 ), p . 1176.

23

c o rn e rsto n e o f a new p sy ch o lo g y , em b arrassin g c h i e f ly f o r
th e re a s o n t l i a t a n y th in g ap p ro x im atin g an ad eq u ate th e o ry o f
b eh av io r m ust p ro v id e f o r th e o cc u rren c e o f ch an g e, p l a s t i c i t y , le a r n in g . The c l a s s i c a l c o n c e p tio n o f th e r e f l e x was
in t h i s r e s p e c t s in g u la r ly d e f i c i e n t . 3 9
E . L . Thorndike noted th e f a i l u r e o f th e m e c h a n istic psychology to
account f o r v a r i a b i l i t y i n human b e h a v io r.

In 1907, lie w ro te:

. . . f o r any human b e in g 's th o u g h t and c o n d u c t, depending as
th e y do upon th e a c tio n o f h is nervous sy stem , w ill sometimes
show m y ste rio u s a l t e r a t i o n s —b e h a v io r u n e x p la in a b le by th e laws
o f i n s t i n c t , a s s o c i a t i o n , and d i s s o c i a t i o n . The nervous system
i s in flu e n c e d n o t o n ly by th e f a c t o r s acco u n ted f o r i n th e s e
th r e e la w s , b u t a ls o by f a t i g u e , d ru g s , s ic k n e s s , th e decay o f
o ld a g e , sh o ck , th e chance v a r ia tio n s o f b lo o d p r e s s u r e ,
m etabolism and th e lik e .h O
S e v e ra l y e a rs l a t e r , Thorndike r e p o r te d a stu d y w hich su p p o rted th e
view t h a t human b e h a v io r i s v a r ia b le from tim e to tim e .

H is stu d y was

concerned w ith p l o t t i n g th e work curve o f f i v e d i f f e r e n t g rad u ate p sy ­
chology s tu d e n ts on f i v e d i f f e r e n t d a y s.

The work c o n s is te d o f adding

numbers f o r one and o n e -h a lf to two h o u rs , r e c o rd in g th e tim e , in se co n d s,
a t th e com pletion o f each row o f 16 exam ples— th u s o b ta in in g th e le n g th
o f tim e re q u ir e d to com plete each row o f a d d i tio n p ro b le m s.

Thorndike

concluded t h a t " th e v a r ia tio n in th e form o f th e work cu rv e on d i f f e r e n t
days i s so g r e a t a s to r e q u ir e c a r e f u l c o n s id e r a tio n o f th e u n r e l i a b i l i t y
o f any d e te rm in a tio n based on o n ly a few d a y s ' r e c o r d s . " ^

39. I b i d .
iiO. E . L . T h o rn d ik e, The Elem ents o f P sy ch o lo g y , The Mason-Henry P r e s s ,
S y ra c u se , New Y ork, 1907, p . 222.
U l. E . L. T h o rn d ik e, M ental Work and F a tig u e and In d iv id u a l D if f e r e n c e s ,
T eachers C o lle g e , Columbia U n iv e r s ity , New Y ork, 1923, p . 5>3.

2k

T horndike gave th e com plete d a ta f o r one s u b je c t on th e tim e r e ­
q u ired to com plete each row o f examples on each d a y .

Computing the

average tim e re q u ir e d to com plete each o f th e n in e rows o f examples used
on f iv e d i f f e r e n t days p ro v id e d a check on th e v a r ia tio n o f o v e r - a l l
e f f ic ie n c y on d i f f e r e n t d a y s .

The fo llo w in g v a r ia tio n s were found in th e

average number o f seconds r e q u ir e d to com plete each row o f exam ples:
1 s t Day

2nd Day

3 rd Day

l*th Day

5 th Day

11*2.7

129.7

150.3

1 1 9 .U

10l*.8

The maximum d if f e r e n c e in th e average tim e re q u ire d to com plete ea ch row
of a d d itio n problem s was 1*5.5 seconds—betw een th e t h i r d and f i f t h d a y s.
In o th e r w ords, th e s u b je c t s p e n t 1*3. h p e r c e n t more tim e p e r row on th e
t i d r d day th a n on th e f i f t h d a y .
between th e f i r s t and t h i r d d a y s .

The minimum d if f e r e n c e was 7 .6 seco n d s—
The s u b je c t sp e n t 5 .3 p e r c e n t more

tim e p e r row on th e t l i i r d day th a n on th e f i r s t d a y . ^
On th e o th e r h an d , Gates re p o r te d a stu d y on v a r ia tio n s i n e f f ic ie n c y
o f p erfo rm in g a d d itio n problem s a t d i f f e r e n t tim es d u rin g th e same day.
The g r e a t e s t v a r ia tio n n o ted d u rin g th e day was 1*.2 p e r c e n t ,
l e s s tlian th e minimum d if f e r e n c e n o ted between days .

)

*3

which i s

Thus i t ap p ears t h a t

v a r ia tio n s i n perform ance on d i f f e r e n t days i s l i k e l y to be g r e a te r th a n
such v a r ia tio n s a t d i f f e r e n t tim es d u rin g th e same day.
Gates a ls o d e p a rts from th e b a s ic co n cep t o f th e m e c h a n istic p sy ­
chology i n h is d is c u s s io n o f human b e lia v io r, c l a s s i f y i n g th e so u rc e s of

1*2. I b i d . , p . 1*9.
1*3. A rth u r I . G a te s , Psychology f o r S tu d e n ts o f E d u c a tio n . The M acm illan
Company, New Y ork, 1931, p . 1*71.

25

v a r i a t i o n a s e x te r n a l and i n t e r n a l :
H ost o f th e complex r e a c tio n s w ith which p sychology i s concerned
m oreover a r e made n o t to a s in g le and sim ple stim u lu s b u t to a
com bin atio n o f f o r c e s . P rom inent among th e f o rc e s a re th e
a c t i v i t i e s g o in g on w ith in th e p e rso n a t th e tim e . The a c t i v i t y
o f any b o d ily mechanism se rv e s a s a p a r t i a l cau se o r s tim u lu s
f o r f u r t h e r a c t i v i t y . Thus b eh a v io r o f a man i s determ ined by
th e combined and c o o rd in a te d e f f e c t s o f what we may f o r conven­
ie n c e d iv id e in to e x te r n a l and i n t e r n a l a c t i v i t i e s , in c lu d in g i n
th e l a t t e r b o th co n scio u s and u n conscious a c t i v i t i e s
A lthough Gates made t h i s s ta te m e n t in 1931, i t i s c lo s e ly p a r a l l e l e d
by Mowrer*s r e c e n t summary o f " le a r n in g Theory” :
. . . I t i s now a p p a re n t t h a t th e s e two form s o f le a r n in g e x i s t
and f u n c tio n , n o t s id e - b y - s id e , b u t e n d -to -e n d : s ig n le a r n in g i s
th e p ro c e s s whereby e x te r n a l ev e n ts come to produce i n t e r n a l
d riv e s t a t e s , and s o lu tio n le a r n in g i s th e p ro c e s s whereby
i n t e r n a l d r iv e s t a t e s produce e x t e r n a l t o v e r t b e h a v io r . We th u s
advance from a s im p le , and p r e t t y c l e a r l y in a d e q u a te , S-R p sy ­
chology to an S-R:S-R p sy ch o lo g y . By draw ing a c i r c l e (0) around
th e R:S p a r t o f t h i s seq u en ce, we n o t only r e d is c o v e r th e ’organism*
b u t we redeem i t from th e s t a t e o f 'e m p tin e s s ' to which extrem e
beliav io rism condemned i t ^ ^and th u s b e g in , r e a l i s t i c a l l y , to examine
th e org an ism -as-a-w h o le
T h u s, th e to p ic a s sig n e d a t a p a r t i c u l a r tim e , o r th e s u b je c t m a tte r
o f a p a r t i c u l a r t e s t ite m , may be co n sid ered as an e x te r n a l f a c t o r , w h ile
th e a c t i v i t i e s going on w ith in a p e rso n a t a p a r t i c u l a r tim e may be con­
s id e re d a s i n t e r n a l f a c t o r s v a ry in g from tim e to tim e .

Inasmuch as th e

to p ic a s s ig n e d , o r th e s u b je c t m a tte r o f a p a r t i c u l a r t e s t ite m , may a f f e c t
th e i n t e r n a l a c t i v i t i e s o f th e in d iv i d u a l, th e two so u rc e s o f v a r i a t i o n
a re in s e p a r a b le .

In g e n e ra l, how ever, th e i n t e r n a l f a c t o r s (th e e f f ic ie n c y

l e v e l o f th e in d iv id u a l) may be co n sid ered as th e p rim ary v a r ia b le when

14i. I b i d . . p . 65.
ii5. Mowrer, 0 £ . c i t . , p . 14.92.

26

th e same c o n te n t f a c t o r s a r e r e p e a te d a few days a p a r t .

L ik e w ise , when

d i f f e r e n t c o n te n t f a c t o r s a r e p re s e n te d d u rin g a s h o r t p e r io d , th e
e f f ic ie n c y l e v e l sh o u ld rem ain f a i r l y c o n s ta n t—th u s th e c o n te n t f a c t o r s
may be c o n sid e re d a s th e p rim ary v a r i a b l e .

SUMmARY

As p re v io u s ly in d ic a te d , th e te a c h in g o f w r itte n e x p re s s io n i s con­
s id e re d a s one o f th e most im p o rta n t e d u c a tio n a l o b je c ti v e s .

A t th e same

tim e , s tu d e n t achievem ent in t h i s a re a has been s u b je c te d to c o n s id e ra b le
c ritic is m .

The f a i l u r e o f s tu d e n ts to a t t a i n th e d e s ir e d g o als h as been

e x p la in e d as fo llo w s :
1) Improvement in language h a b its i s a slow p r o c e s s .
2) Lack o f c o o p e ra tio n by te a c h e rs in o th e r a re a s te n d s to
c o u n te ra c t tiie work o f th e co m p o sitio n t e a c h e r .
3) T eaching methods i n u se may be in a d e q u a te .
R esearch r e l a t e d to th e s e f a c t o r s lias been lim ite d i n i t s v alue by
th e la c k o f v a lid and r e l i a b l e e v a lu a tio n te c h n iq u e s f o r d ete rm in in g s t u ­
d en t improvement in w r i tin g .

A lthough o b je c tiv e E n g lish t e s t s have proved

to be r e l i a b l e in s tru m e n ts , th e y have n o t been v a lid a te d f o r e v a lu a tin g
improvement in th e q u a l ity o f s tu d e n t w r i tin g .

On a lo g i c a l b a s i s , i t

seems d o u b tfu l t h a t o b je c tiv e t e s t s can be developed t h a t w i l l accom plish
t h a t p u rp o se .
On th e o th e r hand, th e d i r e c t e v a lu a tio n o f th e q u a lity and th e im­
provem ent o f s tu d e n t w r itin g has been h in d e re d by two m ajor o b s ta c le s :
l ) R e lia b le r a t i n g s o f sam ples o f s tu d e n t w r itin g have been d i f f i c u l t to

27

o b ta in , and 2) th e r e l i a b i l i t y o f a s t u d e n t 's w r itin g perform ance has
been q u e s tio n e d .
Through th e in te n s iv e t r a i n i n g o f r a t e r s i n th e use o f w r itin g s c a le s
and v a rio u s ty p e s o f sc o re c a r d s , r e l i a b l e r a t i n g s o f s tu d e n t w r itin g
have been o b ta in e d .

A ls o , th e re i s c o n s id e ra b le evidence to in d ic a te

t h a t " c o n te n t f a c t o r s " and " v a r ia tio n s in e f f ic ie n c y " sire th e prim ary
so u rces o f v a r i a t i o n i n s tu d e n t perform ance on t e s t s .

Y et m ost s tu d ie s

conducted to e v a lu a te s tu d e n t improvement i n w r itin g have used a s in g le
com position f o r th e p r e - t e s t and a n o th e r co m p o sitio n on a d i f f e r e n t to p ic
f o r th e p o s t - t e s t .

Even though such e v a lu a tio n p ro ced u res have shown a

sm all o v e r - a l l improvement i n th e q u a l i t y o f w r i t i n g , th e r e s u l t s were
p r a c t i c a l l y m ean in g less f o r e v a lu a tin g in d iv id u a l im provem ent.

In two

s t u d i e s , th e s c o re s r e p o r te d were low er on th e f i n a l p a p e rs th a n on th e
i n i t i a l p a p e rs f o r a c o n s id e ra b le p o r tio n o f th e s t u d e n t s .

I t seems

re a s o n a b le t o assume i n each ca se t h a t th o se s tu d e n ts d id n o t become p o o re r
w r ite r s d u rin g th e y e a r 's t r a i n i n g p e r io d .

Thus i t seems re a s o n a b le to

conclude t h a t th e e v a lu a tio n p ro ced u re was a t f a u l t .

I f th e e v a lu a tio n

pro ced u re was r e s p o n s ib le f o r th e a p p a re n t r e g r e s s io n in w r itin g a b i l i t y
f o r many s tu d e n ts , i t seems re a s o n a b le to assume t h a t th e e v a lu a tio n p ro ­
cedure a ls o r e s u l t e d i n a p p a re n t g a in s w hich w ere ex a g g erate d f o r o t h e r s .
A t b e s t , i t would seem t h a t th e u s e fu ln e s s o f c u r r e n t e v a lu a tio n
p ro ced u res i n t h i s a re a has been lim ite d t o p ro v id in g th e te a c h e r w ith some
evidence t h a t a group o f s t u d e n t s , a s a w h o le, has made some improvem ent,
and t h a t such improvement i s r a t h e r s m a ll.

E v a lu a tio n p ro ced u res have

f a i l e d to p ro v id e th e te a c h e r th e p e r t i n e n t in fo rm a tio n about in d iv id u a l

28

improvement which he needs b e fo re he ca n i n t e l l i g e n t l y approach th e ta s k
of

im proving h i s te a c h in g m ethods and p ro c e d u re s .

As T y le r s t a t e s , one

of

th e im p o rta n t p u rp o ses o f e v a lu a tio n " , . . i s to p ro v id e in fo rm a tio n

b a s ic to e f f e c t iv e guidance o f in d iv id u a l s tu d e n ts . . . .
judgment t h a t he i s
enough.

M erely th e

doing a v e ra g e work i n a p a r t i c u l a r co u rse i s n o t

We need to

f in d o u t more a c c u r a te ly where he i s p ro g re s s in g
1*6
where he i s h av in g d i f f i c u l t i e s . ”

and

F or t h i s p u rp o s e , we n ee d , f i r s t o f a l l , r e l i a b l e and v a lid r a t i n g s
of

th e stu d e n t* s w r i t i n g .

Second, we need t o have a r e l i a b l e sample o f

a s tu d e n t* s w r itin g a t th e tim e h is achievem ent i s b e in g e v a lu a te d .

And,

i f a r e l i a b l e sample can n o t be o b ta in e d , th e n enough sam ples should be
o b ta in e d to p ro v id e

a r e l i a b l e e s tim a te o f w r itin g a b i l i t y .

exam ple, th e q u a l i t y o f a s tu d e n t* s w r itin g v a r ie s

I f , fo r

from to p ic to t o p i c ,

th e n we would need sam ples o f h is w r i tin g on d i f f e r e n t to p ic s ; a n d , i f th e
q u a l ity o f h is w r itin g v a r ie s from tim e to tim e , th e n we would need sam ples
o f h is w ritin g a t d i f f e r e n t tim e s ; a n d , i f th e p s y c h o lo g ic a l p r e s s u r e o f
th e f i n a l ex am in atio n s i t u a t i o n has an a d v e rse e f f e c t on th e q u a l ity o f a
stu d e n t* s w r i t i n g , th e n we would need sam ples o f h i s w r itin g u n d er o th e r
c o n d itio n s ; a n d , i f th e q u a l i t y o f s tu d e n t w r itin g v a r ie s i n amount from
to p ic to to p ic and from day to day in d i r e c t p r o p o rtio n to s tu d e n t a b i l i t y
i n w r i t i n g , th e n we would need more sam ples o f w r i tin g f o r th o se w ith more
a b i l i t y th a n f o r th o s e w ith l e s s a b i l i t y - - i n o rd e r to o b ta in a v a lid e s t i ­
mate o f each s t u d e n t 's w r i t i n g a b i l i t y .

1*6. T y le r , o p . c i t . , p p . 8-9

29

The purpose o f t h i s t h e s i s was to t e s t th e s e b a s ic assum ptions on
th e s t a b i l i t y o f a s t u d e n t 's w r itin g p erfo rm an ce—to d eterm in e w hether a
s in g le p ap er w r i t t e n by a s tu d e n t on a g iv en to p ic a t a p a r t i c u l a r tim e
can be c o n sid e re d a s a r e p r e s e n ta ti v e sam ple o f h is w r itin g a b i l i t y , and
th u s p ro v id e a v a l i d b a s is f o r e v a lu a tin g a s t u d e n t 's a b i l i t y a t any tim e
i n a w r itin g c o u r s e .

CHAPTER I I I

DESIGN AND PROCEDURE FOR CONDUCTING THa STUDY

As p r e v io u s ly s t a t e d , f o u r b a s ic assu m p tio n s a r e in h e r e n t i n th e use
o f a s in g le p a p e r w r i tte n by a s tu d e n t on a g iv en to p ic a t a p a r t i c u l a r
tim e a s a b a s is f o r e v a lu a tin g h is ach iev em en t a t any tim e i n a w r itin g
co u rse.

Such a p r a c t ic e assum es th a t*

1 . any given to p ic p ro v id e s th e same s tim u lu s a s any o th e r t o p i c ,
2 . any g iv e n to p ic e l i c i t s c o n s ta n t re sp o n se s a t d i f f e r e n t tim e s ,
3 . th e p s y c h o lo g ic a l p r e s s u r e in tro d u c e d by th e ex am in atio n s i t u ­
a tio n has no ad v erse e f f e c t on th e q u a lity o f s tu d e n t w r i t i n g ,
and
U. th e q u a lity o f s tu d e n t w r i tin g i s s t a b l e from to p ic to to p ic
and from tim e to tim e , w ith o r w ith o u t th e p r e s s u re o f an
e x a m in a tio n , r e g a r d le s s o f v a r ia tio n s i n s tu d e n t w r itin g a b i l i t y .

A ssignm ents and D esign
S ince th e p u rpose o f t h i s t h e s i s was t o t e s t th e s e fo u r a ssu m p tio n s,
s p e c i f i c w r itin g assig n m en ts and s p e c i f i c c o n d itio n s r e l a t i v e to each
assum ption were p ro v id e d f o r groups o f s tu d e n ts e n r o lle d in th e f i r s t term
o f th e W ritte n and Spoken English^" co u rse a t M ichigan S ta te C o lle g e .

T his

i s a re q u ire d th r e e -te rm ( q u a r te r ) c o u rse i n th e B asic C o lle g e , which i s a
g e n e ra l e d u c a tio n program .

S tu d e n ts r e c e iv e t r a i n i n g in fo u r com munication

1 . I n 1952, th e name o f th e c o u rse was o f f i c i a l l y changed to '•Communication
S k ills .

31

s k ills s

w r itin g , s p e a k in g , re a d in g and l i s t e n i n g .

They a tte n d c l a s s e s

f iv e hours each week—one hour o f l e c t u r e , two hours o f r e c i t a t i o n , and
two hours o f la b o r a to r y .

A ll p a p e rs a r e w r i t t e n in c l a s s d u rin g th e s e

tw o-hour la b o r a to r y p e r i o d s .
R equirem ents f o r th e f i r s t a ssu m p tio n .

To t e s t th e f i r s t assu m p tio n—

t h a t any giv en to p ic p ro v id e s th e same s tim u lu s a s any o th e r t o p ic — i t
was n e c e ss a ry to have s tu d e n ts w r ite p a p e rs on two o r more to p ic s d u rin g
one w r itin g p e r io d , so t h a t th e g e n e ra l e f f ic ie n c y l e v e l o f th e s tu d e n ts
would e n te r in to th e w r itin g f o r each to p ic i n ab o u t th e same m anner.
The s e l e c t i o n o f th e to p ic s p re s e n te d two p ro b lem s, one co n cern in g th e
n a tu re o f th e to p ic s and th e o th e r c o n c ern in g th e number o f to p ic s to be
a s s ig n e d .
A lthough D ie d e ric h

2

has recommended t h a t s im ila r to p ic s be used in

o rd e r to o b ta in th e b e s t e s tim a te o f a s t u d e n t 's w r itin g a b i l i t y , f r e ­
q u e n tly th e t r a i n i n g i n a w r itin g co u rse in v o lv e s more th a n one ty p e o f
w r itin g , o r one ty p e o f t o p i c .

F or exam ple, i n th e W ritte n and Spoken

E n g lish Course a t M ichigan S ta te C o lle g e , th e two assig n m en ts co n sid e re d
n ea r th e end o f th e f i r s t term a re concerned w ith l ) th e developm ent o f
an id e a by th e use o f two o r more m eth o d s, and 2) th e g iv in g o f d ir e c tio n s
o r th e d e s c r ip tio n o f a p r o c e s s .

The q u e s tio n a r i s e s w hether th e q u a lity

o f a s t u d e n t 's w r itin g would be th e same on two s im ila r to p ic s r e l a t e d to
one o r th e o th e r o f th o s e a s s ig n m e n ts .

And, would th e v a r ia tio n s between

2 . P a u l B. D ie d e ric h , "The M easurement o f S k i l l i n W ritin g ," School Review ,
v o l. 55 ( A p r il 19U8), p p . 375-8U.

32

two p a p e rs on s im ila r to p ic s r e l a t e d to one assig n m en t be any d i f f e r e n t
th a n on d i s s i m i l a r to p ic s r e l a t e d to th e two assig n m e n ts?
In o rd e r to r e s o lv e t h i s , i t was n e c e s s a ry to a s s ig n b o th s im ila r
and d i s s i m i l a r t o p i c s .

By com bining two s im ila r to p ic s w ith one d is s im il a r

t o p i c , o n ly th r e e to p ic s were n e c e ss a ry to p ro v id e a s a t i s f a c t o r y t e s t f o r
th e f i r s t assu m p tio n .
The s p e c i f i c s e l e c t i o n o f th e to p ic s was made in c o lla b o r a tio n w ith
th e two i n s t r u c t o r s from th e W ritte n and Spoken E n g lis h D ep artm en t, whose
s tu d e n ts w rote th e p ap ers used i n t h i s stu d y .

S in ce th e p a p e rs were to

be w r i tte n n e a r th e end o f th e te rm , th e s e le c tio n o f th e to p ic s was
lim ite d to th e n a tu re o f th e two co u rse assig n m en ts d u rin g t h a t p e r io d .
I t was a g re e d t h a t th e two s im ila r to p ic s co u ld b e s t be s e le c te d from th e
course assig n m en t co n c ern in g 11th e developm ent o f an id e a by th e u se o f
two o r more methods ," and t h a t th e d is s im ila r to p ic co u ld be s e le c te d from
th e assig n m en t co n cern in g "th e g iv in g o f d i r e c t i o n s o r th e d e s c r ip tio n of
a p r o c e s s ."

T h u s, th e two s im ila r to p ic s co uld be developed by th e use

o f s im ila r m eth o d s, w ith th e c o n te n t p ro v id in g th e m ajor v a r i a b l e .

On

th e o th e r h an d , th e d is s im il a r to p ic would d i f f e r from th e o th e r two to p ic s
n o t only i n c o n te n t b u t, a l s o , i n th e method needed f o r developm ent.
I n s o f a r as p o s s i b l e , to p ic s were s e le c te d c o n c ern in g m a te r ia l common
to a l l s tu d e n ts e n r o lle d i n th e c o u r s e .
w ere;

The two s im ila r to p ic s a s s ig n e d

l ) " I t l a ( I s Not) Too F a r Between C lassroom s f o r S tu d en ts to T ra v e l

D uring th e Ten-4!4inute P erio d A llow ed," and 2) "Textbooks f o r Freslimen a t
M ichigan S ta te C o lleg e Are (Are Not) Too E x p en siv e."

Each s tu d e n t was

p ro v id ed a b ro ch u re o f in fo rm a tio n p e r ta in in g to each t o p i c .

33

The d i s s i m i l a r to p ic a s s ig n e d was:

"Give D ir e c tio n s to a S tra n g e r

a t th e Union B u ild in g E nabling Him to Get to B u ild in g A-6 on South
Campus."

F or t h i s t o p i c , each s tu d e n t was p ro v id e d a map o f th e campus.

(For co n v e n ie n c e , h e n c e f o r th , th e s e th r e e to p ic s w i l l be r e f e r r e d to i n
a b b re v ia te d form a s :

"D istan ce Between C la s s e s " , "C ost o f T ex tb o o k s",

and "G iving D ir e c tio n s " , r e s p e c t i v e l y .)
These th r e e to p ic s p ro v id e d th e n e c e ss a ry v a r ia tio n s i n s ti m u li to
t e s t th e assu m p tio n t h a t "any giv en to p ic p ro v id e s th e same s tim u lu s as
any o th e r t o p i c . "

I t would have been d e s ir a b le to have each s tu d e n t w rite

on a l l th r e e to p i c s —two s im ila r and one d i s s i m i l a r —d u rin g one w r itin g
p e r io d .

Such a p ro c e d u re , how ever, would have in tro d u c e d th e p o s s i b i l i t y

o f f a tig u e d ev e lo p in g to th e e x te n t t h a t th e e f f ic ie n c y l e v e l o f th e
s tu d e n ts would be reduced c o n s id e ra b ly b e fo re th e ta s k was co m p leted .
Such a n e f f e c t would make i t Im p o ssib le to d eterm in e v a r ia tio n s in th e
q u a l ity o f w r i tin g due to th e to p ic s a s s ig n e d .

T h e re fo re , i t was c o n sid e re d

a d v is a b le to r e s t r i c t th e w r itin g assig n m en ts to two to p ic s d u rin g a s in g le
tw o-hour w r itin g p e r io d .
W ith such a r e s t r i c t i o n , i t was n e c e ss a ry to use two groups o f s t u ­
d e n ts i n o rd e r to compare v a r ia tio n s i n th e q u a l ity o f w r itin g between
p ap e rs on s im ila r to p ic s w ith v a r ia tio n s betw een p ap e rs on d is s im il a r t o p ic s .
Group A was asked to w r ite a p ap e r on each o f th e two s im ila r t o p i c s .
Group B was asked to w rite a p a p e r on one o f th e s im ila r t o p i c s , "D istan ce
Between C la s s e s " , and a n o th e r p a p e r on th e d is s im il a r t o p i c , "G iving D i­
r e c tio n s " .

T h is p ro ced u re p ro v id e d a b a s is f o r com paring b o th th e e f f e c t

o f s im ila r to p ic s and th e e f f e c t o f d is s im il a r to p ic s on th e q u a lity of

3b

s tu d e n t w r itin g —th e re b y p ro v id in g a b a s is f o r t e s t i n g th e assu m p tion
t h a t "any giv en to p ic p ro v id e s th e same stim u lu s a s any o th e r t o p i c " .
(P ro v is io n f o r t e s t i n g w h eth er f a tig u e e n te re d in to th e w r itin g o f th e
second paper i s e x p la in e d l a t e r u n d er " C o n tro l o f th e f a tig u e f a c t o r " . )
Requirem ents f o r th e second a ssu m p tio n .

To t e s t th e second assu m p tio n —

t h a t any giv en to p ic e l i c i t s c o n s ta n t re sp o n se s a t d i f f e r e n t tim e s —i t
was n e c e ssa ry to have s tu d e n ts w rite on th e same to p ic on d i f f e r e n t d a y s.
T hus, by r e p e a tin g th e assig n m e n ts d e s c rib e d f o r t e s t i n g th e f i r s t assump­
t i o n , a b a s is was p ro v id e d f o r t e s t i n g th e second a ssu m p tio n .

In f a c t ,

th e assignm en t o f th e same to p ic s to th e same s tu d e n ts on a second day
p ro v id ed s e ts o f p a p e rs on th e th r e e d i f f e r e n t to p ic s as a b a s i s f o r com­
p a rin g th e q u a l i t y o f s tu d e n t w r itin g on d i f f e r e n t d ay s—a b a s is f o r
d e te rm in in g w h eth er v a r ia tio n s i n e f f ic ie n c y from day to day have any
e f f e c t on th e q u a l i t y o f s tu d e n t w r i tin g .
Requirem ents f o r th e t h i r d assu m p tio n .

To t e s t th e t h i r d assu m p tio n —

t h a t th e .p sy c h o lo g ic a l p r e s s u r e o f th e ex am in atio n s i t u a t i o n has no ad v e rse
e f f e c t on th e q u a l i t y o f s tu d e n t w r itin g — i t was n e c e s s a ry to have s tu d e n ts
w rite w ith o u t t h a t p re s s u re a t one tim e and to w r ite on th e same to p ic
w ith tli a t p r e s s u r e a t a n o th e r tim e .
Two o th e r groups o f s tu d e n ts , groups C and D, were used f o r t h i s p u r­
p o se .

They became th e p re s s u re g ro u p , w h ile groups A and B serv ed a s a

c o n tro l—w r itin g on b o th days w ith o u t th e p s y c h o lo g ic a l p re s s u re o f th e
exam ination s i t u a t i o n .
I t was n e c e s s a ry to have th e s tu d e n ts used f o r t e s t i n g th e t h i r d
assum ption ( th e p re s s u re groups C and D) w rite t h e i r f i r s t s e t o f p ap e rs

35

under th e same c o n d itio n s and w ith th e same assig n m en ts used f o r th e
c o n tro l group.
to p ic s*

T h u s, group C was a s s ig n e d to w rite on th e two s im ila r

"D ista n c e Between C la s s e s " and "C ost o f T ex tb o o k s", and group D

was a ssig n e d t o w r ite on th e two d i s s i m i l a r to p ic s :
C la sse s" and "G iv in g D ir e c t io n s " .

"D istan ce Between

Then by adding th e ex am in atio n s i t u a t i o n

as th e o nly v a r ia b le f o r th e s e p re s s u re s tu d e n ts on th e second d a y , a b a s is
was p ro v id ed f o r d e te rm in in g th e e f f e c t o f th e ex am in atio n s i t u a t i o n on
th e q u a lity o f s tu d e n t w r i t i n g .
R equirem ents f o r th e f o u r th assu m p tio n .

To t e s t th e f o u r th assu m p tio n —

t h a t th e q u a l ity o f s tu d e n t w r itin g i s s ta b le from to p ic to to p ic and from
tim e to tim e , w ith o r w ith o u t th e p r e s s u r e o f an ex a m in a tio n , r e g a r d le s s
of v a r ia tio n s i n w r itin g a b i l i t y —i t was n e c e ssa ry to ran k th e s tu d e n ts
ac c o rd in g to t h e i r w r itin g a b i l i t y and th e n to a n a ly z e th e d eg ree and f r e ­
quency o f v a r i a t i o n s i n th e q u a l ity o f w r itin g in r e l a t i o n to th e ran k o f
in d iv id u a ls .
Summary.

(The method o f ra n k in g i s d e s c rib e d i n C hapter V.)

The fo llo w in g w r itin g assig n m en ts p ro v id e d th e b a se s f o r t e s t i n g

each o f th e f i r s t th r e e assu m p tio n s:
1 . F or th e assu m p tio n t h a t any given to p ic p ro v id e s th e same stim u lu s
as any o th e r t o p i c , two p ap e rs were w r i tte n d u rin g a s in g le w r itin g p e rio d
by each s tu d e n t i n Group A on s im ila r t o p i c s , and i n Group B on d i s s i m i l a r
to p ic s .
2 . For th e assu m p tio n t h a t any g iv en to p ic e l i c i t s c o n s ta n t re sp o n se s
a t d i f f e r e n t tim e s , th e i n i t i a l assig n m en ts f o r Groups A and B were r e ­
p e a te d .

36

3.

F or th e assum ption t h a t th e p s y c h o lo g ic a l p re s s u re in tro d u c e d by

th e ex am in atio n s i t u a t i o n lias no a d v e rs e e f f e c t on th e q u a l ity o f s tu d e n t
w r itin g , two p ap ers were w r itte n b y each s tu d e n t i n Group C on s im ila r
to p ic s and i n Group D on d is s im il a r t o p i c s , w ith o u t th e ex am in atio n p r e s ­
su re on th e f i r s t day and w ith t h a t p r e s s u r e on th e second d ay .
D esign i n ta b u la r fo rm .
assignm ents
Key to

The s i m i l a r i t i e s and v a r ia tio n s in th e w r itin g

f o r th e f o u r groups o f

s tu d e n ts a r e shown below i n ta b u la r form :

ta b u la t io n : T opic 1 - ’’D ista n c e Between C lasses*
T opic 2 - ’’C o st o f T extbooks”
T opic 3 - ” G iving D ir e c tio n s ”
W/0 - W ritin g w ith o u t th e ex am in atio n p re s s u re
W - W ritin g w ith th e ex am in atio n p re s s u re
F i r s t Day
S im ila r T opics
T opic 1 Topic 2

Second Day
S im ila r T opics
Topic 1 Topic 2

Group A

W/O

W/0

W/0

W/0

Group C

W/0

W/0

w

w

D is s im ila r T opics
Topic 1
Topic 3

D is s im ila r T opics
Topic 1
Topic 3

Group B

W/O

W/0

W/0

W/0

Group D

W /0

W/0

w

w

C o n tro ls P rovided
A lthough th e arrangem ent o f assig n m e n ts and c o n d itio n s shown above
p ro v id ed th e b a s ic d esig n n e c e ss a ry f o r t e s t i n g th e f o u r assum ptions i n ­
volved i n t h i s s tu d y , c e r t a i n p re c a u tio n s and c o n tr o ls were n e c e s s a ry to
p re c lu d e th e i n t r u s i o n o f e x tra n e o u s f a c t o r s in to th e w r itin g s i t u a t i o n s

37

which would in v a lid a te th e f i n d i n g s .

I t was c o n sid e re d a d v is a b le to

s e l e c t groups o f s tu d e n ts whose sc h e d u le o f c la s s e s co rresp o n d ed w ith th e
d esig n o f th e stu d y so t h a t th e n e c e s s a ry assig n m en ts co u ld be made w ith o u t
c r e a tin g an a r t i f i c i a l s i t u a t i o n .

C o n tro ls were p ro v id e d t o :

l ) p re v e n t

f a tig u e from c o u n te r a c tin g th e e f f e c t o f d i f f e r e n t to p ic s a s s ig n e d d u rin g
th e same w r itin g p e r io d ,* 2) p ro v id e th e a p p r o p r ia te m o tiv a tio n f o r each
w r itin g s i t u a t i o n , and 3) p ro v id e th e a p p r o p r ia te le n g th o f tim e betw een
th e two w r itin g p e rio d s to p re v e n t undue d i s t o r t i o n from new le a r n in g
e x p e rie n c e s o r from memory o f th e f i r s t w r i tin g e x p e rie n c e .
S e le c tio n o f G roups.

Through th e c o o p e ra tio n o f th e head o f th e W ritte n

and Spoken E n g lis h d e p a rtm e n t, and two s t a f f members o f t h a t d e p a rtm e n t,
fo u r groups o f s tu d e n ts were p ro v id ed f o r w r itin g th e a p p r o p r ia te number
o f p a p e r s , and under th e a p p r o p r ia te c o n d itio n s , n e c e s s a ry f o r t h i s s tu d y .
Each s t a f f member was te a c h in g two groups o f b e g in n in g s tu d e n ts i n
th e c o u r s e .

Both i n s t r u c t o r s met each o f t h e i r two groups on th e same

d ay s, Tuesdays and T h u rsd ay s, f o r two h ours each d ay .

Each group c o n ta in e d

a s u f f i c i e n t number o f s tu d e n ts so t h a t , b a r r in g an u n u su a l o ccu rren ce o f
a b s e n c e s, a minimum o f tw enty s tu d e n ts would be p r e s e n t on b o th days th e
w ritin g assig n m en ts were to ta k e p la c e .

T h u s, th e s e f o u r groups would

p ro v id e e ig h ty s tu d e n ts , each o f whom would w r ite fo u r p a p e rs under th e
c o n d itio n s d e s c rib e d a b o v e .
S ince th e s tu d e n ts i n th e s e groups were i n th e h a b it o f w r itin g
p a p e rs i n c l a s s d u rin g a tw o-hour la b o r a to r y , th e w r itin g assig n m en ts f o r
t l i i s stu d y were made to c o in c id e w ith th o se la b o r a to r y p e r io d s —th u s p ro ­
v id in g f o r a minimum d e p a rtu re from th e norm al classro o m p ro c e d u re .

38

C o n tro l o f F a tig u e F a c t o r s :

Having s tu d e n ts w r ite two p a p e rs d u rin g a

tw o-hour p e r io d co u ld e a s i l y r e s u l t i n a c e r t a i n amount o f f a t i g u e d u rin g
th e second h a l f o f t h a t p e r io d .

Such f a t i g u e , i f i t o c c u rre d , m ight w ell

b r in g a b o u t a lo w e rin g o f th e e f f ic ie n c y l e v e l o f th e s t u d e n t s , th e re b y
lo w e rin g th e q u a l i t y o f th e w r itin g d u rin g th e second h o u r.

T h u s, such

f a t i g u e would in v a lid a te th e f in d in g s i n r e l a t i o n • to th e e f f e c t o f th e
t o p ic s on th e q u a l i t y o f th e w r i tin g f o r t h a t p e r io d .
To re d u c e th e p o s s i b i l i t y o f f a t i g u e , th e s tu d e n ts were i n s t r u c t e d to
com plete th e f i r s t p a p e r d u rin g th e f i r s t hour and to ta k e a te n -m in u te
b reak b e fo re s t a r t i n g th e second t o p i c .
To p ro v id e a check on th e e f f e c t o f th e f a t i g u e f a c t o r , e v e ry o th e r
s tu d e n t was in s t r u c t e d to w r ite on th e f i r s t to p ic d u rin g th e f i r s t hour
and on th e second to p ic d u rin g th e second h o u r.

The o th e r s tu d e n ts were

in s t r u c t e d t o w r ite on th e second to p ic d u rin g th e f i r s t hour and on th e
f i r s t to p ic d u rin g th e second h o u r.

T h is p ro ced u re p ro v id e d a b a s is f o r

d e te rm in in g w h eth er th e s tu d e n ts w r i tin g on a s p e c i f i c to p ic d u rin g th e
f i r s t hour d id b e t t e r th a n th o s e w r i tin g on th e same to p ic d u rin g th e
second h o u r.
C o n tro l o f M o tiv a tio n F a c t o r s :

Having a s tu d e n t w r ite a n e s s a y a s a p a r t

o f an ex am in atio n wldlch i s t o d eterm in e h is grade f o r th e term i s u s u a lly
c o n s id e re d a s a m o tiv a tin g f a c t o r , a s w e ll a s a p s y c h o lo g ic a l p r e s s u r e
f a c t o r w hich may d is tu r b him .

T h at i s , th e rew ard o f a good grade o r th e

f e a r o f a p o o r grade w i l l u s u a lly s tim u la te th e s tu d e n t to t r y to do good
w ork.

On th e o th e r h an d , th e r o u tin e w r itin g assig n m e n t i n th e W ritte n

39

and Spoken E n g lis h co u rse a ls o i s a f a c t o r , a lth o u g h a minor o n e, i n
d e term in in g th e s t u d e n t 's end term g ra d e .
I n o rd e r to b e t t e r d eterm in e th e e f f e c t o f th e p s y c h o lo g ic a l p re s s u re
o f th e e x a m in a tio n , i t was d ecid ed to e lim in a te th e grade f a c t o r (a s a
p o s s ib le d is tu rb a n c e ) from th e w ritin g s i t u a t i o n on th e f i r s t day f o r a l l
o f th e s tu d e n ts in v o lv e d , and t o e lim in a te th e grade f a c t o r on th e second
day f o r th e n o n -p re ssu re s tu d e n ts , groups A and B, who se rv e d as th e con­
t r o l group i n r e l a t i o n to th e p r e s s u re f a c t o r .
E lim in a tio n o f th e grade f a c t o r r a is e d a m o tiv a tio n problem f o r a l l
o f th e s tu d e n ts on th e f i r s t d a y 's a s sig n m e n ts, and f o r th e n o n -p re ssu re
s tu d e n ts on th e second d a y 's assig n m e n ts—s in c e i t was d e s ir a b le t h a t a l l
s tu d e n ts sho u ld a tte m p t to w rite good q u a lity p a p e rs f o r a l l assig n m e n ts.
I t was d ec id ed to r e p la c e th e grade f a c t o r w ith an e x p la n a tio n o f
th e a s sig n m e n ts, in d ic a tin g how th e w r itin g assig n m en ts could be of
v alu e to th e s t u d e n t s .

On th e second day th e grade f a c t o r was in v o lv ed

o n ly f o r th e p re s s u re s tu d e n ts , groups C and D.

F or th o se s tu d e n ts on

t h a t d ay , th e grade f a c t o r was i n t e n s i f i e d by p ro v id in g an exam ination
s itu a tio n .
In s tru c tio n s t

The n o n -p re ssu re s tu d e n ts , groups A and B , re c e iv e d th e

fo llo w in g in s t r u c t i o n s to p ro v id e th e n e c e ss a ry m o tiv a tio n i n p la c e o f
th e grade f a c t o r on th e f i r s t day*
I t i s b e lie v e d , a s a r e s u l t o f s e v e r a l s t u d i e s , t h a t i n o rd er
f o r a s tu d e n t to have a f a i r o p p o rtu n ity t o d em onstrate h is
w r itin g a b i l i t y , he sh o u ld w rite on two d i f f e r e n t to p ic s on
th e same d ay . Your grade f o r th e term w ill n o t be a f f e c te d by
th e p a p e rs you w r ite to d a y . However, yo u r p a p e rs w i l l be
graded by a team o f r a t e r s . You w i l l be inform ed o f th e grade
given y o u r p ap ers so t h a t you can compare th o se grades w ith
th o se you have re c e iv e d on p re v io u s p a p e rs . A lso , th o se

Uo

grades sh o u ld give you a b e t t e r id e a o f your a c tu a l w r itin g
a b i l i t y a t p r e s e n t th a n does a grad e on any s in g le p a p e r.
On th e second d a y , th e n o n -p re s su re s tu d e n ts r e c e iv e d th e fo llo w in g
e x p la n a tio n :
As a r e s u l t o f s e v e r a l s t u d i e s , i t i s b e lie v e d t h a t i n o rd e r f o r
a s tu d e n t to d em o n strate h is w r itin g a b i l i t y , he should w rite
on two d i f f e r e n t to p ic s on th e same day and th e n r e p e a t th e p e r­
formance a few days l a t e r . Today you a re ask ed a g a in to w rite
on th e same to p ic s a s s ig n e d a t th e l a s t w r itin g p e r io d . A gain
your grade f o r th e term w i l l n o t be a f f e c t e d by th e grade you
r e c e iv e on th e s e p a p e rs . However, your p ap e rs w i l l be graded by
a team of r a t e r s to d is c o v e r w hether you can w rite b e t t e r on one
to p ic th a n on a n o th e r , and w hether you can w rite b e t t e r on one
day th a n on a n o t h e r . T h is in fo rm a tio n sh o u ld be v a lu a b le to you
d u rin g your n e x t two term s o f work i n th e c o u r s e . The r e s u l t s
o f th e g rad in g w i l l be p o s te d on th e b u l l e t i n board e a r l y n ex t
te rm .
For th e p re s s u re s t u d e n t s , groups C and D, th e e x p la n a tio n f o r th e
f i r s t day*s assignm ent was th e same a s f o r th e n o n -p re ssu re s tu d e n ts .

On

th e second d ay , in o rd e r to p ro v id e th e exam ination p r e s s u r e , th e fo llo w ­
in g e x p la n a tio n was p la c e d i n th e assig n m e n t:
D uring th e l a s t w r itin g p e rio d you w rote two themes which d id n o t
a f f e c t your grade f o r th e te rm . As p ro m ise d , your p ap ers w i l l be
graded by a team o f r a t e r s to d eterm in e w hether you w rote b e t t e r
on one to p ic th a n on th e o th e r t o p i c . These r e s u l t s w i l l be p o s t­
ed on th e b u l l e t i n b o ard a s soon a s p o s s i b le . I t i s f e l t , how ever,
t h a t a f t e r h av in g w r i tte n on a to p ic once you sh o u ld be a b le to do
a b e t t e r jo b o f w r itin g i f g iven a n o th e r o p p o rtu n ity to w rite on
th e same t o p i c . Today you a r e to w rite on th e same two to p ic s
a g a in . S ince you have had a p r a c tic e p e rio d f o r w r itin g th e s e two
p a p e r s , th e themes you w r ite to d a y w i l l be c o n s id e re d a s your f i n a l
and b e s t e f f o r t . They w i l l be c o n s id e re d a s a p a r t o f your f i n a l
ex am in atio n , and th e g rad es on th e s e two p ap ers w i l l be counted in
w ith your sp eech grades and your f i n a l exam ination grade to make
up your grade f o r th e te rm .
(These e x p la n a tio n s were e f f e c t i v e to th e e x te n t t h a t s tu d e n ts began
to in q u ire ab o u t th e r e s u l t s b e fo re th e y were p o s te d .)
I t was f e l t t h a t th e above e x p la n a tio n s f o r th e n o n -p re ssu re s i t u ­
a tio n s would p ro v id e th e s tu d e n ts w ith a reaso n f o r co m p letin g th e

ill

a ssig n m e n ts, b u t t h a t such e x p la n a tio n s would p ro v id e a minimum of
s tu d e n t concern ab o u t th e grades a s s ig n e d t h e i r p a p e r s .

T hus, i f th e

f i n a l exam inatio n p re s s u re has an ad v e rse e f f e c t on th e q u a lity o f s tu d e n t
w r i tin g , t h a t e f f e c t sh o u ld be a p p a re n t from th e v a r ia tio n s i n th e q u a lity
o f s tu d e n t w r itin g noted from day to day f o r th e p r e s s u r e s tu d e n ts in
com parison w ith th o se n o ted from day to day f o r th e n o n -p re ssu re s tu d e n ts .
C o n tro l o f th e tim e f a c t o r x The amount o f tim e e la p s in g betw een th e
w r itin g o f th e f i r s t and second s e t o f p a p e rs on th e same to p ic s was
co n sid e re d in r e l a t i o n to two problem s.

F i r s t , i f th e tim e betw een th e

w ritin g s i s to o s h o r t , th e s tu d e n ts may remember r a t h e r c l e a r l y how th e y
w rote th e f i r s t p a p e r.

As a r e s u l t , th e second p ap er on th e same to p ic

may be m erely a second e d i tio n o f th e f i r s t paper w ith o u t a re -e x a m in a tio n
of th e problem o f d ev elo p in g th e to p ic a s s ig n e d —w hich, o th e rw is e , would
be n e c e ss a ry f o r th e w r itin g o f a new p ap er on th e same to p ic .
On th e o th e r han d , i f too much tim e e la p s e s between th e w ritin g o f
th e f i r s t and second p ap e r on th e same t o p i c , new le a r n in g ex p e rien ce s
may w ell be th e cause f o r many o f th e d if f e r e n c e s which may be n o ted —
in s te a d o f such d if f e r e n c e s bein g caused by v a r ia tio n s in e f f ic ie n c y .
A compromise had to be made betw een th e s e two extrem es in o rd e r to
redu ce to a minimum th e e f f e c t s o f memory and th e e f f e c t s o f new le a r n in g
e x p e rie n c e s .

S in ce th e s tu d e n ts were e n r o lle d in th e W ritte n and Spoken

E n g lish c o u r s e , th e compromise had to fa v o r th e s h o r te r tim e .

O th erw ise,

th e e f f e c t o f new le a r n in g ex p e rien ce s m ight r e s u l t i n v a r ia tio n s in th e
q u a l ity o f w r itin g f o r a c o n s id e ra b le number o f s tu d e n ts .

T h u s, i t was

c o n sid e re d a d v is a b le to sp ace th e two w ritin g p e rio d s w ith o u t any in te rv e n in g

1x2

w r i t i n g p e r i o d s f o r r e g u l a r c l a s s w o rk .
su c h p e r i o d s was one w eek .

The maximum tim e o etw een two

T h e r e f o r e , i t was a r r a n g e d t o hav e t h e two

s e t s o f p a p e r s on t h e same t o p i c s w r i t t e n d u r i n g t h e w r i t i n g l a b o r a t o r y
p e r i o d s o f e a c h o f t h e l a s t two weeks o f t h e t e r m .

T h is arrangem ent

p r o v i d e d a n a t u r a l tim e t o i n t r o d u c e t h e f i n a l e x a m i n a t i o n s i t u a t i o n i n t o
t h e w r i t i n g a s s i g n m e n t f o r t h e p r e s s u r e s t u d e n t s on t h e se c o n d d a y .
Summary.

The d e s i g n f o r t h i s s t u d y c o n s i s t e d o f h a v i n g f o u r g ro u p s o f

s t u d e n t s w r i t e two p a p e r s d u r i n g a s i n g l e w r i t i n g p e r i o d on e a c h o f two
days.

T h ese f o u r g ro u p s w ere g i v e n s p e c i f i c a s s i g n m e n t s u n d e r t h e a p p r o ­

p r i a t e c o n d i t i o n s t o p r o v i d e a o a s i s f o r t e s t i n g t h e f o u r a s s u m p tio n s
w h ich a r e t h e c o n c e r n o f t l i i s s t u d y .
Groups A and B w ro te on s i m i l a r and d i s s i m i l a r t o p i c s , r e s p e c t i v e l y ,
u n d e r t h e same c o n d i t i o n s on e a c h o f two d i f f e r e n t d a y s — t h u s p r o v i d i n g
a o a s i s f o r t e s t i n g t h e e f f e c t o f t o p i c s and t h e e f f e c t o f v a r i a t i o n s i n
e f f i c i e n c y fro m d a y t o d ay on t h e q u a l i t y o f s t u d e n t w r i t i n g , a s w e l l a s
p r o v i d i n g a c o n t r o l group i n r e l a t i o n t o t e s t i n g t h e p r e s s u r e f a c t o r .
Groups C and D a l s o w r o te on s i m i l a r a n d d i s s i m i l a r t o p i c s , r e ­
s p e c t i v e l y , on e a c h o f two d i f f e r e n t d a y s — w i t h t h e e x a m i n a t i o n s i t u a t i o n
on t h e se c o n d d a y p r o v i d i n g t h e v a r i a b l e t o b e t e s t e d i n r e l a t i o n t o i t s
e f f e c t on t h e q u a l i t y o f s t u d e n t w r i t i n g .
I n a d d i t i o n , c o n t r o l s were p r o v i d e d , i n s o f a r a s p o s s i b l e , t o i s o l a t e
th e v a r i a b l e s b e in g t e s t e d .

T h ese c o n t r o l s w ere c o n c e r n e d , p r i m a r i l y ,

w i t h p r o v i d i n g g ro u p s w h ich c o u ld w r i t e -under u n if o r m c o n d i t i o n s , p r o v i d ­
i n g a n e q u a l amount o f tim e f o r w r i t i n g e a c h p a p e r , p r o v i d i n g u n ifo rm

h3

m o tiv a tio n when th e p re s s u re f a c t o r was a b s e n t, and p ro v id in g c o n tro ls
to reduce to a minimum th e e f f e c t s o f f a t i g u e , o f memory, and o f new
le a r n in g e x p e rie n c e s .

CHAPTER IV

METHOD FOR EVALUATING THE QUALITY OF STUDENT WRITING

As e x p la in e d in th e p re c e d in g c h a p te r , assig n m en ts and c o n d itio n s
were p ro v id e d f o r i s o l a t i n g th e v a r ia b le in each assum ption to be t e s t e d .
Y e t, in th e f i n a l a n a l y s i s , th o s e assum ptions co uld n o t be t e s t e d u n le s s
th e e r r o r i n e v a lu a tin g th e q u a l ity o f w r itin g was l e s s th a n th e v a r ia tio n s
in th e q u a l ity of w r itin g from to p ic to to p ic and from tim e to tim e .
T h e re fo re , o b ta in in g r e l i a b l e and v a l id r a tin g s o f th e p a p e rs in v o lv ed in
t l d s stu d y was c o n s id e re d a c r u c i a l f a c t o r in s u c c e s s f u lly t e s t i n g th e
fo u r a s su m p tio n s .
In o rd e r to o b ta in th e most v a lid and r e l i a b l e r a tin g s p o s s i b l e , i t
was c o n sid e re d im p o rta n t to s e l e c t th e b e s t a v a ila b le method f o r r a t i n g
th em es, to d eterm in e wliat p ro ced u res have been most s u c c e s s fu l in u sin g
t h a t m ethod, and to e s t a b l i s h c o n tro ls c o n s is te n t w ith th e assum ptions
b e in g t e s t e d .
S e le c tin g th e R a tin g Method
I n o rd e r

to d eterm in e w hether th e theme r a t i n g method employed by th e

W ritte n and Spoken E n g lish D epartm ent d u rin g
o th e r m ethod,

th e

sh o u ld be used f o r t h i s s tu d y , th e

tu r e on theme r a t i n g methods was rev iew ed .

p a s t sev en y e a r s , o r some
p e r t i n e n t re s e a rc h l i t e r a ­

U5
W ritin g s c a l e s :

S h o r tly a f t e r th e tu r n o f th e c e n tu ry , c o n s id e ra b le

a t t e n t i o n was d ev oted to th e developm ent o f w r itin g s c a le s in o rd e r to
improve th e r e l i a b i l i t y o f r a t i n g E n g lish co m p o sitio n s.
H ille g a s ^ was th e f i r s t to a tte m p t a s c i e n t i f i c developm ent o f a
w r itin g s c a l e , which was to p ro v id e f o r a c c u ra te com parisons between th e
q u a lity o f w r itin g a t th e same sch o o l a t d i f f e r e n t tim es o r a t d i f f e r e n t
sch o o ls a t th e same tim e .

In th e developm ent o f th e s c a l e , no a tte m p t

was made to d e f in e d i f f e r e n t q u a l i t i e s o f w r i t i n g .

M e rit was a term used

to in d ic a te t h a t q u a lity wliich com petent judges a g re e d upon.
The th e o ry su b m itte d as th e b a s is f o r d ev elo p in g th e s c a le was t h a t :
•'D ifferen c es tii a t a re e q u a lly o f te n n o tic e d a r e e q u a l, u n le s s th e d i f f e r ences a r e e i t h e r alw ays o r never n o tic e d .”

2

The u n i t i n t h i s s c a le was

d e fin e d a s t h a t d if f e r e n c e wliich e x a c tly s e v e n ty - fiv e p e r c e n t o f th e
judges a r e a b le to d i s t i n g u i s h .

A ll t h a t was re q u ire d to d e riv e t h a t

u n it was a s e t o f sam ples t h a t v a rie d from each o th e r by sm all d eg ree s of
q u a lity .

When two sam ples were found on w hich s e v e n ty -fiv e p e r c e n t o f

th e judges ag reed i n c a l l i n g one b e t t e r th a n th e o th e r , th e d if f e r e n c e
3

between th e two sam ples was one u n it in th e s c a l e .

T rabue worked o u t two supplem ents to th e H ille g a s s c a le in o rd er t o :
l ) sup p ly a need f o r a supplem entary s c a le composed o f com p o sitio n s o f
th e same g e n e ra l ty p e a s th o se w r i tte n by h assau County p u p i l s , and

1 . Milo B. H ille g a s , "S c a le f o r th e Measurement o f Q u a lity in E n g lish
C om position by Young P e o p le ,” T eachers C o lleg e R eco rd , v o l. 13
(Septem ber 1 9 1 2 ), p . 339.
2 . I b i d . , p . 3hh.
3 . I b i d . , P . 31*7.

it6

2) supply some t e n t a t i v e s ta n d a rd s in d ic a t in g th e q u a lity o f E n g lish
com positions to be ex p e c te d from th e p u p ils o f any given sch o o l grade
These s ta te m e n ts su g g e st t h a t co m p o sitio n s must be judged in r e l a t i o n to
th e ty p e o f w r itin g in v o lv e d .

Y et H ille g a s , i n answer to an o b je c tio n

r a is e d r e l a t i v e to com paring n a r r a tiv e and d e s c r ip tiv e w r i tin g , s ta te d
m erely t h a t th e ju d g es in v o lv e d had n o t o ffe re d any such o b je c tio n s .^
S ince th e sam ple co m p o sitio n s used i n c o n s tru c tin g th e H ille g a s and
Trabue s c a le s were c o m p a ra tiv e ly s h o r t (from ap p ro x im ately $0 to 250
words in l e n g t h ) , i t seems d o u b tfu l t h a t judgm ents were made r e l a t i v e to
any a s p e c t o f w r itin g o th e r th a n form—t h a t judgments co u ld have b een
made r e l a t i v e to s u p p o rtin g m a te r ia ls o r to th e o r g a n iz a tio n o f them .
Lyman concludes th a t*

" O b je c tiv e and s p e c i f i c c a te g o r ie s o f e x c e lle n c e

a re g e n e ra lly in a d e q u a te i n co m p o sitio n s c a l e s ." ^

A ls o , s tu d ie s by Dblch^

and Leonard^ in d ic a te t h a t th e r e i s l i t t l e o r no c o r r e la tio n o f e x c e lle n c e
between th e c o n te n t, th e o r g a n iz a tio n , and th e m echanics o f a co m p osition—
t h a t such c h a r a c t e r i s t i c s should be e v a lu a te d s e p a r a te ly .
Inasmuch a s th e W ritte n and Spoken E n g lish co u rse p la c e s a s much
em phasis on th e u se o f r e le v a n t su p p o rtin g m a te r ia ls and th e o r g a n iz a tio n

It. M. R. T rab u e, "S upplem enting th e H ille g a s S c a le ," Teacliers C o lleg e R e co rd ,
v o l. 18, 1917, P . £1 .
5 . H ille g a s , ojo. c i t . , p . 38U.
6 . R. L . Lyman, " I n v e s t ig a ti o n s in th e F ie ld o f W ritte n C o m p o sitio n s,"
Summary o f I n v e s tig a tio n s R e la tin g to Grammar. Language. and C o m p o sitio n s,
U n iv e rs ity o f C h icag o , 192 9 , p . 195.
7 . E . W. D olch, J r . , "More A ccu rate Use o f C om position S c a le s ," E n g lish
J o u r n a l. v o l. 11 (November 1 9 2 2 ), p p . 536-lUi.
8 . S . A . L eonard, " B u ild in g a S cale o f P u re ly C om position Q u a lity ,"
E n g lish J o u r n a l, v o l. ll* (December 1 9 2 5 ), PP. 780-75.

of them as on w r itin g fo rm , any r a t i n g method which f a i l s to e v a lu a te
such c h a r a c t e r i s t i c s would be u n s u ita b le f o r e v a lu a tin g th e p ap e rs
w r itte n f o r t h i s s tu d y .
A lso , a s T h u rsto n e^ has d e m o n stra te d , th e th e o ry on which th e
H ille g a s s c a le and th e Trabue supplem ents a re based i s l i k e l y to be f a l s e
when a p p lie d to specim ens o f E n g lish co m p o sitio n s—t h a t when we have com­
p le x , o v e rla p p in g s tim u li in v o lv e d , we can n o t ex p ect d if f e r e n c e s wliich
are e q u a lly o f te n n o tic e d to be e q u a l.
T hus, i t seems d o u b tfu l t h a t e i t h e r th e H ille g a s o r th e Trabue
s c a le c o n ta in s u n i t s ap p ro ach in g e q u a lity —as has been cla im e d .

In f a c t ,

such in e q u a li ty o f u n i t s seems to have been su p p o rted by Trabue*s o b se rv a­
tio n o f judgm ents c o n c ern in g two co m p o sitio n s which p r e v io u s ly had been
eq u a ted .

He s t a t e d t h a t "Such la r g e d if f e r e n c e s in th e form o f s u rfa c e s

o f d i s t r i b u t i o n s o f judgm ents were c e r t a i n l y u n e x p e c te d .

They te n d to

make one c r i t i c a l o f th e assum ption t h a t th e v a r i a b i l i t y o f ju d g n en t on
one com positio n i s e q u a l to th e v a r i a b i l i t y on any o th e r .
H eith er H ille g a s nor Trabue r e p o r te d any r e l i a b i l i t y s tu d ie s on th e
use o f t h e i r s c a le s by two o r more p e o p le on th e same s e t o f co m p o sitio n s.
D a rsie r e p o r te d a c o r r e l a t i o n o f .88 betw een two s e t s o f r a t i n g s from th e
use o f th e W illin g s c a l e A c c o r d i n g to Lyman, th e evidence i s about

9 . L . L . T h u rs to n e , "E q u a lly O ften N o ticed D iffe re n c e s ," J o u rn a l of
E d u c a tio n a l P sy ch o lo g y , v o l. 18 (May 1 9 2 7 ), p p . 292-93.
1 0 . T rab u e, 0£ . c i t . , p . $1.
1 1 . M. L . D a r s ie , "The R e l i a b i l i t y o f Judgments Based on th e W illin g
C om position S c a le ," Jo u rn a l o f E d u c a tio n a l R e se a rc h , v o l. 5 (Jan u ary
1 9 2 2 ), p p . 89-90.

1*8
e q u a lly d iv id e d a s t o w hether th e use o f com position s c a le s i s e f f e c t iv e
in re d u c in g th e d i s p a r i t y o f te a c h e r ’s m arks.

He does s t a t e , how ever,

t h a t e x te n siv e t r a i n i n g i n th e use o f su ch s c a le s i s e f f e c t iv e i n t h a t
d ire c tio n .

TO

S o rtin g m ethods*

In a s e r i e s o f s tu d ie s conducted in th e e a r ly 1930*s ,

Sims a ttem p ted to improve th e r e l i a b i l i t y o f theme r a t i n g s by h av in g th e
re a d e rs s o r t th e p ap e rs in to f i v e s ta c k s ac c o rd in g to c e r t a i n s ta n d a rd s .
U t i l i z i n g t h i s m ethod, he o b ta in e d r a t i n g s w ith a c o e f f i c i e n t o f c o n tin g ­
ency o f .7 7 , wliich he i n t e r p r e t e d a s an e s tim a tio n o f th e r e l i a b i l i t y o f
th e s e r a t i n g s .
Sims a ls o used th e same p ro ced u re f o r r a t i n g essay exam inations*
r a t i n g by s o r tin g th e d is c u s s io n q u e s tio n s , one a t a tim e , in to f iv e d i f ­
f e r e n t s ta c k s ac c o rd in g to c e r t a i n s ta n d a r d s .

A gain he o b ta in ed a r e ­

lia b ility of .7 7 .^
P ercen tag e b a s i s * Sims th e n t r i e d a n o th e r a p p ro ac h , which c o n s is te d o f
having th e re a d e rs mark th e p a p e rs on a p e rc e n ta g e b a s i s —w ith a maximum
sco re o f 100 as a b a s is f o r m a rk in g .

He th e n d eterm ined l e t t e r grades by

u sin g th e mean and s ta n d a rd d e v ia tio n f o r d ete rm in in g th e d iv is io n s betw een

12. Lyman, 0£ . c i t . , p . 196.
1 3 . V erner M. Sim s, ’’The O b je c tiv i ty , R e l i a b i l i t y , and V a lid ity o f an
Essay Exam ination Graded by R a tin g ,” J o u rn a l o f E d u c a tio n a l R e se a rc h ,
v o l . 2k (O ctober 1 9 3 1 ), PP . 216-23.
1U. V erner M. Sim s, "Reducing th e V a r i a b i l i t y o f Essay Exam ination Marks
Through E lim in a tin g V a ria tio n s i n S tan d ard o f G rad in g ," Jo u rn al o f
E d u c a tio n a l R e sea rc h , v o l. 26 (May 1 9 3 3 ), PP. 637-1*7.

1*9
g ra d e s .

He concluded t h a t t h i s p ro ced u re was e f f e c t iv e in red u cin g

v a r ia tio n s caused by d i f f e r e n t s ta n d a r d s , b u t d id n o t draw any c o n c lu sio n s
re g a rd in g th e r e l a t i v e m e rits o f th e two system s he had em ployed.1^
Use o f s ta n d a r d s !

S ta ln a k e r used s t i l l an o th e r ap p ro ach .

Over a p e rio d

o f fo u r y e a r s , a tte m p ts were made to re a c h agreem ent on s ta n d a r d s .
r a t e r co n tin u e d to mark p ap ers by a s s ig n in g l e t t e r g ra d e s .
fo u r y e a r s , th e r e l i a b i l i t y o f th e r a t i n g s was o n ly .5 5 .

Each

At th e end o f
He concluded

t h a t , in o rd er to improve th e r e l i a b i l i t y o f theme r a t i n g s , c e r t a i n ch a r­
a c t e r i s t i c s sh o u ld be a g ree d upon by th e te a c h in g p r o f e s s io n .

Then each

c h a r a c t e r i s t i c should be r a t e d s e p a r a te ly on th e b a s is o f f iv e p o in ts —
5 to 0 .

He su g g e ste d such c l i a r a c t e r i s t i c s a s o r g a n iz a tio n , c o h e re n c e ,

and m echanical p e r f e c t i o n s , w ith more to be added.
H ating c l i a r a c t e r i s t i c s :

S ta ln a k e r 's s u g g e stio n has been u t i l i z e d to a

c o n s id e ra b le e x te n t in r e c e n t y e a r s .

As e a r ly as 1917, S a c k e tt re p o rte d

th e use o f a sc o re card i n a stu d y com paring d i f f e r e n t methods o f r a t i n g
c o m p o sitio n s.

H is sc o re c a rd c o n s is te d o f th r e e m ajor c a te g o r ie s , each

having th r e e s u b - d iv is i o n s .

The t o t a l p o s s ib le p o in ts f o r each c a te g o ry

i s shown below*
C o rre c tn e s s —30
S p e llin g and c a p i t a l i z a t i o n —
P arag rap h in g —
P u n c tu a tio n —

15
8
7

15. V erner M. Sim s, "Im proving th e M easuring Q u a litie s o f an Essay Exami­
n a tio n ," J o u rn a l o f E d u c a tio n a l R e se a rc h , v o l. 27 (September 1933),
p p . 20-31”
16. JohnM . S ta ln a k e r , "Q u estio n IV, The E ssay ," E n g lish Jo u rn a l (C ollege
E d iti o n ) , v o l. 26 (F e b ru a ry 1 9 3 7 ), PP. 133-^0.

5o

S entence S tr u c tu r e —20
S yntax—
S im p lic ity —
D ic tio n —
T hought-C ontent— 50
O rig in a lity —
U n ity —
C oherence—

10

7
3
25
15
10

S a c k e tt e v a lu a te d fo u r methods o f g ra d in g co m p o sitio n s:

te a c h e r fs

judgm ent, th e H ille g a s s c a l e , th e B a llo u n a r r a tio n s c a l e , and th e sco re
c a rd .

He found th e s c o re ca rd to be somewhat s u p e rio r to th e o th e r th r e e

m ethods. 17'
A nderson and T r a x le r , u sin g a s im ila r r a t i n g m ethod, o b ta in e d r e l i ­
a b i l i t i e s ra n g in g from .56 to ,9h.
in t h e i r r a t i n g c a rd :

They used e ig h t s e p a ra te c a te g o r ie s

A ccuracy, 6; C o m p leten ess, 6; S p e llin g , 6; Punctu­

a t i o n , 6; Language e r r o r s , 6; Coherence betw een main d i v i s i o n s , 10;
16
O rg a n iz a tio n o f p a ra g ra p h s , 10; and O rg a n iz a tio n o f s e n te n c e s , 1 0 ,
D ie a e ric h re p o r te d a stu d y com paring th e r e l i a b i l i t y o f th e r a t i n g
method d e s c rib e d above (which he c a l l s th e a to m is tic method) w ith th e
r a t i n g o f themes by a s s ig n in g a s in g le sc o re to each p a p e r—based on a te n p o in t s c a le (wiiich he c a l l s th e w h o lis tic m eth o d ).

He found th e a to m is tic

method more r e l i a b l e , b u t more tim e-consum ing, th a n th e w h o lis tic m ethod.
He su g g e sts t h a t th e w h o lis tic approach m ight become a s r e l i a b l e were as
19
much tim e devoted to th e r a t i n g o f each p a p e r .

17. Leroy W. S a c k e tt, “Comparable M easures o f C o m p o sitio n ,"
S o c ie ty , v o l. 5 (P'ebruary 1 9 1 7 ), p p . 233-39.

School and

1 6 . H. A. A nderson, and A. E. T r a x le r , "The R e l i a b i l i t y o f th e Reading
o f an E n g lish E ssay T e s t," School Review , v o l. U8, 19li0, p p . 521-30.
1 9 . P a u l B . D ie d e ric h , "Readers* Methods S c r u tin iz e d i n E n g lish E ssay
S c o rin g ," E d u c a tio n a l T e s tin g S erv ice D evelopm ents, No. 1 (O ctober
1 9 5 1 ), P . 2

51

G erb er, a f t e r u s in g th e a to m is tic method for* r a t i n g b o th themes and
s p e e c h e s, recommended th e fo llo w in g c h a r a c t e r i s t i c s f o r th e r a t i n g o f
speeches and them est

p u rp o s e , c o n te n t, o r g a n iz a tio n , la n g u a g e , and
OA

p r e s e n ta tio n ( s p e c ia l problem s o f o r a l or w r i t t e n d e l i v e r y ) .
F or th e p a s t seven y e a r s , th e a to m is tic ap p ro ach has been used f o r
th e r a t i n g o f com prehensive ex am ination themes o f s tu d e n ts com pleting
t h e i r work i n th e W ritte n and Spoken E n g lish co u rse a t M ichigan S ta te
*

C o lle g e .

D uring tlia t tim e a r a t i n g fo rm , s im ila r to t h a t d e s c rib e d by

G erb er, has been used wliich c o n s is ts o f f iv e c a te g o r ie s .

Each c a te g o ry i s

r a te d on a te n - p o in t s c a l e , a s shown b elo w , (See A ppendix A f o r d e f i n i t i o n
o f each c a te g o r y ) :
TH£ME HATING SCALE
i*ame a n d /o r number______________________

Date_______________

S u p e rio r
U n s a tis fa c to ry
10 9
8
7
6 5 ^ 3 2 1 "
C onventions o f Grammar______________________________________________________
Sentence S tru c tu r e __________________________________________________________
D ic tio n ______________________________________________________________________
O rg a n iz a tio n ________________________________________________________________
C ontent______________________________________________________________________
H ater__________________

T o ta l_________________

In a r e l i a b i l i t y stu d y o f te n p a i r s o f r a t e r s u s in g th e form shown
above, th e p r e s e n t w r i t e r found r e l i a b i l i t i e s ra n g in g from .36 to ,9 b ‘, two
p a i r s o f r a t e r s had r e l i a b i l i t i e s above .9 0 .

T hus, i t ap p ears t h a t , i f

th e theme r a t e r s a r e p ro p e rly s e le c te d and t r a i n e d , th e sc o re ca rd o r
r a t i n g form p ro v id e s a method f o r r a t i n g themes wliich i s as r e l i a b l e as
any method y e t r e p o r te d .
20. John C. G erber, “T e s tin g and E v a lu a tio n i n th e S k i l l s o f Communication,11
C o lleg e E n g lis h , v o l. 9 (A p ril 1 9 ^ 8 ), p p . 375-8U.

52

I t sho u ld be n o te d , how ever, t h a t t h i s method i s s u b je c t to one o f
th e same c r i t i c i s m s made i n r e l a t i o n to th e H ille g a s and Trabue w ritin g
s c a le s .

T hat i s , th e r e i s no assu ra n c e t h a t t h i s method o f r a t i n g E n g lish

com positions p ro d u ces sc o re s c o n s is tin g o f eq u al u n i t s .

T hus, s t a t i s t i c a l

tre a tm e n t o f s c o re s o b ta in e d i n such a manner may be open to q u e s tio n .
A nother p o s s ib le w eakness o f th e r a t i n g method used by th e W ritte n
and Spoken E n g lish d ep artm en t was in d ic a te d in a stu d y by S t a r r i n g .

H is

stu d y concerned th e a c t u a l u se o f th e r a t i n g form by s t a f f members when
c e r t a i n elem en ts in th e themes had b een w eakened.

He had f iv e s e ts o f

themes r a t e d , and th e n weakened a d i f f e r e n t elem ent in each s e t o f them es.
S ta r r in g d isc o v e re d t h a t , when th e weakened themes were r a te d a g a in , th e
r a t e r s d id n o t make a d i s t i n c t i o n betw een weakness in se n te n c e s tr u c t u r e
and weakness i n co n v e n tio n s o f grammar; nor d id th e y make a d i s t i n c t i o n
between w eakness in c o n te n t and weakness in o r g a n iz a tio n .

I n o th e r w ords,

alth o u g h th e r a t i n g form i s a l i s t o f f iv e c a te g o r ie s to be re g a rd e d as
d is c r e te and o f eq u al w e ig h t, i t was n o t so re g a rd e d by th e r a t e r s .

In

a c tu a l p r a c t i c e , th e r a t e r s sco red th e themes on th r e e c a te g o r ie s :

l) a

com bination o f c o n v e n tio n s o f grammar and se n ten c e s t r u c t u r e , 2) d i c t i o n ,
and 3) a com b in atio n o f o r g a n iz a tio n and c o n te n t.

21

These f in d in g s a re v a l i d , p e rh a p s , o n ly f o r th e r a t e r s used in
S t a r r i n g 's s tu d y , o r f o r a s im ila r group o f r a t e r s .

I t should be noted

t h a t S ta r r in g had tw e n ty -fo u r members o f th e W ritte n and Spoken E n g lish
D epartm ent r a t e th e p ap ers used in h is s tu d y .

No a n a ly s is was made to

21. R obert W. S t a r r i n g , A Study o f R a tin g s o f Comprehensive Exam ination
Themes When C e rta in E lem ents Are Weakened ^U npublished Ed. D. T h e s is ) ,
M ichigan S ta te C o lle g e , A ugust 19*>2, pp.~10li-105.

53

d isc o v e r w hether any o f th e r a t e r s d e p a rte d from th e p a t te r n s noted f o r
th e group o f r a t e r s em ployed.
The f in d in g s o f Dolch and Leonard (re p o rte d e a r l i e r ) t h a t th e r e i s
l i t t l e c o r r e l a t i o n i n e x c e lle n c e betw een th e o rg a n iz a tio n and th e c o n te n t
of p a p e rs in d ic a t e t h a t such elem ents can be d is tin g u is h e d by r a t e r s in
r e l a t i o n to d if f e r e n c e s i n e x c e lle n c e .

T hus, i f a c c u r a te e v a lu a tio n s o f

s tu d e n t w r itin g a r e to be made in r e l a t i o n to th e s e p a r a te elem ents
d e s ig n a te d on a r a t i n g fo rm , i t would seem im p o rta n t to have th e r a t e r s
c a r e f u l ly s e le c te d and tr a in e d f o r t h a t p u rp o s e .
S in ce i t lias been d em onstrated t h a t th e r a t i n g method employed by th e
W ritte n and Spoken uinglish D epartm ent can produce h ig h ly r e l i a b l e r a tin g s
when p ro p e r p re c a u tio n s a re ta k e n , and sin c e th e r a t i n g form employed by
t h a t dep artm en t was d esig n ed f o r e v a lu a tin g th e o b je c tiv e s o f t h a t c o u rs e ,
th a t method and t h a t form seemed to p ro v id e th e b e s t a v a ila b le means f o r
e v a lu a tin g th e p ap e rs in v o lv ed i n t h i s s tu d y .
O b tain in g R e lia b le R atin g s
In o rd e r to in s u re th e a p p ro p ria te use o f th e r a t i n g method and th e
r a t i n g form s e l e c t e d , i t was co n sid ered im p o rtan t to determ in e th e number
and ty p e s o f r a t e r s needed f o r th a t p u rp o se , and to use th e most v a lid
method f o r d e te rm in in g th e sc o re f o r each p a p e r.
D eterm ining number o f r a t e r s :

Gerber 99 and D ie d e ric h 23^ b o th ag ree t h a t ,

22. G erber, 0£ . c i t . , p p . 375-8U.
23. D ie d e ric h , "The Measurement o f S k i l l in W riting,'* 0£ . c i t . , p p . 585-92.

$h

in o rd e r to o b ta in th e b e s t e v a lu a tio n o f a them e, two r a t e r s sh o u ld be
employed and t h a t th e av erag e o f t h e i r r a t i n g s should be u se d .

F u rth e r ­

m ore, when th e d i s p a r i t y betw een th e two r a t e r s i s e x c e s s iv e , a t h i r d
r a t e r should be em ployed.

I f th e t h i r d r a t e r a g re e s w ith one o f th e

f i r s t two r a t e r s , th e av e ra g e o f th o s e two r a t i n g s i s u sed .

I f th e r e i s

no c lo se agreem ent betw een any two o f th e tlire e r a t i n g s , th e av erag e o f
th e tlire e r a t i n g s sh o u ld be u se d .
T h is pro ced u re has been fo llo w ed f o r th e r a t i n g o f com prehensive
exam ination themes a t th e co m p letio n o f th e W ritte n and Spoken E n g lish
course a t M icliigan S ta te C o lleg e f o r th e p a s t seven y e a r s .

I t has been

th e p r a c t ic e a t t h i s i n s t i t u t i o n to use a t l i i r d r a t e r when th e d i s p a r i t y
between th e f i r s t two r a t i n g s was in ex cess o f f iv e p o in ts on th e t o t a l
sc o re —th e maximum o f wliich i s £0 p o in ts f o r each r a t i n g .
S e le c tin g r a t e r s t

D ie d e ric h p o in ts out t h a t to o b ta in s i g n i f i c a n t d i f ­

fe re n c e s , when e v a lu a tin g s tu d e n t improvement in w r itin g , o n ly th e most
s e n s itiv e and r e l i a b l e r a t e r s sh o u ld be u s e d .2^

Follow ing t h i s s u g g e s tio n ,

th e re c o rd s o f r a t i n g s by more th a n f o r t y s t a f f members o f th e D epartm ent
o f W ritte n and Spoken E n g lish were exam ined.

Nine r a t e r s were s e le c te d

who ap p eared to have d em o n strated a h ig h d eg ree o f s e n s itiv e n e s s and r e ­
lia b ility .

T h e ir av erag e r a t i n g s were ab o u t th e same and th e d i s t r i b u t i o n s

o f t h e i r r a tin g s were f a i r l y uniform —tli a t i s , th e r a tin g s of each s t a f f
member formed a d i s t r i b u t i o n ap p ro ach in g th e normal c u rv e .

2h. I b i d .

55

From th e s e n in e r a t e r s i t was p o s s ib le to o b ta in th r e e who were
w illin g and a b le t o d ev o te th e tim e n e c e s s a ry to com plete th e ta s k o f
r a tin g each o f th e 320 p a p e rs in v o lv e d d u rin g tlie p e rio d , betw een F a l l
te rm , 1951, and W in ter te rm , 1952.

T h u s, in s te a d o f h av in g t o c a l l in

a t h i r d r a t e r when th e r e was a d i s p a r i t y between th e f i r s t two r a t e r s ,
th r e e r a t e r s judged each o f th e 320 p a p e rs j a l l th r e e r a t e r s were r a t i n g
th e same p ap ers a t th e same tim e and under th e same e x te r n a l c o n d i tio n s .
Such a p ro ced u re sh o u ld p ro v id e f o r more v a lid and r e l i a b l e r a t i n g s th a n
would o th e rw ise be p o s s i b l e .
D eterm ining s c o r e s > A lthough n o t custom ary f o r th e r a t i n g o f th e m es, th e
use o f tlire e r a t e r s a t a tim e has been th e custom ary procedure f o r r a t i n g
th e com prehensive ex am in atio n speeches in th e W ritte n and Spoken E n g lish
course .

I t has been th e p r a c t ic e to u t i l i z e th e r a t i n g s o f

th e two o f

th e tlire e r a t e r s who were in c l o s e s t ag reem en t. Two ex c e p tio n s to t h i s
r u le have been p e r m itte d .

When no two of th e tlire e r a t e r s had a r r iv e d a t

r a tin g s w ith in f iv e p o in ts o f each o th e r , th e average o f a l l tlire e o f th e
r a tin g s were u s e d .

A ls o , when th e d if f e r e n c e s among th e r a tin g s were

eq u a l (su ch a s 2 2 , 2h, and 2 6 ) , th e av erag e of a l l tlire e r a tin g s was u se d .
I t lias been th e con sen su s o f th e hO to 50 s t a f f members in v o lv ed i n
th e above p ro ced u re t h a t such a p r a c tic e i s v a l i d .

Almost in v a r ia b ly th e

r a t e r who has v a rie d c o n s id e ra b ly from th e o th e r two r a t e r s

has reco g n ized

t h a t some d i s t r a c t i v e f a c t o r caused him to r a t e too high o r to o low .
S ince th e s e r e a c tio n s a r e by s t a f f members who have a h ig h se n se o f r e ­
s p o n s i b i l i t y t o th e s tu d e n t whose grade i s b e in g determ ined f o r th re e term s

of h is w ork, th e r e i s good re a s o n to b e lie v e t h a t th e d i s t r a c t i o n s
p o in te d o u t, in most c a s e s , were r e a l and t h a t th e r a t i n g s by th e o th e r
r a t e r s were v a l i d .
The p ro ced u re d e s c rib e d above has r e s u l t e d i n ag reem en t, w ith in f iv e
p o i n t s , by two o f th e tlire e r a t e r s f o r 9b p e r c e n t o f th e speeches r a t e d .
The re c o rd on th e r a t i n g o f themes has been ab o u t th e sam e.

T h is procedure

i s fu n d am e n tally th e same a s th e one u sed f o r th e r a t i n g o f com prehensive
exam ination th e m es,

and th e same a s t h a t recommended by D ie d e ric h .

Under

t h i s sy stem , how ever, th e t h i r d r a t e r does n o t have to be c a lle d i n when
a d isc re p a n c y o c c u rs .

H is r a t i n g has a lre a d y been made and i s a v a ila b le

f o r t a b u la t io n and use a lo n g w ith th e r a t i n g s o f th e o th e r two r a t e r s .
C o n tro l o f F a c to rs A ffe c tin g R a te rs
S f f e c t o f v a r ia tio n s in th e e f f ic ie n c y l e v e l o f r a t e r s .

Since t h i s stu d y

i s concerned w ith th e assum ption t h a t th e e f f ic ie n c y le v e l of th e w rite r
v a r ie s s i g n i f i c a n t l y from day to d ay , a sa fe g u a rd sh o u ld be p ro v id ed
a g a in s t th e p o s s i b i l i t y o f a v a r ia tio n in th e e f f ic ie n c y le v e l o f th e
r a t e r s from day to d ay .
To p ro v id e such a s a fe g u a rd , tlire e r a t e r s were employed to re a d
each o f th e 320 p ap e rs in v o lv e d ; and a l l tlire e were to r a t e th e same papers
a t ab o u t th e same tim e .

T hat i s , each p ap e r was r a te d by R a ter 1 , who

passed i t on to R a te r 2 , who, in t u r n , p assed i t on to R a te r 3 .

In a d d i­

t i o n , th e p ap ers were so o rg an iz ed t h a t a l l fo u r p a p e rs by each s tu d e n t
were r a te d d u rin g th e same r a t i n g p e r io d .

57

E f f e c t o f a c c id e n ta l c o n te n t f a c t o r s on r a t e r s .

S ince tli is stu d y i s

concerned w ith th e assum ption t h a t a c c id e n ta l c o n te n t f a c t o r s ( th e to p ic s )
p ro v id e a so u rc e o f v a r ia tio n in th e w r i tin g p r o f ic ie n c y of s tu d e n ts , th e
stu d y should p ro v id e a sa fe g u a rd a g a in s t th e e f f e c t o f a c c id e n ta l c o n te n t
f a c t o r s on th e q u a lity (th e r e l i a b i l i t y ) o f th e r a t e r s ' work.

I f , fo r

exam ple, a l l o f th e p ap e rs on one to p ic were r a te d on one day and a l l o f
th e p ap ers on a n o th e r to p ic were r a te d on a n o th e r d ay , th e re would be no
way o f d e term in in g w hether th e observed d if f e r e n c e s were due to v a r ia tio n s
in th e e f f ic ie n c y le v e l o f th e s tu d e n ts o r o f th e r a t e r s , o r due to th e
e f f e c t o f a c c id e n ta l c o n te n t f a c t o r s on th e r a t e r s —o r a com bination o f
a l l o f th e se f a c t o r s .
A p a r t i a l sa fe g u a rd f o r t l i i s problem was p ro v id ed by having a l l o f
th e p a p e rs oy each s tu d e n t r a te d on th e same day by a l l tlire e o f th e
r a t e r s — sin c e each s tu d e n t w rote on two d i f f e r e n t to p ic s .

H alf o f th e

s tu d e n ts , how ever, had a t l i i r a to p ic f o r t h e i r second p a p e r.

To p ro v id e

a sa fe g u a rd a g a in s t th e danger o f h aving th e e f f e c t o f a c c id e n ta l c o n te n t
f a c t o r s o f t l i i s to p ic combined w ith v a r ia tio n s i n e f f ic ie n c y le v e l ( o f th e
r a t e r s ) , th e p ap e rs were arra n g e d so t h a t th e to p ic s would be r a te d in
th e fo llo w in g o rd e r:

Topic 1 , Topic 2 , Topic 1 , Topic 3—w ith t l i i s s e r ie s

re p e a te d th ro u g h o u t th e r a t i n g o f th e 320 p a p e r s .

S ince th e e f f e c t of

th e c o n te n t f a c t o r s on th e r a t e r s has been i s o l a t e d , v a r ia tio n s betw een
th e r e l i a b i l i t y o f r a t i n g s on d i f f e r e n t to p ic s can be a t t r i b u t e d to th e
to p ic s .
A rran g in g th e to p ic s in t l i i s o rd e r a u to m a tic a lly in te r s p e r s e s two
s tu d e n ts ' p a p e rs , so t h a t i t would be more d i f f i c u l t f o r th e r a t e r s to
compare d i f f e r e n t p ap ers by th e same s tu d e n t.

A lthough th e r e i s no

58

evidence t h a t such a com parison would s e r io u s ly a f f e c t th e o b j e c t i v i t y
of th e r a t i n g s , th e p ap e rs were arran g ed so t h a t any two p ap ers by a
s in g le s tu d e n t were s e p a ra te d by a t l e a s t two p ap e rs by o th e r s t u d e n t s .
E ff e c t o f f a tig u e on th e r a t e r s .

Since th e f a tig u e f a c t o r was reco g n ized

as a p o s s ib le d i s t u r b in g in f lu e n c e on th e e f f ic ie n c y l e v e l a t which th e
s tu d e n ts would be w r itin g (a r e s t p e rio d b e in g p ro v id ed between th e w r i t­
in g o f th e tvro p a p e rs on each day to c o u n te ra c t such an in f lu e n c e ) , i t
seemed a d v is a b le to ta k e a s im ila r p r e c a u tio n in re g a rd to th e r a t e r s .
C o n seq u en tly , a s ix -h o u r w orking day was p la n n e d .

Each day was d iv id e d

in to ap p ro x im a tely fo u r eq u a l p a r t s o f n in e ty m inutes e a c h .

Two such

p e rio d s were h eld each m orning and two each a f te rn o o n , w ith a 1 5 - 2 0 m inute
r e s t p e rio d in th e m iddle o f th e morning and a fte rn o o n s e s s io n s .

T his

p ro ced u re was fo llo w ed f o r r a t i n g a l l o f th e 320 p ap e rs in v o lv ed i n th e
s tu d y .
In view o f th e arrangem ent o f th e p ap ers (d e s c rib e d above) , th e
e f f e c t o f any f a tig u e f a c t o r s on th e r a t e r s sh o u ld be e v e n ly d i s t r i b u t e d
among th e p a p e rs i n su ch a way as to p reclu d e any s e rio u s e f f e c t on th e
f a c to r s b e in g t e s t e d .
C r i t e r i a f o r R atin g Themes
I n view o f th e f a c t t h a t th e tlire e r a t e r s employed had been u sin g th e
same theme r a t i n g s c a le f o r th e p a s t f o u r y e a rs to e v a lu a te com prehensive
exam ination th em es, i t d id n o t seem a d v is a b le to make any a l t e r a t i o n s i n
t h a t s c a le —s in c e such a l t e r a t i o n s m ight r e s u l t in a lo w erin g o f th e r e ­
l i a b i l i t y o f th e r a t i n g s .

59
On th e o th e r h an d , D ie d e ric h p o in ts o u t t h a t s p e c i f i c c r i t e r i a f o r
ju d g in g p a p e rs r e l a t i v e to a s p e c if ic assig n m en t sh o u ld be s e t a f t e r re a d ­
in g a t l e a s t te n p a p e rs w r i tte n on t h a t a s s ig n m e n t.2^

I t i s im p o rtan t

f o r th e r a t e r s to re a c h an agreem ent on what i s ex p ected o f th e s tu d e n ts
in d ev e lo p in g a p ap e r on a p a r t i c u l a r a s sig n m e n t.

Such agreem ent concerns

th e c o n te n t, p r im a r il y , and th e o rg a n iz a tio n o f th e c o n te n t.

C rite ria fo r

th e se n te n c e s t r u c t u r e , grajnmar and m echanics, and th e d i c t i o n , which make
up th e rem ain d er o f th e g e n e ra l c r i t e r i a l i s t e d on th e r a t i n g s c a le i n u s e ,
rem ain f a i r l y c o n s ta n t f o r a l l assig n m en ts made i n th e W r itte n and Spoken
E n g lish c o u r s e .
A lthough a s p e c if ic assignm ent f o r w ritin g a p ap e r on a d e f i n i t e
to p ic may in f lu e n c e tlie q u a l ity o f th e sen ten ce s t r u c t u r e , grammar and
m ech an ics, and d i c t i o n , th e d e s ire d q u a l ity f o r th e s e elem ents o f w ritin g
rem ains f a i r l y c o n s ta n t.

In c o n t r a s t , th e q u a lity o f th e s u p p o rtin g ma­

t e r i a l , and th e o r g a n iz a tio n o f i t , have to be judged ac co rd in g to th e
t o p i c , th e m a te r ia ls p ro v id e d , an d , to some e x t e n t, th e a d d itio n a l know­
le d g e and id e a s p o sse sse d by th e s tu d e n ts .
The p ap e rs w hich, by random p ro c e d u re , had been e lim in a te d from th e
stu d y i n o rd e r to e q u a liz e th e number o f s tu d e n ts in each group were
u t i l i z e d to determ in e more s p e c i f i c a l l y th e c r i t e r i a f o r r a t i n g th e o rg a n i­
z a tio n and c o n te n t f o r each o f th e tlire e to p ic s in v o lv e d .

25. I b id .

60

Summary
Since i t lias been d em o n strated t l i a t th e r a t i n g method employed by
th e W ritte n and Spoken E n g lish D epartm ent can produce h ig h ly r e l i a b l e
r a tin g s when p ro p e r p re c a u tio n s a r e ta k e n , and s in c e th e r a t i n g form em­
ployed by t h a t d epartm ent was d esig n ed f o r e v a lu a tin g th e o b je c tiv e s o f
t h a t c o u r s e , t h a t method and t h a t form seemed to p ro v id e th e b e s t a v a i l ­
a b le means f o r e v a lu a tin g 'the p ap e rs in v o lv ed i n t h i s stu d y .
T hree r a t e r s , whose re c o rd s in d ic a te d a b i l i t y in making s e n s itiv e
d is c r im in a tio n s , were employed to r a t e each o f th e 320 p ap ers in v o lv e d
in t l i i s s tu d y .

The p ap ers on d i f f e r e n t to p ic s were arra n g e d so t h a t th e

r a t e r s re c e iv e d them i n a l t e r n a t e o rd e r—th u s in s u rin g t h a t th e e f f ic ie n c y
le v e l o f th e r a t e r s e n te re d i n t o th e r a t i n g o f each to p ic i n ab o u t th e
same way.

R est p e rio d s were p ro v id ed a t r e g u la r i n t e r v a l s in o rd e r to

reduce th e p o s s i b i l i t y o f e x c e s s iv e f a tig u e a f f e c t in g th e r a t e r 's w ork.
A fte r re a d in g p ap e rs which had b een e lim in a te d from th e s tu d y , th e
r a t e r s fo rm u late d t h e i r c r i t e r i a f o r e v a lu a tin g th e q u a lity o f w r itin g on
d i f f e r e n t to p ic s —e s p e c i a lly co n cern in g o rg a n iz a tio n and c o n te n t.
tlire e r a t i n g s f o r each p ap e r were ta b u la t e d .

The

The sco re f o r each p ap er

was determ ined by com puting th e av erag e o f th e two r a tin g s w hich were i n
c l o s e s t ag reem en t, when th e d isc re p a n c y betw een th o se two r a t i n g s was n o t
i n ex c ess o f f iv e p o i n t s .

O th erw ise , th e av erag e o f th e tlire e r a tin g s was

u se d .
T hus, th e d a ta was p ro v id ed f o r d e te rm in in g w hether a s in g le pap er
w r i tte n on a given to p ic a t a p a r t i c u l a r tim e p ro v id e s a v a lid b a s is f o r
e v a lu a tin g a s t u d e n t 's w r itin g a b i l i t y a t any tim e i n a w ritin g c o u rse .

CHAPTER V

ANALYSIS OF THE DATA

The purpose o f t l i i s stu d y was t o t e s t f o u r assum ptions in o rd e r to
determ ine w hether a s in g le p ap e r w r itte n by a s tu d e n t on a g iven to p ic
a t a p a r t i c u l a r tim e can be c o n sid e re d as a v a lid b a s is f o r e v a lu a tin g
h is achievem ent a t any tim e i n a w ritin g c o u r s e .
Since th e 80 s tu d e n ts in v o lv ed i n t h i s stu d y were o nly a sm a ll sample
(ab o u t tlire e p e r c e n t) o f th e e n t i r e s tu d e n t p o p u la tio n e n r o lle d i n th e
c o u r s e , and s in c e t l i i s stu d y was concerned w ith v a r ia tio n s o f in d iv id u a l
perform ance o f th e 80 s tu d e n ts who were e q u a lly d iv id e d among fo u r g ro u p s,
i t was n e c e ss a ry to p ro v id e answ ers to th e fo llo w in g q u e stio n s b e fo re
d e f i n i t i v e t e s t s co u ld be p ro v id ed f o r th e fo u r assum ptions*
1 . Were th e s tu d e n ts in v o lv e d in t h i s stu d y r e p r e s e n ta tiv e o f th e
e n t i r e Fresliman p o p u la tio n e n r o lle d i n th e W ritte n and Spoken E n g lish
course?
2 . Were th e s tu d e n ts i n each group r e p r e s e n ta tiv e o f th e e n t i r e
p o p u la tio n in v o lv e d i n th e stu d y ?
3 . Did f a tig u e a f f e c t th e q u a l ity o f w r itin g on th e p a p e rs w r itte n
d u rin g th e second hour o f th e tw o-hour w ritin g p e rio d s? And,
1*. Were th e r e any f a c t o r s wliich a f f e c te d th e average q u a lity o f w r i t­
ing f o r th e d i f f e r e n t groups o f s tu d e n ts —i n r e l a t i o n to t o p i c s , d a y s, or
p re s s u re ?

62

Background A n a ly sis
E ighty s tu d e n ts r e p r e s e n ta ti v e o f Fresliman p o p u la tio n .

F or th e f in d in g s

o f t h i s s tu d y to be g e n e ra lly a p p l ic a b le , i t m ust be assumed t h a t th e
s tu d e n ts in v o lv e d i n th e stu d y were r e p r e s e n ta ti v e o f th e e n t i r e Fresliman
p o p u la tio n e n r o lle d i n th e W ritte n and Spoken E n g lish c o u r s e .

A lthough

th e re was. no s a t i s f a c t o r y method o f t e s t i n g t h i s a ssu m p tio n , th e en ro llm e n t
p ro ced u re in u se o p e ra te d to in s u r e t h a t th e s e l e c t i o n o f any fo u r groups
o f s tu d e n ts would p ro v id e a r e p r e s e n ta tiv e c r o s s - s e c tio n o f th e Fresliman
p o p u la tio n .
Tliat p ro c e d u re c o n s is te d o f a d m ittin g s tu d e n ts in a lp h a b e tic a l o rd e r
tliro u g h o u t th e th r e e -d a y r e g i s t r a t i o n p e r io d .

In o rd e r to e q u a liz e tlie

number o f s tu d e n ts i n each o f th e 93 groups ( r e c i t a t i o n s e c tio n s ) in which
th e s tu d e n ts were b e in g e n r o lle d a t t h a t tim e , th e e n ro llm e n t i n any one
group was lim ite d on each d ay .

T hus, in each o f th e fo u r groups u se d ,

th e s tu d e n t names were d i s t r i b u t e d tliro u g h o u t th e a lp lia b e t— In group A,
from 3 to W; group B, from A to Sj group C , from A to S; and i n group D,
from B to W.

T h u s, i t seems re a s o n a b le to assume t h a t th e 80 s tu d e n ts were

r e p r e s e n ta ti v e o f th e e n t i r e Fresliman p o p u la tio n .
Each group r e p r e s e n ta ti v e o f t o t a l g ro u p .

These e ig h ty s tu d e n ts were

e q u a lly d iv id e d among f o u r groups o f tw en ty s tu d e n ts e a c h .

S ince t h i s

s tu d y was to t e s t assu m p tio n s c o n c ern in g th e e f f e c t o f t o p i c s , o f v a r ia tio n s
in e f f i c i e n c y , and p r e s s u r e on th e q u a lity o f s tu d e n t w r i tin g , and s in c e
th o s e assu m p tio n s co uld be te s te d o n ly by com paring th e r e s u l t s f o r d i f ­
f e r e n t g ro u p s, each group o f tw enty s tu d e n ts sh o u ld be a r e p r e s e n ta tiv e

63

sample ( in r e l a t i o n to t h e i r w r itin g a b i l i t y ) o f th e e n t i r e p o p u la tio n
o f s tu d e n ts in v o lv ed in th e s tu d y —or a method o f e q u a tin g th e fo u r
groups must be u t i l i z e d .
S ince a l l e ig h ty s tu d e n ts w rote one p ap er on th e same to p ic (''D ista n c e
Between C la sse s " ) under th e same c o n d itio n s , a b a s is was p ro v id ed f o r
d eterm in in g w hether o r n o t each group o f tw enty s tu d e n ts was r e p r e s e n ta tiv e
o f th e t o t a l g ro u p .

The method o f a n a ly s is o f variance"*" p ro v id es a t e s t

fo r th e h y p o th e sis t h a t th e d if f e r e n c e s in means f o r th e fo u r groups
in d ic a te r e a l d if f e r e n c e s i n th e av erag e q u a l ity o f w r itin g by th e s e
g ro u p s.
The mean sc o re s o f th e p ap ers w r i tte n on Topic 1 ("D ista n c e Between
C la sse s" ) on th e f i r s t day by each group a re shown below:
Group A

Group C

Group B

Group D

214.5

23. h

2 5 -u

22.ii

A p p lic a tio n of an a n a ly s is o f v a ria n c e produced an F r a t i o of ,7hS7
(as shown b elo w ).

T his does n o t approach th e F r a t i o o f 6.57 needed f o r

s ig n if ic a n c e a t th e £ p e r c e n t l e v e l .

T hus, we can r e j e c t th e h y p o th e sis

t h a t th e re a re s i g n i f i c a n t d if f e r e n c e s among th e fo u r groups in th e average
q u a lity o f w r itin g on Topic 1 .

T h e re fo re i t seems re a s o n a b le to conclude

th a t each o f th e fo u r groups i s r e p r e s e n ta tiv e i n t h e i r w r itin g a b i l i t y
of th e t o t a l group in v o lv e d .

1 . E . F . L in d q u is t, S t a t i s t i c a l A n a ly s is i n E d u c a tio n a l R e se a rc h . Houghton
M if f lin Company, New Y ork, I 9 I4.O, p . 91.

6k

R e s u lts o f A n a ly sis o f V ariance
Source

d.f.

SS

V ariance

F R atio

Between groups
of p ap ers
3
101.5367
33.81*5
17U57
W ithin groups
of papers_____________76________ 3 ^ 9 .3 1 0 0 ________ U5.385__________
F r a t i o needed f o r s ig n if ic a n c e a t ^ l e v e l .............................................8 .^ 7
F r a t i o needed f o r s ig n if ic a n c e a t 1% l e v e l ............................................ 26.27
E f f e c t o f th e f a tig u e f a c t o r .

A lthough p re c a u tio n s were tak en to p re v e n t

f a tig u e from p ro d u cin g v a r ia tio n s in e f f ic ie n c y f o r th e s tu d e n ts w r itin g
p ap ers d u rin g th e second h a l f o f th e w r itin g p e rio d each d ay , a t e s t was
p rovided to d e te rm in e , in s o f a r as p o s s i b le , w hether such f a tig u e d id a l t e r
th e q u a l ity of p ap e rs w r itte n d u rin g th e l a t t e r h a l f o f th o se p e rio d s .
Since a l t e r n a t e t o p i c s , f o r each h a l f o f th e tw o-hour w r itin g p e r io d s ,
were passed out s y s te m a tic a lly to ev ery o th e r s tu d e n t in each s e c t i o n , i t
is re a so n a b le to assume t h a t th e s tu d e n ts r e c e iv in g d i f f e r e n t assig n m en ts
f o r each h a l f o f th e w r itin g p e rio d were e v e n ly d iv id e d r e l a t i v e to t h e i r
w ritin g a b i l i t y .
T hus, w ith o nly two groups of s tu d e n ts in v o lv e d ( f i r s t - h o u r and
se c o n d -h o u r), a t - t e s t may be employed to t e s t th e h y p o th e sis tli a t th e
mean score f o r th e p ap e rs w r itte n on a s p e c if ic to p ic d u rin g th e f i r s t hour
i s s i g n i f i c a n t l y h ig h e r th a n th e mean f o r p ap ers w r itte n on t h a t to p ic
d u rin g th e second h o u r.

Since a l l s tu d e n ts in v o lv ed in th e stu d y w rote

two p ap ers on Topic 1 , th e t e s t may be a p p lie d to th e d if f e r e n c e in means
between a l l o f th e p ap ers on t h i s to p ic w r itte n d u rin g th e f i r s t hour and

2 . I b i d . , p . 51.

65

th o se w r itte n d u rin g th e second h o u r.

The co m p u tatio n , however, re v e a le d

th a t th e mean sc o re f o r th e p ap ers w r itte n d u rin g th e second hour was
s l i g h t l y h ig h e r th a n th e mean o f th e p ap ers w r itte n d u rin g th e f i r s t hour
(a s shown below) :
Means
Papers on T opic 1 , w r itte n d u rin g th e second h o u r.................... 23.925
P apers on Topic i , w r itte n d u rin g th e f i r s t h o u r ...................... 23.735
D if f e r e n c e ........................ 19
Since th e s iz e o f th e means i s j u s t th e o p p o site of what would be
a n tic ip a te d r e l a t i v e to th e f a tig u e f a c t o r s , i t seems re a s o n a b le to assume
th a t th e f a tig u e f a c t o r d id not r e s u l t in a lo w erin g o f th e q u a lity o f th e
w ritin g d u rin g th e second h o u r.

A p p lic a tio n o f th e t - t e s t in d ic a te d t h a t

the d if fe re n c e i n th e two means would have liappened by chance alone b e ­
tween UO and 50 tim es o u t o f a hundred c a s e s .
T h e re fo re , i t seems reaso n ab le to conclude t h a t n e ith e r th e f a tig u e
f a c t o r nor any o th e r f a c t o r r e l a t i v e to th e f i r s t o r second hour o f th e
w r itin g p e rio d liad any s i g n i f i c a n t e f f e c t on th e av erag e q u a lity o f th e
s tu d e n ts 1 w r i t i n g .
E f f e c t o f v a r ia tio n s in e f f ic ie n c y and o f c o n te n t f a c t o r s .

Under normal

c o n d itio n s , i t would be expected t h a t th e e f f ic ie n c y le v e l a t wliich a
group o f s tu d e n ts would be w ritin g a t a p a r t i c u l a r tim e would v ary from
one in d iv id u a l to a n o th e r .
would be a t t h e i r low p o i n t .

Some s tu d e n ts would be a t t h e i r p eak; some
The rem ainder would be working a t e f f ic ie n c y

le v e ls d is t r i b u t e d between th e two ex trem es.

On a n o th e r d ay , under normal

c o n d itio n s , th e same ty p e of d i s t r i b u t i o n would be ex p e cte d , b u t w ith many

66

in d iv id u a ls w orking a t d i f f e r e n t le v e ls o f e f f ic ie n c y th a n th e y were on
th e f i r s t d ay .

T h u s, th e av erag e q u a lity o f th e w r itin g on b o th days

should be a p p ro x im a tely th e same, and th e v a r ia tio n s i n e f f ic ie n c y le v e ls
would n o t r e s u l t in any s i g n i f i c a n t v a r ia tio n s i n th e mean sc o re s o f p ap ers
w r itte n on th e same to p ic s on d i f f e r e n t d a y s.
On th e o th e r h an d , i t may be h y p o th e s iz e d , t h a t i f th e " p r a c tic e
e f f e c t" o r "new le a r n in g e x p e rie n c e s" a f f e c te d most o f th e w r ite r s in th e
same m anner, th e n mean sc o re s on d i f f e r e n t days would be s i g n i f i c a n t l y
d iffe re n t.
L ik ew ise, i f th e a c c id e n ta l c o n te n t f a c t o r s , p ro v id ed by th e d i f f e r e n t
t o p i c s , a f f e c te d most s tu d e n ts in th e same manner ( t h a t i s , i f one to p ic
proved more d i f f i c u l t f o r most s tu d e n ts th a n d id th e o th e r to p ic ) , th e n
th a t e f f e c t would r e s u l t in a s i g n i f i c a n t d if f e r e n c e in th e mean sc o re s
f o r th e two t o p i c s .
In o rd e r to t e s t th e two h ypotheses d e s c rib e d above, a g a in a t e s t
f o r a s i g n i f i c a n t d if f e r e n c e i n means i s r e q u ir e d .

S ince each o f th e 20

stu d e n ts i n each group w rote a p ap er on each o f two d i f f e r e n t to p ic s on
each of two d i f f e r e n t d a y s, fo u r s e ts o f p a p e rs were produced by each
group:

one s e t w r i tte n on th e f i r s t to p ic on th e f i r s t d ay , one s e t on

th e second to p ic on th e f i r s t d a y , one s e t on th e f i r s t to p ic on th e second
day, and one s e t on th e second to p ic on th e second d ay .
Since groups A and B w rote t h e i r p ap ers on b o th days under th e same
c o n d itio n s , w ith no ex am in atio n p re s s u re added on th e second d ay , th e
two f a c t o r s , c o n te n t and e f f ic ie n c y le v e l d e s c rib e d above, a r e th e p rim ary
v a r ia b le s in v o lv e d .

67

A lthough th e f i r s t to p ic a s s ig n e d was th e same f o r b o th g ro u p s, th e
second to p ic a s s ig n e d group B s tu d e n ts was n o t

o n ly

d i f f e r e n t from th e

f i r s t to p ic b u t was o f a d i f f e r e n t ty p e th a n th e second to p ic a s s ig n e d
group A s tu d e n ts .

In o rd e r to d is t in g u is h betw een th e p o s s ib le e f f e c t s

o f two d i s s i m i l a r t o p i c s , a s compared w ith th e e f f e c t s of two s im ila r
t o p i c s , an a n a ly s is o f v a ria n c e was a p p lie d to th e fo u r s e ts o f p ap ers
fo r each o f th e two groups s e p a r a te ly —to determ in e w hether a s i g n i f i c a n t
d if fe re n c e e x is te d betw een any o f th e means o f th e fo u r s e t s o f p a p e r s .^
The means f o r group A a r e shown below :
F i r s t Day
Topic 1__________Topic 2
2 k .5

Second Day
Topic 1___________ T opic 2

23.7

21.9

21.3

For group A, th e F r a t i o o b ta in e d f o r 3 and 76 d eg rees o f freedom
was .7U80 (shown below) .

T h is does n o t approach th e F r a t i o o f 8.57*

which i s needed f o r s ig n if ic a n c e a t th e 5 p er c e n t l e v e l .

T h e r e f o re , i t

seems re a s o n a b le to conclude t h a t th e r e i s no s i g n i f i c a n t d if f e r e n c e b e ­
tween th e means o f any two s e t s o f p a p e rs w r i tte n by group A .
R e s u lts o f A n a ly sis o f V ariance
Source

d.f.

SS

V ariance

F .r a tio

Between s e t s
o f p a p e rs
3
136.91+61+
U5.61+88
.71+80
W ithin s e ts
o f p ap e rs__________76________ 1+637 .8100
________ 61.0238___________
F r a t i o needed f o r s ig n if ic a n c e a t 5% l e v e l ........................................8.57
F r a t i o needed f o r s ig n if ic a n c e a t 1% l e v e l ......................................26.27

3 . E . F . L in d q u is t, l o c . c i t .

68

The means f o r group B a re shown below*
F i r s t Day
Topic 1
Topic 3
25

.h

Second Day
T o p ic~ I
T opic 3

21.6

2 1 .5

2 6 .1

For group B, th e F r a t i o o b ta in e d f o r 3 and 76 d e g re ss of freedom
was .9837 (shown below) .

A gain t h i s does n o t approach th e F r a t i o of

8 .5 7 , which i s needed f o r s ig n if ic a n c e a t th e 5 p e r c e n t l e v e l .

T h e re fo re ,

i t seems re a s o n a b le to conclude t h a t th e re i s no s i g n i f i c a n t d if fe re n c e
between th e means o f any two s e ts o f p ap ers w r i tte n by group B
R e s u lts o f A n a ly sis o f V ariance
Source

d .f .

SS

Between s e t s
of p ap ers
221 .ii073
3
W ithin s e ts
of p ap ers
76
5959.2900
F r a t i o needed f o r s ig n if ic a n c e a t 5$ le v e l
F r a t i o needed f o r s ig n if ic a n c e a t 1% le v e l

V ariance

F ra tio

77.1350

.9837

78.6117

In view o f th e f a c t t h a t no s i g n i f i c a n t d if fe re n c e was found between
any p a i r o f means f o r e i t h e r s e c t i o n , i t seems re a s o n a b le to conclude t h a t :
1) i f th e r e were v a r ia tio n s i n th e e f f ic ie n c y le v e l o f in d iv id u a l w r ite r s
from day to day w hich r e s u lt e d in s i g n i f i c a n t v a r ia tio n s in th e q u a lity
o f t h e i r w r i tin g , th o s e v a r ia tio n s o ccu rred i n such a manner as to ca n ce l
out any s i g n i f i c a n t e f f e c t on th e mean sc o re s f o r groups o f w r i t e r s ; and
2) i f th e a c c id e n ta l c o n te n t f a c t o r s (due to d i f f e r e n t to p ic s ) r e s u lt e d
in s i g n i f i c a n t v a r ia tio n s in th e q u a lity o f w r itin g f o r in d iv id u a l s tu d e n ts
a g ain such v a r ia tio n s o ccu rred i n such a manner as to c a n c e l o u t any
s i g n i f i c a n t e f f e c t on th e mean sc o re s f o r groups o f w r i t e r s .

69

E f f e c t o f p r e s s u re on mean s c o r e s .

The s tu d e n ts in group C w rote on

th e same two to p ic s ( s im il a r ) on each day a s d id th o se in group A .

On

th e second d ay , how ever, th e s tu d e n ts in group C had an a d d itio n a l psy­
c h o lo g ic a l p r e s s u r e f a c t o r ad d ed , by b ein g inform ed t h a t th e p ap ers
w r itte n on t h a t day would be c o n sid e re d as a p a r t o f t h e i r f i n a l e:xamin a tio n .

The s tu d e n ts i n group D w rote on th e same two to p ic s ( d is s im ila r )

on each day a s d id th o se i n group B.

On th e second d ay , however, th e

s tu d e n ts in group D w rote u n d er th e same a d d itio n a l p sy c h o lo g ic a l p re s s u re
as d id th o se i n group C .
A g ain , as w ith groups A and B , one o f th e prim ary v a r ia b le s in v o lv ed
in th e w r itin g of th e f o u r s e ts o f p a p e rs by each o f th e s e two s e c tio n s
i s th e c o n te n t f a c t o r (due to s im ila r to p ic s f o r group C, and to d is s im i­
l a r to p ic s f o r group D) .

S ince i t was d em o n strate d , by th e p re c e d in g

t e s t s , t h a t th e v a r ia tio n s i n e f f ic ie n c y l e v e l from day to day d id n o t
a f f e c t th e mean s c o re s on p a p e rs w r itte n by a group o f s tu d e n ts , th e
a d d itio n a l p sy c h o lo g ic a l p r e s s u r e f a c t o r o f th e ex am in atio n s i t u a t i o n may
be co n sid ered as th e second p rim ary v a r ia b le in v o lv ed i n th e w r itin g by
groups C and D.
A gain, sin c e s im ila r to p ic s were u t i l i z e d by group C and d is s im ila r
to p ic s by group D, i t was deemed a d v is a b le to t e s t th e s ig n if ic a n c e o f
th e d if fe re n c e betw een means f o r each o f th e two groups s e p a r a te ly i n
o rd e r to p ro v id e f o r th e p o s s i b i l i t y o f an i n t e r a c t i o n between th e p r e s ­
su re and s im ila r to p ic s and betw een p re s s u re and d is s im ila r to p ic s having
a s i g n i f i c a n t e f f e c t on th e r e s u l t s .

70

T hus, an a n a ly s is o f v a ria n c e a g a in was used to d eterm in e w hether
th e re were any s i g n i f i c a n t d if f e r e n c e s betw een th e mean sc o re s o f each
of th e fo u r s e ts o f p ap ers w r i t t e n by each group.

The mean sc o re s f o r

group C a re shown below:
F i r s t Day

Second Day

Topic 1

T opic 2

Topic 1

23.k

2 3 .9

2 5 .0

Topic 2
2h.6

For group C, th e F r a t i o o b ta in e d f o r 3 and 76 degrees o f freedom
was .2700 (a s shown b e lo w ), which does n o t approach th e F r a t i o o f 8.57
needed f o r s ig n if ic a n c e a t th e 5 p er c e n t l e v e l .

T h e re fo re , i t seems

re a so n a b le to conclude t h a t th e re was no s i g n i f i c a n t d if fe re n c e between
th e means o f any two s e ts o f p ap ers w r i tte n by group C .
R e s u lts o f A n a ly s is o f V ariance f o r
_______________ GrcnaqaC ______________
Source

d .f .

SS

V ariance

F ra tio

Between s e ts
of p ap ers
3
3^.8690
11.623
.2700
W ith in s e ts
o f papers_________ 76__________ 3271.3500 _______ 1+3 .QUO_________
F r a t i o needed f o r s ig n if ic a n c e a t 5$ l e v e l ..........................
6.57
F r a t i o needed f o r s ig n if ic a n c e a t 1% l e v e l .......................... 26.27
The mean sc o re s f o r group D a re shown below :
F i r s t Day
Topic 1
Topic 3
22.U

1 9 .2

Topic 1
23.9

Second Day
Topic 3
20.9

For group D, th e F r a t i o o b ta in e d f o r 76 and 3 deg rees o f freedom
was l.It.8 (a s shown below) .

T his does n o t approach th e F r a t i o o f 2.73

71

n eed ed f o r s i g n i f i c a n c e a t t h e 5 p e r c e n t l e v e l .

T h e r e f o r e , i t seem s

r e a s o n a b le t o c o n c lu d e t h a t t h e r e i s no s i g n i f i c a n t d i f f e r e n c e b e tw ee n
th e mean s c o r e s o f an y two s e t s o f p a p e r s w r i t t e n b y g ro u p D.
R e s u l t s o f A n a ly s is o f V a ria n c e

fo r

______________ G roup-D _______________
S o u rce

d .f .

SS

V a r ia n c e

F ra tio

2hS.h900

81.63

1.U8

B etw een g ro u p s

of papers

3

W ith in g ro u p s
3'3.25____________________
o f p a p e r s ________ 76____________ 1+199.1650
F r a t i o n eed ed f o r s i g n i f i c a n c e a t
l e v e l .................
2 .7 3
F r a t i o needed f o r s ig n if ic a n c e a t 1% l e v e l ..............................
1+.06
S in c e t h e a p p l i c a t i o n o f a n a l y s i s o f v a r i a n c e f a i l e d t o show any
s i g n i f i c a n t d i f f e r e n c e i n t h e m eans o f t h e f o u r s e t s

o f p a p ers w ritte n

by e i t h e r g ro u p C o r g ro u p D, i t seem s r e a s o n a b le t o

c o n c lu d e t h a t n e i t h e r

th e a d d i t i o n a l p s y c h o l o g ic a l p r e s s u r e o f t h e e x a m in a tio n s i t u a t i o n n o r ,
a g a i n , th e c o n t e n t f a c t o r s (fro m th e u s e o f s i m i l a r an d d i s s i m i l a r t o p i c s ) ,
n o r a c o m b in a tio n o f su c h f a c t o r s had any s i g n i f i c a n t e f f e c t on th e a v e r ­
age q u a l i t y o f w r i t i n g by t h e s e s t u d e n t s .
T h e re w a s, h o w e v e r, one o t h e r c o m b in a tio n o f s t u d e n t s t h a t c o u ld be
u se d t o t e s t t h e e f f e c t o f t h e a d d i t i o n a l p s y c h o l o g ic a l p r e s s u r e on th e
a v e ra g e q u a l i t y o f w r i t i n g oy a group o f s t u d e n t s .

The s t u d e n t s o f a l l

f o u r g ro u p s w ro te on T o p ic 1 ( " D is ta n c e B etw een C l a s s e s " ) on two d i f f e r e n t
d a y s , w ith t h e s t u d e n t s i n g ro u p s A an d B w r i t i n g w ith o u t a n y a d d i t i o n a l
p s y c h o l o g ic a l p r e s s u r e on e i t h e r d a y .

The s t u d e n t s i n g ro u p s C and D ,

h o w ev er, w ro te on T o p ic 1 on th e f i r s t day w i t h o u t a d d i t i o n a l p r e s s u r e ,
o u t on th e s e c o n d d a y t h e p s y c h o lo g ic a l p r e s s u r e o f t h e e x a m in a tio n s i t u ­
a t i o n was a d d e d .

72

T hus, by com bining th e sc o re s on Topic 1 f o r groups A and B on each
day and f o r groups C and D on each d ay , a g a in fo u r s e ts o f p a p e rs a re
p ro v id e d , th e means o f wliich can be t e s t e d f o r s i g n i f i c a n t d if f e r e n c e s .
At th e b e g in n in g of t h i s c h a p te r , i t was d em onstrated t h a t th e re was
no s i g n i f i c a n t d if fe re n c e in th e mean s c o r e s , among th e fo u r g ro u p s, fo r
the p ap e rs w r i tte n on Topic 1 on th e f i r s t day.

A ls o , i t was d em o nstrated

th a t th e r e was no s i g n i f i c a n t d if fe re n c e betw een any two mean sc o re s of
the fo u r s e ts o f p ap ers w r itte n by each group.

By com bining th e sc o re s

on Topic 1 , f o r each day, by groups A and B and by groups C and D , th e
number o f s tu d e n t p ap ers in each group i s douole th e number i n p re v io u s
groups t e s t e d —which should p ro v id e a more r e l i a b l e t e s t f o r th e e f f e c t
of th e f a c t o r s in v o lv e d .

The means f o r th e combined s e c tio n s on Topic 1

a re :
F i r s t Day
Groups A and B
Groups C and D
2h.9

22.9

Second Day
Groups A and 3
Groups C and D
22.5

2U.5

A gain, an a n a ly s is o f v a ria n c e was used to determ ine w hether th e re
was any s i g n i f i c a n t d if f e r e n c e between any two mean sc o re s f o r th e s e fo u r
s e ts o f p a p e rs w r itte n on Topic 1 .

The F r a t i o f o r 3 and 156 deg rees of

freedom was .7957 (a s shown below) .

T his does n o t approach th e F r a t i o

of 6.55 needed f o r s ig n if ic a n c e a t th e 5 p e r c e n t l e v e l .

T h e re fo re , i t

seems re a s o n a b le to conclude th a t th e r e i s no s i g n i f i c a n t d if f e r e n c e b e ­
tween any two o f th e f o u r means—a g a in d em o n stratin g t h a t th e p s y c h o lo g ic a l
p re s s u re o f th e ex am in atio n s i t u a t i o n had no e f f e c t on th e average q u a lity
o f w r itin g by th e groups o f s tu d e n ts in v o lv e d .

A lso ( i n r e l a t i o n to

73

groups A and B com bined), i t has been d em o n strated ag a in t h a t v a r ia tio n s
in e f f ic ie n c y le v e l from day to day d id n o t have a s i g n i f i c a n t e f f e c t on
th e average q u a lity o f w r itin g by th e s e s t u d e n t s .
R e s u lt o f A n a ly s is o f V arian ce f o r
_________ Combined S e c tio n s________
Source

d.f .

SS

V ariance

F ra tio

Between s e ts
of papers
3
130.6815
U3.56
.7957
W ithin s e ts
5U.7U___________
______
of papers_______ 156________ 6539.U960
F r a t i o needed f o r s ig n if ic a n c e a t 5% l e v e l . . . .....................
S'.5>5
F r a t i o needed f o r s ig n if ic a n c e a t 1/b l e v e l ........................... 26.25
E ff e c t o f F a c to rs on In d iv id u a ls
The t e s t s th u s f a r re p o rte d do n o t p re c lu d e th e p o s s i b i l i t y of
s i g n i f i c a n t v a r ia tio n s in w ritin g p r o f ic ie n c y f o r any in d iv id u a l or f o r
any numoer o f in d iv id u a ls in v o lv ed in t h i s s tu d y .

A ll t h a t has been

dem onstrated i s t h a t i f such v a r ia tio n s have o c c u rre d , th e y have o ccurred
in such a manner as to ca n ce l o u t any e f f e c t on th e t o t a l o r mean sc o re s
in each in s ta n c e .
Method o f d e term in in g s i g n i f i c a n t v a r ia tio n s in th e q u a lity o f w ritin g
by in d iv i d u a ls .

In o rd e r to stu d y th e e f f e c t o f th e v a rio u s f a c to r s on

the q u a lity o f th e w r itin g by in d iv id u a l s tu d e n ts , i t was n ec essary to
examine th e sc o re s on p a i r s o f p a p e rs by th e same s tu d e n t and to d e te r ­
mine w hether th e d if f e r e n c e s betw een th o se s c o re s may have been due en­
t i r e l y to th e u n r e l i a b i l i t y o f th e r a t i n g s o r w hether th o se d if fe re n c e s
a c tu a lly in d ic a te v a r ia tio n s in th e q u a lity o f th e stu d e n t* s w r itin g .

7U

A lthough th e method ad o p ted f o r r a t i n g th e p a p e rs in t h i s stu d y i s
co n sid ered by th e e x p e rts in th e f i e l d a s one o f th e most r e l i a b l e and
v a lid methods f o r e v a lu a tin g th e q u a lity o f a s t u d e n t 's w r itin g , th e
sc o re s o b ta in ed by t h a t method a r e n o t amenable to th e u su a l s t a t i s t i c a l
methods f o r d ete rm in in g th e r e l i a b i l i t y and th e e r r o r o f th e r a t i n g s .
In most c a s e s , th e sc o re a s sig n e d each p a p e r was th e average of o n ly two
ra tin g s .

Those two r a t i n g s , how ever, were n o t alw ays by th e same two

p e o p le .

And in some c a s e s , th r e e r a t i n g s were av erag ed in o rd e r to ob­

t a i n th e b e s t e s tim a te o f th e q u a l ity o f a p a p e r .
E b e l, w ith th e a s s is ta n c e o f P ro fe s s o rs E. E. C u re to n , H arold
G u llik s e n , and E. F . L in d q u is t, lias developed a p ro ced u re f o r d eterm in in g
the r e l i a b i l i t y and e r r o r f o r s e t s o f r a t i n g s — s im ila r to th o se used in
t h i s stu d y (See A ppendix A ) .

E bel p o in ts o u t t h a t i f d e c is io n s a re to be

made in p r a c tic e by com paring av e ra g es w hich come from d i f f e r e n t groups
of r a t e r s , th e n th e " b e tw e e n -ra te rs" v a ria n c e should be in c lu d ed as a
p a r t of th e e r r o r te rm .^

(These re q u ire m e n ts a re met by E b e l's p ro c e d u re ,

which i s i l l u s t r a t e d in Appendix B .)
Since th e r e was a p o s s i b i l i t y t h a t a c c id e n ta l c o n te n t f a c t o r s due
to th e to p ic s m ight in flu e n c e th e r a t e r s , as w e ll as th e w r i t e r s , i t
seemed l i k e l y t h a t th e r e l i a b i l i t y o f th e r a t i n g s f o r each to p ic in d ic a te d
t h a t such v a r ia tio n s d id o ccur (a s shown b e lo w ).

T h is means t h a t th e

s ta n d a rd e r r o r o f an observed sco re a ls o would v a ry from to p ic to t o p i c .
S ince th e r e l i a b i l i t y o f th e r a t i n g s v a rie d from to p ic to t o p i c , th e
o v e r - a l l , o r t o t a l , r e l i a b i l i t y and s ta n d a rd e r r o r o f th e r a tin g s were
U. R obert L . E b e l, " E stim a tio n o f th e R e l i a b i l i t y o f R a tin g s ,"
P sy ch o m etrik a, v o l. 1 6 , No. h (December 1 9 5 l) , PP. U07-u2l;.

75

computed i n o rd e r to make more a c c u ra te com parisons o f a s tu d e n t's
o b ta in e d sc o re s on d i f f e r e n t t o p i c s .

The r e l i a b i l i t y and th e s ta n d a rd

e r r o r o f th e r a t i n g s f o r each to p ic and f o r th e t o t a l a re shown below
as computed by E b e l's p ro c e d u re :
S tan d ard E rro r

R e lia b ility

Topic 1
Topic 2
Topic 3

3.2031
3.^583
2.511L6

.80
.77
.91

T o ta l

3.1203

.83

In o rd e r to compare two o b ta in e d sc o re s f o r th e same s t u d e n t, i t
was n e c e ssa ry to compute th e sta n d a rd e r r o r of th e d if f e r e n c e between
th o se two o b ta in e d s c o re s —w liich, ac c o rd in g to McNemar, i s found by
m u ltip ly in g th e s ta n d a rd e r r o r o f th e o b ta in ed sc o re by th e sq u a re ro o t
of tw o.

F u rth e rm o re , t h i s p ro d u ct m u ltip lie d by 1 .9 6 p ro v id es th e

d if fe re n c e ( a t th e 5 p e r c e n t le v e l o f co n fid en ce) between two o b ta in e d
sc o re s wliich i s more th a n i s to be ex p ected on th e b a s is o f ch an ce, i . e . ,
on th e b a s is o f th e u n r e l i a b i l i t y o f th e s c o r e s .

5

In o rd e r to o b ta in as a c c u ra te an e s tim a te as p o s s ib le o f s i g n i f i c a n t
d if f e r e n c e s between th e sc o re s o f a s tu d e n t's p ap ers on two d i f f e r e n t
t o p ic s , th e s ta n d a rd e r r o r o f d if f e r e n c e between two o b ta in e d sc o re s was
computed f o r th e t o t a l number o f r a tin g s in v o lv e d in t h i s s tu d y .

And

a g a in , in o rd e r to o b ta in as a c c u ra te an e s tim a te as p o s s i b l e , th e s ta n d a rd
e r r o r o f th e d if f e r e n c e between two o b ta in ed sc o re s on th e same to p ic was
computed f o r each to p ic s e p a r a te ly —sin c e th e re were some d if f e r e n c e s

5 . Quinn McNemar, P s y c h o lo g ic a l S t a t i s t i c s , John W iley and Sons, New York,
19h9, p . 130.

76

b e tw e e n th e r e l i a b i l i t i e s o f th e r a t i n g s f o r th e p a p e r s on d i f f e r e n t
to p ic s .
E ach r a t e r r e c o r d e d a t o t a l s c o r e f o r e a c h p a p e r i n w hole u n i t s ,
The o b t a i n e d s c o r e s f o r a l l b u t l i t o f th e 320 p a p e r s w ere com puted b y
a v e r a g in g t h e s c o r e s a s s i g n e d b y two r a t e r s ; t h u s t h e o b ta in e d s c o r e s
f o r m o st o f th e p a p e r s w ere e i t h e r i n w hole o r h a l f u n i t s .

T h e re -was

a d i s c r e p a n c y o f m ore th a n f i v e p o i n t s b e tw e e n t h e r a t i n g s f o r ll* o f th e
320 p a p e r s .

F o r th o s e p a p e rs t h e o b ta in e d s c o r e s w ere com puted b y a v e r a g ­

in g th e r a t i n g s re c o rd e d by a l l th r e e r a t e r s .

F o r some o f th o s e p a p e r s

th e o b ta i n e d s c o r e s w ere i n o n e - t h i r d o r t w o - t h i r d s u n i t s .
S in c e t h e o b ta in e d s c o r e s f o r m o st o f th e p a p e r s w ere i n w hole o r
h a l f u n i t s , a n d s i n c e t h e t h e o r e t i c a l d i f f e r e n c e s , n e c e s s a r y t o d e n o te
a s i g n i f i c a n t d i f f e r e n c e i n t h e q u a l i t y o f w r i t i n g , w ere i n odd d e c im a ls ,
t h e minimum d i f f e r e n c e s a c t u a l l y u s e d t o i n d i c a t e a s i g n i f i c a n t d i f f e r e n c e
w ere som ewhat g r e a t e r t h a n t h e t h e o r e t i c a l d i f f e r e n c e s i n e a c h c a s e , a s
i n d i c a t e d b e lo w f o r e a c h t o p i c an d f o r t h e t o t a l .
T h e o r e tic a l D iffe re n c e

Minimum D i f f e r e n c e U sed

T o p ic 1
T o p ic 2
T o p ic 3

6 .8 8
9 .9 8

9 .0 0
1 0 .0 0

7.0U

7.90

T o ta l

8.69

9 .0 0

T h u s , when a d i f f e r e n c e b e tw ee n a s t u d e n t* s s c o r e s f o r two p a p e r s on
T o p ic 1 was fo u n d t o be 9 .0 0 p o i n t s o r m o re , i t seem ed r e a s o n a b l e t o c o n ­
c lu d e t h a t a r e a l d i f f e r e n c e i n t h e q u a l i t y o f h i s w r i t i n g was i n d i c a t e d .
L ik e w is e , s u c h a c o n c lu s io n seem ed r e a s o n a b le when a d i f f e r e n c e i n s c o r e s
o f 1 0 .0 0 p o i n t s o r m ore was fo u n d b e tw e e n two p a p e r s on T o p ic 2 , and o f

77

7.50 p o in ts o r more betw een two p a p e rs on T opic 3 , and o f 9 .0 0 p o in ts or
more betw een two p a p e rs on d i f f e r e n t t o p i c s .
Since each s tu d e n t w rote fo u r p a p e rs (two on d i f f e r e n t to p ic s on
each o f two d i f f e r e n t d a y s ) , th e re were fo u r p o s s i b i l i t i e s f o r d if fe re n c e s
i n th e q u a lity o f a s tu d e n t’ s w r itin g to be n o te d —d if fe re n c e s between th e
papers on:
1.
2.
3.
It.

th e
th e
th e
th e

f i r s t to p ic on d i f f e r e n t d a y s ,
second to p ic on d i f f e r e n t d a y s ,
f i r s t and second to p ic s on th e f i r s t d ay , and
f i r s t and second to p ic s on th e second d ay .

With 80 s tu d e n ts in v o lv ed i n t h i s

s tu d y , th e re were 320 p a ir s o f p a p e rs—

each o f which p ro v id ed an o p p o rtu n ity to n o te a s i g n i f i c a n t d i f f e r e n c e ,
or v a r i a t i o n , in th e q u a l ity o f w r i tin g .
The d if f e r e n c e between th e sc o re s f o r each o f th e fo u r p a ir in g s of
p ap ers by each s tu d e n t was computed and ta b u la te d .

Then th e d if fe re n c e s

wliich equaled o r exceeded th e a p p r o p r ia te minimum d if f e r e n c e , i n each
c a s e , were checked (See A ppendix C )—th u s p ro v id in g a b a s is f o r t e s t i n g
th e f i r s t th r e e assum ptions posed i n t h i s s tu d y .

To t e s t th e f o u r th

assu m p tio n , i t was n e c e s s a ry to ra n k th e 80 s tu d e n ts a c co rd in g to t h e i r
w ritin g a b i l i t y .
Method o f ra n k in g s tu d e n ts ac c o rd in g to w r itin g a b i l i t y .

The 80 s tu d e n ts

were ranked from h ig h to low—u s in g th e h ig h e s t sco re o b ta in e d by each
s tu d e n t o u t o f h is s e t o f fo u r p a p e rs a s a b a s is f o r th e ra n k in g .

The

p re s e n t w r i t e r re c o g n iz e s t li a t to use th e av erag e of th e fo u r sc o re s a s
an in d e x f o r ra n k in g th e s tu d e n ts would r e s u l t i n g r e a te r s t a b i l i t y ,
te n d in g to red u ce th e e f f e c t o f th e e r r o r in r a t i n g s .

78

On th e o th e r h an d , i f th e s c o re s on a l l fo u r pap ers were averaged
in o rd er to o b ta in an in d e x f o r ra n k in g s t u d e n t s , a number o f stu d e n ts
m ight f a l l a t th e same r a n k , even though t h e i r b e s t p ap ers were q u ite
d i f f e r e n t i n q u a l ity .

S in ce th e s c o re on each p ap e r was an average o f

th e r a tin g s re p o rte d by two o r th r e e r a t e r s , th e p ro c e s s o f av e ra g in g
th e sc o re s on th e fo u r p a p e rs by each s tu d e n t, i n o rd e r to o b ta in an
ind ex o f h is r a n k , would seem to r e s t r i c t th e u s e fu ln e s s o f such ra n k in g
f o r t h i s s tu d y .

In f a c t , when a l l f o u r p ap ers by each s tu d e n t were

a v e ra g ed , a s many a s e ig h t s tu d e n ts f e l l a t th e same ran k a t two d i f f e r e n t
p o in ts .
The method used f o r ra n k in g th e sb u d e n ts should be c o n s is te n t w ith
th e b a s ic assum ptions of t h i s s tu d y .

As p re v io u s ly in d ic a te d , th e p r a c tic e

of u t i l i z i n g a s in g le theme a s a r e p r e s e n ta tiv e sam ple o f a s tu d e n t's
w r itin g a b i l i t y f o r th e p u rp o se o f e v a lu a tin g s tu d e n t acliievem ent assumes
tlia t th e re a re no s i g n i f i c a n t v a r ia tio n s i n th e q u a lity o f a s tu d e n t's
w r itin g from day to day o r from to p ic to t o p i c .

I t a ls o assumes t h a t

such s t a b i l i t y of w r itin g perform ance i s tr u e f o r a l l s tu d e n ts , r e g a r d le s s
o f v a r ia tio n s i n a b i l i t y .
I f such assum ptions were v a l i d , i t would make l i t t l e d if fe re n c e
w hether s tu d e n ts were ranked a c c o rd in g to th e h ig h e s t sc o re o b ta in e d , o r
ac co rd in g to th e av erag e o f a l l fo u r s c o re s — sin c e th e o n ly v a r ia tio n s
t h a t would occur would be due e n t i r e l y to e r r o r s in r a t i n g .

A lso , such

v a r ia tio n s would occur j u s t a s f r e q u e n tly f o r th e weak s tu d e n ts a s f o r th e
s tro n g s t u d e n t s .
In t h i s s tu d y , how ever, i t i s contended t h a t such assum ptions may be
f a l s e —t h a t s i g n i f i c a n t v a r ia tio n s in th e q u a lity o f a s tu d e n t's w r itin g

79

may o c c u r n o t o n ly fro m d ay t o d a y an d from t o p i c t o t o p i c , b u t t h a t
su ch v a r i a t i o n s may o c c u r more f r e q u e n t l y f o r s t r o n g s t u d e n t s t h a n f o r
weak s t u d e n t s .
I t i s c o n te n d e d f o r e x a m p le , t h a t i f o n ly one t o p i c i s d i s c o n c e r t i n g
t o a s t u d e n t , a n d h i s e f f i c i e n c y l e v e l i s a t a low p o i n t on th e f i r s t
d ay b u t a t a h ig h p o i n t on th e se c o n d d a y , on t h e l a t t e r day h e may p r o ­
duce one p a p e r o f h ig h q u a l i t y an d one som ew hat lo w e r i n q u a l i t y .

H is

two p a p e r s o n t h e f i r s t d a y m ig h t be q u i t e low i n q u a l i t y i n c o m p a ris o n
w ith h i s b e s t p a p e r .

The s c o r e on t h a t b e s t p a p e r s h o u ld be i n d i c a t i v e

o f t h e s t u d e n t 's w r i t i n g a b i l i t y u n d e r t h e m ost f a v o r a b l e c ir c u m s ta n c e s .
On t h e o t h e r h a n d , t h e a v e ra g e o f th e s c o r e s on a l l f o u r o f h i s
p a p e r s w ould p r o v id e a n e s t im a t e o f h i s a c h ie v e m e n t c o n s i d e r a b l y b e lo w
t h a t p r o v id e d b y t h e s c o r e on h i s b e s t p a p e r — an d som ew hat above t h a t
p r o v id e d by th e s c o r e on h i s p o o r e s t p a p e r .
Thus t h e u s e o f t h e s t u d e n t 's h i g h e s t s c o r e , a s a m ethod o f r a n k i n g ,
w ould l i k e l y p ro d u c e some d i s t o r t i o n due t o e r r o r i n th e r a t i n g s j w h ile
t h e u s e o f t h e a v e ra g e o f a l l f o u r s c o r e s w ould p ro d u c e some d i s t o r t i o n
due t o v a r i a t i o n s i n p e rf o r m a n c e .

I t seem ed a d v i s a b l e , t h e r e f o r e , to

r a n k t h e s t u d e n t s by b o th m ethods i n o r d e r t o p r o v id e a c h e c k on b o th
ty p e s o f d i s t o r t i o n .
E f f e c t o f Com bined F a c to r s
W ith t h e d i f f e r e n c e s b e tw ee n t h e s c o r e s o f t h e f o u r p a i r i n g s o f
p a p e r s by e a c h s t u d e n t com puted a n d t a b u l a t e d ( s e e A p p e n d ix C) , a n d w ith
t h e s t u d e n t s ra n k e d a c c o r d in g t o t h e two m ethods d e s c r i b e d a b o v e , i t was
p o s s i b l e t o d e te rm in e w h e th e r t h e u se o f a s i n g l e p a p e r w r i t t e n by a

80
s tu d e n t on a g iv en to p ic a t a p a r t i c u l a r tim e p ro v id e s a v a lid b a s is f o r
e v a lu a tin g h is achievem ent i n a w r itin g co u rse a t any tim e .
In o rd e r to a r r i v e a t t h a t d e te rm in a tio n , i t was n e c e s s a ry to examine
th e fre q u e n c y and th e d eg ree o f s i g n i f i c a n t v a r ia tio n s i n th e q u a l ity o f w rit
in g f o r th e t o t a l group o f 80 s tu d e n ts , which r e s u l t e d from th e com bination
of f a c t o r s in v o lv ed i n t h i s stu d y — c o n te n t, e f f i c i e n c y , and p r e s s u r e .
T o ta l g ro u p .

I n r e l a t i o n to fre q u e n c y , th e r e were 86 s i g n i f i c a n t v a r i a ­

tio n s i n q u a l i t y o f w r itin g o u t o f 320 o p p o r tu n itie s f o r su ch v a r ia tio n s
to o c c u r.

S ince th e s ta n d a rd e r r o r o f th e d if f e r e n c e betw een two o b ta in e d

sc o re s was computed a t th e 5 per* c e n t le v e l o f c o n fid e n c e , no more th a n
15 such v a r i a t i o n s co u ld be ex p e cte d as a r e s u l t o f th e u n r e l i a b i l i t y of
th e r a t i n g s .

These 86 v a r ia tio n s were d i s t r i b u t e d among k 7 , o r over 58

p er c e n t, o f th e 80 s tu d e n ts in v o lv e d in t h i s s tu d y .
I n r e l a t i o n to d e g re e , th e average o f th e v a r i a t i o n s in ex c ess o f th e
d if f e r e n c e a t t r i b u t e d t o e r r o r in r a t i n g s was f o u r p o in ts —w hich was eq u al
to th e average improvement shown, i n a p re v io u s s tu d y by th e w r i t e r , f o r s t u ­
d e n ts d u rin g a y e a r 's t r a i n i n g i n th e W ritte n and Spoken E n g lish c o u rse .
A ls o , a l e t t e r - g r a d e d i s t r i b u t i o n o f th e sc o re s in d ic a te d t h a t 50 v a r ia tio n s
o f two o r more l e t t e r g rad es p ro b a b ly were due to s i g n i f i c a n t v a r ia tio n s i n
th e q u a lity o f w r i t i n g .

(See A ppendix E ) .

E f f e c t o f combined f a c t o r s on s tro n g and weak s t u d e n t s .

The 80 s tu d e n ts

were ranked by two m ethods—by u s in g th e h ig h e s t sc o re f o r each s tu d e n t,
ana by u s in g th e av erag e o f th e fo u r s c o re s f o r each s tu d e n t.

A f te r ran k ­

in g th e s tu d e n ts by th e f i r s t m ethod, a r e - ra n k in g by th e second method
r e s u l t e d i n th e s h i f t i n g o f o n ly fo u r s tu d e n ts from th e to p iiO to th e
low ^ 0 .

However, o n ly 11 o f th e to p 20 rem ained i n t h a t g ro u p .

On th e

81

o th e r hand, 15 o f th e low 20 rem ained in th e same group.

These s l i i f t s

in ran k p ro v id e some evidence in d ic a tin g g r e a te r v a r ia tio n s in th e
q u a lity o f w r itin g f o r th e s tr o n g s tu d e n ts th a n f o r th e weak s t u d e n t s .
Comparing th e to p 20 s tu d e n ts w ith th e low 20 s tu d e n ts , by both
methods o f ra n k in g , produced th e fo llo w in g fre q u e n c ie s o f s i g n i f i c a n t
v a r ia tio n s :
Top 20
U sing th e
Using th e

h ig h e s t s c o re :
av erag e s c o re :

36

22

Low 20
8
11

Comparing th e to p 1+0 s tu d e n ts w ith th e low 1+0 s tu d e n ts , by b o th
methods o f ra n k in g , produced th e fo llo w in g fre q u e n c ie s o f s i g n i f i c a n t
v a ria tio n s :
U sing th e
U sing th e

h ig h e s t s c o re :
average s c o re :

Top 1+0
63
55

Low 1+0
23
31

I t should be noted t h a t when th e av erag e sc o re i s used f o r ra n k in g
th e s tu d e n ts , th e d if f e r e n c e betw een th e fre q u en cy o f s i g n i f i c a n t v a r i ­
a tio n s f o r th e to p group and th e low group i s re d u c e d .

A p p aren tly t h i s

re d u c tio n i s th e r e s u l t o f th e av e ra g in g p ro c e s s —sin c e th e method of
averages n o t o nly te n d s to reduce th e range o f sc o re s f o r th e e n t ir e group
b ut i t a ls o te n d s to c a n c e l o u t th e r e a l d if f e r e n c e s t h a t may e x i s t b e ­
tween p ap ers by th e same s tu d e n t.

T h e re fo re , i t seems t h a t th e use o f

th e iiig h e s t sc o re by each s tu d e n t to determ ine h is ran k i s j u s t i f i e d f o r
the purpose o f a n a ly z in g th e d if f e r e n c e s betw een s tro n g and weak s t u d e n t s .
Thus, a c c o rd in g to th e d a ta given above, s i g n i f i c a n t v a r ia tio n s i n
th e q u a lity o f w r itin g o cc u rred ab o u t th r e e tim es a s o f te n among th e top
1+0 s tu d e n ts a s among th e low 1+0.

F or th e top 20 and th e low 20 , th e r a t i o

82

i s even la r g e r i n th e same d i r e c t i o n .

(Appendix D shows th e d i s t r i b u ­

tio n f o r each g ro u p .)
A more m ean in g fu l i n t e r p r e t a t i o n i s p ro v id e d by exam ining th e 50
v a r ia tio n s o f two o r

more l e t t e r g ra d e s .

Of t h e s e ,

uO s tu d e n ts w h ile o n ly seven were among th e low iiO. I t

I4.3 were amongth e to p
seems re a s o n a b le

to conclude t h a t v a r i a t i o n s

i n th e q u a lity o f w r itin g o ccu r n o t o n ly more

f r e q u e n tly b u t to a g r e a te r

d eg ree f o r th e s tro n g s tu d e n ts th a n f o r th e

weak s tu d e n ts .
A nother m ean in g fu l apjjroach i s to determ in e th e number o f s tu d e n ts
having v a r ia tio n s o f

two o r more l e t t e r g ra d e s , and

th e

freq u en cy o f such

o cc u rren c es r e l a t i v e t o th e s tr o n g and weak s t u d e n t s .
A t o t a l o f 31 s tu d e n ts had v a r ia tio n s o f two o r more l e t t e r g r a d e s .
Of th e s e 31 s t u d e n ts , 2S were among th e to p AO s tu d e n ts , w h ile o n ly s i x
were among th e low hO.
Of th e 25 s tu d e n ts i n th e
o u t of fo u r p o s s i b i l i t i e s .

to p I4O, 15 had two o r more such v a r ia tio n s

Of th e s i x s tu d e n ts in th e low hO, o nly two

had two or more such v a r i a t i o n s .
In view o f th e v a r i a t i o n s noted in th e q u a lity o f w ritin g f o r over
58 p er c e n t o f th e s tu d e n ts in v o lv e d , and in view o f th e f a c t t h a t th o se

v a r ia tio n s appeared more f r e q u e n tly and to a g r e a te r d eg ree f o r s tu d e n ts
who ranked above th e m edian th a n f o r th o s e below th e median i n w r itin g
a b i l i t y , i t seems re a s o n a b le to conclude t h a t a s in g le p ap er w r itte n by
a s tu d e n t on a g iv e n to p ic a t a p a r t i c u l a r tim e can n o t be co n sid e re d as
a v a lid b a s is f o r e v a lu a tin g h i s achievem ent i n a w r itin g cou rse a t any
tim e , u n le s s t h a t s tu d e n t* s w r itin g a b i l i t y was r a t h e r low; an d , even

63

th e n , a s in g le p a p e r would n o t p ro v id e an i n f a l l i b l e b a s i s f o r such an
e v a lu a tio n .
C o m p a ra b ility o f C o n ten t F a c to rs and V a r ia tio n s
i n E ffic ie n c y

A lthough th e ev id en ce p re s e n te d above c o n tr a d ic ts th e assu m p tion
t h a t a s in g le p ap e r can be used f o r e v a lu a tin g s tu d e n t achievem ent i n
w r itin g , no evidence was p re s e n te d co n cern in g each o f th e f i r s t th r e e
assum ptions in v o lv e d .

I n each o f th o s e tlire e a ssu m p tio n s , a p o s s ib le

source o f v a r ia tio n s i n th e q u a lity o f s tu d e n t w r itin g was i d e n t i f i e d :
l ) C ontent f a c t o r s —from d i f f e r e n t t o p i c s , 2) v a r ia tio n s in e f f ic ie n c y
on d i f f e r e n t d a y s , and 3 ) th e p s y c h o lo g ic a l p r e s s u r e o f an ex am in atio n
s itu a tio n .
S ince a l l 60 s tu d e n ts w rote on two d i f f e r e n t to p ic s on each o f two
d i f f e r e n t d a y s , a b a s is was p ro v id ed f o r d e te rm in in g th e c o m p a ra b ility
o f c o n te n t f a c t o r s and o v e r - a l l v a r ia tio n s in e f f i c i e n c y as so u rc e s o f
th e v a r ia tio n s i n th e q u a lity o f t h e i r w r itin g wliich were n o ted a b o v e .
(The use o f a l l f o u r groups f o r t h i s purpose n e c e s s a r ily in c lu d e d th e
p r e s s u r e f a c t o r under th e v a r ia tio n s i n e f f ic ie n c y and in c lu d e d b o th
s im ila r and d i s s i m i l a r to p ic s under c o n te n t f a c t o r s .

S e p a ra te t e s t s

i s o l a t i n g th e p r e s s u r e f a c t o r and i s o l a t i n g s im ila r and d is s im il a r to p ic s
a re p re s e n te d l a t e r . )
I f i t i s assumed t h a t d i f f e r e n t e f f ic ie n c y le v e ls and d i f f e r e n t
c o n te n t f a c t o r s have com parable e f f e c t s on th e v a r i a b i l i t y i n th e q u a lity
of s tu d e n t w r i t i n g , th e ex p ected fre q u en cy o f s i g n i f i c a n t v a r ia tio n s i n
th e q u a l ity o f w r itin g from each o f th e s e so u rc e s would be ab o u t th e same.

81*

L ik e w ise , i f it. i s assumed t h a t h ig h and low ra n k in g s tu d e n ts have com­
p a ra b le v a r i a t i o n s i n th e q u a l ity o f t h e i r w r itin g from day to day and
from to p ic t o t o p i c , th e ex p ected fre q u e n c y o f s i g n i f i c a n t v a r ia tio n s i n
th e q u a lity o f w r itin g by h ig h and low ra n k in g s tu d e n ts would be about
th e sam e.
T h e r e f o re , to t e s t th e assum ption c o n c ern in g so u rc e s o f v a r i a t i o n ,
i t was n e c e s s a ry to d eterm in e w hether th e d if f e r e n c e betw een th e observed
and th e ex p e cte d fre q u e n c ie s o f s i g n i f i c a n t v a r ia tio n s from d i f f e r e n t
e f f ic ie n c y le v e l s and d i f f e r e n t c o n te n t f a c t o r s may have been due to
chance o r due to r e a l d if f e r e n c e s betw een th e e f f e c t s o f th e two s o u r c e s .
L ik e w ise , to t e s t th e assum ption co n cern in g h ig h and low ra n k in g s tu d e n ts ,
i t was n e c e s s a ry t o d eterm in e w hether th e d if f e r e n c e in th e observed and
th e exp ected fre q u e n c ie s o f s i g n i f i c a n t v a r i a t i o n s f o r th e h ig h and low
ra n k in g s tu d e n ts may have been due to chance o r due to r e a l d if f e r e n c e s
in th e v a r i a b i l i t y o f t h e i r w r itin g .
By ta b u la t in g th e observed freq u en cy o f s i g n i f i c a n t v a r ia tio n s i n
th e q u a l ity o f w r itin g betw een days on th e same to p ic s and betw een to p ic s
on th e same d a y s , f o r th e h igh and low ra n k in g s tu d e n ts s e p a r a te ly , i t
was p o s s ib le to ap p ly th e c h i-s q u a re t e s t ^ to d eterm in e th e c o m p a ra b ility
o f th e s e two so u rce s o f v a r ia tio n in r e l a t i o n t o s tu d e n t w r itin g a b i l i t y .
The ta b u l a t i o n o f th o se observed fre q u e n c ie s i s shown below*

6 . (The m ost a p p r o p r ia te s t a t i s t i c a l t e s t f o r h y p o th eses co n cern in g

fre q u e n c y d i s t r i b u t i o n s when each c l a s s i f i c a t i o n o r c a te g o ry i s
d ichotom o u s.) L in d q u is t, 0£ . c i t . , p p . 33-U 7.

85

Between Days

Between T opics

Combined T o ta ls

High 20
Low 20

17
1*

19
1*

36
8

High 1*0
Low 1*0

35

28

12

11

63
23

T o ta l group

1*7

39

86

T o ta l g ro u p .

The freq u en cy o f s i g n i f i c a n t v a r ia tio n s to t a l e d 86 f o r

both so u rces o f v a r i a b i l i t y .

I f th e two so u rce s were com parable, th e

expected fre q u e n c ie s would be 1*3 f o r each s o u rc e .

A c tu a lly , th e re were

1*7 s i g n i f i c a n t v a r ia tio n s betw een days on th e same to p ic s and 39 betw een
to p ic s on th e same d a y s.

A p p lic a tio n o f th e c h i-s q u a re t e s t in d ic a t e d ,

a t th e 5 p e r c e n t le v e l of c o n fid e n c e , t h a t th e observed fre q u e n c ie s f o r
th e two sources were n o t s i g n i f i c a n t l y d i f f e r e n t from th e ex p ected f r e ­
q u e n c ie s.

T h e re fo re , i t seems re a so n a b le to conclude t h a t th e two

so u rces were re s p o n s ib le f o r com parable v a r ia tio n s in th e q u a lity o f
s tu d e n t w r itin g .
S tro n g and weak s tu d e n ts .

L ik ew ise, th e expected fre q u e n c ie s f o r th e

hig h 1*0 and th e low 1*0 would be 1*3 f o r each group.
quencies were 63 and 23, r e s p e c tiv e ly .

The observed f r e ­

A p p lic a tio n of th e c h i-s q u a re t e s t

in d ic a te d t h a t th e d if fe re n c e betw een th e observed and th e expected f r e ­
quencies was s i g n i f i c a n t a t th e 1 p er c e n t le v e l o f c o n fid e n c e .

In a li k e

m anner, a p p lic a tio n o f th e c h i-s q u a re t e s t to th e fre q u e n c ie s o f h ig h and
low ra n k in g s tu d e n ts f o r each o f th e so u rc e s s e p a r a te ly , in d ic a te d a t th e
1 p e r c e n t le v e l t h a t th e d if f e r e n c e betw een th e observed and ex p ected
fre q u e n c ie s was s i g n i f i c a n t in each c a s e .

T h e re fo re , th e assum ption th a t

v a r ia tio n s in th e q u a lity o f w r itin g by h ig h and low s tu d e n ts a re com­
p a ra b le can be r e j e c t e d .

86

C o m p arab ility o f S im ila r and D is s im ila r T opics
A lthough th e ev id en ce p re s e n te d above in d ic a te d tlia t c o n te n t f a c t o r s
and v a r ia tio n s i n e f f ic ie n c y a re e q u a lly re s p o n s ib le f o r v a r ia tio n s i n
th e q u a lity o f s tu d e n t w r i tin g , th e c o n te n t f a c t o r s in v o lv ed b o th s im ila r
and d is s im il a r t o p i c s .

Since groups A and C w rote on s im ila r t o p i c s ,

w hile groups B and D w rote on d is s im il a r t o p i c s , i t was p o s s ib le to n o te
th e fre q u e n c y o f s i g n i f i c a n t v a r ia tio n s in th e q u a lity o f w ritin g due to
each s e t o f t o p i c s .
T hus, i f i t i s assumed t h a t s im ila r and d is s im ila r to p ic s have com­
p a ra b le e f f e c t s on th e v a r i a b i l i t y i n th e q u a lity o f s tu d e n t w r i tin g , th e
expected fre q u e n c y o f s i g n i f i c a n t v a r ia tio n s f o r each s e t o f to p ic s would
be ab o u t th e same.

L ik e w ise , i f i t i s assumed t h a t h ig h and low ra n k in g

s tu d e n ts have com parable v a r ia tio n s in th e q u a lity o f t h e i r w r itin g on
s im ila r and d is s im il a r t o p i c s , r e s p e c tiv e ly , th e expected fre q u e n c ie s o f
s i g n i f i c a n t v a r ia tio n s f o r h igh and low ra n k in g s tu d e n ts would be about
th e same.
T h e re fo re , a s i n th e p re c e d in g s e c tio n , to t e s t each o f th e s e assump
tio n s i t was n e c e ss a ry to d eterm in e w hether th e d if fe re n c e betw een th e
observed and th e ex p ected f r e q u e n c ie s , in each c a s e , may liave been due
to chance o r due to a r e a l d if f e r e n c e i n th e f a c to r s in v o lv e d .
By ta b u la t in g th e observed fre q u e n c ie s of s i g n i f i c a n t v a r ia tio n s
in th e q u a l ity o f w r itin g on s im ila r to p ic s and on d is s im ila r t o p i c s ,
f o r th e h ig h and low ra n k in g s tu d e n ts s e p a r a te ly , ag a in i t was p o s s ib le
to ap p ly th e c h i-s q u a re t e s t to d eterm in e th e c o m p a ra b ility o f s im ila r
and d is s im il a r to p ic s i n r e l a t i o n to s tu d e n t w ritin g a b i l i t y .

The

£

87

t a b u la tio n o f th o s e observed fre q u e n c ie s a r e shown below :
D is s im ila r
Topics

S im ila r
T opics

Combined
T o ta ls

9

19

1

10
0

High hO
Low i|0

15

13

28

10

1

11

T o ta l group

25

1U

39

High 20
Low 20

T o ta l g ro u p .

1

The fre q u en cy o f s i g n i f i c a n t v a r ia tio n s t o t a l e d 39 f o r b o th

s im ila r and d is s im il a r to p ic s .

I f th e s e two s e ts o f to p ic s produced

com parable v a r ia tio n s in th e q u a lity o f w r itin g , th e expected fre q u e n c ie s
would be 19 1 /2 f o r each s e t .

The observed fre q u e n c ie s were 25 f o r d i s ­

s im ila r to p ic s and lU f o r s im ila r t o p i c s .

A p p lic a tio n o f th e c h i-sq u a re

t e s t in d ic a te d t h a t th e d if f e r e n c e between th e o bserved and ex p ected f r e ­
quencies was n o t s i g n i f i c a n t a t th e 5 p a r c e n t le v e l o f c o n fid e n c e .
T h e re fo re , th e assum ption t h a t s im ila r and d is s im il a r to p ic s have compar­
a b le e f f e c t s on th e v a r i a b i l i t y in th e q u a lity o f s tu d e n t w r itin g cannot
be r e j e c t e d .
S tro n g and weak s tu d e n ts .

However, th e tre n d o f fre q u e n c ie s from h ig h to

low ra n k in g s tu d e n ts su g g e sts t h a t a lth o u g h th e re was no s i g n i f i c a n t d i f f e r ­
ence i n th e fre q u e n c ie s o f 1 5 and 13 on d is s im ila r and s im ila r t o p i c s , r e ­
s p e c tiv e ly , f o r th e h ig h hOt th e re was a d if f e r e n c e i n th e r e s p e c tiv e
A lso , f o r th e d is s im ila r t o p ic s ,
M*

fre q u e n c ie s o f 10 and 1 f o r th e low UO.

th e fre q u e n c ie s f o r th e h ig h UO and th e low liO were 15 and 1 0 , r e s p e c tiv e ly
w h ile on th e s im ila r to p ic s th e fre q u e n c ie s were 13 and 1 , r e s p e c tiv e ly —

88

aH

° f w h ic h s u g g e s t s t h a t d i s s i m i l a r t o p i c s d i d a c c o u n t f o r a c o n s i d e r ­

a b le num ber o f s i g n i f i c a n t v a r i a t i o n s f o r t h e w eak er s t u d e n t s , w h ile
th e s im ila r to p ic s d id n o t.

T liis i n t e r p r e t a t i o n was su p p o rte d by th e a p p l ic a tio n o f th e c h i-sq u a re
t e s t to th e f re q u e n c ie s on d i s s i m i l a r to p ic s (1 5 and 1 0 ) , and to th e
combined fre q u e n c ie s (28 and l l ) f o r th e h ig h iiO and th e low UO, re sp e c ­
tiv e ly .

For th e f re q u e n c ie s on d i s s i m i l a r t o p i c s , th e c h i-s q u a re found

was n o t s i g n i f i c a n t even a t th e 30 p e r c e n t le v e l o f c o n fid e n c e .

Y e t,

f o r th e combined f r e q u e n c i e s , th e c h i-s q u a re found was s i g n i f i c a n t a t th e
1 per cent le v e l.

T h e re fo re , i t seems re a s o n a b le to conclude t h a t s im ila r and d is s im il a r
to p ic s were ab o u t e q u a lly r e s p o n s ib le f o r s i g n i f i c a n t v a r ia tio n s in th e
q u a lity o f w r itin g by th e b e t t e r s t u d e n t s , w h ile o nly th e d is s im il a r to p ic s
were r e s p o n s ib le f o r s i g n i f i c a n t v a r ia tio n s i n th e q u a lity o f w r itin g by
th e w eaker s t u d e n t s .
E f f e c t o f th e P re s s u re F a c to r
As p r e v i o u s l y s t a t e d , t h e t e s t f o r t h e c o m p a r a b i l it y o f c o n te n t
f a c t o r s a n d v a r i a t i o n s i n e f f i c i e n c y i n c l u d e d th e p s y c h o l o g ic a l p r e s s u r e
a s a n e le m e n t i n th e e f f i c i e n c y f a c t o r .

The t h i r d assum ption s t a t e s t h a t th e p s y c h o lo g ic a l p re s s u re i n t r o ­
duced by th e ex am in atio n s i t u a t i o n has no ad v e rse e f f e c t on th e q u a lity
o f a s tu d e n t’s w r i t i n g .

In o rd e r to t e s t t h a t assu m p tio n , i t was n e c e ssa ry

to compare th e q u a l i t y o f w r itin g by in d iv id u a l s tu d e n ts who w rote w ith o u t
p re s s u re on one day and w ith p r e s s u r e on a n o th e r day and th e n to compare
th o se r e s u l t s w ith c o n tr o l s tu d e n ts who w rote on b o th days w ith o u t p r e s s u r e .

89
I f th e p r e s s u r e f a c t o r had no a d v e rs e e f f e c t , th e n th e r e sh o u ld be
no more l o s s e s and no few er g a in s i n s c o r e s , from th e f i r s t to th e second
d a y , f o r th e p r e s s u r e s tu d e n ts tlian f o r th e n o n -p re ssu re s tu d e n ts .

In

o th e r w ords, i f i t i s assumed t li a t th e p re s s u re f a c t o r has no a d v e rse
e f f e c t on th e q u a lity o f w r i t i n g , th e n th e ex p ected fre q u e n c ie s o f s i g n i f i ­
c a n t g ain s and lo s s e s in sc o re s from th e f i r s t to th e second day should
be ab o u t th e same f o r th e p r e s s u r e g ro u p s , C and D, a s f o r th e n o n -p re ssu re
g ro u p s, A and B.

L ik e w ise , i f i t i s assumed t h a t th e h ig h and low ra n k in g

s tu d e n ts have com parable v a r ia tio n s i n th e q u a l ity o f t h e i r w r itin g w ith
and w ith o u t th e p re s s u re f a c t o r , th e ex p ected fre q u e n c ie s o f s i g n i f i c a n t
g ain s and lo s s e s i n th e q u a l i t y o f w r itin g by th e h ig h and low ra n k in g
s tu d e n ts sh o u ld be ab o u t th e sam e.
T h u s, a g a in , to t e s t th e s e two assu m p tio n s i t was n e c e ss a ry to de­
te rm in e w hether th e d if f e r e n c e betw een th e observed and th e ex p e cte d f r e ­
q u e n c ie s , in each c a s e , may have been due to chance o r due to r e a l d i f f e r ­
ences in th e f a c t o r s in v o lv e d .
By ta b u la t in g th e o bserved fre q u e n c ie s o f g a in s and lo s s e s f o r non­
p re s s u re groups A and B and th e p r e s s u r e groups C and D, d iv id e d betw een
th e high and low ra n k in g s tu d e n ts , i t was p o s s ib le to ap p ly th e c h i-s q u a re
t e s t to d eterm in e w h eth er th e d if f e r e n c e betw een observed and ex p ected
fre q u e n c ie s were due to chance o r due to th e f a c t o r s in v o lv e d .
l a t i o n o f th o s e o bserved fre q u e n c ie s i s shown below :

The ta b u ­

90

N o n -p re s s u re
G roups A and B

G ains

High UO

&

L o sse s
2

17
5

13

22

5

7
3

17
6

15

10

2l+

23

25
U7

Low UO
T o t a l g ro u p :
P re ssu re
G roups C and D
H igh LO
Low ItO
T o t a l g ro u p
G rand T o ta l
T o t a l g ro u p .

11

Com bined G ains
an d L o s s e s

10

E x a m in a tio n o f t h e t o t a l f r e q u e n c i e s s u p p o r ts t h e a s s u m p tio n

t h a t th e f r e q u e n c i e s o f g a in s s h o u ld be no fe w e r a n d th e f r e q u e n c y o f
l o s s e s no m ore f o r th e p r e s s u r e groups th a n f o r t h e n o n - p r e s s u r e g r o u p s .

Of th e grand t o t a l o f 2h g a in s , 15 were by th e p re s s u re group and 9 by
th e n o n -p re ssu re group.

Of th e grand t o t a l o f 23 l o s s e s , o n ly te n were

by th e p r e s s u r e g ro u p s, w h ile 13 were oy th e n o n -p re ssu re g ro u p s.

A lthough

th e d a ta in d ic a te a tre n d o f more lo s s e s th a n g a in s by th e n o n -p re ssu re
groups on th e second d ay , and more g ain s th a n lo s s e s by th e p re s s u re groups
on th e second d a y , a p p lic a tio n o f th e c h i-s q u a re t e s t in d ic a t e d , a t th e
5 p er c e n t l e v e l , t h a t th e se observed d if fe re n c e s were no g r e a te r th a n

would be exp ected by ch a n c e .

The d a ta m erely su g g e st t h a t a la r g e r sample

m ight s u b s t a n tia te th e tre n d s in d ic a te d .

T h e re fo re , i t seems re a so n a b le

to conclude t h a t th e p sy c h o lo g ic a l p re s s u re of th e exam ination i s n o t
l i k e l y to have an ad v erse e f f e c t on th e q u a lity o f w r itin g b y a number of
s tu d e n ts .
S tr o n g and weak s t u d e n t s .

The f r e q u e n c i e s o f s i g n i f i c a n t g a in s and l o s s e s

f o r th e low r a n k i n g s t u d e n t s i n b o th t h e p r e s s u r e a n d n o n - p r e s s u r e g ro u p s

91

w ere to o s m a ll f o r a p p l i c a t i o n o f t h e c h i - s q u a r e t e s t .

H ow ever, th e

t r e n d s f o r t h e h i g h and lo w r a n k i n g s t u d e n t s i n b o t h g ro u p s a r e q u i t e
s i m i l a r t o th o s e f o r th e t o t a l g r o u p . : T h e r e f o r e , i t seem s r e a s o n a b l e t o
c o n c lu d e t h a t t h e p r e s s u r e o f th e e x a m in a tio n h ad l i t t l e
e i t h e r h ig h o r lo w r a n k i n g s t u d e n t s .

o r no e f f e c t on

CHAPTER VI

SUMMARY, CONCLUSIONS, AND IMPLICATIONS
Summary.

The p u rp o s e o f t h i s t h e s i s was t o d e te rm in e w h e th e r a s i n g l e

p a p e r w r i t t e n b y a s t u d e n t on a g iv e n t o p i c a t a p a r t i c u l a r tim e c an b e
c o n s id e r e d a s a r e p r e s e n t a t i v e sam ple o f h i s w r i t i n g a b i l i t y — and th u s
p r o v id e a v a l i d b a s i s f o r e v a l u a t i n g a b i l i t y a t any tim e i n a w r i t i n g
c o u rse .
F o r t h a t p u r p o s e , i t was n e c e s s a r y t o t e s t th e f o llo w in g f o u r b a s i c
a s s u m p tio n s in v o lv e d i n t h e u s e o f a s i n g l e p a p e r f o r e v a l u a t i n g a s t u ­
d e n t ’ s w r i t i n g a b i l i t y — t h e a s s u m p tio n s t h a t :
1.

a n y g iv e n t o p i c p r o v id e s t h e same s tim u lu s a s any o t h e r t o p i c ,

2.

a n y g iv e n t o p i c e l i c i t s c o n s t a n t r e s p o n s e s a t d i f f e r e n t t i m e s ,

3.

t h e p s y c h o l o g ic a l p r e s s u r e i n tr o d u c e d b y a n e x a m in a tio n s i t u ­
a t i o n lias no a d v e r s e a f f e c t on t h e q u a l i t y o f w r i t i n g , and

h.

th e q u a l i t y o f w r i t i n g i s s t a b l e from t o p i c t o t o p i c and from
tim e t o tim e f o r a n y s t u d e n t

To

re g a rd le s s o f h is w ritin g a b i l i t y .

t e s t t h e f i r s t a s s u m p tio n , i t was n e c e s s a r y t o

d e te rm in e w h e th e r

th e a s s ig n m e n t o f d i f f e r e n t t o p i c s d u r in g t h e same p e r i o d h a s an y e f f e c t
on th e q u a l i t y o f w r i t i n g .
To t e s t th e s e c o n d a s s u m p tio n , i t was n e c e s s a r y t o d e te rm in e w h e th er
t h e a s s ig n m e n t o f th e same t o p i c on d i f f e r e n t d a y s h a s any e f f e c t on w r i t ­
in g q u a lity .

93

To t e s t th e t h i r d assu m p tio n , i t was n e c e ssa ry to determ ine w hether
th e assignm ent of th e same t o p i c s , w ith and w ith o u t th e p re s s u re o f an
e lim in a tio n , has any e f f e c t on th e q u a lity o f w r itin g .
To t e s t th e f o u r th assum ption i t was n e c e ssa ry to ran k th e s tu d e n ts
acco rd in g to t h e i r w r itin g a b i l i t y and to d eterm in e whether th e frequency
a n d /o r th e d eg ree o f v a r ia tio n s in th e q u a lity o f w ritin g d i f f e r s from
s tro n g to weak s tu d e n ts .
The d a ta f o r t e s t i n g th o se assum ptions were o b ta in e d by h av in g fo u r
groups o f s tu d e n ts (tw enty in each g ro u p ), e n ro lle d in th e W ritte n and
Spoken E n g lish D epartm ent a t M ichigan S ta te C o lle g e , w rite two p a p e rs on
d i f f e r e n t to p ic s on each o f two d i f f e r e n t days..

Two g ro u p s, A and B,

w rote under th e same c o n d itio n s on both d ay s—w ith th e u n d ersta n d in g th a t
th e r e s u l t s would n o t a f f e c t t h e i r grades f o r the c o u rs e .

The o th e r two

g ro u p s, C and D, w rote t h e i r two p ap ers on th e f i r s t day under th e same
c o n d itio n s as groups A and B.

On th e second day, groups C and D were

a ssig n e d to w rite p ap ers on th e same two t o p ic s , b u t w ith th e u n d e rsta n d ­
in g t h a t th e grades a s sig n e d th o se p ap ers would count a s a p a r t o f t h e i r
f i n a l exam ination f o r th e co u rse .

Thus groups A and B provided a b a s is

f o r d eterm in in g v a r ia tio n s due to to p ic s on each d a y , a b a s is f o r determ in­
in g v a r ia tio n s in w ritin g due to e f f ic ie n c y v a r ia tio n s from th e f i r s t to
th e second day, p ro v id ed a c o n tr o l group f o r d eterm in in g v a r ia tio n s due
to th e exam ination p re s su re on groups C and D th e second day.
S teps were ta k en to a s su re t h a t th e e v a lu a tio n sco re a ssig n e d each
o f th e fo u r pap ers w r itte n by each o f th e 80 s tu d e n ts was as r e l i a b l e and
v a lid as could be o b ta in e d w ith c u r r e n t r a t i n g m ethods.

These sc o re s were

9k

an a ly ze d to d eterm in e w hether each group o f s tu d e n ts was r e p r e s e n ta tiv e
o f th e t o t a l group and w hether d i f f e r e n t t o p i c s , e f f ic ie n c y v a r i a t i o n s ,
o r th e p r e s s u r e o f an exam ination had a ry e f f e c t on th e av erag e q u a lity
o f w r itin g by th e a p p r o p r ia te g ro u p s.
A fte r i t had been d eterm ined t h a t th e fo u r groups were com parable,
and t h a t none o f th e th r e e f a c t o r s had any s i g n i f i c a n t e f f e c t on th e
av erag e q u a lity o f w r itin g , i t was th e n p o s s ib le to a n a ly ze th o se f a c to r s
in r e l a t i o n to t h e i r e f f e c t s on d i f f e r e n t in d iv id u a ls w ith d i f f e r e n t w r it­
in g a b i l i t i e s .
In p la n n in g t h i s s tu d y , an a tte m p t was made to p ro v id e th e c o n tro ls
n e c e ssa ry to p re c lu d e th e in te r v e n tio n o f e x tra n e o u s f a c t o r s .

I t seems

a d v is a b le , how ever, to r e s t a t e th e fo llo w in g q u a l if i c a t io n s b e fo re p r e ­
s e n tin g th e c o n c lu sio n s :
1 . The 80 s tu d e n ts in v o lv ed i n t h i s stu d y were o nly a sm a ll sample
(ab o u t tlire e p e r c e n t) o f th e e n t i r e s tu d e n t p o p u la tio n e n r o lle d i n th e
c o u rs e .

However, th e re i s l i t t l e re a so n to b e lie v e t h a t th e s e s tu d e n ts

were n o t r e p r e s e n ta tiv e o f tlia t p o p u la tio n .

T h e re fo re , c o n c lu sio n s

r e l a t i v e to th e 80 s tu d e n ts can be a p p lie d w ith re a s o n a b le co n fid en ce to
th e e n t i r e s tu d e n t p o p u la tio n .
2 . The use o f th e h ig h e s t sco re o f each s tu d e n t to ra n k th e s tu d e n ts
ac c o rd in g to t h e i r w ritin g a b i l i t y could p ro v id e some d i s t o r t i o n .

The

co n c lu sio n s p r e s e n te d , however, a re based on th e a n a ly s e s o f th e d a ta on
th e h ig h iiO and th e low i±0 —and i t has been shown t h a t u s in g th e h ig h e s t
s c o r e , f o r each s tu d e n t, r a t h e r th a n th e average o f a l l fo u r s c o r e s , r e ­
s u lte d i n changing th e p o s itio n o f o nly fo u r s tu d e n ts betw een th e s e two

g ro u p s.

I t would seem , t h e r e f o r e , t h a t th e c o n c lu sio n s co n c ern in g v a r i ­

a tio n s i n th e q u a lity o f w r itin g f o r s tro n g and weak s tu d e n ts sh o u ld be
v a lid .
3 . T here was no a s s u ra n c e t h a t th e r a t i n g method p ro v id e d sc o re s
c o n s is tin g o f eq u a l u n i t s .

T h e r e f o re , th e r e s u l t s o f th e s t a t i s t i c a l

t e s t s may be open to q u e s tio n .

In o th e r w ords, th e c o n c lu sio n s based

upon a s t a t i s t i c a l a n a ly s is o f such sc o re s can n o t be s t a t e d w ith as
much c e r t a i n t y a s would be j u s t i f i e d i f th e s c o re s were known to c o n s is t
o f eq u al u n i t s .
C o n c lu sio n si

From th e a n a ly s e s o f th e w r itin g perform ance by th e groups

o f s tu d e n ts in v o lv e d , i t seems re a s o n a b le to conclude t h a t :
1.

Under norm al c o n d i tio n s , v a r ia tio n s i n th e e f f i c i e n c y o f i n ­

d iv id u a l s tu d e n ts do n o t s i g n i f i c a n t l y a f f e c t th e av erag e q u a lity of
w r itin g by a group o f tw enty o r more s tu d e n ts from day to day on th e same
to p ic s .
2 . When s u f f i c i e n t c a re i s ta k e n to p ro v id e to p ic s w hich a re ap p ro ­
p r i a t e to th e s t u d e n t s 1 background o f e x p e rie n c e s , th e a c c id e n ta l c o n te n t
f a c t o r s r e s u l t i n g from th e assignm ent o f any one o f th o se to p ic s do n o t
a f f e c t th e av erag e q u a lity o f w r itin g by a group o f tw en ty o r more s tu d e n ts
3 . When s tu d e n ts a re a s s ig n e d th e ta s k o f w r itin g a theme a s a p a r t
of a f i n a l e x a m in a tio n , th e average q u a l ity o f th e w r itin g under such
c o n d itio n s w i l l be about th e same as tho u g h th e p a p e rs had been w r i tte n
as an assignm en t wtiich would n o t a f f e c t t h e i r grad es f o r th e c o u rs e .
From th e a n a ly se s o f th e w r itin g perform ance by in d iv id u a l s tu d e n ts
in each g ro u p , i t seems re a s o n a b le t o conclude t h a t :

96

1 . C ontent f a c t o r s due t o d i f f e r e n t to p ic s and v a r ia tio n s i n e f f i c ­
ie n c y from day to day r e s u l t in s i g n i f i c a n t v a r ia tio n s in th e q u a lity o f
w r itin g f o r a c o n s id e ra b le number o f th e s tu d e n ts in v o lv e d , and th e s e
v a r ia tio n s a r e l i k e l y to o ccu r w ith s i g n i f i c a n t l y g r e a te r freq u en cy f o r
s tr o n g s tu d e n ts th a n f o r weak s t u d e n t s .
2 . C ontent f a c t o r s and v a r ia tio n s i n e f f ic ie n c y a re about e q u a lly
re s p o n s ib le f o r such v a r ia tio n s in th e q u a l ity o f s tu d e n ts ' w ritin g a s
o c c u r.
3 . For th e s tro n g s tu d e n ts , d is s im il a r to p ic s a re re s p o n s ib le f o r no
more s i g n i f i c a n t v a r ia tio n s th a n s im ila r t o p i c s .

For th e weak s tu d e n ts ,

how ever, d is s im il a r to p ic s appear to r e s u l t in more s i g n i f i c a n t v a r ia tio n s
in th e q u a l ity of w r itin g th a n s im ila r t o p i c s .
k . P sy c h o lo g ic a l p r e s s u re in tro d u c e d by having p ap ers w r i tte n as a
p a r t o f a f i n a l ex a m in a tio n , does n o t r e s u l t in a s i g n i f i c a n t l y g r e a te r
, fre q u en cy o f g ain s a n d /o r lo s s e s f o r in d iv id u a l s tu d e n ts th an a r e l i k e l y
to occur when such p ap ers a r e w r itte n a s an assignm ent which w i l l n o t
a f f e c t th e s tu d e n ts ' g ra d e s .

T hus, i t ap p e ars t h a t such p re s s u re has no

ad v e rse e f f e c t on a s i g n i f i c a n t number o f s tu d e n ts .
A lthough s i g n i f i c a n t v a r ia tio n s i n th e q u a lity o f w r itin g , e i t h e r
from to p ic to to p ic o r from day to d ay , were n o ted f o r hi o f th e 80 s t u ­
d en ts in v o lv ed in t h i s s tu d y , no such v a r ia tio n s were n o ted f o r th e o th e r
33 s t u d e n t s .

T his does n o t mean, how ever, t h a t th e r e were no v a r ia tio n s

in th e q u a lity o f w r itin g by any o f th o se s tu d e n ts .

I t can be s a i d , o n ly ,

t h a t i f such v a r ia tio n s d id o c c u r, th e y d id n o t exceed th e e r r o r o f th e
ra tin g s .

On th e o th e r h an d , i t seems re a s o n a b le to assume t h a t f o r some o f
th o s e s tu d e n ts , th e two a s s ig n e d to p ic s p ro v id ed ab o u t eq u al s t i m u l i —
r e s u l t i n g i n no s i g n i f i c a n t d if f e r e n c e i n th e q u a l ity o f w r itin g on
th o s e two t o p i c s .

L ik e w ise , some o f th o s e s tu d e n ts may have b een w ritin g

a t a b o u t th e same e f f ic ie n c y l e v e l on b o th days—r e s u l t i n g i n no s i g n i f i ­
c a n t d if f e r e n c e i n th e q u a l i t y o f t h e i r w ritin g on th e same to p ic s on
d i f f e r e n t d a y s.
%

I m p lic a tio n s .

I n C h ap ter I , two m ajor problem s were r a is e d c o n c ern in g

th e use o f a s in g l e sample o f a s t u d e n t 's w r itin g as in d ic a tiv e o f h is
w r itin g a b i l i t y —f i r s t , th e problem o f e v a lu a tin g h is achievem ent ( a s s ig n ­
in g a grade) a t th e c o n c lu sio n o f a w r itin g c o u rs e } a n d , seco n d , th e
problem of e v a lu a tin g in d iv id u a l s tu d e n t improvement from th e b e g in n in g
to th e end o f a c o u r s e .
I n r e l a t i o n t o th e f i r s t p ro b lem , th e f in d in g s from t h i s stu d y c a s t
c o n s id e ra b le doubt upon th e j u s t i f i c a t i o n o f th e custom ary p r a c t ic e o f
u sin g f iv e l e t t e r - g r a d e s to d e s ig n a te achievem ent in a w r itin g co u rse
when a s in g le p ap er p ro v id e s th e b a s is f o r t h a t d e s ig n a tio n .
For exam ple, th e f in d in g s from t t i i s stu d y in d ic a te t h a t v a r ia tio n s
i n th e q u a l ity o f a s t u d e n t 's w r itin g from to p ic to t o p i c , o r from day
to day on th e same t o p i c , p lu s th e e r r o r i n a s s ig n in g a sc o re f o r each
p a p e r , a re l i k e l y to r e s u l t i n v a r ia tio n s o f two o r more l e t t e r - g r a d e s
i n th e e v a lu a tio n o f h is w r itin g a b i l i t y —and t h a t such v a r ia tio n s a re
l i k e l y to occur more f r e q u e n tly f o r s tr o n g th a n f o r weak s tu d e n ts .
T hus, a s tu d e n t m ight have h is w r itin g a b i l i t y r a te d as "A" from one
pap er and a s "G" from a n o th e r p a p e r.

However, i f o n ly one o f th o s e p ap ers

98

had been w r i t t e n a s an in d ic a t io n o f h is w r itin g a b i l i t y , he would have
re c e iv e d on ly one o f th o se r a t i n g s .

I t i s e n t i r e l y p o s s ib le t h a t th e

"AM r a t i n g would n o t have b een th e b e s t e s tim a te o f t h a t s tu d e n t 1s w r i t ­
in g a b i l i t y .

From th e ev id en ce p re s e n te d i n t h i s s tu d y , a grade o f "B1*

m ight have been a b e t t e r e s tim a te —s in c e e r r o r i n r a t i n g could ac co u n t
f o r su ch a d i f f e r e n c e , even when th e m ost r e l i a b l e r a t i n g methods c u r­
r e n t l y a v a ila b le a r e u se d .
On th e o th e r h an d , i f t h a t s t u d e n t 's s in g l e p ap er had re c e iv e d a
r a t i n g o f "C ", th e r e would be no a s s u ra n c e t h a t a r a t i n g o f "A” would n o t
have been a b e t t e r e s tim a te o f M s w r itin g a b i l i t y —s in c e a com bination
o f u n s u ita b le t o p i c , o r a low e f f ic ie n c y l e v e l a t th e tim e th e paper was
w r i t t e n , and e r r o r in r a t i n g co u ld ac c o u n t f o r such a d if f e r e n c e .
Even i f f o u r sam ples o f a s t u d e n t 's w r itin g were o b ta in e d

(cm

two

d i f f e r e n t to p ic s on each o f two d i f f e r e n t d a y s ) , perhaps n o t more th a n a
t h r e e - l e v e l g ra d in g system co u ld be j u s t i f i e d — i n view o f th e amount o f
e r r o r in p r e s e n t r a t i n g m ethods, p lu s p o s s ib le v a r ia tio n s i n th e q u a lity
o f w r itin g from to p ic to to p ic and from tim e t o tim e .

S ince th e f in d in g s

from t h i s stu d y in d ic a te l i t t l e v a r i a t i o n i n th e q u a lity o f w ritin g by
th e weaker s tu d e n ts , an u n s a tis f a c to r y grade m ight be a s s ig n e d from an
e v a lu a tio n o f fo u r p a p e rs w ith c o n s id e ra b le co n fid e n ce t h a t no i n j u s t i c e
i s b e in g i n f l i c t e d upon th e s tu d e n t.

Since c o n s id e ra b le v a r ia tio n s in

th e q u a lity o f w r itin g were n oted f o r th e s tr o n g e r s tu d e n ts , p erh aps no
more th a n two le v e ls o f p a s s in g g rad es co u ld be a s sig n e d w ith co n fidence
i n t h e i r a c c u ra c y .

99

The second problem r a i s e d i n C h ap ter I concerned th e e v a lu a tio n o f
improvem ent i n th e q u a l i t y o f s tu d e n t w r i tin g d u rin g th e p e r io d o f a
w r itin g c o u r s e .

I f an e v a lu a tio n o f o v e r - a l l o r av e ra g e improvement i s

a l l t h a t i s d e s i r e d , i t ca n be o b ta in e d from a s in g le sample o f each s t u ­
d e n t’s w r itin g f o r a p r e - t e s t and a p o s t - t e s t , a s in d ic a te d i n C h ap ter I .
Such an e v a lu a tio n , how ever, p ro v id e s l i t t l e u s e f u l in fo rm a tio n f o r
e v a lu a tin g th e e f f e c t iv e n e s s o f d i f f e r e n t te a c h in g methods w ith s tu d e n ts
who v a ry i n w r itin g a b i l i t y — s in c e v a r i a t i o n s i n q u a l ity o f w r i t i n g , p lu s
e r r o r i n r a t i n g s , w i l l te n d t o n u l l i f y o r e x a g g e ra te th e a c tu a l im prove­
ment f o r a c o n s id e ra b le number o f th e s tu d e n ts in v o lv e d .
From th e r e s u l t s o f t h i s s tu d y , i t would seem t h a t i n o rd e r to d ev elo p
a program f o r e v a lu a tin g in d iv id u a l s tu d e n t improvement i n w r itin g ( f o r
s tr o n g a s w e ll a s f o r weak s t u d e n t s ) , i t would be a d v is a b le to o b ta in
s e v e r a l sam ples o f w r i tin g b y each s tu d e n t— sam ples o f w r i tin g on d i f f e r ­
e n t to p ic s on th e same day and on th e same to p ic s on d i f f e r e n t d a y s .
And such sam ples sh o u ld be o b ta in e d f o r b o th th e p r e - t e s t and th e p o s t ­
te s t.

I f th e to p ic s a s s ig n e d f o r th e p o s t - t e s t a r e n o t t o be th e same as

f o r th e p r e - t e s t , th e n th e y sh o u ld be s im ila r i n n a tu re — e s p e c i a lly f o r
th e weak s tu d e n ts .
Such a program o f p r e - t e s t i n g and p o s t - t e s t i n g would p ro v id e a b a s is
f o r e v a lu a tin g th e im provem ent i n w r itin g p r o f ic ie n c y f o r b o th s tr o n g and
weak s tu d e n ts —re d u c in g th e p o s s i b i l i t y t h a t c o n te n t f a c t o r s a n d /o r
v a r i a t i o n s i n e f f i c i e n c y would a f f e c t th e q u a l ity o f w r itin g i n such a
way a s to n u l l i f y o r e x a g g e ra te a c t u a l im provem ent.
F u r th e r m o r e , i t w ould se e m , from t h e r e s u l t s o f t h i s s t u d y , t h a t
t h e p r e s s u r e o f t h e e x a m in a tio n s i t u a t i o n d o e s n o t r e s u l t i n m ore a d v e r s e

100
v a r i a t i o n s i n th e q u a l i t y o f w r itin g th a n when such p r e s s u r e i s a b s e n t.
T h is does n o t mean, how ever, t h a t su ch v a r i a t i o n s w i l l n o t o c c u r, o c c a ­
s io n a lly .

On th e o th e r h an d , i t seems re a s o n a b le t o assume t h a t , i f

p ro p e r p r e c a u tio n s a r e ta k e n , a p o s t - t e s t may be u t i l i z e d a s a f i n a l ex­
a m in a tio n w ith o u t s e r i o u s l y a f f e c t i n g th e r e s u l t s .
Such a program o f e v a lu a tio n , a s o u tlin e d a b o v e , would e n t a i l a
c o n s id e r a b le amount o f tim e and e f f o r t on th e p a r t o f th e s t a f f members
in v o lv e d .

I t seems to t h i s w r i t e r , how ever, t h a t u n le s s such a program

i s u t i l i z e d f o r e v a lu a tin g in d iv id u a l improvem ent i n a w r itin g c o u r s e ,
t h e r e i s l i t t l e p o s s i b i l i t y o f im proving c o u r s e s , program s^ o r te a c h in g
m ethods co n cern ed w ith t h a t o b j e c t i v e .

And u n le s s improvement does ta k e

p la c e i n th e e f f e c t iv e n e s s o f c o u r s e s , p ro g ram s, o r te a c h in g methods con­
ce rn e d w ith th e improvement o f s tu d e n t w r i t i n g , te a c h e r s i n t h a t a re a
can c o n tin u e t o e x p e c t th e same c r i t i c i s m s th e y have re c e iv e d in th e
p a s t.

BIBLIOGRAPHY
A d k in s, D orothy C . , C o n s tru c tio n and A n a ly s is o f A chievem ent T e s t s .
U. S . Government P r i n t i n g O f f ic e , W ash in g to n , b . C
191*7.
A nd erso n , H. A ., and T r a i l e r , A. E . , "The R e l i a b i l i t y o f th e R eading o f
an E n g lis h E ssay Test*1, School R eview , v o l . 1+8, 191*0, p p . £ 2 1 - 3 0 .
Coward, Ann F . , "A Com parison o f Two Methods o f G rading E n g lis h Composi­
tio n s " , J o u rn a l o f E d u c a tio n a l R e s e a rc h , v o l . 1+6 (O cto b er 1952) .
p p . 8 1 -9 3 .
C ronbach, Lee J . , E s s e n tia l s o f P s y c h o lo g ic a l T e s tin g . H arp er and B ro th e rs
New Y ork, 19& 1--------------------------------- -----------------D a r s ie , M arvin L . , "The R e l i a b i l i t y o f Judgments B ased on th e W illin g
C om position S c a le ," J o u rn a l o f E d u c a tio n a l R e s e a rc h , v o l. £ (Ja n u a ry
1 9 2 2 ), p p . 89-90.
D ie d e ric h , P a u l B ., "The Measurement o f S k i l l i n W r itin g " , School R eview ,
v o l . ££ ( A p r il 191*8), p p . 375-$l*.
, "Readers* Methods S c r u tin iz e d i n E n g lis h E ssay S c o rin g ," Educa­
t i o n a l T e s tin g S e rv ic e Developments , No. 1 (O cto b er 195>l) .
, "The Use o f E ssays to M easure Im provem ent," C o lle g e E n g lis h , v o l.
10 ( A p r il 191*9), p p . 39£-99.
D o lch , Edward W ., J r . , "More A ccu rate Use o f C om position S c a le s ," E n g lish
J o u r n a l, V ol. 11 (November 1922) , p p . £36-1*1*.
E b e l, R o b ert L . , " E s tim a tio n o f th e R e l i a b i l i t y o f R a tin g s ," P sychom etrika
v o l. 1 6 , No. 1* (December 19£l) , p p . 1*07-21*.
E d m isto n , R. W ., and G in g e ric h , C , N ., "The R e la tio n o f F a c to rs o f E n g lish
Usage to C o m p o sitio n ," J o u rn a l o f E d u c a tio n a l R e se a rc h , v o l. 3 6 ,
191*2, p p . 269-71.
G a te s, A rth u r I . , P sychology f o r S tu d e n ts o f E d u c a tio n , The M acm illan
Company, New Y ork, 1931.
G e rb e r, John C . , " T e s tin g and E v a lu a tio n i n th e S k i l l s o f Com m unication,"
C o lleg e E n g lis h , v o l . 9 ( A p ril 19l*8) , p p . 375-81*.
G reene, " E n g lis h —L anguage, Grammar, and C o m p o sitio n ," E n cy clo p ed ia o f
E d u c a tio n a l R e se a rc h , M acm illan, New Y ork, 1950.

102
H a rtu n g , M. L . , and o t h e r s , "A sp ects o f T h in k in g ," A p p ra is in g and R ecord­
in g S tu d e n t P r o g r e s s . H arper and B r o th e r s , N ew Y o r^T T ^U ^
Hawkes, H. E . , and o th e r s , The C o n s tru c tio n and Use o f A chievem ent
E x a m in a tio n s. Houghton M i f f l i n Company, New Y ork, 1936.
H i l l e g a s , M ilo B . , "S c a le f o r th e M easurement o f Q u a lity i n E n g lis h Compo­
s i t i o n by Young P e o p le ," T each ers C o lle g e R eco rd , v o l . 13 (Septem ber
1912) p p . 331-8U.
H uddelson, E a r l , "The E f f e c t o f O b je c tiv e S ta n d a rd s Upon C om position
T eachers* Judgm ents," J o u rn a l o f E d u c a tio n a l R e s e a rc h , v o l. 12
x
(December 1 9 2 5 ), p p . 32 9 -AO.
H u d d lesto n , E d ith M ., M easurement o f W ritin g A b i l i t y a t th e C o lle g e E n tran ce L evel* O b je c tiv e v s . S u b je c tiv e T echniques (P h . D. T h e s is ) ,
New York U n iv e r s ity , 1 9 5 2 ,. 10U numb, le a v e s j p u b lis h e d a s R esearch
B u l l e t i n RB- 5 2 - 7 , E d u c a tio n a l T e s tin g S e r v ic e , P r in c e to n , N, J . , 1952.
L ange, P h il C ., "A rSam pling o f C om position E rr o r s o f C o lle g e Freshmen i n
a Course O ther Than E n g lis h ," J o u rn a l o f E d u c a tio n a l R e s e a rc h , v o l.
h2 (November 1 9 ^ 8 ), p p . 1 91-200.
L eo n ard , S . A ,, " B u ild in g a S c a le o f P u re ly C om position Q u a lity " , E n g lish
J o u r n a l, v o l. lJU (December 1925) p p . 7 6 0 -7 5 .
L in d q u is t, E . F . , S t a t i s t i c a l A n a ly s is o f E d u c a tio n a l R e s e a rc h , Houghton
M if f lin Company, New Y o rk , 19h0.
~
Lockwood, H, R ., C o r r e la tio n o f th e M ental M a tu rity o f One Hundred C o lleg e
Freshm en and T h e ir A b i l i t y to W rite E n g lis h C o m p o sitio n , (M aster *s
T h e s i s ) , U n iv e r s ity o f C h icag o , 1925.
L o e v in g e r, J a n e , "A S y ste m a tic Approach to th e C o n s tru c tio n and E valu­
a t i o n o f T e s ts o f A b i l i t y , " P sy c h o lo g ic a l M onographs, v o l. 61, No. A,
19U7.
Lyman, R. L . , " I n v e s t ig a ti o n s i n th e F ie ld o f W r itte n C o m p o sitio n ,"
Summary o f I n v e s ti g a ti o n s R e la tin g to Grammar. L anguage, and Compo­
s i t i o n , Supplem entary E d u c a tio n a l Monographs No. 3 o 5 The U n iv e r s ity
o f C h icag o , 1929.
McNemar. Q u in n , P sy c h o lo g ic a l S t a t i s t i c s , John W iley and S o n s, New Y ork,
19A9.
M e r r i l l , Maud A ., "The S ig n if ic a n c e o f IQ*s on th e R ev ised S ta n f o rd B in e t S c a le s ," J o u rn a l o f E d u c a tio n a l P sy ch o lo g y , v o l . 29 (December
1938) , p p . d a -^ T .

r

103

M ow rer, 0 . H ., "L e a rn in g T h e o ry ," Review o f E d u c a tio n a l R e s e a rc h , v o l .
22 , No. 5 (December 1 9 5 2 ), p p . U75-95.
P r e s i d e n t 's Commission on H ig h er E d u c a tio n , " E s ta b lis h i n g th e G oals"
( v o l . l ) , H ig h er E d u c a tio n f o r A m erican.D em ocracy. U. S . Government
P r i n t i n g O f f i c e , W ash in g to n , D. C . , 19R7.
S a c k e t t , L ero y W ., "C om parable M easures o f C o m p o sitio n ," S ch o o l and
S o c i e t y , v o l . 5 (F e b ru a ry 2 U , 1 9 1 7 ) , p p . 2 3 3 - 3 9 .
S im s, V erner M ., "The E ssay E x am ination a s a P r o j e c t i v e T e c h n iq u e ,"
E d u c a tio n a l and P s y c h o lo g ic a l M easurem ent, v o l . 8 , 1 9 U 8 , p p . 1 9

-2 0 ,

, "The O b j e c t i v i t y , R e l i a b i l i t y , an d V a l i d i t y o f a n E ssay E x am in atio n
Graded b y R a tin g ," J o u rn a l o f E d u c a tio n a l R e s e a rc h . v o l . 2k (O c to b er
1 9 3 1 ), PP. 2 1 6 -2 3 .
, "R educing th e V a r i a b i l i t y o f E ssay E x am in atio n M arks Through E lim i­
n a tin g V a r ia tio n s i n S ta n d a rd o f G ra d in g ," J o u r n a l o f E d u c a tio n a l
R e se a rc h , v o l . 26 (May 1 9 3 3 ), P P . 637-ii7.
S t a l n a k e r , John N. "The E ssay Type E x am in atio n ," E d u c a tio n a l M easurem ent,
A m erican C o u n c il on E d u c a tio n , W ash in g to n , D. C . , 1 9 5 1 .
, "Q u e stio n XV, The E ss a y ," E n g lis h J o u rn a l (C o lle g e E d i t i o n ) , v o l .
26 (F e b ru a ry 1 9 3 7 ) , p p . 1 3 3 - ^ 0 7
S t a r r i n g , R o b e rt W ., "A S tu d y o f R a tin g s o f Com prehensive E x am ination
Themes When C e r t a in E lem en ts A re W eakened," (u n p u b lis h e d E d. D.
T h e s i s ) , M ich ig an S ta te C o lle g e , 1 9 5 2 , 1 0 9 numb, l e a v e s .
S tro u d , James B . , P sy ch o lo g y i n E d u c a tio n , Longmans, G reen e, and Company,
New Y ork, 19 h 6 .
T h o rn d ik e , E . L . , The E lem ents o f P sy c h o lo g y , The M ason-Henry P r e s s ,
• S y ra c u s e , New Y o rk , 1907.
:
, M e n tal Work and F a tig u e an d I n d iv id u a l D i f f e r e n c e s , T each ers
C o lle g e , Colum bia U n iv e r s it y , New Y o rk , 1 9 ^3 .
T h o rn d ik e , R . L . , " R e l i a b i l i t y , " E d u c a tio n a l M easurem ent. A m erican C o u n c il
on E d u c a tio n , W ashington, D . C . , 1 9 5 l.
T ra b u e , M. R . , "S up p lem en tin g th e H ille g a s S c a le ," T e a c h e rs C o lle g e
R e co rd , v o l . 1 8 , (J a n u a ry 1917) p p . 51-81*.
T r a x l e r , A . E . , and A n d erso n , A . A . , "The R e l i a b i l i t y o f a n E ssay T e s t
i n E n g lis h ," S ch o o l R eview , v o l . .1*3 (Septem ber 1 9 3 5 ), P P. 53U-39.

10k

T y le r , Ralph W ., "P urposes and P ro ced u res o f th e E v alu a tio n S t a f f ,"
A p p ra isin g and R ecording S tu d en t P r o g r e s s . H arper and B r o th e rs ,
New Y ork, 19h2.
V ir tu e , John B ., "The P ro f ic ie n c y E xam ination in E n g lish C om position a t
th e U n iv e rs ity o f K ansas," C o lleg e E n g lis h , v o l. 9 (Ja n u a ry 19^8) ,
p p . 199-203.

APPENDICES

APPENDIX A

THEME HATING SCALE AND DEFINITIONS OF SEPARATE CATEGORIES

S u p e rio r
10
9

b

7

6

£

U n s a tis fa c to ry
G 3
2
1

C ontent
C onventions o f Grammar
S entence S tru c tu r e
D ic tio n
O rg a n iz a tio n

C ontent
C ontent r e f e r s to th e q u a lity and adequacy o f th e s u b s ta n tia tin g ma­
t e r i a l (ex am p les, s t a t i s t i c s , argum ents) employed i n su p p o rt o f id e a s
ex p ressed i n th e p a p e r.

A theme o f c o lle g e c a lib e r sh o u ld concern i t s e l f

w ith m a tte r w orthy o f a d u lt c o n s id e ra tio n and ex p ress a re a s o n a b ly m ature
p o in t o f view .
C onventions o f Grammar
C onventions o f grammar r e f e r s to such m a tte rs as re a s o n a b le s p e l l i n g ,
c o r r e c t p u n c tu a tio n a t m ajor ju n c tu r e s , th e u s u a l gram m atical agreem ents

106

( s u b je c t- v e r b , p ro n o u n -a n te c e d e n t), and th e c o r r e c t use o f posse s s i v e s .
I t r e f e r s a ls o to th e avoidance o f se n ten c e fra g m e n ts, comma f a u l t s ,
p e rio d f a u l t s , and d a n g lin g m o d if ie r s .
Sentence S tru c tu r e
E f f e c tiv e se n te n c e c o n s tru c tio n means th e s t r a t e g i c use o f such
th in g s a s th e p e r io d ic s e n te n c e , s u b o rd in a tio n , and p a r a lle lis m .

It

means t h a t by a v a r i e t y in se n ten c e le n g th , i n sen ten ce s t r u c t u r e , and
in sen ten ce o r d e r , monotony and c h ild is h n e s s o f e x p re ssio n may be avoided
and v a r ie ty and m a tu rity o f e x p re s s io n a c h ie v e d .
wliich a re f r e e from awkwardness ana o b s c u r ity .

I t means se n te n c e s
I t means t h a t s u c c e s s fu l

a t t e n t i o n has been g iven to th e req u irem e n ts o f sen ten ce euphony and
rhytlim .

>

D ic tio n
Good d ic tio n means th e u se o f words w e ll chosen to ex p ress th e w rite r*
m eaning.

I t means th e avoidance o f e x p re s sio n s which a re crude o r t r i t e ,

o f w o rd in e ss, o f pom pousness.

I t means th e use o f ac ce p ted id io m s, o f ex­

p r e s s io n s which a r e vigorous and a l i v e , o f th e s p e c if ic and c o n c re te in
p re fe re n c e to th e g en e ra l and a b s t r a c t .
O rg a n iz a tio n
The s iz e o f th e to p ic should f i t th e le n g th o f th e p a p e r.

The theme

as a whole should have a s i n g l e , c o n tr o llin g id e a o r p u rp o s e , ex p ressed
or c l e a r ly im p lie d , t o which each p a r t o f th e theme c o n 'b rib u te s.

Each

paragraph sh o u ld be re c o g n iz a b le as a u n it ( i . e . d eveloping a s in g le to p ic
o r s u b -to p ic ) in th e development o f th e them e.

The id e as p re s e n te d should

be sm oothly and l o g i c a l l y lin k e d to g e t h e r .

Such lin k in g i s achieved by

a re c o g n iz a b le p a t te r n o f developm ent and by th e use o f such t r a n s i t i o n a l
d e v ic e s a s th e c o n n e c tiv e , p a r a l l e l i s m , pronoun r e f e r e n c e , and r e p e t i ­
tio n .

By th e use o f such th in g s a s p o s i t i o n , p r o p o r tio n , and r e p e t i t i o n ,

t h a t wiiich i s of most im portance in th e theme sh o u ld be made to seem so
to i t s r e a d e r .*

* W ritte n and Spoken E n g lish S y lla b u s , M ichigan S ta te C o lleg e P r e s s ,
E a s t L a n sin g , 19U9, p p . 1 7 - lo .

108

'

APPENDIX B

ESTIMATION OF THE RELIABILITY FOR INCOMPLETE
SETS OP RATINGS*
Whether o r n o t i t i s d e s ir a b le to remove " b e tw e e n -ra te rs " v a ria n c e
in e s tim a tin g th e r e l i a b i l i t y o f r a t i n g s depends upon th e way in which
th e r a tin g s a r e u ltim a te ly used in g ra d in g , c l a s s i f i c a t i o n , o r s e le c tio n .
In any case where d if f e r e n c e s from r a t e r to r a t e r in g en e ra l le v e l of
r a t i n g do n o t le a d to co rresp o n d in g d if f e r e n c e s in th e u ltim a te g ra d e s ,
c l a s s i f i c a t i o n s , o r s e l e c t i o n s , th e " b e tw e e n -ra te rs" v a ria n c e should be
removed from th e e r r o r te rm .

S p e c i f i c a l l y , th e " b e tw e e n -ra te rs " v a r i ­

ance should be removed where th e f i n a l r a tin g s on which d e c is io n s a re
based c o n s is t o f av erag es o f com plete s e ts o f r a t i n g s from a l l o b s e rv e rs ,
o r r a tin g s w hich have been equated from r a t e r to r a t e r such as r a n k s ,
Z -s c o re s , e t c .

L ik ew ise, i f com parisons a re nev er made p r a c t i c a l l y , b u t

on ly e x p e rim e n ta lly , betw een r a tin g s o f p u p ils by d i f f e r e n t r a t e r s , th e
" b e tw e e n -ra te rs " v a ria n c e sh o u ld be removed.

But i f d e c is io n s a re made

in p r a c tic e by com paring s in g le "raw11 sc o re s a ssig n e d to d i f f e r e n t p u p ils
by d i f f e r e n t r a t e r s , o r by comparing av erag es which comes from d i f f e r e n t
groups o f r a t e r s , then th e " b e tw e e n -ra te rs" v a ria n c e should be in c lu d ed
as p a r t o f th e e r r o r te rm s .

* R obert L . E b e l, “E stim a tio n o f th e R e l i a b i l i t y o f R a tin g s ," P s y c h o m e t r i k a ,
v o l. 1 6 , No. h (December 1 9 5 1 ), PP. U11-A12.

109

APPENDIX B - C o n tin u e d

TABLE 2
ANALYSIS OF RATINGS FOR PROBLEM 5 — INCOMPLETE SETS

R atin g s
P u p il 1
P u p il 2
P u p il 3

k

6 6 h U3
o 9 9 li 9 6 5 10 6

5
9
3
17

Sums
Sum o f squared r a tin g s
Sum o f p ro d u c ts (p u p il sum tim es p u p il mean)
P ro d u ct o f sum and mean
Sum o f sq u ares
For t o t a l
656 — 76L.L706
F or p u p ils
765.3333 — 76U.JU706
F or e r r o r
9 3 .529ii — 20.6627
Kean square
F or p u p ils
20.6627 + 2
For e r r o r
72.6667 + li;
*
Average v alu e of k
10.U31U — 5.1905
R e lia b ility
10.1i3lii + (U. 1176) (5.1905) =

Sum
25
66
23
HL
858
785.3333
76ii.L706
93.529ii
20.6627
72.6667
10 .ii3lU
5.1905
.I6ii8

T able 2 i l l u s t r a t e s a p p lic a tio n of t h i s form ula to a sim p le problem
in wiiich th e ta b le of r a tin g s i s incom plete and th e so u rces o f r a tin g s
a re n o t i d e n t i f i e d .

In t h i s case o n ly two components o f th e v a ria n c e ,

a t t r i b u t a b l e to p u p ils and e r r o r , a re s e p a r a te d .

Thus any d if fe re n c e in

g e n e ra l l e v e l o f r a t i n g s between th e v a rio u s r a t e r s i s a u to m a tic a lly in ­
cluded in th e e r r o r te rm .

'.S;

110

APPENDIX C

DIFFERENCES BETWEEN PAIRS OF PAPERS
\

( S i g n i f i c a n t d i f f e r e n c e s a r e u n d e r l i n e d . N o ta tio n s e q u a l: T1 - T o p ic 1 ,
T2 - T o p ic 2 j D1 - F i r s t day f D2 - Second day • W/0 - W ith o u t p r e s s u r e j
W - W ith p r e s s u r e . S i g n i f i c a n t g a in s an d l o s s e s fro m f i r s t day t o seco n d
d ay a r e i n d i c a t e d b y p l u s a n d m inus m a rk s .)

________________________________
Group_A______________________________
S iM
T u
____________ T1_________________T2
D1 W/0 ________ D2 W/0
U|
D1
D2
D1
D2
T1 - ' T2
T1 T2
E E

W/0

W/0

W/0

W/0_________________________________

NK
T

2

1 2 .0 +

lU .l -

k .l

2 2.0

3

6 .5
lii.o 3 .0
2 .0
1 .0
6 .5
5 .5
1 .0
6 .0
1 .5
6 .0
2 .7

8 .5

U .5

2 .5

5 .0
3 .0
2 .0
5 .5
8 .0
5 .0
3 .0
o .o
11.0
1 0.7
1 0.5
5 .0
1 .5
0 .0
1 .5
1 .5
2 2.5
5 .0

3 .5
1 .0
0 .0
0 .0
1 8.0
1 .0
2 .0
o .5
5 .5
2 .7
8 .0
7 .0

1 5.5
5 .0
0 .0

k

5
6
8
9
10
11
12
15
16
17
18
19
21
22
23
2h

25

h.$

1 1.5 k.o

1 3 .0 +
3 .0
1 3 .0 6 .5

+
-

k.O

1 .5
2 .0
5 .5
6 .0

k.S

3 .5
0 .5
k.o

5 .5
U .o
lii.O

0 .2
7 .5
6 .0
2 .5
1 6.5
1 .0
0 .0
5 .5

I ll

APPENDIX C - C o n tin u e d

Group C
ro

1

s; o
c>

T1
D1

w /o

T2
-

D2

y

D1 W/0
T1
T2

D2 W,
T1
T2

26

1 .0

2 .5

5 .o

3 .5

27

e.o

2 .0

2 .5

3 .5

26

5 .0

1 .5

2 .0

1 .5

29

6 .0

2 .0

7 .5

3 .5

30

2 .0

7 .0

6 .0

1 .0

31

1 3 .5 +

2 .5

h.S

U .5

32

20.5 +

9 .0

7 .5

22.0

33

0 .2

3 .5

8.3

5 .0

35

1 .0

2 .5

7 .5

U .o

36

6 .0

2 .5

5 .5

0 .0

37

2 .5

6 .0

9 .0

1 2.5

'36

3 .0

3 .0

0 .0

6 .0

39

2 .5

1 .5

2 .0

6 .0

hO

1 1 .5 -

7 .0

0 .5

3 .0

ui

U.5

7 .5

1 2 .0

0 .0

U2

6 .5

0 .5

3 .0

.9 .0

5 .0

1 .0

1 1 .0

28.0

5.9

1 .0

0 .5

3 .0

2 .5

6 .0

8.0

U3

1 5 .0 +

hh

1 6 .h

U5

2 .0

h6

-

u .5 +

1 7 .5 +

112

APPENDIX C - C o n t in u e d

Croup B

Dl

T1
D2

____ Wo

yt

T3

Dl W /0

D2
W

D l W /0
T1 T2

D2 W/0
T1 « T2

he

3 .0

0 .5

1 3.0

U9

2 .5

7 .0

6 .5

9 .5
1 1 .0

51

0 .0

U .o

5 .5

1 .5

52

b.5

6 .5 -

5 .5

9 .5

5U

6 .8

5 .o

U .5

2 .7

55

1 9.5 -

U .o

2 1 .5

2 .0

56

0 .5

5 .0

5 .5

1 .0

57

2 .5

0 .0

8 .0

5 .5

56

6 .5

U .5

1 1 .5

7 .5

59

2 .5

1 2 .5

3 .5

61

10.2 -

3 .5

lU .o

7 .3

62

5 .5

1 .5

1 .5

U .5

63

2 .5

7 .5 +

7 .0

2 .0

61*

0 .0

0 .7

3 .7

3 .0

1 .0

1 3 .0

U .o

65

1 0 .0 +

'

11.5 +

67

ii.o

9 .5 +

6 .0

66

9 .0 -

2 .5

8 .5

7 .5
2 .0

69

1 3 .0 +

9 .5 0 .0

6 .0

1 6.5

6 .5

5 .5

-

0 .5

U .o

70

1 .0

71

7 .0

-1 0 .5

113

APPENDIX C - C o n tin u e d

Group D
T1
Dl
W/0

D l W/0

T3
D2
W

Dl
W/O

D2
W

T1

T3

T1

D2 W
- T3

73

U.O

U .o

1.0

1 .0

7U

1.0

6.0

9 .0

U .o

75

1 Q .5

0.0

6.0

1 6.5

76

10.0

8 .5 -

o .5

1 8 .0

77

9 .0

0 .5

U .5

U .o

79

9 .5 -

0 .5

1 1.5

2 .5

80

2.0

1 .5

1 .5

2.0

81

U .o

1 3.5

u .5

2.0

83

1 3 .0

8U

1 5.0

6.0 -

o.O

3 .0

0.0

2.0

1.0

86

0 .5

15.0

0 .5

1 5 .Q

87

11.0

1 Q .5

2.0

1 .5

88

1 Q .5 -

8.0

8.0

1 Q .5

89

2 .5

0 .5

1 1 .5

iU .5

90

U .5

1 5.5

1 9.0

1 .0

92

6 .5

3 .5

2 .5

7 .5

93

9 .5 -

2.0

:i . 5

9 .0

9b

8.0

7 .5 -

5 .0

1 Q .5

95

5 .0

11.0

0 .5

1 5 .5

-

-

lilt

APPENDIX D
FREQUENCY OS' SIGNIFICANT VARIATIONS FOR STRONG AND WEA.K STUDENTS
N o te:

T1 - T o p ic 1 , T2 - T o p ic 2 , T3 - T o p ic 3 j
Dl - 1 s t D ay, D2 - 2nd Dayj
D1-D2 - D if f e r e n c e b etw een p a p e r s on t o p i c i n d i c a t e d
f o r d i f f e r e n t days
T l- T 2 ,3 - D if f e r e n c e b etw e en p a p e rs on d i f f e r e n t
t o p i c s f o r d ay s i n d i c a t e d
Top 1+0
Top 2 0 , i n d e s c e n d in g o r d e r ,
p r e s e n te d on t h i s p a g e .

Low 2 0 , i n a s c e n d in g o r d e r ,
p r e s e n te d on t h i s p a g e .

Second 20 o f to p hO c o n tin u e d
on n e x t p a g e .

Second 20 o f
on n e x t p ag e

-p
? £-1
3 'i T 1
£ ID1-D2
56
83
22
55
6U
It
88
kk
1|2
1+8
32
1+3
58
61
75
9
37
69
2
10

1 3 .0
1 3 .0
1 9 .5

ABOVE MED.
H igh
20
D
l
T 2 ,3
D1-D2
T1-T2 ,3
8 .0
—
-

llt.O
1 0 .5
l6 .lt
2 0 .5
1 5 .0
1 0 .2
1 0 .5
1 3 .0
1 2 .0

Low ItO

-

6 .0
1 7 .5
—
—
—
—
-

9 .5
llt.l
—

2 1 .5
-

2 8 .0
1 3 .0
U .5
llt.O
1 8 .0
9 .0
—
-

•p
_ 0) <
D
D2
H-g T1
T1-T2 ,3 &§D1-D2
1 5 .0
1 6 .5
-

1 5 .5
1 0 .5
-

9 .0
9 .5
2 2 .0
1 1 .0
1 6 .5
1 2 .5
1 6 .5
2 2 .0

73
25
63
8057
62.
30
26'
21
7U
11
81+
86
52
92
3
5
68
6
1+9

—
-

lo w hO c o n tin u e d

BELOW MED.
Low
20
D2
Dl
T 2 ,3
D1-D2
T1-T2 ,3
T1-T2 ,3
—
7 .5

mm

mm

-

-

-

—

-

-

-

—

-

-

mm

-

-

-

—

-

—

-

-

-

—
—
—
1 5 .0
8 .5
-

mm

9 .0
-

—
-

-

—
-

1 5 .0
9 .5
mm

-

-

-

-

-

-

9 .0

-

-

-

-

-

-

-

-

-

1 1 .0

115

APPENDIX D - C o n tin u e d

Top AO (C o n t'd .)
Secbnd
20
o f to p AO i n descend­
"S
u
C
O
0
)
in
g
o
r
d
e
r
.
TJ 0
12 g T1
Dl
D2
T2,3
-PO^5
P
C
D1-D2 D1-D2
T1-T2 ,3
T1-T2 ,3
16
17
96
2ii
27
71
33
67
15
65
76
93
19
AO
29
A5
59
9u
Ai
31

.

10.7

mm

—

1 0 .5

-

-

-

13.5

-

9 .5

13.0

-

-

1 0 .5

-

-

11.0
-

10.0
10.0
9.5
u .5

1 0 .5
1 1 .0
-

9 .5

-

-

-

-

-

-

-

-

-

-

-

13.0

8 .5

-

-

-

-

-

-

18.0
9.0
-

-

-

-

-

-

-

-

-

-

1 1 .5

u .5
7 .5

-

-

-

13.5

Notes

2 2 .5

1 A.0

—

-

12.5
-

1 2 .0
•

-

1 0 .5
-

Low AO (C ont*d.)
^Second 20 o f low AO i n ascending
$ b o rd e r.
p gi T i
D2
Dl
T2,3
-p
3
coaDl-D2 D1-D2
T 1-T2,3
T1-T2 ,3
51
81
22 1 3 .0
28
70
9 .0
77
95
12
35
—
5U
18
79
9 .5
36
33
A6 u . 5
90
6
39
—
67
—
89

w —

1 3 .5

1 1.5

-

16.5

-

-

-

-

-

-

-

-

-

-

1 5.5

1 1 .0

-

-

-

-

-

-

-

-

-

-

-

-

-

-

11.5

-

-

-

-

-

-

-

-

-

-

15.5

1 9 .0

-

-

-

-

-

-

-

9 .5

-

-

u .5

114.5

S tu d en ts numbered from 2 to 25 and from A8 to 71 w rote on both
days w ith o u t th e exam ination p r e s s u r e . S tu d en ts numbered from 26
to A6 and from 73 t o 96 w rote on second day w ith t h a t p re ssu re
add ed . S tu d en ts numbered from 2 to A6 w rote on s im ila r to p ic s ;
w h ile s tu d e n ts numbered from A8 to 96 w rote on d is s im il a r t o p i c s .

116

APPENDIX E

LETTER-GRADE VARIATIONS
The mean and s ta n d a rd d e v ia tio n o f th e 320 s c o re s were used to form
a d i s t r i b u t i o n (a p p ro x im a te ly normal) o f l e t t e r g rad es from A to F f o r
a l l o f th e p a p e rs .

Then a check was made to d eterm in e how many l e t t e r -

grade v a r ia tio n s may iiave been due to e r r o r i n r a t i n g s and how many were
due to s i g n i f i c a n t v a r ia tio n s i n th e q u a l i t y o f w r i t i n g .
Most o f th e v a r ia tio n s o f one l e t t e r grade may have b een due to
e r r o r i n r a t i n g s , w h ile m ost o f th e v a r ia tio n s o f two o r more l e t t e r grades
p ro b a b ly were due to s i g n i f i c a n t v a r ia tio n s i n th e q u a l ity o f w r itin g —a s
shown below :
V a ria tio n s o f one l e t t e r g ra d e , w hich may have been due
o n ly to e r r o r i n th e r a t i n g s ...................................................................... 105
V a r ia tio n s o f one l e t t e r g ra d e , w hich p ro b a b ly were n o t
due o n ly to e r r o r i n r a t i n g s b u t w hich o cc u rred a s a r e ­
s u l t o f s i g n i f i c a n t v a r ia tio n s i n th e q u a l i t y o f w r i t i n g . . . .

33

V a ria tio n s o f two l e t t e r grades w hich may have been due
t o e r r o r i n r a t i n g s .......................................................................................

5

V a r ia tio n s o f two or more l e t t e r grad es w hich p ro b a b ly were
n o t due o n ly to e r r o r i n r a tin g s b u t w hich o c c u rre d a s a r e ­
s u l t o f s i g n i f i c a n t v a r ia tio n s in th e q u a l i t y o f w r i t i n g . . . .

50