.
.
.
H ,
. ..\ I .
.
. . - l. . n . .
n J n | . I n .
I . V O I
. . I ‘ .
. . -- .
.. . .. .
. q . n _ .
‘ . .. . I u. . .c .
. , ,.I - . . .
9— | Q I a
. u -
~ 0 .y I I I n
1 V x 1. ‘
U 0 I W 7 n
. . . u .. u . .
.. . 5 IA. .. ‘ 1 . A
,. x . .i. .. . ‘ . .
5 .fi. - n Y . i
‘ v . f . .. r.
u n ‘I . I : 4 n
v a 4 2.
t - I l I I "‘ t x l I. . .
, . l . .
.. . 1 . . . ...
i .. .
.I ‘
I .7 . . I. . I .
. v. .
x I. I I I
x I. . ‘ .
-H . - . . ,
\ It _ | A . I
It ‘ s ‘. . , n I .
r . . . _-
x, . n . . '
Hr . v 4 1 ..
Visa. . A t .
_. a. _
\ . .
I O I w u '
F ‘ . .
- . . . ‘ .
. .a .
_ . A .
n. _ ~
‘ . .
‘ .
.c‘v ,~ \ I u ‘ I
- . \ . .
to r n 1| . I
‘ - . . . . .
l .mv‘ ' n I D .
I . .
- .z
-n. - - . . . . . ..
I \l I x . I ~ I
w ‘ w x . - . . . .. .
A 1 hi . v . .
. . I l . . .; r
‘ v. . .. n . .I . w y
. . I . . Q. . . I . I. . I
I -0 A w .1 ..v . n .
. u. ‘ ,
A w ‘ I ‘4‘ I .
. .. I . I ..
I.” . . . 0 ,. ..> . H I v ‘ . I v , . ‘ . . EVIV‘IIIIIIQIRI
.4 a. .I .. «I» u 4 I v , y ‘ 9...) 5.1.0.3?-
. :J... . y - l r. ‘ . ‘ H . L . . 4hr 5.0.11.0.
' U sv» I b .» .I. | ‘ , , , . 2.2Iv II 1" I'lv
. ) V I l. I ‘ I I I I . ‘ , Vy .
i . I ‘ . ‘
l - I p r A ‘5 ‘ I . w
| , . r . , ., . .f v .0 . .. ..u‘ n.. U , ~ I: 7 I 1 .colrlxd‘vfir) J I} . 3‘73 c'i,’ | I!
"-1. ‘ .. .Voll ‘ I..xu....:..|.u. 1...}...l H .54.. n.4|!:..i..th 71%} at” .1... _ .I 1.. e y .i r.‘ .4 1. -I . . .. . ... s a A». 1...: :3. Ir... a".
m .I2 . ‘.. .
I

   

 

|||"‘|||

IIIIIIIIIIIIIIIIIIIIIIIIIIIIIII

300579 1581

This is to certify that the

dissertation entitled

LEARNING LEGAL REASDNING IN LAW SCHOOL
THE DIFFERENCES BETWEEN FIRST AND THIRD YEAR STUDENTS

presented by

Charles John Senger

has been accepted towards fulfillment
of the requirements for

Ph.D. degree in_Edu.C.a1'.iQLL

M/%
y MaJOYproy

 

 

Date jflf/f/

MS U is an Affirmative Action/Equal Opportunity Institution 0-12771

 

LIERARY
Michigan State
University

 

 

 

 

PLACE IN RETURN BOX to remove this checkout from your record.
To AVOID FINES return on or before date due.

r fl 4:

DATE DUE DATE DUE DATE DUE

 

MAY 1 0 157?}

it; >an 2
2; —~
1 .

f a

m
N
W
323,

 

 

 

 

 

 

 

I!

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

MSU Is An Afiirmmive ActioNEqual Opportunity Institution

 

 

LEARNING LEGAL REASONING IN LAW SCHOOL:
THE DIFFERENCES BETWEEN FIRST AND THIRD YEAR STUDENTS

By

Charles John Senger

A DISSERTATION
Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of
DOCTOR OF PHILOSOPHY
Department of Counseling, Educational Psychology and Special Education

1989

ABSTRACT

LEARNING LEGAL REASONING IN LAW SCHOOL: THE DIFFERENCES
BETWEEN FIRST AND THIRD YEAR STUDENTS

By

Charles John Senger

The purpose of this study was to use new methods to compare the legal reasoning of
students who had completed almost one year of traditional American legal education
with students who had completed almost three years. Traditional legal education was
defined as giving a central role to the use of the case method of teaching with classroom
discussion of appellate court opinions. Legal reasoning was defined as reasoning as
used in the casebook method, including the idea of reasoning from example to example
using analogies based upon previously decided cases. Data was collected from volun-

teers who both wrote answers to essay questions and thought out loud as they worked.

Analysis of both the written answers and the transcribed protocols of the thinking aloud
comments revealed, contrary to expectations, no support for the hypothesis that the
third year students’ legal reasoning, as defined, would be superior. Three areas were
advanced as sources of possible explanations for the finding of no significant differences
between the first and third year students’ reasoning. Those areas were: the group
selection procedures used, particularly the reliance upon volunteers; the definition of

legal reasoning used since it might not have captured what the law students were

learning; and the design of this study, including motivational and scaling issues, possible
interference between the processes of writing essay answers and thinking aloud,
possible triggering by the use of essay questions of a specialized response different from
the legal reasoning being learned, and the possible presence of a nonmonotonic
learning process with the result that the third year students were measured too early in

their careers to reveal any differences.

The researcher’s conclusion is that the most likely explanations for the finding of no
difference are those explanations associated with the design of this study. However, this
study also develops data and findings which will be valuable for future research and

educational decision making.

Copyright © 1989 by Charles John Senger.

All rights reserved.

 

Dedicated to
my wife Bettie
and our children
Paul, John, and Carolyn

 

ACKNOWLEDGMENTS

Although these brief acknowledgements cannot name everyone who has assisted me
in this research, I nevertheless wish to extend thanks to my family, colleagues, and
committee members. A special thanks also is owing to the participants in this study.

Dr. Beverly Hunt introduced me to both Dr. Stephen Yelon and to the connection
between educational psychology and legal education. In the early stages of this project,
she provided the inspiration and a listening car when I needed it most. Likewise my
brother, Philip Senger, encouraged me at all stages of this program and, in addition,

introduced me to the work of Dr. David Schon.

Virginia Allen, Daniel Hearn, Steven Ramey, and Phillip Stevens are among the
attorneys who contributed ideas to this project. John Michaud provided library refer-
ence help. likewise my colleagues often provided assistance, especially Professors

Peter Kempel, Daniel McNeal, Roger Needham, John Nussbaumer, and John Rooney.

At Michigan State University, Dr. Lawrence Lezotte guided me through my master’s
program. Dr. Lee Shulman shared his insights regarding professional education and
research approaches to it. Likewise at critical stages of my program, important contri-
butions were made by Dr. Lawrence Alexander, Dr. Christopher Clark, the late Dr.
Robert Ebel, Dr. Frederick Erickson, Dr. Susan Florio, and Dr. Andrew Porter.

The members of my committee contributed extensively both in the classroom and in
personal consultations. Dr. Carol Scotton served on my committee while she was at
Michigan State University. She taught me much about linguistics and the law. Dr.

Richard Houang took Dr. Scotton’s place on the committee and helped me put into

vi

practice what he had taught me about statistics and research design. Dr. Douglas
Campbell guided me through earlier field work research projects but this particular
study reflects only a small part of the methodology that he shared. Dr. Walter
Hapkiewia helped me throughout my program and chaired my Guidance Committee.
Very early on he helped me to see how research on topics like Bobo dolls and
kindergarten classrooms could have important connections to questions about legal

education.

Dr. Stephen Yelon, my Dissertation Director, deserves thanks that cannot be put into
words. My previous legal education and experience frequently has made it difficult for
me to consider matters from a psychological perspective. Dr. Yelon repeatedly helped
me to find bridges between what I had learned and what I was attempting to learn. Even
more important has been the motivation supplied by his example because he shows, in

his own teaching, that it is possible to put good education theory into practice.

Finally, my wife Bettie, and my children Paul, John, and Carolyn, helped me with the
many aspects of our lives that were affected by my education and research program.

The thanks owed them can only be understood by those who have been through similar

circumstances.

vii

I _

 

 

 

Table of Contents

CHAPTER I - INTRODUCTION .......................................... 1

The need for the study of legal education. ........................ l

The research question: ........................................ 3

Key terms: definitions and background information. ............... 4

Legal reasoning: ....................................... 4

Traditional legal education: ............................. 4

One year of law school: ................................. 4

Background information: ................................ 4

Plan for this study. ............................................ 6
CHAPTER H - REVIEW OF THE LITERATURE ......................... 10
Review of law school research. ...................................... 10
Bryden’s study. .............................................. 11

Essay questions as data. ...................................... 13

Legal reasoning and its coding. ................................ 25
General literature on thinking, reasoning, and problem solving. .......... 33
Thinking aloud protocols as data. .............................. 35
Summary. ......................................................... 37
CHAPTER III - PROCEDURE ........................................... 39
Participant selection procedures ...................................... 40
Population. ................................................. 4O
Informed Consent. ........................................... 42
Sampling .................................................... 43

viii

Data collection. .................................................... 44

Data Analysis. ..................................................... 46
First stage data analysis. . . . .................................. 48
Second stage data analysis. .................................... 51

CHAPTER IV - FINDINGS .............................................. 54

Introduction. ...................................................... 54

Written essay scores. ............................................... 55
Group difference findings. .................................... 61
Data reliability and significance findings. ....................... 61

Scorer reliability indices. ............................... 62
Statistical tests of significance. .......................... 62

Thinking aloud data. ............................................... 64

Typed protocols. ............................................ 64
Researcher’s examination of the protocols. ............... 64

Computer searches of the protocols. ..................... 70

Researcher’s notes. .......................................... 71
Evidence of Levi’s reasoning by analogy. ................. 72

Essay question validity .................................. 73

Uncontrolled factors. .................................. 73

Group background data. ............................................ 74
Summary. ......................................................... 76
CHAPTER V - DISCUSSION AND CONCLUSIONS ....................... 77
Discussion ........................................................ 77

Introduction. ................................................ 77

Group selection procedures as an explanation. .................. 79

The definition of legal reasoning as an explanation. .............. 80

Research design as an explanation .............................. 82

Summary and discussion. ..................................... 90

Conclusions ....................................................... 93

LIST OF REFERENCES ................................................. 96
APPENDICES ..........................................................

Appendix A - Informed Consent Form ............................... 107

Appendix B - Invitation to volunteer to participate .................... 109

Appendix C - Response letter to volunteers .......................... 110

Appendix D - Protocol questions .................................... 1 11

Appendix E - Warm up instructions ................................. 1 16

Appendix F - Participant’s legal experience ........................... 117

Appendix G - Classification Guide .................................. 118

Appendix H - Practice question and answers .......................... 122

Appendix I - Pearson product moment correlations .................... 137

 

Table 4.1
Table 4.2
Table 4.3
Table 4.4

Table 4.5

LIST OF TABLES

Findings Summary ............................................. 54
Scorer Reliability Measures ..................................... 62
Essay Score t Tests ............................................ 63
Group Background Data ........................................ 75
Background Questionnaire Data ................................. 75

Figure 4.1
Figure 4.2
Figure 4.3
Figure 4.4

Figure 4.5

LIST OF FIGURES

Group Mean Essay Scores ...................................... 56
LL Scores .................................................... 57
FF Scores .................................................... 58
SNS Scores ................................................... 59
TANK Scores ................................................. 60

CHAPTER I
INTRODUCTION

Law school has been described as a place for the accumulation of
learning. First-year students bring some in; third-year students take none
away. Hence it accumulates (White, 1983, p. 23).

The purpose of this study is to probe what students learn in law school, particularly the
avowed goal of teaching students legal reasoning or, as it is often phrased, "to think like
a lawyer." The effectiveness of the three year legal education program will be tested by
comparing the performance of first and third year law students. The task will be essay
questions of the type used throughout the three years of law school. To evaluate
performance on the essay questions, four types of data will be collected: numerical
scores of the essay answers; thinking aloud protocols made while the students write
their answers; notes taken by the researcher while observing the students writing; and
background information on the participants. The multiple types of data will allow
comparisons to be made which have not been attempted in previous studies. If the
groups of students differ, as it seems they should, then this study will have a better

chance than previous studies to document the nature of those differences.

The need for the study of legal education.
The effectiveness of legal education is important for many of the same reasons that

have led to the intensive study of medical education’s effectiveness. Both programs
require the investment of substantial amounts of time and money; law school is a three
year, graduate program. Both programs serve as gatekeepers controlling entry into the
respective professions; completion of an accredited law school program is a prerequi-

site to taking the bar examination in most states.

In some ways, it is even more important to study legal education than medical education
because the substance of what is taught in law school has been more often challenged.
An early example is when King James, acting on the advice of Bancroft, Archbishop of
Canterbury, said that the law was founded upon reason and therefore the King, using

reason, could decide cases without judges. Lord Coke answered:

That true it was, that God had endowed His Majesty with excellent
science, and great endowments of nature; but His Majesty was not
learned in the laws of his realm of England, and causes which concern
the life, or inheritance, or goods, or fortunes of his subjects, are .not to
be decided by natural reason but by the artificial reason and judgment
of law, which law is an act which requires long study and experience,
before that a man can attain to the cognizance of it.

As will be noted in the review of the literature, three centuries later there are still those
who would argue that legal reasoning is merely common sense or a specialized knowl-
edge base and that three years of special law school education are not necessary. These
challenges take on added urgency when one considers the progress of work on artificial

intelligence and expert systems (e.g., Clancey & Shortliffe, 1984; Gardner, 1987).

In addition to questions about what is taught in law school, the teaching methods also
have been questioned. The questions persist even though legal education has used the
case method for teaching since the 1870’s (Stevens, 1983). Stevens traces the political
and educational arguments over traditional legal education and shows that, even very
soon after the 1870’s, questions were raised about whether law school education was
appropriate for students other than the particularly quick or talented (1983, p. 118).

Likewise questions have been raised about whether legal education is good preparation

1 Prohibitions Del Roy, 12 Co. Rep. 63, 65; 77 Eng. Rep. 1342, 1343 (1608).

l—""

for persons who go into business (Slade, 1989) or even the courtroom (Burger urges
curb, 1973). Although these questions about business or courtroom skills may appear
to address separate areas of concern, it has been recognized that they have direct
implications for law school programs (American Bar Association, 1979). They also have
importance both for the students and the society which they will serve (Auerbach, 1976;
Carrington, 1984). As Paul (1988) has noted, many students come to law school with
virtually no training in complex problem solving and, if they do not receive it in law

school, then law schools are ”guilty of profound failure" (1988, p. 925).

Despite the parallels to medical education, and despite the seriousness of the questions
raised, legal education has been subjected to far less empirical study than, for example,
medical schools. The review of the literature will show that much has been written but

most of it is based upon anecdotal data gathered from experience teaching in law school.

Unlike many previous works, this study will gather several forms of data and use both
qualitative and quantitative methods of analysis. Even if Lord Coke was right, even if
legal reasoning is something special acquired over a long period of time, this study’s
multiple data collection and analysis methods offer hope of isolating at least the
contribution made by the last two years of the three year law school program. Isolating
the difference between the legal reasoning of first and third year law students might
then contribute to the resolution of the controversies about both the nature of legal

reasoning and the effectiveness of legal education.

The research questlon:
How does the legal reasoning of law students who have completed almost one year of

traditional American legal education compare to that of law students who have com-

pleted almost three years?

 

I

 

Key terms: definitions and background Informatlon.

Legal reasoning:
Legal reasoning is the set of cognitive skills which traditional legal education empha-

sizes. In this study, the definition of that set of cognitive skills will follow the work of
Levi (1949), including the idea that legal reasoning is primarily reasoning from example

to example using analogies based upon previously decided cases.

Tradltlonal legal educatlon:
Traditional legal education means law school teaching which follows the casebook

model. The model teaches legal problem solving through application of legal reasoning
to the facts and law of the problems. The model emphasizes the use of: sequences of
appellate opinions illustrating the facts, law, and legal reasoning of prior legal prob-
lems; classroom questioning by the professor focusing on applying what is learned from
the appellate opinions to new sets of facts; and essay examinations testing what is

learned again focusing on application to new sets of facts.

One year at law school:
One year of law school means that thirty semester credit hours have been completed

(typically ninety credits over three years for a total law school program).

Background lntormatlon:
The preceding definitions, particularly the definition of "traditional legal education,"

show that this study will primarily address the teaching of legal reasoning via the
casebook method. The origin and development of the casebook method will be
presented in Chapter II as part of the review of the literature. However, to understand
the plan of this study, some background information is appropriate in this introduction.

Reading the preceding definition of "traditional legal education" does not necessarily
tell one very much about the casebook teaching method being described. One can

 

 

envision a professor asking a student to give the facts of a case, to summarize what the
court did, and to explain the reasons with comparisons and contrasts with other cases.
However, even that exchange between the professor and student can take many forms
as is illustrated, for example, by Turow’s (1977) descriptions of his first year law classes.
Despite the possible variations, the gist of the method was perhaps best expressed by
Harry W. Jones (cited in Kelso, 1972), an acknowledged master teacher, when he

explained how he used sequences of cases:

1. A general sequence for class learning: encourage students with respect
to what they know and have learned; create doubt on some matter; then
resolve that doubt or break it into the ingredients of an answer.

2. One very helpful device for creating and resolving doubt is to state
hypotheticals or give professional skill variations such as challenging the
students to draft a document that will restructure a transaction to change
the way in which law applies to it (p. 608).

Jones’ explanation also helps show why the case method of teaching is sometimes called
"Socratic." Although working with larger numbers than Socrates did, the law professor
also uses questions to encourage students in what they know, alert them to what they

do not know, and lead them to ways in which the unknowns might be resolved.

The definition of "traditional legal education" also is not meant to imply that only the
casebook method of instruction is used in law schools. Levin (1989), for example,
outlines the many teaching approaches currently in use. However, as will be further
developed in the review of the literature, the case method appears to form the core
around which most law schools build the rest of their curriculum. This is especially true .

when one examines the portion of the curriculum designed to teach legal reasoning.

 

 

 

Finally, in regard to the definition of "legal reasoning,” that phrase will often be used
interchangeably with the phrase "legal thinking.” The reason is that legal literature often
uses "legal reasoning" as synonymous with ”thinking like a lawyer.” Therefore, to the
extent that legal literature is used in this study, it will be difficult to maintain a rigid
distinction between "legal reasoning” and ”legal thinking.” However, it is not the intent
of this writer to suggest that thinking and reasoning are, in fact, synonymous terms. To
a lesser extent, the review of the literature will illustrate a blurring of the lines among
the concepts of "thinking," "reasoning,” and ”problem solving." References to prior
studies will follow the terminology selected by the original writer but following that

usage is not meant to imply that the writer of this study considers the concepts
synonymous.

Plan for this study.

This study has the opportunity to pioneer new approaches to the study of legal
education. The review of the literature will show that previous studies of legal education
have been based primarily upon the teaching experience of the various writers. This
study will break new ground by using multiple data collection and analysis methods
taken from both the legal research tradition and the research on thinking, reasoning,

and problem solving more generally.

Since this study attempts to break new ground, the review of the literature will have to
explain how several different research approaches have been selected and combined.
From the legal research tradition, the review will show how the work of Bryden (1984)
is closest to the goal of this study. Using Bryden’s work as a starting point in the legal
research tradition, the review will also show that legal research on legal reasoning has
usually focused on the case method of teaching and the use of essay questions as a
closely associated measurement method. The use of essay questions turned out, by

hindsight, to be central to the results of this study. Therefore the review of the literature

 

 

will provide an extended treatment of how the case method has a central role in legal

education. That central role led to the use of essay questions in this study.

In examining legal literature, the review also will show why the selected definition of
legal reasoning is appropriate for this study. Levi’s definition has especially close ties
both to the case method of instruction and to problem solving studies more generally.
Specifically, his definition can be linked to the work of Polya, Stemberg, Schon, and
Collins. Tracing some of these links will help the review’s transition to the literature

on thinking, reasoning, and problem solving more generally.

The larger body of literature on thinking, reasoning and problem solving will not only
help put this study in perspective but it will also provide an additional research tool in
the form of thinking aloud protocols. Although other research approaches have been
fruitful, and although the use of thinking aloud protocols has occasioned some contro-
versy, nonetheless thinking aloud protocols provide an easily integrated supplement to
the previously selected measurement approach using essay questions. Previous re-
search indicates that having persons say what they are thinking does not appear to
interfere with a task like writing answers to essay questions. Using both research
approaches will allow the collection of both the product data, the written answers, and
also process data, the thinking aloud protocols.

Chapter III will be devoted entirely to explaining in more detail how the essay questions
and thinking aloud protocols were collected. The outline of the design was that first
and third year law students would write answers to essay questions selected to allow
the demonstration of the students’ legal reasoning. While the students were writing,
they were asked to verbalize what they were thinking. Data was collected from each

student in individual sessions so that the researcher could observe and videotape the

process. Four kinds of data thus were produced: protocols transcribing the students’
thinking aloud comments; written student answers; notes taken by the researcher while
observing the students work on the essay questions; and background information about

the participants.

Both the written answers to the essay questions and the background information about
participants were examined using quantitative methods. The written essay answers
were first classified by independent evaluators. The classifications they used were tied
to Levi’s definition of legal reasoning. Their classifications were converted to numbers
so that group means for first and third year students could be computed and statistically
compared. likewise background measures such as undergraduate grade point averages

were computed and compared.

The thinking aloud protocols and the researcher’s notes were examined using qualita-
tive methods. Selected portions of the students’ thinking aloud comments were tran-
scribed into written protocols. Those protocols, in conjunction with the researcher’s

notes, were then examined for patterns in the students’ legal reasoning.

Chapter IV reports the findings of this study. Contrary to the researcher’s expectation,
the overall finding was that there was no significant difference between the legal
reasoning of the first and third year student groups. Even more contrary to the
researcher’s expectations, the only exception was data from one essay question that

showed the first year student group doing significantly better.

Chapter V tackles the problem of discussing and explaining these findings. It reviews
a number of possible explanations, including the inherent limitations of this study, the

 

problems attendant to the use of essay questions, and the nature of what is being

learned.

The preceding paragraphs give an overview of this study. The next steps are to place
this study within the larger research context and to provide a foundation for the research
choices made in the design and conduct of this study. Both of these steps are taken in

the next chapter on the review of the literature.

CHAPTER II
REVIEW OF THE UTERATURE

Chapter I provided an overview of this study, its research question, definitions, and
research heritage. In particular, it noted that this study of differences in the legal
reasoning of first and third year law students is preceded most closely by the work of
David Bryden (1984). Therefore legal literature, and especially Bryden’s work, will be
the starting point of this review. Reviewing the legal literature will contribute to this
study both a definition of legal reasoning and a basis for coding its presence in essay
question answers. Then this review will place the present study within the larger body
of research on thinking, reasoning, and problem solving more generally. In designing
this study to go beyond what Bryden had done, especially the technique of using
thinking aloud protocols as a data collection method will be drawn from that larger

field of research.

Review of law school research.

Formal, empirical study of law school learning is scarce. Teich (1986) and the report
of the Special Committee for a Study of Legal Education (American Bar Association,
1980) are good overviews of prior work. Much of that work has consisted of surveys of
opinion regarding legal education (e.g., Kahn, 1980). Some studies compared the
effectiveness of various teaching methods (e.g., Kimball and Farmer, 1979; Lorensen,
1968) or testing methods (e.g., Wood, 1924; 1925; 1927). Closer to this study is the work
of Philips (1982) in using ethnographic techniques to trace the learning of legal jargon
or "cant.” However, there is one prior study on which this study can build directly. That
study was done by Bryden (1984).

10

 

11

Bryden’s study.
Bryden studied law school learning by comparing the performance of students who

were at early and late stages of their law school program. Bryden’s participants were
drawn from law students at three excellent American law schools. Random sampling
was attempted in selecting students in their last semester of their third year of law
school. Since about half of those asked declined to participate, the eighty-seven
students who did volunteer cannot be considered a representative sample as Bryden
himself notes (1984, p. 482). Beginning first year students were given a general
invitation and Bryden then selected those whose law school entrance examination
scores most closely matched the already selected third year students. To encourage

students to participate, prize money was offered for excellent performance in the study.

Bryden’s research procedure was to administer a four question, three hour essay
examination to all the participants. The third year students took the examination during
their last semester in law school and the first year students took it early in their first

SCHICSICI'.

The essay questions were similar to those used for regular law school examinations.
Fact patterns, new to the participants, described legal problems such as the possibility
of making a legally binding contract in certain circumstances or the possible conse-
quences for a pet cat that was alleged to have bitten two neighbors. Selected legal

authorities, both cases and statutes, were included in an appendix.

Students were to find "issues," which are problems usually arising from gaps, ambigu-
ities, or conflicts in the interrelationship of existing law with given facts. The questions
were designed to raise certain types of issues that law students would be equipped to

isolate and solve. For example, a statute was ambiguous in its possible application to a

12

set of facts. Students were expected to know that interpreting the statute according to
its purpose would be one way to resolve the issue. Likewise two cases in the appendix
could have been applied to another set of facts. Students were expected to know how
to use the cases based on the distinction between a case’s "holding" (its binding legal
authority) and its ”dicta” (words which might be persuasive but which were not neces-

sary to the decision and thus not binding on a future court).

Bryden and a student each coded the essay answers by counting the number of legal
issues which were raised by the examinees and in addition they noted whether the
examinees resolved each issue that was raised. Bryden then compared the examinees’
responses to what he, as a law school professor, considered to be the issues raised in
the questions. Bryden’s scoring procedure is consistent with typical law school practice
(Delaney, 1982; Kelso & Kelso, 1984; Nickles, 1977; Swygert, 1983; Wood, 1924, 1925,
1927).

Bryden’s findings were that "[t]he seniors were nearly always more proficient than the
entering freshmen, yet hardly anything was said by a majority even of the seniors." In
other words, even the third year law students often failed to raise and discuss issues that
Bryden felt were presented by the questions. Bryden supports this summary conclusion
by giving footnotes detailing what percentage of first and third year students raised each
issue. N o inferential statistics were applied to the data. He does, however, provide
extensive examples of how the first and third year students responded to some of the

questions.

Bryden admits that there are a number of methodological problems with the study. He
acknowledges, for example, that his coding techniques were subjective but he compen-

sates by sharing much of the data so that the reader can draw independent conclusions.

’ \

13

A similar technique would have been helpful in regard to the sampling difficulties which
he encountered; unfortunately no explanation is given for the large number of students

who declined to participate.

One further research question that stands out in Bryden’s work is the question adopted
for pursuit in the present study. Bryden compared the legal reasoning ability of persons
starting law school with those almost finished with the three years of law school. His
finding of relatively small differences between those two groups not only was troubling
to him as a legal educator but also led him to question whether any difference at all
would be found if the group to be compared to third year students were not starting
students, as he had done, but rather students who had completed one year of law school.

That question, of course, is the one taken up by the present study.

This study will take from Bryden, not only the research question comparing first and
third year law students, but also his dependence upon essay questions as a source of
data. As will be clearer when the results of this study are discussed, the use of essay
questions, for better or worse, was the single most important design decision that was
made. Therefore, this review will devote a fair amount of space to explaining why essay
questions seem to be an appropriate data gathering source when studying law school

learning.

Essay questions as data.
Should essay questions be the heart of this study’s data collection? On the one hand,

other tasks have proven fruitful in the analysis of analogous problem solving tasks. For
example, Elstein, Shulman, and Spraflca (1978) used actors as patients to study the
diagnostic skills of physicians. Legal clinics associated with most law schools would
present similar research opportunities. However, if the goal is examining changes in

legal thinking, then essay questions are closely tied to the law school educational

 

‘

14

methods for teaching thinking. The reasons for this close tie are developed in the
following paragraphs.

The principal tie between essay questions and law school’s teaching of legal thinking
comes from the idea that essay questions are like a written continuation of the typical
classroom process. As noted in the introduction, the traditional casebook method uses
sequences of appellate court Opinions. The facts, rules, and reasons of each opinion are
compared and contrasted with the others as well as being tested against new hypothet-
icals proposed by the professor. Discussions typically are between the professor and
individual students but all students are expected to follow along vicariously and be
prepared to step in and carry the discussion if called upon. This process is described in
more detail in, for example, Kelso (1972) or Turow (1977). An essay question can be
seen as a written hypothetical to which each student must explicitly respond individually
rather than implicitly as part of the classroom group. Thus essay questions closely fit

the classroom teaching method.

The second part of the tie between essay questions and law school’s teaching of thinking
comes from the idea that the casebook method is the primary method of teaching legal
thinking. Law schools use a number of teaching methods including lectures, simula-
tions, and clinics (Levin, 1989). However, since about 1870, the casebook method has
had a special place in the teaching of legal thinking. The following paragraphs will show
this special place by briefly tracing the development of the casebook method.

These materials serve a function in addition to tracing the casebook method. The
materials to be quoted are not the product of formal research efforts. All could be
dismissed as speculation based upon anecdotal data. However, all the casebook method

discussion comes from law professors who have made important contributions to legal

15

literature. What they say, therefore, at least is data about the professors themselves and
their own beliefs about the teaching of legal reasoning. Reading quotations such as
those that follow is perhaps the only way to appreciate the depth of the professors
shared belief that their teaching dramatically impacts the way that their students think.
This belief about the possibility of teaching thinking is important for the present study
even if what the professors say about the casebook method is flawed, at least from a

research methodology standpoint.

The casebook method began at Harvard as away of teaching, not so much legal thinking,
but doctrine. In the words of Christopher Iangdell, its founder:

Law, considered as a science, consists of certain principles or doctrines.
To have such a mastery of these as to be able to apply them with constant
facility and certainty to the ever-tangled skein of human affairs, is what
constitutes a true lawyer; and hence to acquire that mastery should be
the business of every earnest student of law. Each of these doctrines has
arrived at its present state by slow degrees; in other words, it is growth,
extending in many cases through centuries. This growth is to be traced
in the main through a series of cases; and much the shortest and best, if
not the only way of mastering the doctrine effectually is by studying the
cases in which it is embodied. But the cases which are useful and
necessary for this purpose at the present day hear an exceedingly small
proportion to all that have been reported. The vast majority are useless,
and worse than useless, for any purpose of systematic study. Moreover,
the number of fundamental legal doctrines is much less than is commonly
supposed; the many guises in which the same doctrine is constantly
making its appearance, and the great extent to which legal treatises are
a repetition of each other, being the cause of much misapprehension. If
these doctrines could be so classified and arranged that each should be
found in its proper place, and nowhere else, they would cease to be
formidable from their number (pp. vi, vii. 1871).

Langdell’s basic message was that law could be a science if it restricted itself to the
development of doctrines drawn from the primary data of decided cases. The common-

place analogy was that law would be like chemistry with the library as the laboratory.

16

In Langdell’s method, students of the law would learn, not from treatises or appren-
ticeships, but from casebooks of court opinions under the guidance of, not practicing
attorneys, but professional teachers. This had the theoretical advantage of making legal
education more suited for a university setting and the practical advantage of allowing

a few teachers to instruct large numbers of students.

If law schools still professed to use Langdell’s case method primarily to teach doctrine,
then the method would have little relevance for the present study of legal thinking.
However, as noted by Stevens (1983, pp. 119-120), Langdell’s emphasis on doctrine was
not totally accepted once it became apparent that the number of legal doctrines to be
learned was greater than that predicted by Langdell. Gradually the emphasis turned to

producing a "trained mind.”

At first the changes were small as the case method’s purpose moved from doctrine to
mind training. In 1892, William Keener, Dean of Columbia Law School, wrote that at

least one modification of the method seemed appropriate:

I think that time can be saved without the sacrifice of substantial benefit
in the first year, by discussing in some subjects the elementary principles
as stated by text writers before discussing the cases on a given topic. This
simply for the purpose of acquainting the student with legal modes of
thought and expression (p. 149).

After about forty more years, the change from conveying doctrine to teaching legal
thinking had gone much further. In 1931, Thomas Konop, based upon his twenty-six

years of experience, wrote the following:

The purpose of a law school is not merely to impart knowledge of the
principles of the law - to simply stuff the memory of the students, but to
train students to think legally.... The decided cases are the authoritative

17

sources of the Common Law. The study of the law should proceed from
these sources (p. 282).

Likewise in 1952, after quoting the previously quoted passage from Langdell, Morgan
says:

I have had considerable experience in the classroom with second and
third year students whose instruction in the first year has been of varied
sorts. And that experience has demonstrated to my satisfaction that men
who have not had rigid training by the case method in their first year are
likely to have developed habits of superficial thought and a tendency to
accept plausible generalizations and to support attractive fireside solu-
tions of difficult problems by ethically appealing but inapplicable legal
maxims; they still lack the capacity to discriminate between the legally
important and the legally immaterial elements in a complicated situa-
tion, although many of them can write a pretty essay on the general
principles which might be but probably are not pertinent upon careful
analysis. Unless such habits are eliminated and unless such a capacity is
developed, the student will continue to be unfit to do the kind of work
required of every really good lawyer in any branch of professional activity

(p. 387).

Morgan goes on to acknowledge that other teaching methods, including lectures, have
a place in law school. However, his conclusion still is that the case method is the "best
available method” for giving a student "a sure foundation” to acquire the capacities and
qualities that the student needs (p. 391). In a comment that foreshadows the results of
the present study, Morgan’s conclusion also notes that a foundation is given because
three or four years of education are insufficient to equip a student with the needed

capacities and qualities.

Reference to more recent articles shows that the case method now is subject to more
criticism but it continues to be thought of as the primary way to teach legal thinking.
For example, Watson (1968) uses his experience both as a law professor and as a

psychiatrist to address the case method of teaching. In his view, the common application

18

of the method has detrimental effects for the student both in law school and in future
legal practice. In law school, the method can lead to psychologically defensive reactions
which hinder learning. Although some students adapt successfully, others are driven by
stress to perform less well than they are able. In legal practice, the after effects of the
case method may leave a student "unemotional" (p. 13 1) and thus less able to work with
people in a service profession. This tracks with the critique of, for example, Redmount

(1981).

Watson, however, seeks modifications to the case method, not its abolition:

Let me emphasize that it is a superb teaching technique which must
remain a fundamental part of good legal education. There is no question
of its effectiveness in developing enormous intellectual skill. However,
modifications could be made which would greatly enhance its value (p.
145).

Principal among Watson’s suggested modifications are various ways for students to see
both the extent of their progress and the need for working with their emotions. Watson
also suggests, among other things, that students be given heroes to emulate, that law
professors receive interdisciplinary training to sharpen their teaching and counseling
skills, that practitioners be involved in the teaching process, and that apprenticeship

arrangements be considered to provide what law school cannot.

Finally, Wade (1984) is a good example of a more recent article that supports the use
of the case method to teach legal thinking. However, these references to Langdell,
Keener, Konop, Morgan, Watson and Wade are not meant to be taken as saying that
all legal writers agree with the case method as traditionally used. To help place the
preceding references in perspective, the following paragraphs will cite examples of

writers who urge more dramatic changes in the case method. However, even while these

19

writers debate particular aspects of the case method, the implicit assumption will
continue to be that the case method plays a central role in the teaching of legal
reasoning. This latter point shows why even persons seeking modifications to the case
method are still important to this present study. If legal writers across a wide spectrum
agree that the case method plays such a role in teaching legal reasoning, then the
foundation is laid for an argument to be made shortly that it is advisable to continue to
use essay questions as part of the data collection due to the close association that essay

questions have with the case method.

The first example of a person seeking more sweeping changes to the case method system
is Llewellyn. For example, Llewellyn (1948) noted that: ”For it is obvious that man could
hardly devise a more wasteful method of imparting information about subject matter
[emphasis in original] than the case-class. Certainly man never has" (p. 215). Llewellyn’s
proposed remedy was to supplement the cases with outside readings. Llewellyn also

proposed that the current definition of the case method was too narrow:

It is not the judicial decision which is the essence of the "case"; it is instead
the concrete problem-raising situation - so that, as I see it, any introduc-
tion of the so-called ”problem method” into law teaching is really but an
expansion of the essential merits of case-teaching, an expansion ob-
scured only by a current mis-emphasis upon the idea of a "case" as being
at best the official report of a judicially decided cause [emphasis in

original] (p. 217).

Finally, Llewellyn (p. 218) stated that law schools have not given attention to the type
of books which could be substituted for casebooks nor have they combined the available
teaching approaches to form "a reasonably rounded, reasonably reliable body of training
for a whole student body” [emphasis in original]. This Llewellyn linked to a social
responsibility to make a law degree "a reliable mint-mark" (p. 218) [emphasis in
original].

20

Llewellyn suggested changes go further than any of the others because he would
substitute hypothetical problems for appellate court cases. This goes to the heart of
Langdell’s desire to treat cases as original data. However, even in suggesting that the
law school case method become more like the case method of, for example, a business
school, Llewellyn at least keeps much of the classroom teaching method in place. The
discussion materials change from appellate cases to hypothetical problems but much

of the rest stays the same.

Another who seeks fundamental changes in the case method is Gross (1984). Gross’
concern is with the much larger question of what conception of the legal system should
law school convey to students. Much of his discussion rests on the premise that law
schools rely too greatly on a nomological model of knowledge and rationality, that it is
too often assumed that phenomena are completely determined by laws that can be
precisely identified, defined, and proved (p. 393). However, as part of that larger issue,
Gross argues (pp. 426-427) that the case method of teaching is too narrow for two
reasons. The first reason is that the case method teaches only some of the skills used in
legal practice. As Gross notes, the teaching of these broader skills has been treated by,
for example, Kreiling (1981). Even if the clinical model of instruction described by
Kreiling is not adopted, Gross would address the teaching of those broader skills, in
part, by including in the curriculum more writing projects. The second reason for
arguing that the case method is too narrow is that the case method, and especially the
essay examinations used in conjunction with it, teach students to reason with overly
abstracted rules rather than with the legal authority itself. In other words, students
learn, especially in preparation for the time pressure imposed by examinations, to
reduce cases to simple rules rather than to reason with them in the fuller process
described by Levi. Gross argues that addressing this problem could be one part of

making students more reflective learners.

21

As fundamental as Llewellyn’s and Gross’ suggested changes might appear to be, they
are not the most extreme position in current law school thought about the case method.
The most caustic critiques flow from writers like Kennedy and Boyle who would turn
the method upon itself. For example, Kennedy admits that law school teaches some
important skills (1982, p. 595) but he is extremely critical of current teaching, including

the use of the case method:

Law schools teach these rather rudimentary, essentially instrumental
skills in a way that almost completely mystifies them for almost all law
students. The mystification has three parts. First, the schools teach skills
through class discussions of cases in which it is asserted that the law
emerges from a rigorous analytical procedure called "legal reasoning,"
which is unintelligible to the layman but somehow both explains and
validates the great majority of the rules in force in our system. At the
same time, the class context and the materials present every legal issue
as distinct from every other, as a tub on its own bottom, so to speak, with
no hope or even any reason to hope that from law study one might derive
an integrating vision of what law is, how it works, or how it might be
changed (other than in an incremental, case-by-case, reformist way).

Second, the teaching of skills in the mystified context of legal reasoning
about utterly unconnected legal problems means that skills are taught
badly, unselfconsciously, to be absorbed by osmosis as one picks up the
knack of ”thinking like a lawyer." Bad or only randomly good teaching
generates and then accentuates real differences and imagined differ-
ences in student capabilities. But it does so in such a way that students
don’t know when they are learning and when they aren’t and have no way
of improving or even understanding their own learning processes. They
experience skills training as the gradual emergence of differences among
themselves, as a process of ranking that reflects something that is just
"there” inside them.

Third, the schools teach skills in isolation from actual lawyering experi-
ence. "Legal reasoning" is sharply distinguished from law practice, and
one learns nothing about practice...

Teachers convince students that legal reasoning exists, and is different
from policy analysis, by bullying them into accepting as valid in particular

22

cases arguments about legal correctness that are circular, question-beg-
ging, incoherent, or so vague as to be meaningless (p. 596).

Boyle (1985) demonstrates how Kennedy’s ideas can be put into practice in the

classroom. According to Boyle:

If there is one issue on which there is general consensus among law
teachers, it is that legal education is even sicker than legal scholarship.
First year angst, second and third year burnout, not enough theory, not
enough practical skills - these themes come up again and again (p. 1005).

The tone of this quotation conveys the passion with which Boyle approaches the topic
of law school teaching. Boyle argues, along with Kennedy, that legal reasoning tech-
niques should be taught explicitly rather than simply being modelled. To illustrate how
that can be done, Boyle provides descriptions of his day to day classroom teaching. Thus
his typical practice is to give students a handout which explains the usual types of
arguments based on precedent and policy. He argues that students can thereby learn
in thirty minutes how to interpret words either by their dictionary meaning or their
purpose, how to phrase a rule broadly or narrowly, and how to select among common
pairs of policy objectives such as morality as freedom versus morality as security or the

need for a firm rule versus the need for a flexible standard.

On the one hand, Boyle’s use of a handout is consistent with the previous remarks of
Keener and even the texts of Langdell’s other followers included material other than
just collections of appellate case reports (e.g., Ames, 1893, 1894, 1904). On the other

hand, the handout is a call for a dramatic change. The summary of his handout states:

Neither the precedential nor the ”policy" arguments are capable of
providing a neutral, ”correct" answer. You must learn to use these
arguments, if you wish to convince decision makers. You cannot rely on
them, or the system they justify, to provide you with moral or political
guidance. In fact, the most dangerous thing about these arguments is that

23

they tend to discourage committed moral or political thought about the
legal system by making you feel insecure or powerless (until you learn
them) and at the same time by offering tools to knock down any sugges-
tion for change. But this cuts both ways. Remember that it also means
that the arguments which people give you about why the status quo is
"simultaneously natural, inevitable and just" - that these arguments are
also vulnerable to deconstruction. The choice is yours (p. 1063).

The preceding references to Llewellyn, Gross, Kennedy, and Boyle demonstrate
forcefully that not all law professors today agree with the case method of teaching as it
has evolved from the time of Langdell. However, for the purposes of this study, the
point again is that the case method, and therefore essay testing, are an integral part of
the law school experience even in the eyes of those who seek major modifications in
the method. In addition, the preceding references demonstrate the depth of these law
professors’ belief that their teaching has a major impact on their students’ legal

The preceding discussion presents an indirect argument for the importance of essay
questions as a source of data on the learning of legal thinking. The argument was that
the case method of instruction has historically played a central role in the teaching of
legal thinking. Therefore it was argued that the learning of legal thinking should be
measured by a test that was very similar to the case method of instruction. Essay
questions filled that role since they were like written hypotheticals forming the next

step beyond the classroom discussion.

Another way to demonstrate the importance of essay questions. is more direct but less
explanatory. Reference will again be made to the work of Wood (1924; 1925; 1927) at
Columbia Law School. Wood worked with law school professors to construct twenty-
two objective tests in twelve courses over three years. The purpose of the tests was to

study the ability of those tests to measure reasoning ability as compared with the like

24

ability of traditional essay tests. Although some of the tests consisted of all objective
questions, other tests were part objective and part essay. The objective test portion of
a combined examination consisted of from 60 to 200 questions and the essay portion
typically was three or four questions (1924, p. 247). In discussing the ability of the
objective tests to predict future performance in law school, Wood drew this conclusion
from some of the 1927 data:

Every correlation presented, without exception, clearly and consistently
favors the conclusion that the new-type [objective] examinations give
more accurate and more reliable measures of reasoning ability than
old-type [essay] examinations of equal or greater time allowance, both
when we compare the new and old-type examinations singly or in com-
bination. In the face of such data, it seems impossible that the idea could
persist that the objective examinations measure only factual knowledge
and isolated bits of information (1927, p. 795).

Despite Wood’s conclusions, law school examinations still are deminated by essay
questions. More specifically, essay questions dominate at the institution where this
research was conducted. The persistent use of essay questions,.in the face of Wood’s
critique, more directly shows how deeply embedded essay questions are in legal

education practice.

This study continued the use of essay questions due to both the direct, and indirect
arguments which have been detailed here. Although alternative data collection meth-
ods had been fruitful in other studies, particularly the use of live actors as simulated
clients or patients, still the use of essay questions appeared reasonable in light of the
connections between essay questions and legal education. Due to the importance of
the decision to use essay questions, those connections with legal education have been
developed at some length. However, as mentioned earlier, the decision to use essay

questions will be revisited when the results of this study are discussed. The decision will

25

be revisited because the use of essay questions may be the single most important factor
underlying the results of this study. The reasons for the importance of using essay
questions will include the insights of Gross presented above, especially his idea that
students learn, not Levi’s reasoning by analogy, but an oversimplified method of using
abstracted rules.

Legal reasoning and its coding.
Even if Bryden’s research question about first and third year students is adopted, and

even if essay questions are used to collect data, the further question remains of whether
Bryden’s coding scheme should be maintained. Bryden coded "issues" based upon
interpreting a statue according to its purpose, distinguishing between the "holding" and
"dicta” of a case, and the like. Although this approach provides indicators of legal
reasoning which are clearly tied to the case method of instruction, a review of legal
literature shows that ”legal reasoning” has been defined in ways that permit other coding
methods.

As an overview, legal reasoning, sometimes called "thinking like a lawyer", has been
examined from a number of directions. Some have focused on the legal education
process either from the perspective of law professors (e.g., Boyle, 1985; Brown, 1983;
Cox, 1976; Friedland, 1986; Kelso, 1972; Llewellyn, 1960) or law students (e.g., Turow,
1977; White, 1983). Kahn (1980) constructed a taxonomy of legal thinking skills based
largely on survey data from law students, professors, and practitioners. Moreover the
American Bar Association has sponsored numerous studies of legal education. A good
example is the 1980 report of the Special Committee for a Study of Legal Education.

Gee and Jackson (1982) provide a good overview of legal education studies.

Legal education and legal reasoning have also been studied from other perspectives,
most of which could have been used as the basis of this study. Teich (1986), as

26 .

mentioned earlier, provides a good summary of prior empirical research on law school
teaching methods, including the previously referenCed studies of Kimball and Farmer
( 1979), Lorensen (1968), and Wood (1924, 1925, 1927). Crombag, de Wijkerslooth, and
Serooskerken (1975) attempted to reduce legal problem solving to a flow chart.
Schwartz (1980) and Thielens (1980) describe the socialization process of which
thinking like a lawyer is a part. Philips (1982) used ethnographic techniques to study
socialization from the aspect of learning the particular "cant" or language in which legal
thinking is expressed. Sarat and F elstiner (1986) used similar techniques to describe
what an attorney does during an actual client conference. Frank (1947) wrote about
clinical legal education. Boyum and Mather (1983), Jensen and Horvitz (1979), Nagel
and Neef (1979), and Spaeth (1979) all use decision theory to model legal reasoning.
Brkic (1985) describes it from the perspective of formal logic. Gardner (1987) and Tito
(1987) explain legal reasoning in the context of artificial intelligence. Stevens (1983)
provides a historical perspective by tracing how thinking like a lawyer has been taught
in America since 1850 and Consalus (1978) covers the Colonial period. Parallels have
been drawn between legal reasoning and taking photographs (Conniff, 1985), writing
history (Berlin, 1960), and interpreting literary works (Dworkin, 1985).

In addition to the preceding, two other perspectives could have been used as the basis
of this study. One would be a philosophical perspective. This is an attractive possibility
since this is the most elemental level on which to consider what thinking like a lawyer
really means. Burton (1985), Golding (1984), Goldstein (1987), and MacCormick
(1978) give good introductions to legal reasoning in a philosophical context. Dworkin
(1978), Fuller (1964), Hart (1961), and Kelman ( 1987) are good examples of the vast

literature developing the concepts in more detail.

A

27

The other attractive perspective from which to work would be based upon the lists or
catalogues of legal education objectives developed by, for example, Cort and Sammons
(1980), Holmes (1976), and Strong (1950). Wangerin (1986) reviews much of this work
and adds some contributions of his own. Each of these lists attempts to present an
overview of the capacities which a law student needs to become a lawyer. Thus Holmes
includes descriptions not only of legal analysis and synthesis, but also such skills as legal
drafting, counseling, and negotiation (p. 579). He also attempts to present common
ground for all the different law school teaching approaches.

Cort and Sammons developed detailed definitions of generic competencies for use in
evaluating law students performance in a law school clinical setting. Their approach
also received support when the State Bar of California Bar Examiners (1988) adopted

parts of it in the practice competency section of their bar examination.

However, for a study of the changes taking place in law students during law school, one
definition of "legal reasoning” seems especially attractive because it is so closely tied to
the essential ideas behind the casebook method. The definition was developed by a law
school classroom professor, Edward Levi, a long-time professor of law at University of
Chicago. Based upon his own experience in teaching from appellate opinions, Levi

(1949, pp. 1-2) said:

The basic pattern of legal reasoning is reasoning by example. It is
reasoning from case to case. It is a three-step process described by the
doctrine of precedent in which a proposition descriptive of the first case
is made into a rule of law and then applied to a next similar situation.
The steps are these: similarity is seen between the cases; next the rule of
law inherent in the first case is announced; then the rule of law is made
applicable to the second case.

28

The balance of Levi’s book consists of extended examples of legal reasoning drawn
from sequences of appellate opinions. Part of one of those sequences, the one on the
law of product liability, is included in the torts textbook (Prosse'r, Wade, and Schwartz,
1982) which all participants in this research project had studied. J aff (1986) and Suber
(1988) provide examples of how Levi’s definition fits closely with law school teaching.

Levi’s approach also has the advantage of describing a process which makes it more
suitable for this study than the doctrine oriented approach of Langdell (1871), the
closely associated definition of Holmes (1976), or Cort and Sammons’ (1980) product
oriented analysis. Levi’s description also is at a convenient level of abstraction. On the
one hand, Ievi himself stresses (e.g., 1949, p. 7) that legal reasoning is not primarily a
deductive process applied to an existing rule; reasoning by example is the more
fundamental level of logic which is used to select rules or premises to be applied in a
given situation. On the other hand, it does not propose to settle which of two conflicting
analogies necessarily must be selected. That selection might well involve philosophical
analysis on which there is no present consensus in the legal field. In other words, using
Levi’s definition provides the advantage of allowing a middle level of process analysis
to which law students would have received repeated indirect exposure in their case

method study of appellate opinions.

As will be developed in more detail in Chapter III, Levi’s definition provides the basis
for a system of coding different from the issue counting of Bryden. Levi’s definition
focuses on the role of case analogies in legal reasoning. Thus, if a person is using legal
reasoning as described by Levi, they will mention cases. References to cases might
prove to be easier or less ambiguous to code than the categories used by Bryden. Thus,
although counting case references would be an indirect measure of legal reasoning, it

still might be a better. method than Bryden’s counting of issues. In addition, Levi’s

29

definition has the advantage of describing legal reasoning even when the reasoning
does not complete all the steps described by Levi. If a student uses only a rule, as Gross
(1984) would predict, this still can be described in Levi’s terms as being a truncated
version of the more complete process. The use of the rule is counted even if the rest of
the process is absent. Thus the coding still can work even if, for example, a deductive

reasoning process is being used.

Levi’s definition of legal reasoning, although developed in the legal context, also has
the advantage of being paralleled by problem solving findings in other contexts. These
parallels help link the concept of legal reasoning to the more general field of problem
solving. It is not difficult, for example, to make connections between Levi’s definition
and the works of Polya or Stemberg. One of the closest parallels comes from the
protocol analysis work of Schon (1983; 1987). Schon studied, among other things, the
interactions between a teacher and student of architectural design. Based upon his
observations, Schon isolates three levels or conceptions of thinking during problem
solving. The first level is the use of facts, rules, and procedures to solve instrumental
problems nonproblematically (1987, p. 39). Teaching this level of thinking is a form of
technical training in which, as Schon puts it:

[s]tudents would be expected to-acquire the material by reading, listening
and watching, familiarizing themselves with examples of practice prob-
lems matched to the appropriate categories of theory and technique.
Coaching would consist in observing student performance, detecting
errors of application, pointing out correct responses. (1987, p. 39)

The second level of thinking is where problem solving is viewed as ”thinking like a
". On this level, Schon states:

studentswill still learn relevant facts and operations but will also learn
the forms of inquiry by which competent practitioners reason their way,

30

in problematic instances, to clear connections between general knowl-
edge and particular cases. The standard drills of law school classroom
and the medical clinic exemplify this view. In a practicum of this kind,
there is presumed to be a right answer for every situation, some item in
the corpus of professional knowledge that can be seen, eventually, to fit
the case at hand. (1987, p. 39).

The third level of thinking is an extension of the second level but concentrates not on
rules of inquiry assumed to be available to solve all problems but instead focuses on
what Schon calls "re'flection-in-action". This concept is developed in both of his books

but he states the general idea as follows:

We will see students as having to learn a kind of reflection-in-action that
goes beyond statable rules - not only by devising new methods of reason-
ing, as above, but also by constructing and testing new categories of
understanding, strategies of action, and ways of framing problems.
Coaches will emphasize indeterminate zones of practice and reflective
conversations with the materials of a situation. (1987, pp. 39-40).

Both Levi (1949) and Schon (1987) leave open the question of how a problem solver
finds an analogy or sees a new way of framing a problem. In the legal context, some
insight on this process may be provided by the work of Collins (1985). His studies also
used protocols to study legal teaching. His subjects included well-known professors like
Max Beberman, Richard Anderson, Arthur Miller and Roger Schank.

Collins identified ten major strategies which inquiry teachers use in choosing cases
(1985, pp. 580-581). The ten can be arranged in five pairs of cases: positive and negative
paradigm; near miss and near hit; maximal and minimal relation; independent or
dependent variable range; and insufficient or unnecessary factor counter example.
Although Collins warned that the list is incomplete, especially for law schools where
the cases may be in the form of hypotheticals, still the pairs add detail to optimal

31

learning sequences such as that suggested by Gagne (1977, pp. 119-120) for concept
learning.

In addition to the listing of types of cases used in inquiry teaching, Collins listed eight
strategies that are used to force students to learn reasoning skills. Those eight were:
asking students to form hypotheses; asking students to test hypotheses; asking students
to make predictions; asking students to consider alternative predictions; entrapping
students into revealing their misconception; tracing consequences to a contradiction;
encouraging students to formulate alternative hypotheses; and encouraging students

to question authority.

Collins’ (1985) findings supplement the findings of Bryden, Levi, and Schon. Bryden
counted the number of issues raised by his Subjects. Levi referred to the use of analogies
as central to legal thinking. Schon made reflective thinking the center of his discussion.
However, as noted earlier, Bryden, Levi, and Schon all left much unsaid about how the
subject originally would isolate an issue or pick an analogy or begin a new cycle of
reflective thought. More generally, Dillon (1982) has noted that not much is known
about "problem finding” as opposed to problem solving.

In this regard, Collins (1985) might have helped. Although a distinction might be drawn
between techniques of teaching problem solving and techniques of problem solving,
still Collins’ list of case types and teaching strategies might have served‘as guides when
measuring problem solving learning. This possibility arose from the research of those
like Bandura (1965) who found that modeled behavior was likely to be imitated. In the
present study, law school professors will have modeled certain techniques in the process
of teaching problem solving. When student problem solving is observed, one could

hypothesize that some of those techniques might appear even though it must be granted

32

that the task facing the student is very different from that which faced the professor.
For the professor, the techniques might help in selecting the next question to ask in
order to assist a learner. For the student, the techniques might help in finding an
appropriate analog to solve a legal problem. Thus Collins’ list of techniques might
have provided clues on what to look for when legal reasoning is examined in protocols.
However, as will become clearer in Chapters IV and V, the data actually gathered in
the present study did not lend itself to analysis using Collins’ lists.

Despite the nice meshing of Levi’s definition with the work of Schon and Collin’s, not
everyone would agree that reasoning by analog is the best definition of what students
learn in law school. In part this has been foreshadowed in the exchange between King
James and Lord Coke which was quoted in Chapter I. Just as King James felt that he
had science and reason and therefore he did not need specially educated judges to
decide cases, so some scholars would support the possibility that law students come to
law school already equipped with the reasoningprocess described by Levi and therefore
the scholars question how time is spent in law school. On a very general level, this would
be supported by Minsky’s (1986) observation that our language is riddled and stitched
with analogies and metaphors. In the legal context, Mudd (1983, pp. 705-706) has
questioned how legal thinking differs from what students learn in undergraduate
courses. Homstein (1981) likewise argues that legal reasoning is good reasoning
applied to legal materials. Fejfar (1986), relying upon Lonergan (1957), states that:
"Much of a law student’s education, as well as a lawyer’s practice, involves common
sense, rather than purely logical or scientific understanding." Part of Lonergan’s

description of common sense sounds much like Levi’s description of legal reasoning:

Common sense, on the other hand, never aspires to universally valid
knowledge and it never attempts exhaustive communication. Its concern
is the concrete and particular. Its function is to master each situation as
it arises. Its procedure is to reach an incomplete set of insights that is to

33

be completed only by adding on each occasion the further insights that
scrutiny of the occasion reveals. (Lonergan, 1957, pp. 176-177)

If legal reasoning is common sense, then interesting linkages might exist with work
currently being done on informal logic (e.g., Blair & Johnson, 1980). However, if legal
reasoning is common sense already possessed by the students, then this could have a
dramatic effect on a study, such as this one, .which attempts to measure law school
learning. This topic will come up again in Chapter V during the discussion of the results
of this study. In any case, the debate over the nature of legal reasoning helps to illustrate
how a review of the legal literature soon leads to literature on thinking, reasoning, and

problem solving more generally. It is to that literature that this review now turns.

General literature on thinking, reasoning, and problem solving.

Reviewing legal education literature provides not only a starting point in Bryden but
also a basis for using essay questions in the study of legal reasoning. However, further
insights into the present research question are provided by work that has been done in
regard to thinking, reasoning, and problem solving more generally. This has already
been demonstrated by, for example, reference to the work of Schon and Lonergan. In
addition, this section of the review will draw from that body of work especially the idea
of using thinking aloud protocols to supplement the data being gathered in the essay

questions.

It is not the purpose of this review to exhaustively cover prior work on thinking,
reasoning, and problem solving. That literature stretches back even beyond the dia-
logue of Socrates with Menon (Plato, __B.C./ 1956). For a person who wishes to
review more of that literature, at least the more recent portions have been surveyed in

the works of a number of authors. Good examples include the works of Mayer (1983),

34

Nickerson (1988), Nickerson, Perkins, and Smith (1985), and Shuell (1986). Edited
works include: Baron and Stemberg (1987); J ohnson-Laird and Wason (1977); and the
two volume set by Sega], Chipman,'and Glaser (1985). Frederiksen ( 1984), Hill’s (1979)
bibliography, and Rowe (1985) concentrate on problem solving. Edited works on
problem solving include Groner, Groner, and Bischof (1983) and Tuma and Reif
(1980). Bloom and Broder (1950) discussed theproblem solving process of college
students. Chi, Glaser, and Farr (1988) edit a collection on the nature and acquisition
of expertise. Kennedy (1987) concentrates on the development of expertise in the

context of professional education.

The preceding reviews show that, as was the case for legal education, some programs
which teach thinking have been the subject of relatively little formal research. For
example, the works of de Bono (1967; 1976; .1986), Polya (1954; 1957), and Whimbey
and Lochhead (1981) have been widely accepted and used but much of that acceptance
and use appears to be based upon intuitive appeal rather than extensive formal
evaluation. Of course this does not mean that no evaluation of the programs has been
done. Lochhead (1985), for example, reports on how parts of his program have been
evaluated. However, some programs have been subjected to much more evaluation.
For example, IE or FIE (Feuerstein Instrumental Enrichment) program of F euerstein
(Feuerstein, Hoffman, Jensen, & Rand, 1985) is a program which has been formally
tested a number of times as illustrated by the review of Savell, Twohig, and Rachford
(1986).

The reviews also show that, compared to the legal literature, there is an even greater
split between studies which support or undermine the proposition that thinking,
reasoning or problem solving can be taught. On the supporting side are studies like
Nisbett, Fong, Lehman, and Cheng’s (1987) success in teaching inferential rules,

35

Ghatala, Levin, Pressley, and Lodico’s (1985) results in teaching cognitive strateg
monitoring to second graders, and Woodward, Carnine, and Gersten’s (1988) reported
improvements to problem solving skills through the use of structured teaching and

computer simulations.

Even closer to this study would be successful programs based on Stemberg’s (1977;
1982) work on analogies. Polya’s work would have fit into this category but, as Newell
(1983) notes, Polya’s work is often overlooked. In any case, Alexander, White, Haensly,
and Crimmins-Jeanes (1987) is a good example of a successful program based upon
Stemberg’s work. It is especially interesting for the law school context because, al-
though based upon work with fourth graders, it supports the proposition that direct

classroom instruction can help with analogical reasoning.

On the other hand, there are also studies which undermine the proposition that
thinking, reasoning, and problem solving can be taught. For example, if behavioral
decision theory studies are consulted, a number would call into question the very kind
of learning which law school attempts to provide. Johnson (1988) is a recent summary
of the implications of those studies across contexts such as graduate admissions office
decisions (Dawes, 1971), economic forecasting (Armstrong, 1978), and clinical psychol-
0g (Goldberg, 1970). Likewise the studies of Elstein et al., (1978) and Feltovich,
Johnson, Moller, and Swanson (1984) indicate that learning a knowledge base is the
key to expert problem solving.

Thinking aloud protocols as data. .
Reference to the wider field of thinking, reasoning, and problem solving thus does not

provide unequivocal support for the proposition that thinking, reasoning, and problem
solving can be taught in schools, even if the school is a law school. However, the studies

do provide something else for this study. Many of the studies in other areas use, not the

36

essay questions of law school, but thinking aloud protocols produced by making a
transcript of what a person says when asked to say what they are thinking while problem
solving. N ewell and Simon (1972) demonstrated the use of the method with puzzle
solving but it soon was used in expert-novice (Elstein et al., 1978; Feltovich et al., 1984)

and naive-novice (Voss, Blais, Means, Greene, and Ahwesh, 1986) comparisons.

Collection of thinking aloud protocols has not met universal approval. Nisbett and
Wilson (1977) present detailed arguments against them, including the idea that think-
ing aloud interferes with the task being studied. However, Ericsson and Simon (1984)
answer many of those arguments, at least in a context where concurrent verbal expla-
nations are close to the mental verbalizations likely to be involved in the task. For
example, concurrent verbalization might not interfere with execution of a mental task
like a Tower of Hanoi problem but might well interfere with a task like hitting a
baseball.

Likewise Johnston and Afflerbach (1985) addressed the problem of experts not verbal-
izing what they were doing because the processes appeared to be elemental. Johnston
and Afflerbach believed that the processes were not elemental but rather were auto-
mated from long practice. They found that selecting the appropriate level of difficulty
for a question could increase the amount and quality of data gathered. Increasing the
difficulty of the questions apparently caused the experts to move more toward novice
performance. When operating closer to the novice level, the automated processes

disappeared and the steps in the thinking process were rendered accessible.

Although studies such as those of Ericsson and Simon and Johnston and Afflerbach
address some of the criticisms of thinking aloud protocol methodolog, they do not

answer all the questions, particularly in regard to combining thinking aloud with the

37

task of writing answers to essay questions. In addition, alternative research methodol-
ogies are available. As mentioned earlier, Johnson (1988) collects a number of studies
which have used judgment and decision theory to study experts. Shulman and Elstein
(1975) explain various judgment and decision theory approaches. They also distinguish
those approaches from process tracing approaches, including thinking aloud protocols,

and give examples of contributions attained by each approach.

N 0 one research approach has emerged as the method of choice. However, given the
emphasis of this study on reasoning rather than, for example, decision making, it would
appear that the use of thinking aloud protocols would be a good research approach to

supplement the data collection from essay questions.

Summary.

The goal of this study is to examine the differences in the legal reasoning of first and
third year law students. Because the starting point for this study would be Bryden’s work
based in the legal research tradition, this review first examined works in that scholarly
tradition. That review showed that the use of essay questions was deeply embedded in
both law school teaching theory and practice. The key to the wide-spread use of essay
questions appeared to be their close connection to the type of discussion typically

conducted in a law school class using the case method of teaching.

Regarding the definition of the key phrase "legal reasoning", a number of options
presented themselves in the literature. The basis for these ranged from the Langdell’s
doctrinal emphasis to the practice oriented descriptions of Cort and Sammons. For
purposes of studying the effect of classroom learning, the definition developed by a

classroom professor was selected as most closely tied to the case method of instruction.

38

The process described in Levi’s definition also had the advantage of facilitating coding
and of paralleling the processes described by, for example, Schon.

Although Bryden’s study provided the research question and initial outline for this
study, and although review of other legal research both provided a definition of legal
reasoning and reinforced the advisability of using essay questions to elicit data, the
more general literature also made contributions. The impetus for conducting this study
was increased by the possibility of contributing to the ongoing controversy about
whether thinking, reasoning, and problem solving can be taught. In addition, the
general literature provided an additional research tool in the form of thinking aloud
protocols. This tool would assist data collection on participants’ thinldng processes at

the same time that the written products in the form of essay answers would be gathered.

Much of this study’s design has taken shape during this review of the literature.
However, the more detailed description of the resulting research plan now will be taken

up in Chapter III.

I CHAPTER III
PROCEDURE

In order to examine the differences between the legal reasoning of first and third year
law students, this study used a new combination of research techniques drawn from
both previous legal and social science research. From the legal research area, the
central role of the law school case method of study led to the use of essay questions to
collect data about legal reasoning. The definition of legal reasoning, and the essay
scoring scale presented in this chapter, likewise were tailored to measuring the legal
reasoning of persons who had studied using the case book method. From the broader
area of research on thinking, reasoning, and problem solving, the technique of using
thinking aloud protocols was drawn. The basic idea was that thinking aloud protocols
would capture data about the students’ thinking processes at the same time as they
created their essay answer data. Thus there would be from first and third year law
students both process data, the thinking aloud protocols, and product data, the essay
answers, which could be analyzed for differences. The rest of the data collection and

analysis would be built around the core of product and process data.

Since this study breaks new ground, the design selected emphasizes exploration of the
students’ legal reasoning rather than an in depth replication of an existing line of
research. The emphasis on exploration led to multiple approaches both for data
collection and data analysis. For data collection, four kinds of data were collected from
both first and third year law students: product data in the form of essay question
answers; process data in the form of thinking aloud protocols; general data from the
observations of the researcher as the students worked on the questions; and background

data collected to double check the equivalency of the two groups of students who were

39

40

being compared. For data analysis, the methods included: visual inspection and statis-
tical analysis of numerical data like the essay answer scores and the background
undergraduate grade point average data; computer searches of the essay answers for
the frequency with which key terms were used; and the researcher’s individual scanning
of both the thinking aloud protocols and the researcher’s notes to determine if thinking

patterns could be isolated and compared.

The starting point for explaining the design will be the details of how the students were
selected and grouped for study. Then the data collection and data analysis methods will
be presented in more detail. The central concern of each step in the design will be to
assist in answering the question of how the legal reasoning of first year law students

compares to that of third year students.

Participant selection procedures.

Population.
The population to be sampled consisted of students at an accredited midwestern law

school. Selecting this population had both benefits and drawbacks when compared to
the populations used in other studies. On the benefit side, selecting students from an
accredited, but not "top twenty”, law school provided an opportunity to study students
from a population that has not been represented in prior studies. Bryden’s students, for
example, were only from top law schools. In addition, the particular midwestern school
selected had a much higher than average number of required courses. Thus the three
year curriculum was more uniform than at schools which allow more elective courses.
This increased uniformity lessened the uncontrolled variation in the educational

experience of the third year students who were selected to participate.

41

A more subtle curb for uncontrolled variation was that the students at the selected
institution were all "part time" students. Part time students take fewer credit hours per
semester but still complete the program in three years due to the availability of summer
sessions. The course sequences that they take are the same as would be taken by full
time students. However, the commuting or employment responsibilities typical for a
part time student often shield the part time student from some informal educational
influences. The full time student is more likely to be present at a law school outside of
required class hours. This allows full time students the luxury of an opportunity, for
example, to have a coffee break in the lunchroom with a professor. A part time student
dashing from the office to class would not get that same opportunity. This leads to the
somewhat ironical result that the part time students selected in this study may have
been more representative of the results of traditional legal education than full time
students would be because the part time students have had less exposure to informal

sources of legal education.

The drawbacks to using this population include those associated with selection at only
one institution and with the use of volunteers. Each of these drawbacks will be discussed

separately in the following paragraphs.

The ability to generalize from the results of this study will be qualified by the decision
to select students at only one institution. However, in addition to the benefits men-
tioned previously, this decision also appears reasonable in light of the results of
Bryden’s study. Bryden found that the results were "suggestively consistent" from school
to school (1984, p. 500). This finding is not surprising in light of the uniformity in content
and method in traditional legal education. Therefore restrictions on the ability to

generalize may not be as severe as they might first appear.

42

The drawbacks of using volunteers are well documented in the literature. However,
given the nature of the highly intensive type of testing involved in this proposed study,
it appeared that there was no reasonable alternative to the use of volunteers, at least if

student rights are to be adequately protected.

Informed Consent.
Since the principal researcher is himself a law professor who could be viewed as an

influential person regarding future decisions affecting student participants, special care
was devoted to obtaining the free and informed consent of participants. Fortunately
the research site had two characteristics which greatly assisted the protection of student
participants. The first characteristic was that almost all courses at the law school use
anonymous grading; the only exceptions were courses in which the principal researcher
could easily avoid involvement. The second characteristic was that all participants had
a minimum of twenty semester credit hours of law school and thus already had studied
the law of Torts. Torts includes the study of the doctrine of informed consent and thus
participants in this study were better equipped to make a free and informed decision.

The primary method of insuring informed consent was through the use of the consent
form which accompanied the participation invitation letter (Appendix "A"). The con-
sent form contained numbered paragraphs setting forth the following points. Paragraph
1 summarized the purpose, procedures, and duration of the research. Paragraph 2
expressed the participant’s agreement both that the research has been explained and
that the participant understands it, including the following two risks. One risk is the
stress associated with the taking of a three hour essay examination. The other is the
possibility that the results of the essay examination may not be in accordance with the
participant’s own self image. Paragraph 3 explained that the principal researcher
retained the right to grade participants in future law school courses if those courses use

the school’s anonymous grading procedure. It also explained that the principal re-

43

searcher would not join in law school decision making concerning an individual
participant unless the participant made a written request asking the principal re-
searcher to join in a particular decision process. Finally, paragraph 3 indicated that
prospective participants should not sign the consent form unless they understood the
role of the principal researcher as it was set forth in paragraph 3. Paragraph 4 stated
that the participants were free to discontinue participation at any time without recrim-
ination. Paragraph 5 stated that all results would be treated with strict confidence
regarding the identity of any participant but that individual participants would be able
to obtain their own results if they so desire. Paragraph 6 stated that participation
involved no guaranteed benefits to the participants other than the agreed fee paid for
their participation. Paragraph 7 stated that signing the form indicates that the signer
freely consents to participate.

The consent form also included the phone numbers of two persons from whom

additional information could be obtained by prospective participants.

Sampling.
Volunteers were solicited from students who were in either their third semester or in

their eighth or ninth semester of law school. An invitation was mailed to each student
followed by classroom announcements. The letters (Appendix "B”) included a tear-off
section on which students could volunteer by writing their name, address, and tele-
phone number. Participation was encouraged by offering twenty-five dollars to each

volunteer who was chosen for participation.

Forty-eight students volunteered. Of the forty-eight volunteers, eight were not eligible
for participation: five had cumulative grade point averages below a 2.0 and three had
completed between forty-five and sixty credit hours and thus did not qualify for

inclusion in either the first or third year group.

44

Each of the remaining forty volunteers were sent a letter (Appendix ”C') asking them
to schedule a data collection session at a time that would be convenient for them. Using
letters to invite the students avoided the possible coercive effects attendant to an in
person solicitation from a person, in this case a law professor, who could be viewed as
being in a position of authority over the student. Letter invitations also standardized
the information given to each student and thus avoided the possible confounding effects

of information that might slip out in a conversation with the student.

Sessions were scheduled at all times of day on all days of the week to accommodate
participants. Nevertheless, of the forty eligible volunteers, only thirty-one took the
additional step of signing up for a session and this number was reached only by
extending the sign up period three weeks longer than originally planned. Fortunately,
of the thirty-one students, sixteen were first year and fifteen third year. One first year
student was used for a trial run of the data collection process and this left an even
number of students in the two groups. Since about one hundred ten first year students
were eligible, between fourteen and fifteen per cent actually participated. For the third
year students, the participation rate was between six and seven percent since fifteen

participated out of a total eligible pool of about two hundred thirty.

Data collection.

Data was collected at individual administrations of four essay questions (Appendix "D")
during a three hour session. The four essay questions were drafted by the principal
researcher. All of the questions also were reviewed by a panel of two law school
professors. Each of these professors had well over ten years of legal education experi-
ence. This panel approved the questions as to form, substance, and likelihood of
eliciting "legal reasoning" as defined in this study.

45

The legal subject matter covered by the questions was confined to the areas of
substantive law covered in the first year of law school: contracts, crimes, civil procedure,
property, and torts (torts are civil wrongs, other than breach of contract, such as
defamation and civil fraud). This offset the superior substantive knowledge of the third
year students with the more recent exposure of the first year students to the materials.

Previous experience with law school examinations indicated that three hours would be
more than enough time to complete the four questions, even with time allowed for
verbally describing one’s thoughts while writing. Only one student came close to using

the full three hours and even that student did not complain of time pressures.

The essay questions were drafted to present problems for which there exists no single,
easy answer. This was done in the hopes of eliciting more legal reasoning by avoiding
automated processes in the same way as Johnston and Afflerbach (1985) did in their
reading comprehension study. However, the absence of a single, simple answer did not
mean that the questions were impossible or even very difficult. The questions were well

within the capabilities of even the first year law student participants.

As previously noted, a three hour block was scheduled to accommodate the student’s
schedule. Data was collected in two ways. First, as in Bryden’s study, students were
asked to write answers to each of the four essay questions. Second, the students were
asked to verbally describe what they were thinking while they were writing the essay
answers. Avideotape camera and recorder were used to capture these verbal responses.
No time limit was given for each question but the students were asked to complete the
four questions within the three hour block and a clock was provided to assist the
students in keeping track of the time. Both the number of questions and the time

allowed would be typical for a law school examination.

43

To make data collection seem more like a lawyer’s typical office work, and less like a
law examination, students wrote their answers while seated at a desk in an office.
Writing was done on yellow legal pads rather than examination bluebooks. Students
were told that could take a break at any time to get a cup of coffee, use the washroom,

etc.

Prior to beginning writing, each student signed a statement attesting to the fact that the
student had not discussed the essay questions with any of the other participants, either
directly or indirectly. Each student was also given warm up instructions (Appendix "E")
and two scrambled letter problems so that they could practice the thinking aloud
procedure. The two scrambled words used, in the order given, were "koro" (rook) and
”npepha" (happen). After the students had worked through these words, they were
asked to select one set of questions from forty sets spread face down on the desk. The
sequence of questions in each set had been systematically varied so that the question
that was first in the first set became the second question in the second set and so forth.
The resulting forty sets of questions then had been shuffled. By selecting one of the

face down sets, the student received a randomly assigned sequence of questions.

After completion of the essay questions, participants were asked to fill out a question-
naire (Appendix "F"). This questionnaire collected data on participants’ legal experi-
ence outside of the required courses. Included was information on elective courses
taken, employment experiences if legally related, and close associations, if any, with
attorneys. This information served to give some indication of what other factors may

be affecting the performance of participants.

47

Data Analysis.

The students’ written responses and their verbal comments were analyzed in two stages.
First, the written answers were anonymously classified by a panel of eight coders, each
of whom had experience with the law. Second, the principal researcher examined the
students’ written answers in conjunction with the students’ concurrent verbal comments
and the researcher’s observation notes in an attempt to further understand what the
students did and did not do and why. The goal of both stages of the analysis was to
answer the primary question of how the legal reasoning of students who have completed
almost one year of traditional legal education compared to the legal reasoning of those

students who have completed almost three years.

Both stages of data analysis followed the work of Levi as summarized in Levi’s
previously quoted definition of legal reasoning. To assist the evaluation of students’
legal reasoning, that definition was used as the basis for a classification scale (Appendix
"G") which was given to the panel of eight coders who evaluated the student answers.
The scale’s classifications were defined according to the type of authority used by the
student and whether the facts of the question or previous legal case were used in the
student’s answer. The types of authority recognized included nonlegal authority, legal

rules, and prior case law such as a reported decision of an appellate court.

This approach to coding was chosen as the best way to indirectly assess the extent to
which students were using Levi’s reasoning by analog. If a student were to use all three
of Levi’s steps, the student would have to mention the facts of the previous case, the
facts of the present problem, how the two sets of facts compared and contrasted with
each other, the disposition of the previous case, the legal rule which could be developed

from consideration of both sets of facts, and the disposition for the present problem.

43

Thus, the extent to which a student worked with both the facts of the cases and of the
problems would be one indication of reasoning by analog. If, on the other hand, a
student mentioned only a common sense or. legal rule, it would indicate deductive

reasoning that stops short of using all of Levi’s steps.

The coding classifications forced the coders to separately consider whether the answer
was based upon the use of facts and whether it used a given type of authority. Breaking
the coding process down into these steps was an attempt to achieve higher reliability
in coding than if the coders were simply given Levi’s definition and asked to evaluate
the extent of its use in an answer. Breaking the coding process into separate steps for
facts and authority also attempted to achieve higher reliability than would have been

achieved with the use of issue counting in the manner done by Bryden.

Letter combinations, such as "NLAA" (nonlegal authority alone) or ”CF-C&P" (Case
with Facts of the Case & this Problem) were used to indicate both the type of authority
being used and whether that authority was used in conjunction with facts or not. The
combinations used were: NLAA (Non Legal Authority Alone); NLAF (Non Legal
Authority with Facts); RA (Rule Alone); RF (Rule with Facts); CA (Case Alone); CF
-C or P (Case with Facts of Case or this Problem); and CF - C&P (Case with Facts -
Case and this Problem). The use of letters instead of numbers helped the coders
remember what each classification was and it also avoided the inference that one
classification was necessarily better than another. The principal researcher met with
each coder and used a practice question and answers (Appendix "H") to assure that the
classifications were applied correctly by the coder. The practice answers were obtained
from student responses to a classroom exercise which was otherwise unconnected to

this study.

49

FlfSt stage data analysis.
The first stage of data analysis was classification of the student answers by the eight

coders mentioned above. Each of the four essay question answer sets was randomly
assigned to two coders who worked independently. Thus each coder classified thirty
essay answers. Prior to giving the answers to the coders, each answer to each question
was assigned a random identification number. Coders therefore classified the answers
without any knowledge either of the student writer’s identity or of the group, first or
third year, to which the student belonged.

After the answers had been classified and returned to the principal researcher, the
classifications were converted to numbers to assist analysis. The numbers were assigned
in the order the classifications appear in the classification guide (Appendix "G"). Thus
"NLAA" (Non Legal Authority Alone) became ”1", "NLAF” (Non Legal Authority with
Facts) became "2", "RA” (Rule Alone) became "3", "RF" (Rule with Facts) became "4",
"CA” (Case Alone) became "5", ”CF-C or P" (Case with Facts of Case or this Problem)
became "6", and "CF-C&P” (Case with Facts of Case and this Problem) became "7". The
participant’s final score for each question was the total of the two numbers resulting
from the coders’ classifications. Thus is if a particular essay answer received a ”5" from
one coder’s classification and a "4" from the second coder’s, then the participant’s final
score for that answer was a ”9". Higher numbers were assumed to indicate more
complete conformity with Ievi’s definition of legal reasoning and thus the classification

scale was assumed to be an ordinal scale.

Since the classification scale has not been validated in prior research, the scale also was
recoded into two dichotomous variables which reflect key elements of Levi’s definition
of legal reasoning. The two variables were based upon whether facts were used in the

answer or not and on whether a rule (common sense or legal) or a case was used. The

50

recoding was done with the researcher acting as a tie-breaker if the two scorers’
classifications would lead to different results. Since both the seven point scale and the
dichotomous variables were based upon Levi’s definition, it was expected that both
coding methods would lead to similar results. However, the recoding was done to allow
that assumption to be verified.

Student classification scores next were sorted back into first and third year student
groupings. The scores were then analyzed in three ways. First, the scores were graphed
for visual examination of how each group of students performed on each question. Then
mean scores for each question were computed and graphed for both the first and third
year student groups. Finally, interpretation of the scores was assisted by the computa-
tion of several statistics. To examine how well the classification scales worked, a
Spearman-Brown scorer reliability estimate was computed as well as correlations
between the original seven point scale and the dichotomous scales produced by the
recoding. To examine the results obtained from the classification scales, a Student’s t
test of significance for the differences between the student group means on each
question was computed as well as an analysis of covariance using undergraduate grade

point averages and Law School Admission Test scores as covariates.

All analysis of the data, whether informal visual inspection procedures or formal
statistical tests, were conducted based upon a null hypothesis of no difference between
the two groups. The alternative hypothesis was that a third year student group mean
score on a question would be greater than that of the first year student group. Due to
the small sample size, a significance level of a = .05 was chosen for the statistical tests.
The power of the test (Hayes, 1973, pp. 419-420) was .94 based upon a sample size of

fifteen when the percentage of variance accounted for was .25 or more. The further

51

assumption for the power computation was that the approximation was based upon a

normal distribution.

To gain more information about the participating students, and to determine whether
the writing ability of the first and third year students was similar, the students’ answers
were analyzed using the computer program Grammatik III (Wampler, 1988). This
program indicated the reading level of what the students wrote. The idea behind using
the test was that, if the reading levels would be similar, then there would be one less

source of variation that could influence coding of the answers.

Second stage data analysis.
The primary goal of the second stage of data analysis was to search for explanations of

whatever findings were made in the first stage analysis. The search process proceeded
in two steps. First, even as the students worked on the essay questions, wrote their
answers, and concurrently made verbal comments, the principal researcher observed
their words and actions. While observing the principal researcher sat in a corner of the
office behind where the student was working. Notes could therefore be made without
cuing the participants as to desired responses. The only prompt that was given was the
statement: "Please remember to speak up.” Since the students each received the
questions in packets in which the questions had been sequenced at random, abbreviated
notations were adopted to allow the researcher to make unambiguous references to
the questions. Thus, in appendix "D", the four essay questions are labeled: "FF' (free
food); "LL” (Larry Landowner); "SNS" (Saturday night special); and 'TANK” (Tanker).
As indicated by the words in parentheses, the abbreviated notations were selected
based upon some of the facts in the essay questions. Second, all portions of the
videotapes relating to one of the questions ("FF“) were transcribed into typewritten
protocols. Both the notes of the principal author and the typewritten protocols were

searched for explanations of the first stage findings.

52

At each step of the second stage of data analysis, guidance was provided by Levi’s
definition of legal reasoning. The panel of coders used classifications designed to
indirectly indicate the use of Levi’s reasoning by analog. The principal author used
Levi’s definition directly as a guide when observing the students, viewing the video-
tapes, and examining the typewritten protocols. In addition to personally examining the
protocols, the principal researcher used a computer text search program (Zylndex,
1987) to determine if certain key words were used by the students. Key words were
chosen based on the likelihood of their appearance if a student was using Levi’s
reasoning by analog. The words chosen included "analog", "case", ”decision", "exam-
ple”, "common law", ”rule”, and "principle”. When any of these words were located by
the computer search, the principal researcher re-examined the passage to assure that
the material had been not been missed during the researcher’s examination of the

protocol.

The second stage of data analysis also provided an opportunity to obtain data or insights
not based upon Levi’s definition of legal reasoning. Since collection and analysis of
concurrent verbal comments went beyond what had been done in previous studies, so
also efforts were made to remain open for the possibility of new and unexpected
findings. For example, it may have become clear that, contrary to expectations based
on previous studies, legal reasoning is learned only very late in law school. First year
students might mimic an automated process and only much later realize the underlying
legal reasoning implications. Such a finding would be difficult to make based upon
examination only of the results of participants’ problem solving but it might appear in
the more complete data preserved by thinking aloud while problem solving. In any case,
such a finding would be an example of a pattern of problem solving that might be

discovered in the second stage of data analysis.

53

This chapter has detailed the design of this study, the types of data to be collected, and
the methods of analysis to be used. The essay question data, the thinking aloud
protocols, the researcher’s observation notes, and the background data all were aimed
at collecting data about the legal reasoning of first and third year law students. The
visual inspection techniques, the statistical analyses, and the computer searches all were
selected as means of comparing the data about the students’ legal reasoning. The

findings from all this will be presented in Chapter IV.

CHAPTER IV
FINDINGS

introduction.

The findings are subdivided into three subsections corresponding to three groups of
data: written essay scores; thinking aloud data (including thinking aloud protocols and
the researcher’s observation notes); and group background data. Each of these three
groups of data were examined to answer the research question of how the legal

reasoning of first year students compared to that of third year students.

The researcher’s expectation was expressed in the alternative hypothesis that third year
students’ legal reasoning would be better than that of first year students. As summarized
in table 4.1, the findings from the three groups of data do not support the researcher’s

expectation.

Table 4.1

Findings Summary

 

The overview provided by table 4.1 will be developed in more detail in the next three
subsections. As indicated in part by table 4.1, those subsections will present the

54

55

following findings. First, the written essay score data revealed that the first year average
scores, compared those of the third year students, were higher for each of the four
questions. Because the first year students scored higher, and because this was exactly
the reverse of the researcher’s expectation expressed in the directional alternative
hypothesis, no essay results could have been statistically significant using a one-tailed
t test. If the alternative hypothesis had simply been that the means of the first and third
year student groups were not equal, then a two-tailed t test would have indicated that,
for one question, the difference would have been statistically significant. This finding
of significance for one question would not change if analysis of covariance techniques
were used with undergraduate grade point averages and law school admission test
scores as covariates. Likewise the results changed little if the seven point essay scoring
scale was recoded into two dichotomous scales and then reanalyzed. Second, the
thinking aloud data provided more data on the legal reasoning of the students. How-
ever, no pattern of differences was found between the first and third year students.
Finally, the group background data revealed no significant differences in the groups’

undergraduate grade point averages, law school admission test scores, or writing levels.

Written essay SOOI'OS.

Written essay scores were the first data examined to determine whether the legal
reasoning of first year students differed from that of third year students. Group average
scores were computed for each question and then compared for differences in the
performance of the first and third year student groups. In addition, estimates of
reliability and tests of significance for the scores also were computed. These computa-

tions produced the following findings.

z>mz

I'I'JWOO

 

\ .

 

 

\V /

 

 

 

\ /

 

 

 

l l I l
LL FF SNS TANK

QUESTIONS

Figure 4.1 Group Mean Essay Scores.

3rd yr.

ist yr.

57

k\\\\\\\\\\\V
3rd yr

1 st yr.

 

 

 

 

 

 

 

 

 

 

 

 

 

12

10

8 6 4 2 o

FREQUENCY

SCORES

Figure 4.2 LL Scores.

«nzmcomwm

12

10

 

W
3rd yr.

 

   

-
1 st yr.

 

 

 

 

 

’/////.

/

 

’.///

 

 

.a—i

 

(a:

 

 

 

 

SCORES

Figure 4.3 FF Scores.

t\\\\\\\\\\\\\‘

3rd yr.

1 st yr.

 

 

 

 

 

 

 
   

 

 

SNS \\\\\\\‘\\

SIS

 

 

 

 

 

 

 

12

10

F

R E Q U E N C Y

14

12

SCORES

Figure 4.4 SNS Scores.

-<OZU1COIT17U‘11

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

12
W
3rdyr.

1° -

1styr.
a
6 \
4 C
\
§
\
8‘ Q L
\
0 I i ll | lg; l |

l l
9 11 13
2 4 6 8 10 12 14

SCORES

Figure 4.5 TANK Scores.

61

Group difference findings.
The group average scores for each question are graphed in figure 4.1. Visual inspection

of this graph leads to two findings. One finding directly addresses the question of group
differences. The other is the basis of later discussion about the type of reasoning being

used by the groups.

The first finding is that the first year students, compared to third year students, scored
higher on all four of the essay questions (appendix "D" - questions labeled by letter
abbreviations taken from the facts of the question). The difference ranged from a low
of .87 on the "SNS" question to a high of 1.87 on the "LL" question but the direction of
difference was always the same. In addition, the highest average score of the third year
students was on the ’Tank" question and yet that average score only equals the lowest

average first year score which was on the "SNS" question.

The second finding is that the highest average score is 8.87 and that score came from
the first year student group. This average score is some evidence of the reasoning
process being used by the students. Since two scorers evaluated each question, and since
their scores were added together for reporting purposes, 3 average score of 8.87 could
be produced by each scorer reporting a average score of 4.435. A score of ”4" corre-
sponded to the use of rule with the facts of the case and a ”5" to use of a case without
facts. This finding about the central tendency of the data can also be made by visually
examining the distribution of the data in figures 4.2 to 4.5.

Data reliability and significance findings.
The reliability and significance of the essay score findings were checked by estimating

the scorer reliability indices and by running various tests of statistical significance. The

findings from these steps are presented in the next two subsections.

62

Scorer reliability lndlcee.Comparisons based on the average scores were
checked by computing scorer reliability estimates. Table 4.2 sets forth both the Pearson
Product Moment correlations and the Spearman-Brown reliability estimates. "FF" has
the highest correlation coefficient at .824 and thus the highest reliability estimate at
.903. The lowest set of numbers are for "Tank" at .513 and .678.

Table 4.2

Scorer Reliability Measures

 

Statistical tests of slgnlflcance.Finally, the essay score differences were
tested using several statistical tests. The first test was Student’s t . As mentioned earlier,
no statistical tests were required to reject the alternative hypothesis because the first
year students unexpectedly scored higher on all four questions. As an aid to further
interpretation of the data, however, Table 4.3 sets forth the results of a two-tailed test
for each of the four questions. Using .05 as a decision point, only the results for the
"Tank" question show a statistically significant difference between the scores of the first
and third year students.

63

Table 4.3

Essay Score t Tests

 

Jam—u EF ENS TANK—
rou 1§t E 3rd y; 1§t 2; 3rd 3 lg; n firg E 1st fl erg y;
Mean 8.867 6.8 7.333 6.6 7.2 6.333 8.6 7.2

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3‘39“?” 3.523 2.651 2.375 2.898 2.336 2.024 1.920 1.320

ev12_ttron

Standard Error 0.910 0.685 0.613 0.748 0.603 0.523 0.496 0.341
Student’st -1.816 -1.172L -1.086 -2127
Prob >14 0.080 0.251 0.287 0.028
Degrees of 28 28 28 28
Freedom

 

In addition to the t test, an analysis of covariance was applied using undergraduate grade
point averages and law school admission test scores as covariates. The probability levels
associated with the univariate F tests were: FF - 0.257; LL - 0.144; SN 8 - 0.404; and
TANK - 0.034 (squared multiple "R" of .204). Thus, for the TANK question, the mean
scores for the first and third year students differed significantly even when undergrad-
uate grade points and law school admission test scores were used as controls. However,
when all four dependent variables were included in a multivariate analysis of covari-

ance, the Hotelling-Lawley trace statistic was 0.255 and thus not significant.

Statistical tests also were applied to the recoded scores produced by splitting the seven
point scale into two dichotomous scales. One scale was based upon whether a student’s
answer used facts or not. The other scale was based upon whether the answer used a
rule or a case. Thus a student’s answer to, for example, the "LL" question would produce
an "LL” score from one to seven, a "LLFACI‘S" score of 0 or 1, and a "LLRULE" score
of 0 or 1.

64

Pearson product-moment correlations showed that, in general, the two dichotomous
scales were weakly correlated with each other and more strongly correlated with the
seven point scale. For example, the scores for question "LL" on the seven point scale
had correlations of .436 and .836 with the corresponding scores on the dichotomous
scales but the corresponding dichotomous scales had a correlation of only .139 with
each other. The full correlation matrix is presented in appendix "I”.

The scoring results on the dichotomous scales were tested again using analyses of
covariance with undergraduate grade point averages and law school admission test
scores as covariates. The test for FFFACI‘S, at .046, was the only set of scores to show
a significant difference between the first and third year students. A corresponding t test
is not reported for FFFACI‘S because for that set of scores, as well as for TANKFACI‘
and TANKRULE, one of the group scores had a standard deviation of 0.0 and therefore
the STATS module of SYST AT (Wilkinson, 1988) did not complete the computations.

Thinking aloud data.

N 0 pattern of differences in the groups’ legal reasoning was found either in the thinking
aloud typed protocols or in the researcher’s notes made during the thinking aloud
process. However, some data was relevant to a determination of the type of legal

reasoning being used by the students.

Typed protocols.
The texts of the thinking aloud protocols first were examined by the researcher. Search

results then were supplemented by computer search techniques. The findings from each

technique are presented separately in the next two subsections.

Researcher’s examination of the protocols.The researcher’s examination

of the protocols was guided by Ievi’s description of legal reasoning. The first step was

65

to find evidence of Levi’s legal reasoning by analog. The evidence for each group of

students then could be examined to detect differences between the groups.

As discussed earlier, Levi distinguished legal reasoning from pure deductive reasoning.
Rather than selecting and applying a rule, legal reasoning, according to Levi, developed
the applicable rule by comparing the facts of the present problem to those of previous
cases. To find evidence of this process, the protocols were searched especially for

citations to previous cases and their facts.

The search revealed that none of the fifteen protocols contained evidence of all three
steps of Levi’s process. Nevertheless, the protocols can be grouped in accordance with
Levi’s description of legal reasoning. To assist the reader, all references to protocols
will be by use of the randomly assigned numbers which identified them in the research
process but an extra digit has been added to indicate that the protocol is from a third
or first year student. Thus protocol number 700 became 3700 and 906 became 3906 to
indicate they were from third year students. Likewise 942 became 1942 and 50 became
150 to indicate first year student authorship.

Five of the fifteen protocols contained evidence of using a prior case by analog. Three
protocols (3700, 1942, and 1713) contained that evidence in response to both parts "A”
and "B" of the question. One contained it only in response to part "A” (1475) and one
only for part "B” (3906). In two of the five protocols, 3906 and 1942, the evidence was
only in the thinking aloud portion of the protocol and not in the student’s written

answer.

Protocol number 3700 is an example of the type of evidence found:

66

I’m gonna play around with the Hadley v Baxendale issue which limits
damages. Although it’s a contract case, maybe I could somehow correlate
the two. I can’t, I really can’t use Palsgmf because that means you’re
liable for all damages that result from a tort, so I’m gonna try to use

Hadley.

The balance of the protocol further developed the reasons why the Hadlexl case was
not directly applicable and yet could be used to advantage to address the essay question
presented. Thus the protocol developed both the facts of the present case and those of
the previous case. However, neither it nor any other protocol illustrated Ievi’s next
step of explicitly comparing those sets of facts to develop a rule. Instead, the rule of the
prior case was stated and then applied to the present problem. Two other protocols
(3680 and 1349) contained arguments based upon analogies. However, both based the
analogies upon factual hypotheticals rather than upon previous cases. For example,
protocol number 3680 argued that the property owner would have a duty to safeguard
persons leaving the gm from street hazards since the owner already had a duty to

safeguard from other hazards like fighting and smoking.

Two protocols at least mention a potential analog. One, 3223, contains the statements:
"It wasn’t like a slip and fall.” and ”I wonder if you could go as far as saying this was like
an attractive nuisance." Iikewise number 1688 said "it could be classified as an attractive
nuisance” and also referred to ”tort law", ”invitee" and "licensee". If ”slip and fall”,
”attractive nuisance”, etc. are taken as short hand references to a line of established
cases, then these references would qualify as evidence of Levi’s reasoning by analog
with cases. Even if the references are taken only as short hand statements of legal rules,

they still give evidence at least of using a rule by analog.

1 Hadley v. Baxendale, 156 Eng. Rep. 145 (Exch. 1854).

67

Ievi’s description of legal reasoning conceded that the law can sometimes appear to
be simply deductive reasoning. The other six protocols illustrate this possibility. For

example, protocol number 1564 stated:

You gotta be kidding. Ummm. This would be a lot easier ifI knew which
kind of course I was supposed to be writing this for but we’re going to
talk in, ummm, torts and address it from that standpoint. Ifwe address it
from that standpoint then, then we want to address the issue of negli-
gence and if we address the issue of negligence then I’ll wanna address
it a four element: duty, breach, proximate cause, and injury.

Later in the same protocol, the student declined an opportunity to draw an analog to
a prior famous case. The student said: ”I wanna talk about zone, danger and all that
shit.... I don’t wanna get into Cardozo and all that.” "Zone of danger” and Judge Cardozo
had been mentioned by other students in reference to the Ealsgmfz case. Instead of
developing an analog to that case, the student said: "the courts have held that a injury
must be foreseeable to, ummm, connect liability to the tortfeasor.... Here it is unfore-

seeable that a teenager eating supper in the building would be killed by an auto.”

The protocol providing the most abbreviated version of Levi’s (1949) process was that
of a third year student (3926). It stated: ”Again, the facts are not really important at all."

The resulting answer to the first part of the question was only four sentences long:

Plaintiff would contend that the owner of real estate such as that in
question (gm) may not delegate duties regarding the care and mainte-
nance of its rental property. That is, the real estate company would be
strictly liable for this nondelegable duty.

Public policy forbids the delegation of such duties when the landowner

2 Palsgraf v. Long Island R.R. C0., 248 NY 339, 162 NE. 99 (1928).

68

retains such a benefit that the delegation is not permitted. Here, the real
estate company retains such a benefit that they remain liable notwith-
standing their rental of the gm to the school.

N 0 facts, cases, or hypotheticals were in the thinking aloud protocol section correspond-

ing to this answer.

These last two protocols, 1564 and 3926, illustrate the range of ways in which a student
could use a general rule or principle to solve a legal problem. Instead of referring to
the facts of a prior case for an analog, as Levi would suggest, the student in 1564 first
selected a general area of law, torts, then a general area within torts, negligence, and
then a general rule of negligence law to apply to the facts of the problem. The student
in 3926 likewise uses what the student apparently considers a general principle to
answer the question but 3926 does not integrate the facts into the answer as 1564 had

done.

The other four protocols use rules in a manner that is within the range illustrated by
1564 and 3926. Excerpts from those four protocols will give the reader an indication of
what the students were doing. 3822 searched for ”an area of liability" which the student
could not remember. 3467 referred to ”theories espoused by the estate", "agency
theory”, "tenant situation" and "the rule of law that I can’t think of right now.” 1937
referred to the legal categories of "invitees, licensees, and trespassers", the legal theory
of negligence, and a "standard" which the student could not remember. 150 said "Kind
of attracted real little kids to the place" and therefore "the real estate company will have
a higher duty.”

Finally, one protocol, 3680, provided some evidence of Schon’s (1983; 1987) reflection-

in-action. That protocol began with comments that the second part of the question

69

would be easy to answer, proceeded with further discussion of the distinctions between
the two parts of the question, and then eventually realized that both parts of the
question could be seen as aspects of a deeper problem. This is the closest that any

protocol came to clearly illustrating Schon’s process.

The research design presented in Chapter III envisioned the use of Collins’ factors to
investigate how students found the analogies that they used. However, findings for this
part of the research design are not reported even though analysis using Collins’ factors
initially looked promising. Pursuing this type of analysis and reporting the results was
not done for a reason that can be best illustrated by referring again to protocol number
3700. That protocol’s selection of cases to cite can be explained without recourse to the
sophistication of Collins’ approach. Both Hadley and Balmaf are landmark cases in
the development of the law. A leading reference book on the law of torts calls Ealsgmf
"the most discussed and debated of all torts cases” (Keeton, Dobbs, Keeton, & Owen,
1984, p. 284). Hadley has similar fame in the law of contracts since it announces rules
that have been uniformly adopted by American courts (Simpson, 1965, p. 396). Since
protocol number 3700 was one of the better examples of the use of case analogies, and
since even that protocol’s interpretation did not require the application of analysis using
Collins’ factors, a decision was made not to pursue the originally planned stage of
analysis using Collins’ factors.

In summary, the researcher’s examination of the thinking aloud protocol data provided
little evidence of Schon’s process and little need to use Collins’ factors but some
evidence of legal reasoning as described by Levi ( 1949). Although none of the protocols
illustrated all three steps of Levi’s process, still they could be grouped in accordance
with his description of legal reasoning. Five protocols came close to illustrating Levi’s

process with prior cases. T’wo protocols used analogies to hypotheticals. Two protocols

70

at least mentioned the possibility of an analog. The rest of the protocols, six in all, gave
evidence of deductive reasoning but differed in the extent to which they used the facts
of the problem. However, all these groups of protocols came from almost equal
numbers of first and third year students. No pattern of differences was found between

the two groups of students.

Computer searches of the protocols.The computer program ZyIndex pro-
duced frequency reports for various word combinations. The word searches were on
words whose occurrence would be expected if a student were using Levi’s approach to
legal reasoning. Given Levi’s emphasis on the use of prior cases by analog, the words
selected for search were: analog (and its forms); case; decision; example; common
law; principle (and its forms); rule; Palsgraf (a landmark decision in the problem area);
and policy. For reasons that will be explained later, a search was also made for the letters
"IRAC‘. Those letters stand for "Issue, Rule, Application, and Conclusion".

Some of the search terms were completely absent from the protocols. For example, the
search specification for the word analog was ”analog". The asterisk serves as a wild
card so that all forms of the word will be retrieved. Thus analog, analogize, and
analogous would all be reported. ZyIndex reported no occurrences in the fifteen
protocols. The same was true for the word ”decision" using "decisio‘" as the search

specification. Likewise "principl‘" never occurred.

The most frequently used word was "rule". Fourteen of the fifteen protocols had at least
one occurrence. In five of the protocols, the word "nlle’ was used in the sense of "rule
of law” or "courts rule" (3467, 1937, 1475, 1942, 1713). The balance of the occurrences
reflected the use of the word "rule" in the problem itself. Thus many referred to the

"city rules” and the ”program rules”.

71

The next most frequently occurring search term was ”case”. Nine protocols used the
word ”case” at least once. In four of the protocols, 1475, 1942, 3700, 1713, ”case” was
used in the sense of a legal decision. Thus 3700 referred to the "Palsgraf case”. The other
five protocols used "case" in different senses. Examples include: ”in which case”; "in this

case”; ”make a case for”; and, "in case there’s a little something I missed".

"Palsgraf“, correctly or incorrectly spelled, occurred in four protocols (1942, 3700, 17 13,
and 3906).

None of the other search terms occurred in more than one protocol. "Example" was
used by 1937 in the sense of "for example”. The same protocol also contained the only
occurrence of the phrase "common law”. "Policy" was used by 3926 in the sense of ”public

policy argument”. Finally, "IRAC" was used in protocol number 1942.

In summary, "rule” was the most frequently occurring word followed by "case" and
”Palsgraf”. "Example”, ”common law”, "policy" and IRAC' never appeared in more than

one protocol. ”Analog", "decision”, and "principle” never appeared.

HOSGEI‘CI'IGI’S notes.
Only limited notes were taken by the researcher during the thinking aloud process but

those notes nonetheless made three contributions to this study. As an overview, the
three contributions were as follows. First, the notes support the finding that most of the
students were not demonstrating the use of Levi’s reasoning by analog. However, they
also provide evidence that some of the students may have been using Levi’s process
even if it did not evidence itself in their answers. Second, the notes support a finding
that at least one of the essay questions was so close to material studied by the first year

students that they probably had a decided advantage in attempting to answer it. Third,

72

the notes provide some evidence of uncontrolled factors which may have influenced

this study. Each of these three contributions will be treated in a separate subsection.

Evidence of Levi’s reasoning by analogy.The first contribution of the
researcher’s notes is that they provide further support for a finding that the students
revealed little evidence of using Levi’s reasoning by analog. As had been the true for
the written answers and the typed protocols, references by analog to previous cases or
decisions were absent from a majority of the students’ answers. However, a review of

the notes also reveals that the students’ answers to one question are an exception.

One essay question concerned the disputed ownership of buried money. The person
who buried the money, the owner of the land where it was buried, and the hunter who
found it, all claimed the money. While thinking aloud on this question, a majority of all
students, ten third year and eleven first year, referred to prior cases. However, the essay

scores show such use only by two third year and six first year students.

This same essay question also provides some evidence of students using Levi’s reason-
ing process even when they give no evidence of it in their written answers. Student
number 371 is one example. There the student spoke of two prior cases and explicitly
noted that the essay question’s situation was ”analogous" to those cases. However, the
student’s written answer was scored four and two by the two scorers. Iikewise student
number 3209 mentioned two prior cases and said: "Make Harry’s parallel that - but he’s

a trespasser." Again the written answer received a score of four from both scorers.

Evidence of Levi’s reasoning by analog was present in a few answers other than 371,
3209, and the two discovered in the thinking aloud protocols mentioned earlier. Review
of the researcher’s notes shows the following instances of a student referring to the idea

of using a case by analog even though it is not included in the students written answer:

73

3223 on 'Tank" question (scores of four and two); 3476 on ”SNS"(scores of four and
two); 3676 on 'Tank” (scores of four and four); and 3467 on ”SNS" (scores of one and
one). Thus the researcher’s notes reveal six instances, including 371 and 3209, where
there is evidence of Levi’s process even though it is not reflected in the written essay

scores. All six of these instances come from third year students.

Essay question valldlty.The second contribution from the researcher’s notes
is a finding that the first year students may have enjoyed a substantial advantage in

answering at least one of the questions.

The question that the first year students may have had an advantage on was the buried
treasure question treated in the last subsection. As noted there, a majority of both third
and first year students remembered prior cases like the question presented. However,
a number of first year students remembered details not mentioned by the third year
students. For example, two first year students remembered the name of the case they
had studied and one even remembered the name of the justice who wrote it. Another
mentioned that it had been the first case that they had studied in law school. No similar
references were made in the remarks of the third year students. Thus the more detailed
and recent memory apparently possessed by the first year students may have given them
an advantage in making it more likely that they would mention the case and thereby

enhance their scores.

Uncontrolled factors.The researcher’s notes provide some evidence that
news reports may have given some of the first year students an advantage in answering
the 'Tank" question. Two things happened to make this possible. First, the secretary
scheduling the thinking aloud sessions had given the third year students preference in
assigning times. Thus twelve of the first fifteen persons to do the questions were third

year students. Second, about one week into the month of sessions, student 942 men-

74

tioned that the 'Tank” problem sounded like something the student had heard about
on the news. At that point, twelve of the thirty sessions had been completed, including
eleven of the fifteen third year student sessions. Three other students later made similar
comments 1564; 150; and 1349. As indicated by the numbers, all of these students were
first year students.

Group background data.

Four types of background data were collected. Each of the four sets of data provided a
different basis for judging how similar the groups of students were on measures other
than the amount of legal education completed. Undergraduate grade point averages
[UGPA] and Law School Admission Test [LSAT] scores were two of the types col-
lected. In addition, the computer program Grammatik III computed a Flesch-Kincaid
Grade Level for the essay answers that the students wrote. These grade level figures
were computed to safeguard against the possibility that especially the essay scores might
reflect differences in the writing abilities of the groups rather than any difference in
their legal education. Finally, questionnaires completed by the students provided
information about: parents or friends who were attorneys; legal employment experi-
ence; and experience gained from practice type law school courses. This data was
collected to give an indication of how much an opportunity the students had to learn

legal reasoning outside of traditional law school courses.

Table 4.4 presents both the undergraduate grade point averages, LSAT scores, and
Flesch-Kincaid Grade Level figures for the two groups. Since the group averages
differed on each of the measures, the table includes the results of using Student’s t to
test whether the differences are significant. Those tests (again, two-tailed) confirm that
the group averages do not differ significantly.

75

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 4.4
Group Background Data
_Mmllrs PA EAT MEW—
rou 1§t 1; 3rd fl 1st 2 3rd E __1_st vr 3rd vr
Mean 2.8613 2.9153 529.87 499.53 10.4 8.6667
Standard
D . . 0.3753 0.3740 91.961 68.333 2.1974 2.7689
evratlon
Standard Egor 0.0969 0.0966 23.744 17.644 0.5674 0.7149
Student’s t 0.3947 -1.0254 -1.8991
Prob >th 0.6961 0.3139 0.0679
Degrees of 28 28 28
Freedom

 

 

 

 

Table 4.5 condenses the information collected on the student questionnaires. Third

year students gave more affirmative responses to all the questions. As might be

expected from the composition of the groups, nine of the third year students had taken

some of the practice type courses offered late in the law school curriculum. None of

the first year students had taken such a course. For the other questions, the numbers

were closer. For having a parent or friend who was an attorney, the numbers were seven

to six and for legal employment experience it was eleven to seven.

Table 4.5

Background Questionnaire Data

 

 

 

 

 

 

 

Group Attorney Parents Legal Practice Legal Employment
1st yr 6 0 7
3rd yr 7 9 1 1

 

 

 

In summary, the group background data indicates that third year law students had more

opportunities for exposure to legal reasoning in setting outside the traditional law

76

school courses but they did not differ in undergraduate grade point averages, LSAT
scores, or writing level as measured by the Flesch-Kincaid Grade Level method.

Summary.

As indicated earlier in Table 4.1, the findings do not support the researcher’s expecta-
tion, expressed in the alternative hypothesis, that third year law students would dem-
onstrate better legal reasoning, as defined by Levi, than the first year law students. The
finding of no difference between the legal reasoning of the first and third year students
was supported by the essay question scores, the thinking aloud protocols, and the
researcher’s observational notes. This result, the possible reasons for it, and its impli-
cations for both legal education and the teaching of problem solving more generally,

will now be discussed in Chapter V.

CHAPTER V
DISCUSSION AND CONCLUSIONS

Discussion

introduction.
The research question of this study asked how the legal reasoning, as defined by Levi,

of first year law students compared with that of third year law students. Essay question
answers, thinking aloud protocols, researcher observation notes, and student back-
ground data were collected. Visual inspections, statistical analyses, and computer

searches examined the data and compared the results for the two groups of students.

The findings of this study are consistent with the prior work of Bryden (1984). As
previously noted, Bryden raised the possibility that the small differences he found
between first and third year law students would disappear if the first year students had
the benefit of a little more legal education. This study introduced such a change. Instead
of using beginning students as Bryden had, students were selected with twenty credit
hours completed over two semesters. As Bryden would have predicted, no difference

was found.

The expectation in this study, expressed in the alternative hypothesis, was that third
year law students would demonstrate superior performance if Bryden’s research ap-
proach were modified. This study explored modifications regarding both the definition
of what was being measured and the different data collection techniques which could
be used. After reviewing a number of alternatives contained in the literature, Levi’s
(1949) definition of legal reasoning was adopted. The literature likewise provided
multiple data collection and analysis approaches. If the expected result had occurred
and third year students had performed better, then Bryden’s questions about legal

77

78

education would have been largely answered and this study would have been additional

support for the proposition that expertise can be taught.

The expectation of this study was not fulfilled. N 0 difference was found between the
legal reasoning, as defined, of first and third year law students. Reasoning by analog
data was sparse so there was little to which Collins’ factors could be applied. Schon’s
reflection in action was almost totally absent. Moreover, although the findings are
consistent with the work of Bryden, it is difficult to accept the null hypothesis because
those same findings appear counter-intuitive. From the perspective of a teacher, it
would seem that a group with an average of 74.3 credit hours of legal education would
do better than a group with an average of only 20.5, especially when what is being
measured is what law schools profess to teach, that is, legal reasoning. A contrary
finding of no difference might be taken as a direct challenge to present legal education
methods. In addition, as discussed in the review of the literature, it would provide
further support to those who question whether thinking and problem solving skills can
be taught.

Rather than accept the null hypothesis, this discussion will review other explanations
for the findings. Once again, the review of the literature is a fertile source of possible
explanations of why this study’s expectation was not met. Thus this discussion will begin
with a section considering the possibility that group selection procedures are respon-
sible for the finding of no difference between the first and third year students. Then the
following sections will inquire whether the responsible factor or factors stem either
from the way legal reasoning was defined or from the research design that was chosen.
Those sections will be followed by a summary and by conclusions based upon the

discussion.

79

Group selection procedures as an explanation.
The first possible explanation of this study’s findings of no difference between the

groups is based on the argument that the findings are not a reflection on legal
education’s effectiveness, either at the school studied or others, but rather are a mere
artifact of the two particular groups which were selected and studied. This possibility
is perhaps the most difficult to dismiss and, ultimately, perhaps must be simply acknowl-
edged based upon the practical limitations of research in general and this study in
particular.

One of the more concise presentations of this possibility’s underlying reasons is
included in the work of Campbell and Stanley (1966). In their terminolog, the essay
score analysis in this study would be a static-group research design for which the internal
and external threats to validity are well defined. One of those internal validity threats
is the "selection" factor which arises from the failure to have complete randomization

in the process of choosing participants.

The ”selection” factor, the possibility that differential recruitment influenced the
composition of the groups studied, cannot be ruled out in this study. Here there was
little information on why some students participated and why many others did not.
Students mentioned time problems even though sessions were held at any time conve-
nient for the student. Other said that the $25.00 was not enough compensation com-
pared to their outside employment prospects. Unmentioned but probably present was
a fear to expose their abilities to a professor whom they might have for a teacher in a

future class. Also unmentioned but likely to be present was a hope by some to obtain
special help in test taking.

80

Each of these reasons could lead to the selection problems raised by Campbell and
Stanley. Students with great time pressures might be more apt to be top or bottom
performing students. Iikewise employment concerns might be more important for
bottom students, because they would be less likely to have scholarships, or top students
because of their enhanced employment prospects. However, little data was available
regarding the reasons for student participation and no formal data was available about

what implications those reasons might raise.

Selection of students at only one law school also brings with it threats to the external
validity of this study. The possibility exists that the institution studied was not typical
and therefore the results should not be generalized to other schools. As noted by
Bryden (1984), it is of some help that most law schools use many of the same textbooks

and often teach from them in similar ways. However, generalizability concerns remain.

Campbell and Stanley (1966) also explain that all threats to validity cannot be over-
come. Some assumptions remain, and might be false, even in the best designed study.
Here, practical constraints necessitated more than the minimum number of assump-
tions. Therefore it must be allowed, especially for the statistical analysis of the essay
scores, that the way the students were selected may be responsible for any difference
or lack of difference found in this study. The balance of this discussion is presented in
the context of the limitations raised by Campbell and Stanley.

The definition of legal reasoning as an explanation.
The second possible explanation of this study’s finding of no difference between first

and third year students’ legal reasoning is based on the idea that this study attempted
to measure the wrong educational outcome. Both the review of the literature and the

data provide some support for the possibility that the "legal reasoning” supposedly

81

learned by law students is, not Levi’s (1949) legal reasoning by analog, but rather a

further application of the students’ common sense to a new substantive knowledge base.

The review of the literature included references to scholars who would view legal
reasoning as common sense. Included in those references were Fejfar (1986), Lonergan
(1957), and Minsky (1986). If legal reasoning is basically common sense, and if law
students come to law school already equipped with those skills, then an alternative
educational outcome from a legal education would be acquisition of a body of substan-
tive knowledge. As noted in the review of the literature, this would be consistent with
the medical education findings of Elstein et al. (1978) and F eltovich et al. (1984). It
would also be consistent with the view of legal education that students sometimes have

(White, 1986).

In addition, the data here provide some support for the possibility that the definition
of legal reasoning used in the study is responsible for the finding of no difference. For
example, the computer word searches revealed no occurrences in the transcribed
protocols of the word "analog." Also, the first year students did best on the "LL"
question which was most closely related to the material that they had already studied.
In addition, even the highest scores of the groups averaged only between "4" and "5"
whereas students using Levi’s reasoning would be expected to score at least "5.” These
findings do not decide the issue but they are consistent with the view that expertise
acquisition is more the learning of a large data base of information than the learning

of a skill like Levi’s reasoning by analog.

At first glance, these challenges appear to go to the heart of this study. The basic
assumption here was that law schools teach students to do "legal reasoning” and that

Ievi’s ( 1949) reasoning by analog was the best description of what that meant.

82

However, even if students come to law school with some of those skills already in place,
and even if learning a substantive body of knowledge is a primary goal, still the data
and results in this study are not necessarily rendered superfluous. It can even be argued

that the results become more interesting if these challenges are correct.

If law students already know how to reason by analog, then it would seem even more
likely that third year law students would achieve higher scores on this stu‘dy’s measures.
The third year students’ presumably larger body of legal knowledge would form the
basis for more analogies than would be available to a first year student. Thus it might
be argued that the challenges raised in this section, far from explaining the finding of
no difference in the performance of the first and third year students, actually provide
additional incentive for seeking explanations of why so little reasoning by analog data
was obtained. Even if the study’s basic assumption about the nature of legal reasoning
was incorrect, still the data collection methods cast a wide enough net to catch any
differences between the two groups of students. Therefore this set of challenges raise
important questions but, by questioning the basic assumption of the study, they do not
automatically provide an explanation for why this study found no difference between
the two groups.

Research design as an explanation.
Even if this study’s student selection procedures were appropriate, and even if assump-

tions about the definition of legal reasoning did not compromise the results, the
possibility still exists that aspects of this study’s design would cause it to fail to detect
differences between the first and third year students’ legal reasoning. The findings in
Chapter IV prompt consideration of at least five aspects of the study’s design as
candidates to explain the study’s results. First, it may be that the design did not motivate
the students sufficiently to elicit performance on the level addressed by Levi’s defini-

tion of legal reasoning. Second, the measurement scales used for scoring the essay

83

questions may not have correctly implemented Levi’s definition of legal reasoning.
Third, the design might have actively interfered with what was being measured. Fourth,
the study’s measurements may have been collected at the wrong time. Finally, the
study’s design may have been weakened by the absence of additional control groups.
Each of these possibilities will be discussed separately in the following paragraphs.

Student motivation is one possible explanation for this study’s results. The level of
student motivation might have been important in at least two ways. One is that the first
year students, having been more recently introduced to legal reasoning, might have had
more motivation to demonstrate their ability to use the new way of thinking. This
additional motivation might have increased first year student performance enough to
mask any difference that the third year students might have otherwise shown. The
second way that motivation might have been important is if neither first or third year
students were sufficiently motivated to perform very well. The uniformly low level of

performance again might have masked the presence of differences between the groups.

This study included no formal measures of student motivation. However, the
researcher’s observations of the students during data collection provide some anecdotal
data about student motivation. Based on that data, it can be said that some students
appeared to throw themselves completely into the task while others appeared willing
to take the participation money and run. Moreover the latter category appeared to be
a small minority of both the first and third year student groups. No other motivational
patterns stood out from the data. However, due to the type of data used in drawing
these conclusions, the possible role of motivation cannot be ruled out as an explanation
for the results here. Even though it would be contrary to the researcher’s personal
impressions, it may be that no differences were found between the first and third year

students’ legal reasoning because of the influence of motivation on group performance.

84

The essay scoring scales are a second aspect of the study’s design that may explain the
finding of no difference between the first and third year student groups. The seven point
scale was developed from Levi’s definition in order to achieve more reliable scoring
than had been achieved in previous studies where legal ”issues" had typically been
counted. To explore whether the seven point scale was really one ordinal scale or a
combination of several dichotomous scales, the data was recoded into two dichotomous
scales and the results were compared with the results from the seven point scale. The
two dichotomous scales were based upon two key elements of Levi’s definition - the
use of facts and the use of cases. The seven point scale’s classifications used combina-

tions of these elements rather than separating them into two different scales.

On the one hand, the findings indicate that the scales worked well. Even though the
students used reasoning on a lower level than expected, the scales were broad enough

to describe and capture the process in a way that could be analyzed as planned.

On the other hand, the scales did not appear to ease scoring decisions as much as had
been hoped. Even after being checked out at the orientation session with the re-
searcher, each of the scorers called the researcher at least once to clarify how the scoring
classifications should be applied. Most of the questions concerned how much evidence
was necessary to conclude that a student’s answer was using a legal rule or a case. The
possible combination of factors proved to be extensive. Thus, rather than eliminating
the ambiguity involved in deciding what an "issue" is and how it should be counted, the
scoring scale may have merely placed the ambiguity into a new setting. Naturally
ambiguity from any source would introduce measurement error which could conceal

the effects that were being measured.

85

Even though the scales appeared to be broad enough to measure the form of legal
reasoning used by the students, it also appears that the breadth of coverage may have
sacrificed the precision needed to measure variations among the students. Particularly
on the 'TANK" question, the histogram of the scores shows little variability. Thus it
appears that the scoring scale may have been too coarse a scale to make fine distinctions

among student answers.

A third way that aspects of the study’s design could have been responsible for the study‘ 5
results was if the design interfered with the data collection process. Design interference
could take several forms, each of which might so interfere with data collection so as to
explain why no difference was found between the legal reasoning of the first and third
year students. The first way that the design could interfere would be if the students
mistook the data collection process as a legal examination for which they had learned
a specialized method of response. Bryden (1984) recognized the possibility that a legal
examination might evoke such a response from students. As a consequence, this study
was designed so as to reduce the likelihood that the problems would be viewed as part
of a legal examination. For example, the students wrote on lawyers’ yellow pads rather
than bluebooks and they worked at an office desk. Nevertheless, it was clear from
various remarks of the participants that the ”examination factor” had not been removed.
Thus, in one protocol, the student said: "Can’t panic here just because I’m taking an

exam.”

Another protocol provided a clue as to how the students might respond if they believed
that they were taking an examination. That protocol, number 1942, referred to IRAC,
an acronym standing for: Issue, Rule, Application, and Conclusion (Kelso & Kelso,
1984, p.481). Although perhaps suitable for some contexts, IRAC would appear to favor
deductive reasoning over Levi’s reasoning by analog. Searching student outline banks

86

and similar materials was not a part of this study’s design. An informal review of several
student course outlines was at first assuring in that it showed that students had been
exposed to typical lines of appellate cases which are used to teach reasoning by analog.
However, the informal review also supported the possibility that an IRAC type ap-
proach to examination questions may be part of law school’s hidden curriculum. As
noted in the review of the literature, Gross (1984) details how students may be learning
an oversimplified legal reasoning process for use on examinations. One key part of the
oversimplification is that students work with abstracted legal rules rather than with the
authority itself (pp. 426-427).

If this study’s data collection process was viewed by students as a legal examination,
and if students learn a specialized method of response for legal examinations, then that
might explain why no difference was found between the first and third year students.
The specialized responses might not accurately reflect what was being learned in law
school. The students might be learning Levi’s (1949) reasoning by analog but the
study’s design might interfere with their demonstration of it because a different
response would be triggered by the data collection procedure. The students also might
not be learning Levi’s reasoning by analog, as Gross would maintain, but instead an
oversimplified process of using abstract rules. In either case, essay questions would
trigger exactly the kind of rule dominated responses that were found in this study. As
noted earlier, the essay scores averaged, at best, between "4" and "5" with a "4" denoting
the use of rule with facts and a "5" denoting the use of a case alone. Use of a case, without

its facts, is like using the case as shorthand for a rule in the manner described by Gross.

Rule dominated responses also may have occurred because, as noted earlier, the
students may not have been sufficiently motivated to engage in the more involved

reasoning demands of Levi’s process. However, whether caused by lack of motivation

87

or the triggering of a specialized response, an additional impact of receiving rule
dominated responses is that the use of rules rather than case analogies leaves little room
for analysis using Collins’ factors. Iikewise a lack of motivation or the triggering of a
specialized response could explain why Schon’s reflection-in-action did not appear

more often in the data.

The design of this study may have interfered with data collection in another way. Going
beyond the problems of students thinking that they were taking an examination, Groen
and Patel (1988) maintain that two factors are responsible for the results of investiga-
tions, like this one, which are designed along the lines of studies such as de Groot (1965)
and Elstein et al. (1978). The first factor is that pattern recognition methodolog, which
proved useful in studying chess problems presented on de Groot’s chess board, is not
well adapted to verbally complex domains. In other words, chess pieces on a board may
present a pattern for recognition but abstract words present a different type of problem
requiring different methods. The second factor is that nonroutine, difficult problems
will trigger the backward (hypothetico—deductive) reasoning found by Elstein et al.
(1978). Groen and Patel (1988) themselves found forward reasoning when a different

research design was used.

Here backward (deductive) reasoning was found. Largely absent was Levi’s reasoning
by analog which, according to Levi (1949), is an imperfect form of forward or inductive
reasoning. Moreover the essay questions used in the study deliberately included
difficult portions so as to avoid the automated processing described by Johnston and
Afflerbach (1985). However, it would seem that the easy portions of the essay questions
would have avoided difficulties raised by Groen and Patel. Since students provided
little evidence of Levi’s reasoning by analog even for the easy portions of the questions,

88

Groen and Patel’s two factors may not be a complete explanation of why no difference

was found between the first and third year students.

Finally, the design of this study may have interfered with data collection due to the
peculiar demands associated with writing an essay response while thinking aloud. Prior
research has coupled thinking aloud with primarily mental activities such as problem
solving (Ericsson & Simon, 1984) or reading (Johnston & Afflerbach, 1985 ). However,
prior research leaves open the possibility that a task like writing may interfere with the
collection of thinking aloud data or vice versa, especially when the task is being
performed by a person who is not yet an expert. Here the data also raised the possibility
of this type of interference because several participants commented on the difficulty of
both writing and thinking aloud. Likewise some participants wrote and thought aloud
in separate steps, not concurrently. An example of how this type of interference could
be important is provided by data in the researcher’s notes showing participants using
analogies even though they did not include them in their written answers. Since all six
of those participants were third years students, the absence of the analogies from their
written answers, and thus from their scores, might have had an important impact on
statistical comparisons of group scores. However the analysis of the data did not focus
on this source of interference and thus the possibility of its impact here can be

acknowledged but not resolved.

A fourth reason why students may not have manifested Levi’s reasoning by analog,
even though they had learned it, is that the process involved may not be a monotonic
function of learning, that is, that persons who have learned a certain amount may not
perform as well as persons who have learned either more or less than that amount.
Lesgold, Glaser, Rubinson, Klopfer, Feltovich, Wang (1988) give the example of a baby

whose locomotion may become less efficient when automatic and secure creeping gives

89

way to toddling. In their own work, Lesgold et al. (1988) found that third and fourth
year radiologist residents performed less well than either experts or first and second
year residents on some films. Strauss and Stavy (1982), quoted by Lesgold et al., state
that one possible reason for nonmonotonicities is: "[o]scillation between a familiar but
inadequate mental representation of the problem situation, and an improved but still

new and ’untrusted’ representation system which is correct."

Here the findings of Lesgold et al. might explain why even third year students used very
general rules to solve the problems rather than applying the more sophisticated
reasoning process which presumably they had learned. If, as Herbert Simon (Chase &
Simon, 1973) has proposed, it takes 10,000 to 20,000 hours (five to ten work years) to
become a chess master, then third year students might be at too early a point in their
development to show a difference based upon the change in their reasoning. If a
nonmonotonic process is involved, the third year students might even do worse than
the first year students on some questions, as was the case here and for Groen and Patel’s

radiolog residents.

The fifth aspect of the study’s design that might affect at least the interpretation of the
results is that no data was collected either from pre-law students or from lawyers several
years after their graduation from law school. The presence of these control groups
would have shed light on a number of the possible interpretations of the data. For
example, the possibility that law students come to law school already equipped with
legal reasoning would have been checked by data from pre-law students. Likewise the
presence of law graduates with several years experience would have checked the
possibility that a nonmonotonic process was being learned. However, for the present,

the contribution of such control groups must await future studies.

Summary and discussion.
This discussion began with a recognition of the possible conclusions that could be drawn

from a finding of no difference between the legal reasoning of first and third year law
students. It was recognized that those conclusions could go to the heart of both present
legal education methods and the possibility of teaching expertise in general.

Since the present system of legal education has been in effect for about one hundred
years (Stevens, 1983), and since it therefore can arguably be entitled to a benefit of the
doubt, this discussion has focused on explanations of the finding of no difference

between the groups and yet not overthrow legal education as we presently know it.

Reviewing the three sections of challenges discussed above, it would appear that a
"middle of the road" interpretation of the findings is probably the best approach.
Extreme interpretations are, of course, possible. The limitations explained by Campbell
and Stanley (1966) can be used to attribute the findings to peculiarities of the individual
students studied with no justification for drawing any wider conclusions either for the
law school involved or for law schools in general. At the other extreme, one could argue
that the finding of no difference, even though one group had about three and a half
times as much instruction, means that this study is one more on the list of studies which
prove that it is misleading to claim that expertise can be taught. At a minimum, this
would mesh with previous calls for reducing the law school curriculum from three to

two years (e.g., Packer, & Ehrlich, 1972).

In between the extreme interpretations is the possibility of interpreting these findings
in a "middle of the road” fashion. From the one extreme, that interpretation could allow
that the study’s design, while not rendering the data superfluous, nevertheless could be
modified both to better address the problem of motivating participants and to reduce

91

the "examination factor”. For example, the use of simulated clients might be seen by
students as being more realistic and thus increase motivation while still allowing data
collection without students automatically applying a specialized set of responses that
they may have learned for examination taking. From the other extreme, that middle
position interpretation could allow that legal education may contain a "hidden curric-
ulum" at variance with what the process is claimed to be. For example, it may be that
knowledge base acquisition and deductive reasoning play a larger role than is often
acknowledged. This could lead to measurement tools more focused on the knowledge
base acquisition and deductive reasoning rather than upon legal reasoning as defined

by Levi.

In addition to taking bits of wisdom from more extreme interpretive positions, a middle
position might also concentrate on the possibility that a nonmonotonic process is being
learned and that therefore increased learning will not always be accompanied by
immediate improvements in performance. Although this would be frustrating for both
students trying to learn expertise and teachers trying to measure that same learning,
still it would appear to better represent what is involved in the acquisition of expertise.

Finally, taking a middle position would allow these findings to be more easily integrated
with the findings of Bryden (1984). That study used traditional legal education mea-
surement techniques. Issues were counted and totals compared for beginning and third
year students. Here data was collected on the level of how issues are identified, on the
level of legal reasoning by analog as defined by Levi (1949). Uses of analogies by first
and third year students were counted and compared. In both studies, no significant

differences were found when the data was examined as a whole.

92

Although neither Bryden’s study nor this one can be taken as dispositive, still both
together can be read as presenting a stronger case that some modifications would
strengthen legal education. Bryden urged greater use of programmed learning. This
study would support at least greater recognition of the role of knowledge acquisition.
It does not seem sufficient to rely on teaching legal reasoning or "thinking like a lawyer"
as a set of skills apart from a large knowledge base.

In addition, both studies can be read as supporting the idea, often expressed before,
that legal educators might spend more time explicitly describing the process they are
teaching. Absent that explicit teaching, it appears that students may be quickly learning
a specialized response to essay examinations. Although in one sense that shows that
legal education can be very effective, it appears that the specialized response is learned
more as part of a hidden curriculum than as an explicitly desired learning outcome.
Moreover, it appears that such a specialized response, since it treats the law superfi-
cially as a set of overly abstracted rules, and since it does not involve use of law’s richer

resources, would be detrimental to students in their future practice of law.

Any changes in legal education will encounter predictable obstacles (Cramton, 1982).
However, attempting to overcome such a specialized response by continuing to teach
legal reasoning implicitly through the use of the case method may be effective for some
but it probably is not the best method, particularly if a nonmonotonic process is being
learned. Since a nonmonotonic process may delay positive feedback about what has
been learned, students might become discouraged when long hours of work appear to
produce no positive results and even seem to make matters worse. They may opt for
what appears, at least short term, to work better. The fact that some students might
succeed despite these obstacles would be of little comfort to educators if it appears that

93

the educational system could do more for many others to whom that educational system

has held itself out as a source of help.

Conclusions

Reviewing the preceding findings and discussion, some conclusions will be drawn about
the questions addressed in this study. Those conclusions then lead to two levels of
recommendations. On one level, recommendations can be made concerning future
work on these questions. On another level, recommendations can focus on the broader

challenge of teaching expertise.

Many possible explanations could be responsible for this study’s finding of no differ-
ence between the legal reasoning, as defined by Levi (1949), of first and third year law
students. This researcher concludes that the most likely candidates are interference
between the task of writing and the thinking aloud process, the ”examination” factor
with its associated specialized response, and the nonmonotonic nature of expertise
learning. These appear the most attractive alternatives, not only because they would
account for the results, but also because they would most directly lead to future

remedial action.

If the results are due to interference between writing and thinking aloud or to partici-
pants thinking that they are taking a law school examination which calls for a specialized
response, then data gathering can be modified to better capture the reasoning that they
are learning. The suggestions of Groen and Patel (1988), for example, would attack part
of this problem. If a nonmonotonic process is being learned, then longitudinal studies
could receive more emphasis. To the extent that repeated measures present special

problems and opportunities, reference can be made to the testing and measurement

94

advice of Willett (1988). In any case, conclusions and recommendations on this level
allow practical steps to be taken toward resolving the larger questions of how problem
solving and expertise in general can be defined, taught, and measured.

On a broader level, this researcher declines to conclude that the teaching of expertise
is impossible or that legal education is in need of a revolution. However, it would appear
that this study, read in light of other studies on the acquisition of expertise, supports at
least one more general recommendation. As noted previously, the recommendation
has been made before but it merits repetition. The studies support a call for legal
educators, and other teachers of expertise, to be more explicit about what we teach, no
matter how that goal is defined. If we teach a type of reasoning, we can identify and
explain it. If we teach a knowledge base, we can acknowledge its importance and
perhaps tailor our teaching methods to make its acquisition most effective and efficient.
If we teach various analysis or learning skills, we can try to help students to see their
progress even as their performance may appear to deteriorate. If acquisition of exper-

tise is a long, hard process, we may have to examine how to instill a love of life-long

learning.

It is likely that legal education, and the teaching of expertise in general, requires
teaching to be explicit in all the ways outlined in the preceding paragraph, plus perhaps
other ways of being explicit that have not yet been discovered. If that is the case, then
we should acknowledge the magnitude of the task and be unwilling to be satisfied with
short-term, partial solutions such as over-reliance upon any one teaching or testing
method. Hopefully this study can help in the search for longer-term solutions both by
documenting part of the legal education experience for the benefit of others who seek

to teach expertise and by perhaps helping to forge links between legal education and

95

research on teaching expertise in general so that legal educators could take better
advantage of what has been learned by others.

Hopefully this study also will help build a greater appreciation for what has already
been accomplished in the teaching of expertise. Although, for example, law school
testing, and state bar examinations, can be easily criticized, and although they could
certainly be better, still this study would support a view that they may be preferable to
many alternatives. If a nonmonotonic process is involved, then heavy emphasis on
testing the acquisition of knowledge may be appropriate to test a student who has only
been in the field for three years or less. It may also, of course, raise questions about
whether testing at a later point might be appropriate, not only to protect the public
served by the expert, but in addition to provide incentive for the expert to continue the

process of gaining expertise.

By raising the possibility of what amounts to the recertification of experts, this set of
conclusions has obviously moved far beyond the subjects and context of this study. It is
an appropriate point to observe that much research remains to be done but that this
study must, indeed, conclude. The torch passed by Bryden (1984) and others has been
carried here but it again must be passed on. Further conclusions will be best made when

more data has been gathered and examined.

LIST OF REFERENCES

Alexander, P. A., White, C. S., Haensly, P. A., & Crimmins-Jeanes. Training in
analogical reasoning. American Educational Research Journal, 1987, 24, 387-404.

American Bar Association. Report and recommendations of the task force on lawyer
competency: The role of the law schools. Chicago: ABA Press, 1979.

American Bar Association. Special Committee for a Study of Le al Education. Law
schools and professional education: Report and recommendations 0 the special commit-
tlege 8r a study of legal education of the American BarAssociation. 'cago: ABA press,

Ames, J. B. Cases on torts (Vol. 1) (2nd. ed). Cambridge, MA: Harvard Law Review
Publishing Association, 1893.

Ames, J. B. A selection of cases on the law of bills and notes (Vol. 1). Cambridge, MA:
Harvard Law Review Publishing Association, 1894.

Ames, J. B. A selection 0 cases in equity jurisprudence (Vol. 2). Cambridge, MA:
Harvard Law Review Pub ' hing Association, 1904.

Armstrong, J. S. Long range forecasting: From crystal ball to computer. New York: John
Wiley & Sons, 1978.

Auerbach, J. S. Unequal Justice. New York: Oxford University Press, 1976.

Bandura, A. Influence of model’s reinforcement contingencies on the ac uisition of
imitative responses. Journal of Personality and Social Psychology, 1965 , I, 5 9-595.

Baron, J. B., & Stemberg, R. J. (Eds). Teaching thinking skills: Theory and practice.
New York: W. H. Freeman, 1987.

Berlin, I. The concept of scientific history. History and Theory, 1960, 1, 1-31.
Blair, J. A., & Johnson, R. H. (Eds). Informal logic. Pt. Reyes, CA: Edgepress, 1980.

Bloom, G. S., & Broder, L. J. Problem-solving processes of college students. Chicago:
University of Chicago Press, 1950.

96

97

B0 1e, &ghe anatomy of a torts class. TheAmerican University Law Review, 1985, 34,
-1 .

Boyum, K. O., & Mather, L. (Eds.). Empirical theories about courts. New York:
Longman, 1983.

Brkic, J. Legal reasoning. New York: Peter Lang, 1985.

Brown, J. J. Honing the legal mind: The classroom experience. Stetson Law Review,
1983,12, 653-669.

Bden, D. P. What do law students learn? A pilot study. Journal of Legal Education,
19,34, 479-506.

B97§erL urges curb on trial lawyers not fully trained. New York Times, 27 November
1

Burton, S. J. An introduction to law and legal reasoning. Boston: little, Brown, 1985.

Campbell, D. T., & Stanleyzllc Experimental and quasi-exp experimental designs for
research Chicago. Rand Mc ally College Publishing, 1966.

Carrington, P. D. Of law and the river. Journal of Legal Education,D 1984, 34, 222-228.
Chase, W. G., & Simon, H. A. Perception in chess. Cognitive Psychology, 1973, 4, 55-81.

Chi,M. T. H., Glaser, R., &Farr, M. J. (Eds). Thenatureoferpertise. Hillsdale, NJ:
Lawrence Erlbaum Associates, 1988.

Clancey, W. J. & Shortliffe, E. H. Readings tn medical artificial intelligence. Reading,
MA: Addison-Wesley, 1984.

Collins, A. Teaching reasonin skills. In Seg al, J. W., Chijiam an, S. F., & Glaser, R.
Thinkingandlearningskills (V0. 2). Hillsdale, gNJ: Lawrence rlbaum Associates, 1985.

Conniff, G. Common ground. New Haven: Yale University Press, 1985.

Consalus, C. E. legal education during the colonial period, 1663- 1776. Journal of Legal
Education, 1978, 29, 295-310.

98

Cort, H. R., & Sammons, J. L. The search for "good lawyering": A conce t and model
of lawyering competencies. Cleveland State Law Review, 1980, 29, 397-4 0.

Cox, M. P. Part-time legal education: The Kelso report and more. Journal of Legal
Education, 1976, 27, 473-508.

gagtloggl. C. The current state of law curriculum. Journal of Legal Education, 1982,

Crombag, H. F. M., de Wijkerslooth, J. L., & van Tu 1 van Serooskerken, E. H. On
solving legal problems. Journal of Legal Education, 19 5, 27, 168-202.

Dawes, R. M. A case study of graduate admissions: Application of three principles of
human decision making. Amencan Psychologist, 1971, 26, 180-188.

de Bono, E. The five-day course in thinking. New York: Basic Books, 1967.

de Bono, E. Teaching thinking. London: Maurice Temple Smith Ltd., 1976.

de Bono, E. de Bono’s thinking course. New York: Facts on File Publications, 1986.
de Groot, A. D. Thought and choice in chess. The Hague: Mouton, 1965.

Delaney, J. M. How to do your best on lawschool exams. New York: John Delaney, 1982.
Dillon, J. T. Problem finding and solving. Journal of Creative Behavior, 1982, 16, 97-111.
Dworkin, R. Taking rights seriously. Cambridge, MA: Harvard University Press, 1978.
Dworkin, R. A matter of principle. Cambridge, MA: Harvard University Press, 1985 .

Elstein, A. S., Shulman, L. S., & S rafka, S. A. Medical problem solving. Cambridge,
MA: Harvard University Press, 19 8.

Ericsson, K A., & Simon, H. A. Protocol analysis: Verbal reports as data. Cambridge,
MA: MIT Press, 1984.

Fejfar, A. J. Insight into lawyering: Bernard Loner an’s critical realism applied to
jurisprudence. Boston College Law Review, 1986, 27, 1-719.

99

Feltovich, P. J., Johnson, P. E., Moller, J. H., & Swanson, D. B. LCS: The role and
development of medical knowledge in diagnostic expertise. In Clancey, W. J ., &
Shortliffe, E. H. (Eds.). Readings in medical artificial intelligence. Reading, MA: Addi-
son-Wesley, 1984.

Feuerstein, R., Hoffman, M. B., Jensen, M. R., & Rand, Y. Instrumental enrichment,

an intervention program for structural cognitive modifiability: Theo and ractice. In

Se al, J. W., Chipman, S. F ., & Glaser, R. (Eds.). Thinking and learnrng sla (Vol. 1).
' ale, NJ: Lawrence Erlbaum Associates, 1985.

 

Frank, J. A plea for lawyer-schools. Yale Law Journal, 1947, 56, 1303-1344.

Frederiksen, N. Implications of cogfitive theo for instruction in problem solving.
Review of Educational Research, 19 , 54, 363-4 7.

Friedland, S. I. The use of appellate case re ort analysis in modern legal education:
How much it too much? Nova Law Journal, 986, 10, 495-516.

Fuller, L. L. The morality of law. New Haven: Yale University Press, 1964.

Gagne, R. M. The conditions of learning (3rd ed.). New York: Holt, Rinehart and
Winston, 1977.

Gardner, A. L., An artificial intelligence approach to legal reasoning. Cambridge, MA:
MIT Press, 1987.

Gee, E. G., & Jackson, D. W. Current studies of legal education: Findings and
recommendations. Journal of Legal Education, 1982, 32, 471-505.

Ghatala, E. S., Levin, J. R., Pressle , M., & Lodico, M. G. Trainin cognitive strateg-
monitoring in children. American Educational Research Journal, 1 85, 22, 199-215.

Goldberg, L. R. Man versus model of man: A rationale, plus some evidence for a
method of improving clinical judgment. Psychological Bulletin, 1970, 73, 422-432.

Golding, M. P. Legal reasoning. New York: Alfred A. Knopf, 1984.

Goldstein, L. (Ed.). Precedent in law. Oxford: Clarendon Press, 1987.

100

Groen, G. J ., & Patel, V. L. The relationship between conNrIprehension and reasoning
in medical expertise. In Chi, M. T. H., Glaser, R., & Farr, . J. The nature of expertise.
Hillsdale, NJ: Lawrence Erlbaum Associates, 1988.

Groner, R., Groner, M., & Bischof, W. F. (Eds.). Methods of heuristics. Hillsdale, NJ:
Lawrence Erlbaum Associates, 1983.

Gross, P. W. Intellect beyond law: The case of legal education. Cleveland State Law
Review, 1984,33, 391-448.

Hart, H. L. A. The concept of law. Oxford: Oxford University Press, 1961.

Hays, W. L. Statistics for the social sciences (2nd ed.). New York: Holt, Rinehart and
Winston, 1973.

Hill, C. C. Problem solving: Learning and teaching. New York: Nichols Publishing
Company, 1979.

Holmes. E. M. Education for com etent lawyering - Case method in a functional
context. Columbia Law Review, 197 , 76, 535-580.

Homstein, A. D. The myth of legal reasoning. Maryland Law Review, 1981, 40, 338-347.

J aff, J. Frame-shifting: An empowerin methodolo for teaching and learning legal
reasoning. Journal of Legal Education, 986, 36, 249- 7.

 

Jensen, H. L., & Howitz, J. S. A theoretical fi'amework for quantifying legal decisions.
Jurimetrics Journal, 1979, 20, 121-139.

Johnson, E. J. rtise and decision under uncertainty: Performance and rocess. In
Chi, M. T. H., G aser, R., & Farr, M. J. (Eds.). The nature of expertise. ' dale, NJ:
Lawrence Erlbaum Associates, 1988.

J ohnson-Laird, P. N., & Wason, P. C. (Eds. . Thinking: Readings in cognitive science.
Cambridge: Cambridge University Press, 19 7.

Johnston, P., & Afflerbach, P. The rocess of constructing main ideas from text.
Cognition and Instruction, 1985, 2, 20 -232.

101

Kahn, E. S. Reading/thinking skills of lawgers, law professors, and law students (Doc-
toral dissertation, Rutgers Universi , 198 ). DissertationsAbstracts International, 1980,
41, 1525-A. (University Microfilms ternational No. 8023599).

Keener, W. A. Methods of legal education. Yale Law Journal, 1947, 56, 143- 149.

Keeton, W. P., Dobbs, D. B., Keeton, R. E., & Owen, D. G. Prosser and Keeton on the
law of torts (5th ed.). St. Paul, MN: West Publishing, 1984.

11(9esl’r7nan, M. A guide to critical legal studies. Cambridge, MA: Harvard University Press,

Kelso, C. D. Teaching teachers: A reminiscence of the 1971 AALS law teachers clinic
and a tribute to Harry W. Jones. Journal of Legal Education, 1972, 24, 606-628.

Kelso, R. R., & Kelso, C. D. Studying law: An introduction. St. Paul, MN: West
Publishing, 1984.

Kennedy, D. Legal education and the reproduction of hierarchy. Journal of Legal
Education, 1982, 32, 591-615.

Kennedy, M. M. Inexact sciences: Professional education and the develo ment of
expertise. In Rothkopf, E. Z. (Ed.). Review of Research in Education (Vol. 1 ). Wash-
ington, DC: American Educatlonal Research Association, 1987.

Kimball, E. L, & Farmer, L. C. Comparative results of teaching evidence three ways.
Journal of Legal Education, 1979, 30, 196-212.

Konop, T. F. The case system - A defense. Notre Dame Lawyer, 1931, 6, 275-283.

Kreiling, K R. Clinical education and lawyer com tency: The process of learnin to
learn from experience through properly structure clinical supervision. Maryland
Review, 1981, 40, 284-337.

Langdell, C. C. A selection of cases on the law of contracts. Boston: little, Brown, and
Company, 1871.

Lesgold, A., Glaser, R., Rubinson, H., Klopfer, D., Feltovich, P., & Wan , Y. Expertise
in a complex skill: Diagnosing x-ray ictures. In Chi, M. T. H., Glaser, ., & F arr, M.
J. The nature of expertise. Hillsdale, J: Lawrence Erlbaum Associates, 1988.

102

Ilngeg, E. H. An introduction to legal reasoning. Chicago: University of Chicago Press,

:LfVin, B. F. Experience, not education. National Law Journal, February 6, 1989, pp. 13;

ligwleléyzzr, K. N. The current crisis in legal education. Journal of Legal Education, 1948,
, 1 - .

Llewellyn, K N. The brarnble bush. Dobbs Ferry, NY: Oceana Publications, 1960.

Lochhead, J. Teaching analytic reasoning skills through pair problem solving. In Segal,
J. W., Chipman, S. F., & Glaser, R. (Eds.). Thinkingand learnmgskills (vol. 1). Hillsdale,
NJ: Lawrence Erlbaum Associates, 1985.

Lonergan, B. J. F. Insight. New York: Philosophical Library, 1957.

Lorensen, W. D. Concentr ting on a single jurisdiction to teach criminal law - an
experiment. Journal of Legal ation, 1968, 20, 361-365.

MacCormick, N. Legal reasoning and legal theory. Oxford: Clarendon Press, 1978.
Mayer, R. E. Thinking, problem solving, cognition. New York: W. H. Freeman, 1983.
Minsky, M. L. The society of mind. New York: Simon and Schuster, 1986.

Morgan, E. M. The case method. Journal of Legal Education, 1952, 4, 379-391.

Mudd, J. 0. Thinking critically about "thinking like a lawyer". Journal of Legal Educa-
tion, 1983, 33, 704-711.

Nagel, S. S., & Neef, M. G. Decision theory and the legal process. Lexington, MA: D. C.
Heath, 1979.

N ewell, A. The heuristic of George Polya and its relation to artificial intelligence. In
Groner, R., Groner, M., & Bischof, W. . (Eds.). Methods of heunstrcs Hillsdale, NJ:
Lawrence Erlbaum Associates, 1983.

Hillelg 713., & Simon, H. A. Human problem solving. Englewood Cliffs, NJ: Prentice-

103

Nickerson, R. S. On im roving thinking through instruction. In Rothkopf, E. Z (Bd.).
Review a Research in atton (Vol. 15). Washington, DC: American Educational
Researc Association, 1988.

Nickerson, R. S., Perkins, D. N., & Smith, E. E. The teaching of thinking. Hillsdale, NJ:
Lawrence Erlbaum Associates, 1985.

213333, S. H. Examining and grading in law schools. Arkansas Law Review, 1977, 30,

Nisbett, R. B., Fong, G. T., Lehman, D. R., & Cheng, P. W. Teaching reasoning. Science,
1987, 238, 625-63 1.

Nisbett, R. E., & Wilson, T. Tellin more than we can know: Verbal reports on mental
processes. Psychology Review, 197 , 84, 231-259.

1:39;“, H. L, & Ehrlich, T. New directions in legal education. New York: Mcgraw-Hill,

Paul, J. A bedtime story. Virginia Law Review, 1988, 74, 915-934.

Phili S. U. The language socialization of lawyers: Acquiring the "cant". In S indler,
$19.88 .). Doing the ct nography of schooling. New York: Holt, Rinehart and inston,

Plato. Great dialogues of Plato. Warmington, E. H., & Rouse, P. G. (Eds), Rouse, W.
H. D. (trans). New York: New American library, 1956.

gplya, g 51l41athematics and plausible reasoning (2 vols.). Princeton: Princeton University
ess, .

Polya, G. How to solve it (2nd ed.). New York: Doubleday, 1957.

Prosser, W. L., Wade, J. W., & Schwartz, V. E. Cases and Materials on Torts(7th ed.).
Mineola, NY: Foundation Press, 1982.

Redmount, R. S. Law learning, Teacher-student relations, and the legal profession.
Washington University Law Quarterly, 1981, 59, 853-886.

Rowe, H. A. H. Problem solving and intelligence. Hillsdale, NJ: Lawrence Erlbaum
Associates, 1985.

104

Sarat, A., & Felstiner, W. L Law and strategy in the divorce lawyer’s office. Law &
SocietyReview, 1986, 20, 93-134.

Savell, J. M., Twowig, P. T., & Rachford, D. L. Empirical status of Feuerstein’s
"instrumental enrichment” (FIE) technique as a method of teaching thinking skills.
Review of Educational Research, 1986, 56, 381-409.

Schon, D. A. The reflective practitioner. New York: Basic Books, 1983.
Schon, D. A. Educating the reflective practitioner. San Francisco: J ossey-Bass, 1987.

Schwartz, A. J. Law, lawyers, and law school: Perspectives from the first-year class.
Journal of Legal Education, 1980, 30, 437-469.

Se al, J. W., Chipman, S. F., & Glaser, R. (Eds.). Thinking and learning skills (2 vols.).
Hri' lsdale, NJ: Lawrence Erlbaum Associates, 1985.

Shuell, T. J. Cognitive conceptions of learning. Review of Educational Research, 1986,
56, 411-436.

Shulman, L. S., & Elstein, A. S. Studies of problem solving, judgment, and decision
making: Im lications for educational research. In Kerlinger, F. N. (Ed.). Review of
Research in %ducation (Vol. 3). Itasca, IL: F. E. Peacock Publishers, 1975.

Slgagge, MZ. Law degree wanes as passport to business job. New York Times, 27 January
1 , p. 1.

Spaeth, H. J. Supreme court policy making. San Francisco: W. H. Freeman, 1979.

State Bar of California, the Committee of Bar Examiners. Information regarding
performance tests. San Francisco: State Bar of California, 1988.

Stemberg, R. J. Intelligence, information processing, and analo ' al reasoning: The
componential analysis of human abilities. New York: John Wiley Sons, 1977.

Stemberg, R. J. (Ed.). Handbook of human intelligence. Cambridge: Cambridge Uni-
versity Press, 1982.

Stevens, R. Law school. Chapel Hill: The University of North Carolina Press, 1983.

105

Strauss, S., & Stavey, R. U-sha d behavioral growth: Imglications for theories of
gzgegggment. In Hartup, W. W. Ed.). Review of Child Deve pment Research, 1982, 6,

Stro F. R. A new curriculum for the College of Law of the Ohio State University.
Ohio tate Lanournal, 1950, 11, 44-56.

Suber, P. Analogy exercises for teaching legal reasoning. Journal of Law & Education,
1988, 17, 91-98.

Sggflrst, M. 1. Putting law school grades in perspective. Stetson Law Review, 1983, 12,

Teich, P. F. Research on American law teaching: Is there a case against the case system?
Journal of Legal Education, 1986, 36, 167-188.

Thielens, W. Jr. The socialization of law students. New York: Arno Press, 1980.

Tito, C. Artificial intelli ence: Can computers understand why two legal cases are
similar? Computer/Law oumal, 1987, 7, 09-437.

Tuma, D. T., & Reif, F. (Eds.). Problem solving and education: Issues in teaching and
research. Hillsdale, NJ: Lawrence Erlbaum Associates, 1980.

Turow, S. One L New York: G. P. Putnam’s Sons, 1977.

Voss, J. F., Blais, J ., Means, M. L., Greene, T. R., & Ahwesh, E. Informal reasoning
and subject matter knowlque in the solving of economics problems by naive and novice
individuals. Cognition and nstruction, 1986, 3, 269-302.

Wade, J. W. Some observations on the present state of law teaching and the student
response. Journal of Legal Education, 1984, 35, 753-772.

Wampler, B. B., & Williams, M. P. Grammatik III (computer software). San Francisco:
Reference Software, 1988.

Wangerin, P. T. Skills training in "legal analysis”: A systematic approach. University of
Miami Law Review, 1986, 40, 409-48 .

Watson, A. S. The quest for professional competence: Psychological aspects of legal
education. University of Cincinnati Law Review, 1968, 37, 93-166.

106

Whimbey, A., & Lochhead, J. Problem solving and comprehension (2nd ed.). Philadel-
phia: Franklin Institute Press, 1981.

White, D. R. The official lawyer’s handbook. New York: Simon & Schuster, 1983.

White, J. B. Doctrine in a vacuum: Reflections on what a law school ought (and ought
not) to be. Journal of Legal Education, 1986, 36, 155-166.

Wilkinson, L. SYSTA T(computer software). Evanston, IL: SYSTAT, Inc., 1988.

Willett, J. B. Questions and answers in the measurement of change. In Rothko f, E. Z
(Ed.). Review 0 Research in Education (Vol. 15). Washington, DC: American uca-
tional Researc Association, 1988.

ggozdésB. D. The measurement of law school work. Columbia Law Review, 1924, 24,

$2.05? B. D. The measurement of law school work II. Columbia Law Review, 1925, 25,
1 1.

$052? D. The measurement of law school work III. Columbia Law Review, 1927, 27,

Woodward, J ., Carnine, D., & Gersten, R. Teaching problem solvin through computer
simulations. American Educational Research Journal, 1988, 25 , 72- 6.

ZyIndex (computer software). Arlington Heights, IL: ZyLab, 1987.

APPENDICES

Appendlx A

Informed Consent Form

Prior to signing this form, please read all of the following paragraphs. The paragraphs

are numbered to make it easier to ask questions about specific points.

1. The research project in which you would participate is being conducted, in part, to
fulfill one of the requirements of Professor Senger’s doctoral program at Michigan
State University. The purpose of the research project is to study the type of learning
which goes on in law school. Learning will be measured by essay questions much like
those typically used on law school examinations. The one difference will be that, in
addition to the usual written answers, participants will be asked to say aloud what they
are thinking while they read the essay questions and prepare their written answers.
There will be four essay questions which will be answered in a three hour time period.
Each participant will write the questions ‘while alone in a room with the principal
reseacher (Professor Senger); this will allow the process to be videotaped while
preserving anonymity for the answers. The essay question writing sessions will be
scheduled at your convenience over the next few weeks. Analysis of the data and writing

a dissertion reporting the results will be completed within one year.

2. Signing this form expresses your agreement both that the research has been explained
to you and that you understand the explanation. Please note that this research involves
all the risks usually associated with taking an essay examination including both the risk
of stress while answering the questions and the risk that one’s performance may not

meet one’s expectations.

107

108

3. Professor Senger agrees not join in law school decision making concerning individual
participants unless a participant makes a written request asking Professor Senger to
join in a particular decision. Professor Senger will retain the right to grade participants
in future law school courses but only if those courses use the school’s anonymous
grading procedure. If any part of this paragraph is not understood, please do not sign
this form.

4. Please note that, even if you agree to participate, you may end your participation at

any time without fear of reprisal.

5. All results of this study will be treated with strict confidence regarding the identity
of any participant. Individual participants will be able to obtain their own results if they
wish. The videotapes will be viewed only by Professor Senger. The videotapes will be

erased after this research project is completed.

6. There are no guaranteed benefits to participants other than the agreed upon fee for
their participation.

7. Signing this form will indicate that you have freely consented to participate in the
research referred to in the preceding paragraphs. Ifyou so agree, please sign and date

this form in the space provided below.

Participant’s signature Date

If you decide not to participate, could you briefly indicate why? There is no obligation

to say anything but the information would greatly assist analysis of the data that is
collected.

Appendix B

Invitation to volunteer to participate

June 4, 1987

HELLO:

Want a chance to earn $25, help with some research on legal education, and learn more

about law school skills? How? Well, more details will be provided if you are selected,

mtg): basis igea involves taking something like a law school exam. Three hours or less
nee e .

To indicate your interest in participating, please print our name and telephone number
on this sheet and return it to the secretaries on the fth floor. Please act before June
12th. Prospective participants will be randomly selected from those who volunteer and
will be notified the following week. Even if you volunteer and are selected, you will be
free to decline to participate if you happen to change your mind.

Thank you for your consideration,

Charles J. Senger

Professor of Law-

 

Yes, I would like to be considered:

Name

 

Phone Number

 

109

Appendix c

Response letter to VOIUI‘I‘IBOI‘S

June 20, 1987

Dear :

Thank you for your expression of interest in participating in my study of legal education.
As the next step, I would like you to call my secretary to schedule a three hour block.
The time can be at your convenience - any time, any day that is open. What we can then
do is to discuss the project in more detail and, if you wish to go ahead, we can complete

the process in that time block.
If you have difficulty making an appointment, or if you have further questions, please
do not hesitate to call me. If I am not at the office, please call me at home (517)

349-9105.

Thank you again for your offer to help. I look forward to working with you and together

maybe we can make a contribution to legal education.

Sincerely yours,

Professor Senger

110

Appendix D

Protocol questions

QUESTION 1.

(Please remember to say out loud what you are thinking while you read and prepare
your answers.)

Free food! It was too good to be true. But it was true, as Sally’s teenage sister explained.
All one had to do was to show up at 6:00 P.M. at what used to be the neighborhood
school gym. A free hot supper was provided for all teenagers who came, Monday
through Saturday.

Sally was only nine, and her ounger sister was two and a half, but the hot lunches

sounded great to someone w 0 did not eat re arly. The next day, Sall decided to

Lake her younger sister on a two block walk to e gym to try to get both 0 them a free
ot supper.

Unfortunately, the charitable organization running the supper program enforced the
rules limiting the sup rs to teenagers. As the lady in the gym explained it, the city of
Detroit had started e program to help keep teenagers off the street. Although all
money for the program came from private donations, the city rules on eligibility still
had to be followed.

As Sally argued with the lady, she did not notice that her younger sister had slipped
away. By the time that Sally checked for her sister, she was too late. The two an alf
year 01 had gone out the front door of the gym and, in attempting to run across the
street, had been hit and killed by a car.

The estate of the child sued the driver of the car, the city of Detroit, the charitable
organization which ran the suppers, and the present owner of the gym, a local real estate
company. In the trial court, all defendants were found liable. e defendants now

appeal.

111

 

112

Using the bluebook that is supplied, please write an answer to the following two
questions.

A. Assume that you are an attorney who represents the estate of the deceased child.
Please explain how you would argue to an appellate court in support of the trial court’s
ruling that the estate has a cause of action against the local real estate company which
currently owns the gym.

B. Assume that you are an attorney who represents the local real estate company. Please
explain how you would argue to an appellate court in their defense against the trial
court’s ruling that the estate has a cause of action against the real estate company.

 

113

QUESTION 2.

(Please remember to say out loud what you are thinking while you read and prepare
your answer.) '

Danny Dealer didn’t know what to do. His drug dealing profits were high but both the
cops and some rival gangs were hot on his heels. If he got caught by the gangs, it would
all be over. If he got caught by the cops, at least he wanted to save the money for his
"retirement”.

Given that banks had to report large cash transactions, Dann knew that going near a
bank would on] bring the cops closer. In desperation, Danny rove out into the country
to bury a box fu l of money in some woods. The ground was harder than c ected, Danny
was out of shape, and time was short, but at least he got the box covere The next day
Danny was arrested and put in jail.

Some time later, Harry Hunter passed through the woods where the box had been
buried. As Harry walked, he noticed the corner of a box sticking out of the ground,
apparently uncovered by recent flooding. Harry was unable to di up the box without
a s ovel but he broke open the expose corner. The sight of all e money in the box
almost caused him to famt.

Harry took the money from the box and filled both his back pack and game bag. He
then took the money to the local sheriff’s department and explained where the box was
located. Harry could not believe what happened in the next few days. All his efforts
resulted in an unending headache as seermngly eve one around went after the money.

Landowner, who owned the woods, said at the money should go to the
landowner. Dann Dealer read about the find in the newspaper and Danny’s lawyer
sags that Danny s ould get it. The Internal Revenue Service, the State Treasury, and
o ers, all join the fight. After a short trial, the trial court ruled that, as a matter of law,
Danny Dealer was entitled to the money. All the other claimants now appeal.

Using the bluebook that is supplied, please write an answer to the following two
questions.

A. Assume that you are an attorne who represents Larry Landowner. Please explain
how you would ar e to an apple ate court against the trial court’s ruling that only
Danny Dealer, an not Larry downer, is entitled to the money.

B. Assume that you are an attorney who represents Harry Hunter. Please explain how
gm would argue to an a pellate court a must the trial court’s ruling that only Danny
ealer, and not Harry unter, is entitle to the money.

114

QUESTION 3.

(Please remember to say out loud what you are thinking while you read and prepare
your answer.)

It was almost closing time when the robber burst into George Grocer’s store. George
knew what was happenin , grabbed the shotgun which he kept under the counter, and
attempted to shoot. U ortunately the shotgun (jammed and the robber’s handgun
didn’t. George fell to the floor with a chest woun .

After grabb' the money in the cash drawer, the robber ran to leave on] to see two
police cars pu ling into the store’s parking lot. A gun battle broke out an the robber

eld off the police for about an hour until he was overpowered b a police officer who
had slipped in through a back delivery door. The olice co seated the robber’s
handgun, a "Saturday Night Special Model" made by C eap Arms Inc. Inspection of the
weapon showed that "Saturday Night Special Model" was a good name for the hand
since it was poorly manufactured from cheap quality materials. Looking at it ma e it
harl'id to believe that the robber had fired it almost sixty times in the battle with the
po ce.

George Grocer sued Chea Arms Inc. George claiming that Cheap Arms Inc. was liable
under a theory of products 'ability. The trial court ruled, as a matter of law, that George
Grocer had no products liability cause of action against Cheap Arms Inc.

Using the bluebook that is supplied, please write an answer to the following two
questions.

A. Assume that you are an attorney who represents George Grocer. Please explain how
you would argue to an appellate court against the trial court’s ruling that George Grocer
as no products liability cause of action against Cheap Arms Inc.

B. Assume that you are an attorney who represents Cheap Arms Inc. Please explain
how you would argue to an appellate court in support of the trial court’s ruling that
George Grocer has no products liability cause of action against Cheap Arms Inc.

 

115

QUESTION 4.

(Please remember to say out loud what you are thinking while you read and prepare
your answers.)

Co ral Paul Tanker joined the Arm in 1985 and, after basic training, was assigned
to ort Huachuca, Arizona, the wor dwide Army communications center. Shortly
thereafter he met Ser eant Patricia Smith. Over a period of time, the two became
involved in a sexual re ationship.

In accordance with standard operating procedure, Corporal Tanker had been tested
for AIDS when he had arrived at the Army base. Due to an administrative error, it was
not until four months later that he was informed that he had tested positive for the
AIDS virus. He told Ser eant Smith the bad news and the two of them broke ofi their
relationship out of fear t at Sergeant Smith might catch the disease.

Three months later Sergeant Smith visited Corporal Tanker and they again had sex
which resulted in Sergeant Smith’s pregnancy. Although Corporal Tanker’s company
commander had ori nally been understanding, this incident led to the commander
decidng to bring criminal charges against Co oral Tanker. Co oral Tanker was
char e with aggravated assault which is define as assault "with a angerous weapon
or 0 er means or force likely to produce death or grievous bodily harm."

Testimony at trial revealed that some medical experts estimate the chance of contract-
ing AIDS to be as high as 50 rcent for each sex act. Sergeant Smith’s monthly tests
for AIDS were revealed to all ave been negative but doctors also testified that patients
sometime do not test positive until four or five years after exposure.

The Erisal court convicted Corporal Tanker of aggravated assault. Corporal Tanker now
appe .

Using the bluebook that is supplied, please write an answer to the following two
questions.

A. Assume that you are an attorney who represents Cogoral Tanker. Please explain
32w you wguld argue to an appellate court that Corpor Tanker’s conviction should
reverse .

B. Assume that you are an attorne who represents the Arm rosecutor. Please explain
ll;(citizzrgitiinw‘clmld argue to an appe ate court that Corporal anker’s conviction should
e .

 

Appendix E

Warm up instructions

In this experiment we are interested in what you think about when you find answers to
some questions that I am oing to ask on to answer. In order to do this I am going to
ask you to THINK ALOU as you wor on the roblem given. What I mean by thinking
aloud is that I want you to tell me EVER Y'IHIN G you are thinking from the time you
first see the question until you have completed your answer. I would like you to talk
aloud constantly from the time I present each problem until your answer is completed.
I don’t want you to try to plan out what you say or try to explain to me what you are
saying. Just act as if you are alone in the room speaking to yourself. It is most important
that you keep talking. If you are silent for any long period of time, Iwill ask you to please
keep talking. Do you understand what I want you to do?

116

Appendix F

Participant’s legal experience

Please answer the following questions which will assist with data interpretation:

1. Have you ever been employed in a legally related job? If so, please explain briefly.

2. Please circle any of the following course names if you have taken that course:

Trial Workshop; Sixty-Plus Law Clinic; Client Counseling;

National Moot Court; Law Review.

3. Are either of your parents an attorney? If so, please explain briefly.

4. Prior to going to law school, did you have close friend who was an attorney? If so,
please explain briefly.

5. Prior to today’s experience, have you discussed either the facts or issues of these
gueséions, or the thinldng aloud procedure, with anyone else? If so, please explain
me y.

117

APPENDIX G

CLASSIFICATION GUIDE

1. INTRODUCTION

The essay question answers you read will fall within one of several classifications. The
classifications used here have been chosen to reflect both the type of authority being
used and whether facts are used with that authority. The task is to classify each answer
according to its type.

For purposes of this classification process, there is no implication that one type is
necessarily better or worse than another. likewise, the intent here is not to grade the
answers as would be done in a typical essay examination. A number of other factors
would have to be considered if grading were the goal. Once an answer is classified, there
is no need to further evaluate it.

The following paragraphs will describe the classifications.

2. CLASSIFICATIONS

WWW

NLAA - Non Legal Authority Alone - will be the classification if the student’s
written answer addresses the question in terms of common sense or authority other
than legal authority. As will become clearer in the descriptions of the other answer
types, legal authority would be a legal rule or court decision.

As an example of this type, a student might say that one party should win because it was
”fair" or "just" to reach that result. Likewrse a student mi t say that a given result should
be reached because a Harris poll showed that the majority of the people in America
agreed with that result.

118

119
I NIEE-Il I 151° 'IE

NLAF - Non Legal Authority with Facts - will be the classification if the student’s
wntten answer addresses the question in terms of eneral common sense or authority
other than 1e a1 authority, as in type NIAA, but interweaves some of the facts of
the problem into that common sense or non legal authority.

As an example of this type, a student might say that one party should win because it was
"fair". However, the student would also show what facts of the problem made it more
fair. Perhaps one party had been more responsible for the nustake that caused the
parties a problem. Maybe one par had been in a more powerful bargaining position
and had taken unfair advanta e of t at position. Anyway, the main characteristic which
dis ' ' hes this type from c previous one is that the facts of the problem are used
with e non legal authority to support the conclusion.

MEL-Bum:

Type RA - Rule Alone - will be the classification if the student’s written answer

addresses the uestion in terms of an announced 1e al rule. For example, a student

might say that t e rule of law in this state was "ABC " and therefore the result in the
resent case must be "XYZ". The clear use of a legal rule distinguishes this answer type
om the preceding types.

W

Type RA - Rule with Facts - will be chosen if the student’s written answer addresses
the question in terms of an announced legal rule which is interwoven with the facts of
the roblem. For example, a student might say that the general rule was "ABCD". The
stu cut would then tie the facts of the problem to the elements of that rule. If
application of the rule required that the injured party had lost more than $100, the
student would show how the problem involved a loss of at least that magnitude.

Interweaving the facts with the rule distinguishes this type of answer from type RA
above. In type RA, a conclusion might be reached through the use of the legal rule but
the facts were not explicitly treate in light of the requirements of the rule.

W

Type CA - Case Alone - will be chosen if the student’s written answer addresses the
question in terms of a prior court decision. For example, the student might say that the

120

"Doc” case held that the rule was ”ABCD" and therefore the result in the present case
would be "XYZ".

The explicit reference to a prior court decision in what distinguishes this type from types
RA and RF above. In 5 RA and RF, the rule is taken from a seeming y given body
of law. The RA or rule is divorced from the prior facts of the case in which it
originally arose. In type CA, at least the prior case is explicitly acknowledged as the
source of the rule

Wm

Type CF -C or P - Case with Facts of Case or this Problem - will be chosen if the student’s
answer addresses the question in terms of a rule of a prior court decision which is
interwoven with the facts of either the prior case or of the present problem. For
example, the student might say that the "Doc" case held that the rule was ABCD". The
student would then tie the elements of that rule to the facts of the lErgesent problem.
Alternatively, the student might explore the facts of the "Doc” case fore reaching a
result for the present problem.

Type CF -C or P is distinlgushed from CA much like type RF is distinguished from

RA. Just as type interweaves acts of the problem with the announced rule
w ereas type RA just states the rule and erhaps a conclusion, so type CF-C or P
interweaves facts of the problem or case witE the rule from the prior case whereas type
CA just states the case rule and perhaps a conclusion.

Type CF-C&P - Case with Facts of Case and this Problem - will be chosen if the
student’s answer addresses the question in terms of a rule of a prior court decision.
However, unlike type CF-P, the facts of the prior case are explicitl compared or
contrasted with the facts of the resent problem before the rule of t e prior case is
ap lied to the resent problem. or example, the student might say that the "Doc" case

e was "ABC ". The student would then note that "Doe" involved facts "EFGH" and
the present problem involved facts "EFGK". Since the facts were not identical but still
quite similar, the student would then conclude that the "Doc" case rule could be used
by analogy in the present problem. In using the rule, the student ma modify its
requirements to accommodate the ways in “[11101] the present problem di ers from the
prior case.

Although this type answer shares much with the precedin types, the key distinction is
that the student recognizes that the prior rule is being use by analo . In type RA, RF,
CA, and CF-P, the prior rule is taken as an absolute which is app ed to the present
problem and, at most, facts from only the problem or the case were interwoven. In type

 

121

CF-C&P, the student is giving evidence of the implications of using the prior case rule
as an analogy. Those implications are twofold. First, that the prior case rule should only
be used based upon a similarity between its facts and the present facts. Second, that the
prior case rule may itself be modified in order to fit the special requirements of the
present problem.

The CF -C&P answer thus shows recognizes that the rule of the prior case cannot be
selected for application until consideration has been given to how the facts of the prior
case compare and contrast with the facts of the present problem. The rule of the prior
case also is not taken to be absolute but rather is su ject to broader or narrower
definition in light of the facts of the present problem.

3. CONCLUSION

The essay questions answered by the students have two parts. In addition, for a
particular uestion, a student sometimes will break the legal problem into several
smaller pro lems or issues with each smaller problem then receiving separate treat-
ment. It may be that the treatment of one part of the problem seems to fit into one
answer ‘tvylple and another part of the answer fits a different type. In such situations, the
answer receive multiple classifications. Thus one answer might be classified as types
RA, RF, and CA because its different parts fit those classification types.

Finally, a student ma use multi 1e common sense rinciples, rules, or cases in treating
a given issue. If that appens, p ease write "multip e" along with the classification that
is otherwise appropriate.

THANK YOU!

Appendix H

Practice question and answers

[Practice question]

Plaintiff was injured while riding a roller coaster when a branch from an overhanging
tree struck her in the eye, shattering her eye lasses. The plaintiff brou t suit against
the owner of the roller coaster, alleging that t e owner negliglently faile to ensure that
the route travelled by patrons of the roller coaster was cc of obstructions. The
defendant denied that the amusement device was negligently maintained and that the
plaintiff was injured while riding it. As part of his case-in-defense, the defendant sought
to have the roller coaster manager testify that thousands of people had ridden the
amusement device in the three months prior to the plaintiff’s accrdent, and no rider
had complained of having been struck by a branch or limb of a tree during a ride.

A. Assume that you are an attorney representing plaintiff. Please explain how you would
33%: to an appellate court that the roller coaster manager’s testimony should not be
a 'tted into evidence?

B. Assume that you are an attorney representing defendant. Please explain how you
would argue to an appellate court that the roller coaster manager’s testimony should
be admitted into ev1dence?

[As noted in Chapter III, this question is taken from a classroom exercise in a class that

was using as a text Paul R. Rice’s Evidence: Common law and federal rules of evidence

(New York: Matthew Bender, 1987). This question is taken from problem 2 on page
76 of that text.]

122

123

NLAA

The roller coaster manggfir should be allowed to testify about the prior incidents. If he
can’t testify, who can? 0 is in a better position to know? What s the harm in letting
him tell his story?

This witness has taken a day off from work to try to contribute what he can. He has
waited all day. Now we are going to refuse to let him testify? Why?

prlaintiff is free to testify about his ride, and no one can sa anything about other rides,
how is the jury going to get the big picture? Doesn’t simp e fairness require that both
sides get their say?

If there are some objections that plaintiff wants to make about the testimony, let’s
e lain those objections to the jury and let them weigh the evidence in that light - but
at east let them hear it.

124

NLAA

If this roller coaster owner has to pit?! for this accident, there soon won’t be an
amusement rides around. The story is ways the same; the deep pocket or person wi
money should pay. But is that fair. y should the ride owner bear the loss?

It used to be in this country that everyone was proud to take care of themselves. It was
the pioneer spirit. If a wagon wheel broke, they fixed it and, if help was needed, the
community pgched in. If a barn had to be rebuilt after a lightening storm, then the
farmer and ' friends did it. It wasn’t always easy but self reliance instilled a special
kind of pride.

Now everyone who has a problem immediately wants someone else to pagfor it. Maybe
the government. Maybe a corporation. Maybe a home town business. ut the key is
that it’s always someone else.

If this roller coaster manager can’t show the tfilmy what’s going on, then we invite the

jury to make the wrong person pay. Of course '5 evidence should be allowed in. Justice
demands it.

 

125

NLAF

The large number of prior safe rides should hel defendant. If the number had been
low, then the lack of complaints could have been or other reasons. Maybe others didn’t
report because their injuries were minor. Maybe they didn’t know who should get the
report. Maybe they didn’t know that they could sue the owner. Maybe they had just
been almost hit and had ducked out of the way.

Although there are other explanations, the thousands of persons who rode safely seems
to more than cover the possibilities. If the wind was a factor, it must have blown hard
on one of the revious rides. The same for laintiff’s height, etc. If riders were etting
hit, it’s unlike y that plaintiff would be the rst to complain out of all those peop e. The
greater the number of nonoccurrences, the more likely that a true reflection of the
actual condition of the ride will be revealed.

Another way to explain it is to refer to the saying: "History repeats itself.” Thus it is
worthwhile to examine the history of passenger experience on e roller coaster. If no
one else was ever hurt, then it would tend to suggest this particular occurrence was a
fluke or the fault of the rider.

Prior history especially helps since it was supposedly a tree branch which hit plaintiff.
A tree branch grows very slowly. To say that it was out of the way for 3 months and then
suddenly grew across the path is hard to believe. If the branch was a hazard, it would
have been reported reviously. Since no such history of reports exists, it wasn’t a hazard.
It wouldn’t be just ' defendant can’t show that history.

 

126

Defendant argues that the rides of thousands of ople over the three prior months are
substantially similar to the ride taken plaintiff and there were no com laints. The
manager can lay a foundation 1b}; testifying to the customs and proce ures in the
operation of the roller coaster. at standard would set both times of operation and
procedures of maintenance and checkin . The maintenance inspection, for example,
might be staff walking or riding the "rails to check for possible problems like the tree
branch alleged to have caused this accident. Defendant can testify the time eriod he
chose is all within the same amusement season. Defendant can argue that ere have
been no changes in the route or trackage. In sum, the situation has remained substan-
tially the same. Defendant would ask that the trial court admit the evidence based Epon
the similarities and the trial court’s denial, if one, would be an abuse of discretion. ven
if the prior safe rides are not perfectly similar to the ride in question, that should only
go to the weight to be given t e evidence, not to its admissibility.

The defendant’s basic argument is that he had no notice of a problem. The testimony
that thousands of people had ridden the device would be relevant to show this lack of
notice. This testimony might be relevant to show that the branch had shifted into a
dangerous position just before the plaintiff was injured and therefore show that the
defendant may not have been able to foresee such an occurrence.

127

Evidentiaryfiiuestions like this should not be made to seem more difficult than they
really are. e roller coaster owner wishes to show that his ride was safe. He wishes to
show safety by evidence of prior nonoccurrences.

According to the law of this state, evidence of prior nonoccurrences is admissible as
long as substantial similarity is shown. In this case, substantial similarity of circum-
stances is obvious. Therefore, the evidence should be admissible. The law of our state
permits and, as a matter of fact, demands no less.

Plaintiff has, of course, objected to the introduction of this evidence. The motivation
for that objection is not hard to see. This evidence is very relevant for defendant’s case
and naturally plaintiff would like to keep it away from the jury. However, the rule is
that it’s admissible and nothing that plaintiff says changes that.

128

The manager’s testimony is probative as it serves to discredit the plaintiff’s contention
of negligence. It serves to clarify the conditions under which the alleged injury occurred.
It also serves to verify the organization’s routine practice. Exclusion would, in fact,
prejudice the defendant as it would be logical to assume that if the branch injured one
person, it would injure another.

The rules of evidence allow relevant evidence to be admitted. This evidence should
therefore be admitted. To deny the jury here an opportunity to weigh the facts in light
of the manager’s testimony would deny them an opportunity to consrder circumstances
which may ave probative value in determination of whether the incident, in fact,
occurred - something which must be determined prior to determination of whether the
park was negligent.

Anyway, it is sufficiently similar that it should be admitted. This would be circumstantial
evidence but it is relevant to the defense of my client and its probative force is greater
than its possible prejudice to the plaintiff.

129

RF

Plaintiff here alleges that he was struck by a branch as he rode in a roller coaster.
Defendant wishes to show that such an occurrence was unlikely, if not impossible.
Defendant’s means of proving that it was unlikel is to show that thousands of other
riders completed the ride with no one saying an ing about any branches in the way.

The general rule is that rior nonoccurrences can be shown if the circumstances of those
nonoccurrences are ciently similar to make them relevant. That similari is easily
shown here. Defendant’s roller coaster is the same one that has carried all t e riders.
The location of the ride hasn’t changed in at least three months. There is no showing
that the cars on the ride have been changed in any wa - no Windshields added or
removed, etc. There is no showing of changes in the hei t of persons allowed to ride.
For example, it would have been important if all the nor nonoccurrences happened
when the only permitted riders had been children un er 48 inches in height.

Since the prior nonoccurrences happened under sufficiently similar circumstances to
assure relevancy, the eneral rule will allow the defendant to prove those nonoccur-
rences. To deprive de endant of that opportunity would be un'ust and contrary to the
Almericiciln system of justice. If defendant is to have a fair trial: this evidence must be
a mitte . -

130

The modern principles of evidence would allow the testimony by the manager that no
one had been hit before on the ride. If there was a chance that the jury might emphasize
it too much, the adverse arty can use their persuasive skills to convince the jury that
it didn’t really carry muc weight.

Evidence about nonoccurrences is allowed if the events are close in time, the circum-
stances are static, and there is enough frequency to insure reliability.

Here the roller coaster manager’s testimony is about nonoccurrences. The rule would
support the admission of the evidence as long as the circumstances and use of the roller
coaster had not chan ed much. Since at least some of the prior riders had been on the
ride the same day as p aintiff, it seems like] that the nonoccurrences were under similar
circumstances and should be admitted. ousands of people riding the same roller
coaster with no one getting hit by a tree branch is pretty persuasive unless the plaintiff
is 8 feet tall or maybe was some ow standing up in the seat!

131

CA

Defendant’s attomeé’would offer the manager’s testimony to prove the alleged defect
did not cause plainti s injuries. Show as support no other complaints have been made.
Try to get people from the community to testify re: their rsonal knowledge from use
at the park, etc. Hard to do because someone coul have been injured yet not
complained.

The Jeep rollover cases said that prior occurrence evidence is admissible if those
occurrences were sufficiently similar to the one sought to be proven.

Here defendant wants to show lack of accidents rather than accidents. Still, the similar
situation rule should apply. The prior roller coaster rides were similar. According to
the Jeep rollover cases, the court should let defendant show them.

 

132

CA

The main issue is negligence. The defendant should have the right to establish that
nobody else riding the coaster has ever been injured in this manner. Without this
testimony, it is almost impossible to defend against this charge.

Under the Stanley case, the prior lack of accidents when all the persons rode the coaster
should be allowed in. Stanley held that the absence of prior accidents was admissible
evidence to prove lack of notice of a problem.

Of course it would help if it were corporate ractice to document all problems with the
gpéster. That type of monitoring would be p show what conditions were the same or
erent.

In summary, it will be best under Stanley if plaintiff can tie in facts about: closeness in
time; static conditions; and frequency 0 use. Here, plaintiff should be in good shape.

133

CF-C or P

Plaintiff was hit by a tree branch while riding a roller coaster. In order to hold the owner

of the ride responsible, plaintiff has to show that the owner was negligent - that the

phwnelr lens; or should have known about the unsafe branch and yet did not act to protect
e p am .

The owner now seeks to rove that he had no inkling that the branch could be a
problem. To prove that, e wishes to present the testimony of the roller coaster
manager. The manager will testify that thousands of persons had ridden the coaster in
the last three months and no one had complained about the tree branch.

Prior occurrences or nonoccurrences are relevant to a question of notice of a defective
or dangerous condition. In Marois v Paper Converting Machine Company held that
evidence of prior occurrences were admissible if the prior occurrences happened in
substantially similar circumstances.

In the present problem, substantial similaritLpf circumstances could be shown by
reference to a number of factors. One is that t ' is not a carnival ride which is moved
from lace to place - it’s been in the same spot for at least three months. Another is
that e tracks keep the ride within known bounds - unlike a "go-kart", for exam le. If
these and like factors are enough for substantial similarity, then the evidence 5 ould
be admitted under the Marois rule.

134

(CF-C or P)

The prior safe rides on the roller coaster are nonoccurrences compared to the occur-
rence of a tree branch allegedly striking a rider in the present problem. Evidence of
prior nonoccurrences can be introduced if certain conditions are satisfied. For example,
the court in Simon said that such evidence would be considered in light of three factors:
the closeness in time of the non occurrences; the static or nonstatic condition and
circumstances of the instrumentality in question; and the frequency with which the
instrumentality was used during the period of nonoccurrence.

In the present problem, only two of the factors might be satisfied. Some of the non
occurrences involving other riders were on the same night as the accident and the
number of those riders may have been fairly high. However, even if those two factors
are satisfied, the tree branch may not have been a static condition. The wind, for
example, might have blown the branch into the path of the roller coaster.

Thus, according to Simon, the nonoccurrence evidence should not be admitted because
all the factors are not satisfied. Therefore the offered nonoccurrence evidence from
the other roller coaster riders will not be accepted.

135

CF-C&P

Evidence that something has not happened before is only admissible if it is relevant.
But just because something has not happened before does not mean that it can’t happen.
Although arguments thus might be made both Wirs, Stanley v Schiavi was a lot like
what happened in this problem. Here we have a ro er coaster and proposed testimon
about the absence of prior accidents. Stanley involved a mobile home with a three inc
step leading from the main part of the mobile home into the living room. Stanley had

tripped over that step and sued the mobile home dealer.

The mobile home dealer sou t to rove that the dealer had no notice that the step
was defective or dangerous. e de er therefore attempted to introduce evidence that
several hundred persons had walked through the mobile home model without anyone
tripping. The a pellate court stated that the evidence was admissible as long as
su stantial simi arity of conditions was shown or an adequate number of incidents was
offered. Since several hundred persons had safe] ne otiated the step, the court held
tléat angtllequate number of incidents had been offxere and the evidence was therefore
a mum e.

 

With the roller coaster, the facts of the problem likewise do not show whether the
conditions were substantially similar at the time of the accident and at the time others
rode the ride. However, the proposed testimony again concerns a large number of prior
nonoccurrences. The Stanley opinion presented this as an alternative prerequisite for
admitting the evidence.

The logic apparently is that a sufficiently hi number of prior situations assures that
at least some of them were substantially simi ar. Whatever the limits of that logic might
be, at least it is persuasive for facts such as those presented b this roller coaster
situation. Here even more persons rode safely than had been invo ved in Stanley. Thus
the evidence should be admissible.

136

CF -C&P

The roller coaster owner has a problem. Even if thousands of ople have had safe rides
on the roller coaster, it doesn’t automatically mean that evi ence of the safe rides will
be admissible in court. The prior rides mi t have been safe for reasons totally
unassociated with the accident bein liti ated in court. Therefore, before the evidence
can go to the jury, the court must eci e whether it is relevant - whether it tends to
make more probable a fact material to the trial.

The case involving the train whistle perhaps could tprovide the court guidance in
deciding whether to admit the manager 5 testimony in '5 problem. In the train whistle
case, the plaintiff sought to prove that a train had not blown its whistle before it struck
plaintiff’s truck at a crossing. Plaintiff wished to present the testimony of other persons
who would testify that the train often failed to whistle at that crossing. Those persons
lived near the tracks with several having homes just about at the point where the whistle
should have been blown.

 

According to the train case, the key to the admission of nonoccurrence testimony is that
the prior nonoccurrence situations must be substantially similar to the situation that is
being litigated at the trial. Although the court noted that some factors probably would
be different each time the train went by, still it concluded that enough was the same to
allow the testimony into evidence.

 

The situation with the roller coaster is much the same - it even can be viewed as a small
train. The only difference is that the persons testifying would be those on the "train”
rather than in homes alongside the tracks.

In some ways, the roller coaster situation is even more likely than the train to involve
similar circumstances each time it runs. The train whistle depended upon a voluntary
human act. The en 'neer decision to blow the whistle could have been influenced in a
number of wa s. ere, however, the presence or absence of a tree branch does not
appear to be e result of a voluntary act. In addition, even though some things might
be different each time the roller coaster went by, it’s unlikel that a tree branch could
grow fast enough to suddenly hit a person where no one ha been hit before.

Since the court in the train whistle case held that prior nonoccurrences were sufficiently
similar to be admissible, the prior safe rides likewise should be able to be shown in this
problem through the testimony of the roller coaster manager. The roller coaster rides
are more apt to be the same than the engineer blowing the whistle, allowing this
evidence in will not broaden the train whistle case’s ruling about what incidents are
similar enough to be admissible.

Appendix I

PBBI'SOII pI’OdUC‘I moment correlations

LL FF SNS TANK CLASS
LL 1.000
P? 0.371 1.000
SNS 0.388 0.432 1.000
TANK 0.394 0.537 0.092 1.000
CLASS -0.325 -0.216 -0.201 -0.403 1.000
“GPA -0.004 0.291 0.216 0.159 0.074
LSAT 0.283 0.212 0.375 0.186 -0.190
LLFACTS 0.436 0.233 0.325 0.300 -0.111
FFFACTS 0.332 0.445 0.376 0.428 -0.333
SNSFACTS 0.406 0.425 0.467 0.061 -0.000
TANKFACT 0.340 0.366 0.411 0.096 '0.186
LLRULE 0.836 0.163 0.345 0.295 -0.302
SNSRULE 0.350 0.085 0.587 -0.138 0.000
TANKRULE 0.392 0.391 0.091 0.784 '0.267
FFRULE 0.052 0.699 0.088 0.403 0.111
UGPA LSAT LLIACTS FPFACTS SNSFACTS
UGPA 1.000
LSAT 0.170 1.000
LLFACTS 0.326 0.080 1.000
FFFACTS 0.378 0.172 0.630 1.000
SNSFACTS -0.049 0.195 0.356 0.356 1.000
TANKFACT 0.122 0.104 0.557 0.557 0.695
LLRULE -0.172 0.139 0.201 -0.050 0.161
SNSRULE -0.006 0.108 0.089 0.089 0.071
TANKRULE 0.016 0.068 0.089 0.089 0.071
FFRULE 0.185 -0.349 0.111 0.111 0.089
TANKIACT LLRDLB SNSRULB TINKRUL! PTRULB
TANKFACT 1.000
LLRULE 0.112 1.000
SNSRULE 0.050 0.443 1.000
TANKRULE 0.050 0.443 -0.071 1.000
FFRULE 0.062 0.050 .0.089 0.356 1.000

BARTLETT CHI‘SQUARE STATISTIC:

137

266.311 DP- 105 PROB-

.000

PF

SNS

TANK
CLASS
UGPA
LSAT
LLFACTS
FFFACTS
SNSFACTS
TANKFACT

SNSRULE
TANKRULE
FFRULE

"GPA

LLFACTS
PFFACTS
SNSFACTS
TANKFACT
LLRULB
SNSRULE
TANKRULE
FFRULE

TANKFACT
LLRULE
SNSRULB
TANKRULE
FFRULE

NATRIX OF PROBABILITIBS

LL

0.000
0.044
0.034
0.031
0.080
0.985
0.130
0.016
0.074
0.026
0.066
0.000
0.058
0.032
0.784

UGPA

0.000
0.370
0.079
0.039
0.797
0.521
0.364
0.974
0.933
0.327

NUMBER OF OBSERVATIONS: 30

138

0.000
0.017
0.002
0.251
0.119
0.261
0.215
0.014
0.019
0.047
0.390
0.655
0.033
0.000

LSAT

0.000
0.676
0.365
0.301
0.584
0.463
0.569
0.721
0.059

0.000
0.014
0.014
0.792

SNSRDLB

LLRACTS

TANK
0.000
0.630 0.000
0.287 0.027
0.253 0.401
0.041 0.324
0.080 0.107
0.040 0.018
0.009 0.747
0.024 0.613
0.062 0.114
0.001 0.466
0.633 0.000
0.645 0.027
FTEACTS
0.000
0.000 0.000
0.053 0.053
0.001 0.001
0.287 0.792
0.640 0.640
0.640 0.640
0.559 0.559
TANKRULB
0.000
0.708 0.000
0.640 0.053

0.000
0.696
0.314
0.559
0.072
1.000
0.326
0.105
1.000
0.153
0.559

SNSEACTS

0.000
0.000
0.395
0.708
0.708
006‘0

FFRULE

0.000