IOLOL.

5...:

15?;

z... n. :
$an 2 V

.c‘

1
.15...-
x..1u\| x.

 

 

€311.13

f. ‘ .. .113.
.Ix't
.ilb:

. x. . av;
53.1....
3: Uzi...

 

31.:
I £12333“. I
u: L. #7:. . pl

 

’93:»?!
r. .:
:1..,L.I,$r? .t 93.15.17).
5 T it! x .14 $2.: r.

 

. 1., .6.» A...

 

 

. :5: ya .1.
‘ E .53.;

 

‘ . .m.‘_.£..§§ “if?!

f: . ..I..

 

ll.

”by?“

Illllll MilliWilli. l‘il\l\\\\ll\l\\l

 

LIBRARY
Michigan State
University

 

 

 

This is to certify that the

dissertation entitled

PERCEIVED CONCEPTUAL BASES AND ACCEPTABILITY OF
CURRICULUM-BASED ASSESSMENT AMONG SCHOOL PSYCHOLOGISTS

presented by

David Fugate

has been accepted towards fulfillment
of the requirements for

 

 

Ph.D. , degreein Education

Harvey 2W . Clariz i

 

 

 

Major professor

Date //‘.5” ([3

MS U i: an Affirmative Action/Equal Opportunity Institution 0- 12771

 

PLACE IN RETURN BOX to remove this checkout from your record.
TO AVOID FINES return on or before die due.

     

DATE DUE DATE DUE DATE DUE

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

[Ii—J
.L——_____=______,

ll
F—ll l

MSU Ie An Affirmative ActiorVEquai Opponunity Institution
emprnS-pt

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

|J

 

 

PERCEIVED CONCEPTUAL BASES AND ACCEPTABILITY OF CURRICULUM-
BASED ASSESSMENT AMONG SCHOOL PSYCHOLOGISTS

BY

David Fugate

A DISSERTATION

Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of

DOCTOR OF PHILOSOPHY
Department of Counseling, Educational Psychology
and Special Education

1993

ABSTRACT

PERCEIVED CONCEPTUAL BASES AND ACCEPTABILITY OF CURRICULUM-
BASED ASSESSMENT AMONG SCHOOL PSYCHOLOGISTS

BY

David Fugate

This study dealt with two separate research topics that
have become increasingly popular over the last decade. The
first line of research investigated was that dealing with
Curriculum-based Assessment (CBA). CBA has become a buzz-
word for alternative assessment techniques based upon the
use of the school's current curriculum. The second line of
research investigated was the behaviorally oriented research
dealing with treatment acceptability. The latter research
appears to have great potential for understanding how and
when CBA is used as well as investigating its potentials and
shortcomings in actual practice. Despite this potential,
there has been a paucity of research investigating the
acceptability of CBA among practicing school psychologists.

This study investigated these two areas of research in
a survey of CBA use and perception. The literature pertain—
ing to both CBA and treatment acceptability was reviewed.
Following this review a rationale was presented for isolat—
ing factors from the treatment acceptability literature with

possible significance for the investigation of the

acceptability of CBA. Purported advantages and disadvantag-
es of CBA were identified from the review of CBA literature.
The extent to which practitioners agreed or disagreed with
the identified advantages and disadvantages was investigat-
ed. In addition, the study explored the perception of
practitioners with regard to the conceptual bases of CBA.
Also, the possibility of relationships between demographic
factors of school psychology practitioners and the accept-
ability of CBA was investigated.

The results of the study suggested that there was only
slight agreement among practitioners with regard to the
conceptual bases of CBA. The majority of factors related to
treatment acceptability were also found to be related to the
acceptability of CBA. There was little consensus found
among practitioners with regard to the proposed advantages
and disadvantages of CBA. Finally, there was no evidence of
a relationship between the demographic factors investigated

and the acceptability of CBA.

ACKNOWLEDGEMENTS

The completion of this document would not have been
possible without the assistance and support of the following
people.

I would like to thank Dr. Lucian Parshall and Denise
Smith with the Michigan Department of Education, Office of
Special Education. Their technical support was invaluable.

I would also like to thank my committee members, Dr. A
Herbert Burks, Jr., Dr. Martha Karson, and Dr. William
Mehrens. Their editorial and technical input was truly
valued. I would also like to thank Dr. Deborah Bennett for
her role as my "unofficial" committee member. Her advice
and support were appreciated.

I would like to thank Dr. Harvey Clarizio, my major
advisor. His challenging comments and keen editorial skills
made this document possible. I am honored to consider him
my mentor.

Finally, I would like to thank my family. This effort
would not have been possible without the support and pa-
tience of my wife, Ann, and my children, Samantha and

Brianna.

iv

TABLE OF CONTENTS

INTRODUCTION . . . . . . . . . . . . . . . . . . . . 1
REVIEW OF THE LITERATURE . . . . . . . . . . . . . . 6
History of CBA . . . . . . . . . . . . . . . . . 7
Definitions of CBA . . . . . . . . . . . . . . . 8
Examples of CBA . . . . . . . . . . . . . . . . . 11
Conceptual Bases of CBA . . . . . . . . . . . . . 18
Treatment Acceptability . . . . . . . . . . . . . 22
Proposed Advantages of CBA . . . . . . . . . . . 30
Possible Disadvantages of CBA . . . . . . . . . . 34
Acceptability of Curriculum-Based Assessment . . 36

Summary of Literature and Implications for the

Study of CBA Acceptability . . . . . . . . . . . 38
METHODOLOGY . . . . . . . . . . . . . . . . . . . . . 45
Identification of Research Questions . . . . . . 47
Measurement Instrument . . . . . . . . . . . . . 56
Sample . . . . .i. . . . . . . . . . . . . . . . 64
Statistical Analysis . . . . . . . . . . . . . . 78
RESULTS AND DISCUSSION . . . . . . . . . . . . . . . 81
Research Questions . . . . . . . . . . . . . . . 82
Demographic Data . . . . . . . . . . . . . . . . 96
Limitations . . . . . . . . . . . . . . . . . . 101
Discussion . . . . . . . . . . . . . . . . . . 104

SUMMARY AND RECOMMENDATIONS

Implications for Future Research

Implications for Practice

APPENDIX A: Cover Letter

APPENDIX B: Results Letter and Summary Statistics

LIST OF REFERENCES

vi

113

119

123

126

127

130

LIST OF TABLES

Table 1. Summary of CBA Models . . . . . . . . . . 17
Table 2. Tenets from Assessment Models . . . . . . 49
Table 3. Common Factors from Treatment

Acceptability and CBA Research . . . . . . . . . . . 53
Table 4. Purported Advantages and Possible
Disadvantages of CBA . . . . . . . . . . . . . . . . 54
Table 5. Questionnaire: Items, Purpose and

Source . . . . . . . . . . . . . . . . . . . . . . . 57
Table 6. Demographic Questions . . . . . . . . . . 61
Table 7. Questionnaire . . . . . . . . . . . . . . 62
Table 8. Summary Statistics . . . . . . . . . . . . 83
Table 9. Item Means, Standard Deviations:

Conceptual Model Items . . . . . . . . . . . . . . . 86
Table 10. Correlation Matrix: Treatment Acceptability
Items . . . . . . . . . . . . . . . . . . . . . . . . 87

Table 11. Item Means, Standard Deviations:
Advantages/Disadvantages of CBA . . . . . . . . . . . 97

Table 12. Analysis of Variance of Theoretical
Perspective and CBA Acceptability . . . . . . . . . . 98

Table 13. Analysis of Variance of Training Institution
and CBA Acceptability . . . . . . . . . . . . . . . . 98

Table 14. Analysis of Variance of Degree Held and
CBA Acceptability . . . . . . . . . . . . . . . . . . 98

Table 15. Correlations: Demographic Items and
CBA Acceptability . . . . . . . . . . . . . . . . . . 99

vii

ANOVA

CBA

CBM

CBM-ID

CR-CBA

CBE

NST

NASP

TEI

MET

IEP

LIST OF ABBREVIATIONS

Analysis of Variance

Curriculum-based Assessment

Curriculum-based Measurement

Curriculum-based Measurement-Instructional Design
Criterion Referenced-Curriculum Based Assessment
Curriculum Based Evaluation

Norm-referenced Test

Nationally Standardized Test

National Association of School Psychologists
Treatment Evaluation Inventory

Assessment Rating Profile

Multidisciplinary Evaluation Team

Individualized Educational Program

viii

INTRODUCT I ON

Over the last decade there has been increasing inter-
est in the use of the current school curriculum as a basis
for assessment of educational problems. This technique has
come to be known as Curriculum-based Assessment, or CBA.

CBA techniques have been offered as a means to both supple-
ment and supplant the more traditional standardized, norm-
referenced tests. Supporters of CBA have written prolifi-
cally about the technical adequacies and practical advan-
tages their technique offers over traditional assessment ap-
proaches (see Rosenfield & Shinn, 1989 and Bagnato, 1986 for
examples of reviews). Over this period of time, CBA advo-
cates have also espoused various distinct models of CBA
(Shinn, Rosenfield, & Knutson, 1989). Recently, the concep-
tual underpinnings of CBA have also been more clearly delin-
eated (Knutson & Shinn, 1991), allowing practitioners to
consider the research on CBA from an established conceptual
perspective. What has not been investigated is the extent
to which practitioners are aware of, or agree with, the
proposed conceptual bases of CBA.

Entire issues of journals, as well as a number of
texts, have been dedicated to the study of CBA techniques.

It would seem that CBA is a popular, timely alternative to

2
traditional assessment. However, despite this proliferation
of press advancing the utilization and advantages of CBA
techniques, there has been a paucity of research with regard
to their popularity among practitioners. The question
remains to be answered: Who is using CBA techniques and for
what purposes?

In a separate line of research, behaviorally oriented
researchers have endeavored to identify those factors that
contribute to a treatment's acceptability, a construct that
has proven to be quite important to both providers and
recipients of treatments. These researchers have demon-
strated that an effective technique is not necessarily a
utilized technique. There are other pertinent issues and
dimensions of a treatment that help to determine how that
treatment is perceived by both those who use the treatment
and those who will receive the treatment. Although CBA is
not a treatment per se, it has been suggested that one of
its advantages over traditional assessment is the strong
link it provides between assessment and teaching. In this
sense CBA might be perceived as an educational treatment of
sorts. Also, CBA techniques have been based upon behavioral
models of assessment, providing further support for the
appropriateness of considering the implications that the
research on treatment acceptability may have for the accept-
ability of CBA techniques. While the research on treatment
acceptability has been applied, albeit sparingly, to the

field of education, to date the two lines of research (i.e.,

3
CBA and treatment acceptability) have crossed only once
(Shapiro & Eckert, 1993). This is unfortunate, as the
research on treatment acceptability may prove as useful to
the investigation of assessment techniques like CBA as it
has in investigating behavioral treatments.

The present study attempted to link these two formerly
disparate lines of research. Subsequently, the implications
from these research findings were utilized to assess the
degree of acceptability of CBA by those most likely to be
responsible for its implementation--current practitioners of
school psychology.

The present study also offered some definitions of
CBA, provided a brief history of its development, and con-
sidered the different models discussed in the literature.
The proposed conceptual underpinnings of CBA were then dis-
cussed. The assessment models upon which CBA is supposedly
based were reviewed and characteristics were identified for
considering the link between these assessment models and
CBA. Next, the separate line of research dealing with
treatment acceptability were reviewed. From this consider-
ation, the rationale for considering the link between CBA
and the behaviorally based empirical research on treatment
acceptability was offered. The study then considered the
empirical implications that have been forthcoming from the
behavioral research on treatment acceptability over the last
decade. From this line of research, a number of pertinent

factors were delineated that, it was proposed, should have a

4
direct impact on the acceptability of CBA techniques as
well. Next, the research on CBA was revisited in order to
provide the background for the proposed advantages suggested
by its advocates that may influence how the technique is
perceived by practitioners as well as to provide more
rationale for linking the lines of research. Conversely,
the research that has identified possible disadvantages of
CBA techniques was reviewed to consider its possible impact
upon the acceptability of CBA techniques. Finally, the
fledgling research on CBA acceptability was discussed.

The results of these various reviews pointed to a number
of factors from the research (i.e., conceptual underpinnings
of CBA, empirical evidence from treatment acceptability, and
the consideration of advantages and disadvantages of CBA)
that should have an influence on how CBA is perceived by
school psychology practitioners. These factors provided the
basis for the questionnaire designed to assess the conceptu—
al understanding and degree of acceptability of CBA tech-
niques among practitioners. The various factors also pro-
vided for an analysis of how the perception of acceptability
is formed. How CBA is perceived, and how those perceptions
are formed, will have important implications for both crit-
ics and proponents of CBA. The implications of the research
are important for consideration of future research in order
to address perceived deficits, investigate perceived advan-

tages, and enhance utilization of CBA techniques among

5
practitioners, where indicated. The present study was an

attempt at providing this knowledge.

REVIEW OF THE LITERATURE

The review of the literature addressed two areas of
research that have been characteristically distinct. The
first area was that of CBA. The history of CBA was very
briefly reviewed. Next, some definitions of CBA were of-
fered as well as some consideration of the difficulty in—
volved in defining what constitutes CBA. Following this,
some models of CBA were presented to provide the reader with
some familiarity with popular CBA techniques. The focus of
the literature review then shifted to the three major areas
that provided the themes for the current investigation. The
first consideration pertained to the research that has
traced the conceptual underpinnings of CBA. This literature
provided a context with which to compare current models of
CBA. Next the review examined a somewhat separate body of
research--the empirical investigation of the acceptability
of behavioral treatments. This research provided the inves-
tigational framework for the present investigation as well
as the several themes that appeared promising in the inves-
tigation of the acceptability of CBA. The next area of
review focused on some of the proposed advantages and disad-
vantages of CBA that have been the mainstay of current

research on CBA. This area provided a number of potential

factors that may affect practitioners' perceptions of CBA.
Finally, the very recent and limited area of CBA acceptabil-
ity was reviewed. The limitations of the current research
pointed out the need for further investigation. A summary
of the literature review concluded this section while set-

ting the stage for the study's rationale.

W

The history of CBA, not unlike the understanding of
the concept itself, was somewhat unclear. In an article by
Tucker (1985), the first use of CBA was traced to Gickling
in 1981, as a method of assessment for school psychologists.
In a much earlier article, Eaton and Lovitt (1972) discussed
the use of achievement tests versus direct and daily mea-
surement based on the curriculum. While no specific use of
the term CBA was found in that article, Eaton and Lovitt
(1972) discussed the use of curriculum measurement as a
replacement for standardized achievement tests in a manner
that certainly suggests their work anticipates, if not
directly provides the foundation for, the work of later CBA
advocates. They suggested the use of their technique for
assessing, and subsequently placing, students for instruc—
tional purposes. In a manner anticipating much of the work
to follow, they also pointed out the disadvantages of the
use of standardized achievement tests. It seemed reasonable
to consider the Eaton and Lovitt (1972) article as one of

originators of the CBA method, if not the term itself.

8

Of course, this consideration of history was not meant
to suggest that the use of curriculum as a means of measur—
ing student achievement was only as recent as 1972. The use
of quizzes and tests based on what has been taught most
likely predated written history. However, the particular
use of the terms Curriculum-based Assessment and Curriculum-
based Measurement, and their employment by school psycholo-

gists, seemed to be a relatively recent development.

I E' 'l' E SEE

Perhaps one of the most perplexing tasks involved in
understanding Curriculum-based Assessment (CBA) involved
defining the term. What is CBA? According to Tucker
(1985), it was nothing new. He defined CBA as assessing the
student in terms of the curricular requirements of the
particular student's school setting: "In curriculum based
assessment the essential measure of success in education is
the student's progress in the curriculum of the local
school" (p. 199).

It has been suggested that Curriculum-based Assessment
may be defined as "the practice of obtaining direct and
frequent measures of a student's performance on a series of
sequentially arranged objectives derived from the curriculum
used in the classroom" (Blankenship & Lilly, 1981, p. 81).

Perhaps adding to the possible confusion of defining
the concept of CBA was the alternate, and often overlapping,

use of the term Curriculum-based Measurement (CBM). The

9
primary proponent of this term was Stanley Deno. Deno
wrote,

An emerging alternative to commercial standard-

ized tests and to informal observations is cur-

riculum-based measurement (CBM) that combines the

advantages of both. Through standardizing obser-
vation of performance in the curriculum, CBM

generates reliable data that is valid with re—

spect to widely used indicators of achievement

such as achievement test scores, age, program

placement, and teachers' judgement of competence

(Deno, 1985, p. 219).

From an analysis of the statements presented above,
one saw that there was considerable overlap between CBA and
CBM and was unclear as to how the terms were actually dif—
ferentiated. Also, neither term was particularly concise
and both seemed to be somewhat short of an operational
definition. Perhaps Gickling and Thompson (1985) were
presenting an example of the "state of the art" when they
chose to label their article "A personal view of curriculum-
based assessment." In doing so, these authors acknowledged
the degree of diversity currently involved in the practice
of CBA. However, for the sake of clarification, some effort
towards distinguishing between the terms and arriving at
some level of understanding as to what constitutes CBA
seemed warranted.

There did appear to be some consistency with regard to
what constituted a CBA technique. Tucker (1985) offered the
following characteristics:

1. Is based on the curriculum as defined by local

boards, state and federal law, etc. as it applied to that

10
particular grade (e.g., each grade, school class would have
its own "test").

2. Assumed that curricular outcomes are known and
measurable.

3. Provided a valid measure of achievement.

4. Provided instructionally useful data (some propo-
nents have argued it does so, whereas standardized achieve—
ment tests do not).

5. Was the ultimate example of teaching the test.

More recently, Tucker (1987) offered the following
requirements of CBA: Measurement materials are drawn from
the curriculum, measurement is ongoing, and assessment
information is used to formulate instructional decisions.
It appeared that most proponents of CBA agreed upon the
following characteristics: (1) it is an assessment tech-
nique based upon and utilizing as its test domain an estab-
lished curriculum currently being used to instruct the
student in some specific skill area; (2) the aforementioned
technique is employed to gain an understanding of the
student's current level of functioning in the skill area as
opposed to the employment of a standardized achievement
test. Other characteristics of CBA that seemed less widely
agreed upon included: (1) CBA is based upon a developmental
curriculum; (2) the results of CBA are used to adjust the
student's curriculum to the current (as measured) level of
functioning; (3) CBA is employed on an ongoing basis with

regular monitoring of the student's performance, typically

11
through the use of graphs, and linked to the student's
program of instruction. Of course this list of character-
istics may not be allsinclusive nor universally accepted,
but it did provide the beginning of a definition for what
constitutes CBA.

It appeared that the distinction between CBA and CBM
has focused on the level of specificity. CBA was seen as
the more generic of the terms, encompassing a wide variety
of approaches and authors. CBM, on the other hand, has
tended to be more closely associated with the work of Stan-
ley Deno and others involved in the research conducted at
the University of Minnesota. Fuchs, Fuchs, and Hamlett
(1990) suggested that CBM qualifies as a technique of CBA
because it meets the three criteria outlined by Tucker
(1985). Additionally, they noted that CBM is unique as a
result of two additional properties. The first property
they cited was that CBM is conducted on samples from an un-
changing pool of stimulus materials composing the year long
curriculum. The other additional feature was that CBM is

highly prescriptive and standardized.

W

It was not possible, within the scope of this study,
to discuss or even list the numerous examples of CBA within
the literature. As Shinn and Hubbard (1992) have pointed
out, CBA is not one set of unified testing strategies or

procedures. Rather, CBA encompassed a range of approaches

12

from teacher-made criterion referenced tests (Blankenship,
1985), procedures designed to determine a student's frustra—
tional, instructional, and independent academic levels
(Gickling & Thompson, 1985), to projects designed to provide
local norms for entire school districts and beyond (Deno,
1985). This perspective seemed reasonable and demonstrated
the difficulty involved in analyzing "the model of CBA."
What was possible, however, was to consider a few models
that provided examples of an operational definition of CBA.

While CBA could theoretically be applied to any school
curriculum, it seemed that the most popular use of the model
involved the measurement of reading, perhaps due to the
perceived simplicity involved in measuring this particular
curriculum area. One popular CBA model of reading was that
proposed by Deno (1985). Deno's technique consisted of
having students read aloud from basal texts. The number of
words read aloud correctly served as the criterion of mea-
surement for reading achievement. Deno claimed that this
relatively simple task had high criterion validity as demon—
strated by significant correlations of his CBA technique (he
refers to it specifically as Curriculum-based Measurement)
with standardized achievement tests. The number of words
read correctly (he makes it a point to note this is in
contrast to measuring number of words read incorrectly) was
expected to rise over the course of the child's development
in a fashion similar to increases in height and weight

(Deno, 1985). Deno has carried this technique further by

13
gathering data from various classes, schools, and districts
in a manner that allowed him to make comparisons (assuming
one has followed proper standardized procedures, etc.),
among students in a manner that might be described as the
construction of a locally normed achievement test. While
the technique itself employed by Deno was certainly parsimo-
nious, the work involved in gathering local norm data may
have complicated the technique.

Another model of CBA applied to reading was that
offered by Gickling and Thompson (1985). This particular
technique also involved reading aloud from the basal texts
but incorporated other criteria as well. It should be
mentioned that the authors explicitly stated the need for a
developmentally based curriculum upon which to base the CBA.
In this way, they suggested, the curriculum itself provided
a developmental measure of reading ability. Such a proposal
assumed that the developmental curriculum was validly and
reliably constructed in such a manner as to make these
interpretations meaningful. Interestingly, more recent
research (Fuchs & Deno, 1992) has suggested that CBM is
quite robust in the face of curriculum variation. These
authors investigated the effects of curriculum variation on
the technical features of criterion validity and developmen-
tal growth rates. Their findings suggested that CBM's
technical features (i.e., criterion validity and developmen-

tal growth rates) remained strong despite variation in the

14
curriculum. It remains to be seen whether this robustness
will generalize to other CBA techniques as well.

In addition to the requirement of reading text aloud,
the Gickling and Thompson (1985) technique also attempted to
establish levels of performance. They defined these as
frustrational, independent, and instructional. They sug-
gested that when the curriculum is too easy or too hard, the
child will struggle with boredom or frustration, respective-
ly. Instead, it is believed that an instructional level
consisting of 93 to 97% known words with a 3 to 7% challenge
is desirable. In this way, a level of curriculum matched to
the child's current level of functioning could be identi-
fied. The explicit intent of this model was the application
of the assessment results to effect changes in the curricu-
lum. That is, the curriculum would be modified to address
the level of the individual.

The model proposed by Blankenship (1985) focused on
the connection between curriculum assessment and instruc-
tion. She offered the following definition of CBA:

...the practice of obtaining direct and frequent

measures of the student's performance on a series

of sequentially arranged objectives derived from

the curriculum used in the classroom (Blankenship

& Lilly, 1981, p. 81). The essence of the ap-

proach is the linking of the assessment to our-

riculum and instruction (p. 234).

This technique called for the use of CBA's over several days
within a classroom in order to make individual instructional

decisions. This appeared to be a purely "teacher driven"

model. In her paper, Blankenship (1985) outlined a generic

15

model that could be applied to a variety of curriculum
areas. Her steps included: list the skills presented in the
material selected; examine the list to see if all important
skills are presented; decide if the resulting, edited list
has the skills in logical order; write an objective for each
skill on the list; prepare items to test each listed objec-
tive; prepare testing materials for student use, presenting
the items; plan how the CBA will be given; give the CBA
immediately prior to beginning instruction on the topic;
study the results to determine those students who have
mastered the skills targeted for instruction, possess suffie
cient knowledge of the necessary prerequisite skills and are
ready to begin instruction, and those lacking the prerequi-
site skills; re-administer the CBA after instruction on the
topic. Study the results and identify those students who
have mastered the material and are ready to begin a new
topic; those making progress, but in need of more instruc-
tion; and those making insufficient progress, requiring the
teacher to modify some aspect of instruction; and periodi-
cally re-administer the CBA throughout the year to assess
for long-term retention. While the model proposed by Blank-
enship (1985) was perhaps the most generalizable, the level
of measurement knowledge it seemed to require was consider-
able.

Given that there seemed to be a proliferation of CBA
models, and no clear means with which to differentiate them,

Shinn, Rosenfield, and Knutson (1989) proposed a set of

16
dimensions upon which CBA models could be compared (see
Table 1). The dimensions included: relationship to assess-
ment and decision making; relationship of assessment data to
instruction; type of student response; focus of material
when student progress is monitored; test formats; technical
adequacy; and utility for making other decisions. They
suggested that these dimensions could be utilized to provide
meaningful comparisons amongst what they identified as the
(at least) four current models of CBA: Curriculum Based
Assessment for Instructional Design (CBA-ID); Criterion-
Referenced-Curriculum Based Assessment (CR-CBA); Curriculum,
Based Measurement (CBM); and Curriculum-Based Evaluation
(CBE). While it is unlikely that any assessment model can
evaluate all aspects that might differentiate between the
various versions of CBA, the model proposed by Shinn et al.
(1989) seemed to provide a useful means for delineating
among these varied techniques. In addition, they made
explicit the fact that different CBA techniques do not
provide similar information. By assessing relative
strengths and weaknesses along the dimensions, practitioners
of CBA techniques would be more likely to employ the "cor-
rect" technique for their particular application. In addi-
tion, models such as that proposed by Shinn, et al. (1989)
reflected an increasing awareness among CBA proponents of

the need to critically examine their models.

Table 1.

17

Summary of CBA Models

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Features CBM-ID CBM (Deno) Cit-CBA CBE
(Gickling) (Idol\Blank- (Howell)
enship)
Relation to Instruction: Student Instruction- Instruc-
Assess. & al Planning Progress a1 Planning tional
Decision Monitoring Planning
Making
L Timing in Ongoing, Ongoing, Pre-testing Ongoing,
Instruc— most of with most of and mastery' with most
tional assessment assessment testing, most of assess-
Planning’ preceding after ini- of assessment ment pre-
initial in- tial preceding ceding
struction instruction initial in— initial
struction instruc-
tion
Type of Production Production Variable ‘Variable
Student
Response
Focus of Short Term Long Term Short Term Short Tern:
Material for
Monitoring
Progress
Test Format Short Short Varies Varies I
Duration Duration
Data on Content Reliability Content va- Content
Technical Validity, Criterion- lidity Validity
Adequacy Construct Related
Validity Validity
Construct
u Validity
Adapted from Shinn, Rosenfield, and Knutson (1989).

18
W

A recent article by Knutson and Shinn (1991) clarified
the conceptual basis from which CBA works as an assessment
technique. The authors noted that there has been some
confusion with regard to what constitutes CBA. It was
suggested that CBA is perceived as a set of essentially
similar, if not unified, assessment techniques. In order to
clarify the conceptual basis for their particular model
(CBM) the authors presented it as a procedure built upon the
foundation provided by three interrelated models of assess-
ment.

It was suggested that the basic measurement strategies
of CBM were derived from a behavioral assessment perspective
(Deno, Mirkin, & Shinn, 1979). Knutson and Shinn (1991)
noted the similarity between the desirable criteria devel-
oped for CBM measures by Jenkins, Deno, and Mirkin (1979)
and the principles of behavioral assessment delineated by
Barlow, Hayes, and Nelson (1984). The first principle
suggested by Barlow et a1. (1984) was that behavioral as—
sessment measures socially valid behaviors, that is, a
meaningful measure of the presenting problem. Knutson and
Shinn (1991) suggested that this corresponded to the tenet
of Jenkins et al. (1979) that a measurement system must
generate data that provide a fair representation of tasks
that students are taught. The second shared principle was
that of repeated measurements over time. In terms of CBM,

this aspect of the measurement system allowed for the

19
teacher to make instructional changes when expected rates of
progress were not obtained (Jenkins et al., 1979). The
behavioral perspective has called for measures prior to,
during, and after the treatment in order to determine effi-
cacy. Both systems touted the necessity for measures that
were of short duration and administrative ease (Knutson &
Shinn, 1991). Another principle common to both was preci-
sion. Knutson and Shinn (1991) noted the importance of
"molecular" measures in the behavioral realm and the need
for CBM to be sensitive to small changes over time. Final-
ly, both the behavioral assessment model and CBM shared the
principle that each intervention plan be treated as a test-
able hypothesis with respect to its efficacy.

Knutson and Shinn (1991) suggested that the second
assessment model upon which CBM was based was the ecological
model. This model (Lentz & Shapiro, 1986) emphasized the
structured analysis of the student's environment (e.g.,
instruction curriculum, fidelity of curriculum implementa-
tion, and interactions with the student's performance).
Lentz and Shapiro (1986) outlined three major assumptions
that defined educational assessment from an ecological per—
spective. The first of these was that remediating academic
deficits in the classroom was the purpose of psychological
services. As a result, they suggested that assessment data
were valuable only to the extent that the information yield-
ed promoted meaningful outcomes. The second major assump-

tion was the multiplicity of causal factors that affect poor

20
performance. According to this assessment model an examina-
tion of the student's performance, the classroom environ-
ment, and the interaction between these was necessary for
understanding and ameliorating academic problems. Finally,
remediation of academic problems required direct, effective
instruction of deficit skills and/or manipulation of the
environmental variables that may be contributing to the
problem. Lentz and Shapiro (1986) suggested that the prin-
ciples of behavioral assessment and ecological assessment be
combined in order to improve educational assessment. Knut-
son and Shinn (1991) have argued that the integration of CBM
into practice required adoption of the underlying ecological
assessment principles and incorporation of its data collec-
tion strategies in addition to the adoption of the behavior-
al assessment principles cited previously. The importance
of linking assessment to intervention was exemplified by
both approaches.

Knutson and Shinn (1991) cited one final philosophical
underpinning of CBA in suggesting that it is closely linked
to the problem-solving model conceptualized by Deno (1989).
In this data-based model educational assessment was driven
by the specific referral problem and the educational deci-
sion to be made (Deno, 1989; Shinn, Nolet, & Knutson, 1990).
Deno (1989) stated that this model was grounded on three
underlying assumptions. The first was that a problem exists
when there is a discrepancy between a student's actual

behavior and that which is expected of him or her. Similar

21
to the two previous models, problems were conceived as
situational in nature rather than residing within the child.
The second premise of this model was that since general
education fails to educate all students effectively, these
students will demonstrate the aforementioned discrepancy.
In this case, some modification of environmental factors
would have been necessary. Deno (1989) suggested it was a
value judgement as to what constituted a discrepancy so
severe as to require allocation of services beyond general
education. The final premise was that the task of educators
was to work towards solving the educational problems that
students experience in schools (Knutson & Shinn, 1991). The
challenge of planning effective interventions required that
implementors were capable of evaluating the effects of their
interventions and generated multiple interventions when
initial ones failed.

Knutson and Shinn (1991) stated that the integration
of CBM into practice was based on the principles of all
three of these interrelated models. They argued that CBM
was not designed as a series of tests to be used in isola-
tion. Rather, implementation was based on the following
fundamental underpinnings derived from the assessment mod-
els: (a) academic behaviors that students exhibit in the
classroom must be observed directly and frequently, (b)
academic deficits must be viewed in an ecological context as
an interaction between student behavior and academic perfor-

mance, and (c) student performance data using CBM provide a

22
basis for making decisions in a problem solving model in-
tended to meet students' educational needs.

The extent to which the conceptual underpinnings
delineated By Knutson and Shinn (1991) as constituting the
basis for CBM applied to other models of CBA was, of course,
dependent upon the specific model in question. However, the
major premises of the models cited appeared to fit closely
with the characteristics of CBA discussed in the sections on
the definition and models of CBA. To the extent that CBA
techniques have a common conceptual background, the princi-
ples cited by Knutson and Shinn (1991) appeared to general-
ize well to the majority of CBA techniques. The degree to
which an individual practitioner identifies with the princi-
ples derived from the models, or the models themselves, has
not been investigated. It is suggested that this informa-
tion is important. The degree to which practitioners agree
upon the conceptual ties of assessment models to CBA, and
the characteristics of the models which have been proposed
as underlying CBA, may help clarify the question of exactly
what constitutes a CBA technique. It is hoped that this
information may lead to a clearer delineation of the charac-
teristics of CBA and a more uniform approach to investiga—

ting its utility and limitations.

I I E I l'J'l
The proponents of CBA have suggested that their tech-

nique is a viable alternative to traditional assessment;

23
others have expressed reservations. Regardless of the
effectiveness of CBA, or whether or not it is superior to
nationally standardized tests (NST's), the question of its
acceptability remained. As Mehrens and Clarizio (1993) have
suggested, assessment/intervention linkages must be effec-
tive and acceptable if practitioner usage and adherence to
procedures are to be achieved.

Knutson and Shinn (1991) have argued that CBM has
strong conceptual ties to a behavioral model of assessment.
It has been suggested that the basic measurement strategies
of CBM were derived from a behavioral perspective (Deno,
Mirkin, & Shinn, 1979). An emphasis on direct measurement
of behavior in measurable and observable terms was one of
the characteristics shared by CBM and behavioral assessment
(Knutson & Shinn, 1991). Because CBM could be conceptual-
ized as a behavioral model for academic assessment, there
was reason to believe the research in behavioral psychology
on behavioral treatment acceptability was pertinent to CBA
as well. As CBM represented just one example of CBA tech-
niques, it was conceivable that the treatment acceptability
research was applicable to other CBA techniques (i.e., those
with an emphasis on direct measurement of observable behav-
ior), as well.

Research on behavioral treatment approaches with
utility for education began to receive attention in the late

1960's (Miltenberger, 1990). Initially, these studies

24
focused almost exclusively upon the relative effectiveness
of the treatments. Subsequent to this focus on
effectiveness was a realization that in order for a treat-
ment to be utilized it must not only be effective, it must
also be acceptable to those responsible for its implementa-
tion (Kazdin, 1977; Wolf, 1978). Central to this argument
was the notion that, despite its effectiveness or efficien-
cy, practitioners might ignore a treatment that is not
acceptable to the individuals receiving the treatment or
their advocates (e.g., parents, administrators, etc.).
Kazdin was the first to define treatment acceptability and
attempt to operationalize the concept in a way that would
lend itself to being measured (Miltenberger, 1990). Central
to Kazdin's (1980) definition of treatment acceptability
were the judgements of the treatment by consumers of the
treatment. He suggested that a treatment may be judged
acceptable based upon the perception that it is appropriate
to the problem, fair, reasonable, non-intrusive, and consis-
tent with notions about what a treatment should be. Much of
the subsequent investigation of treatment acceptability has
focused on differentiating treatment acceptability and
treatment effectiveness. Kazdin (1980) was successful in
demonstrating that a measure of treatment acceptability (the
Treatment Evaluation Inventory or TEI), was able to readily
differentiate treatments according to their relative degree

of acceptability.

25

The first factor of treatment acceptability to be
empirically investigated was the relationship between treat-
ment acceptability and effectiveness. The two concepts
appeared to be closely related, but not identical. An
effective treatment was one that changed a behavior in the
desired direction, whereas an acceptable treatment was de—
fined more subjectively as suggested by Kazdin (1980).
Thinking on the relationship of the concepts has flowed in
both directions. One possibility was that a treatment that
was more effective would be more acceptable. However, the
converse also seemed reasonable: the more acceptable a
treatment was the more effective it was likely to be due to
factors such as treatment compliance and integrity (Von
Brock & Elliott, 1987). Attempts to delineate this rela-
tionship have appeared in the research literature. Kazdin
initially (1981) suggested that there appeared to be no
relationship between a treatment's efficacy and its accept-
ability. Subsequent researchers found the constructs of
effectiveness and acceptability to be positively related
(Kirgin, Braukmann, Atwater, & Wolf, 1982). Following this
work, the focus turned to exploring the relationship of
these constructs. Shapiro and Goldberg (1986) were able to
demonstrate that treatments viewed as equally effective were
not necessarily viewed as equally acceptable. Intervening
factors appear to affect the relationship between accept-

ability and effectiveness.

26

One factor related to the acceptability of a treatment
was whether it was perceived as being positive or negative
in nature. Witt, Elliott, and Martens (1984) demonstrated
that teachers rated positive interventions (reinforcements)
as more acceptable than negative interventions (punitive or
response cost contingencies). Despite the fact that some
"negative" interventions may be very effective, if they were
not viewed as acceptable they may not be utilized by treat-
ment consumers.

Von Brook and Elliott (1987) further differentiated
the concepts of treatment acceptability and effectiveness
and attempted to explore their relationship. Utilizing an
analogue study with treatment variation (time-out, token
economy, and response cost), effectiveness variation, and
problem severity variation, they hoped to show how previous—
ly identified factors interacted to affect treatment accept-
ability. The authors were able to show that treatment
acceptability and effectiveness were distinct factors, but
that they were closely related. In addition, they were able
to shed some light upon this relationship. There appeared
to be a strong relationship between effectiveness and ac-
ceptability. When teachers viewed a treatment (time-out) as
less acceptable, they also rated it as less effective. This
finding was interpreted as further delineating a relation-
ship from previous research that positive interventions were
consistently more acceptable or more effective than negative

interventions (Witt, Elliott, & Martens, 1984). Another

27
finding was that problem severity and effectiveness informa-
tion interacted to affect the ratings of treatment accept-
ability. Interestingly, the effectiveness information
appeared to have the greatest influence when the problem
severity was milder. The authors suggested that teachers
may feel there is more latitude in dealing with milder
problems, and tend to rely more on personal judgement or
past experience when problems become severe.

In a review of the literature on the acceptability of
behavioral interventions, Reimers, Wacker, and Koepl (1987)
were able to identify several other factors related to
acceptability. In addition to the factors of problem sever-
ity and positive or reductive characteristics of the treat-
ment, Reimers et al. also noted that the amount of time
needed to implement a procedure has differential effects on
acceptability. Kazdin (1982) first suggested that proce-
dures that require more time or resources, such as token
economies, may not be implemented consistently unless done
so in controlled settings. Witt, Elliott and Martens,

(1984) also found that teachers' ratings of acceptability of
a treatment varied as a function of the time needed to
implement the procedure. They found that as the time needed
for implementation increased, acceptability decreased. In
another study, Elliott, Witt, Galvin, and Peterson (1984)
were able to demonstrate that two factors, time and positive
or reductive characteristics of the intervention, interact

to differentially affect the acceptability of the treatment.

28
Their results suggested that teachers perceive interventions
as more positive when they both require less time to imple-
ment and are positive in nature.

Another possible factor involved in treatment accept-
ability identified by Reimers et al. (1987) was that of side
effects. Kazdin (1981) found that the stronger the adverse
side effects, the more likely it was that treatment accept-
ability would suffer.

A number of researchers have also suggested that the
consultee's level of understanding affects treatment accept-
ability. Reimers et al. (1987) noted evidence that a treat-_
ment's acceptability can be modified by improving the con-
sultee's understanding of the treatment. Kazdin (1980)
showed that the acceptability of a reductive technique
(isolation) could be increased through providing variations
of the same technique (i.e., isolation and contracting, and
withdrawal of attention with isolation as a backup). Walle,
Hobbs, and Caldwell (1984) found that parents who tried
various treatments in an experimental setting viewed them as
more acceptable than parents who only received a description
of them. They suggested that the positive ratings may have
been influenced by the understanding the parents received
from having attempted the techniques. Singh and Katz (1985)
demonstrated that providing instruction on treatments in-
creased raters' acceptability of all the treatments. Based

on the available evidence, Reimers et al. (1987) concluded

29
that the consultee's understanding of a technique may impact
acceptability.

Another factor identified by Reimers et al. (1987) was
that of the relative cost of the treatment. While they
noted that there was no empirical evidence to date to sup-
port their hypothesis, the authors suggested that if a
treatment is perceived as costly it may also be seen as
unacceptable. They noted this possibility would be espe-
cially likely in a situation where two or more treatment
options were available.

Another possible factor identified by Reimers et al.
(1987) was that of treatment integrity. Although once again
there appeared to be little empirical support, the authors
suggested that acceptability and integrity may interact.
They hypothesized that treatments perceived as less accept-
able by consultees may be implemented with less integrity.
This could be especially problematic with those treatments
that require strong integrity.

The research on treatment acceptability has yielded
several factors for consideration in investigating how
acceptable a treatment is likely to be perceived. These
factors included: effectiveness; characteristic of the
treatment (i.e., positive vs. reductive); cost-effective-
ness; prevalence of side-effects; time; understanding of the
treatment; problem severity; and treatment integrity. There
was evidence from the research on CBA that suggested some of

these same factors are pertinent to assessment as well.

30
Therefore, it seemed reasonable to consider what implica-
tions the research on treatment acceptability may have for
the assessment of the acceptability of CBA.

There appeared to be considerations from both concep-
tual and empirical research for assessing the acceptability
of CBA. However, the majority of CBA research might be
better described as experiential in nature. That is, what
observations have arisen from researchers and practitioners
who have instituted and studied CBA techniques? Both propo-
nents and critics have been vocal. To what extent do these
ideas affect the perception of CBA among practitioners?

This was considered in the following pages.

W

Inherent in considering the proposed advantages of CBA
was the consideration of the cited alleged shortcomings of
nationally standardized, norm-referenced assessment proce-
dures. Mehrens and Clarizio (1993) offered the following
list of suspected deficiencies that have appeared in support
of CBA: The neglect of the treatment utility of assessment
(Hayes, Nelson, & Jarrett, 1987); the mismatch between test
and curricula; costs in terms of time, material, and spe-
cialized personnel; insensitivity to small treatment gains
in a child's curriculum; the technical inadequacy of
standardized tests used to diagnose handicapping conditions
(Ysseldyke & Thurlow, 1984); localizing the problem within

the child; and cultural bias. The authors offered the list

31
not as empirically proven deficits, but rather as an example
of popular beliefs among the proponents of CBA cited for the
necessity of considering their alternative to assessment.

In addition to the suggested deficits associated with
standardized, norm-referenced tests, CBA proponents have
offered a number of proposed advantages associated with the
use of this alternative assessment procedure.

Tucker (1985) pointed out two main functions of as-
sessment--identification of handicapping conditions for
purposes of placement and eligibility, and gathering data
for instructional planning for the individual being
assessed. He argued that traditional assessments are heavi—
ly loaded on the identification and placement objective to
the neglect of the instructional planning objective. Tucker
(1985) also suggested that traditional assessment is over-
used (he does allow that it is, at times, appropriate) and
that traditional assessment has resulted in teachers ne-
glecting "good practices" in favor of referral to special
education. Finally, he argued that CBA provides a useful
screening instrument that might be utilized to circumvent
"unnecessary" special education referrals.

Galagan (1985), an attorney, has suggested that the
use of standardized tests may in fact be illegal. His
viewpoint was one of the strongest examples of basing CBA
upon the inadequacy of the current practice. His criticisms
of norm-referenced standardized achievement and intelligence

tests (I refer to them collectively as NRT's for

32
simplicity's sake with the understanding that norm referenc-
ing and standardization are actually characteristics of this
group of assessment devices), included: cultural and social
bias, unreliable and unvalidated for determining character—
istics of handicapping conditions, convenient means of
classifying and excluding children, and little use for
educational planning.

Galagan (1985) proposed that the deficiencies of NRT's
are sufficient enough to outlaw their use. He argued that
alternatively, CBA was inherently meeting the letter of the
law as set forth in the Education for All Handicapped Chil-
dren Act (EHA). He also argued that referred students are
placed students, citing the findings of Ysseldyke (Yssel-
dyke, 1983 cited in Galagan, 1985) that a high percentage of
students referred for special education is placed in special
education. Galagan (1985) suggested that NRT's are biased
and that CBA can discriminate between groups without bias by
utilizing the actual curriculum of the students.

Deno's (1985) criticisms appeared similar to those
cited generally by CBA proponents, such as previously de-
scribed by Tucker (1985). For example, he also cited bias
regarding curriculum content, inadequacy for making deci-
sions about individual students, and poor contribution to
educational planning in reference to standardized achieve-
ment tests. He is also a proponent of frequent assessment
of student performance outcomes via the curriculum. In

addition, Deno (1985) cited evidence that there is a very

33
poor match between actual student performance and teacher
judgement of student performance. That is, there is evi-
dence that teachers tend not to rely on formally developed
standardized tests nor on carefully constructed teacher
measures; rather, they rely on informal assessments of
questionable validity. Actually a case could be made for
support of both measures based on this evidence, and cer-
tainly models of CBA such as that proposed by Blankenship
(1985) would be affected by this tendency. In any case, it
is this gap where Deno (1985) believed CBM was most usefully
applied. Another of the advantages of CBM over NRT cited by.
Deno (1985) was that of improved communication. CBM, being
based on the curriculum, a medium teachers are quite famil—
iar with, leads to ease of communication. In a sense it
would have a certain social desirability or face validity
among teacher populations. It remains to be seen whether or
not it has a similar "validity" among testing practitioners
such as school psychologists. Another cited advantage was
increased sensitivity to small gains due to frequent test-
ing, gains that would be missed with traditional assessment
procedures. Deno (1985) is an advocate of norm-referenced
testing for special education decision making. However, he
argued for the ease of constructing local norms based on the
simplicity of the CBM technique. Deno (1985) suggested that
this classroom normative data may be more useful for the
classroom teacher than the more extensive norms produced by

NRT's. Proponents of CBA have also suggested that using the

34
local curriculum provides a better match between instruction
and assessment (Tucker, 1985). Another argument for the use
of CBM has been cost-effectiveness (Deno, 1985) in that
curriculum materials are already prevalent and have been
paid for. Proponents of CBA have typically noted their
technique to provide high curricular validity, having been
based upon the actual curriculum.

Mehrens and Clarizio (1993) noted that several charac-
teristics have been claimed by the CBM technique. These in—
cluded: technical or psychometric adequacy as suggested
through studies of reliability and validity; clear and
effective communication of judgements regarding student
progress; sensitivity to small but important gains in stu-
dent performance over relatively short periods of time
(i.e., days or weeks); frequent and continuous assessment of
achievement without practice effects; simplicity in adminis-
tration which facilitates development of local norms; cost
effectiveness with respect to time required to administer
the test and cost of materials; collection of data essential
to special education decision making with respect to screen-
ing, eligibility, and program evaluation; and superiority
over traditional assessment methods in terms of satisfying
federal requirements for a valid, unbiased, comprehensive
psychoeducational assessment. While the authors concluded
that many of these purported advantages remain to be demon-

strated, they also suggested that some of the advantages

35
cited for CBM may constitute reasons for their acceptance

and use by school personnel.

E .1] E' i :E El
The possibility that the advantages cited by propo-
nents of CBM may influence the acceptability of CBM has been
suggested by Mehrens and Clarizio (1993). Conversely, they
also suggested that there may be reasons for believing that
resistance by school personnel will hinder the effectiveness
of CBM techniques. Possibly foremost among the reasons for
this resistance is the lack of consensus regarding the
claimed advantages of CBM over standardized, norm-referenced.
tests (Mehrens & Clarizio, 1993). Other possible disadvan-
tages have also been identified, including: lack of time to
develop appropriate tests and system-wide norms, inexperi-
ence and lack of expertise in using CBM procedures, in—
creased threat of accountability, limitation of the tech-
nique to the elementary school level, possibility of teach-
ing to the test, and disagreement with a situation-centered
definition of handicapping conditions (Mehrens & Clarizio,
1993). Additionally, there has been concern expressed
regarding the degree of standardization necessary to reli-
ably administer CBM techniques (Derr & Shapiro, 1989).
Another area of concern has been whether or not CBM tech-
niques alone yielded enough information for making program
decisions during the course of instruction (Frank & Gerken,

1990). Taylor, Willits, and Richards (1988) also expressed

36

reservation regarding areas of concern associated with CBA.
These included: assumptions regarding the content of the CBA
instrument, appropriateness of CBA as the only information
source for individualized education programs, and recogni—
tion of CBA's limitations as a procedure/model under the
current education system. These possible limitations may
prove equally influential on the perceived acceptability of
CBM as the possible advantages.

The final source of research for this study of CBA

acceptability was the current research on the topic itself.

E 1.]. E 2 . J -E i E
As has been noted, there is a paucity of research on
the acceptability of CBA despite a rich literature on CBA
and treatment acceptability. In fact, there are no pub-
lished articles to date on the topic of CBA acceptability
(E. S. Shapiro, personal communication, March 10, 1993).
Shapiro and Eckert (1993) recently completed a study
on the acceptability of CBA among school psychologists. The
authors utilized a national sample of school psychologists
who were members of the National Association of School
Psychologists (NASP). A hypothetical case description of a
10-year-old child with "academic difficulties" was mailed to
500 participants in the study. Equal numbers of two treat-
ment conditions were mailed with the description. In one
condition the psychologist used a typical battery of assess-

ment instruments including the Wechsler Intelligence Scale

37
for Children-Revised and the Peabody Individual Achievement
Test. The other condition consisted of a description of the
psychologist "directly assessing the student's skills via
probes taken directly from the child's reading and math
curriculum" (p. 10). Of the surveys returned, 249 were
usable, resulting in a return rate of 49%.

Shapiro and Eckert (1993) analyzed the acceptability
of the two techniques via the respondents ratings of the
techniques on the Assessment Rating Profile (ARP). The
authors reported that both sets of assessment measures were
found to be rated as acceptable, but that the CBA technique
was found to be rated significantly and consistently as more
acceptable than the standardized test condition.

The Shapiro and Eckert (1993) study represented an
important initial investigation into the acceptability of
CBA. However, limitations of the study and unexplored areas
of investigation suggested the need for further CBA accept-
ability research.

The Shapiro and Eckert (1993) study may represent a
somewhat biased sample of school psychologists in that only
members of NASP were sampled. All school psychologists are
not NASP members and there may be non-random differences
between those who are and those who are not. The authors
also noted the return rate was slightly below 50%. Citing
"rules of thumb" for return rates for survey methods, Babbie
(1989) noted that a very good response rate is 70%, a good

response rate is 60% and 50% would be considered adequate.

38
Although the 49% return rate may be very close to adequate
and the authors received very nearly equal response rates
for both conditions, it is conceivable that some response
bias existed. The authors also noted that the use of a
within-subjects design might have resulted in different
findings. Additionally, they pointed out that the choice of
tests described in the hypothetical case may have affected
practitioners' ratings of acceptability. The use of an
analogue design, although consistent with the treatment
acceptability studies, also was noted as a limitation. The
authors suggested that although CBA may be reported as more.
acceptable, it may not actually be implemented. In fact,
they found in a previous study (Shapiro & Eckert, in press)
that although 45% of the respondents to their survey report-
ed using some form of CBA in their practice, more specific
questions designed to directly assess use of CBA found that
only 18% used CBA on a regular basis.

The Shapiro and Eckert (1993) study also left impor-
tant areas of investigation unanswered. The degree of
knowledge, experience, and familiarity of practitioners with
CBA was not considered in the study. The authors made no
attempt to analyze the relationship of many of the factors
identified from the consideration of treatment acceptability
and CBA literature with CBA acceptability. Although the
Shapiro and Eckert (1993) study represented an important
initial investigation into CBA acceptability, the limita-

tions of the study, areas left uninvestigated and even the

39
authors themselves, suggested the need for continued re-

search.

WWI-WI i EZEEE 1.].

The discussion of possible factors influencing the
acceptability of CBA has included implications from the
three assessment models that have been put forth as
providing the conceptual basis, the empirical findings from
the research on treatment acceptability, the experiential
suggestions from CBA literature, and the fledgling CBA
acceptability research.

The consideration of the conceptual basis of CBM
(Knutson & Shinn, 1991) suggested that the principles of
behavioral assessment, the ecological model, and the data-
based problem solving model should be important in determin—
ing how an individual perceives CBA. That is, to the extent
that a practitioner perceives as acceptable any or all of
these models, or any or all of the principles put forth by
these models, the practitioner should also perceive CBA as
acceptable.

In considering the implications of the findings from
the empirically based treatment acceptability research, an
important consideration was the extent to which the factors
identified in that body of research could be directly ap-
plied to CBA acceptability. The suggestion that the factors
from the treatment acceptability literature may be directly

applied to the acceptability of CBA may be questioned.

40
Although CBA may not be perceived as a treatment by some, it
is in fact regarded as an assessment procedure that can be
used to develop and modify instructional programs (Fuchs,
Fuchs, & Hamlett, 1990). Also, although there appears to be
only limited research on the direct evaluation of the ac—
ceptability of assessment techniques, it would seem reason-
able that many of the same factors that influence practi-
tioners' and consultees' perceptions of acceptability of
treatments would also apply to their perceptions of assess-
ment acceptability. In addition, the research on the con-
ceptual basis of CBM provided a strong argument for concep-_
tualizing CBA as an assessment technique that is directly
linked to treatment. Fundamental to the principles of
behavioral assessment (Jenkins et al., 1979) was the use of
the assessment to modify treatment. In fact, the assessment
may determine when a treatment is considered effective or
when it is to be discontinued. The ecological model (Lentz
& Shapiro, 1986) also strongly advocated the linking of
assessment with treatment, noting that successful treatment
is dependent upon accurately assessing the individual's
performance, the academic environment, home environment,
community peers, community, etc. In addition, the interac-
tion of these variables must also be considered. Finally,
treatment will likely involve some modification of the
environment as well as remediating or providing instruction
to the student in order to overcome areas of deficit or

mismatch with the environment. The identification of

41
treatment goals is dependent upon the assessment. It seemed
reasonable, then, to suspect that many of the factors that
affect treatment acceptability might also affect the accept-
ability of CBA.

Specifically, there appear to be a number of factors
that have been cited as important issues in the CBA litera-
ture that have also been cited as factors in the treatment
acceptability literature. The inclusion of factors common
to these bodies of literature in the present study appeared
warranted.

Perhaps foremost among these "commonalities" was the
concept of effectiveness. Proponents of CBA techniques have
suggested that CBA is effective as an assessment technique
(in some cases more effective than traditional tests, e.g.,
Galagan, 1985), and a technique that is effective for in-
structional planning (Blankenship, 1985; Fuchs, Fuchs, &
Hamlett, 1990). The assessment models also call for proper
assessment (measures of socially valid behavior) in order to
have effective treatments (Knutson & Shinn, 1991). The
literature on treatment acceptability clearly pointed to a
relationship between a treatment's acceptability and its
effectiveness, although the direction of that relationship
was not exactly clear (Reimers et al., 1987; Miltenberger,
1990). It seemed reasonable to expect that in order for CBA
to be viewed as an acceptable technique, it must also be

viewed as an effective technique.

42

Deno (1985) has suggested that cost-effectiveness is
an advantage of CBA over traditional techniques. As Reimers
et al. (1987) have proposed, cost may affect treatment
acceptability, especially when two or more options are
available. With CBA and nationally standardized and norm-
referenced tests (NST's and NRT's) available to the practi-
tioner, this point also seemed pertinent to the issue of
assessment acceptability.

Reimers et al. (1987) suggested that the existence of
side-effects may negatively affect a treatment's acceptabil-
ity. Some proponents of CBA have suggested that CBA may be»
less stigmatizing than NRT's (Galagan, 1985). If stigmati—
zation of students could be viewed as a side-effect of
assessment, it seemed reasonable to consider the possible
effect of this factor on acceptability.

Time has also been a factor mentioned in the conceptu-
al and experiential discussions on CBA and the treatment
acceptability research (Deno, 1985; Knutson & Shinn, 1991,
Reimers et al., 1987). To the extent that practitioners
view CBA as more or less time-consuming, one might also
expect to see their perceptions of its acceptability vary.

Understanding of the treatment, both in terms of
knowledge (Singh & Katz, 1985) and experience (Walle et al.,
1984), has been noted to be an important factor in treatment
acceptability as well (Reimers et al., 1987). Although no

direct consideration of understanding is apparent in the CBA

43
literature, it seemed possible that understanding of CBA may
also affect its acceptability.

Another possible link is the area of problem severity.
The research on treatment acceptability has demonstrated
that the severity of the problem may interact with the
acceptability of the intervention. Similarly, one might
expect that the difficulty or severity of the assessment may
differentially affect the acceptability of the treatment
technique. While advocates of CBA have not limited the use
of their technique to any particular range of severity, it
was conceivable that practitioners may perceive that CBA is_
more or less effective dependent upon the severity of the
suspected handicap.

In contemplating the implications from the experien-
tial research, the statement from Mehrens and Clarizio
(1993) was worth considering. While these authors concluded
that many of the purported advantages remained to be demon-
strated, they also suggested that some of the advantages
cited for CBM may constitute reasons for their acceptance
and use by school personnel. Conversely, they also suggest-
ed that there may be reasons for believing that resistance
by school personnel will hinder the effectiveness of CBM
techniques. It was important to include these factors in
the investigation of CBA acceptability. By including these
factors, it may become more evident whether practitioners
perceive the advantages or disadvantages cited in the expe—

riential literature to more accurately reflect reality.

44

Another area of research that suggested the need for
further investigation of CBA acceptability was that of the
fledgling research on CBA acceptability. The work of
Shapiro and Eckert (1993) represented an important initial
investigation into CBA acceptability. However, the limita-
tions of that study, the research questions left uninvesti-
gated, and the authors themselves, all called for further
investigation of CBA acceptability.

In conclusion, the consideration of these various
findings and their interrelated implications provided the
rationale for the investigation into the acceptability of
CBA. There has been very little research to date assessing
the acceptability of CBA. In proposing such an investiga-
tion the foremost consideration may be whether such an
effort is worthwhile. That such an investigation might be
worthwhile should be apparent from the previous paragraphs.
As Mehrens and Clarizio (1993) have noted, acceptability is
essential for practitioner usage and adherence to proper
technique. Reimers et al., (1987) also pointed out that
unless an intervention is acceptable it may not be imple-
mented with integrity. As integrity is an important issue
with CBA (Derr & Shapiro, 1989), the assessment of CBA
acceptability among practitioners appeared to be a worth-

while endeavor.

METHODOLOGY

The focus of the first part of this study was the
conceptual bases suggested by Knutson and Shinn (1991). It
seemed reasonable to inquire as to what extent the conceptu-
al bases suggested as the underpinnings of CBM by these
authors are also perceived as underlying CBA by practitio-
ners. In order to assess the extent to which practitioners
agree that the proposed assessment models have strong con-
ceptual ties to CBA, practitioners were asked to check
whether or not an assessment model had ties to CBA. In
addition, the tenets of the three models suggested by Knut-
son and Shinn (1991) were included in the questionnaire to
assess the degree to which practitioners agreed that the
characteristics representing the assessment models were also
characteristic of CBA.

The investigation of practitioners' perceptions of CBA
acceptability had little direct research upon which to draw.
As a result, the research questions utilized to investigate
this area were, by necessity, derived from other bodies of
research. For the purposes of the present investigation,
consideration of two sources provided the material for

research questions.

45

46

One source for potential research questions was that
of the empirically based treatment acceptability research.
Although this body of research has a relatively short histo—
ry, a number of factors that influence the acceptability of
behavioral treatments have already been consistently identi-
fied. The question of the degree to which the acceptability
research on behavioral treatments generalized to CBA accept-
ability was a reasonable one. Yet the overlap between this
research and the third source suggested there was potential
for generalization.

The literature from authorities on CBA identified
factors important to CBA (e.g., Mehrens & Clarizio, 1993;
Deno, 1985; Tucker, 1985) that have also been identified in
the treatment acceptability research. This area of overlap
suggested some common factors worth including in the study
of CBA acceptability. There were also factors identified as
potential advantages and disadvantages of CBA by authorities
on CBA that have not been identified in the treatment ac-
ceptability research._ A number of these proposed advantages
and disadvantages of CBA have been discussed in the preced-
ing sections. The extent to which the claims of both propo-
nents and opponents of CBA are perceived to be true was
investigated.

The remainder of the research questions included demo-
graphic items about the characteristics of the respondents
that might have been related to their perceptions of CBA

acceptability.

47

II 'E' I' E E l : .

W

The conceptual bases suggested a number of components
that are consistent with the three conceptual models pro—
posed by Knutson and Shinn (1991). Their input was selected
for inclusion in the present study due to the article's
recency, explicitness, and comprehensiveness in identifying
conceptual models that supposedly have directly influenced
CBM. This may be an arguable point. As mentioned in the
section on CBA models, considerable confusion exists between
what characteristics are necessary for an assessment model‘
to be considered CBA. However, Shinn, Rosenfield, and
Knutson (1989) mentioned four common principles of CBA, in-
cluding: (a) student assessment in classroom instructional
materials, (b) short-duration testing, (c) frequent and
repeated measurement, and (d) graphed data to allow for
monitoring of student progress. As pointed out in that
article, CBM may be considered an example of a particular
CBA model. CBM shared these common characteristics with the
other popular CBA models, according to Shinn et al. (1989).
While Knutson and Shinn (1991) chose to focus on CBM in
discussing the conceptual underpinnings of this assessment
technique, it seemed clear that CBM is an example of the
more generic technique of CBA. While some variations in CBA
models are noted (Shinn et al., 1989; see Table 1), it also

seemed clear that the various models have much in common,

48
including assessment in classroom instructional materials,
short-duration testing, frequent and repeated measurement,
and graphed data to allow for monitoring of student prog-
ress. These same factors also seemed quite consistent with
the factors identified by Knutson and Shinn (1991) as being
components of the behavioral, ecological, and problem-solv-
ing assessment models. It seemed reasonable to expect that
if these three assessment models do provide the conceptual
bases for CBM, they should also contribute to conceptual
bases of the more generic model of CBA, of which CBM is a
specific example. This expectation was based on the fact
that CBA and CBM have much in common such as using the
curriculum to assess students, use of repeated measurements,
short duration assessment, and graphing or plotting results.
As CBM grew out of CBA techniques, it is possible that the
assessment models that supposedly inspired CBM might have
first inspired CBA. Yet, it was not clear to what extent
practitioners perceived CBA as sharing a conceptual back-
ground with CBM based upon the behavioral, ecological, and
problem-solving models. The purpose of including research
questions based upon the three assessment models identified
by Knutson and Shinn (1991) in the present study was to
investigate this possibility. Therefore, a number of prin-
ciples were identified that seemed to constitute the major
tenets of the three models proposed by Knutson and Shinn
(1991). A summary of these is provided in Table 2. It is

uncertain as to whether the three models constitute clearly

49

Table 2. Tenets from Assessment Models

 

EHAVIORAL ASSESSMENT MODEL:

 

Assessment measures socially valid behaviors.

 

 

B

1.
2. Measures are sensitive to small improvements.
3.

During the assessment repeated measurements are
conducted.

 

client's behavior to document improvement\treatment
effectiveness.

 

.4. The assessment system must use quantified measures of
IIECOLOGICAL ASSESSMENT MODEL:

1. Remediation of academic deficits is the purpose of
psychological services. This is possible only if the
assessment provides information that promotes meaningful
outcomes.

2. Poor performance may have multiple causes: the stu-
dent's performance, classroom environment, or an interac—
tion between the two.

3. The remediation of academic problems requires direct,
effective instruction of deficit skills and/or manipula-
tion of variables that are contributing to the poor
performance.

IIDATA BASED PROBLEM-SOLVING MODEL:

 

1. A problem exists when there is a discrepancy between
the student's actual performance and that which is ex-
pected.

2. General education fails to educate all students:
problems are situational in nature.

 
 

i 3. Practitioners must evaluate the effects of interven-
‘ tions and generate multiple interventions.

    

50
distinctive assessment approaches. However, research ques—
tions were chosen to reflect the distinctive characteristics
of the assessment models in order to delineate between them
to the extent possible.

The tenets of the behavioral assessment models as
identified by the present study were as follows: the neces—
sity of measuring socially valid behavior, the sensitivity
of measures to small gains, the importance of repeated
measurements, and the use of quantified measures of the
student's behavior to document improvement\treatment effec-
tiveness. The factors included from the ecological model
were the need for the assessment to: provide information
that promotes meaningful outcomes, encourage the direct
instruction of deficit skills and/or manipulation of vari-
ables contributing to the student's poor performance, and
consider the situational or ecological factors. From the
problem-solving model the factors of generating multiple
interventions from assessment and definition of a problem as
a discrepancy between performance and expectation were
included as representative of this particular approach.

3 E E I E l'J'l 1 2E!
Literature

The overlap among factors identified as affecting
treatment acceptability and the proposed advantages and
disadvantages of CBA provided a number of research questions
for the present study. The research on the acceptability of

behavioral treatments has led to the identification of a

51
number of factors that affect the perceived acceptability of
treatments. As noted previously, some have argued that CBM
techniques were derived from a behavioral perspective (Deno,
Mirkin, & Shinn, 1979; Knutson & Shinn, 1991). I have also
suggested that the behavioral model (in addition to the
ecological and problem-solving models) may be perceived as
providing a conceptual basis for the majority, if not all,
of CBA models. If CBA may be perceived of as a behaviorally
based assessment model, is there reason to believe that the
same factors important to the acceptability of behavioral
treatments also affect the acceptability of assessment
models? Given that CBA links assessment to treatment, there
is reason to believe that the treatment acceptability fac-
tors generalize to CBA acceptability as well. Although CBA
is an assessment technique, it is said to be directly linked
to instruction due to the fact that the material for assess-
ment is the student's current curriculum. That is, both
assessment and decision making are said to be "curriculum
referenced" (Marston, 1989). As noted by Shinn et a1.
(1989), CBA provides for frequent assessment and graphing or
monitoring of students' progress. When desired improvements
are not noted, changes in the classroom instruction are
suggested (Blankenship, 1985; Fuchs, Fuchs, & Hamlett,
1990). Whether or not the assessment yields prescriptive
information for instruction appears to depend upon the
particular model. For some models, assessment intentionally

leads to direct changes in instructional planning (e.g.,

52
Gickling & Thompson, 1985; Blankenship, 1985; Howell, 1986).
For others, assessment is designed more to monitor student
progress and indicate when there may be a problem versus
being prescriptive in nature (Deno, 1985). There is also
some evidence from recent research suggesting factors that
affect treatment acceptability also affect CBA acceptability
(Shapiro & Eckert, 1993).

Although the specific impact of the assessment on
instruction may vary, it appears that CBA is directly linked
to instruction, the educational equivalent of treatment. It
does not seem unreasonable, then, to suspect that the same
factors affecting treatment acceptability of behavioral
interventions also affect CBA acceptability, given its
behavioral influence and link to instruction.

All of the factors that have already been identified
as affecting the acceptability of treatments (see Table 3)
have also been identified as important themes or factors in
the CBA literature (for a more comprehensive list of factors
identified from the CBA literature see Table 4). These
"common" factors would appear to have potential for affect-
ing acceptability of CBA as well, and all of them have been

included in the present study.

53

Table 3. Common Factors from Treatment
Acceptability and CBA Research

 

1. Effectiveness of the treatment impacts its accept-
ability.

 

2. Whether the treatment is positive or negative in
nature affects its acceptability.

 

3. Problem severity affects the acceptability of the
treatment.

 

4. The amount of time necessary to implement the treat-
ment affects its acceptability.

 

5. The amount and/or severity of side-effects from the
treatment affects its acceptability.

 

6. The level of understanding of those using or impacted
by the treatment affects acceptability.

 

7. Treatments perceived as more acceptable may be imple-
mented with more integrity and those implemented with
more integrity may be perceived as more acceptable.

 

8. The relative cost of the treatment affects accept-
ability.

 

 

Ir’ 1
W

In addition to the factors common to both the empiri-
cal treatment acceptability research and the CBA literature,
a number of research questions based only on the CBA litera-
ture were included. As noted in the review of the litera-
ture pertaining to CBA, there continues to be some contro-
versy surrounding the purported advantages and disadvantages
of CBA. It was beyond the scope of the present research to
attempt to prove or disprove these claims. Rather, it was
felt that the knowledge of practitioners' perceptions of
these claims should be assessed. It is possible that the
belief of a proposed advantage or disadvantage may influence

the use of CBA. The present study attempted to assess the

54

Table 4. Purported Advantages and Possible
Disadvantages of CBA

 

PURPORTED ADVANTAGES OP CBA i

1. CBA is a useful screening device for determining if there
is a need for additional evaluation and\or services.

education assessments.

 

3. CBA is less likely to be culturally biased.

 

fl 2. CBA is useful for meeting the federal requirements for special fl

4. CBA allows for frequent assessment.

 

5. CBA provides more instructionally useful data than traditional
assessment.

 

6. CBA allows for better, more meaningful communication with teach-
ers.

 

 

7. CBA is sensitive to small treatment gains.

CBA allows for easily constructed local norms.

8
9. CBA is cost-effective.

 

10. CBA has high curricular validity.
11. CBA has demonstrated psychometric adequacy.

12. CBA provides for clear and effective communication of judgements
and decisions.

POSSIBLE DISADVANTAGES OF CBA

 

1. There is a lack of time to develop local norms for a school or
district using CBA.

 

EL=1=L==L==

2. Practitioners lack experience and expertise in the use of CBA
techniques.

 

3. Many practitioners and school personnel express non-acceptance of
a situational centered definition of handicapped.

 

 

LE

  
 
  
  
   
  
    

4 CBA does not provide adequate data for accountability purposes.

 

5. CBA requires significant standardization that might be neglected.

 

content .

 

fl 6. CBA relies upon invalid assumptions with regard to curriculum

7. CBA is not very applicable at the secondary school level.

 

8. Teaching to the test would be more easily accomplished with CBA
‘ than with traditional assessment.

 

I . CBA is an inadequate assessment technique used in isolation.

   

55
degree to which practitioners endorse some of the claims of
CBA reported in the literature. Table 4 provides a list of
advantages and disadvantages identified by the author from
the review of CBA literature. For the purposes of identify-
ing research questions, attention was given to those claims
that have been listed as both advantages and disadvantages
of CBA and those featured most prominently in the research.

The purported advantages and disadvantages included in
one form or another in the present study were that: (1) CBA
is perceived as meeting the requirements of special educa-
tion laws, (2) CBA may have less potential for stigmatizing
labels, (3) CBA may have less potential for cultural bias,‘
(4) CBA allows for more frequent assessment, (5) CBA may
provide more instructionally useful data, (6) CBA is sensi—
tive to small gains, (7) CBA allows for easily constructed
local norms, (8) CBA has demonstrated psychometric adequacy
(i.e., content validity), (9) users (i.e., practitioners)
may lack the experience or (10) knowledge necessary for use
of CBA, (11) CBA may not be applicable at the secondary
school level, (12) teaching to the test may be more readily
achieved with CBA than with traditional assessment, and (13)
the need for standardization of CBA may be neglected.

These claims were chosen based upon the rationale that
they reflected some of those most often cited in the CBA
literature. They also tended to be somewhat controversial.
Mehrens and Clarizio (1993) noted that many of the positive

claims have not been substantiated in any meaningful way to

56
date. It was important to assess whether these advantages
are perceived as such by the practitioners who may be likely
to employ them.

Based upon the preceding considerations of the litera-
ture and research dealing with CBA and treatment acceptabil-
ity, three broad questions were asked: (1) To what extent do
practitioners agree with authorities that CBA has strong
conceptual ties to the three conceptual models, and to what
extent do they agree with authorities that CBA demonstrates
the characteristics of these assessment models?, (2) Do the
factors identified from the treatment acceptability research
affect the acceptability of CBA?, and (3) To what extent do
practitioners agree with the proposed advantages and disad-

vantages of CBA?

WW

Given that the only current measurement instrument for
assessing the acceptability of CBA was based upon an ana-
logue technique (Shapiro & Eckert, 1993) and the present
research utilized a direct survey approach of practitioners'
perceptions, a new instrument was devised. Although the
present CBA acceptability measure was loosely modeled after
those instruments currently utilized to measure treatment
acceptability, it was devised to specifically include those
factors identified as common to both CBA and the treatment
acceptability research previously discussed. Table 5 pro-

vides an overview of each item, its intended measurement

[ _

57

 

Table 5. Questionnaire: Items, Purpose and Source
Questionnaire Itum Intended Purpose Source
1. CBA is an accept- Intended to measure Treatment

able assessment tech-
nique

overall perceived ac-
ceptability of CBA

Acceptability re-
search, Conceptual
research

 

2. I have used CBA
regularly.

Intended to measure
degree of understanding
of CBA techniques by
looking at use

Treatment
Acceptability re-
search

 

L 3. CBA does not rely
: on national norms for
‘ comparisons of exam-

Also intended to mea-
sure degree of under-
standing of CBA tech-

Treatment
Acceptability re-
search and CBA

 

technique

degree to which the
respondent perceives
CBA to be an effective
assessment technique

‘ inees niques, but in a less research
' obvious fashion
1 4. CBA is an effective Intended to measure the Treatment

Acceptability re-
search

 

S. CBA is less apt to

use stigmatizing 1a-
bels than traditional
assessment

Intended to measure if
CBA is perceived to
result in fewer "side-
effects"

Treatment
Acceptability re-
search

 

6. CBA is less cost-

. effective than tradi-
. tional assessment

Intended to measure if
CBA is perceived to be
more or less costly
than traditional as-
sessment

Treatment
Acceptability re-
search CBA re-
search

 

E 7. CBA is less likely

to be culturally bi-

: ased

Intended to measure
perception of CBA as
less prone to bias, in
a straightforward man—
ner

CBA research

 

 

; 8. CBA is not effec-

' tive for assessing

1 children with both

1 mild and severe handi-
lcaps

Intended to measure
impact of problem se-

‘verity on perceived

acceptability

Treatment
Acceptability re-
search and CBA
research

 

 

9. CBA provides more
socially valid
measures of behavior
than traditional as-
sessments

Intended to assess per-
ception of CBA as mea-
suring valid behavior

Conceptual research

 

10. CBA takes more
time than traditional
assessment techniques

 

Intended to assess how
the factor of time ef-
fects acceptability

 

Treatment
Acceptability re-
search, Conceptual
research and CBA
reseangh

 

 

I Questionnaire Item

58

Table 5 (cont'd)

Intended Purpose

 

Source

 

. 11. CBA emphasizes a

more positive approach
to assessment and re-

 

Intended to measure the
degree to which CBA is
seen as a positive (as

Treatment Accept-
ability research

 

‘ mediation opposed to negative or
reductive) technique
5 12. Standardized ad- Intended to measure the CBA research

ministration is not
essential for CBA as-

' sessment techniques.

attitude of the respon-
dent with regard to the
necessity for standard-
ization of CBA tech-
niques

 

’ 13. CBA has higher
; content validity than

traditional assessment

Intended to measure
perception of CBA as
having higher content

CBA research

 

F techniques. validity than CBA in a
. straightforward manner
14. It is easier to Intended to measure the CBA research

' construct local norms
. with CBA than with
f traditional tests.

respondents attitude
regarding ease of con-
structing local norms
with CBA

 

3 15. Frequent testing

is necessary for as-

? sessment .

Intended to measure the
respondents attitude
regarding need for fre-
quent testing

CBA research and
Conceptual research

 

a 16. CBA is more likely

to meet the re-
quirements of assess-
ment under special
education law

Intended to measure the
respondents perception

of the degree to which

CBA meets special edu—

cation requirements

CBA research

 

17. CBA provides data
that allows for direct
instruction of deficit
skills

Intended to measure the
respondents perception
of the degree to which
CBA is more instruc-
tionally useful than

'traditional techniques

CBA research and
Conceptual research

 

18. CBA is not as sen-
sitive to small im-
provements or gains as
traditional assessment
techniques

Intended to measure the
degree to which CBA is
seen as sensitive to
small gains

Conceptual research
and CBA research

 

 

19. CBA is more likely
to consider the situa-
tional or environmen-
tal factors than is

traditional assessment
E

 

Intended to measure the
degree to which CBA is
seen as consistent with
an ecological assess-
ment model

 

Conceptual research

 

Questionnaire Item

59

Table 5 (cont'd)

Intended Purpose

 

Source

 

20. CBA is less likely
to generate multiple
interventions than tra-
ditional assessment

Intended to measure
the degree to which
CBA is seen as being
consistent with a pro-
blem-solving model

Conceptual research

 

)
)u

21. CBA uses quantified
measures of the stu-
dent's behavior to doc-
ument improvement and
intervention effective-
ness

Intended to measure
the extent to which
practitioners perceive
CBA to use quantified
measures of student
behavior to assess
effectiveness

  

Conceptual research

 

22. CBA defines a
problem as a discrep-
ancy between the stu-

i dent's performance and
; what is expected

Intended to measure
the perception of CBA
as utilizing a prob—
lem-solving model def-
inition of a problem

  

Conceptual research

 

23.

I would consider
myself knowledgeable

: with regards to CBA

techniques

Intended to measure
degree of perceived
knowledge of CBA inde-
pendent of actual use

Treatment Accept~
ability research

 

 

24.

CBA provides infor-
mation that promotes

. meaningful outcomes

Intended to measure
perception of CBA as a
technique that pro-
vides information that
leads to meaningful
outcomes

Conceptual research

 

 

g 25. CBA is more appli-
. cable to elementary

. school students than

. secondary school stu-

dents

Intended to measure
perception that CBA
may be more applicable
for elementary school
students than second-
ary school students

CBA research

 

 

3 26. Teaching to the
1 test is more easily

accomplished with CBA

3 than with traditional

 

assessment

 

Intended to measure
the perception that

' CBA lends itself to

teaching to the test
more readily than tra-
ditional assessment

 

CBA research

6O
purpose, and its source from the literature review. In
addition, the instrument allowed for some basic assessment
of demographics in order to assess for possible differential
effects due to level of education, training program, years
of practice, diagnostic caseload, or conceptual perspective.
These items are provided in Table 6. A copy of the instru-
ment is provided in Table 7. The instrument follows the
basic design of the Behavior Intervention Scale (Elliott &
Von Brock Treuting, 1991), an instrument devised to measure
teachers' perceptions of treatment acceptability and per-
ceived effectiveness of classroom interventions. The format
consists of a seven-point Likert-type scale with questions
derived to measure the effect of the factors previously
identified through the literature review and rationale in
the preceding discussion. In addition to the inclusion of
items devised to measure the identified factors, the present
instrument also was designed to assess practitioners' actual
experience and perceptions of CBA rather than employing an
analogue-type design.. It is suggested that even those
practitioners who have not actually used CBA have some level
of familiarity with CBA (even if it is slight) and have
perceptions of CBA that are associated with its acceptabili-
ty. Also, the more generic concept of CBA was chosen for
the purposes of this study rather than using a specific
model of CBA (e.g., CBM, CBA-ID, etc.). While some finer
discriminations may have been lost, it was felt that utiliz-

ing the generic term would result in the most representative

61

Table 6. Demographic Questions

ee Intended 9...... +

      
       
   

:My own theoretical perspec-
'tive could best be described
'as: (1) Behavioral (2) Cogni-
3tive (3) Eclectic (4) Other

Intended to acquire a theo-
retical orientation for
purposes of assessing for
interaction effects between
orientation and responses.

  
  

  
   
 

University attended for de-
gree in school psychology.

Intended to gather demo-
graphic data for further
analysis.

 

Date degree was received.

Intended to gather demo-
graphic data for further
analysis.

 

Highest degree held.

 

Intended to gather demo-
graphic data for further
analysis.

 

Number of years experience as
a teacher.

Intended to gather demo-
graphic data for further
analysis.

 

Approximate number of stu-
dents you expect to assess
this year.

Intended to gather demo-
graphic data for further
analysis.

 

Approximate number of years
in practice as a school psy—
chologist.

Intended to gather demo-
graphic data for further
analysis.

 

Approximate percentage of
time spent as a school psy-
chologist.

 

 

Intended to gather demo-
graphic data for further
analysis.

 

 

62

Table 7. Questionnaire

NOTE: If you have not formed a percepeien on any item, you may indicate this by
omitting that item.

PART 1: Please indicate (by placing an 'X' on the line(s)) which of the
following assessment models you feel have provided a conceptual basis for CBA:
Behavioral Assessment Model

Ecological Assessment Model

Data Based, Problem-solving Model

Please provide the following information:

My own theoretical perspective could best be described as: (1) Behavioral
(2) Cognitive (3) Eclectic (4) Other (please indicate)

 

University attended for degree in School Psychology (if you received school
psychology degrees from more than one school, please indicate degree): Michigan
State University University of Michigan Western Michigan Universi-
ty Central MIcEIgan University Eastern Michigan University Other
(indicat

Date degree in school psychology was received

e)

Highest degree held: B.A.___ M.A.___ Ed.s.___ Doctorate___

Number of years experience as a teacher (if any)

Approximate number of years in practice as a school psychologist
Approximate number of students you expect to assess this school year
Approximate number of students you assessed last school year

What percent of your time is spent as a school psychologist %

Please estimate the percentage of time you spend on the following activi-
ties:(total-100t)

Assessing students for special education eligibility %
Assessing students for other purposes I

Attending Multidisciplinary Team (MET) and Individual Educational Plan (IEP)
meetings

Writing results of assessment and MET reports_____%

Providing consultation services to school staff or parents %

Providing counseling or therapy services to students I

Providing in-service or other educational presentations %

Providing direct interventions or behavior modification in the classroom %
Otheriplease specify) . %

Approximately what year do you plan to retire from your position as a school
psychologist

 

(OVER PLEASE)

63

Table 7 (cont'd)

Please indicate the extent to which you would agree or disagree with the
following statements pertaining to CBA. SCORING CRITERIA:

1. STRONGLY DISAGREE, 2. DISAGREE, 3. SLIGHTLY DISAGREE, 4. NEITHER AGREE NOR
DISAGREE OR DON'T KNOW, 5. SLIGHTLY AGREE. 6. AGREE. 7. STRONGLY AGREE

+

 

1. CBA is an acceptable assessment technique. 1 2 3 4 S 6 7

2. I have used CBA regularly. 1 2 3 4 s 6 7

3. CBA does not rely on national norms for 1 2 3 4 S 6 7
comparisons of examinees.

4. CBA is an effective assessment technique. 1 2 3 4 S 6 7

5. CBA is less apt to use stigmatizing 1 2 3 4 S 6 7
labels than traditional assessment.

6. CBA is less cost-effective than traditional 1 2 3 4 5 6 7

assessment.

7. CBA is less likely to be culturally biased. 1 2 3 4 s 6 7

8. CBA is not effective for assessing children 1 2 3 4 S 6 7
with both mild and severe handicaps.

9. CBA provides more socially valid measures 1 2 3 4 5 6 7
of behavior than traditional techniques.

10. CBA takes more time than traditional 1 2 3 4 S 6 7
techniques.

11. CBA emphasizes a more positive approach to 1 2 3 4 S 6 7
assessment and remediation.

12. Standardized administration is not 1 2 3 4 S 6 7
essential for CBA assessment techniques.

13. CBA has high content validity. 1 2 3 4 S 6 7

14. It is easier to construct local norms with 1 2 3 4 S 6 7
CBA than traditional tests.

15. Frequent testing is necessary for proper 1 2 3 4 5 6 7
assessment.

16. CBA is more likely to meet the requirements 1 2 3 4 S 6 7
of assessment under special education law.

17. CBA provides data that allows for direct 1 2 3 4 S 6 7
instruction of deficit skills.

18. CBA is not as sensitive to small gains 1 2 3 4 S 6 7
or improvements as traditional techniques.

19. CBA is more likely to consider the 1 2 3 4 S 6 7

situational/environmental factors than is
traditional assessment.

20. CBA is less likely to generate multiple 1 2 3 4 S 6 7
interventions than traditional assessment.
21. CBA uses quantified measures of the 1 2 3 4 S 6 7

student's behavior to document improvement
and intervention effectiveness.

22. CBA defines a problem as a discrepancy 1 2 3 4 S 6 7
between the student's performance and what
is expected.

23. I would consider myself knowledgeable with 1 2 3 4 S 6 7
regards to CBA techniques.

24. CBA provides information that leads to 1 2 3 4 5 6 7
meaningful outcomes.

25. CBA is more applicable to elementary school 1 2 3 4 S 6 7
students than secondary school students.

26. Teaching to the test is more easily accom- 1 2 3 4 S 6 7
plished with CBA than traditional assessment.

64
sample of knowledge of and beliefs towards CBA as it is
possible that many practitioners are unfamiliar with the
various CBA models. In fact, other researchers have also
noted what seems to be considerable confusion among practi-
tioners with regard to what constitutes CBA and, as a re-
sult, it may be more prudent to employ a "generic" approach
to CBA acceptability research (E.S. Shapiro, personal commu-

nication, March 10, 1993).

Sample

The study utilized the entire population of certified
school psychologists in the State of Michigan. During the.
1992-93 school year, there were 869 certified school psy-
chologists reported to be working in the State of Michigan
(Michigan Department of Education (MDE), 1992). Question-
naires were sent to all 869 of the school psychologists
utilizing the MDE data-base and official letterhead for the
cover letter. While only 175 returned questionnaires would
be necessary to obtain a significant correlation at the .01
level with obtained cOrrelations of .15 or higher (Borg &
Gall, 1983), the over-sampling offered some protection in
the event that the return rate was low and allowed for
generalizations to the entire state for MDE data-keeping
purposes.

A recent MDE survey of school psychologists obtained a
return rate of approximately 63% (L. Parshall, personal

communication, March 1, 1993). Therefore, a second mailing

65
of questionnaires was deemed to be unnecessary. A postcard
thanking participants for returning the questionnaire or
reminding them to do so if they had not was sent to the
entire sample 10 days following the initial mailing of
questionnaires, to increase the return rate, however.

In order to differentiate between respondents and non—
respondents, the return envelopes were numbered. The num-
bers corresponded to the numbers assigned to the practition-
ers in the data-base. The actual names were removed from
the data base, leaving only the demographic information, in
order to assure anonymity. The questionnaires were separat-
ed from the numbered return letters as they arrived. A .
cover letter was sent with the questionnaires assuring
participants that the numbers on the return envelopes were
used solely to identify demographic characteristics of
respondents versus non-respondents and that it was not
possible for the researcher to match questionnaires with
names. Also, participants were offered the opportunity to
receive a summarization of the research results by including
their mailing address on the cover letter and returning it
with their questionnaire. They were also offered the option
of phoning the researcher or mailing a request for the
summary. Finally, all individual responses were considered
confidential by the researcher. There were no identifi-
cation numbers on the questionnaires. Thus, a method of
identifying the demographic characteristics was possible

while maintaining anonymity. The demographic data available

66
to compare the respondents to non-respondents on demographic
variables included age, education level, certification,
ethnicity, and sex.

The study provided considerable information from
actual practitioners of school psychology regarding an
initial attempt to assess CBA acceptability. The sample
consisted of the entire population of certified school psy-
chologists in the state of Michigan. The information ob-
tained should be representative of practitioners in the
state of Michigan and allow for generalization within the
state with reasonable confidence. Although such a sample
might have restrictions in terms of generalizing to other
states, it is uncertain as to what extent practitioners vary

in practice and perception from state to state.

W

The research on CBA and treatment acceptability does
provide a basis for some directional hypotheses. In other
situations there is simply no evidence upon which to base
predictions and none were forthcoming.

Three major questions were identified. Each of these
is followed by the research questions designed to provide
information to answer the major question. Demographic
questions were also included in the questionnaire in an
attempt to gather information that might bear on acceptabil—

ity of CBA.

67
Questign_1: To what extent do practitioners agree with
authorities that CBA has strong conceptual ties to the three
conceptual models, and to what extent do they agree with
authorities that CBA demonstrates the characteristics of

these assessment models?

Reseazgh_Qn§stian Do practitioners agree that CBA

has strong conceptual ties to the behavioral assessment
model?

Qperatignal_nefinitignz This question was answered by
examining the mean of the practitioners' ratings on the
items derived from the behavioral assessment model (items 9,

15, 18, and 21).

Reseazgh_Qnestign: Do practitioners agree that CBA

has strong conceptual ties to the ecological assessment
model?

Qpe;atignal_pefiinitign: This question was answered by
examining the mean of the practitioners' ratings on the
items derived from the ecological assessment model (items

17, 19, and 24).

Reseazgh_QuestiQn: Do practitioners agree that CBA

has strong conceptual ties to the problem-solving model?

QperaLional_Definitian This question was answered by

examining the mean of practitioners' ratings on the items

derived from the problem-solving model (items 20 and 22).

68

Researgh_QnestiQn: To what extent is CBA perceived to
measure socially valid behavior?

Qpexatignal_nefinitign: This question reflects tenet
number one from Table 2 that behavioral assessment models
measure socially valid behavior. This was operationalized
by examining the extent to which practitioners agree (i.e.,

mean score) with item 9.

Reagaxgh_QuesLiQn: To what extent do practitioners
agree that frequent testing is necessary for proper assess-
ment?

Qpexatignal_nefinitign: This question was operation-
alized by examining the mean of practitioners' ratings on

item 15.

Reseaxgh_QuestiQn: To what extent do practitioners

agree that CBA is sensitive to small gains?

QperaLiQnal_Definitian This was operationalized by

examining the mean of practitioners' ratings on item 18.

Reseaxgh_QnesLignz To what extent do practitioners
agree that CBA uses quantified measures of the student's
behavior to document improvement and intervention effective-
ness?

QperaLiQnal_DefiiniLian This was operationalized by

examining the mean of practitioners' ratings on item 21.

69

Reseaxgh_QnestiQn: To what extent do practitioners
agree that CBA considers situational or environmental fac-

tors in assessment?

Qperafiignal_nefinitign: This was operationalized by

examining the mean of practitioners' ratings on item 19.

Researgh_Questian To what extent do practitioners

agree that CBA provides information that promotes meaningful

outcomes?

Operational_nefinitignz This was operationalized by

examining the mean of practitioners' ratings on item 24.

Researgh_Qngatian To what extent do practitioners

agree that CBA generates multiple interventions?

Operational_nefinitignz This was operationalized by

examining the mean of practitioners' ratings on item 20.

Reseaxgh_QnestiQn: To what extent do practitioners

agree that CBA defines a problem as a discrepancy between

the student's actual performance and that which is expected?

Operational_nefinitign: This was operationalized by

examining the mean of practitioners' ratings on item 22.

Reseaxgh_Questign: To what extent do practitioners

agree that CBA provides data that allow for direct instruc-

tion of deficit skills?

QnexaLiQan_Definitign: This was operationalized by

examining the mean of practitioners' ratings on item 17.

70

Question I was answered by examining the percentage of
practitioners who endorse the model as providing a conceptu-
al basis for CBA and the mean ratings of the items pertain-
ing to the three assessment models. If practitioners do
agree that the three models represent the conceptual under-
pinnings of CBA, then the items pertaining to the specific
models should be strongly endorsed by the practitioners
(i.e., mean ratings above 4.8) and the majority (i.e., more
than 50%) should endorse the particular model. The extent
to which practitioners agree that CBA demonstrates the
characteristics of these three models was answered by exam-
ining the means of the individual items. Additionally, the
extent to which practitioners agree that the characteristics
of a specific model are shared with CBA was analyzed. This
was operationalized by examining the mean ratings of the
unique combination of items representing the specific mod-
els. The behavioral model is represented by items 9, 15,
18, and 21. The ecological model is represented by items
l7, l9, and 24. The problem-solving model is represented by
items 20 and 22. An average mean score (i.e., the sum of
the means of the ratings for each item divided by the number
of items), representing each of the models was computed.
The extent to which practitioners agree that this model
represents the conceptual bases of CBA was examined by
comparing the average means of the three models. Given that
a majority of practitioners did not agree that the models

had strong conceptual ties to CBA, a search for additional

7l

conceptual models may be indicated. Alternatively, it is
possible that the results may suggest the need for addi—
tional educational emphasis on the conceptual underpinnings
of CBA among practitioners. The findings specific to the
characteristics of the various models also indicated the
need for additional investigation into the characteristics
that are considered representative of the models or educa-

tion of practitioners with regard to the respective models.

Questign_11: Do the factors identified from the treatment

acceptability research relate to acceptability of CBA?

Reseaxgh_9nestignz Does the finding that treatments

perceived as effective treatments tend to be perceived as
acceptable also generalize to CBA?
QuaraLiQnal_DflfiiniLiQn: It is predicted that high
perceptions of acceptability will be associated with high
levels of effectiveness. This hypothesis was operation-
alized by examining the correlation between acceptability

and the item measuring effectiveness (items 1 and 4).

Researgh_Questian Is practitioner perception of the

cost of CBA related to perception of its acceptability?

Qneraticnal_nefinition: This was investigated by

examining the correlation between acceptability and cost

(items 1 and 6).

Researgh_Qnestign: Do practitioners perceive of CBA

as having less potential for "stigmatizing labels" (an

72
educational "side-effect" of traditional assessment), and is
this related to acceptability?

Qperatignal_nefinitign: The extent of generalization
of this factor to CBA acceptability was measured by examin-
ing the correlation between acceptability and the item
measuring the potential for CBA to be less stigmatizing

(items 1 and 5).

Research_Question: Is CBA perceived as a more posi-

tively focused assessment technique than traditional assess-
ment and is this related to acceptability?
Qpezatignal_nefinitignz This was assessed by examin-
ing the correlation between acceptability and the item
measuring the extent to which CBA is perceived as a positive

technique (items 1 and 11).

Reseazgh_QnestiQn: Is CBA use related to perceptions

of it as an acceptable technique?

Qpexatignal_nefinitign: This was operationalized by
examining the correlation between acceptability and use of
CBA (items 1 and 2). Also, the number of practitioners who
responded that they use CBA regularly provided an answer to
whether the interest in CBA suggested by the current prolif-

eration of research is reflected by practitioner use.

Researgh_QnesLign: Does practitioner knowledge of CBA

techniques relate to whether or not they tend to view it as

an acceptable technique?

73
Qpexatignal_DefiniLign: Examining the correlation
between acceptability and the item measuring familiarity
with the CBA literature operationalized this research ques-

tion (items 1 and 23).

Reseaxgh_QnestiQn: What is the minimal level of

knowledge of CBA among practitioners, and does this relate
to its acceptability?

Qperatignal_nefiinitignz The response to the item
regarding CBA's use of national norms and the item regarding
acceptability provided some insight into the degree to which
practitioners are aware of one of the more common tenets of
CBA and how this relates to acceptability (items 1 and 3).
This item also provided a less obvious check with the accu-
racy of the response to the items measuring actual CBA use

and familiarity with the CBA literature.

Reseaxgh_QuesLiQn: Does practitioner perception of
the amount of time necessary for CBA assessment relate to
perceptions of acceptability?

Qpexatignal_nefinitignz The possibility of this was
operationalized by examining the correlation between accept-

ability and the item measuring time (items 1 and 10).

Research_QuestiQn: Are high levels of acceptability

associated with belief that CBA is appropriate for assessing

a wide range of problem behavior?

74
Qperatignal_nefinitign: This question was operation-
alized by examining the correlation between acceptability
and the item measuring the perception that CBA is useful for

assessing a range of problem behaviors (items 1 and 8).

In order for question II to be answered in an affirma-
tive manner, the majority of factors had to be significantly
correlated (at the .05 level) with the acceptability item.
The degree to which the individual factors were associated

with CBA acceptability was also assessed in this manner.

Questign_111: To what extent do practitioners agree with

the proposed advantages and disadvantages of CBA?

Researgh_QnesLign: To what extent do practitioners

perceive CBA as a technique that meets special education

requirements for assessment?

Qnerational_nefinition: This was operationalized by

examining the mean of item 16.

Reseazgh_QnestiQn: To what extent do practitioners

perceive that CBA has less potential for stigmatizing la-

bels?

Qperational_nefinitign: This item was operationalized

by examining the mean of item 5.

Researgh_QnestiQn: To what extent do practitioners

believe that CBA is less culturally biased than traditional

tests?

75

QperaLiQnal_DefiniLiQn: This item was operationalized

by examining the mean of item 7.

Reseaxgh_QuestiQn: To what extent do practitioners

perceive that CBA provides instructionally useful data?

Qpera;ignal_nefinitionz This was operationalized by

examining the mean of item 17.

Researgh_Qnestignz To what extent do practitioners

perceive that CBA is sensitive to small gains?

Qpexatignal_flefiinitign: This was operationalized by

examining the mean of item 18.

Researgh_QnesLiQn: To what extent do practitioners

believe that CBA allows for more easily constructed local

norms?

Operational_nefinitignz This was operationalized by

examining the mean of item 14.

Reseaxgh_Qnestign: To what extent do practitioners

perceive that CBA is cost—effective?

Qpfixafiignal_DefiniLiQn: This was operationalized by

examining the mean of item 6.

Eeseaxgh_Qnestian To what extent do practitioners

perceive that CBA has demonstrated psychometric adequacy

(i.e., content validity)?

Operational_nefinitign: This was operationalized by

examining the mean of item 13.

76

Reseazgh_Questignz To what extent do practitioners

agree that they have used CBA regularly?
Qpexational_DefinitiQn: This was operationalized by

examining the mean of item 2.

Research_QuestiQn: To what extent do practitioners

perceive that they are knowledgeable with regard to CBA

techniques?

Operational_nefinitign: This was operationalized by

examining the mean of item 23.

Reseazgh_QnesLign: To what extent is CBA perceived as

more applicable to assessing elementary school children than

secondary school children?

Qperatignal_nefinitign: This was operationalized by

examining the mean of item 25.

Reseaxgh_Qnestian To what extent is teaching to the

test perceived to be more easily accomplished with CBA than

with traditional assessment?

QperaLiQnal_DefiniLiQn: This was operationalized by

examining the mean of item 26.

Researgh_QuestiQn: To what extent are practitioners

aware of CBA's requirement for standardized administration?

Querafiional_DefinitiQn: This was operationalized by

examining the mean of item 12.

77

Question III was answered by examining the means of
the various items. In order for question III to be answered
affirmatively, a majority of the items had to be rated in
the affirmative direction. Similarly, if a majority of the
items were rated in a negative manner, the overall question
would be answered negatively. The final possibility was
that most of the items would be rated close to "neutral"
(i.e., 4.0). In this case, it was possible that question
III could not be answered in either direction. In order to
determine the extent of agreement with individual items,
item means were assessed. While there was a certain amount
of inherent subjectivity involved in determining the degree
of significance of a mean rating, the means were compared
with the seven-point scale for purposes of this study.

Although there was not any empirical evidence on which
to base specific predictions associated with the demographic
data, such data might have revealed possible relationships
between the demographic factors and CBA acceptability. For
example, one possible consideration might be the extent to
which training programs with different theoretical emphases
are associated with practitioners' acceptance of CBA. If
practitioners from a common program received their training
based upon a particular theoretical background, would they
tend to view the acceptability of CBA similarly? Examining
the relationship between respondents' training programs and
their degree of acceptance of CBA allowed for consideration

of such a notion. Other demographic relationships were also

78
possible, including that of the effects of experience, the
level of training, and when the degree was granted. There
were no specific predictions made regarding these demograph-
ic factors, but they were included in order to gather evi-

dence for the identification of possible relationships.

5"JEJ'

The statistical analysis assumed the use of interval
data and a continuous variable. While it was apparent that
a seven—point scale may not accurately represent an interval
scale, and was more accurately called an ordinal scale, it
was also arguable that the variable "degrees of agreement"
was a continuous variable with an approximately normal dis-
tribution in the population. If this was so, then it was
reasonable to treat ordinal data as interval data for sta-
tistical purposes (Crocker & Algina, 1986). Therefore, the
analysis treated the data as interval under these assump-
tions.

Much of the research involved the simple computation
of summary statistics. The question of whether the concep-
tual bases of CBA were perceived by practitioners to be the
same as those reported by Knutson and Shinn (1991) for CBM
was addressed by examining the number who endorsed the
various models and the summed means of the items represent-
ing those models. The item means were compared to the
seven-point scale (i.e, 1=strongly disagree, 2=disagree,

3=slightly disagree, 4=neither agree nor disagree or don't

79
know, 5=slightly agree, 6=agree, 7=strongly agree), to
provide a descriptive summary of practitioner agreement or
disagreement for the item.

For the section on treatment acceptability, a multiple
correlation technique was used to address the question of
association of the items to the acceptability item., A table
was constructed so that the relationship of the items to the
acceptability item can be readily observed. In situations
where the questions were "negatively worded" to avoid a
positive or negative response bias, the values of the scale
were reversed in order to allow for the outcome data to be
in the direction predicted.

The section on the purported advantages and disadvan-
tages followed the same procedure as the section dealing
with the items representing assessment models. That is,
item means were compared to the seven point-scale as noted
above.

Although no specific predictions were made regarding
the demographic data, the results did allow for some post—
hoc analysis of possible relationships such as the relation-
ship of demographic characteristics to CBA acceptability.
This analysis consisted of two separate techniques. On
those items where the respondents could be divided into
meaningful groups (i.e., theoretical perspective, university
attended, and highest degree held), an ANOVA was conducted

with the ratings on the acceptability item as the dependent

80
variable. For those items which did not allow for meaning-
ful groups to be constructed (i.e., date degree was
received, years as a teacher, years as a school psychol-
ogist, number of students assessed last year, number of
students expected to assess this year, and percentage of
time as a school psychologist), a correlation was computed
between the individual items and the acceptability item.
Also, summary statistics were provided. For example, data
regarding the characteristics of the respondents and non-
respondents was included.

A measure of reliability of the questionnaire was
provided, although it should be noted that the study did not
intend to produce an internally consistent questionnaire
measuring a single domain. Rather, the questionnaire was
designed to measure practitioner perceptions on a number of
items, possibly sampling from different domains. In any

case, coefficient alpha was computed and found to be .77.

RESULTS AND DISCUSS ION

Questionnaires were mailed to 869 school psychologists
in the state of Michigan. Of those 869, 64 were returned
for insufficient addresses or other reaSons making delivery
to the school psychologists not possible. One questionnaire
was returned by the school indicating that the practitioner
was deceased. Two other questionnaires were returned indi-
cating that the addressees were not school psychologists.
Further screening of the mailing data base revealed that
there were 15 duplicates in the original mailing list.
Therefore, the population was, in effect, reduced to 787.

Of these 787, 348 usable questionnaires were returned. This
resulted in a return rate of approximately 44%. Although
this rate was lower than 50%, the rate desired by the au-
thor, it may not be unusual for this area of research. In
the only other known survey of CBA acceptability (Shapiro &
Eckert, 1993), the return rate was 49%. The researchers
noted that, although the return rate was acceptable, it may
have resulted in a response bias due to self-selection into
the study. Also, despite the author's assertion that the
return rate was acceptable, authorities on survey research
tend to disagree. Citing "rules of thumb for survey re-

search," Babbie (1989) notes that a very good response rate

81

82
is 70%, a good response rate is 60%, and 50% would be con-
sidered adequate. These same cautions would appear warrant-
ed in the present study.

Of the 348 returned questionnaires, 166 requests for a
copy of the final data were received. A summary of the
overall results (see Table 8), including a narrative de-
scription of the conclusions and the item summary statis-

tics, was sent (see Appendix C).

W
Questign_1: To what extent do practitioners agree with
authorities that CBA has strong conceptual ties to the three
conceptual models, and to what extent do they agree with
authorities that CBA demonstrates the characteristics of
these models?

The extent to which practitioners agreed with authori-
ties on the conceptual models was analyzed by examining the
percentage of respondents who endorsed those models as
providing a conceptual basis for CBA. The behavioral as-
sessment model was endorsed by approximately 34% of the
respondents. The ecological model was endorsed by approxi-
mately 25%. The data-based problem solving model was en-
dorsed by 45%.

The degree to which practitioners agreed with the
authorities was further assessed by examining the ratings of
the individual items pertaining to the models. The level of

agreement or disagreement on the items was determined by

83

Table 8. Summary Statistics
Mean Ratings and Standard Deviations on Questionnaire Items:

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

IlFreq. testing necessary for proper assessment SL-A

 

CBA more likely to meet special education law SL-D

 

 

 

 

 

 

Range 1-7

( '” ”“7 ’ ‘“ "‘““"' "”'" “" “ ‘7 1 ‘“ “ ‘“ 7 “ ' ‘77 ” 1
1 ITEM LEVEL ITEWI STD .
( ENDORSED MEAN DEV (

§CBA is acceptable assessment technique SL-A 5.2 1.3
?I have used CBA regularly SL-D 2.5 1.5 .
:CBA does not rely on national norms A 5.5 1.4 ;
!CBA is an effective assessment technique SL-A 5.1 1.3 E
:CBA less apt to use labels N/DK 4.6 1.5 !
?CBA cost-effective N/DK 4.2 1.4 ‘

!CBA less culturally biased N/DK 4.1 1.4

lCBA not effective for range of handicap SL-A 4.5 1.5
jCBA provides more socially valid measurement N/DK 4.2 1.3 i
{CBA takes more time for assessment N/DK 3.7 1.4 )

'CBA emphasizes more positive approach SL-A 4.8 1.4

“Standardized administration not necessary N/DK 3.6 1.5
CBA has high content validity SL-A 5.2 1.3 )
Easier to construct local norms with CBA SL-A 4.5 1.5 1

4.6 1.4

3.1 1.4

5.5 1.2

5.1 1.4

4.6 1.4

4.6 1.4

4.7 1.3

4.2 1.5

 

 

 

 

  
     
 

 

CBA provides data for direct instruction A “
CBA not as sensitive to small gains SL-A
CBA considers sit./environ. factors SL-A
"CBA less likely to generate mult. interven. SL-A
CBA uses quant. meas. of student behavior SL-A
CBA defines problems as envir./expect. N/DK
discrep. -
"I am knowledgeable with regard to CBA N/DK 3 .6
CBA provides info lead. meaningful outcome SL-A
CBA more appropriate to elementary than N/DK 3.6 1.4

secondary

 

 

 

 

   

Teaching to test more easily accompnished

LEVEL OF ENDORSEMENT: SD=strongly disagree, D=disagree, SL-D=slightly
disagree, N/DK=neither agree nor disagree or don't know, SL-A=slightly
agree, A=agree, SA=strongly agree

DEFINITION OF INTERVALS: 1 to 1.49=strongly disagree, 1.5 to 2.49=dis-
agree, 2.5 to 3.49=slightly disagree, 3.5 to 4.49=neither agree nor dis-
agree or don't know, 4.5 to S.49=slightly agree, 5.5 to 6.49=agree, 6.5
to 7=strongly agree

84
examining the item mean and referring to the seven-point
scale (i.e., 1 to l.49=strongly disagree, 1.5 to 2.49=dis—
agree, 2.5 to 3.49=slightly disagree, 3.5 to 4.49=neither
agree nor disagree or don't know, 4.5 to S.49=slightly
agree, 5.5 to 6.49=agree, 6.5 to 7=strongly agree).

The behavioral model was represented by items 9, 15,
18, and 21. On a scale ranging from one to seven, item 9
received a mean rating of 4.21. Item 15 received a mean
rating of 4.59. Item 18 received a mean rating of 5.14.
Item 21 received a mean rating of 4.76. Therefore, of the
three items representing the behavioral model, two received
a rating within the slight agreement range, and one fell
within the neutral range. The overall mean rating (i.e.,
the sum of the ratings of the items divided by the number of
items) of the behavioral model was 4.67. This rating fell
within the range of the "slightly agree" rating, suggesting
that practitioners had only a tendency, if any, toward
slight agreement on the items representing the behavioral
model.

The ecological model was represented by items 17, 19,
and 24. The mean rating on item 17 was 5.54. The mean
rating for item 19 was 4.63. Item 24 received a mean rating
of 5.15. The first item fell within the range of agreement,
the other two received ratings of slight agreement. The
overall rating for the model was 5.10, within the range of

slight agreement. Therefore, it appears that practitioners

85
tend to slightly agree with the items representing the
ecological model.

The data-based problem solving model was represented
by items 20 and 22. The mean rating on item 20 was 4.57.
The mean rating on item 22 was 4.16. The first item fell
within the range of slight agreement, the second item was
rated as neutral or don't know. The overall rating of this
model was 4.36. This rating fell within the "neutral"
range, suggesting that practitioners neither agree nor dis-
agree, or don't know about the items representing the data—
based problem solving model.

The results of the analysis of the items describing
the conceptual models suggested that there is a tendency for
practitioners to slightly agree with the items representing
the ecological model. However, practitioners appeared to
neither agree nor disagree, or didn't know about the items
representing the data-based problem solving model and dis-
played only a tendency toward slight agreement on the items
representing the behavioral model (see Table 9 for item
statistics summary). Overall, of the 9 items representing
the conceptual models, 1 received a rating of agreement, 5
received ratings of slight agreement, and 3 were rated as
neutral or don't know. These results appear to be consis-
tent with the relatively weak endorsement of the theoretical
models (ranging from 25 to 45%). It would seem then, that
practitioners show only mild agreement with authorities

regarding the conceptual basis of CBA. A more parsimonious

86

Table 9. Item Means, Standard Deviations:
Conceptual Model Items

‘ ITEM LEVEL
( ENDORSED

»- _. _ _ 1* ",7 1,, __ _ *1. _¥__1V_ .fi. fi__

 

CBA provides more socially valid measures of N/DK
i behavior than traditional techniques. (B)

 

Frequent testing is necessary for proper SL-A
, assessment. (B)

 

, provements as traditional techniques (reverse
' coded). (B)

 

CBA uses quantified measures of the student's SL-A
, behavior to document improvement and inter-
= vention effectiveness. (B)

 

I
|
’ CBA is not as sensitive to small gains or im- SL-A
1
I
l
1

; CBA provides data that allows for direct in- A
, struction of deficit skills. (E)

 

' CBA is more likely to consider the situation- SL-A
al/environmental factors than is traditional
assessment. (E)

 

I
( CBA provides information that leads to mean- SL-A
ingful outcomes. (E)

 

CBA is less likely to generate multiple in- SA-A
terventions than traditional techniques (re-
verse coded). (D)

 

CBA defines a problem as a discrepancy be- N/DK
tween the student's performance and what is
expected.n(D)

C__. ._ _.__ _

 

 

 

 

.
I

 

CODE: B=behavioral, E=ecological, D=data-based problem solving

LEVEL OF ENDORSEMENT: SD=strongly disagree, D=disagree, SL-D=slightly
disagree, N/DK=neither agree nor disagree or don't know, SL-A=slight1y
agree, A=agree, SA=strongly agree

87

explanation, perhaps, is that practitioners may be unaware
of the theoretical sources of the various characteristics.
Qng§;1Qn_II: Do the factors identified from the treatment
acceptability research relate to the acceptability of CBA?

This question can be answered by examining the corre—
lations between the acceptability item (number 1) and the
items derived from the treatment acceptability research.

Table 10 presents a summary of the item correlations.

Table 10. Correlation Matrix: Treatment
Acceptability Items.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

1

2

3
fl 4 .72 .19 .15 1.0
n 5 .33 .10 .04 .41 1.0
II 6 .21 .09 -.10 .22 .29 1.0

8 .30 .001 .09 .34 .04 .13 1.0

10 .07 .14 -.06 .06 -.02 .19 .18 1.0

n 11 .50 .16 .12 .51 .47 .20 .22 -.05 1.0
I! 23 .21 .43 .20 .25 £12 .12= .03 .07 .14 1.0

 

Does the finding that treatments perceived as effec-
tive treatments tend to be perceived as acceptable also
generalize to CBA? This was operationalized by examining
the correlation between the acceptability item and the item
measuring effectiveness (items 1 and 4). The correlation
between these items was .72 (p5.01). It would appear that

practitioners who perceive CBA to be an effective technique

88
also tend to perceive it as an acceptable technique. This
finding was consistent with the research on treatment
acceptability that noted effective treatments tend to be
perceived as more acceptable than less effective treatments
(Kazdin, 1977; Wolf, 1978).

Is practitioner perception of the monetary cost of CBA
related to their perception of its acceptability? This
question was answered by examining the correlation between
items 1 and 6. The correlation between these items was .21
(p$.01). Although the correlation was statistically signif-
icant, it accounted for only about 4% of the variation.
Therefore, although it would appear that practitioners'
perception of CBA as cost-effective tended to be associated
with their perceptions of it as an acceptable technique,
this relationship was relatively weak. The finding of a
positive relationship was consistent with the treatment
acceptability research.

Does practitioner perception of CBA as having less
potential for "stigmatizing labels" related to acceptabili-
ty? The correlation between items 1 and 5 provided evidence
to answer this question. The correlation was .33 (p$.01).
The finding suggested that practitioners who perceive CBA as
having less potential for stigmatizing labels also tended to
perceive it as a more acceptable technique. This finding
was also consistent with the research on treatment accept-

ability.

89

Is perception of CBA as a more positively focused
assessment technique related to its acceptability? This
question was answered by examining the correlation between
items 1 and 11. The correlation was .50 (ps.01). Practi—
tioners who view CBA as more positively focused than tradi-
tional techniques also tended to view it as a more accept-
able technique. Once again, this finding was consistent
with the treatment acceptability research.

Is CBA use related to perceptions of it as an accept—
able technique? This was answered by examining the correla-
tion between items 1 and 2. The correlation between these
items was .24 (p$.01). There appeared to be a mild tendency
for CBA use to be associated with perceptions of it as an
acceptable technique. This finding was also consistent with
the research on treatment acceptability.

Does practitioner knowledge of CBA relate to accept-
ability? The correlation between items 1 and 23 was exam-
ined to answer this question. The correlation was .21
(p$.01). It appeared that practitioner knowledge of CBA
tended to be somewhat associated with perceptions of accept-
ability. This finding was also consistent with treatment
acceptability research.

Is a minimal level of knowledge of CBA among practi-
tioners related to its acceptability? The correlation
between items 1 and 3 was analyzed to answer the question of
relationship. The mean of item 3 provided a measure of

practitioner knowledge. The mean of item 3 was 5.51. This

90
indicated that practitioners were aware that CBA does not
rely on national norms (considered a minimal level of knowl-
edge of CBA). The correlation between items 1 and 3 was .15
(ps.01). Although this relationship was statistically
significant, it accounted for a scant 2% of the variation.
Practitioner knowledge of CBA may have tended to be associ-
ated with perceptions of it as an acceptable technique, but
to a limited degree. Despite the weakness of the relation-
ship, this finding was consistent with treatment acceptabil-
ity research.

Are perceptions of acceptability associated with
perceptions of CBA as appropriate for assessing a wide range
of problem behaviors? The correlation between items 1 and 8
was assessed to provide evidence for answering this ques-
tion. The correlation was .30 (ps.01). Practitioners'
perceptions of the range of appropriateness of CBA assess-
ment tended to be associated with its acceptability. This
finding was consistent with the treatment acceptability
research.

Does practitioner perception of the amount of time
necessary for CBA relate to perceptions of its acceptabili-
ty? Examining the correlation between items 1 and 10 pro-
vided the evidence to answer this question. The correlation
was .07. This correlation was not statistically signifi-
cant. This finding was interpreted as not consistent with
the research on treatment acceptability. Although it is not

possible to provide a definitive explanation for this

91

inconsistency, one possibility is that school psychology
practitioners did not associate less time with the increased
acceptability of a technique. That is, they may have be-
lieved that an acceptable assessment technique must involve
a certain minimal amount of time. Although some school psy-
chologists reported feeling pressured to complete a number
of assessments within a short period of time, they may have
believed that more valid, and acceptable, assessment actual-
ly involved more time. It is possible that assessments that
took more time were actually perceived as more acceptable,
provided that they did not require too much time (e.g., more
than one hour). Another possibility would be the existence
of a curvilinear relationship. Practitioners may have
believed that more acceptable techniques fall within a given
range of time. Techniques may be perceived as more accept-
able if they are neither too brief nor too long, but fall
within this range. It is also possible that the wording of
the item (i.e., "CBA takes more time than traditional tech-
niques") contributed to the finding. Future research should
include variations of this question to more clearly delin-
..eate this factor. Further investigation will be necessary
to replicate this finding and provide evidence to support
the explanations offered.

In order to answer Question II in an affirmative
manner a majority of the items had to be strongly associated
with acceptability (i.e., correlations significant at p=.05

or more). All but one of the items was significant at the

92
p$.01 level. With the exception of the time factor, it
would appear that the same dimensions influencing the ac-
ceptability of treatments operate to some degree to influ-

ence the acceptability of CBA.

Questign_111: To what extent do practitioners agree with
the proposed advantages and disadvantages of CBA?

This question was answered by examining the means of
the various items that relate to the advantages and disad-
vantages of CBA that have been put forth in the CBA litera-
ture. Once again, the level of agreement or disagreement on
the items was determined by examining the item mean and
referring to the seven-point scale (i.e., 1 to 1.49=strongly
disagree, 1.5 to 2.49=disagree, 2.5 to 3.49=slightly dis-
agree, 3.5 to 4.49=neither agree nor disagree or don't know,
4.5 to 5.49=slightly agree, 5.5 to 6.49=agree, 6.5 to
7=strongly agree).

To what extent do practitioners perceive that CBA
meets the special education requirements for assessment?
This question was ansWered by examining the mean of item 16.
The mean for this item was 3.15, representing slight dis-
agreement. This rating indicated that practitioners slight-
ly disagree with the statement that CBA meets the special
education requirements for assessment.

To what extent do practitioners perceive that CBA has
less potential for stigmatizing labels? This question was

answered by examining the mean of item 5. The mean for this

93
item was 4.62. This rating fell within the range of slight
agreement. Practitioners appeared to have a tendency to-
wards slight agreement. However, there was no real consen-
sus with regard to this item.

To what extent do practitioners believe that CBA is
less culturally biased than traditional tests? This ques-
tion was answered by examining the mean of item 7. The mean
on this item was 4.07, showing that respondents as a group
were essentially neutral or undecided in their perceptions
of CBA as less culturally biased. From a practitioner's
standpoint, the jury is still out on this important issue.

To what extent do practitioners perceive that CBA
provides more instructionally useful data? This question
was answered by examining the mean of question 17. The mean
for this item was 5.54, indicating that practitioners report
agreement in perceiving that CBA provides more instruction-
ally useful data than traditional techniques.

To what extent is CBA perceived to be sensitive to
small gains? This question was answered by examining the
mean of item 18. The mean for this item was 5.14, indicat-
ing that practitioners slightly agreed with regard to CBA
being sensitive to small gains.

To what extent do practitioners believe that CBA
allows for more easily constructed local norms? The mean of
item 14 was assessed in order to provide information to

answer this question. The mean was 4.49. This rating fell

94
within the neutral range. The results suggested that prac-
titioners have not come to a consensus on this item either.

To what extent do practitioners perceive that CBA is
cost-effective? The mean of item 6 was assessed to answer
this question. The mean of this item was 4.25. Practi-
tioners neither agreed nor disagreed, or do not know about
the relative cost-effectiveness of CBA.

To what extent is CBA perceived as having demonstrated
psychometric adequacy (i.e., content validity)? The mean
of 5.19 on item 13 showed that practitioners have a tendency
towards slight agreement on the demonstrated psychometric
adequacy of CBA.

To what extent do practitioners agree that they have
experience in the use of CBA techniques? This question was
answered by examining the mean of item 2. The mean of this
item was 2.49. This rating fell within the "disagree"
range. Practitioners did not perceive themselves as being
experienced with CBA techniques.

To what extent do practitioners perceive themselves to
be knowledgeable with regard to CBA techniques? This ques-
tion was answered by examining the mean of item 23. The
mean of this item was 3.49. This rating fell within the
slightly disagree range. Practitioners tended to slightly
disagree with the statement that they are knowledgeable with
regard to CBA techniques.

To what extent is CBA perceived as more applicable to

assessing elementary school children than secondary school

95
children? This question was answered by examining the mean
of item 25. The mean of this item was 3.65, suggesting that
practitioners tended to neither agree nor disagree, or do
not know about CBA being more applicable at the secondary
level.

To what extent is teaching to the test perceived to be
more easily accomplished with CBA than with traditional
assessment? This question was answered by examining the
mean of item 26. The mean of this item was 4.79, suggesting
slight agreement.

To what extent do practitioners believe that CBA
requires standardized administration? This question was
answered by examining the mean of item 12. The mean of that
item was 3.65, suggesting practitioners neither agreed nor
disagreed, or do not know about CBA requiring standardized
administration.

In order for Question III to be answered in an
affirmative manner, the majority of the questions had to be
rated in an affirmative direction. Of the thirteen items,
only one item fell within the range of "agree": provides
more instructionally useful data, and only four were rated
as "slightly agree": less potential for stigmatizing labels,
sensitive to small gains, has demonstrated psychometric
adequacy, and teaching to the test is more easily accom-
plished. Due to the considerable variation and number of
relatively weak ratings, it was not possible to answer Ques-

tion III in an affirmative manner. Practitioners tended to

96
be neutral or undecided about the majority of purported
advantages and disadvantages of CBA included in the study.
That is, there was no convincing evidence that they have
reached a strong consensus, either pro or con, on most of
the items included in the present study (see Table 10). The
two notable exceptions were the tendency for practitioners
to agree that CBA provides instructionally useful data and
their tendency to disagree that they are experienced with

regard to CBA techniques.

W

Despite the fact that no specific predictions were
made based upon the demographic data, these data were ana-
lyzed for evidence of possible relationships between the
demographic data and CBA acceptability.

In order to investigate the possibility of a relation-
ship between theoretical_pexspegtixe and acceptability, an
ANOVA was conducted. None of the results was statistically
significant (see Table 11). The results suggested that the
theoretical perspectiVe of practitioners was not related to
their perception of CBA as an acceptable technique.

The possibility of a relationship between the practi-
tioner's training_institntign and the acceptability of CBA
was also investigated via ANOVA. The results of this analy-
sis were also not statistically significant (see Table 12).
The results suggested no relationship between training

institution and practitioner perception of CBA.

97

Table 11. Item, Means, Standard Deviations:
Advantages/Disadvantages of CBA

 

 

CBA meets the special education requirements
or assessment?

 

f
.To what extent do practitioners perceive that SL-A 4.62 1.5
CBA has less potential for stigmatizing 1a-
abels?

'To what extent do practitioners perceive that SL-D 3.15 1.40
2

 

To what extent do practitioners believe that N/DK 4.07 1.44
CBA is less culturally biased than tradition-
.al tests?

 

:To what extent do practitioners perceive that A 5.54 1.23
CBA provides more instructionally useful da-
ta?

;To what extent is CBA perceived to be sensi- SL-A 5.14 1.40
tive to small gains?

 

 

FTo what extent do practitioners believe that N/DK 4.49 1.49
CBA allows for more easily constructed local
.norms?

 

To what extent do practitioners perceive that N/DK 4.25 1.40
CBA is cost-effective?

 

To what extent is CBA perceived as having SL-A 5.19 1.31
demonstrated psychometric adequacy (i.e.,
content validity)?

 

 

To what extent do practitioners agree that D 2.49 1.54
they have experience in the use of CBA tech-
niques?

l[To what extent do practitioners perceive SL-D 3.49 1.63

themselves to be knowledgeable with regard to
CBA techniques?

 

To what extent is CBA perceived as more ap- N/DK 3.65 1.37
plicable to assessing elementary school chil-
dren than secondary school children?

 

To what extent is teaching to the test per- SL-A 4.79 1.34
ceived to be more easily accomplished with
CBA than with traditional assessment?

 

 

 

 

 

 

To what extent do practitioners believe that N/DK 3.65 1.55
CBA requires standardized administration?
M

 

LEVEL OF ENDORSEMENT: SD=strongly disagree, D=disagree, SL-D=slightly
disagree, N/DK=neither agree nor disagree or don't know, SL-A=slightly
agree, A=agree, SA=strongly agree

Table 12.

98

Analysis of Variance of Theoretical
Perspective and CBA Acceptability

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

:47
H Source of Sum of Degrees of Mean Signif.
Variation Squares Freedom Square F of F
Main Effects 12.197 3 4.066 2.418 .066
Residual 563.184 335 1.681
Total 575.381 338 1.702
1 I--====-=- III-H-fl-iaalnnl-El-Il
Table 13. Analysis of Variance of Training
Institution and CBA Acceptability
“ I---I
Source of Sum of Degrees of Mean Signif.
Variation Squares Freedom Square F of F
Main Effects 16.818 7 2.403 1.410 .200
Residual 572.737 336 1.705
“ Total 343 1.719

 

589.555
--==-41

 

 

 

 

 

 

 

The possibility of a relationship between the degree

held_in_eeheel_peyehelegy and CBA acceptability was investi-

gated via ANOVA as well.

not statistically significant (see Table 14).

The results of this analysis were

The results

suggested that the degree held in school psychology was not

related to practitioners' perception of CBA as an acceptable

technique.

Table 14.

Analysis of Variance of Degree Held and
CBA Acceptability

 

 

 

 

 

 

 

 

 

 

 

Source of Sum of Degrees of Mean Signif.
Variation Squares Freedom Square F of F
Main Effects 3.598 3 1.199 .703 .551
Residual 571.152 335 1.705 “
Total 574.749 338 1.700 n
=-

99
In order to investigate the remainder of the demo-
graphic data, correlations were computed between the accept-

ability item and the demographic data.

The possibility that the date_the_degne_e_in_s_cho_ol
peyehelegy_mee_xeeeiyed and acceptability of CBA are related

was investigated. The correlation between these items was
.0898 and was not statistically significant (see Table 15).
The results suggested no relationship between the date the
degree was received and the acceptability of CBA.

The possibility of a relationship between the number
efi_yeeze_ee_e_eeeeher and the acceptability of CBA was also
investigated with this technique. The correlation between
these items was .0308 and was not statistically significant
(see Table 15). The results suggested no relationship

between teaching experience and acceptability of CBA.

Table 15. Correlations: Demographic Items and
CBA Acceptability

 

 

 

 

 

 

 

 

 

. Date degree received. .0898 “
Number of years experience as a teacher .0308 m
|
Number of years experience as a school psy- -.0718 F
chologist
Number of students expected to assess this -.0310
year
If
Number of students assessed last year -.0159
Percentage of time spent as a school psy- .0536
chologist
None of the above correlations was significant using the
.05 level as a minimum criterion.
=

 

 

 

100

The possibility of a relationship between the number
Qf_yeere_ae_a_eeheel_peyehelegier and the acceptability of
CBA was investigated. The correlation between these items
was -.0718 and was not statistically significant (see Table
15). The results suggested that the number of years as a
school psychologist was not related to the acceptability of
CBA.

The possibility of a relationship between the number
WW and the accept-
ability of CBA was investigated. The correlation between
these items was -.0310 and was not statistically significant
(see Table 15). The results suggested that the number of
students expected to assess in the year of the study was not
related to perceptions of CBA acceptability.

The possibility of a relationship between the number
W to the study and the
acceptability of CBA was investigated. The correlation
between these items was -.0159 and was not statistically
significant (see Table 15). The results suggested that
there was not a relationship between the number of students
assessed the year prior to the study and perceptions of CBA
acceptability.

The possibility of a relationship between the pereenb;
WW and the acceptability
of CBA was investigated. The correlation between these
items was .0536 and was not statistically significant (see

Table 15). The results suggested that the percentage of

101
time spent as a school psychologist was not related to

perceptions of CBA as an acceptable technique.

Some limitations were inherent in the design of this
study. Perhaps foremost was its associative nature. The
research design called for an assessment of the relationship
between identified factors and acceptability. It was not
possible to determine the cause of practitioners' percep—
tions of acceptability of CBA utilizing the present design.
Also, although a rationale for the choice of factors was
provided, it was not possible to determine if the "correct"
factors were identified. It was not possible to determine
if these factors were ideal or even adequate.

The degree of generalizability of the data may be
limited based upon the obtained sample. The present study
obtained a return rate of approximately 44%. This rate was
lower than the minimal target rate of 50% (Babbie, 1989).
However, this rate may not be unusual for this area of
research. In the only other known survey of CBA acceptabil-
ity (Shapiro & Eckert, 1993), the return rate was 49%.
Those researchers noted that, although the return rate was
acceptable, it may have resulted in a response bias due to
self-selection into the study. Those same cautions would
appear warranted in the present study.

Another concern centered around the possible

102
differences between respondents and non-respondents. An
analysis was conducted to compare the respondents and non-
respondents on the demographics of sex, age and ethnicity.
A chi-square analysis was conducted on the demographics of
sex and ethnicity. Among the respondents, 42% were male,
57% were female. Among the non-respondents, 45% were male
and 55% were female. The results of the chi-square analysis
revealed that the groups were not significantly different
using p=.05 as the minimum level of statistical signifi-
cance. The analysis of ethnicity revealed that approximate-
ly 92% of the respondents were white; about 6% were African-
Americans; and less than 1% were Hispanic, American Indian,
or Asian. Among the non-respondents, approximately 93% were
white, approximately 6% were African-American, and 1% were
American Indian, Hispanic, or Asian. The chi-square
analysis was not statistically significant at the p=.05
level, suggesting that the two groups did not differ signif-
icantly on the demographic variable of ethnic composition.
In order to investigate the demographic variable of age, a
t-test of means was conducted. The analysis revealed that
the mean age of respondents was 44.5 years and the mean age
of non-respondents was 45.6 years. The t-test of means was
not significant at the p=.05 level, this suggested that the
two groups did not differ significantly along the demograph-
ic variable of age. Although it was not possible to con-
clude that the two groups did not differ along some other

important dimension, there were no significant differences

103
between respondents and non-respondents with respect to age,
sex, and ethnicity.

A related concern was the possibility of a positive
response bias among the volunteer respondents. Although the
subjects did not appear to know much about or use CBA, only
one item that was critical of CBA received a rating of
slight agreement. This finding lead to the suspicion of the
presence of a response bias.

Another limitation of the study was the limited prac-
tical knowledge level of the practitioners with regard to
CBA. The mean response to the item "I have used CBA regu-
larly" was 2.49, indicating that practitioners did not agree
with this statement. As a result, it was not clear whether
non-usage or irregular usage stemmed from a lack of knowl-
edge about CBA techniques or for other reasons.
Unfortunately, the scale did not provide the option of a "no
basis for judgement" response. This option could have
provided additional information regarding where practitioner
knowledge was lacking.

It remains to be demonstrated how generalizable the
findings are. The present study was offered not as a defin-
itive answer to the issue of acceptability of CBA as an
assessment model, but as an initial attempt towards under-
standing how practitioners perceive the factors identified
as influencing the degree to which they find CBA to be

acceptable.

104
E' .

The analysis of the results regarding the perception
of the three assessment models cited as providing a concep-
tual framework for CBA (i.e., ecological, data—based problem
solving, and behavioral) suggested that practitioners showed
only mild agreement with authorities. Practitioners seemed
unaware of these models' relationship to CBA. The evidence
for this statement was the fact that few practitioners (25%
for the ecological, 34% for the behavioral, and 45% for the
data-based problem solving) endorsed the models as having
conceptual ties to CBA. From the relatively low percentages
of endorsement, it seemed possible that either practitioners
did not perceive these models to underlie CBA or that they
were unfamiliar with the models. Practitioners also showed
only mild agreement with authorities regarding the extent
that CBA demonstrated the characteristics of these assess-
ment models. The results of the analysis on the items
representing the conceptual models revealed that there was a
tendency for practitioners to slightly agree with the items
representing the ecological model. However, practitioners
appeared to neither agree nor disagree, or did not know
about the items representing the data-based problem solving
model and displayed only a small tendency toward slight
agreement on the items representing the behavioral model
(see Table 9 for item statistics summary).

These results appeared to be consistent with the rela-

tively weak endorsement of the theoretical models (ranging

105
from 25 to 45%). It would seem then, that practitioners
only slightly agreed with authorities regarding the concep-
tual basis of CBA. It was also possible that practitioners
may have been unaware of the theoretical sources of the
various characteristics. The implication of this finding
pointed to the need for additional emphasis and education
regarding the origins of CBA techniques. Although CBA
advocates have written prolifically about its use and merit,
the neglect of its origins appeared to be manifested in
practitioner confusion. The need for clarification is not
limited to historical accuracy. In order to evaluate the
utility and validity of any assessment technique, it is
necessary to consider the theoretical bases upon which it is
founded. In the case of CBA, it appears that the utility of
the technique may have preceded evaluation of the theory or
theories which have led to it. Careful investigation of the
bases of CBA may suggest important considerations for prac-
titioner use of the CBA model (e.g., the need to gather
local norm data, standardized assessment procedures, defini-
tion of what constitutes a problem or handicap, etc.). The
impact of such consideration upon the use of CBA is unknown
at this time. However, possibilities would include a more
definitive definition of CBA, a clearer delineation of its
potential use and applicability, and a more realistic analy-
sis of its strengths and limitations. Future research may
wish to utilize an analogue method similar to the one em—

ployed by Shapiro and Eckert (1993). Detailed descriptions

106
of the model (including its label) could precede a case
description employing an assessment consistent with the
model. It would be informative to investigate the impact of
the description of the model upon acceptability.

The analysis of the results pertaining to treatment
acceptability research suggested that much of that litera-
ture may apply to the determination of CBA acceptability.
The results obtained suggested that practitioner perceptions
of factors identified as important to treatment acceptabili-
ty generalized to some degree to CBA acceptability as well.
These results were also consistent with the findings of
Shapiro and Eckert (1993) who, utilizing a different ap-
proach, also demonstrated the importance of considering the
factors from treatment acceptability when determining the
acceptability of CBA. Given that two recent studies have
come to similar conclusions, it seems worthwhile to explore
other areas of generalizability for the treatment accept-
ability factors. A likely candidate would be assessment in
general. Can the acceptability of assessment techniques be
determined using the factors identified by the treatment
acceptability research? Future research should focus on
investigating the generalizability of the present findings
to other forms of assessment. If consistent, there would be
evidence that the factors influencing treatment acceptabili-
ty may be applied to the acceptability of assessments. Such
information should prove informative to practitioners who

are responsible for choosing assessment techniques

107
acceptable to themselves as well as those who will be influ-
enced by the results (i.e., other school personnel, chil-
dren, and parents). In addition, test manufacturers may
wish to consider the influence of these factors in designing
assessment techniques that are consistent with factors
influencing acceptability. Future research should also
continue to investigate the possible existence of excep-
tions. The factor of time is an example. The present study
did not support the generalization of the time factor to CBA
acceptability. Replication of this result in future re-
search is necessary before it can be determined that time
does not, in fact, influence the acceptability of CBA. If
this result proves consistent, an important question to
answer is why practitioners do not feel that less time-
consuming techniques are more acceptable than more
time-consuming techniques. The possibility of a curvilinear
relationship between time and acceptability should also be
investigated. The discovery of an "acceptability range"
with regard to time would have important implications for
both practitioners and test manufacturers.

The analysis of the results pertaining to the purport—
ed advantages and disadvantages of CBA suggested that there
was considerable uncertainty or lack of knowledge among
practitioners with regard to the CBA literature. It ap-
peared that practitioners tended to be neutral or undecided
about the majority of purported advantages and disadvantages

of CBA included in the study. That is, there was no

108
convincing evidence that they have reached a strong consen-
sus on any of the items included in the present study. It
appeared that practitioners did not, as a group, agree that
CBA possessed all the advantages that have been cited in the
CBA literature. None of the purported advantages received a
rating of strong agreement. Only one item received a rating
of agreement; practitioners agreed that CBA provides in-
structionally useful data. It would be informative to
investigate this area further. The present study did not
analyze what practitioners consider to be instructionally
useful data (e.g., designing, implementing, or evaluating)
and how CBA provides it. This needs to be addressed in
further research. However, if CBA effectively provides
information linking assessment to intervention, as
practitioners tended to believe, then this technique may
prove useful beyond an adjunctive role to school psycholo-
gists.

The results also revealed that practitioners have a
tendency towards slight agreement in perceiving that CBA has
less potential for stigmatizing labels, that CBA is sensi-
tive to small gains, that CBA has demonstrated psychometric
adequacy, and that teaching to the test is more easily
accomplished with CBA.

Practitioners had a tendency to perceive CBA has less
potential for stigmatizing labels. It was possible that
practitioners believed this to be so as a result of the

observation that CBA does not meet all the requirements of

109
special education law. More research is necessary to deter-
mine if this is so.

It also appears that practitioners had at least a
tendency to agree that CBA is sensitive to small gains. The
present study did not analyze what constitutes a small gain
nor did it investigate how practitioners would use this
information. Further investigation in this area would be
useful to answer these questions.

Practitioners also appeared to agree that CBA has
demonstrated psychometric adequacy. Further research is
necessary to more clearly delineate what procedures CBA
users must follow to insure that their techniques are con-
sistent with those reported in the literature (e.g., Derr &
Shapiro, 1989).

Practitioners also slightly agreed with the statement
that teaching to the test is more easily accomplished with
CBA. This may not be a desirable attribute in every situa-
tion. Mehrens & Kaminski (1989), noted that if one wishes
to infer to a broader domain from a sample of items then
teaching too directly to the items or the specific sample of
objectives lowers the validity of the inference. Quite
often we use test information for making predictions about
the likelihood of success in future endeavors (e.g., college
aptitude tests, achievement tests for placement purposes,
etc.). In these situations we are more concerned about
obtaining an adequate sample of the subject's knowledge from

a broader domain, we are less concerned with how well the

110
subject has mastered enly the content of the test itself.
Practitioners need to consider when mastery of the test
content itself is worthwhile versus sampling knowledge from
a broader domain to which one wishes to infer.

The results also suggested that practitioners have not
reached consensus with regard to the limitations either.

The only limitation that received a rating of even slight
disagreement was the statement that CBA meets the special
education requirements for assessment. The present study
did not investigate what requirements practitioners perceive
CBA does not meet. Future research will be necessary to
answer this question as well. This result raised other
questions as well. Given the fact that only one critical
item was endorsed by the respondents, the possibility of a
response bias among the volunteer respondents should be con-
sidered. This is especially pertinent given the fact that
less than 50% of the practitioners responded. Opinions
among non-respondents might have differed from respondents
in the negative direction. Future research should include
additional efforts to attain higher response rates in order
to address this issue.

There was also some consensus regarding practitioners'
perceptions of CBA as an acceptable technique (mean rating =
5.16). The rating on this item suggested that practitioners
tended to slightly agree that CBA is an acceptable assess-
ment technique. What was perplexing was the evidence that

even this level of acceptability has been achieved despite

111

the fact that very few practitioners have actually used CBA.
The mean of the item stating "I have used CBA regularly" was
only 2.49. This rating fell within the range of "disagree".
It seemed that practitioners were using CBA infrequently, if
at all. Shapiro and Eckert (1993) found that actual use was
even lower than perceived use. The compelling question
then, is how are practitioners determining whether CBA is
acceptable? Additionally, how are they determining the
relative advantages or disadvantages? The vast literature
pertaining to CBA testifies to its popularity among writers
and researchers (e.g., Rosenfield & Shinn, 1989; Bagnato,
1986). Why is this popularity not manifested in practition-
er use? This finding was unexpected, given the favorable
statements made in the literature. More research is
necessary to determine the role of CBA in assessment as well
as to clarify how perceptions of CBA acceptability are
formed among practitioners. Future research should investi—
gate these issues. Studies should focus on delineating what
factors differentiate groups of practitioners who employ CBA
regularly versus those who do not. Additionally, groups who
perceive CBA to possess mostly advantages versus those who
perceive CBA to possess mostly disadvantages could be com-
pared.

The analysis of the results on the influence of demo-
graphic characteristics suggested that, with respect to the
demographics included in the study, demographics do not play

an important role in determining the acceptability of CBA.

112
Future research should attempt to replicate these findings
to rule out demographic effects. In addition, other demo—
graphics should be investigated. For example, geographic
region may influence acceptability. Studies employing a
national sample could be employed to investigate this possi-

bility.

SUMMARY AND RECOMMENDATIONS

This study has reviewed the research on CBA and CBM,
and the research dealing with treatment acceptability. At
present there is a paucity of research dealing with the
acceptability of CBA. The purpose of this dissertation was
to identify and investigate possible theoretical sources of
influence on practitioners' perceptions of CBA.

Authorities (Knutson & Shinn, 1991) have identified
three assessment models that they believe have provided the
conceptual underpinnings of CBM. The extent to which prac-
titioners agreed with the authorities that these models were
linked to CBA was investigated. Also, the extent to which
practitioners agreed with authorities that CBA demonstrated
the characteristics of these models was investigated. In
order to investigate these questions practitioners were
asked to endorse any or all of the models they perceive to
have provided a conceptual basis for CBA. In addition,
practitioners were asked to rate whether selected character-
istics of the three models were also characteristics of CBA.
The analysis of the results suggested practitioners seemed
unaware of these models, or at least their relationship to
CBA. Most practitioners (25% for the ecological, 34% for

the behavioral, and 45% for the data-based problem-solving)

113

114
failed to endorse the models as having conceptual ties to
CBA. The results of the analysis on the items representing
the conceptual models revealed that there was a tendency for
practitioners to slightly agree with the items representing
the ecological and behavioral models. However, practition-
ers appeared to neither agree nor disagree, or did not know
about the items representing the data-based problem solving
model. These results appeared to be consistent with the
relatively weak endorsement of the theoretical models. It
was suggested that practitioners may only slightly agree
with authorities regarding the conceptual basis of CBA or
that practitioners may be unaware of the theoretical sources
of the various characteristics.

This study also investigated the possibility that
factors identified as influencing the acceptability of
treatments also affect the acceptability of CBA. The re-
search on treatment acceptability was reviewed. A number of
factors that have been found to influence treatment accept-
ability were identified. These factors were then included
in the questionnaire on CBA acceptability. Practitioners
were asked to rate their level of agreement with the fac—
tors. These ratings were then correlated with their overall
ratings of CBA acceptability.

The majority of the factors identified as influencing
the acceptability of treatments were also found to be posi-
tively correlated with CBA acceptability at the p5.01 level.

Those factors included: effectiveness, cost, lower potential

115
for "stigmatizing labels," the perception of CBA as a more
positively focused assessment technique, CBA use, practi-
tioner knowledge, and utility for assessing a range of
handicapping conditions. These findings were consistent
with the results of the treatment acceptability research.

One finding that was not consistent with the treatment
acceptability research was that regarding the factor of
time. The correlation between this item and the acceptabil-
ity item was not statistically significant. One possible
explanation offered was that school psychology practitioners
did not associate time with the acceptability of a tech-
nique. That is, they may have believed that an assessment
technique is acceptable despite the fact that it may involve
a significant amount of time (e.g., an hour). Another
possibility offered was the existence of a curvilinear
relationship between acceptability and time.

This study also examined the degree to which practi-
tioners agreed or disagreed with the proposed advantages and
disadvantages of CBA reported in the CBA literature. A
review of the literature was utilized to identify a number
of purported advantages and disadvantages of CBA. Practi-
tioners' mean ratings of agreement with the statements of
purported advantage and disadvantage were examined to deter-
mine if subjects tended to agree or disagree with the state-
ments.

Taken together, the findings suggested that practitio-

ners are undecided about the majority of purported

116
advantages and disadvantages of CBA. The results did reveal
that practitioners had a tendency towards agreement in
perceiving that CBA provided more instructionally useful
data than traditional techniques. The present study did not
analyze what practitioners considered to be instructionally
useful data (e.g., designing, implementing, or evaluating)
and how CBA provides it. This needs to be addressed in
further research. However, if CBA effectively provides
information linking assessment to intervention, as practi-
tioners tended to believe, then this technique may prove
useful beyond an adjunctive role to school psychologists.
The results also revealed that practitioners showed a ten-
dency toward slight agreement in perceiving that CBA has
less potential for stigmatizing labels, that practitioners
had a tendency towards slight agreement with regard to CBA
being sensitive to small gains, that practitioners had a
tendency towards slight agreement on the demonstrated psy-
chometric adequacy of CBA, and that practitioners had a
tendency toward slight agreement that CBA lends itself to
teaching the test. It would be informative to investigate
these areas further. If CBA does not lead to stigmatizing
labels, is its usefulness for special education purposes
limited? Further research will be necessary to examine the
utility of CBA in meeting special education requirements.
It also appeared that practitioners slightly agreed that CBA
is sensitive to small gains. The present study did not

analyze what constitutes a small gain, if it is reliable,

117
and if it is educationally useful. Nor did it investigate
how practitioners would use this information. Further
investigation is necessary to answer these questions. The
belief that CBA lends itself more readily to teaching to the
test needs to be investigated. The present study did not
attempt to define when this is a desirable attribute.
However, consideration should be given to the domain of
knowledge to which one wishes to infer. It has been noted
that if one wishes to infer to a broader domain from a
sample of items then teaching too directly to the items or
the specific sample of objectives lowers the validity of the
inference (Mehrens & Kaminski, 1989). Teaching to the test
may be most desirable in those instances where the test
content is the identified domain. In other situations it
may be more appropriate to consider whether the test ade-
quately samples knowledge from a broader domain, thereby
increasing its utility in predicting performance in that
domain. Practitioners also slightly agreed that CBA has
demonstrated psychometric adequacy. Further research is
necessary to more clearly delineate what procedures CBA
users must follow to insure that their techniques are con-
sistent with those reported in the literature (e.g., Derr &
Shapiro, 1989). The results also suggested that practition—
ers have not reached consensus with regard to the limita-
tions either. The only limitation to receive a rating of
disagreement was the statement that CBA meets the special

education requirements for assessment. The present study

118
did not investigate what requirements practitioners perceive
CBA does not meet. Future research will be necessary to
answer this question as well. There was also some evidence
that practitioners lacked experience with CBA techniques.
It was concluded that there continues to be considerable
uncertainty with regard to the purported advantages and
disadvantages of CBA. The absence of strong beliefs pro or
con might indicate that practitioners are simply not ade-
quately informed at this time to make definitive judgements
regarding the value of CBA.

Finally, this research investigated a number of demo—
graphic variables for evidence of possible influence on,
acceptability. An ANOVA was conducted on the factors that
allowed meaningful groups to be constructed. The results
suggested that the theoretical perspective of the practi-
tioner was not related to the perception of CBA as an ac—
ceptable technique. It is possible that no relationship was
found due to the eclectic nature of school psychologists.
Sixty percent of the psychologists responding to the ques-
tionnaire considered themselves to be eclectic. The results
also suggested that there did not appear to be a relation-
ship between training institution and practitioner percep-
tion of CBA. Finally, the results suggested that the degree
held in school psychology was not related to practitioners'
perception of CBA as an acceptable technique.

In order to investigate the remainder of the demo-

graphic data, correlations were computed between the

119
acceptability item and the demographic data. The results
suggested that there was not a relationship between the date
the degree was received and the acceptability of CBA. The
results also suggested that there did not appear to be a
relationship between teaching experience and acceptability
of CBA. Also, the number of years as a school psychologist
was not related to the acceptability of CBA. Further, the
number of students expected to be assessed in the year of
the study was not related to perceptions of CBA acceptabili-
ty nor was there a relationship between the number of stu-
dents assessed the year prior to the study and perceptions
of CBA acceptability. Finally, the results suggested that
the percentage of time spent as a school psychologist was

not related to perceptions of CBA as an acceptable tech—

nique.

Witch

With regard to the findings on the conceptual under-
pinnings of CBA, the implications pointed to the need for
additional emphasis and education regarding the origins of
CBA techniques. Although CBA advocates have written exten—
sively about its use and merit, the minimal endorsement of
its theoretical foundations suggested a neglect of its theo-
retical origins. The need for clarification was noted to be
not limited to historical accuracy. Rather it was suggested
that to evaluate the utility and validity of any assessment

technique, it is necessary to consider the theoretical bases

120
upon which it was founded. In the case of CBA, it appears
that the utility of the technique may have preceded evalua—
tion of the theory or theories which have led to it. Care-
ful investigation of the bases of CBA may suggest important
considerations for practitioner use of the CBA model (e.g.,
the need to gather local norm data, standardized assessment
procedures, definition of what constitutes a problem or
handicap, etc.). The impact of such considerations upon the
use of CBA was noted as being unknown at this time. Howev-
er, possibilities included a more definitive definition of
CBA, a clearer delineation of its potential use and applica-
bility, and a more realistic analysis of its strengths and
limitations. Future research may wish to utilize an ana-
logue method similar to the one employed by Shapiro and
Eckert (1993). Detailed descriptions of the model (includ-
ing its label) could precede a case description employing an
assessment consistent with the model. It would be informa-
tive to investigate the impact of the description of the
model upon acceptability.

With respect to the findings regarding the factors
from the treatment acceptability research, the analysis of
the results suggested that much of that literature may apply
to the determination of CBA acceptability, and may general-
ize to assessment acceptability as well. The results sug-
gested that practitioner perceptions of factors identified
as important to treatment acceptability generalized to CBA

acceptability as well. These results were also perceived as

121
being consistent with other research on CBM acceptability
(Shapiro & Eckert, 1993). It seems worthwhile to explore
other areas of generalizability for the treatment accept-
ability factors. A possible candidate suggested was assess-
ment in general. The question was: can the acceptability of
assessment techniques be determined using the factors iden-
tified by the treatment acceptability research? Future
research should focus on investigating the generalizability
of the present findings to other forms of assessment. Such
information could prove informative to practitioners who are
responsible for choosing assessment techniques acceptable to
themselves as well as those who will be influenced by the
results (i.e., other school personnel, children, and par-
ents). Additionally, test manufacturers may wish to consid-
er the influence of these factors in designing assessment
techniques that are consistent with factors influencing
acceptability. Further, future research should also contin-
ue to investigate the possible existence of exceptions such
as the factor of time. The present study did not support
the generalization of the time factor to CBA acceptability.
Replication of this result in future research is necessary
before it can be determined that time does not influence the
acceptability of CBA or affects it in a curvilinear manner.
Important questions may be why practitioners do not feel
that less time-consuming techniques are more acceptable than
more time-consuming techniques or whether a range of

time/acceptability exists.

122

With respect to the findings regarding the reported
advantages and disadvantages of CBA, the analysis of the
results suggested that there was a lack of consensus among
practitioners with regard to the CBA literature. It ap-
peared that practitioners, as a group, mildly endorsed only
a few of the advantages that have been cited in the CBA
literature. Similarly, they have not reached consensus with
regard to the limitations either. Practitioners have ex-
pressed slight agreement with regard to whether CBA is an
acceptable technique. Even this limited level of accept-
ability has been achieved despite the fact that very few
practitioners have actually used CBA. Future research
should focus on the questions such as: how practitioners
determine whether CBA is acceptable, and how they determine
the relative advantages or disadvantages of CBA. Another
suggested area of investigation noted was why the apparent
popularity of CBA was not manifested in practitioner use.
More research is necessary to determine the role of CBA in
assessment as well as to clarify how perceptions of CBA
acceptability are formed among practitioners. Suggested
studies included investigating what factors differentiate
groups of practitioners who employ CBA regularly versus
those who do not and comparing groups who perceive CBA to
possess mostly advantages versus those who perceive CBA to
possess mostly disadvantages.

Finally, the analysis of the results on the influence

of demographic characteristics suggested that, with respect

123
to the demographics included in the study, demographics did
not play an important role in determining the acceptability
of CBA. Replication and extension of this study is neces-
sary to further investigate demographic variables. One

demographic variable worthy of study is geographic region.

I 1' . E E .

The lack of consensus with regard to the theoretical
foundations of CBA, and the apparent confusion or unaware-
ness regarding the theoretical sources of the characteris-
tics of CBA, suggested the need for additional educational
emphasis on the conceptual underpinnings of CBA. To appro-
priately analyze the strengths or weaknesses of an assess-
ment technique, the theoretical foundations of that tech-
nique should be considered. It appeared that practitioners
were not aware of the models that have purportedly provided
the conceptual premises of CBA techniques. In-service
education and pre-service education could provide practitio-
ners with the knowledge necessary to decide how to properly
use CBA. Additionally, this education would alert practi-
tioners to the limitations (Mehrens & Clarizio, 1993) and
considerations (Derr & Shapiro, 1989) that must be addressed
if one chooses to employ CBA.

The results regarding treatment acceptability also
suggested implications for practice. Support for the
generalization of the factors from the treatment acceptabil-

ity research to the acceptability of CBA suggested that

124
practitioners might begin considering these factors when
determining whether CBA is acceptable for them and their
clientele (e.g., children, parents, teachers, etc.). The
noted exception of the time factor suggested not only the
need for further study, but consideration on the part of
practitioners as to how time affects their determination of
whether an assessment technique is acceptable.

The lack of consensus among practitioners with regard
to the purported advantages and disadvantages of CBA also
suggested implications for practice. Perhaps foremost among
these was the apparent need for more information about CBA
at the practitioner level. It was not uncommon for respon-
dents to "apologize" for their lack of knowledge regarding
CBA. Some, in fact, noted they were unable to complete the
questionnaire because they were not sure they knew what CBA
was. The result of this apparent dearth of knowledge may
have been manifested in the lack of consensus observed.
There appeared to be a need for additional in-service or
other educational experiences designed for the practitioner.
The observation that practitioners did not appear to be
using CBA also suggested implications for practice. Those
practitioners experienced in the use of CBA may be called
upon to offer their expertise to other practitioners with
regard to when these techniques are appropriate and how they
should be employed.

It appeared that CBA was not as burning an issue among

practitioners as it was among academicians. It would seem

125
appropriate to conclude that, at this time, there continues
to be considerable tentativeness regarding the utility of
CBA among practitioners of school psychology. Concomitant
with that uncertainty is the need for additional training
and experience before definitive judgements regarding the
value of CBA can be made. Informed judgements can be made

only on the basis of knowledge.

APPENDICES

APPENDIX A

COVER LETTER

APPENDIX A

COVER LETTER

David J. Fugate

1409 M. Rundle

Lansing Michigan 48910
(517)-372-8983

Dear Michigan School Psychologist.

I am writing to request your assistance. As many of you are aware, there
continues to be controversy in our field with regard to what form assess-
ment can and should take. With that in mind,I have undertaken a research
project in order to find out what you, the school psychologists in the
state of Michigan, believe about various assessment issues. The major
focus of this effort is on your perceptions and beliefs about curriculum-
based assessment. The Michigan Department of Education (MDE) has generous-
ly agreed to sponsor this research.

Enclosed is a return envelope and a questionnaire. Please note that the
return envelope has been numerically coded. This number corresponds to the
MDE data-base used to generate the mailing list as well as to track
demographic information with regard to school psycnologists in the state.
The number on the envelope will be used to compare the demographic charac-
teristics of respondents with non-respondents. No attempt will be made by
the researcher to link questionnaires with names and there is_no identify-
ing information on the questionnaires. In addition, names will be removed
from the researcher's data base after the mailing list has been generated
in order to further safeguard your anonymity. Any information received
will be treated under the guidelines of the American Psychological Associa-
tion's Ethical Principles of Psychologists and code of conduct.

If you should desire to receive a summary of the study results, I unuld be
happy to send you a copy. In order to generate a mailing list for research
results. please complete the information below and simply include it with
your questionnaire. Again, mailing requests will be separated from
individual questionnaires immediately and the researcher will make no
attempt to match names with questionnaires. If you find this process
objectionable or inconvenient, you may request a copy by phoning me
directly at the number provided above or by writing to me. Be sure to
include your complete mailing address.

' ' Thank you for taking the time
to assist with this project.

 

 

 

Sincerely,
Please send me a copy of the
research summary: David J. Fugate,
Name (optional) School Psychologist
Street Address Michigan State University
City,Zip Doctoral Candidate

 

126

APPENDIX B

RESULTS LETTER AND SUMMARY STATISTICS

APPENDIX B

RESULTS LETTER AND SUMMARY STATISTICS

Dear Michigan School Psychologist:

Below you will find a summary of the results from the survey
on CBA that you completed in spring of 1993. The overall
results of this survey suggested that practitioners did not
feel they had much knowledge about or experience with, CBA
techniques. They also tended to only mildly endorse the .
majority of the items. This suggested that a strong consen-
sus had not been reached with regard to CBA among practitio-
ners. The lack of consensus may indicate that the strongest
advocates of CBA may be the academicians, not practicing
school psychologists.

If you have any questions or would like more detailed infor-
mation about the results, please do not hesitate to contact
me at the numbers listed below. Your cooperation in this

project was greebly appreciated!

Sincerely,

 

David Fugate

School Psychologist
(517)-651-2536
(517)-699-2099

127

1128

DEMOGRAPHIC DATA
Return rate: 448

Percent Endorsing Model as Having Conceptual Ties to CBA:-
Behavioral Model: 49, Ecological Model: 36.3, Data Based Prob-solve: 64.5

Theoretical Perspective of School Psychologists (percents):
Behavioral: 23.6, Cognitive: 10.9%, Eclectic: 60.6, Other: 2.3

College Attended (percents): MSG: 18.1, CMU: 21.8, 0 of M: 6.3
EMU: 7.2, HMO: 12.4, 0 of D: 8.0, Wayne: 10.3, Other: 14.7

Date Degree Received: Mean Year: 1979, Std dev: 7.4

Highest Degree Held: BA: .3, MA/MS: 32.5, 80.8.: 50.0, Ph.D: 14.7
Number Years as Teacher: Mean: 3.1 Std dev: 4.8

Number Years as School Psychologist: Mean: 13.5 Std dev 7.3
Number of Students Assessed This Year: Mean: 85.2 Std dev: 34.0
Number of Students Assessed Last Year: Mean: 80.3 Std dev: 33.4 .
Percent of Time as School Psychologist: Mean: 94.3 Std dev: 16.8
Percent of Time Assessing-Spec Ed: Mean: 44.0 Std dev: 8.6
Percent of Time Attending MET/IEP: Mean: 12.8 Std Dev: 7.3
Percent of Time Writing Results/Reports: Mean: 16.0 Std dev: 8.6
Percent of Time Consulting: Mean: 11.9 Std dev 9.4

Percent of Time Counseling: Mean: 3.7 Std dev 6.0

Percent of Time Educational Presentations: Mean 2.6 Std dev: 4.3
Percent of Time Behav Mod in Class: Mean 4.0 Std dev 6.0

Percent of Time Other: Mean 12.4 Std dev 16.5

Year Plan to Retire: Mean: 2007 Std dev 8.0

129

LEVEL OF ENDORSEMENT, MEAN RATINGS AND STANDARD DEVIATIONS
ON QUESTIONNAIRE ITEMS: Range 1-7

I TEM LEVEL S TD
ENDORSED MEAN DEV

CBA is acceptable assess. technique: SL-A 5.2 1.3
I have used CBA regularly: SL-D 2.5 1.5
CBA does not rely on national norms: A 5.5 1.4
CBA is an effective assess. technique: SL-A 5.1 1.3
CBA less apt to use labels: N/DK 4.6 1.5
CBA cost-effective: N/DK 4.2 1.4
CBA less culturally biased: N/DK 4.1 1.4
CBA not effective for range of handicap SL-A 4.5 1.5
CBA provides more socially valid meas.: N/DK 4.2 1.3
CBA takes more time for assess: N/DK 3.7 1.4
CBA emphasizes more positive approach: SL-A 4.8 1.4
Standardized administration not necessary: N/DK 3.6 1.5
CBA has high content validity: SL-A 5.2 1.3
Easier to construct local norms with CBA: SL-A 4.5 1.5
Freq. testing necess. for proper assess.: SL-A 4.6 1.4
CBA more likely to meet spec. ed. law: SL-D 3.1 1.4
CBA provides data for direct instruction: A 5.5 1.2
CBA not as sensitive to small gains: SL-A 5.1 1.4
CBA considers sit./environ. factors: SL-A 4.6 1.4
CBA less likely to gener. mult. interven.: SL-A 4.6 1.4
CBA uses quant. meas. of student behav.: SL-A 4.7 1.3
CBA def. prob. as envir./expec. discrep.: N/DK 4.2 1.5
I am knowledgeable with regard to CBA: N/DK 3.5 1.6
CBA prov. info lead. meaningful outcome: SL-A 5.1 1.1
CBA more app. to elem. than secondary: N/DK 3.6 1.4
Teaching to test more easily accomplished: SL-A 4.8 1.3

LEVEL OF ENDORSEMENT: SD=strongly disagree, D=disagree, SL-
D=slightly disagree, N/DK=neither agree nor disagree or
don't know, SL-A=slightly agree, A=agree, SA=strongly agree

DEFINITION OF INTERVALS: 1 to 1.49=strongly disagree, 1.5 to
2.49=disagree, 2.5 to 3.49=slightly disagree, 3.5 to
4.49=neither agree nor disagree or don't know, 4.5 to
5.49=slightly agree, 5.5 to 6.49=agree, 6.5 to 7=strongly
agree

LI ST OF REFERENCES

LIST OF REFERENCES

Babbie. E. (1989). Ihe_practice_of_social_research. Belmont.
CA: Wadsworth.

Bagnato, S. J. (1986). Curriculum-based assessment for the
young exceptional child: Rationale and review. Ten;

1cs_1n_Earlx_Ch1ldhood_Snecia1_Education. 6. 97-110.
Barlow, D. H. Hayes, S. C. & Nelson, R. O. (1984). The

. I O I
see s.ss ._ -s s

c11n1cal_and_educational_settinssi New YOrk: Pergam-

mon Press.

Blankenship, C. S. (1985). Using curriculum- based assessment'

to make instructional decisions. Exceptional_Chil;
dren. 52. 233- 238.

Blankenship, C.S., & Lilly, M.S. (1981). Mainstreaming

New York: Holt,
Rhinehart & Winston.

Borg, w.R., & Gall, M. (1983). Educational_research1__An
inbreduerien. New York: Longman.

Crocker. L.. & Algina, J. (1986). 1ntrodustion_to_class1cal
and_medern_reeb_rheery New York: Holt, Rinehart and

Winston.

Deno, S. (1985). Curriculum- based measurement: The emerging

alternative Exceptional_Ch11dren 52. 219-232.

Demo, S. (1986). Formative evaluation of individual student
programs: A new role for school psychologists.

School_Bsxcholog¥_Re11ew. 15. 358-374.

Deno, S. (1989). Curriculum-based measurement and special
education services: A fundamental and direct rela-

tionship. In M. R. Shinn (Ed. ), CurriQuJHm-baggd mea-
surement1_Assessing_spec1al_children (pp. 1-17). New

York: Guilford Press.

130

131

Demo, 8., Mirkin, P., & Shinn, M. (1979). Beberierel_per;

‘ ‘ so 4‘ q ‘ «‘0 s ‘4 o o- 0 2. ‘0 0° -
dren (Monograph No.12). Minneapolis, MN: University
of Minnesota, Institute for Research on Learning Dis-
abilities.

Derr, T., & Shapiro, S. (1989). A behavioral evaluation of
curriculum- based assessment of reading. Jeurnel_ef

Esxchoeducational_Assessment 1, 148-160.

Eaton, M., & Lovitt, T. (1972). Achievement tests versus
direct and daily measurement. In G. Semb (Ed.), Be-
haM1or_Anal¥sis_and_Education1 Lawrence. KS: Univer-

sity of Kansas.

Elliott, S. (1986). Children's ratings of classroom inter—
ventions for misbehavior: Findings and methodological

considerations. Journal_of_School_Bs¥cholog¥. 24. 23-
35.

Elliott, S., & Von Brock Treuting, M. (1991). The behavior
intervention rating scale: Development and validation.
of a pretreatment acceptability and effectiveness

measure. JQurnal_of_School_25¥sholos¥. 22. 43-51.

Elliott, S., Witt, J., Galvin, G., & Peterson, R. (1984).
Acceptability of positive and reductive behavioral
interventions: Factors that influence teachers' deci-

sions. Journa1_of_Sshool_Bsxcholog¥. 22. 353-360.

Frank, A. & Gerken, K. (1990). Case studies in curriculum-

based measurement Educat1on_and_Tra1n1ng_in__Meneal
Retardation 25, 113-119.

Fuchs, L., & Deno, S. (1992). Effects of curriculum within

curriculum- based measurement. Exeepr1enel_gh11dren,
53, 232- 242.

Fuchs, L., Fuchs, D., & Hamlett, C. (1990). Curriculum—
based measurement. A standardized, long- term goal
approach to monitoring student progress. Aendemie

Theranx 25. 615- 632.

Galagan, J. (1985). Psychoeducational testing: Turn out

the lights the party's over. Exeeptienal_flh11dren,
52, 288- 299.

Gickling, E., & Thompson, V. (1985). A personal view of

curriculum-based assessment. Exeept1enal_£h11dren,
52, 205-218.

132

Hayes, S.C., Nelson, R.O., & Jarrett, R. B. (1987). The
treatment utility of assessment: A functional ap—
proach to evaluating assessment quality. Amerieen

Bsxshologist 42. 963- 974.

Howell, K.W. (1986). Direct assessment of academic perfor-
mance. School_2s¥cholog¥_fie11ew. 15. 324-335.
Jenkins, J. R. Deno, S. L. & Mirkin, P. K. (1979). Measuring

pupil progress towards the least restrictive environ-

ment Learn1ng_Disah111t¥_Quarter1¥ 2. 81 92

Kazdin, A. (1977). Assessing the clinical or applied impor—
tance of behavior change through social validation.

Journal_of_Applied_8ehayior_Anal¥s1s 1, 427- 452.

Kazdin, A. (1980). Acceptability of time out from rein-
forcement procedures for disruptive child behavior.

Beharior_Iherap¥ 11, 329- 344.

Kazdin, A. (1981). Acceptability of child treatment tech-
niques: The influence of treatment efficacy and ad-

verse side effects. Bebexier_Iherepy,12, 493- 506.

Kazdin, A. (1982). The token economy. A decade later.
Journal_of_Applied_Beha119r_Anal¥sis 15. 331-346.

Kirgin, K., Braukmann, C., Atwater, J., & Wolf, M. (1982).
An evaluation of teaching— family (Achievement Place)

group homes for juvenile offenders. Jeurnel_ef_Ap;
plied_Behavior_Anal¥sis. 15. l- 16

Knutson, N., & Shinn, M. (1991). Curriculum-based measure-
ment: Conceptual underpinnings and integration into
problem—solving assessment. Jeurnal_ef_$ebeel_£ey;
QthQQY, 22. 371-393-

Lentz, F.E., & Shapiro, E.S. (1986). Functional assessment

of the academic environment. Sebeel_£eyehelegy_3e;
view, 15. 347-357.

Marston, D.B. (1989). A curriculum-based measurement
approach to assessing academic performance: What it
is and why do it. In Shinn, M. R. (Ed.), Currieulum;

° (pp.
18-78). New York: Guilford Press.

Mehrens, W., & Clarizio, H. (1993). Curriculum based
measurement: Conceptual and psychometric consider-

ations. Esxcholog¥_1n_the_Schools 30. 241- 254

133

Mehrens, W., & Kaminski, J. (1989). Methods for improving
standardized test score: Fruitful, fruitless or fraud-

ulent? Educat1onal_Measurementi__lssues_and_Eractice.
e, 14- 22.

Michigan Department of Education (1992). Ieehnieel_Menual.
Lansing, MI: Department of Education.

Miltenberger, R. (1990). Assessment of treatment accept-

ability. A review of the literature. Iep1ee_1n_Eerly
Ch1ldhood_Special_Education 10. 24— 38

Reimers, T., Wacker, D., & Koepl, G. (1987). Acceptability
of behavioral interventions: A review of the litera-

ture. School_Es¥cholog¥_Re11ew. 16. 212-227.

Rosenfield, S., & Shinn, M. (1989). Mini-series on Curri-
culum-based assessment. School_£s¥chology_8eriew. 18.
297-370.

Shapiro, E. (1987). Intervention research methodology in
school psychology. School_2sycholog¥_3eyiew. 16. 290-‘
305.

Shapiro. E., & Eckert. T. (1993). Acceptabilit¥_of_curricu;
1umehased_aasessment_b¥_school_ps¥cholog1sts. Manu-
script submitted for publication.

Shapiro, E., & Eckert, T. (in press). Knowledge, use, and
attitudes about curriculum based assessment among

school psychologists. Journal_of_Scthl_Es¥chology.

Shapiro, E., & Goldberg, R. (1986). A comparison of group
contingencies for increasing spelling performance
among sixth grade students. SehQQl_BS¥£th99¥_R§¥ifiHn
15, 546-557.

Shinn, M., & Hubbard, ‘D. (1992). Curriculum- based measure-
ment and problem solving assessment. Basic procedures

and outcomes. Eocus_on_Excsptional_Ch11dren 24. 1-
20.

Shinn, M. R. Nolet, V. & Knutson, N. (1990). Best practices
in curriculum- based measurement. In A. Thomas & J.

Grimes (Eds. ), Best_pract1ces_1n_school_ps¥chology_11
(pp. 287- 308). Washington, DC: National Association

of School Psychologists.

Shinn, M., Rosenfield, S., & Knutson, N. (1989). Curriculum
based assessment: A comparison of models. Seheel

Psychology_3e11ew. 18. 299-316.

134

Singh, N. & Katz, R. (1985). On the modification of
acceptability ratings for alternative child treat-

ments. Behaxior_Mod1fication. 9. 375- 386

Taylor, R., Willits, P., & Richards, S. (1988). Curriculum-
based assessment: Considerations and concerns. Ding;

nostique. 14. 14-21.

Tucker, J. (1985). Curriculum- based assessment: An intro-
duction. Exceptional.§h11dren. 52. 199- 204.
Von Brock, M., & Elliott, S. (1987). Influence of treatment

effectiveness information on the acceptability of

classroom interventions. Journa1_of_School_Bs¥cholQ;
SE. 25. 131-144.

Walle, D., Hobbs, S., & Caldwell, H. (1984). Sequencing of
parents training procedures. Effects on child noncom-
pliance and treatment acceptability. Beheyier_medifii;
cation. 8. 540- 552

Witt, J., Elliott, S.N., & Martens, B.K. (1984). Accept-
ability of behavioral interventions used in class-
rooms: The influences of amount of time, severity of
behavior problems, and type of intervention. Benny;

ioral_Disorders. 9, 95-104.

Witt, J., & Martens, B. (1983). Assessing the acceptability
of behavioral interventions used in classrooms. Rey;

cholog¥_1n_the_Schools. 20. 510- 517.

Wolf, M. (1978). Social validity: The case for subjective
measurement or how applied behavior analysis is find-

ing its heart. JQurnal_of_Anplied_Behayior_Ana1ysis.
11, 203- 214.

Ysseldyke, J., & Thurlow, M. (1984). Assessment practices
in special education: Adequacy and appropriateness.

Educational_25¥chologist, 9. 123-136.