PLACE N RETURN BOX to roman this chockout from your rocord.
TO AVOID FINES return on or bdoro duo duo.

DATE DUE DATE DUE DATE DUE

 

7-59)”- - W. I

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

MSU I. An Affirmative Action/Equal Opportunity Institution
mum-9.1

 

 

 

Evaluating the Efficacy of Computer Courseware
Designed to Support Concept Learning in College-Level Physiology

By

James Bolanle Obielodan

A Dissertation

Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of

DOCTOR OF PHILOSOPHY

Department of Counseling, Educational Psychology
and Special Education

1996

 

r I
raw an .,
dash-L .L .‘.L

‘W.’ ".3: ~'.-
Ana‘s“. '3;

.
'd'J-n-‘n
‘ b§»§‘u‘“

‘
.. 3‘5 a '3
"vui:§.‘§\~

“.|‘ 1:. .
M “‘P ~ - n—Ji

ABSTRACT

EVALUATING THE EFFICACY OF COMPUTER COURSEWARE
DESIGNED TO SUPPORT CONCEPT LEARNING
IN COLLEGE-LEVEL PHYSIOLOGY
By

James Bolanle Obielodan

Claims of benefits of instructional uses Of new technologies are Often based on
anecdotal statements from faculty and students rather than on empirical and objectively
measured data secured by educational research methods. This study was conducted to:
(1) determine the effects of using computer courseware programs, designed to teach
college-level physiology concepts, on students’ learning achievements; (2) assess the level
of importance of courseware characteristics in promoting effective learning as perceived
by experts; (3) assess the extent to which the quality Of courseware characteristics helped
in promoting effective learning as perceived by students and experts; and (4) construct a
framework for analyzing and evaluating computer-based multimedia courseware.

Exam grades for students in Physiology 431 course, offered at Michigan State
University during the Fall Semester of 1995, were collected and analyzed as a measure of
the effect of courseware programs on learning achievement. The students and two groups
of experts interacted with and evaluated the programs using evaluation instruments
constructed by the researcher. Data collected from the subjects were analyzed using

simple descriptive statistics, oneway analysis of variance (AN OVA), multidimensional

_,-. 0 ~-

.
L»-..b"
.

 

c.- l\ ..-
Hun“-

:2 ---‘

-. b. ‘i

 

u‘ ‘
'P 9
«I -£

k..

- -.- .
x, ""‘
~“~‘

scaling, and stepwise regression procedure. Their comments on the programs were
compiled, categorized, and analyzed.

The major findings of the study were: (1) students who used all three programs
significantly performed better than other students; however, gender and GPA were the
significant predictors of learning achievement; (2) only courseware 2, the program on
Control Systems, enhanced learning achievement. Interestingly, while the students rated
the courseware as least in quality, the experts rated it as best; (3) only students’ class
level and previous experience using courseware had a significant relationship to their
perceptions regarding the quality of courseware characteristics; (4) nearly all the students
liked the programs and expressed deep interest in using similar courseware in other
courses; (5) the experts rated “presenting complete, accurate, or current lesson content”
as the most important characteristic in promoting effective learning; (6) gender and age
were the demographic factors that differentiated the experts in their perceptions regarding
courseware characteristics; (7) while the experts were more stringent in rating the quality
of courseware characteristics, the students were more detailed as to the specific ways the
programs enhanced learning; and (8) while most of the students liked the way humor was
used in the programs, most of the experts did not.

In conclusion, “A Framework for Analyzing and Evaluating Computer-Based
Multimedia Courseware” was constructed based on the results and findings of the study

in light of established theoretical paradigms in educational psychology.

(pr

l. _
MM)

396

Copyright by

James Bolanle Obielodan

1996

DEDICATION

To the praise and glory of God,
in whom I live and move and have my being,
the source of all wisdom,
the perfect evaluator of life,

and the ultimate and righteous judge of all.

ACKNOWLEDGMENTS

My special gratitude to Dr. Norman Bell, my advisor and the chairperson of my
doctoral committee, for his enduring patience, valuable suggestions, professional
assistance, and guidance at every step of my graduate studies and doctoral research. His
trust and confidence in me to strive for excellence has made this study possible, for which
I am deeply honored and grateful.

Special thanks to other members of my committee, Dr. Leighton Price, Dr. Timothy
Little, and Dr. Kent Creswell, for their support, advice, and helpful comments with the
dissertation from the proposal stage to the final version. I am especially grateful to Dr.
Leighton Price for his painstaking attention to details, incisive comments, and the
personal trips he made to my office to encourage and shepherd me through my data
analysis. I am deeply honored by his friendship and grateful for his mentoring.

Words are inadequate to express my sincere and profound gratitude to Dr. Thomas
Adams, Professor of Physiology at Michigan State University, for enthusiastically
allowing me to use his courseware programs and the Physiology 431 class for the study.
His support, encouragement, and invaluable suggestions in conducting the study provides
the model by which I may always measure my achievement as a scholar and my humanity
toward those who may seek my help and guidance. Thanks to him for all his help.

Special thanks to the students in the Physiology 431 class, during 1995 Fall semester
at Michigan State University, and the experts who participated in the study for their time

and support. They meant much to me than words can describe.

vi

‘-Q

in---

‘3.

hh“

'o‘

 

 

Special thanks to Dr. Paul Hunt, Vice Provost of Libraries, Computing and
Technology and to my friends, colleagues, and the management of the Computer
Laboratory at Michigan State University, in particular, Dr. Lewis Greenberg, Diana
D’Angelo and, Robert Matson for their unqualified support, encouragement, and the
opportunities provided me at work, which, in many significant ways contributed to the
completion of this study.

Special thanks to Dr. Joshua Bagakas, Dr. Castelle G. Gentry, Dr. John Vinsonhaler,
Dr. Richard McLeod, Dr. Joe Byer, Dr. Ralph Putnam, Dr. Marie Emery, Dr. Stephen
Yelon, Dr. Randy Fotiu, Dr. Richard Houang, Dr. Leroy Olson, Dr. Craig Philips, Dr.
Kristine Morrissey, Dr. Mary Garrett, Dr. Mark Moallemian,Tory Sawyer, Mary Pern'n,
Lynn Anderson, Dr. Joachim Bayo Oludaja, Dr and Mrs. Usman Adarnu, Dr. and Mrs.
Stephen Abraham, Drs. Oluwadare and Remi Aworuwa, Dr. and Mrs. Samuel Oleka, and
Dr. and Mrs. Olufemi Fatunmbi for their friendship and the many special ways in which
their support and interaction encouraged me every step of the dissertation process.

Thanks to Mr. David Bolig and Nancy Lucas, of Michigan State University Libraries,
for going beyond the call of duties to assist me in obtaining important resources for my
research.

Sincere thanks and gratitude to my special family friends — Dr. James and Susan
McGillicuddy, Mr. Larry and Barbara McMullen, Dr. and Mrs. Arthur Rupprecht, Mr.
John and Becky Lillyman, Mr. and Mrs. Jim Flickinger, Mr. Richard and Vangi
DeMaster, Dr. Robert and Nancy Brown, and late Mr. Ross Sams, Jr. — whose
friendship, love, kindness, moral support, and prayers have always brought me hope,

comfort, courage, and inspiration to succeed.

vii

Very special thanks go to my mother, Serah Obielodan, my brother, Joseph Obielodan,
and other relatives for their unwavering love and support for me over the years. My
success in life is in large measure due to their confidence in my abilities and capabilities,
their moral and prayer support, and their enduring patience with me in my quest for
“encyclopedic” knowledge. Most especially, special and deep thanks to my uncle and his
wife, Michael and Lydia Obielodan, who, out of very little resources, raised me through
my early years. Their unqualified love and support in sponsoring me from elementary
through secondary education laid the foundation for my academic achievements and
spiritual development. Thanks to them for being my hero and a model example of hard
work, sacrificial love, and spiritual integrity.

Finally, there are many other fiiends and individuals who, although it is impossible to
mentioned them individually, nonetheless, have inspired, encouraged, and supported me
along the way, some consciously, some never knowing. Thanks to them for their love and

kindness. God bless.

viii

l_.o

uo~ ‘

win.

-5.- I

TABLE OF CONTENTS

LIST OF TABLES ...................................................................................................
LIST OF FIGURES .................................................................................................
ABBREVIATIONS .................................................................................................

CHAPTER I .............................................................................................................
INTRODUCTION ..................................................................................................
Statement of the Problem .............................................................................
Purpose of the Study ...................................................................................
Significance of the Study ..............................................................................
Research Questions ......................................................................................
Assumptions of the Study ...........................................................................
Theoretical Framework for the Study ..........................................................
Definition of Terms ......................................................................................
This Dissertation Plan ..................................................................................

CHAPTER II ...........................................................................................................
REVIEW OF THE LITERATURE .........................................................................
Introduction ..................................................................................................
Effective Teaching and Effective Leaming ...................................................
Technology in Education ..............................................................................
Computers as Information Processing Systems ...............................

Technological Horizons in Education ...............................................

Academic Uses of Computer Courseware .......................................

Cognitive Information Processing Theory ...................................................
Information Processing Components ...............................................

Gagné’s Theory of Instruction .........................................................

Evaluation .....................................................................................................
Definition of Evaluation ...................................................................

Function of Evaluation .....................................................................

Evaluation Models and Approaches ................................................

Computer Courseware Evaluation ...................................................
Multidimensional Scaling (MDS) Procedure ................................................

CHAPTER III ..........................................................................................................
RESEARCH DESIGN AND METHODOLOGY ..................................................
Introduction ..................................................................................................
Subjects ........................................................................................................
Instrumentation ............................................................................................
Questionnaire Distribution and Data Collection ..........................................

Data Analysis ...............................................................................................
Analyzing Data Collected from the Student Group .........................

ix

nOi

13
16
18

20
20
20
20
22
22
24
28
29
29
34
43
43
45
45
48
51

55
55
55
56
57
66
69
69

Analyzing Data of Respondents in the Expert Rating Form

Group ......................................................................................... 71
Analyzing Data of Respondents in the Expert Courseware
Evaluation Group ....................................................................... 74
Summary ...................................................................................................... 75
CHAPTER IV .......................................................................................................... 77
PRESENTATION AND ANALYSIS OF DATA FROM STUDENTS ............... 77
Introduction .................................................................................................. 77
Section One — Description of Students’ Demographics ............................. 78
Section Two —- The Effects of Computer Courseware on Students’
Learning Achievement ............................................................................. 84
Research Question Al ...................................................................... 84
Research Question A2 ...................................................................... 86
Section Three — Students’ Perceptions Concerning Quality of
Computer Courseware ............................................................................ 88
Research Question B1 ...................................................................... 89
Research Question B2 ...................................................................... 91
Research Question B3 ...................................................................... 103
Summary ...................................................................................................... 124
CHAPTER V ........................................................................................................... 126
PRESENTATION AND ANALYSIS OF DATA FROM EXPERTS ................... 126
Introduction .................................................................................................. 126
Section One — Description of Experts’ Demographics .............................. 126

Section Two — Perceptions of Respondents in the Expert Rating Form
Group Regarding the Level of Importance of the Characterisitcs of

Computer Courseware in Promoting Effective Learning .............................. 133
Research Question C1 ...................................................................... 134
Research Question C2 ...................................................................... 154
Research Question C3 ...................................................................... 158

Section Three — Perceptions of Respondents in the Expert Courseware
Evaluation Group Regarding the Quality of Computer Courseware

in promoting effective learning ................................................................ 160

Research Question D1 ...................................................................... 162

Research Question D2 ...................................................................... 164

Research Question D3 ...................................................................... 179

Summary ...................................................................................................... 188

CHAPTER VI .......................................................................................................... 189
SUMMARY, DISCUSSION, CONCLUSIONS, AND

RECOMMENDATIONS .................................................................................. 189

Summary of the Study ................................................................................. 190

Findings and Discussions ............................................................................. 193

Limitations of the Study .............................................................................. 219

Conclusions and Implications ...................................................................... 221

X

Recommendations ........................................................................................ 229
A Framework for Analyzing and Evaluating Computer-Based
Multimedia Courseware ................................................................... 231

APPENDIX A

Computer Courseware Evaluation Form ...................................................... 241
APPENDIX B

Computer Courseware Evaluation Checklist Rating Form .......................... 249
APPENDIX C

Computer Courseware Evaluation Checklist Form ...................................... 263
APPENDIX D

Letter requesting Dr. Thomas Adams’ permission to conduct the study

using his courseware programs and Physiology 431 class ........................... 280
APPENDIX E

Letter requesting students in the Physiology 431 class to participate in

the study ...................................................................................................... 282
APPENDIX F

Letter, email or phone messages to experts requesting them to participate

in the study .................................................................................................. 286
APPENDIX G

Cover letters that accompanied the evaluation instruments for the

experts .......................................................................................................... 291
APPENDIX H

Follow up messages to the experts .............................................................. 293
APPENDIX I

MDS coordinates of the experts’ perceptions regarding the level of

importance of the characteristics of computer courseware in promoting

effective learning ........................................................................................... 295
LIST OF REFERENCES ......................................................................................... 298

Table 2-1:

Table 3-1:

Table 3-2:

Table 4-1:

Table 4-2:

Table 4-3:

Table 4-4:

Table 4-5:

Table 4-6:

Table 4-7:

Table 4-8:

Table 4-9:

Table 4—10:

LIST OF TABLES

Events of instruction and their relation to processes of learning .....

Cronbach’s alpha reliability coefficient of the items on the

Computer Courseware Evaluation Form .........................................

Cronbach’s alpha reliability coefficient of the items on the

Computer Courseware Evaluation Checklist Rating Form. .............

The distribution of the student respondents by use of

courseware and participation in research questionnaire ..................

The distribution of students who used courseware programs
but did not participate in research questionnaire by

courseware programs used. ..............................................................

The distribution of the student respondents by use of
courseware and participation in responding to research

questionnaire ....................................................................................

The distribution of the student respondents by academic

discipline and gender ........................................................................

The distribution of the student respondents by GPA ....................

Number and percentage of the student respondents with

previous experience using computer productivity tools .................

The distribution of the student respondents by total credits

earned at MSU .................................................................................

One way analysis of variance (AN OVA) results of differences in
learning achievements (as measured by adjusted raw scores)
among students who used and those who did not use the

courseware programs .......................................................................

Regression results for the prediction of adjusted raw scores
(learning achievements) by the students demographic

characteristics ..................................................................................

Means, standard deviations, and ranks of the students’
perceptions regarding the quality of the characteristics of

sections of the courseware programs under study ..........................

xii

38

6O

65

79

79

8O

80

81

82

83

85

88

91

 

.3...»

 

2“:-
‘5‘.»

‘u
(I:
V In

Table 4-1 1:

Table 4-12:

Table 4-13:

Table 4-14:

Table 4-15:

Table 4-16:

Table 4-17:

Table 5-1:

Table 5-2:

Table 53:

Analysis of variance results of the student respondents’
perceptions of the quality of characteristics of the courseware
sections/composites in helping them to better learn Physiology
431 concepts by class level .............................................................

Analysis of variance results of the student respondents’
perceptions of the quality of characteristics of the courseware
sections/composites in helping them to better learn Physiology
431 concepts by previous experience with computers using
courseware .......................................................................................

Analysis of variance results of the student respondents’
perceptions of the quality of characteristics of the courseware
sections/composites in helping them to better learn Physiology
431 concepts by gender ...................................................................

Analysis of variance results of the student respondents’
perceptions of the quality of characteristics of the courseware
sections/composites in helping them to better learn Physiology
431 concepts by academic discipline ...............................................

Analysis of variance results of the student respondents’
perceptions of the quality of characteristics of the courseware
sections/composites in helping them to better learn Physiology
431 concepts by previous experience with computer

programming ....................................................................................

Analysis of variance results of the student respondents’
perceptions of the quality of characteristics of the courseware
sections/composites in helping them to better learn Physiology
431 concepts by computer ownership ............................................

Regression results of student respondents’ perceptions regarding
the quality of the characteristics of the courseware programs
under study in promoting effective learning ....................................

The distribution of respondents in the Expert Rating Form
Group by range of teaching experience in years ..............................

The distribution of respondents in the Experts Rating Form
Group by previous experience using computers as productivity
tools .................................................................................................

The distribution of respondents in the Expert Courseware
Evaluation Group by range of teaching experience in years ............

xiii

94

95

96

97

98

99

102

128

129

131

Table 5-4:

Table 5-5:

Table 5-6:

Table 5-7:

Table 5-8:

Table 5-9:

Table 5-10:

Table 5-11:

Table 5-12:

Table 5-13:

The distribution of respondents in the Expert Evaluation Group
by previous experience using computers as productivity tools ......

Means, standard deviations, and ranks of the importance of
characteristics of computer courseware that engender effective
learning. ............................................................................................

Means, standard deviations, and ranks of characteristics of
computer courseware sections .........................................................

Analysis of variance results of perception of respondents in the
Expert Rating Form Group regarding the level of importance of
the characteristics of composite sections of computer courseware
that can produce effective learning by gender ..................................

Analysis of variance results of perception of respondents in the
Expert Rating Form Group regarding the level of importance of
the characteristics of composite sections of computer courseware
that can produce effective learning by age in years .........................

Regression results of the perceptions of respondents in the
Expert Rating Form Group regarding the level of importance of
the characteristics of courseware in promoting effective learning

Means, standard deviations, and ranks of experts’ perceptions
regarding the quality of the characteristics of sections of the
courseware programs under study ...................................................

Analysis of variance results of the perceptions of respondents in
the Expert Courseware Evaluation Group regarding the quality of
the characteristics of the courseware sections/composites in
helping them to effectively learn the lesson material (Physiology
431 concepts) by gender ..................................................................

Analysis of variance results of the perceptions of respondents in
the Expert Courseware Evaluation Group regarding the quality of
the characteristics of the courseware sections/composites in
helping them to effectively learn the lesson material (Physiology
431 concepts) by age .......................................................................

Analysis of variance results of the perceptions of respondents in
the Expert Courseware Evaluation Group regarding the quality of
the characteristics of the courseware sections/composites in
helping them to effectively learn the lesson material (Physiology
431 concepts) by faculty with expertise in courseware
development ....................................................................................

xiv

132

136

140

156

157

158

163

166

167

168

o
3‘.
\d‘.

I‘-
N.‘

It;
nhi. '

‘I

\‘r

n
v,.

Table 5-14:

Table 5-15:

Table 5-16:

Table 5-17:

Table 5-18:

Table 5-19:

Table 5-20

Analysis of variance results of the perceptions of respondents in
the Expert Courseware Evaluation Group regarding the quality of
the characteristics of the courseware sections/composites in
helping them to effectively learn the lesson material (Physiology
431 concepts) by staff with expertise in courseware

development ....................................................................................

Analysis of variance results of the perceptions of respondents in
the Expert Courseware Evaluation Group regarding the quality of
the characteristics of the courseware sections/composites in
helping them to effectively learn the lesson material (Physiology
431 concepts) by expertise in content area .....................................

Analysis of variance results of the perceptions of respondents in
the Expert Courseware Evaluation Group regarding the quality of
the characteristics of the courseware sections/composites in
helping them to effectively learn the lesson material (Physiology
431 concepts) by teaching experience in years ................................

Analysis of variance results of the perceptions of respondents in
the Expert Courseware Evaluation Group regarding the quality of
the characteristics of the courseware sections/composites in
helping them to effectively learn the lesson material (Physiology
431 concepts) by approximate number of courseware programs

developed. ........................................................................................

Analysis of variance results of the perceptions of respondents in
the Expert Courseware Evaluation Group regarding the quality of
the characteristics of the courseware sections/composites in
helping them to effectively learn the lesson material (Physiology
431 concepts) by previous experience using computer
courseware and multimedia programs .............................................

Analysis of variance results of the perceptions of respondents in
the Expert Courseware Evaluation Group regarding the quality of
the characteristics of the courseware sections/composites in
helping them to effectively learn the lesson material (Physiology
431 concepts) by previous experience with computer

programming ....................................................................................

Analysis of variance results of the perceptions of respondents in
the Expert Courseware Evaluation Group regarding the quality of
the characteristics of the courseware sections/composites in
helping them to effectively learn the lesson material (Physiology
431 concepts) by previous experience assembling computers ........

XV

169

170

171

172

173

174

175

Table 5-21 Regression results of the perceptions of respondents in the
Expert Courseware Evaluation Group regarding the quality of the
characteristics of the courseware programs under study in
promoting effective learning ............................................................ 178

xvi

Figure 4-1

Figure 5-1

Figure 5-2

Figure 5-3

Figure 5-3(a)

Figure 5-3(b)

Figure 5-3(c)

Figure 5-3(d)

Figure 5-3(e)

Figure 5-3(f)

LIST OF FIGURES

The distribution of the student respondents by class level

The distribution of respondents in the Expert Rating Form
Group by approximate number of courseware programs
developed ................................................................................

The distribution of respondents in the Expert Courseware
Evaluation Group by approximate number of courseware

programs developed ................................................................

MDS of perceptions of the respondents in the Expert Rating
Form Group regarding the level of important of the
characteristics of computer courseware in promoting
effective learning. [Variable view of all the characteristics] .....

Enlarged view of the upper region of the three-dimensional
plot of the variables ................................................................

Enlarged view of the upper right region of the three-
dimensional plot of the variables ............................................

Enlarged view of the lower right region of the three—
dirnensional plot of the variables ............................................

Enlarged view of the lower region of the three-dimensional
plot of the variables ................................................................

Enlarged view of the lower left region of the three-
dirnensional plot of the variables ............................................

Enlarged view of the upper lefi region of the three-
dimensional plot of the variables ............................................

xvii

82

128

131

141

145

I47

148

149

151

152

"i [:1

L h...
A-

I

.2.

\ VAL

A-"~

\c-A u

v-9’L'
a- ~.~.

-u—
54‘s.

Abbreviations used in three-dimensional plots of the perceptions of respondents
in the Expert Rating Form Group regarding the level of importance of
the courseware characteristics in promoting effective learning.

Variables
CCSCRNL
CCSCRNE
CMENU
CGRAPH
CEPBEHA

CLOGORG
CICHUNK

CINTERA
CNAVIGT
CI-IELP
CAPNOVP
CGASITU
CSSCRNL
CLGFACT
CERSKIL

CNISIMP
CLBWREV

CAPCONC
CAPPSIM
CERSREV
CNIACCU
COBJCON
CLGSKIL
CUSEHOW

CCLAOBJ
CISTRAT

Characteristics

Consistency of the screen layout and elements (graphics, color, button,
text, etc.

Coherence among screen elements or components (background,
foreground, pages, navigational tools, etc.)

The use of MENU to access information

The use of graphics in enhancing learning of the course materials
Provides opportunity for the learner to practice making choices related
to the attitude learned

The logical organization of lesson content

The amount (chunk) of information presented on each screen or in each
lesson unit

The level or degree of interactivity the learner has with the lesson
materials (opportunity for the learner to participate)

The use of courseware navigational tools (buttons, hotwords, etc.)
The use of HELP function to get context sensitive information

Tests the learner on solving a variety of novel or unique problems
Introduction focuses learner attention on the learning situation or context
Simplicity of the screen layout and elements (graphics, color, button,
etc.)

Provides instruction to guide the learner in linking simple to more
complex lesson content

Provides additional opportunities for the learner to practice the skills
learned

Presents information in simple and concise statements

Review of relevant content (prerequisite skills) at the beginning of the
lesson .

Tests the learner on applying concepts and rules to new situations or
examples

Tests the learner on making choice of action in a real or simulated
situation

The use of spaced reviews including a variety of examples throughout
the lesson

Presents complete, accurate, or current lesson materials

Match between courseware objectives and lesson content

Requires the learner to practice the skills learned and provides feedback
on the learner’s performance

Directions on how to use the courseware

Statement of objectives

The use of instructional strategies (e.g., hints, examples, illustrations,

xviii

crasroa
CFEE'JBK

Ff‘t'r!

(CANTER
or ’N '
;LCO\C

CLFACT
CZEROB

WWII

CEPPRC‘B
CPFFACT

CPFCOXC

P' l“- .
L.CU.\TR
R" “r-

ucfiLIJNC
(firm-us,
.3351

CPFCOXS
N F

»~:CO.\'C
~51:.iCT

”1“\
grim

‘
. Va.

CTESTOB
CF EEDBK
CGATASK
CILCONC

CILFACT
CILPROB

CNIVARY
CEPPROB

CPFFACT
CPFCONC
CPFPROB
CREVCON
CREVQUE
CLCONTR
CSRCONC
CNISKIL
CPFCONS
CLGCONC
CSRF ACT
CSRSKIL

CINTAUD
CPFSKJL

.CEPSKIL

CSRESUB

CLGPROB

CAPSKIL
CNIBEHA

analogies, flowcharts, outlines)

Match between performance tests and lesson Objectives

Feedback to students responses

Introduction focuses learner attention on the tasks and skills to be
learned

Describes and demonstrates concepts, principles, and procedures to be
learned

States what the learner will be able to say or do after the lesson
Clarifies the general nature of problem solutions expected from the
leamer

Presents varied examples of concepts and rules or principles

Provides a variety of opportunities for the learner to practice or use
cognitive strategies learned

Confirms the correctness of learner’s response to question about
statements of facts

Confirms the correctness of learner’s use or application of concepts and
rules

Confirms the correctness of learner’s solution to problems presented
Review of the main concepts and principles at the end of the lesson
Review questions to clarify the lesson content

Learner control over the rate of information presentation

Stimulates recall of prerequisite concepts and principles or rules
Presents instruction on and demonstrates proper steps required for the
motor skill performance desired

Shows direct consequences of personal actions taken or decisions made
by the learner

Provides instruction to guide the learner in following the proper
sequence of steps to master the lesson content

Stimulates recall of relevant facts and context that will make the learning
tasks or processes meaningful

Stimulates recall of skills, situations and actions that will be involved in
the learner making personal choices or decisions

Identification of intended audience

Provides feedback on the degree of accuracy and timing of the learner’s
performance of skills

Provides opportunity for the learner to practice performing all physical
skills or tasks taught in the lesson

Stimulates recall of subskills (part skills) and the process of integrating
subskills (executive subroutine) neccesary to learn the performance of
the total motor skills desired

Provides prompts and hints to help the learner find novel solution to
stated problem

Tests the learner on using all physical skills taught in the lesson
Presents information to be learned about desired behaviors or attitudes
by modeling or illustrating actions and consequences related to the
behaviors or attitudes

xix

an “I"!
I I .
\‘kk \'

ranfi

sink-’9‘“

ARR“ PI

LEEJLI

”v ~fI'V

V...“

A. v-‘ .“D
U--\
fin I-v y

Uo~&-‘ L-

WAAA

..L§U

F :A')‘,

5‘5 Us "bJK

“"111 '
Luv. l“ L

CE?CC)\

rho-h.“ A1

I
“Luau
‘1 'V'MI

1\‘t'

V'ldg.

.diCRVL

Ann... .

-.,_ .
hrs. “L1

LLTCO
:2;D§KS
:fiBCO‘

1“TA“...

:L~U\\%

ixfszz
its:

”FA

H~U\S

CRCCONT
CLGBEHA

CSRBEHA
CILSKIL

CLEVDIF
CILBEHA

CNICOGS

CPFORIG
CCOMPLX
CEPCONC

CCERROR
CVISUAL
CBSCRNL

CSEFECT

CTXTCOL
CHLINKS
CTABCON
CGLOSSA

CTXTSIZ
CTXTSTY

. CICONS
CANIMAT
CVIDEO
CAUDIO
CNIPROB
CEPF ACT
CAPF ACT

Statement or demonstration of the courseware relevance to the learner
Describes or demonstrates the desired behavior or attitude by
simulating, modeling, or illustrating actions and consequences related to
the behavior or attitude

Stimulates recall of skills, by modeling or illustration, that the learner can
imitate in learning new behaviors or attitudes

Demonstrates performance of motor (physical) skills expected of the
leamer

The level of difficulty of lesson materials for the intended audience
Provides example of the kind of behavior or attitude (action choice)
expected of the learner

Describes and shows the use of mental strategies or procedures to solve
problems (e.g., strategy of asking proper questions; a strategy to “work
the problem in steps backward from the goal”)

Confirms originality of learner’s solution to problems presented

The complexity of lesson structure for the intended audience

Asks the learner to apply concepts and rules to instances not previously
encountered in the lesson

The ease of correcting errors (e.g., typing errors)

General visual appeal of the overall screen design

Balance in the screen layout and elements (graphics, color, button, text,
etc.)

The use of special effects (text animation, highlighting, transition effects,
etc.)

The use of color to enhance the readability of text

The use of branching or hyperlinks

The use of INDEX or TABLE OF CONTENT to access information
The availability of GLOSSARY to provide meaning of terms or
additional information

The use of text sizes to enhance the readability of text

The use of text styles to enhance the readability of text

The use of icons to communicate meaning or expected action

The use of animations in enhancing learning of the course materials

The use of video in enhancing learning of the course materials

The use of audio in enhancing learning of the course materials

Presents novel problems for the learner to solve .

Asks the learner for information in his/her own words

Tests the learner on restating lesson information in his or her own words

Note: Wherever the first letter C of an abbreviation is followed by l, 2, or 3, the
abbreviation signifies a component characteristic of courseware 1, courseware 2, or
courseware 3 respectively.

-- .-‘ .:
neg} .

5“.” '3‘ ‘
Justin») On

“.'Q '9)v _‘\
A-.. .15. 50’.

“-0 Ham.» .0
1 . ar-
.. by“... not»

C

“lit-Yam p?-
n...‘.... HQ}

"2-1.1 241...;
'hw‘t *aufi

CHAPTER I

INTRODUCTION

Statement of the Problem

The jury is still out among researchers and experts on the characteristics and objective
measures of successful computer courseware. For the purpose of this study, successful
computer courseware refers to any computer-based educational material (computer-based
or computer-assisted instructional program with or without interactive audiovisual media
components) that produces specified educational outcomes including learning motivation,
learning achievement, and transfer of knowledge skills in effective and efficient ways.

Although the emerging technologies have been much heralded for their sophisticated
capabilities to provide effective instruction (Bosco, 1986; Cronin & Cronin, 1992;
DeBloois, 1988; Fletcher, 1990; Gayeski & Williams, 1985; Gold, 1989; Kalowski, 1987;
Kearsley & Frost, 1985; McNeil and Nelson, 1991; Smith, 1987), claims of benefits from
academic uses of new technologies are often based on anecdotal statements from faculty
and students rather than on evaluation data secured by educational research methods
(Boettcher, 1993). The lack of empirical, or objectively measured data, on the success, or
efficacy, of computer courseware has generated a number of important issues and
questions concerning the need for and viability of academic uses of computer-related
technology systems (Hannafin, 1985; Boettcher, 1993). For instance, educational

psychologists are concerned about the kinds of knowledge and knowledge representations

”H: L'
'I an

us“

.. .5! ngfl

Ll bUaibho ..

0 ‘~ '
o. s“ Q
>- ".‘ ‘ 01‘]

L’s-.o-‘_~
I»

.32...” E.
I‘su-AB' ~
~

-7921- T
M’ .1‘3. A

or ‘9'.” :9” 4.

n- »&.m .1st

u g _
t4... , _.

4 w- a
9. \ 1 'l
‘bxi' I. c L,

.. Q-'-
.ri’. 4 ‘ ‘\‘
sea...“-

~ I
“I"; .3; - F
nun-”ask, f“

:..‘21‘3~9".
-~. ~“‘5A$.

3‘ ‘3!»-

."‘§‘..¢‘ are

2

for which emerging technologies are most suitable. A closely related issue is the concern
about the kinds of learning processes that emerging technologies best facilitate. There is
also concern about the extent to which computer courseware incorporates research
findings on instructional and learning theories about effective instruction and effective
learning. Educators want to know how students learn from computer-based educational
materials. They also want to know the cognitive effects that emerging technologies have
on both students and faculty. Courseware developers are interested in knowing the
factors that influence the effective and efficient utilization of the full instructionai
potentials of emerging technologies. Educational researchers and psychometricians are
concerned about how to authentically measure or assess students' learning and
achievement from using computer courseware. And administrators, publishers, and even
students, are interested in knowing the worth of computer courseware.

The lack of empirical data on the effects of technology on learning was clearly noted in
the Joe Wyatt project (Boettcher, 1993), the most seminal work to-date on the benefits of
the use of information technology. As Boettcher (1993, p. 32), the Committee Chair of
the Joe Wyatt project, stated:

The committee accepted a wide range of type of data—including anecdotal
statements from faculty and students—as evidence of benefit. The criteria for
documenting the success of the story were straightforward, but not rigorous,
despite a strong research orientation by the committee. Many of the projects

measured student and faculty attitudes toward the use of the computer, rather
than examining quantitative and qualitative measures of the effect of technology on

learning.

o t

.- ..,o 1«Io-3 p
'...-H\ s

t-

- I
. V
Each} 11'. L1

22 33‘. 2'16 .1:

'w
pt.
.
H‘ --‘u
had. no.

I
’1‘».

‘U‘ i L.A

131” ~ I
5&- AAAA

?

't n‘
a tie III
m: I
In». . y .

“31“.
0.45,.
o

3

The paucity of traditional research-based data on the effectiveness of new information
technologies may be attributed to the difficulty in applying "traditional requirements of
educational research" to the instructional uses of the new technologies. The difficulty lies
basically in the inappropriateness of traditional research methods as a means of securing
data on the usefulness of information technology to provide effective instruction. In the
words of (Boettcher, 1993):

prominent research measures in use for most of the past fifiy years reflected,
and appropriately so, the characteristics of a curriculum and its associated
teaching and learning resources. The characteristics of these new teaching and
learning strategies and resources are different from those available in the past.
The new resources provide much richer environments; they engage students much
more interactively; the outcomes are less rigid and often can be only generally
anticipated. The new learning tools, such as computer-assisted design tools and
exploratory data analysis, and the new stimulated multimedia environments
encourage and enable students to become actively engaged with interdisciplinary
concepts, content, and relationships. The new learning tools require decision
making and problem solving and, ofien, the actual production of designs and
products.

Evaluation measures have ofien focused on testing students' knowledge of
factual dates, times, and places, as well as static concepts and situations. Instead,
we need to be testing students' understanding of relationships, links, dynamic
processes, and strategies.

Once information technology transforms a course from a lecture class to a
laboratory class and revises objectives, strategies, and content, then applying
traditional evaluation methods becomes difficult. New and more powerful
research design strategies and evaluative measures are needed to identify and
capture data on the generative, interactive environments and on the generative
constructionist learning that we believe is occurring.

Boettcher's assertions, in the Joe Wyatt project report, clearly present four issues of
interest for further research on the academic uses of computer-related technology. The

first issue is to determine the kind of educational goals that faculty seek to achieve in

developing and/or using computer courseware. The second issue is to determine what the

 

3 ‘4
nib . b...-

'-: ~¢ .3‘
.u— ¢bL§

' Q
‘0: §~.§,
uab .“ _,

'DI“
. f-

a:
I

4

effects of computer courseware are on teaching and learning processes. The effects may
be specified or incidental, they may be enhanced knowledge and comprehension skills,
they may be generative or high order cognitive skills (e.g., application, analysis, synthesis,
and evaluation), and they may be quantifiable or may only be described. The third issue is
to establish how the efficacy, or benefits, of computer courseware on teaching and
learning processes can be measured. A closely related issue is concerned with the validity
and reliability of the measures of effects obtained. The fourth issue is to determine the
characteristics of successful computer courseware. This study is concerned about all but

the first issue.

Purpose of the Study
This study was conducted to investigate factors and assess characteristics that

determine the success or efficacy of computer courseware designed to support students’
learning of college-level physiology concepts. The study serves four purposes. The first
purpose was to determine the effects of using computer courseware programs designed to
teach college-level physiology concepts on students’ learning achievement. The second
purpose was to assess the level of importance of computer courseware characteristics in
promoting effective learning as perceived by experts. The third purpose was to assess
the impact of the quality of computer courseware characteristics in promoting effective
learning as perceived by both students and experts. The fourth purpose was to construct

an evaluation framework that courseware developers, instructional designers, researchers,

5

project managers, curriculum designers, educators, administrators, program evaluators,
and manufacturers of software programs can use to analyze and assess computer

courseware programs for the characteristics of instructional quality or efficacy.

Significance of the Study

The study makes a fundamental assumption that the success, or efficacy, of computer-
based educational materials depends in large part on the application of instructional and
learning theories to the development of computer courseware. Yet, many courseware
programs produced and used today for classroom instruction, including college-level
academic instruction, are developed based on intuitive hunches rather than on
systematically documented, research-based instructional and learning theories. It must be
underscored, of course, that in most cases the hunches come from many years of valuable
classroom teaching experiences. The challenge posed by academic uses of non-theory
based courseware programs is to understand the level of success of such programs, as well
as the factors and characteristics that make them successful in promoting effective
learning. One major importance of this study, therefore, is in constructing a framework
for developing and evaluating successful courseware programs from the findings of the
study. The framework will serve as a guide for courseware developers to ensure that their
courseware programs fully incorporate research-based findings on the characteristics of
effective instruction. It will also serve as a checklist for educators and anyone else

concerned about selecting sound, quality courseware.

.0“.

\

\O‘-.-
u

.
'3". " ’3 T
”0.. *‘

-,~¢ \

.‘ykbb. .2!

an ,-
in ' ‘
my}.
.

“‘ V..-
,M -

\

.2 u '
‘ :- his..-

 

6

This study extends the work accomplished on effective teaching in the classroom
setting to the electronic teaching and learning environments which offer capabilities to
teach and learn in ways not possible through conventional teaching (Boettcher, 1993, pp.
3233). Like research on effective teaching in the classroom, this study was also
concerned with the characteristics of effective instruction. However, the scope of this
study went beyond research on effective instruction per se. In addition to assessing
instructional activities and presentation of intellectual skills, which constitute the primary
focus of research on effective teaching in the classroom, this study sought also to examine
how unique pedagogical content and strategies are represented in computer courseware
programs to promote effective learning. Another importance of the study, therefore, is
gaining useful insights on the kinds of unique learning capabilities and effects that are

made possible by the unique capabilities of computer-related technologies.

Research Questions

The research questions that guided this study were based on the purpose of the study.
The main purpose was to investigate factors and assess characteristics that determine the
success of computer courseware and to construct a fi'amework for evaluating the efficacy
or instructional quality of computer courseware programs. Several major sources of
information influenced the formulation of the research questions, namely: literature

review; my graduate classes in educational psychology; my nine years of experience to-

if: .s 1.3 [0

\a; J" '1'; 4’)
.0,...' H“; U5

I. 1
Iowa-'- Air-4 4'
“ A A :u
inn-EU“ he}

I I‘
I, «0’4 ‘l 4 -.
aim-11b MIX
I l .
*1 --~ - -
22's....lrt‘n. In
‘ VI

1 : I ‘ ’
LR-Wj Cami

"‘30? so

“Hib bl n “I:

73: :eseac

“-33:11

#1
s:
.1" ’
I

7
date as a Hypermedia Consultant at Michigan State University developing and working

with faculty to develop and use computer courseware; informal discussions with various
courseware developers and experts at various professional conferences (Society for
Applied Learning Technology, Consortium for Inter-Institutional Collaboration In
African and Latin American Studies, etc.); my interaction with various electronic listservs
devoted to discussions on issues relating to emerging technologies and educational
psychology, including: EdTech listserv, American Educational Research Association
(AERA) Curriculum listserv, and Problem-Based Learning (PBL) listserv; and my
interaction with various World Wide Web (WWW) sites.

The research questions that guided the study were:

Broad uestions

A. What are the effects of computer courseware on students’ learning achievement?

B. What are the student respondents’ perceptions regarding the quality of the
characteristics of computer courseware programs used in this study?

C. What computer courseware characteristics do experts perceive as important in
promoting effective learning?

D. What are the experts’ perceptions regarding the quality of the characteristics of

computer courseware programs used in this study?

Specific 1

.- .
5f ”‘2'." I
H; “nub“

".5 '1‘)".
.2. .sxh‘a 5

I
Au-
5..

Specific Questions
The main questions were subdivided into specific questions to further define the scope of

the research. The subsidiary questions are presented under each main question below.

A. What are the effects of computer cOurseware on students’ learning achievement?
Research Question A1: Does courseware usage predict the students’ level of
learning achievement?

Research Question A2: To what extent is the students’ level of learning
achievement dependent on factors other than the use of the computer courseware,
for instance, the student respondents’ demographic characteristics (gender, age,
GPA, academic discipline, class level, previous experience with computers, and

computer ownership)?

B. What are the student respondents’ perceptions regarding the quality of the
characteristics of computer courseware programs used in this study?
Research mestion B1: What are the student respondents’ perceptions regarding
the quality of the characteristics of the computer courseware programs used in
this study in promoting effective leaming?
Research Question B2: Do the student respondents’ perceptions regarding the
quality of the computer courseware characteristics vary by their demographic
characteristics such as: gender, class level, academic discipline, previous experience

using computer productivity tools, previous experience using computer

9

courseware, previous experience with computer programming, or computer
ownership?
Research Question B3: What other features of the courseware programs do

student respondents perceive as unique and helpful in promoting effective

learning?

What computer courseware characteristics do experts perceive as important in
promoting effective learning?

Research Question C1: What are the experts’ perceptions regarding the level of
importance of computer courseware characteristics in promoting effective
learning?

Research Question C2: Do the experts’ perceptions regarding the level of
importance of computer courseware characteristics in promoting effective learning
vary by their demographic characteristics such as: gender, age, professional
discipline, teaching experience, courseware development experience, experience
using computer productivity tools, and computer ownership?

Research Question C3: What other features do experts perceive as important to

the success of computer courseware in promoting effective learning?

10
D. What are the experts’ perceptions regarding the quality of the characteristics of

computer courseware programs used in this study?

Research Question D1: What are the experts’ perceptions regarding the quality of
the characteristics of the computer courseware programs used in this study in
promoting effective learning?

Research Question D2: Do the experts’ perceptions regarding the quality of the
computer courseware characteristics vary by their demographic characteristics
such as gender, age, professional discipline, teaching experience, courseware
development experience, experience using computer productivity tools, and
computer ownership?

Research Question D3: What other features of the courseware programs used in
this study do experts perceive as unique and helpful in promoting effective

learning?

Assumptions of the Study
An increasing phenomenon in educational practice today is the integration of computer
courseware into classroom instruction by the same instructor who developed the
program. There is no question that the instructors are experts in their academic
disciplines. They may or may not be well grounded in instructional and learning theories,
however. The challenge in this situation, as in any situation in which computer
courseware is integrated into classroom instruction, is determining the success of academic

uses of courseware programs developed by the same instructor who teaches the class. A

. - 1-
, $1.2 rend

- s
..¥

.1“- mji’b '
J. nut-55 ~

A

Uit’pt:

I
q' ~ ‘."‘~ "’4
LA OLA-up 3'.
A

~"‘. oh 3".
.2“ punk...

t .
art-flu! ‘
uni—abs; as

‘ 2“‘\,‘\
untia‘cini.
”fly“- ‘

. 4 .
~-~-.I> \“
‘ V

”fi'q 4» k-
;““‘“": L‘-

“I
"9‘- u p
I .16 .5de

9‘5

*w.
' I
my}, ”a“:

:5; 2"}; 9
Nu.. to

I.-.: .
, 9 'H

“4446‘

3. ,
91:53,. 3.1 Ga .
. 0.5.‘5‘~

IA-
‘ '..=

33-}

\

‘V in.
§.\L.‘~'3%1

11

closely related and heretofore elusive challenge is determining factors or characteristics
that make computer courseware programs successful in promoting effective learning.

One possible solution to the problem of understanding factors that affect the success
of computer courseware programs is to examine the programs in light of research-based
instructional design theories. Most instructional design theories are derived from a
plethora of research findings, especially the results of effectiveness studies in the
classroom. This study assumed, therefore, that successful computer courseware
programs will at least manifest characteristics of effective instruction and effective
leaming based on findings from research in the classroom.

The study assumed also that different courseware programs will, in varying degrees,
incorporate sound instructional design theories and principles. The differences are
assumed to be due to a combination of differences in courseware development experience
and in the knowledge of instructional theories and design principles. Thus the more
experienced the courseware developers, the higher the degree to which their courseware
programs will manifest characteristics that promote effective instruction and learning and,
therefore, the more likely it may be that the programs will succeed in promoting effective
learning.

From the foregoing discussions, it is assumed that the different courseware programs
used in this study will manifest different levels of characteristics of effective instruction
and effective learning and, therefore, will differentially affect students’ levels of leaming
achievements. In this case, the differences are assumed to be due to progression in the

developer’s experience in courseware development.

12

Another assumption of the study was that students will pay careful attention to
courseware programs and gain significant learning achievements only if the courseware
programs are fully integrated into academic curriculum. Thus, even when a program is
well designed, consistent with instructional design principles, the full potential of the
program to engender effective learning may not be fully realized if the program is made
available only as an option.

The study assumed also that the students taking the course in which the computer
courseware programs are used are fairly homogenous with respect to academic abilities.
The students have fairly equivalent educational experience.

More importantly, the study assumed that the extent to which computer courseware
succeeds in promoting effective student learning processes depends in large part on the
degree to which the programs provide cognitive tools that challenge and allow the learner
to find solutions to real life problems and to construct new knowledge. Only by fully
integrating computers as tools for problem solving and knowledge construction into
every aspect of an academic curriculum will there be any significant impact of computers
on “meaningful learning” and academic achievement.

In sum, this study makes a fundamental assumption that the success of computer
courseware programs in promoting learning achievement is a function of several factors
including: the pedagogical content of the courseware programs, pedagogical strategies used
in the courseware programs, how the programs are used, environmental variables that

conditioned the use of the programs, characteristics of the learners, and the components

I
O

b \v.»

F...)

‘fi.

I“

13

of the programs that interact with learners’ characteristics and environmental variables to
produce the Observed effects.

Finally, the study assumed that Gagné’s theory of instruction and multidimensional
scaling procedure are appropriate theoretical framework and methodology to study the
extent to which research-based findings about the characteristics of effective instruction
and effective learning are incorporated into computer courseware programs. The
combination of both the theoretical framework and the procedure will provide non-biased
portrayal or representation of the content and potential effects of computer courseware.

These assumptions form the underlying rationale and thrust for this study. The
justifications for the assumptions will be established in the section for theoretical

framework and under literature review in the following chapter.

Theoretical Framework for the Study

Computer courseware is essentially an information processing system. As such, a
meaningful understanding of the factors that influence the success of computer
courseware programs requires an information processing theory-based framework for
conceptualizing effective teaching and effective learning with computer courseware.
Gagne’s theory of instruction, based on cognitive information processing theory, provides
a versatile theoretical framework for conceptualizing the psychological processes, or
factors, that influence the success of courseware programs as instructional, as well as

information systems. The theory provides a wealth of empirical research-based

u“...-

 

.Nl

a” ‘5‘

\
,.'.'ss.
1

a.) an.
X»; n

1 . -
.

I

ha a
7" ‘1“-

.‘4 we -.

\.
i‘

.3- .
~~-.S;
I‘

a'-..
~

'st.

'3'

n-.
h' -‘A
\‘2,‘..:]
"5'
A“ 3“ _
K C
‘-a
.l.
“at m
. u,
s I.
K ““
'I.
a, a

14

information for constructing an evaluation framework for deciphering components of the
courseware programs that impact on the success of the information systems.

The cognitive information processing theory holds that learning is an active mental
process of acquiring, remembering, and using knowledge. The process involves attention,
selective perception, rehearsal, encoding, and retrieval of information that when properly
orchestrated results in meaningful retention of information (Woolfolk, 1995; Travers et
al., 1993). Meaningful retention or learning occurs when people obtain and process
information in ways that solve problems or generate new insights (Woolfolk, 1995, p.
241). Retention is greatly influenced by the kind and strength of cues used in encoding
information. The stronger and more varied the cues involved in information processing
the more the information will be retained (Borsook and Higginbotham—Wheat, 1992;
Driscoll, 1994, p. 96). The multi-sensory nature of new computer-related technologies
offers a great potential of using cues that can significantly enhance learning processes. As
Borsook and Higginbotham-Wheat (1992) noted, “Strong cues, such as those cues which
may be generated in hypermedia, are more likely to be present at recall.”

Gagné’s theory of instruction is a particularly suitable framework for conceptualizing
effective teaching and learning because it combines “a cognitive information processing
perspective on learning with empirical findings of what good teachers do in their
classrooms.” (Driscoll, 1994, p. 330). Driscoll asserts also that the theory has served
well as a “basic framework for a prominent instructional design theory” (p. 330). The
theory offers a well-tested and validated comprehensive approach for conceptualizing

multiple instructional methods for multiple learning outcomes (Driscoll, 1994, p. 333).

w-
k... “A

”h.

 

15

Multidimensional scaling (MDS) procedure was utilized in this study to examine
patterns of perceptions among experts with regards to the level of importance of
characteristics of computer courseware in promoting effective learning. The MDS
technique provides the ability to examine relationships among entities in similarity or
dissimilarity matrices and offers a best fit to an Euclidean distance space with relatively
few dimensions (Wilkinson, et al., 1992). The result of MDS analysis helps to simplify
abstract and complex sets of relationships by reducing a matrix to a spatial analogue in
which each entity is represented by its coordinates on a few dimensions (Kruskal and
Wish, 1978). The MDS technique permits the use of data which satisfies minimal
assumptions regarding level of measurement and satisfies some basic triangulation
assumptions regarding relationships among entities (Kruskal and, Wish, 1978; Donthu and
Rust, 1989; Wilkinson, et al., 1992; Jones and Koehly, 1993).

The combination of Gagné’s instructional theory, as a way of conceptualizing
instructional and learning processes, and multidimensional scaling technique, as a
procedure for understanding patterns perceptions, offers a set of powerful frameworks
for understanding factors and characteristics that make courseware successful in
promoting effective learning. The operationalization of the frameworks in this study
should provide useful information on more effective ways to assess the results of

academic applications of computer courseware.

16
Definition of Terms

In this study, important concepts are defined as follow:

Computer courseware — refers to any interactive educational material delivered by
means of computers, or computer-related information technologies, including multimedia
or hypermedia technologies, CD-related technologies, videodisc technology.

Courseware characteristics — refers to characteristics of effective learning and
effective instruction incorporated into the computer courseware programs.

Courseware quality -- refers to the degree to which courseware characteristics
match research findings on effective instruction and effective learning using the rating
scale: very good, good, fair, poor, very poor, and not applicable.

Effective instruction —— refers to the achievement of specified educational
(instructional and learning) outcomes.

Efficacy of computer courseware —— refers to effective and efficient achievement of
educational outcomes (including learning motivation, learning achievement, and transfer of
knowledge skills) specified for the use of computer courseware. It refers also to the
power (potential/ability) of any computer-based educational material (computer-based or
computer assisted instructional program with or without interactive audiovisual media
components) to produce specified educational outcomes including learning motivation,

learning achievement, and transfer of knowledge skills in effective and efficient ways.

 

...3'
nag-y,

A
‘3‘.

17

Efficient instruction — refers to the achievement of specified educational
(instructional and learning) goals within the time framework (estimated or scheduled) for
the completion of learning the computer courseware.

Expert — refers to any individual (university faculty or staff) with at least three years
expertise or professional experience in college-level physiology instruction, classroom
teaching using computer courseware, computer courseware development, instructional
design, or research and evaluation.

Evaluation — is defined as a systematic process of gathering, synthesizing, and
interpreting information in support of making prudent decisions about the quality of the
object evaluated based on defined criteria

Learning processes — refers to how information is acquired, stored, manipulated,
and retrieved.

Meaningful learning — refers to appropriate and effective use of relevant
information, problem solving skills, and knowledge transfer, as well as generating useful
information. It also refers to effective use of problem solving skills.

Learning motivation —— refers to interest in learning. For the purpose of this study,
the term refers to the computer courseware used in this study to generate and sustain
students’ interests in learning the content materials presented.

"The same instructor" — refers to a faculty member who developed computer
courseware and also uses it as an integral component of an academic curriculum.

Pattern of courseware use — refers to patterns of students’ behavior in using the

computer courseware programs. Pattern of use is classified as follows: did not complete

M: '3'
' l

-4 V‘
r I

“1“:

3a....»

‘-’€

W.-

 

--5\
l

W Z!
Hadul

no),
u...‘

..
s-,.

.3-
..3

 

18

any program, completed one program at least once, completed some program(s) once and
some program(s) two or more times, and completed two or more programs two or more
times.

Time on courseware — refers to the amount of time students spent interacting with
computer courseware programs relative to the time specified for completing the programs.
Time on courseware is categorized as follows: below specified (expected) time, within
specified (expected) time, above specified (expected) time.

Successful computer courseware — refers to any computer-based educational
material (computer-based or computer-assisted instructional program with or without
interactive audiovisual media components) that produces specified educational outcomes
including learning motivation, learning achievement, and transfer of knowledge skills in

effective and efficient ways.

The Dissertation Plan
Chapter One of this dissertation presents the context for the study which includes the
introduction, statement of the problem, the purpose, the significance of the study,
research questions, assumptions, and the theoretical fiarneworks for the study. The
chapter closes with the definition of terms and concepts used in the study.
Chapter Two presents a review of literature. It delineates the theoretical constructs

underlying the study.

, 1’". \I
:3..th

A32; 3
g...» b

, .
32.15.:

A§_~‘. .0;
r2“... fish
I

,.
-.~_ J~t
$.15sz

'15 "'2'
51. 51M

Vofl)‘.l
: "‘r u

U~~_o:,

I
qu _
. .44; k4

‘if;pa‘.
"' h baa

19
Chapter Three describes the design of the study. It contains a description of the

selection of subjects for the study, the methods employed for the study, the development
of the research instruments, sources of data, and procedures for data collection and
analysis.

Chapter Four presents data analyses addressing research questions A — effects of
computer courseware in students’ learning achievements. It also contains analyses of data
collected to answer research question B — students’ perceptions regarding the quality of
the characteristics of computer courseware used in the study.

Chapter Five presents data analyses addressing research questions C — experts’
perceptions regarding the level of importance of computer courseware characteristics in
promoting effective learning. It also contains analyses of data collected to answer
research question D — experts’ perceptions regarding the quality of the characteristics of
computer courseware used in the study.

Chapter Six presents the summary of findings, discussions, and conclusions of the
study. It also contains recommendations for further studies and policy implications. In
this chapter, a prescriptive framework for evaluating the success, or efficacy, of computer

courseware based on the findings of the study and literature review is presented.

\~.-

5..»-

“"A

Wb<

‘O.

u
\-

f

'2
l 3'
‘I
‘1
;-- i
‘31 l
l
v.
M.
Q Q
.1
u R.‘
'q

CHAPTERII

REVIEW OF THE LITERATURE

Introduction

This chapter reviews the literature from five major areas pertinent to this study: (1)
effective teaching and effective learning; (2) technology in education; (3) cognitive
information processing theory; (4) evaluation; and (5) multidimensional scaling. This
review provides the basis for investigating ways of understanding and evaluating factors

and characteristics that promote effective learning via computer courseware.

Effective Teaching and Effective Learning

Society has always been interested in understanding how teaching and learning occur.
Various aspects of these topics have engaged the attention of researchers and scientists
who study human thought processes and behavior. Research on these topics has
traditionally focused on teaching and learning processes in the classroom. With the
advent of modern technologies, however, research on teaching and leaming processes has
also focused on the role and effects of technology on the way we teach and learn.

In a comprehensive review of more than 650 studies of effective instruction conducted
by Group for the Study of Effective Teaching (Lillie et al., 1989) listed sixteen

characteristics of effective instruction which Should also be the guiding "research-based

20

21

instructional principles" for the design and development of computer-based instructional

software.

The review pointed out that students learn more when instruction (i.e., lessons

and activities) incorporates these principles:

PNFMfPPT‘

9.

Beginning lessons with review

Beginning lessons with an introduction

Presenting instruction fluently and precisely

Using understandable language and concepts

Using relevant examples and demonstrations

Ensuring high rates of success

Presenting instruction at a brisk pace

Making smooth transitions within and between lessons
Making assignments and instructions clear

10. Summarizing the main points of the lesson

11. Maintaining reasonable standards

12. Checking student performance routinely

13. Posing questions one at a time

14. Providing instructional feedback

15. Affirming correct responses

16. Providing sustained feedback after incorrect responses

Similar factors that influence effective teaching have been identified by others. For

instance, Mastropieri and Scruggs (1987, p. 4) identified the following as "general teacher-

effectiveness" variables: time on task; content covered in terms of scope and sequence,

objectives, and pacing; providing information, i.e., teacher delivery of content;

questioning; feedback; guided and independent practice; and formative evaluation.

These principles have serious pedagogical implications. Any instruction that

incorporates these principles can be expected to be effective. Conversely, any instruction

that lacks any or some of these characteristics will be less than optimal in quality and

effectiveness, and perhaps will be ineffective.

U

~

Q; «~
”:5 5L
I ‘
“I; 3!-
us like
.'I‘.

m 1'
' MK...

;.I- ~-
\h‘“;

5”} \r

22
Clearly, the body of literature on effective teaching is replete with very useful

information on principles of effective instruction for either the classroom setting or
computer-based instructional software. However, most of the work remains essentially
prescriptive rather than a descriptive study examining the extent to which courseware

programs actually incorporate the principles which facilitate learning processes.

Technology in Education
The computer technology has been hailed as perhaps the most significant innovation on
the landscape of education since the advent or invention of printing technology. This
section of the review of literature examines some of the important advances in computer-

based technologies and their implications for instruction and learning.

Computers as Information Processing Systems

Computers are information processing systems with unprecedented capabilities. Their
information processing capabilities far surpass those of other technologies including print,
radio and television (I-Ieinich, 1996; Kozma, 1991). For instance, unlike television that
offer the viewer little or no control over the nature of televised information, the computer
allows the user to work with data and information in dynamic and multi-dimensional
ways. Whereas television viewers only have control over sound and video components of

the information they receive (essentially by being able to turn them on or off), computer

Q ~~I

ufiA-fi.

CI-
)2
I

..'~‘

I .
:Ip"
5'

.“'M

\--. .
’v:\ ‘
a V...:

23

users can work with and dynamically manipulate various forms of data and information
including text, graphics, audio, and video.
Computers can juxtapose, or transform, information in one symbols system to
that in another A learner can type in printed text, and a computer can with a
voice synthesizer transform it into speech. The computer can take equations,
numerical values, or analog signals and transform them into graphs. (Kozrna, 1991,
p. 195).

Since the multi-symbol processing capability of computers make it possible for
humans to process information in a variety of ways, computers hold a great potential for
a wide variety of applications in education and training. Computers can facilitate teaching
and learning at all levels of education and cognitive processes (Kleinman, 1984, p. 120).
They can empower teachers to provide a wide variety of instructional materials to a
variety of learner groups in highly effective, efficient, and motivating ways. Computer-
related technologies, such as hypermedia systems, can provide opportunities for
collaborative work and thereby foster meaningful interaction between the teacher and
students or among students (OTA, 1991). The flexible information processing
capabilities of computers make them powerful tools for learners to control their own
learning experiences. Computers can enable learners to control the type and amount of
information accessed, set the pace of learning, and experience dynamic and meaningful

interaction with information. Computers can provide learning experiences which are not

possible or very difficult to accomplish through other means.

'Jq-n-‘l \ "
i
ye... -

...——“-’—

Orr: :11:
xi: horizo:
1"“. sin

YrAv‘u "tort

:3".ng ~. ..
lib. 2.1110 L'gl 5..

'9- ”)4 9,35“,-
hush “Sui

1.5.5::de toil,

2“)”: ‘F‘

:fi'w‘l td.

:‘1-3'4

“W ““1101“;

Ir
“1;"
DH, , r
.53“..ng ‘.
1

Dsk

wig jO'

L “bx—3'11
-

7‘05.“

Auk

:Qi‘ua‘

«Sou

€343, 511d t

2 4
Technological Horizons in Education

Over the past decade, several forms of new information technologies have emerged
over horizons in education and with important implications for educational practice.
These technologies are commonly referred to in terms of either the functions they
perform or the very nature of the specific technology. By functions, new information
technologies may be referred to as new technologies, emerging technologies, computer-
related technologies, multimedia, hypermedia, or hypertext. Often the technologies are
referred to in terms of the specific nature of the technologies such as interactive video,
digital video interactive (DVI), CD-ROM, compact disc interactive (CD1), or virtual
reality. The element that is common to all the technologies is the utilization of computers
to dynamically access and display a wide variety of “nonsequentially” organized and

stored information including text, graphics, audio and video.

Hypertext, Hypermedia, and Multimedia

Distinctions have been made in the literature among technologies commonly referred to
as hypertext, hypermedia and multimedia (Heinich et al., 1996). Hypertext is a term used
to refer to computer-based “nonsequential documents” composed of text, audio, and
visual information which are linked and annotated into a web of related chunks of
information. In contrast, hypermedia is a generic term used in referring to a “computer
hardware and software system that allows the composition and display of nonsequential
documents that may include text, audio, and visual information and in which related

information may be linked into webs by author or user.” (Heinich et al., 1996, p. 407).

l '., .
Fl.finlric‘

tips 0-

ms.
-.'-'-U
s

I II
.0. sq

.rushsé“

. . ,

w . 4

flannel.
a...

25

Multimedia refers to computer hardware and software systems for the composition and
display of presentations that incorporate text, audio, graphics, animation, and video
images.

The constant and increasing change in the sophistication of new technologies has
practically blurred any substantive differences among hypertext, hypermedia and
multimedia technologies. Little or no differences exist today in the nature of the
technologies used for creating hypertext, hypermedia, or multimedia applications. As
such, the three terms are used interchangeably in this study.

Hypermedia systems hold great promise for providing effective instruction and
meaningful learning experiences. They function in the same manner as the human mind
processes information. The mind can organize or structure information by establishing
relationships, links and associations among symbols, concepts and other forms of
information. Similarly, hypermedia allows hyperlinks and annotation among related
chunks of information. The similarity between hypermedia capabilities and the
associative properties of the human mind offer the potential of extending human
information processing abilities. Hypermedia can empower or enable individual learners
to engage in active, as well as creative learning experiences. Also the multisensory
capabilities of hypermedia technologies offer the potential for meaningful, “real-world”
or “all-at-once”, learning experiences for individuals and groups. As Heinich et a1. (1996,
p. 261) pointed out, “individual learners respond differently to various information
sources and instructional methods, so the chances of reaching an individual are increased

when a variety of media is used.”

 

I.
(b
3

I; ‘
..>y
.

,.._..._‘
5.0.5.5..

'—‘~~
. u

\p_‘
l‘ -ad‘.
d

c‘ _
”a- 9.
e.

26

Interactive Video

Computer-based interactive video system combines the technologies of computers and
television to provide a multimedia learning environment in which audio, graphical, and
video materials are presented under the control of the computer and/or the interactive
responses of the users (Heinich, 1996; Semrau and Boyer, 1994; and Cronin and Cronin,
1992). As Heinich et al. (1996) described the technology, “The video portion of
interactive video is provided through a videocassette, videodisc, or compact disc. The
images can be presented in slow motion, fast motion, or frame by frame... The audio
portion of a videodisc may occupy two separate audio channels, making possible two
different narrations for each motion sequence.”

Interactive video offers various levels of interactivity, random access, and branching
capabilities. The Constant Angular Velocity (CAV) videodisc has capacity to hold 30
minutes of motion video or 54,000 flames of still images on each side. Combining these
capabilities with powerful decision-making algorithms of authoring software programs
offer the potential for providing learning environments in which learners can interact with
materials at their own pace. A typical interactive video program provides “fully
interactive response-driven instruction featuring embedded questions, response feedback,
and branching within a lesson. In addition, student response records can be used to help
make instructional decisions.” (Heinich et al., 1996, p.268). Interactive video has played

an important role in providing problem-solving and simulated learning experiences

a“:
I->

- t
'0'.

wk.

:95;

*C.

‘ho ‘

Nfi‘

3‘1
"v.1

I“ ‘
5U.

27

especially in areas or fields that are too dangerous, costly, or take too much time to

provide real-life learning experiences.

CD-ROM

CD-ROM is a high capacity optical storage medium with high quality audio
capability. CD-ROM discs can hold over 650 megabytes worth of digital information
including text, graphics, photo images, animation, audio, and video. The high capacity,
ideal portability, durability, versatility for multimedia development, and decreasing cost
of CD-ROM have gained the technology wide acceptability in education. CD-ROM is
currently the preferred medium for a wide variety of applications in education including:
making available significant and timely research databases such as the Educational
Resources Information Center (ERIC) databases; publishing multimedia encyclopedias
and other electronic books such as titles published by Discis Books; and delivery of
interactive multimedia instructional programs such as: the Pathology Lab Review CD-
ROM program, Exploring the Michigan State University Libraries, and a number of other
CD-ROM based programs in which the present researcher has been involved in producing
at Michigan State University. The search capabilities of CD-ROM technology can foster
the attitude and skill of exploring interrelationships among concepts and ideas as between

people, places, and events.

28

DVI/CDI

Digital video interactive (DVI) is a variant of the compact disc technology. DVI
combines the powers of the computer, CD-ROM technology, and a video compression
and decompression board to provide 72 minutes of full-motion video. A closely related
technology to DVI is compact disc interactive (CDI) technology. CD1 consists of a
highly intelligent special compact disc player connected to a standard television set. It is
used in providing programs or information that integrate text, audio, graphics, animation,

and broadcast quality video.

Academic Uses of Computer Courseware

Computer courseware programs have been classified under several nomenclatures.
Programs have been classified based on the type of instruction and/or learning experiences
they provide and how a program is integrated into curriculum.

The traditional classification system used for categorizing educational software
programs are: drill and practice, tutorial, problem solving, simulations, and intelligent
tutoring systems. This category system is based on the role(s) that programs are
designed to serve in learning processes. The roles are determined by the kinds of events
of instruction the programs provide (Wager and Gagné, 1988).

How programs are integrated into an academic curriculum is another basis of

classifying computer courseware programs. According to this category scheme, four basic

WK

..Hu
run?

'H"r

at...

‘0‘ .

r' ~b;

‘ngs I

.p

I.

O
t,
s, a

29
types of academic software exist (Syllabus, September 1994): stand-alone content

software, software specifically designed for use with a textbook, software specifically

designed for classroom interaction, and studyware.

Cognitive Information Processing Theory

Information Processing Components

The way the human memory functions is the primary lens through which information
processing theorists view learning. The human memory is conceived to function much
like the computer system. As Woolfolk (1995, p. 243) noted:

Like the computer, the human mind takes in information, performs operations on
it to change its form and content, stores the information, retrieves it when needed,
and generates responses to it. Thus, processing involves gathering and
representing information, or encoding; holding information, or storage; and getting
at the information when needed, or retrieval. The whole system is guided by
control processes that determine how and when information will flow through the
system.

According to the cognitive information processing view, the human memory consists
of three major components or processes. These are sensory memory, short-term
memory, and long-term memory. Basically, the processes involve encoding of

information “in the sensory register where perception determines what will be held in

short-term memory for further use. Thoroughly processed information becomes part of

 

 

 

 

50.712333
U

1523 SCflSC

. ,

ovum...“

txbbuut‘g.
b

spent: al

“-15%“

\. eu\ IA
'1" "’1 n
>5u.\1a101.
”“(TDv

whet“
b

‘F‘Tan

y'“~ I e .

Pr. _-,
with" “A.

rifle—J

n)

30

long-term memory and can be activated at any time to return to working memory.”
(Woolfolk, 1995, p. 244).

The sensory memory or register refers to a system 'of receptors which functions to
hold sensory information very briefly. Each of the five senses (hearing, seeing, tasting,
touching, smelling) has a sensory memory associated with it. And although they all
operate alike (Driscoll, 1994, p. 69), each sensory memory functions in a unique way. As
Woolfolk (1995, p. 245) noted, “The content of sensory memory resembles the
sensations from the original stimulus. Visual sensations are coded briefly by the sensory
register as images, almost like photographs. Auditory sensations are coded as sound
patterns. It may be that the other senses also have their own codes.”

In sensory memory, both perception and attention play critical roles in the selection
and organization of information for further processing. It is perception that allows us to
attach meaning to raw information received through our senses. The construction of this
meaning is a function of both objective reality and prior knowledge. On the other hand,
attention is the ability to focus on certain stimuli and ignore others. Woolfolk (1995, p.
246) underscores the importance of attention as follows:

The first step in learning is paying attention; students cannot process something
they do not recognize or perceive. Many factors in the classroom influence
student attention. Eye-catching or startling displays or actions can draw attention
at the beginning of a lesson... Bright colors, underlining, highlighting of written or
spoken words, surprise events, and changes in voice level, lighting, or pacing can
all be used to gain attention.

Once information in sensory memory is transformed into patterns of images or sounds

or other types of sensory codes, it passes into the short-term memory system. Short-

wt“

‘1

.55

5., .
.45 l

.

-JQ

5.4.6.

9.1-!
O b b‘
a

a..

u‘
‘a4"‘

 

31

term memory is working memory which holds a limited amount of information briefly.
According to Woolfolk (1995, p. 247) “The information in short-term memory may be in
the form of images that resemble the perceptions in sensory memory, or the information
may be structured more abstractly, based on meaning.”

A number of processes can help to foster the retention of information in short-term
memory. One strategy to keep information activated in short-term memory is to rehearse
information mentally. Rehearsal which involves repeating information in the mind is
known as maintenance rehearsal. Whereas rehearsal which involves associating new
information with prior knowledge is known as elaborative rehearsal. Another strategy for
keeping information activated in short-term memory is by the control process of
chunking. This process involves grouping individual bits of data into meaningful larger
units.

Information that is well activated in short-term memory (by control processes such as
rehearsal and chunking) is commonly assumed to be well-learned. The well-learned
information passes into long-term memory for permanent storage. The capacity of long-
term memory is practically unlimited. However, much time and effort are required to
process information for permanent storage in or retrieval from long-term memory.

The contents of long-term memory are regarded to be either visual images, or verbal
units, or both. Thus, some cognitive psychologists maintain that information coded both
visually and verbally is easier to remember. Others “suggest that many images are
actually stored as verbal codes and then translated into visual information when an image

is needed.” (Woolfolk, 1995, p. 250)

“3"“

“an“

(I)

"i
*1 I

'I0.

In.“

A.'

bL',

W.
0‘.

pm
In

32

The third stage in the information processing theory is the long-term memory. This
memory represents the permanent storage of information. The long-term memory is
commonly categorized as semantic, episodic, and procedural. While the episodic memory
is memory for specific events, semantic memory “refers to all the general information
stored in memory that can be recalled independently of how it was learned.” (Driscoll,
1994, p. 86). In the words of Martindale (1991, p. 181), “Semantic memory contains the
basic elements of knowledge, and episodic memory is made up from these elements.”

Semantic memory is also referred to as memory for meaning. This memory is stored
as propositions, images, and schemas. Woolfolk (1995, p. 250) defines images as
“representations based on perceptions—on the structure, or appearance of the
information...Images mirror what they represent—they preserve the physical attributes
and spatial structure of information.” Thus, images are useful in fostering
conceptualization and proceduralization of practical operations as well as abstract
reasoning. A schema, on the other hand, is an abstract data structure that organizes vast
amounts of information. It “is a pattern or guide for understanding an event, a concept, or
a skill.” (Woolfolk, 1995, p. 251)

Procedural memory is the third category into which long-term memory is
distinguished. This is the memory for how to do things. According to Woolfolk (1995,

p. 254), “Procedural memories are represented as condition-action rules, sometimes called
productions. Productions specify what to do under what certain conditions: if A occurs,

then do B.... The more practiced the procedure, the more automatic the action...”

5‘35.

bed-

a?”
si—I

END
“UL lhl

"-4
‘ 0.
In. F

Af-‘p -\

“UL:

‘. no .
~04.

35w

.e‘\.

33 r,-
'
ml .
‘ul 4“

T

01M

M
Qt,“

1 n
‘u.

,1 l“:
““uq‘

33

Information processing in long-term memory, and thus learning, is affected by
elaboration, organization, and context (Woolfolk, 1995, p. 254). Elaboration promotes
learning as we apply our schemas and draw on prior knowledge to construct an
understanding of new information. In the case of organization, Woolfolk (1995, p. 254)
noted that: “Material that is well organized is easier to learn and to remember than bits
and pieces of information, especially if the information is complex or extensive. Placing a
concept in a structure will help you learn and remember either general definitions or
specific examples. The structure serves as a guide back to the information when you need
it.” As for the role of context in information processing, learning improves when the
context in which information will be recalled is linked with the physical and emotional
aspects of the original context in which the information is processed.

Retrieval is another psychological process that plays a crucial role in learning. One
way in which information is retrieved fi'om long-term memory is through the spread of
activation. This means that remembering, or retrieval of a piece of information, activates
the recall of associated information. Another means of retrieving information from long-
term memory is through reconstruction, a problem-solving process of “recreating
information by using memories, expectations, logic, and existing knowledge.” (Woolfolk,
1995, p. 255). Thus, given the right cues, information can be effectively retrieved from
long-term memory.

In sum, the cognitive information processing theory suggests a number of
psychological processes that are necessary to foster effective learning. The processes

include raising attention, expectations, motivation, activating prior knowledge, and

+

36.13?
1

N" "

.
g”... I“

“VJ
.,’

a4. . ‘3.
“have“
l‘por‘ . A

up at

the tie.)
Lu; -1»

R.‘ '9

'3‘
5L...)\

‘twn
fl
‘-.
“fires;
Mic“

34
performance monitoring (Shuell, 1992). A number of important theories of learning and

instruction have evolved from the cognitive information processing theory in the efforts
to apply it as a framework for conceptualizing educational practice, including the
educational uses of computer courseware, as information processing systems. One such
instructional theory is Gagné’s theory of instruction. The next section briefly describes
the theory as a framework for conceptualizing factors that influence the success of

courseware programs in promoting effective learning.

M’s Theory of Instruction

Gagné’s theory of instruction is a model example of a comprehensive instructional
theory built on information processing theory. The theory incorporates multiple learning
outcomes and provides multiple instructional models. As Driscoll (994, p. 333) noted,
“Gagné’s theory incorporates three major components: a taxonomy of learning outcomes,
specific learning conditions required for the attainment of each outcome, and the nine

events of instruction.” Each of the components are briefly described in the subsequent

paragraphs.

Learning Outcomes
Gagné (1985) describes five major categories of learning outcomes: (1) verbal
information; (2) intellectual skills; (3) cognitive strategies; (4) attitudes; and (5) motor

skills. Verbal information refers to “knowing what” or “knowing about” organized bodies

 

 

s‘nna
tens»

Co

3"” . I
a“ due

H

:.
u. e

‘vie:‘zl
“”45“.

T‘s-«i.
'“vynl
I

35

of information. Knowledge of verbal information is demonstrated in learners’ ability to
state previously learned material such as facts, concepts, principles, and procedures.

Intellectual skills refer to procedural knowledge. They are learning outcomes that
involve “knowing how” to solve a problem, apply a rule, or use a concept. Intellectual
skills are divided into five, hierarchically ordered levels: discriminations, concrete
concepts, defined concepts, rules, and higher order rules.

Cognitive strategies are mental processes by which learners control their own feelings,
learning processes, thinking, and constructing or choosing solutions to problems.
According to Gagné (1985), cognitive strategies represent the executive control functions
of the human information processing system. “As such,” remarked Driscoll (1994, p.
341), “learners employ cognitive strategies to monitor their own attention, to help
themselves better encode new information, and to improve their success at remembering
critical information at test time.” There are no specific, observable learning outcomes
directly associated with cognitive strategies. As Driscoll (1994, p. 341) pointed out,
“Frequently, because cognitive strategies are employed in the service of other learning
goals, it is the attainment (or not) of those goals which is noticed.”

Attitudes as learning outcomes refer to previously learned mental states that influence
choice of personal action toward some objects, persons, or events. Attitudes may be
learned as a means for achieving other learning outcomes or they may be learned as ends in
themselves.

Motor skills refer to skillful execution of bodily movements. Gagné (1991) described

motor skill as the “capability that enables the individual to execute some smoothly and

36

precisely timed pattern of muscular movement such as is seen in swimming, in planing the
edge of a board, or in printing a letter.”

The differences inherent within the learning outcomes mean that each of the learning
outcomes will differ with respect to instructional content, instructional conditions , and
assessment procedures that are all designed to engender the learning outcome (Hannum,
1988). As Travers et al. (1993) aptly remarked, “Each of the learning outcomes demands
a different set of conditions for optimizing learning, retention, and transferability.” The
next section describes various kinds of learning conditions that are required to bring about

the learning outcomes.

Learning Conditions

Learning conditions are the essential building blocks for effective instruction. (Gagné
and Driscoll, 1988). Both instructional goals and learning outcomes are achievable by the
kinds of internal conditions activated and/or external conditions constructed to facilitate
learning processes.

Verbal information is best learned when the information is presented in meaningful
chunks and in a wide variety of meaningful contexts. Instructional strategies often used to
facilitate effective learning of verbal information include imagery, cognitive organizers,
themes, mnemonics. The strategies should be used to direct learners’ attention to
significant points by variations in print or speech (Driscoll, 1994, p. 345). The

multisensory capabilities of multimedia technologies offer a great potential for providing

huh»

. -Ow'
ism

.
v.0.)
.u‘bsl.

«a: 1“
5418‘ s

I;~‘i.
Uhulu

-
3“ng
Mus.

(aka;

F

37

learning experiences that draw learners’ attention to distinctive features and ensure that
cues used during initial learning match the retrieval cues.

In the case of intellectual skills, learning conditions must direct learners’ attention to
distinctive features of the concept or rule to be learned. Retention and transfer of
intellectual skills are best facilitated by providing spaced reviews and opportunities for
practice using a variety of examples and problems.

Conditions that promote effective learning and use of cognitive strategies include:
demonstration of strategies to be learned, providing a variety of contexts for practice
using the strategy, and providing informative feedback as to the creativity or originality of
the learning outcome (Driscoll, 1994, p. 345).

Attitudes are learned primarily by modeling. The following conditions are required to
effectively learn or establish attitudes: creating an expectancy of success associated with
the attitude to be learned, making the modeling experience relevant to the learners’ needs,
providing opportunities to demonstrate choice of personal action related to the desired
attitude, and providing feedback for successful performance ( Driscoll, 1994, p. 349).

Motor skills “consist of component skills that must be mastered separately before
they can be assembled into the single, terminal performance.” (Driscoll, 1994, p. 350).
Therefore, conditions for effective motor skill learning include: presenting learning
guidance to cue the executive subroutine, providing opportunities for practice, presenting
immediate feedback concerning the accuracy of performance, and providing

encouragement to use mental practice.

CO

1|.

Q ‘_

38

In sum, learning conditions are essential factors for achieving learning outcomes. They

constitute the building blocks for instruction. Conditions of learning include internal and

external conditions. While internal conditions refer to prerequisite skills, external

conditions refer to various ways that instructional events outside of the learner function

to activate and support internal processes of learning. Gagne' referred to pedagogical

strategies as events of instruction (Gagné, 1985). The events of instruction provide ways

of determining prescriptions to facilitate different types of learning outcomes.

Events of Instruction

Gagné proposed nine events of instruction to facilitate activation and regulation of

executive control processes that modify flow of information as it passes through the

stages of memory. Table 2-1 presents the nine events of instructions together with the

internal processes that they support.

Table 2-1: Events of instruction and their relation to processes of learning

 

 

Instructional Event Internal Learning Process

1 Gaining attention Attention; alertness

2 Infonning learner of the objective Expectancy

3 Stimulating recall of prerequisite Retrieval of prior learning to working
leaming memory

4 Presenting stimuli with distinctive Pattern recognition; selective
features perception

5 Providing leaming guidance Semantic encoding; cues for retrieval

6 Eliciting the performance Retrieval and responding

7 Providing feedback about performance Reinforcement
correctness

8 Assessing the performance Cueing retrieval; retention

9 Enhancingretention and transfer Retention, retrieval, andjeneralization

 

(-

“It

. .

39

The nine events of instruction are necessary ingredients for effective learning. Thus,
the purpose of “gaining attention” is to focus learners on the tasks to be learned.
Providing learners with instructional objectives help them to know the criteria to be
mastered and when they have been achieved. Recall of relevant prerequisite information
into active memory is crucial to the acquisition of complex or integrative skills. The
presentation of the stimulus material is the heart of instruction. The stimulus material
presented depends on the goal of the instruction. However, as Driscoll (1994, p. 353)
aptly remarked, “For all types of outcomes, the stimulus presentation should emphasize
distinctive features or essential elements of the desired outcome in order to facilitate the
processes of pattern recognition and selective perception.” To that end, “techniques such
as simplifying a display, voice inflection, underlying, encircling, pointing at, or enlarging
might be used to be sure that learners attend to relevant instructional features.” (Okey,
1991). In terms of providing learning guidance, the purpose is to help and ensure that
learners acquire the particular capabilities specified in the objectives. The instructional
event involves modeling or showing learners what appropriate actions constitute correct
performance (Okey, 1991). Eliciting performance is the instructional event that requires
the learner to demonstrate mastery of the newly learned stimulus material. Its purpose is
to show whether or not the learned capability has been stored in long-term memory. The
response required from the learner must match that required in the objective as a way of
providing practice that is relevant to the desired outcome. Informative feedback is

provided to apprise learners of the correctness or accuracy of their performance.

40

Feedback functions to correct learners’ misconceptions or provide extended elaboration
on the knowledge or skill to be learned. The next event of instruction is assessing
performance. Performance assessment is concerned with exhibiting complete
demonstration of the learning outcomes at the close of the instruction. The purpose is to
establish the degree of retention of the new learning as well as to provide additional
practice. Retention of all learned capabilities is further enhanced by providing additional
practice in a wide variety of instances. Learning in a variety of contexts enhances the
ability to transfer or apply knowledge learned in settings and situations different from the
initial learning contexts.

For effective application of the events of instruction, the events must be designed and
implemented in ways that meet the specific requirements of each type of learning
outcome. The need for differential implementation of instructional events stems from
different functions served by the events. Aronson and Briggs (1983) classified the nine
events of instruction into two main categories according to their functions. One category
consists of events that “function in essentially the same way regardless of the type of
learning” (p. 93). The events in this category include gaining attention, informing the
learner of the objective, eliciting performance, and providing feedback. The essential
function of this group of events is to facilitate an expectancy of the results of learning.
The second category consists of events that “function entirely differently depending on
the type of learning” (Aronson and Briggs, 1983, p. 83). Included in this category of
events are stimulating recall of prerequisite leamings and providing learning guidance.

Two factors account for the differential effects of the events of instruction. First, “The

41

nature of the essential prerequisites for stimulating recall of prerequisite [earnings is
different for each type of outcome because the internal conditions are different.”
(Aronson and Briggs, 1983, p. 93). The second factor is that “the function of learning

- guidance will be different for each type of learning because the external conditions are
different. Thus, the instructional prescription is often determined by the kind of learning
guidance that is required.” (Aronson and Briggs, 1983, p. 93). Ultimately, practical
applications of the events of instruction depend on the characteristics of the learner and
the nature of the learning situation.

As Driscoll (1994, p. 351) pointed out, “the manner in which the events are
implemented may vary greatly depending upon the instructional delivery system that is
chosen.” As such, the process of applying the events of instruction often involves
making prescriptions for the use of various media. Instructional media refers to “all
means by which stimuli are presented to provide the eventsof instruction. In this sense,
media include the full range of audiovisual materials, print, and the voices of the teacher
and learners.” (Aronson and Briggs, 1983, p. 93). The use of media must be matched with
learner characteristics and task variables. As a rule of thumb, prescriptions for media use
should be sequel to and determined by specification of the requirements for providing the

instructional events for a given instructional sequence.

Summary
The cognitive information processing theory, out of which grew Gagné’s theory of

instruction, was presented as a basic framework for conceptualizing instructional and

42

learning processes. Information processing theory specifies or describes conditions and
processes that control how humans acquire and use information. In a similar fashion,
Gagné’s theory of instruction stipulates a variety of ways in which learning outcomes,
conditions of learning, and events of instruction can be orchestrated to facilitate encoding
of information or enhance meaningful learning. Whereas “the conditions of learning
include both internal events (such as previously encoded information) and external events
(such as methods of elaboration to facilitate encoding) the events of instruction refer to
methods or procedures designed to facilitate the specific processes (such as encoding,
retention, retrieval, etc.) thought to occur during learning.” (Driscoll, 1994, p. 333).

Gagné’s theory of instruction has practical use for instructional design and evaluation
of learning systems and processes. As Aronson and Briggs (1983) stated:
“Understanding how instructional events function in different learning situations and for
different types of learning provides direction to the designer in making appropriate
instructional prescriptions.” And, as Travers et a1. (1993, p. 371) pointed out, Gagné’s
events of instruction serve as a model that “provides an instructional basis for analyzing
the interaction of internal events with external events, which makes the model applicable
to instruction of many forms in a wide variety of settings.” Lessons designed based on
Gagné’s instructional design theory will manifest specified learning outcomes, well
organized instructional events that are appropriate for each kind of learning outcome, and
how instructional events are exemplified for different kinds of outcomes. The key

elements of the learning outcomes, the conditions of learning, and the events of instruction

43

formed the substance of the characteristics or variable categories for content analysis of

the computer courseware programs under study.

Evaluation
This section of the literature review focuses on evaluation. The section is organized into
the following parts: Definition of Evaluation, Function of Evaluation, Evaluation Models

and Approaches, and Evaluation of Computer Courseware.

Definition of Evaluation

Evaluation has been defined differently by different people relative to its purpose and
function (Wolf, 1984, p. 2). One of the earliest and highly influential definitions of
evaluation was proposed by Ralph Tyler. Tyler (1950, p. 69) defined evaluation as “the
process of determining to what extent educational objectives are being realized.” Basic to
the Tylerian view of evaluation is the principle that “evaluation should judge that a
program is good if, and only if, its objectives are achieved.” (Alkin and Ellett Jr., 1994).

Stemming from Tyler’s conception of evaluation is Provus’ Discrepancy Evaluation.
Provus (1971) defined evaluation as the comparison of performance to some standards to
determine whether discrepancies existed. Any observed “discrepancy information” is
utilized for program improvements.

A widely accepted definition of evaluation has been that of the systematic
investigation or assessment of the worth or merit of some educational objects (Scriven,

1967; Joint Committee, 1981; Brinkerhoff, et al., 1983). For instance, Smith and Glass

.5»

44
(1987, p. 30) defined evaluation as “the process of establishing value judgments based on

evidence about a program or product.” Borg and Gall (1983, p. 733) defined educational
evaluation as “the process of making judgments about the merit, value, or worth of
educational programs, projects, materials, and techniques.”

Evaluation has also been defined in terms of providing information for decision-
making. One of the best representations of this decision-oriented definition is provided
by the Phi Delta Kappa (PDK) National Study Committee on Evaluation. The PDK
defined evaluation as “the process of delineating, obtaining, and providing useful
information for judging decision alternatives (Stufflebeam, et al., 1971, p. 40). Similarly,
Beeby (1977) defined evaluation as “the systematic collection and interpretation of
evidence, leading, as part of the process, to a judgment of value with a view to action.”

The definition of evaluation considered most appropriate for this study is that given
by Dick and Carey. According to Dick and Carey (1991, p. 227), evaluation involves
“gathering information in support of making prudent decisions.’ The process
encompasses “clarifying the decisions to be made, deciding what information will be
useful in aiding the decisions, and then gathering, synthesizing, and interpreting the
information. Interpretation requires making judgments about the quality of the object
evaluated that are based on defined criteria.” (p. 227).

For the purpose of this study, therefore, evaluation is defined as a systematic process
of gathering, synthesizing, and interpreting information in support of making prudent

decisions about the quality of the object evaluated based on defined criteria. The primary

45

goal of evaluation is improvement of quality and determination of success of programs,

projects, materials, and techniques.

Function of Evaluation

Evaluation serves formative and summative functions (Scriven, 1967; Brinkerhoff, et
al., 1983). Formative evaluation is concerned with providing data to assist in improving
on-going program or project development. In contrast, summative evaluation is
concerned with providing data for determining the worth, value, or overall effectiveness of
programs, products, or procedures. Stufflebeam (1971) referred to formative evaluation
as proactive evaluation intended to serve decision-making, and to summative evaluation as
retroactive evaluation meant to serve accountability.

Formative evaluation is often conducted by internal evaluators, that is, individuals or
teams who are directly involved in the development of the program. The individuals or
team members may perform a dual role by being both developers and evaluators. In
contrast, summative evaluation is often conducted by external evaluators, or a team
comprised of internal and external evaluators. External evaluators are individuals not

directly involved in the development of the program and may be from outside the

organization.

Evaluation Models and Approaches

Evaluation models can either be descriptive or prescriptive in nature. As Alkin and

Ellett Jr. (1994), succinctly stated, “Evaluation models either describe what evaluators do

46
or prescribe what they should do.” Alkin and Ellett Jr. (1994) described prescriptive and

descriptive models as follow:
(a) A prescriptive model is a set of rules, prescriptions, prohibitions, and
guiding frameworks, which specify what a good or proper evaluation is and how
evaluation should be carried out. Such models serve as exemplars.

(b) A descriptive model is a set of statements and generalizations which
describes, predicts, or explains evaluation activities. Such models are designed to
offer an “empirical theory.

The focus of this study is concerned primarily with the prescriptive models of
evaluation. Therefore the review of literature on evaluation models focused essentially on
evaluation procedures and approaches.

A variety of prescriptive models and approaches have been developed as guidelines or
strategies for conducting evaluation studies (Kemp et al., 1994; Dick and Carey, 1991;
Alkin and Ellett, 1994; Flagg, B. N. 1990). Attempts to describe and explain various
evaluation procedures and approaches have led to a number of classification schemes.
Alkin and Ellett Jr. (1994) pointed out that the basis for determining a category system
for an evaluation model is the relative emphasis or importance placed on each of the three
component parts of the evaluation model. The three aspects of an evaluation model are:
methodology, values, and uses.

In relation to program evaluation, Worthen (1990) classified evaluation models into
five categories of approaches:

1. Performance-Objectives Congruence Approaches, which conceive of

evaluation as the process of determining the extent to which specified educational

objectives are actually attained. Discrepancies between the learner’s performance

47

and stated objectives provide information for revising the educational program or
instruction. Originally formulated by Tyler, this evaluation approach had great
influence on the formulation of Hammond’s EPIC evaluation model, Provus
discrepancy model of evaluation , and P0pham’s instructional objectives
approach.

2. Decision-Management Approaches, which describe appropriate evaluation
activities throughout the life-cycle of an educational program. This evaluation
approach is best represented by Stufflebeam’s Context, Input, Process, and
Product (CIPP) evaluation model (Stufflebeam, 1971) and Alkin’s UCLA
Evaluation Model (Alkin, 1969).

3. J udgment-oriented Approaches, which require experts to render a
professional opinion about the quality of an instructional program, procedure, or
product. This approach is represented by Scriven (1967), Stake’s Countenance
Model (1967), and Eisner’s “connoisseurship model” (1979).

4. Adversarial Approaches, which refer to all evaluations in which there is
planned opposition in the points of view of different evaluators or evaluation
teams. As Worthen (1990, p. 45) stated:

One evaluator (or team) would serve as the program’s advocate, presenting
the most positive view of the program possible from the data, while
another evaluator (or team) would play an adversarial role, highlighting any
extant deficiencies in the program. Incorporation of these opposing views

within a single evaluation reflects a conscious effort to assume fairness and
balance and illuminate both strengths and weaknesses of the program.

48

Models of adversary evaluations in education include judicial, congressional
hearings, and debate models.

5. Pluralist-Intuitionist Approaches, which consider the evaluator as “a
portrayer of different values and needs of all the individuals and groups served by
the program, weighing and balancing this plurality of judgments and criteria in a
largely intuitive fashion.” (Worthen, 1990, p. 45).

In addition to the classification schemes described in the foregoing discussion, other
category systems have been used as frameworks for conceptualizing evaluation models
and approaches. As stated earlier, evaluation models are often categorized by the
functions and needs they serve. For instance, evaluation models and procedures have
been categorized depending on whether they serve formative and summative functions.

Four types of formative evaluation have been identified in the literature on evaluation
models (Tessmer, 1993; Dick and Carey, 1991). These include one-to-one trials, small-

group trials, field trials, and expert review.

Computer Courseware Evaluation

Several models of computer courseware evaluation can be deciphered from the
literature on computer courseware. The traditional or classical model of computer
courseware evaluation involves comparison of computer courseware (CAI, CBE, CAL,
IV 1, etc.) to another method of instruction, such as a different form of instructional
technology, or traditional classroom instruction. The general thrust of the comparative

experimental studies model has centered on finding statistically significant effectiveness

49

differences between computer courseware and another method of instruction. For the
most part, the results of the comparative studies model of courseware evaluation, like
those from the forebear media studies, have been “no significant differences” in
effectiveness among technologies under comparison (Clark, 1983; DeBloois, 1988;
Reeves, 1991). The classical approach has revealed little or no information concerning
factors or features that influence the success of computer courseware. Information about
how to improve or develop better courseware is not engendered as well (Reeves, 1991, p.
90).

A second model of computer courseware evaluation is the courseware checklist
approach. Typically, a courseware evaluation checklist consists of items concerned with
“how well the courseware fits within the context of the total course, the appropriateness
of text and graphics presentations, test provisions, and to lesser extent, how well user
characteristics have been considered.” Criswell and Swezey (1984, p. 43). Courseware
evaluation checklists are often constructed as design evaluation criteria to guide formative
evaluation of the courseware development process. Typically, courseware checklists are
used in evaluating computer courseware for purposes of selecting educational software for
curriculum adoption. As Criswell and Swezey (1984, p. 43) pointed out, courseware
evaluation checklists are particularly useful “where experimental data are not available.”
A fundamental shortcoming of many courseware checklists is their lack of grounding in
theory of instruction.

Another model of computer courseware evaluation is the computer modeling

approach. Reeves (1991, pp. 93-95) delineated two forms of computer modeling

50

approaches to understanding the impact of computer courseware. The first approach is
program modeling which involves structured decomposition. Reeves (1991 , p. 94))
defined structured decomposition as “the breaking of an instructional program into its
component parts from the broadest level of global conceptualization to the lowest level of
generality at which meaningful decisions can be made.” Once an instructional program is
structurally decomposed, the “program components as well as input and output measures
relevant to the educational context can be analyzed using statistical procedures such as
commonality analysis...” (p. 94).

The second approach to computer modeling “involves adapting existing theories of
instruction into a model” for analyzing technology-based interactive instruction.
According to Reeves (1991, p. 94), “Extensive measurements of each element in the
model will be subsequently analyzed using a variety of path analysis and ‘soft modeling’
approaches The goals of this analysis are to test the overall predictability of the model
and to estimate the relative instructional effectiveness of each of the model’s elements.”
One way to operationalize the computational modeling approach is “to construct models
of the effective dimensions of IMM (interactive multimedia) such as interactive video,
collect relevant data, analyze the data with computer modeling methods, and thus
improve our understanding of effective instructional dimensions “bit by bit.” Reeves was
deeply convinced in the promise that the computational modeling approach holds to
“provide us with the new ‘tongues’ we need to preserve complexity and advance the

science and art of IMM.” (1991, p. 95).

51

Like other forms of courseware evaluation, however, the computational modeling
approach presents difficulty in estimating the impact of computer courseware. As
Reeves (1991, p. 94) eloquently stated the problem, “direct measurement is elusive.”

Human learning is a complex phenomenon. The complexity is further compounded
when the context and means of pedagogy involves computer courseware. No one
methodology can provide unequivocal support for the worth or usefulness of computer
courseware. As such, a meta-analytic or multidimensional methodological model of
courseware evaluation is required to obtain authentic and sufficient data to establish what
works, or does not, with regards to academic uses of computer courseware. The present
study coalesces salient features of different forms of courseware evaluation into a
framework model tailored for evaluating characteristics of computer courseware in

promoting effective learning.

Multidimensional Scaling (MDS) Procedure
Wilkinson, et al. (1992, p. 124) defined multidimensional scaling (MDS) as “a
procedure for fitting a set of points in a space such that the distances between points
correspond as closely as possible to a given set of dissirnilarities between a set of
objects.” The set of objects may be “stimuli, individuals, or the relationships between
stimuli and individuals” (Jones and Koehly, 1993, p. 95).
The input for the MDS procedure can be either proximities data or preference data

(Donthu and Rust, 1989; Kruskal and Wish, 1978). While proximities data represent how

52

similar or dissimilar pairs of objects are, or are perceived to be, preference data represent
how objects are ranked. The greater the similarity between objects or variables, the closer
the data points in a spatial or geometric configuration that constitutes the MDS results.

The utility of the MDS procedure lies in its power to reveal the latent structure of
data points or relationships among variables (Kruskal and Wish, 1978). Jones and
Koehly (1993, p. 95) assert that multidimensional scaling “can be used to reveal and
quantify the structure of complex stimulus domains, isolate and identify individual
differences in perception, cognition, and preference, and measure changes in perceived
structure over time, across subject populations, and experimental interventions.” As a
data reduction technique, the MDS procedure “provides a visual representation of a
reduced set of data which is often easier to interpret than the data itself.” (Donthu and
Rust, 1989). It can be used to transform a series of unidimensional relationships into
multidimensional expressions, thereby allowing relationships to be shown in both visual
and numerical formats (Kress and Synder, 1994; Donthu and Rust, 1989). The MDS
procedure is thus especially robust in providing greater insight into patterns of
relationships between or among variables.

There are two major multidimensional scaling models. An MDS model can be either
metric or nonmetric (Donthu and Rust, 1989; Jones and Koehly, 1993). According to
Donthu and Rust (1989, p. 48), “Fully metric methods require ratio-scaled distances as
input. MDS uses ratio-scaled distances to find a configuration whose interpoint distances
are proportional to the input data.” The nonmetric MDS models may be “fully

nonmetric” or simply “nonmetric”. The fully nonmetric MDS models assume only rank

53
order of input distances. As Donthu and Rust (1989, p. 48) asserted, the objective of the

fully nonmetric MDS models “is to find a space of minimum dimensionality and
determine the rank order of each point on each dimension in turn.” One shortcoming of
the fully nonmetric MDS models is that they do not provide a configuration of points in
space. In short, the fully nonmetric output provides hardly any useful insight into new
information. In contrast, the simply nonmetric methods combine the best aspects of fully
metric and fully nonmetric approaches. According to Donthu and Rust (1989, p. 49),
“Nonmetric methods accept rank ordered (ordinal) input data and find a configuration
whose rank order of ratio-scaled distances best produces the original input ranks.” In
essence, nonmetric MDS models allow monotonic transformations of nonmetric data into
ratio-scaled output.

Jones and Koehly (1993, p. 104) listed four types of data that can be collected for the
MDS procedure: (1) any type of proximity measure for all pairs of stimuli; (2) objective
measures of the stimuli; (3) ratings of the stimuli along unidirnensional attribute scales;
and (4) personal information about the respondents. As they pointed out, “Information
of the last three types is used as an aid to interpretation of the dimensions and
configuration derived via MDS of the dissimilarities data, and to explore or identify
individual and group differences in perception or construal of the stimuli.” (p. 104).

The MDS procedure has been used in many fields for a variety of different purposes
(Donthu and Rust, 1989; Kress and Snyder, 1994; Kruskal and Wish, 1978). The
following list, from the editor’s introduction to the text by Kruskal and Wish (1978, p. 6),

describe some of the more common uses of the MDS techniques:

54

- Psychologists have used them to understand the perception and
evaluation of auditory stimuli (such as speech and musical tones), visual
stimuli (such as colors and faces), and social entities (such as personality
traits and social situations).

- Sociologists have used these methods to determine the structure of groups
and organizations, based on members’ perceptions of one another and their
interaction patterns.

- Anthropologists have used these methods for comparing different cultural
groups, based on their beliefs, language, and artifacts.

- Economists and marketing researchers have used these methods for
investigating consumer reactions to a wide variety of product classes.

- Educational researchers have used these methods to study the structure of
intelligence, of different test batteries, and of classroom environments.
In the present study, the multidimensional scaling procedure is employed to determine
specific patterns of relationships that may exist in the perceptions of the experts
regarding the level of importance of the characteristics of computer courseware in

promoting effective learning.

CHAPTER III

RESEARCH DESIGN AND METHODOLOGY

Introduction

This study was conducted to investigate factors and assess characteristics that
determine the success, or efficacy, of computer courseware designed to support students’
learning of concepts in college-level physiology. There were four purposes for
conducting the study. The first purpose was to determine the effects of using computer
courseware programs designed to teach physiology concepts on students’ learning
achievement. The second purpose was to assess the level of importance of computer
courseware characteristics in promoting effective learning as perceived by experts. The
third purpose was to assess the impact of the quality of the characteristics of the
computer courseware programs used in this study in promoting effective learning as
perceived by both students and experts. The fourth purpose was to construct an
evaluation framework that courseware developers, instructional designers, researchers,
project managers, curriculum designers, educators, administrators, program evaluators,
and manufacturers of software programs can use to analyze and assess computer
courseware programs for the characteristics of instructional quality, or efficacy.

The research design for this study utilized an achievement test to collect data on

students’ performance based on the use of computer courseware in the academic

curriculum under study. Three evaluation checklists, in the form of survey

55

56

questionnaires, were developed by the researcher and employed to collect data about
perceptions regarding the characteristics of computer courseware in promoting effective
learning from both the students who used the courseware programs in the Physiology 431
class and some selected experts. Statements describing those characteristics were rated by
one group of experts in terms of the level of importance of each characteristic and by a
second group of experts in terms of the quality of the characteristics as manifested in the

courseware programs under study.

Subjects

Three groups of subjects participated in this study. The first group consisted of the
students who used the three computer courseware programs under study. This group of
students took the Physiology 431 class taught by Dr. Thomas Adams, in the Physiology
Department at Michigan State University, during the Fall Semester of 1995. The letter
that requested the permission of the faculty to use the class and courseware programs for
the study is presented in Appendix D.

The students represented three categories of academic disciplines or career paths
which were: pre-medicine, natural science, and humanities. Random assignment of
student subjects to treatment groups was not possible due to ethical considerations. It
would have been unethical to allow some students to use all the courseware programs
while denying other students access to some or all of the courseware programs. However,
all the students in the Physiology 431 class were encouraged to use the courseware

programs under study on a voluntary basis. Dr. Adams had indicated that in the past,

57

while a vast majority of students in the class used the programs, a few students did not.
It was assumed, therefore, that the self selection process would operate in a similar way
in this study.

The second group of subjects consisted of experts selected mostly from Michigan
State University. A few other experts were selected from other institutions of higher
learning, but only one of them returned the survey questionnaires on time and with
sufficient or complete response. The experts were comprised of faculty with experience
in courseware development and integration into academic curriculum, staff with
professional careers in courseware development, instructional designers, educational
researchers or psychometricians, and educational psychologists. The experts rated the
level of importance of the characteristics of computer courseware programs in promoting
efl‘ective learning.

The third group of subjects consisted of experts with professional experiences similar
to those of the experts in the second group. They were drawn also mostly from Michigan
State University. These experts assessed the extent to which the quality of computer

courseware characteristics helped in promoting effective learning.

Instrumentation
Four instruments were used for the research data collection. The first instrument was
a set of achievement tests that were developed and administered by the faculty who
taught the course in which the courseware programs were used. The achievement tests

were administered to measure students’ learning achievement in Physiology 431 class

58

where the courseware programs under study were used. The other instruments were
constructed based on literature review (Good and Brophy, 1995; Driscoll, 1994; Kemp,
1994; Okey, 1991; Carey and Dick, 1991; Gagné et al., 1991; and Lillie and Stuck, 1989).

The second instrument, developed by the researcher, was the Computer Courseware
Evaluation Form (Appendix A) that students filled out in conjunction with using the
different computer courseware programs under study. The instrument had two parts.
Part A centered on the characteristics of courseware content and the characteristics of
instructional procedures used in the courseware programs. Measurement of Part A was
based on a six-point, Likert-type response that allowed the respondents to rate their
opinion or perception about the quality of the characteristics of the courseware programs
used in the study in promoting effective learning.

The rating scale ranged from “not applicable” to “superior” as follows:

1 - Not applicable (or absent) 4 - Average
2 - Inferior 5 - Above average
3 - Below average 6 - Superior

Part B of the Computer Courseware Evaluation Form requested students who used the
courseware programs in the Physiology 431 class and also volunteered to participate in
the study to complete the following demographic information: gender, age range in years,
GPA, academic discipline, class level, total credits earned at Michigan State University,
previous experience in using computer productivity tools, and computer ownership. The
student respondents also supplied information regarding the average number of times per

week courseware programs were used, average amount of time per week courseware

59

programs were used, their opinion on the helpfulness of the courseware programs in
learning the lesson materials presented, whether they would recommend the courseware
programs to other students, and whether they would like to use similar courseware
programs in other college courses. Due to the composition of the student group and the
nature of their responses concerning a number of demographic variables, some of the
personal demographic characteristics were rccoded after data collection in order to have
meaningful data analysis and interpretation.

Also, the students were requested, in open-ended questions, to write any comments
about the courseware programs used in the study. The comments were requested as a
way to uncover any features of the programs that may not be indicated as courseware
characteristics in Part A of the evaluation instrument but, nonetheless, were perceived by
the students as unique and helpful in effectively learning Physiology 431 lesson materials.

In order to establish the validity of the Computer Courseware Evaluation Form
instrument, the four doctoral committee members at Michigan State University initially
critiqued the instrument for internal consistency, appropriateness, and clarity of
questions. Similar information was obtained by pre-testing the instrument with a group
of students in the Physiology 431 class, the population from which the sample for this
study was drawn. The subjects who participated in the pilot study indicated that the
instrument was clear and meaningful. Comments provided both by the doctoral
committee members and students in the pilot study group were used in revising the

evaluation instrument.

60

The Cronbach’s alpha reliability coefficient was used to estimate the internal
consistency measure of the reliability of the evaluation instrument in measuring the
students’ perceptions regarding the quality of the characteristics of computer courseware
programs in promoting effective learning. Table 3-1 presents the alpha values obtained.

In general, 0.6 is the minimum acceptable alpha level of internal consistency (Ary,
Jacobs, and Razavieh, 1990, pp. 281-282). In this study, all of the items examined
relating to the Computer Courseware Evaluation Form had a reliability coefficient of
greater than 0.6.

Table 3-1: Cronbach’s alpha reliability coefficient of the items on the Computer
Courseware Evaluation Form

 

 

 

 

 

 

 

 

 

 

1 Description Number
1 of Items Index i
a The quality of representation of courseware l lesson 18 .9733
content
Ilb The quality of representation of instructional procedures 12 .9386
used in courseware 1
c The quality of representation of all courseware l 30 .9813
characteristics
d The quality of representation of courseware 2 lesson 18 .9696
content
n e The quality of representation of instructional procedures 12 .9554
used in courseware 2
f The quality of representation of all courseware 2 30 .9809
characteristics
g The quality of representation of courseware 3 lesson 18 .9763
content
h The quality of representation of instructional procedures 12 .9362
used in courseware 3
i The quality of representation of all courseware 3 30 .9824
characteristics
j The quality of representation of all characteristics in all 3 115 .9362
courseware ro s

 

 

 

 

61

In order to enhance the credibility of the study, the researcher requested the three
professors who team-taught the Physiology 431 class to explain the importance of the
study and encouraged the students to participate. The professors cooperated and
explained the significance of the study and the students’ participation in several class
sessions. In addition, a cover letter accompanied each questionnaire to explain the
purpose of the study and request the participation of the students in the study (see
Appendix E).

The third instrument was the Computer Courseware Evaluation Checklist Rating Form
(see Appendix B). The researcher designed the instrument to allow experts to rate their
perceptions regarding the level of importance of characteristics of computer courseware in
promoting effective learning. There were three parts to the instrument. Part A was
related to the courseware structure and contained three sub-parts, program structure,
screen layout and design, and media use. Part B was concerned about lesson content. It
contained nine sub-parts delineating Gagné’s Events of Instruction and Learning
Outcomes. Part C requested experts responding to the questionnaire instrument to
provide the following general information: gender, age range in years, professional
discipline, range of teaching experience in years, approximate number of courseware
programs developed, experience with computers, and computer ownership. The
instrument was designed to collect information from experts to establish weights for
characteristics of instruction and learning perceived to be critical to the success or efficacy

of computer courseware.

62

The rating scale used was as follows:

0 - Not applicable

1 - Low
2 - Medium
3 - High

The fourth instrument, Computer Courseware Evaluation Checklist Form, was
constructed to allow experts to evaluate the quality of the characteristics of computer
courseware programs used in this study in promoting effective learning (see Appendix C).
Experts were asked to interact with the three courseware programs under study -—Action
Potential, Control Systems, and Synaptic Mechanisms (Parts 1 and II) —— and then used
the evaluation framework to evaluate the programs. There were three parts to the
evaluation instrument. Part A dealt with evaluating computer courseware structure.
Structure referred to how the courseware programs were structurally and functionally
organized. Part B dealt with analyzing and evaluating courseware lesson content (i.e.,
instructional content and strategies). This part required experts to assess the learning
outcomes and instructional events manifested in the courseware programs under study.
Essentially, the experts assessed how facts, concepts, principles, procedures, and
attitudes were presented in the courseware programs and if they were orchestrated to
facilitate effective learning. One aim of this study was to determine whether courseware
types had any differential effects on students’ learning or performance achievement. To
that end, the evaluation instrument was constructed with three column matrices. Each of
the column matrices contained the same rating scales and was associated with one

particular courseware among the three computer courseware programs under study.

63
Measurement of both Part A and Part B was based on a six-point, Likert-type

response that allowed the respondents to rate their opinion or perception about the
quality of the courseware characteristics. Quality referred to the extent to which, or how
well, a particular characteristic was presented or represented. The rating scale ranged

from “not applicable” to “very good” as follows:

1 - Not applicable (or absent) 4 - Fair
2 - Very poor 5 - Good
3 - Poor 6 - Very Good

The “not applicable” rating meant that the particular characteristic did not apply to
the objectives of the courseware, or it was absent even though applicable, or that the
expert was not able to evaluate it. Blank spaces were provided within each section of
both Part A and Part B for the experts to include other characteristics or features of
computer courseware that they thought should be assessed and to then rate the
features using the same scale used for other features. Information thus collected along
with other comments and suggestions were used in constructing the final instrument,
“A Framework for Analyzing and Evaluating Computer-Based Multimedia
Courseware” in Chapter VI, Conclusions and Implications.

Part C of the Computer Courseware Evaluation Checklist Form focused on general
information about the courseware evaluator. This part requested the experts responding
to the questionnaire instrument to provide information about their gender, age range in
years, professional discipline, range of teaching experience in years, approximate number

of courseware programs developed, experience with computers, and computer ownership.

64

Also, the experts were requested, in open-ended questions, to write any comments about
the courseware programs used in the study. The comments were requested as a way to
uncover any features of the programs that may not be indicated as courseware
characteristics in Parts A and B of the evaluation instrument but, nonetheless, were
perceived by the experts as unique and helpful in effectively learning the lesson materials
that the courseware programs present. Such unique features, possibly latent in the
courseware programs, may provide additional information that leads to a better
understanding of the factors that influence the success or efficacy of computer
courseware.

Like the evaluation form used with the student respondents, several steps were taken
to ensure the validity of the data collection instrument used with the experts. First, the
instrument was developed based on the review of literature on academic uses of computer
courseware and on educational psychology principles of effective instruction and
effective learning. Researchers, faculty members who have developed and used computer
courseware in academic curriculum, as well as educational researchers were also consulted
in constructing the questions on the questionnaires or instruments. Second, the
instrument was initially critiqued for clarity and appropriateness of questions by the four
doctoral committee members at Michigan State University. The doctoral committee’s
comments and suggestions were used in revising the evaluation instrument. Third, the
revised questionnaire instrument was pre-tested to determine whether the questions on
the evaluation instrument were clear and meaningful from the respondents’ point of view

and whether the questions were able to elicit and measure the information desired.

65

Another reason for pre-testing the revised questionnaire was to determine the average
time it would take respondents to complete the instrument. The pilot study involved a
focus group comprising of ten experts (including one Physiology 431 graduate assistant)
drawn from various departments at Michigan State University and three regular students
from the Physiology 431 class. The focus group members were given the instrument a
week prior to the discussion meeting so that it could be reviewed and responded to ahead
of time in preparation for the discussion meeting. In the discussion meeting, the focus
group members shared and discussed their views and perceptions regarding the evaluation

instrument in terms of the following topics:

1. Clarity and adequacy of instructions or directions for using the instrument
2. Clarity of the questionnaire items
3. Adequacy of the instrument to elicit and measure the desired information

or whether the instrument missed any characteristics that should be
included as a way to measure the efficacy of computer courseware
4. Whether the instrument contained any redundant items
5. Whether any questionnaire item was confusing and should be
stated differently
The focus group discussion meeting was videotaped and audiotaped. The tapes as
well as notes of important comments and suggestions written during the meeting were
later analyzed for any information that might be necessary for further modification and to
better enhance the clarity and understanding of the instrument.
The instrument was checked for reliability. The Cronbach’s alpha reliability

coefficient was used to estimate the internal consistency measure of the reliability of the

evaluation instrument in measuring the experts’ perceptions regarding the quality of the

66

characteristics of computer courseware programs in promoting effective learning. Table

3-2 presents the alpha values obtained from the reliability analysis.

Table 3-2: Cronbach’s alpha reliability coefficient of the items on the Computer
Courseware Evaluation Che Rati Form

        

 

 

 

 

 

 

of Items Index 1

a The level of importance of the characteristics of “program 19 0.7786
structure” in promotirg effective learning

b The level of importance of the characteristics of “screen 14 0.8622
layout and design” in promoting effective learning

c The level of importance of the characteristics of “media 5 0.8797
use” in promoting effective learnifi

"d The level of importance of the characteristics of 12 0.9203

“prerequisite skills” in promoting effective learning

e The level of importance of the characteristics of 30 0.9137
“presenting new lesson content” in promoting effective

_1__°_£
The level of importance of the over all characteristics in 80 0.9592
romotin effective learnin via com uter courseware

 

 

 

 

   

Questionnaire Distribution and Data Collection

As explained earlier, each of the three subject groups that participated in this study
responded to a particular instrument designed to collect a set of specific data from the
group. Accordingly, separate steps were taken in distributing the questionnaire
instruments to the groups.

Three main steps were taken in handling the evaluation instrument with the student

group. The first step was to obtain the permission of the professor who developed the

67

courseware programs under study and also taught the Physiology 431 class in which the
programs were used (see Appendix D). In addition to getting Dr. Adams’ permission to
use his courseware programs and the Physiology 431 class for the study, I sought and
obtained his consent and support to introduce me and to explain the purpose and nature
of the study to the class, as well as to request that the students participate in the study
on a voluntary basis. The cooperation of both the faculty and students continued
throughout the semester. For instance, before the beginning of several class lectures, Dr.
Adams or one of the other two professors who team-taught the class reminded the
students to complete and return the evaluation questionnaire to me. The introduction,
announcements, and reminders were done verbally, accompanied by projection of
overhead transparency notes. The second step in distributing the questionnaire
instruments involved the researcher going to the classroom and handing out copies of the
instrument to the students who volunteered. A cover letter explaining the purpose and
nature of the study, as well as obtaining the students’ voluntary consent to participate in
the study accompanied the evaluation instruments (see Appendix E). The third step with
the student group involved the researcher visiting ten class lecture sessions, usually 15
minutes before and after the lectures, to collect completed questionnaires. Ten class visits
were made to ensure a high questionnaire return rate.

In the case of the expert groups, four main steps were taken in distributing the
evaluation instruments. First, the researcher sent introductory letters, email messages, or
made phone calls to various experts primarily at Michigan State University (see

Appendix F). A few other experts were contacted at other institutions of higher learning

68

in the mid-Michigan area to seek their voluntary consent to participate in the study.
Most of the experts gave their consent verbally over the phone. A few experts agreed to
participate through email. The second step taken was to hand-distribute the evaluation
instruments to the experts who volunteered to participate in the study. In most cases,
the researcher delivered the instruments to the experts in their offices. In one instance,
however, the instrument was delivered to an expert at home who, in turn, distributed
copies of the instrument to two other experts. A cover letter explaining the purpose and
urgency of the study accompanied the evaluation instruments (see Appendix G). The
third step in handling the evaluation questionnaires with the experts involved follow-up
phone calls or email messages to the experts to ensure a high return rate. With the
exception of the experts who had returned their questionnaires with their names written
on them before the follow up messages were sent, the experts were sent a generic message
to avoid the feeling of being singled out and thereby offended (see Appendix H).

The fourth step involving the experts was concerned with the collection or returns of
the evaluation questionnaires. Although all the evaluation instruments were accompanied
by self-addressed stamped envelopes, most of the experts returned their questionnaires
along with the envelopes to the researcher by hand. Since most of the experts were from
Michigan State University, some of them returned the questionnaires and envelope via
campus mail. Both means of returning the instruments and envelopes saved the
researcher a lot of time and money in data collection. A few of the experts returned the

questionnaires using the self-addressed stamped envelopes.

69

Data Analysis

Analyses of data for this study were conducted under three major parts. Part one
examined data collected from the students in the Physiology 431 class. Part two
examined data collected from the group of experts who responded to the Computer
Courseware Evaluation Checklist Rating Form. Part three focused on data collected fi'orn
the group of experts who used the courseware programs and responded to the Computer

Courseware Evaluation Checklist Form.

Analging Data Collected from the Student Group

Part one of the data analyses quantitatively addressed research questions A and B,
along with their subsidiary questions, using data obtained from survey of students and
from course grades. The aim of data analyses in part one was to determine the effects of
courseware usage and the factors that predict students’ learning achievement in
Physiology 431 course. As a precursor, simple descriptive statistics using the Statistical
Package for the Social Sciences (SPSS) were obtained in terms of frequencies, percentages,
means and standard deviations. Frequency and percentage distributions were utilized to
represent the demographic characteristics of the subjects. In order to determine the
effects of courseware usage on the students’ learning achievements, one way analysis of
variance (AN OVA) was used to test for statistical significance of mean differences in

adjusted course grades among students who used all the courseware programs, those who

70

used one or two programs, and those who did not use any of the programs. The .05
alpha-level, which is conventional for behavioral social science research (Cohen, 1977),
was used for all tests of statistical significance in this study. Once differences in adjusted
grade scores were established among the groups of students, the Tukey post-hoe test
analysis was conducted to determine which group of students performed better. Further
analysis on the effect of courseware usage on the students’ learning achievement was
undertaken using stepwise regression analysis. The purpose of the regression analysis
was to determine the extent to which the amount of courseware usage predicted students’
learning achievement while controlling for the students personal demographic
characteristics.

The next phase of part one of the data analyses centered on the students’ perceptions
regarding the quality of the computer courseware programs in promoting effective
learning. The aim of this phase of the analyses was two-fold. One aim was to determine
whether the students’ perceptions regarding the quality of the courseware programs in
promoting effective learning were related to and/or predicted their learning achievements.
The second aim was to determine whether the students’ perceptions were related to
and/or predicted by their demographic characteristics. For parsimony and economy in
managing and conducting the data analyses, related characteristics on the courseware
evaluation questionnaire were constituted into sets or groups of composite variables.
Following the categories used for the items on the computer courseware evaluation
questionnaire responded to by the students (see Appendix B), three sets of composite

variables were formed for each courseware as: courseware structure, media use, and

71

instructional procedures. A total of nine sets of composite variables were thus formed for
the three courseware programs used in the study. Correlation procedure was employed
to determine relationships between students’ learning achievements and their perceptions
regarding the quality of the courseware programs in promoting effective learning. One
way analysis of variance (AN OVA) was then used to determine whether the students’
perceptions of the courseware characteristics varied by their demographic characteristics
such as gender, academic discipline, class level, and computer ownership. The mean,
standard deviations, F-values, and P-values were obtained and analyzed for composite
sections of each computer courseware program. Further analyses, using stepwise
regression procedures, were conducted to determine whether learning achievement was
predicted by perceptions and also whether perception was predicted by the students
demographic characteristics.

An analysis was also conducted on the comments that the students provided to open-
ended questions concerning other features of the courseware programs that they perceived
as helpful in promoting effective learning. The responses were compiled and categorized

under concepts or themes that emerged from the contents of the comments.

Analging Data of Respondents in the Expert Rating Form Group

Part two of the data analyses quantitatively addressed research question C and its
subsidiaries. Simple descriptive statistics in terms of frequencies and percentages were
employed to represent the distribution of the respondents in the Expert Rating Form

Group. Other descriptive statistical measures were computed in the form of means,

72
standard deviations, and ranks on the experts’ perceptions regarding the level of

importance of computer courseware characteristics in promoting effective leaming. The
aggregate mean ratings of the perceptions were computed based on the three-point Likert-
type response which ranged from “low” to “high” as follows:

1 - Low

2 - Medium
3 - High

The mean ratings were interpreted as follow:

1.00 - 1.99 — Low

2.00 - 2.50 — Medium

2.51 - 3.00 — High

As in part one, related characteristics on the Courseware Evaluation Checklist Rating

Form were constituted into a set of composite variables. Following the categories used
for the items on the computer courseware evaluation questionnaire responded to by the
experts (Appendix C), five sets of composite variables were formed for each courseware
as: courseware structure, screen layout and design, media use, prerequisite skills, and new
lesson content. One way analysis of variance (AN OVA) was used to examine the
relationships among the experts’ perceptions regarding the level of importance of
courseware characteristics and their demographic characteristics such as: gender, academic
discipline, class level, and computer ownership. The mean, standard deviations, F-values,

and P-values were obtained and analyzed for composite sections of each computer

courseware. Further analysis, using stepwise regression procedure, was conducted to

73

determine whether the experts’ perceptions were predicted by their demographic
characteristics.

A more detailed and intensive analysis was conducted to identify differences and
similarities in the way that characteristics that can promote effective learning via
computer courseware were perceived. The experts’ demographic characteristics were not
pertinent to the analysis and therefore not included. Multidimensional scaling technique
(MDS) was used to examine the data to determine whether patterns of structure or
relationships existed among the characteristics as perceived by the experts. The MDS
procedure was used only in this phase of the data analyses where there were enough cases
or variables to compute proximity measures for the data collected.

In order to obtain the clusters or patterns of relationships, distances among all the
characteristics that can promote effective learning via computer courseware were
calculated from the raw data using SPSS. A three dimensional Euclidean distance matrix
was used as the scaling model. The MDS procedure produced three dimensional
coordinates of all the variables. The results were imported into Systat to generate a three-
dimensional plot of the relationships matrix.

An analysis was also conducted on the comments that the experts provided to an
open-ended question concerning other features or characteristics not listed on the
questionnaire that they thought were important in promoting effective learning via
computer courseware. The responses were compiled and categorized under concepts or

themes that emerged from the contents of the comments.

74

Analyzing Data of Respondents in the Expert Courseware Evaluation Grow

Part three of the data analyses quantitatively addressed Research Question D and its
subsidiaries. The aim of the data analyses was to determine what the experts’
perceptions regarding the quality of the courseware programs used in the study were as a
measure of the potential of the programs to engender effective or successful learning.

Simple descriptive statistics in terms of frequencies and percentages were employed to
represent the composition of the respondents in the Expert Courseware Evaluation
Group who used the courseware programs and responded to the Computer Courseware
Evaluation Checklist Form. Both descriptive and inferential statistics were used to obtain
a profile of the experts’ perceptions.

The first step in profiling the experts’ perceptions was to form composite variables
from a set of related characteristics on the courseware evaluation questionnaire. Five sets
of composite variables were formed for each courseware, making a total of 15 sets of
composite variables for the three courseware programs used in the study. One way
analysis of variance (AN OVA) was used to determine whether there were significant
differences among the experts in their perceptions regarding the quality of the courseware
characteristics by their demographic characteristics such as gender, age, professional
discipline, teaching experience, number of courseware programs developed, previous
experience with using computers as productivity tools, and computer ownership. The
mean, standard deviations, F -values, and P-values were obtained and analyzed for

composite sections of each computer courseware. Further analyses, using stepwise

75

regression procedures, were conducted to determine whether the experts’ perceptions
vary by their demographic characteristics.

The last data analysis conducted centered on the responses to open-ended questions
concerning other features of the courseware programs that the experts perceived as
helpful in promoting effective learning. The responses were compiled and categorized

under concepts or themes that emerged from the contents of the experts’ comments.

Summary

This chapter contains the design and methodology used in this study. It describes the
composition of subjects involved in the study. Four instruments used for the study were
described. The first instrument was simply the exams administered to the students in the
Physiology 431 class. The other three were evaluation survey questionnaires which were
tested with samples of the target population to establish their content validity. The
process of questionnaire distribution and data collection was discussed.

Next, data analyses procedures were presented. The analyses were conducted in three
parts. Part one addressed the research questions concerned with the Student Group. Part
two addressed the questions concerned with experts who responded to the Computer
Courseware Evaluation Checklist Rating Form. Part three addressed the questions
concerned with experts who responded to the Computer Courseware Evaluation
Checklist Form. Regression analysis was used to determine the extent to which

courseware usage predicted the students learning achievement. Both simple descriptive

76

and inferential statistical procedures were used to determine relationships and differences
in the experts’ perceptions regarding the characteristics of computer courseware programs
in promoting effective learning. In the case of the experts’ perceptions regarding the level
of importance of characteristics that promote effective learning via computer courseware,
a more in-depth and intensive analysis was conducted using multidimensional scaling
technique. The MDS procedure was explained.

Comments to open-ended questions were compiled and categorized under concepts or

themes that emerged from the contents of the comments. The comments were analyzed.

CHAPTER IV

PRESENTATION AND ANALYSIS OF DATA FROM STUDENTS

Introduction

The data presented in this chapter was gathered from the students in Physiology 431
at Michigan Sate University. The data was gathered during the Fall semester of 1995.

Over 400 students registered for the Physiology 431 class during that semester,
however, only 55 students volunteered and participated in the study. Nonetheless, data
on exam grades for most of the students, as well as data on which students used and
which did not use the prograrhs were available and, therefore, collected. The only
exceptions were a few students who had incomplete grade records or those who did not
provide sufficient information on the data collection instruments used in the study. For
instance, students who skipped multiple sections of the evaluation questionnaire or those
who did not correctly indicate their student identification numbers on the form that asked
them to indicate courseware programs used were not included in the data analyses. The
student identification number was a necessary identifier in coding students’ demographic
variables to match exam grades.

The report in this chapter is presented in three sections. Section one presents the
demographic descriptions of the students in the Physiology 431 class. It presents
information on the students’ participation and/or use of the courseware programs under

study. Section two addresses the questions relating to the effects of computer

77

78

courseware on students’ learning achievement. Section three contains information about
the students’ perception regarding the quality of the characteristics of the computer

courseware used in this study.

Section One — Description of Students’ Demographics
There were three major groups of students involved in this study. The first group

consisted of 76 (19.2%) students who neither used the courseware programs nor
responded to the evaluation questionnaire that accompanied the programs. The second
group consisted of 265 (66.9%) students who used the programs, but did not respond to
the evaluation questionnaire. The third group consisted of 55 (13.9%) students who used
the programs and responded to the questionnaire. Nearly all of the students in this third
group, 53 (96.4%), used all the three courseware programs under study. Of the two
exceptions, one student used only courseware 1 (Action Potential) and the other student
used only courseware 2 (Control Systems). None of the 55 students in the third group
self-selected to use courseware 3 (Synaptic Mechanisms 1 and II) alone or to use a
combination of courseware l and 2, courseware 1 and 3, or courseware 2 and 3. The self-
selection nature of the study made it difficult to have an even distribution of the research
participants to different study groups. The frequency distribution of the students’ use of

courseware and their participation in the research questionnaire is presented in Table 4-1.

7 9
Table 4-1

The distribution of the student respondents by
use of courseware and participation in research questionnaire

 

 

 

 

(n=396)
Nocourseware useand no participation Mi 6 129.
Used courseware but no participation 265 66.9
Used courseware and participated 55 13.9

 

 

 

 

 

Of the 265 students who used the courseware programs, but did not participate in

responding to the evaluation questionnaire, a frequency distribution of the programs they

used is presented in Table 4-2.

Table 4—2

The distribution of students who used courseware programs
but did not participate in research questionnaire
by courseware programs used.

f n=265 _
--
No No.
Courseware 1 — Action Potential 98. 9

 

 

rCourseware 2 -— Control Systems |24o 90.6 25 9.4 I265 100
CoursewareB—S na-tic Mechamsms 243 91.7 22 8.3 265 100

 

 

Of the 55 students who used the courseware programs and participated in responding

to the research questionnaire, a frequency distribution of their responses regarding the

programs they used is presented in Table 4-3.

8 0
Table 4-3

The distribution of the student respondents by
use of courseware and participation in responding to research questionnaire

  

(n=55)
--
7 ,, s No No.

‘ Courseware 1 — Action Potential 54 98.2 100

      

 

I Courseware 2—c6mml Systems I 54 98.2 1 1.8 I 55 100 JI
Courseware 3 - S na tic Mechanisms 53 96.4 2 3.6 55 100

 

A total of 150 questionnaires were distributed to the students in the Physiology 431
class. Out of that number, 55 questionnaires were adequately completed and returned.
The data analyzed and presented in this chapter were derived primarily from those
sources. The total return rate was 36.7%. The distribution of the respondents by

academic discipline and gender is presented in Table 4-4.

Table 4-4

The distribution of the student respondents by academic discipline and gender

1 Gender Pre- Natural Humanities Other Total
Medicine Science
7 No. % No. % No. % ,, No. % No. %

 

 

Male 19 34.5 6 10.9 1 25 46.3
Female 21 38.2 5 9.1 1 1.8 2 3.6 29 53.7
Total 40 72.7 1120 f 1 g 1.8 2 , 3.6 54* 100

" One missing value observed
Of the 55 valid respondents, 40 (72.7%) were pre-medical students, 11 ( 20%) were

majoring in natural science disciplines, one (1.8%) was in humanities, and two (3.6%)

81

were in the “other” category. The “other” category consisted of one graduate student and
one Life Long Education student. Of these 55 student respondents, 25 (46.3%) were
males, 29 (53.7%) were females, with one missing value observed.

In terms of distribution by age, the student respondents consisted of four (7.3%) who
were below 20 years, 41 (74.5%) between 20 and 25 years, six (10.9%) between 26 and
30 years, and three (5.5%) over 30 years.

The students’ GPAs, before taking the Physiology 431 class and, therefore, prior to
using the courseware under study, ranged between 2.5 and 4.0. Table 4-5 presents the
GPA distribution.

Table 4-5

The distribution of the student respondents by GPA

under 3. 0
3.0 3.5

3.51 - 4.0
Missin values

 

 

 

 

Several other classification schemes were used in studying the nature of the student
respondents. By class level, the student respondents consisted of four (7.4%)
sophomores, 26 (48.1%) juniors, 17 (31.5%) seniors, seven (13%) graduates, with one
missing value observed. A barchart of the class level distribution in presented in

Figure 4-1.

82

Figure 4-1
The distribution of the student respondents by class level
30 q-
25 a

20‘

15‘

10'

 

 

Sophomore Junior Senior Graduate

The type of previous experience with computers is another classification scheme used.
The distribution of the student respondents according to previous experience in using
computer productivity tools is presented in Table 4-6.

Table 4-6

Number and percentage of the student respondents
with revious ex rience usin com uter roductivi tools

 

 

 

 

 

 

 

 

Type of With previous With no previous Missing Total
Computer experience experience Values

Ex rience No. % No. % No. % No. %
Wordprocessing 50 90.9 4 7.3 l 1.8 55 100
Spreadsheet 38 69.1 16 29.1 1 1.8 55 100
Database 29 52.7 25 45.5 1 1.8 55 100
Communication 46 83.6 8 14.5 1 1.8 55 100
Pro ' 26 47.3 28 50.9 1 1.8 55 100
Multimedia 14 25.2 40 72.7 1 1.8 55 100
Games 41 74.5 13 23.6 1 1.8 55 100
Courseware 32 58.2 22 40 1 l .8 55 100
Assembling 6 10.9 48 87.3 1 1.8 55 100
Corn uters

 

 

 

 

 

83

The respondents were asked to indicate whether or not they owned computers. 29
(52.7%) of the students responded yes, 24 (43.6%) responded no, with two (3.6%)
missing values observed. Also the respondents were examined as to the total credits
earned at Michigan State University prior to taking the Physiology 431 class and using
the courseware programs under study. The distribution of the responses ranged from
zero to 140 credits. This distribution is presented in Table 4-7.

Table 4-7

The distribution of the student respondents by total credits earned at MSU

MSU Cams

0.
8

22

11
5
9

55

 

 

 

 

The student respondents were asked whether or not the courseware programs helped
them to better learn the concepts presented in Physiology 431 class. Nearly all the
students, 53 (96.4%), responded positively. The only exceptions were 2 (3.6%) missing
observation values.

Finally, the students were asked as to whether or not they would like to have similar
courseware programs in other college courses. 53 (96.2%) respondents said yes; only

two (3.6%) respondents said no.

grat
whi
Vari.
0011;
any I
for a
Pics:
of 5n
used
No I3

Mgr;

84
Section Two — The Effects of Computer Courseware on Students’

Learning Achievement.
This section examined the effects of computer courseware usage on students’ learning
achievement. Data analyses for the section were guided by research question A and the

subsidiary questions under it.

Research Question A1

Does courseware usage predict students’ level of learning achievements?

The final grade in Physiology 431 class, as measured by the adjusted raw score, was
used as the outcome in this analysis. The adjusted raw score was a linear combination of
grades for all three exams written in the Physiology 431 class during the semester in
which the use of the courseware programs in the class was studied. One way analysis of
variance (AN OVA) was used to compare differences among students who used all the
courseware programs, those who used one or two programs, and those who did not use
any of the programs in terms of the adjusted raw score. This information was available
for all 395 students who attended the Physiology 431 course. The AN OVA results
presented in Table 4-8 indicate a statistically significant difference among the three groups
of students (F=24.75, P<0.05). Tukey’s post-hoc results revealed that students who
used all three programs significantly performed better than those who used just one or

two programs, who in turn, performed significantly better than those who did not use any

program.

 

If

51'.

85

Table 4-8

One way analysis of variance (AN OVA) results of differences in learning achievements
(as measured by adjusted raw scores) among students who used and those who did not
use the courseware-1m -

-m

 

Groupl 80. 2857 20. 0150 24. 7460 0.000 1
(No Courseware Used)
Group2 44 82.4091 15.7721

(Courseware l and 2,
Courseware l and 3 or
Courseware 2 and 3)

 

 

 

 

 

 

 

Group 3 275 94.3818 14.9448
(Courseware l, 2 and 3)
. Total - 396 90. 7551 17.0797 ,.

In sum, the students who used the courseware programs significantly performed better
than those who did not. In addition, the more courseware programs used, the better the

students’ performance.

86

Research Question A2
To what extent is the students’ level of learning achievement dependent on
factors other than the use of the computer courseware, for instance,
demographic characteristics (gender, age, GPA, academic discipline, class
level, previous experience with computers, and computer ownership)?

The final grades in the Physiology 431 class, as measured by the adjusted raw scores,
was further examined to determine if students’ learning achievements were due mainly to
the use of the computer courseware or whether other factors account for the observed
significant differences in the adjusted raw scores. Unlike Research Question A1,
information for Research Question A2 was available for only those 55 students who
responded to the questionnaire. Two stepwise regression models were constructed to fit
the data.

The first regression model was designed to determine whether the amount of
courseware usage alone predicted learning achievement as measured by adjusted raw
scores. The amount of usage of courseware l, courseware 2, and courseware 3 were used
as predictors. The amount of courseware usage was computed as the product of the
average number of sessions a program was used per week by the average amount of time

(in minutes) spent on the program per session. As Table 49 indicates, the amount of

usage of courseware 2 alone was a significant predictor of the adjusted raw score (I3=

0.331, P<0.05). For this predictor, one unit increase in the use of courseware 2 improves

the adjusted raw score by 0.33 standard deviations. Usage of courseware 2 alone

pr:
of
81':

ha

0“

87

accounts for 10.9% of the variance in learning achievement as measured by adjusted raw
scores.

The second stepwise regression model was designed to determine the extent to which
the amount of courseware usage predicted students’ level of learning achievement while
controlling for other personal demographic characteristics of the respondents. Adjusted
raw score was used as the outcome. Factors used as control variables were gender, GPA,
academic discipline, class level, previous experience using computer courseware or
multimedia programs, previous experience in computer programming, previous experience
using databases, previous experience using spreadsheet, and computer ownership. The
selection of these control variables was a function of the availability of sufficient data for
any particular variable. Table 49 presents the results of the analysis. The results
revealed that the main predictors of adjusted raw scores were the students’ GPA and
gender. Students with better GPAs performed significantly better than students with

lower GPAs (B= 0.472, P<0.05). A difference of one standard deviation in GPA was

predicted to cause a difference of 0.472 standard deviation in adjusted raw score. In terms
of gender, male students performed significantly better than female students where, on
average, achievement score for male students was about 0.338 standard deviations above
that of female students.

In sum, when computer usage is the sole predictor of adjusted raw score, the amount

of usage of courseware 2 (i.e., Control Systems) is the only significant predictor of

88

learning achievement. The effect is greatly reduced, however, when the students’ gender

and GPA were entered into the prediction model.

Table 4-9

Regression results for the prediction of adjusted raw scores (learning achievements)
‘ by the students’ demographic characteristics

Model Step Predictors B -2- P-value n

Adjusted Courseware 2 .331 00145"

Score (Control Systems)
GPA 0.472 0.0002“

 

 

 

 

 

 

 

 

 

vGender g 2- ,, ,, 0.338 f g 0.0065"I

" Significant at 0.05 level.
Model 1 — Using only courseware usage as predictors
Model 2 — Using courseware usage and demographic characteristics as predictors.

Section Three — Students’ Perceptions Concerning Quality of
Computer Courseware
This section answers the research questions relating to the student respondents’
perceptions regarding the quality of the characteristics of computer courseware programs
under study. The students were asked to rate their perceptions regarding the quality of

the characteristics of the programs in promoting effective learning. The Computer

89

Courseware Evaluation Form (see Appendix A) was used in collecting data to answer the
research questions relating to the students’ perceptions.
The major question guiding this section of the report was
Research Question B:
What are the student respondents’ perceptions regarding the quality of the
characteristics of computer courseware programs used in this study?
The subsidiary questions were:
Research Question Bl: What are the student respondents’ perceptions regarding
the quality of the characteristics of the computer courseware programs used in
this study in promoting effective learning?
Research Question B2: Do the student respondents’ perceptions regarding the
quality of the computer courseware characteristics vary by their demographic
characteristics such as: gender, class level, academic discipline, previous experience
using computer productivity tools, previous experience using computer
courseware, previous experience with computer programming, or computer

ownership?

Research Question B3: What other features of the courseware programs do the
student respondents perceive as unique and helpful in promoting effective

learning?

Students’ perceptions regarding the quality of the characteristics of the computer
courseware programs in promoting effective learning

The student respondents were asked to rate how they perceived the quality of each
courseware program under study in promoting effective learning. The mean of the quality
ratings and standard deviations were computed for the characteristics of each composite

section of the courseware programs. The possible mean ratings ranged from the lowest,

90

1.00, indicating that the set of composite characteristics was rated inferior, to the highest,
5.0, indicating that the set of composite characteristics was rated superior. Table 4-10
presents the means, standard deviations, and ranks of the students’ perceptions regarding
the quality of the characteristics of the courseware sections.

From the information on the table, two patterns of mean ratings of students’
perceptions of quality were observed. The first pattern relates to how the courseware
sections compared or were perceived across the three programs. The mean ratings of
quality of instructional procedures characteristics were higher (4.34, 4.33, and 4.29) than
the mean ratings of courseware structure characteristics (4.28, 4.27, and 4.26), which, in
turn, were higher than the mean ratings of the characteristics of media use (4.13, 4.13,
4.06). The second pattern of perception of quality relates to how the three courseware
programs compared in terms of the mean ratings of quality of each specific courseware
section. The highest mean ratings in each courseware section went to courseware 3, the
second highest mean ratings in each courseware section went to courseware 1, except in
instructional procedures which received third place mean rating. The third and last mean
ratings in each courseware section went to courseware 2, except in instructional
procedures section that received second place mean rating in that section category. In all,
the students perceived courseware 3 as the best in terms of the quality of courseware
characteristics in promoting effective learning, followed by courseware 1, and then

courseware 2.

’1'.)

91
Table 4-10

Means, standard deviations, and ranks of the students’ perceptions regarding the quality
of the characteristics of sections of the courseware programs under study

   
    

   

 

  

  
   
   

 

 

H‘
Courseware l 4. 27 5 ‘

] Courseware Structure Courseware 2 55 4.26 .54 6

' Courseware 3 55 4.28 .51 4
Courseware l 55 4.13 .88 7

I Media Use Courseware 2 55 4.06 .90 9

' Courseware 3 55 4.13 .83 7

5 Courseware 1 55 4.29 .61 3

I Instructional Procedures Courseware 2 55 4.33 .59 2
Courseware 3 1

 

 

 

 

 

 

The results of mean ratings of the quality of each courseware component were

correlated with the students’ grades to determine whether there was a significant
relationship between the students’ perception regarding the quality of courseware
programs and their learning achievement. The results of scatter plots and correlation
analysis indicated no linear or even curvilinear relationship between grades and perception
of courseware quality. The perception of courseware quality had no significant
relationship with students’ learning achievement.

A similar result was obtained in analyzing the data to determine any relationship
between students’ perceptions regarding the quality of courseware characteristics and
amount of courseware usage. No significant relationship was observed between

perceptions regarding the quality of courseware characteristics and learning achievement.

92

Effects of demographic characteristics on the student respondents’ perceptions
regarding the quality of the characteristics of the courseware programs under
study

One way analysis of variance (ANOVA) was used to determine whether students’
perceptions regarding the quality of computer courseware characteristics vary by their
demographic characteristics such as gender, academic discipline, class level, and computer
ownership. The results are presented in Tables 4-11 through 4-16.

As the tables indicate, no significant differences in the students’ perceptions were
observed between or among groups by gender, academic discipline, computer ownership
or, surprisingly, previous experience with computer programming. However, significant
differences in perceptions regarding the quality of the characteristics of certain courseware
sections were observed among the students by their class level and previous experience
using computer courseware.

As shown in Table 4-11, junior students differed significantly from senior students in
their perceptions regarding the instructional procedures component of all three
courseware programs — courseware 1 (F=7.0, P=.01<.05); courseware 2 (F=4.9,
P=.03<.05); courseware 3 (F=6.3; P=.02<.05). In all three courseware programs, the
junior students were more positive in their perceptions regarding the quality of
instructional procedures characteristics than the senior students — courseware 1(mean for
juniors = 4.41, mean for seniors = 3.95); courseware 2 (mean for juniors = 4.41, mean for

seniors = 4.03); courseware 3 (mean for juniors = 4.44, mean for seniors = 4.02).

N0 SI."

qualir;

93

However, both class levels did not differ in their perceptions regarding the quality of
other dimensions of the programs.

As shown in Table 4-12, significant differences in perceptions regarding the quality of
the characteristics of certain courseware sections were observed between student
respondents with previous experience using computer courseware and those without
previous experience. Students who had no previous experience using computer
courseware had more positive perceptions of the quality of courseware characteristics
than students who had previous experience in terms of the following:

Courseware 1: courseware structure (F=8.65, P=.005<.05)
media use (F=4.76, P=.34<.05))

instructional procedures (F=5.89, P=.02<.05)

Courseware 2: courseware structure (F=5.63, P=.02<.05)
instructional procedures (F #74, P=.03<.05)

Courseware 3: courseware structure (F=7.87, P=.01<.05)
instructional procedures (F=5.46, P=.02<.05)
No significant difference was observed in either group in their perceptions regarding the

quality of media use in courseware 2 and courseware 3.

94

8. v ._ 380.008 .
1 8.38.. .5008. I an 50.10: n z.

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

.505 2: m0 v30. w 5.0. S 003.. 3.0 an 005.0000:— _0:0_.0=.=m=_ II n 00038500
0m _ m 38.. E. :u and S mums... 08.0. on 8D 0:002 I m 0.5308500

, mum. N038 gun. E _.v 2 23. n33. em 05.086 0.8308500 I. m 0.338500
.88 wmowé 2.3. 58.0 2 2.3.. :10 em 005—0000...— _0:0_.0.50=_ .l N 0.338500
. ~52. nth.— wamh. mead E a fin. 0:3” mm 8D 0:002 l. N 05308500
Sam. 3.0... $2. 0. .6 2 new? and. 0N 05.03% 0.538500 In N 0.338.500

. 4m. 5. 08.: ~30. 036 t 2.3. 80.0 0N 0.05—0000.5 .50505051 _ 0.5308500
09%. 88. 035. 03.0 2 2.3 .36 on 8D 0:002 I. _ 005.008500
. :2. 8E 33.11.21 I was : mm. 1. .6.“ -i mass wssasso In mafiatmso,

02.3-.— 0=_a>-...— n— m :00: z a m :00: 2

00.000 .1 ,- . 0.4.5:. I m:d.§._nwalw0 000308000

 

 

.05. 88.0 .3 $000.80 .9 30.08.30 500. 00:00 0. 505 wet—0: 5 00300050020008
0.0308000 0... ..0 0030500820 00 3:50 05 .00 00300200 3000:0080. 50050. 05 .00 350.0.— 00§t0> .«0 «mam—52
_ Te 030...

95

00. v .— 0.30.6.0 .
8.0.»... 2.1.... u 0.0 Jigsaw. z

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

.28 0.00.0 0000. .000 NN 03.0. 00.0 N0 0050000.... 300.8000... II n 03308000
.00.. mm..0.N m .0... 05:0 NN 0000. 0.0.0 Nm 83 0.00.2 I. m 0.3308000
00.00 3...: 000m 0.0.0 NN 00Nm. ...0..0 Nm 05.03% 03380000 I: n 03380000
_. .000 00m...0 ...00. N000 NN 2.00. 00.0 NM 0000000000 300000000. I. N 03308000
GEN. 00.. . .. 000... NON.0 NN 0000. 0.0.0 Nm 8: 0.005. I. N 03308000
.0.N0 . 20.0 0Nn0. 000.0 NN 0000. 00.0 NM 20.00am 03308000 II N 0338500
0.... .0 .0000 0000. Nmm .0 NN 0.00. ..m . .0 Nm 0000000000 300000500. I . 03308000
. 0.0.0 N020 n .0... 000.0 NN 0000. v.00... Nm 8: 0.003. .l . 03308000
:Hmwmo. 000... Ram. 0.lmi0 1 _ N0 . 003.1,- ..JN..0 - Nm - 80.00am 03308000| . 03380000
. 00.3... 00.03-..— a m 000.). z a m 0005. z
_ 0000.00.70 000.300 07; 0000.00.70 000.30.— _ .[ [00%.200300500 000308000

 

03308000 00.00 80.00500 5.3 300.0070 000.>000 .3

0.000000 .00 30.0.0000 030. 0000.. 0. 0.0... 00.0.00. 0. 03.000000050000000

03308000 0... 00 0000.00.00.20 ..0 0:000 2.. .0 000000000 3000000000 0.0.00.0 05 00 8.0000 003.3.» 00 0.00.05.
N.-0 0.00...

 

Mm — IV nU—r-~.h-

96

1...»... 23...... new .332. z

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

SS. 88. can. on? 8 2%. «a? a 85:82.. _§_§§=_ l n £33.80
an. 38. 32. :3. a n8». 8:. R a: 332 l m 2.3838
88. 88. SR. 32. a 2:. RN... 9.. 232% 25858 I m 239.38
22. :8. ”New. :2. a :3. e? 2 8589.2.— _§_§_§_ l N 23838
«an. 32. 38. $3. a $8. :3. a a: 282 I ~ $3858
8 a. a 8. 8%. :3 an 3.3. 8?. mm 228% 232:8 l N 2.5858
2:. :8. 8% £3. a 88. :3. mm 85382; _§_§5§ | _ 25838
.8”. 88. SS .2 .v a 9;. R _ a. a a: £52 I _ 339....8
£3. 88. 2m“ 33 a -- -- male. a2... ,3 2323.38 I _ 2.5858

 

,, .502 -

 

 

333038335330 2:59:30

.bvcuw .3 «3330 Re awe—cmmmsm ES. .583 3 ER: 9.3—2. 5 §_§Eoo\u:owooa
253358 2: .3 85303820 .3 5:9... 05 .3 33022. Eco—3:83. .53.» 05 .3 8.32 8533 .3 £95.92

m Te 03...!

97

3...»... 23...... .8 33...: u z

 

 

 

 

 

 

 

 

3.3. 338. Kmm. 9.03 : «one. Nani cc 305—3003:— 3323335 I n 0.3303530
$5 meme. hm 5. 33:. Z «8”. on .3 cc 033 3:302 I n 0.3303530
33. :8. van. mad 2 3mm. 3N6 .3 0.5335 03303530 I n 0.3303530
53. N 53. 35mm. mmmé : R3. and co mes—3003...— 3332259: I N 0.3303530
33mm. axon. 3mm?— emad Z 23. 38.3 av 9.5 3:302 I N 03303530
88. VNNN. 53v. 80v Z 39%. Swé cc 0.5335 0.338.530 I N 03303530
332.. 32:. Nam»: 33:. 2 83 384. cc 3053003.:— 3333033— I _ 03303530
«~33. mmcc. No.3. 3:. Z 83 n26 3v 83 3:302 I _ 0.333.530 ,
$2.. 32 _. mvmn. mmmé : mmmm. awmé ice 053.5% 0.33%30I- 03I30E330

 

 

 

 

 

 

 

 

 

 

 

 

00:35 3.532

 

 

. 533%..33330 0.338.530

05—383 230—333 .3 3303333 .9 .3336»...— 330. 30:03 3. 305 9.3.0: 5 33333330333338

03308333 05 .3 83303823 .3 3:33 05 .3 3330803 33033338.. 30333 05 .3 9.32 00333 .3 ammb3<

v _ -v 033%

98

8338.. 2.3.5. -33 £332.. 2

 

 

 

 

 

 

 

 

 

_ 03.3.»...— 03.3.»-..—

00:0...2.3m 5330.... 37.

 

 

 

 

 

 

 

 

00=0t033m 5330...—

VNVM ma .3 3mmm 3.3. ”N «N3. .‘eN;~ 3N 305.3038.— .3333533. I m 03303530
333.. 23.. «.33. EN”. «N 8.3. 333 3N 03... 3.303. I m 0.3303530
N33. .33..N 33v. m3m.v 3N VNmn. M33. 3N 0.5.0335 03303530 I m 0.3303530
.33. 32.3. N3nm. 5.}~ «N 533. SN3 3N 30.5.3003... 3.3.3.533. I N 0.3303530
«new. .33.. 33mm. 32.3 ”N 233. N83 3N 03: 3.302 I N 03303530
v .2. «333.. .m .m. 336 ”N 8mm. ”3. .v 3N 0.5355 03303530 I N 0.3303530
98v. 333. 3.3. mcmé «N same. hNNé 3N 3053003.... 33.3.53:. I . 03303530
N33.. N32... N33. 3nd. «N N33. 833 3N 03: 3:00.). I . 0.3303530
New .. c .3c.N N33. 32.3 MN 8cm Na . .v 3N. 230:“.M0W333I01330 I... .033w0m5301

    

53200930333330 0.3303530

35:53..qu 3033530 5.3 0030.30.30 33330.... .3
3300330 .9. 33.3.3.3... :30. 30:03 3. 305 33.3.2. 3. 30333353033338 03303530 0... .3

m . .v 0.33...

30.33.0333 .3 3.33.3 05 .3 3.30203 330333330.— 30.53 0... .3 33.330.— 00.353 .3 3.33.5?

99

80:80 2.18. -00 .332. z

 

 

 

 

 

 

 

 

rl.

 

 

 

 

 

 

 

 

 

 

 

 

Ewe. :3 $3. and. cm 3.3. wand am 00500005 30030.50:— l n 05300500
mug. 00». was. 2 3V em 58. m 3.0 an 00: 0:002 I m 0.5300500
RE. «mm _. 5n 3. Sad en Sun. find. an 05.0.5m 0.5300500 I: m 0.5300500
:3. 53. ~80. wand en mnwe. momé am 00.50000...— _0:0_.03.0:_ I. N 0.5300500
wmmm. 2.9V.— Svm. Rod vm .NS. _NN.v 0N 00: 0:002 ..I N 05300500
030. 33. 30v. 09.4. cm W Em. 80v 0N 253m 0.5300500 I N 05300500
co _ 0. name. 300. mend «N 300. and 0N 00.500005 50002505 I. _ 05300500
3:. mm _ n. 32.. 03.0. cm N000. 3N6 an. 00: 0:002 I. _ 0.5300500
.5. 23. 1mg. 32 ixlaém. 325.0 - 122......“ E28: _ 03.0.6.

00.0.0...— 0=_s>-h a— m :00: z a m :00: z 1 - i

I ..._..__m.5,..._...w...._.a c.2583 33.30.2025 2.5.5

020.0030 0050500 .3 0.000000 :3. 30.205..— So. 00:00 0. 505 9.5.0: 5 08000502000080

0.0300500 05 00 03.05.0820 .«0 5:50 05 .00 00500200 .05—0.50002 50.050 05 .00 0:008 00001550 $020.2
3 -0. 030k

100

Further analysis was conducted on the data using the stepwise regression procedure to
determine demographic factors that might be significant predictors of the students’
perceptions regarding the quality of the characteristics of courseware programs under
study. The results are presented in Table 4-17. As the table shows, only GPA and the
students’ class level were significant predictors of the students’ perceptions regarding the
quality of computer courseware characteristics.

In terms of GPA, the range 3.51 to 4.0 was observed as a significant predictor of the

quality of the characteristics of both courseware 1 structure (B=.313, P=.022) and
courseware 2 structure (B=.310, P=.024). As a predictor, GPA in the range 3.51-4.0

accounted for approximately 9.8 percent of the variance in the students’ perceptions
regarding the quality of the characteristics of courseware 1 structure and 9.6 percent of
the variance in the perceptions regarding the quality of the characteristics of courseware 2
structure (see Table 4-17). In general, the results revealed that students with GPAs in the
range 3.51-4.0 were more likely than others to have a more positive perception of the
quality of the characteristics of courseware structures.

Class level was a significant predictor of the students’ perceptions regarding the
quality of the characteristics of instructional procedures of all three courseware programs:

courseware l instructional procedures (B=-.3 82, P=.005), courseware 2 instructional
procedures (B=-.347, P=.011), and courseware 3 instructional procedures (l3=-.373,

P=.005). As a predictor, being a senior student explained about 15 percent, 12 percent,

101

and 14 percent of the variances in the perceptions regarding the quality of instructional

procedures of courseware 1, courseware 2, and courseware 3, respectively (see Table 4-
17). The results revealed that senior students were more likely than other groups to be

more negative in their perceptions regarding the quality of the characteristics of

instructional procedures of the courseware programs under study.

102

l l 000.- 05.50
00..- 0.3.50 02:802..
.23 '00.0 53.800... - 0 203858
08. .08. 3.0 08. 00.0 20. 310.0
0.0. 0.00. o... 00.0-0.0
<00 23250- 02038.80
00.. :8. 8.0 .8. 00.0- $0.- 5200
000. 0...- 000.- 03.5.0 85.38...
.06.. 9.0.0 3233.0... - 0 203358
0:. .08. 00.0 08. 00.0- 000.- 08.80
000. $0.- 3..- 08...... 02:88...
.93 03.0 52.250... - . 2033.58
08. 00.0 0.0. 310.0
. . .8. 00.0.0.0
(00—9 25¢qu u — 0.3300500

 

 

 

.03. 0e... 2. 2.20.3.0 .

    

   

 

 

 

 

    

    

   

 

 
  

 

    

   

 

 

   

0:

 

 

 

 

    

00.0.» - -
_ ... 5.1.0 01.05- ,.

   

 

 

laces. 53.02..

  

 

0.5.80. 02.00000 950505 E .0050 000.5 0533.5 0.5300500 05 00
0000.00.08.30 0.: 00 3:50 0... mam—:09. 0500:0000.— 50050 .00 00.350200 0:. 00 0.5000 0200030:

0. _ -v 050,—.

103

Other features of the courseware programs the student respondents perceived as
unique and helpful in effectively learning Physiology 431 course materials

The students were asked to comment on their experiences using the courseware
programs, particularly on any features of the programs not indicated as characteristics on
the evaluation instrument, that they perceived to be helpful in effectively learning
Physiology 431 course materials. The students’ responses were compiled and grouped
according to the respective courseware programs in cases where a specific comment
applied to a particular program, or according to themes in cases where comments applied
to all three programs. Comments under courseware sections and themes were further
categorized under positive or negative perception. Sections and themes with comments
that are either all positive or all negative were categorized as such. The themes under
which the comments were classified were derived from instructional and learning
principles expressed in the comments themselves. When a comment contained more than
one idea, and so could fit under several themes, a subjective decision was made as to the
best theme under which it should be classified. Nebulous comments or statements that,
although sounded interesting, but did not indicate how specific courseware features
promoted or could facilitate effective learning, were not included in this report. For
instance, statements like “the programs are excellent”, or “the programs are very helpful,”
were not reported since such statements did not explain specific ways in which the
programs were excellent or helpful. The classifications, cOmments, and discussion of the

data are presented next.

104

Comments common to all three courseware programs
Overall usefulness and utility of the courseware programs
Positive views or perceptions:

The courseware programs were favorably appraised as offering complete tutorial
instructions and serving well as a useful synthesis of the Physiology 431 course lecture
and reading materials. As such, the programs were considered to be a useful supplement
to classroom instruction. One student commented: “I liked the way the computer
program paralleled the study guide. This really reinforced the lectures and made it easy to
take notes.” Another student remarked: “I thought these were so helpful! I read the book
and went to all the lectures, but the CA1 programs put it all together in my head and
really prepared me for the test questions.” Another student stated, “The courseware
directly reinforced everything we had been taught in lecture and in the readings. It was
very complete, and therefore could be used to answer questions fi'om both the lecture and
readings.” Yet another student remarked: “The programs clarified a lot of points that
were confusing in the book. They are a good supplement to class lectures.” In the words
of another student:

The programs helped me tremendously. I used them as a reference and was able
to increase my understanding with them. I feel I truly learned the material from
that part of the class (in which the programs were used), particularly because the

material was presented and reinforced in three different ways - lecture, book and
computer.

105

Virtually all the student respondents described the programs as a useful tool for
enabling a “thorough review” of the extensive lesson concepts and principles covered in
the course. As one student remarked:

I believe these computer programs really helped. There was a lot of information
that Dr. Adams presented where we had to take rapid notes and even though he
would go over everything several times, the computer programs really helped tie
everything together and clearly showed how the different systems worked. It is
very valuable because you can go through it at your own pace and can read as
much or as little as you want.

Another feature of the courseware programs perceived to be helpful in promoting
effective leaming of the Physiology 431 lessons was the use of examples and the
application of abstract concepts to real life situations. According to one student: “The
application of theory to real life situations was educational.”

The systematic, clarity, and completeness of lesson content presentation was another
salient feature of the courseware programs identified as helpful in promoting effective
learning. In the words of one student: “The courseware is the most helpful aid to any
class I have ever taken in a course with so much information. I liked how it tied it all
together in a systematic manner." Another student remarked: “I loved them. They were

about the only thing I studied. I liked how clearly all pieces fit right in the program. The

less I have to do to get the information the more time I have to learn it.”

106

Negative views or perceptions:
Some of the comments that the students provided to the open-ended questions
revealed negative perceptions about certain features of the courseware programs as well as
negative views concerning the way in which the programs were used in the Physiology
431 course. The documentation and analysis of such negative comments are crucial to a
meaningful and fruitful formative evaluation of any product development.
One major criticism of the courseware programs under study was the wordiness of the
presentation of information. According to one student: “Some of the screens had too
much reading information (text).”
Some students criticized the prominence that the programs had in the course. In the
words of one student:
It seemed as though we were tested on the courseware programs rather than the
material covered in the book and lecture. I heard from students who had taken the
class earlier that the courseware was all that was needed. It seems as though it
became the center of the material to be tested rather than merely a learning
supplement.

Another student critiqued:
It should be used (both by professor and students) only as supplementary to
lecture. It was stressed too much. Having the book, lecture, computer was too
much redundant material to wade through when a student has other classes that
they must attend to. I think in a class such as physiology, such programs should

cover material completely separate from that covered in class and the book (i.e.,
clinical situations). Then it would be more interesting and successful.

107

One student even declared: “After the time spent in lecture, reviewing notes, reading text,
I put the computer programs at a low priority. The text was often too similar to the text
book.”

Also, some students argued that the programs presented essentially the same
information covered by other course materials and, as such, were redundant. For instance,
one student remarked: “The material on the courseware should not be too repetitive of
information found in course readings and lectures. Sometimes I found the courseware a

bit too repetitive of what we had already learned.”

Individualized Instruction
Positive views or perceptions

Another major theme that emerged from the students’ comments was the usefulness of
the courseware programs in providing individualized instruction. According to one
student: “The programs allow you to work at your own pace to ensure understanding.”
Another student claimed: “1 think that the software was excellent. It accented the points
from lecture with Dr. Adams and I could work at my own speed.” In the words of
another student: “Definitely, without it, material cannot be learned as well for students

like myself. It opens up options and is more fair for certain people.”

l O 8
Stimulating and sustainifl learner interest

Positive views or perceptions

A number of students remarked that the computer courseware programs helped to
stimulate and sustain their interests in learning the lesson content. For instance, one
student commented: “I found the computer to keep my interest longer than reading a
textbook.”

Several characteristics were identified as factors that made the courseware programs
stimulating and interesting. First, some students claimed that the courseware programs
helped them to “visualize concepts more clearly.” In the words of one student: “It
helped you stay motivated, visualize what you are doing, and check yourself afterward. I
loved it.” Second, some students claimed that the programs were “fun” to interact with.
According to one student: “It’s hard to discipline oneself to sit down to read a book.
These programs accomplished the same task in a different way. A way that is fun and
less of a chore.” Third, the programs were considered to be interesting because of the
nature of humor used. In the words of one student:

I would really recommend these programs because they hold so much information.
They allow you to work on them when you want. The humor and little pictures
are great! They are hilarious! Its kind of like a little kids game - makes learning fun
and with harder concepts.
Fourth, the programs were perceived to use instructional strategies that promoted
positive attitude toward learning. For instance, one student noted:
Dr. Adams programs were exceptional because he explained why answers were

correct and more importantly why the others were wrong. And psychologically,
even when I did get answers wrong, phrases like “I can see why you chose this

109

answer but...” and “good try but...” didn’t leave me feeling defeated. It helped me
to keep a good attitude and lower stress level while studying.

Relevance of the programs to learner needs
Positive views or perceptions:

The relevance of the courseware programs to the students’ needs to perform well in
exams was cited as one characteristic that some students perceived made the programs to
be helpful in promoting effective learning. According to one student: “The programs were
more useful for me in studying for the test than the textbook. They cleared up any
confusion I had.” Another student even claimed:

The programs were extremely helpful, informative and fun. I plan on using these

and the others to study for the MCAT. It helped very much to go through them

before and after the material was presented in lecture.
According to another student, the programs offer an “excellent way of reviewing and
bringing together material before exams.” In the words of one student: “The courseware
programs helped much during preparation for exams. They clarified many concepts and
organized numerous materials for an easier understanding.” Another student claimed:
“The programs better prepared me for the exam. I could get a feel for the kind of
questions that would be asked and see my weak areas.” In the words of another student:
“The material was clear, focused and relevant to the class. It helped out in that it

simplified things. It made it much easier to understand a concept.”

110

Facilitated the use of students’ leaminggtyfi
Positive views or perceptions:

The courseware programs were perceived to enable some students to use their learning
styles. According to one student: “I learn better by trial and error. So this optional
technique for leaming was exceptional.” Another student stated:

Everyone can’t learn by reading and lecture only. The multimedia software uses
sight, sound and active learning in the form of review questions to enhance the
student’s unique learning abilities.
In the words of one student: “The programs provided clarification to concepts discussed
in class. They helped me to visualize key concepts, something that is very important in
my learning process.” A similar sentiment was expressed by another student who noted:
“Visual learning at my own space made it easier to grasp concepts.” One student was
simply impressed by the use of computer courseware in academic curriculum. According

to the student: “It was helpful to see course information via a third media in addition to

lectures and assigned readings.”

Promoted meaningful learning

Positive views or perceptions:
Another important feature of the courseware programs that the students perceived

was their potential to promote meaningful learning. One student claimed: “Quiz

111

questions at the end and throughout the programs enabled me to apply concepts rather
than just grasp or memorize the material.” Another student asserted that the programs
“provide excellent review and related material to the way exam questions were to be
asked. Question why’s and help with explanations clarified lesson material very well.”

In a similar vein, one student noted: “The programs were very helpful in stressing main
ideas. They were also helpful in developing problem solving skills and thinking.”
Another student asserted: “This is a very good way to be involved actively in the learning
process.” One student claimed: “The courseware enhanced my understanding of the main
topics and also further explained the more obscure topics.” The same sentiment was
expressed by another student who noted: “The courseware was extremely helpful in
aiding my understanding of the various topics.” In the words of one student: “I believe
the computer aided program for Physiology was extremely helpful and informative. It
clarifies things that were unclear in class. This program should continue to be used in this

course.”

InteractivityI learner and navigational control

Positive views or perceptions:

Interactivity, learner and navigational controls were perceived as essential features of
the courseware programs that helped in promoting effective leaming. According to one
student: “The courseware is unique in that it allows the subject to complete the lesson on

his/her own time and at an appropriate pace as determined by the learner.” Another

112

student noted: “I think this courseware is an extremely effective learning aid. It forces the
student to interact with the material instead of trying to passively absorb it through
review of notes or reading. This interactions enhances learning. The courseware also

provides a constant reference that can be accessed at an individualized pace.”

Negative views or perceptions:

The students noted a shortcoming in terms of navigational restrictions in certain parts
of the courseware programs. In the words of one student: “I wish that there was a means
of going backwards to refer to things I realized later in the tutorials that I was still

working on. Also, I wish I could have printed some of the material.”

User-fi'iendliness
Positive views or perceptions:

Ease of use and user-friendliness are crucial to the success of any computer courseware
in promoting effective learning. With regards to the courseware programs under study,
one student claimed: “1 like how they were user-friendly and the student using it could
determine how fast or slow to go through the material, or could go back over a difficult

concept.”

113

Feedback
Positive views or perceptions:

The feedback features of the courseware programs were appraised favorably for their
usefulness in promoting effective learning. According to one student: “The review
questions were very helpful. It allowed me to not only see what I was supposed to leam,
it gave me the reasons why my answer was wrong. Telling students why they got a
question wrong is the most helpful thing to me. I learn from making mistakes.” Another
student remarked: “I think the big thing it did was when I punched in a wrong answer, it
told me why I was wrong.” One student claimed: “I thought some of the feedback
statements were humorous!” In the words of one student: “Very good features (of the

programs) include the use of examples and feedback.”

Aesthetics and visual apge_als
Negative views or perceptions:

In general, the aesthetics and visual appeals of the courseware programs were rated
poorly. Such a perception could negatively influence the acceptance and usage of
courseware programs. In the words of one student, for instance, “The multiple colors on
the screen strain the eyes.”

One related issue was the quality and accuracy of the textual materials of the

programs. In general, the students felt that the programs presented too much textual

114

information. Some spelling mistakes were also detected. According to one student: “The
text and explanations on this program were a bit extensive, too much like text book.
Hurts eyes to read screen full of text. But informative indeed!” Another student
remarked: “The programs need to be proofread carefully!!! I was thrown off by
misspelled and wrong words.” Another student suggested: “... cut down (text). Use the
computer to convey information in non-traditional ways, using graphics and case
histories.” Yet another student prescribed: “Cut text in half. Key words and phrases

only. I found myself rushed for time trying to read full screen text. Narrate instead.”

Courseware availability and accessibility

Positive views or perceptions:

Comments provided about courseware availability centered primarily on the need to
develop similar programs on other topics in physiology 431 course, topics in other
physiology courses, and topics in other sciences, particularly the physical sciences. In
the words of one student: “I believe that the CA1 was tremendously helpful. I really
wish that it was available for the rest of 431 and I hope it is available for 432 (i.e.,
Physiology 431 and 432 courses)” The same sentiment was echoed by another student
who wrote, “The computer-aided instruction was extremely beneficial. I only wish they
had one for each test, such as one on the GI (Gastrointestinal) tract and ANS (Autonomic
Nervous System)” In the same vein, another student wrote, “These programs are great

and should definitely be continued. The lack of computer assistance on the

115

Gastrointestinal tract topic is reflected in the lesson grade I received on that recent test as

compared to my first test with which computer aided instruction was available.”

Negative views or perceptions:

An issue closely related to availability is the view that courseware programs should be
delivered across multiple platforms. The programs under study are strictly Windows-
based. However, a comment by one student argued for cross-platform compatibility of
courseware programs. In the words of the student: “The programs were useful. But need
to accommodate both PC and Mac users.” Expressing the same sentiment, one student
remarked:

Having to go to a computer lab is a detriment. Although disks are available to

check out, there weren’t enough copies to go around when needed. Could

something be put on the intemet (e.g., a home page on WWW)? Or, if a disk (Mac

or IBM) was available for purchase with the study guide, this would help.
According to another student: “The material was very useful and well written. I enjoyed
learning the material this way. I just wish that the software was available for people with
Macintosh computers.”

On the issue of accessibility, the students’ comments centered primarily on the need
to make the procedure for successfully using the courseware programs in the
microcomputer labs on campus foolproof, especially for students who do not personally
own computers. According to one student:

I feel that the students should have access to the program without the use of the

login name/password. I didn’t get mine for awhile and everytime I would try to
study for physiology using the computer programs I was locked out. I had

116

problems getting a password. And many, even the computer techies, were unsure
what the glitch was. All the while, I was left out in the cold having to study with
others when they used the computer.
In the words of another student: “I think the computer-aided instructional programs are a
great asset to this course. The only drawback is that for working students who live off
campus (and do not own a computer for home use), it’s difficult to get adequate time on
campus computers.”
A protest by one student is noteworthy for linking accessibility to computer
courseware with computer literacy as a prerequisite for the use of courseware in academic
curriculum. According to the student:
I live off campus and have limited time to come here (i.e., the computer labs on
campus) for use of computers. I thought the directions were not specific enough.
The one attempt I made to use the computer, I was unable to access the programs.
I’m probably as close to computer illiterate as you can get.

The same sentiment was echoed by another student who wrote:
These programs are useful in that they strengthen a students understanding of
course materials. However, what is not fair and not right is that MSU’s book of
classes available never indicates if computers are used in the class; and for those

who do not know much about how to use computers, it’s not a good thing at all
not to notify students before signing up for the class.

Time to complete courseware usage
Positive views or perceptions:
The students’ comments revealed a mixed view on the time required to complete the

courseware programs. While some students felt that a considerable amount of time could

117
and should be devoted to the programs, others held a contrary point of view. According

to one student, for instance, “I did all of the programs in a short amount of time, maybe a
week. If I would have done them over the course of a few weeks, or even several, and
also more than once I think I would have learned more efficiently. That was my own

fault. The courseware was very helpful though.”

Negative views or perceptions:
On the contrary, another student complained, “The problem with them is that they
take much time to finish and look at such that you end up spending more hours studying

them. You might not spend that much time studying the book.”

Appropriateness of using computer courseware
Positive views or perceptions:

The student respondents also provided comments on academic uses of computer
courseware programs in general. An examination of the comments revealed mixed feelings
on the part of the students. Some students liked and recommended wider applications as
well as broader integration of computer courseware. Others deplored and protested
against the use of computer courseware in academic courses in which computers is not the
subject matter of discipline.

As an example of the camp that supported full integration of computer courseware

into academic curriculum, one student claimed the applicability of computer courseware

118

in all subject areas. In the words of the student: “Every class can use these type of
programs.” Another student remarked: “I would like to see more of these programs for
biochemistry and physics classes.” Also, one student recommended:
Courses such as the General Inorganic Chemistry, Organic Chemistry, and most of
all Biochemistry should have courseware programs similar to the one in
Physiology. All of these sciences require the learner to know large amounts of
information from nomenclature to real life applications. These programs could
ease the learning process and can virtually minimize it.

A less enthusiastic student, but nonetheless supportive of academic uses of computer
courseware, commented: “I don’t think they (i.e., courseware programs) should serve to
replace lecture or readings but as a compliment to the usual learning methods.” Another
student argued: “In some classes that (i.e., courseware programs) would be good, but not

all classes.” According to another student: “Hard sciences are the only courses in which I

believe that the additional material may be helpful.”

Negative views or perceptions:

Some students outrightly opposed academic uses of computer courseware. For
instance, one student protested: “I think computer material in a non-computer class being
required reading is wrong. I managed a 3.5 without the programs and without much

studying and thus I think they were a waste of time.”

119

The same instructor effects
Positive views or perceptions:

The students’ comments revealed that the use of the courseware programs by the same
instructor who developed them influenced the students’ acceptance and usage of the
programs. In the words of one student:

I feel one reason this was so helpful in learning the class material is because the
professor put the programs together. Because of this, there was no doubt that the
material presented was important and relevant to the class itself. I used
courseware programs in other courses and I had the same experience. I felt they
were a wonderful tool for learning and I very much respect the professor for
taking the time to put such a tool together for us.
For some students, the programs capsulized the topics, concepts, and principles that
were important to the instructor. The programs, therefore, served as a tool for focusing
the students’ attention on lesson materials. One student responded: “Overall, I thought
that the CA1 programs enhanced my understanding of the materials in a way that
indicated how the professor wanted me to learn and understand the materials, not just
how the textbook authors thought about the material.” One student noted: “I really wish

more classes would put these (kinds of programs) together because you can really know

what your professor is focusing on and what the most important material is!”

120

Specific comments about Courseware l (i.e., Action Potential)
Positive views or perceptions:

The courseware program on Action Potential was appraised to serve well as a useful
synthesis of classroom lecture on the topic. In the words of one student: “This program
was very helpful to tie the whole thing together. Graphics in this program were very
helpful.” According to another student: “Action Potential reinforced what we learned in
class and the practice questions really helped me in terms of comprehension and picking
up main points to be learned.”

The clarity of the courseware was also identified as another feature that made the
program to be a useful learning tool. According to one student: “The CAI program on
Action Potential was relatively straight forward.” Another student remarked that the
program was a “Good set up and easy to understand.”

One student noted that the program helped to promote thinking skills. The student
asserted: “This was very detailed view of the Action Potential and quite thought

provoking. Many of the questions were worthy of extra attention.”

Negative views or perceptions:

A few of the comments about the courseware on Action Potential indicated negative
views or needed improvement. In the words of one student: “In a few of the screens I
could not go back to a specific of the courseware without starting over. Use animation
and audio (sound of an Action Potential).” Another student remarked: “More extensive

difficult review questions would be helpful.” According to another student: “The

12 1
threshold-frequency stuff was weird. I didn’t like the Previous or Next page stuff. I like

to click on what I need to see.”

Comments about Courseware 2 (i.e., Control Systems)
Positive views or perceptions:

The comments provided concerning the courseware on Control Systems indicate that
some students liked how the program used graphics illustrations, examples, and analogies
to clarify and simplify lesson concepts. In the view of one student: “It (the program on
Control System) was the best one! It showed many examples and excellent diagrams
concerning Control Systems in physiology.” In the words of another student: “This
program was very useful and helpful. The graphics and examples made it easier to
understand concepts of Control Systems.” According to one student: “This was a
difficult subject for me but the courseware made things easier to understand. The
examples and test questions were also helpful.” Another student remarked: “Assumes
that students have never seen Control Systems before. I hadn’t.”

The program was well received especially as an important supplement to classroom
lecture on Control Systems. In the words of one student: “This was probably the most
helpful (of the courseware programs) because there was no information in the book. I had
to rely strictly on lecture and study guide.” According to another student: “As I had
missed the presentation of this material in lecture format, I felt that the tutorial itself

prepared me very well for the upcoming subject examination.” Also, one student stated:

122

“The program almost directly paralleled the study guide (used in the class) and was very

helpful.”

Negative views or perceptions:

Mixed views were expressed concerning the use of examples in the courseware
program on Control Systems. According to one student, for instance, “I feel that more
examples should have been given. More problems to work through would have helped
more.” A similar view was expressed by another student in the statement:
“Physiological examples are really helpful here. I think you could include more.” One
student even claimed: “It was short, and didn’t go over the material in enough detail so
you can understand completely.” However, a different point of view was expressed by
another student who complained:

The Control Systems explained the basic lesson, but in giving examples went a
little beyond my knowledge of the body’s systems. Because of this, I simply
skipped over that example - but was left wondering if I was to know that much
material. Even though I understood Control Systems, I felt rather disillusioned

afier such a difficult example. This negative aspect is what sticks in my mind
about the courseware.

Comments about Courseware 3 (i.e., Synaptic Mechanisms I and II)
Positive views or perceptions:
One feature that some students identified as an important strength of the courseware

on Synaptic Mechanisms was application of concepts to real life situations. In the words.

123

of a student: “This tutorial served as a good review of the information, and allowed me to
develop enough knowledge of the application of the material to other situations.”

Some students praised the program for simplifying and clarifying what would
otherwise have been a difficult subject matter. According to one student: “The program
was very good at explaining how to tell whether or not it is a peripheral nerve or a
(neural/dorsal) root that is damaged when doing a neurological examination. Even though I
missed the question on the exam #1, I know the answer and how to determine it. I missed
it due to time restriction and had to just fill in an answer.” Another student said, “What
was most helpful were the hints and the explanation that follow the selections.”

Other features that some students identified as useful in promoting effective learning
were the use of examples, review questions, and graphics animation. According to one
student: “Synaptic Mechanisms I and II were useful - the examples and questions helped
me quite a bit. Great program.” Another student wrote: “The actual moving animations

meant the most to aid in understanding the mechanisms... Keep the programs, they’re

great!”

Negative views or perceptions:

Some comments were made concerning drawbacks in the program as well as aspects in
which the program needs improvement. According to one student: “J ust like Control
Systems and Action Potential, I could not go back during certain parts, almost got lost in
the program. Animation would be very useful.” Another student wrote: “Synaptic

Mechanisms was fine but a little bit harder (compared to other programs) to keep focused

 

124

on the program. It was good but missing maybe some details.” One student remarked: “I
wished there was more here. I still felt a little lost after completing the courseware and
reading. Perhaps more check your work questions here would help.” According to one
student: “The only problem I noticed was that (endplate) potentials were not explained
clearly enough in either the programs or the book, or class.” Also, one student
commented: “Some of the screens contained too much text. This caused fatigue.”

Another student noted: “Some explanations were wordy.”

Sum mary
Findings related to the effects of computer courseware on students’ learning
achievement were presented in this chapter. Comments made by students regarding
features of the computer courseware programs that aided effective learning of the lesson
concepts were complied and classified under four major sections:
1. Comments common to all the three courseware programs
2. Comments specific to courseware 1
3. Comments specific to courseware 2
4. Comments specific to courseware 3
Section one, comments common to all three courseware programs, was further divided
into themes to make the numerous and diverse comments provided about the programs

more comprehensible and meaningful. The themes that emerged from the comments were:

0 Overall usefulness and utility of the courseware programs
0 Individualized Instruction

125

Stimulating and sustaining learner interest
Relevance of the programs to learner needs
Facilitated the use of students’ learning styles
Promoted meaningful learning

Interactivity, learner and navigational control
User-friendliness

Feedback

Aesthetics and visual appeals

Courseware availability and accessibility
Time to complete courseware usage
Appropriateness of using computer courseware
The same instructor effects

A summary of the findings, conclusions, discussions, and recommendations regarding

the effects of computer courseware on learning achievement is presented in Chapter VI.

CHAPTERV

PRESENTATION AND ANALYSIS OF DATA FROM EXPERTS

Introduction

The data presented in this chapter were gathered from experts drawn mostly from
Michigan Sate University. The only exception was data collected from an expert from a
community college because of the respondent’s academic background in educational
systems development and previous extensive professional experience in courseware
development at Michigan State University. The data from the experts were collected
during the Fall 1995 and Spring 1996 semesters.

The report in this chapter is presented in three sections. Section one describes the
demographic characteristics of the experts involved in the study. Section two addresses
the questions relating to the experts’ perceptions regarding the level of importance of
computer courseware characteristics in promoting effective learning. Section three
presents information about the experts’ perceptions regarding the quality of the

characteristics of computer courseware programs in promoting effectively learning.

Section One — Description of the Experts’ Demographics

Two groups of experts participated in the study. The first group consisted of 32
experts who responded to the survey questionnaire on the level of importance of the

characteristics of computer courseware in promoting effective leaming. This group of

126

127
experts is referred to as the Expert Rating Form Group (ERFG). The second group

consisted of 34 experts who used and evaluated the same computer courseware programs
that the students in Physiology 431 class used and evaluated. This group of experts is
referred to as the Expert Courseware Evaluation Group (ECEG).

Several demographic characteristics were collected about the experts in an attempt to
understand factors that influenced their perceptions. Such an understanding will, in turn,
shed some light on factors that exert influence on the evaluation of educational software.
The demographic characteristics of the Expert Rating Form Group will be described first,
followed by a description of the Expert Courseware Evaluation Group demographic
characteristics.

The Expert Rating Form Group was comprised of an equal number of male (50%) and
female (50%) respondents. By age distribution, 13 (40.6%) of the respondents in the
Expert Rating Form Group were below 35 years old, 11 (34.4%) of them were between
36 and 45 years old, and those at least 46 years old constituted 25 % of the group. In
terms of teaching experience, five (15.6%) of the respondents in the Expert Rating Form
Group had between one and ten years of experience. Another five (15.6%) respondents
had between 11 and 20 years of experience. Eight (25%) of the respondents had at least
21 years of teaching experience. 14 (43.8%) respondents either had no teaching experience

or did not provide that information.

 

12 8
Table 5-1

The distribution of respondents in the Expert Rating Form Group
by range of teaching experience in years
(n=32

'Teachin Exerrence in Years - 2 “-

 

 

 

 

 

 

 

23 (71.9%) of the respondents in the Expert Rating Form Group indicated having
expertise in courseware development. Nine (28.1%) respondents indicated having no
courseware development expertise.

Data was also collected on the approximate number of courseware programs the
experts had developed. As Figure 5-1 shows, 18 (56.3%) of the respondents had
developed between one and three programs. Six (18.8%) respondents had developed at
least four programs. Seven (21.9%) respondents indicated that they had not developed
any programs.

Figure 5-1

The distribution of respondents in the Expert Rating Form Group
by approximate number of courseware programs developed

4 or more
1-3

 

 

129

The respondents were asked to indicate whether or not they had previous experience
using computer productivity tools. As Table 5-2 shows, all the respondents indicated
having experience using word processing and communication tools. 75% of the
respondents had experience using spreadsheet programs, 72% had database experience,
63% had computer programming experience, 91% had multimedia experience, 66% had
experience in computer games, 72% had experience using courseware programs, and 25%
had experience in assembling computers.

Table 5-2

The distribution of respondents in the Expert Rating Form Group
orevious ex . rience usin. com nuters as roductivi tools

Type of Computer Had previous No previous
Experience oexperience oexperience

 

 

 

 

 

 

 

 

 

 

 

I
I
I WordprocessinL 100

Spreadsheet 24 75 8 25 32 100
I Databases 23 71.9 9 28.1 32 100 f
I Communication 32 100 32 100 w
I Proamu: 20 62.5 12 37.5 32 100
I Multimedia 29 90.6 3 9.4 32 100
I Games 21 65.6 11 34.4 32 100 *
I Courseware 23 71.9 9 28.1 32 100 I
I Assemblin_ Com .uters 8 24 75 32 100

In a related, but separate question from experience with computer productivity tools,
this group of respondents was asked to indicate whether or not they owned computers.
The distribution of computer ownership consisted of 30 (94%) respondents in the Expert

Rating Form Group who owned a computer, only two (6.3%) did not.

130

Similar data presented for the Expert Rating F orm Group were also collected for
respondents in the Expert Courseware Evaluation Group. The analysis of the data is
presented separately since this group consisted mostly of different respondents and
responded to a different survey questionnaire.

The Expert Courseware Evaluation Group was composed of 22 (64.7%) males and 12
(35.3%) females. By age distribution, eight (23.5%) of the Expert Courseware Evaluation
Group were under 25 years old, 11 (32.4%) were between 26 and 35 years old, and 14
(41.2%) were over 36 years old. The Expert Courseware Evaluation Group consisted of
five (14.7%) respondents with expertise in the content area covered by the courseware
under study, and 29 (85.3%) respondents with no expertise in the content area. Nearly
threeoquarters (73.5%) of the respondents in the Expert Courseware Evaluation Group
indicated that they had expertise in courseware development. Nine (26.5%) of the
respondents in this group had no previous experience in computer courseware
development.

In terms of teaching experience, six (17.6%) of the respondents in the Expert
Courseware Evaluation Group had not more than 10 years experience, three (8.8%) had
between 11 and 20 years experience, nine (26.5%) had at least 21 years of experience. As
Table 5-3 indicates, 16 (47.1%) respondents either had no teaching experience or did not

provide that information.

131
Table 5-3

The distribution of respondents in the Expert Courseware Evaluation Group
by range of teaching experience in years
n=34
I m _
None (or missing values) 47.1
1-1016 17.6
11-20 3 8.8
21 and above 9

   
   
     
     
 

   

 

 

 

 

 

 

 

Data was also collected on the approximate number of courseware programs developed
by the respondents in the Expert Courseware Evaluation Group. As Figure 5-2 shows,
19 (55.9%) of the respondents had developed between one and three courseware
programs, seven (20.6%) had developed more than three courseware, and five (14.7%) of
the respondents indicated that they had no experience in developing computer

courseware. Three (8.8%) of the respondents did not answer this question.

Figme 5-2

The distribution of respondents in the Expert Courseware Evaluation Group
by approximate number of courseware programs developed

4 or more

 

 

 

132

Another demographic characteristic examined for the respondents in the Expert
Courseware Evaluation Group was previous experience using computer productivity
tools. As table 54 shows, all the respondents indicated having experience using word
processing and communication tools. 88.2% of the respondents had experience using
spreadsheet programs, about 80% had database experience, about 65% had computer
programming experience, about 91 % had multimedia experience, over, 60% had experience
in computer games, about 82% had experience using courseware programs, and over 44%
had experience in assembling computers.

Table 5-4

The distribution of respondents in the Expert Courseware Evaluation Group
b revious ex rience usin corn uters as roductivi tools

 

 

 

 

 

 

 

 

Type of Computer Had previous No previous Total
Experience experience experience

No. % No. % No. %
Wordprocessing 34 100 34 100
Spreadsheet 30 88.2 4 l 1 .8 34 100
Databases 27 79.4 7 20.6 34 100
Communication 34 . 100 34 100
Pro ' 22 64.7 12 35.3 34 100
Multimedia 3 l 91 .2 3 8.8 34 100
Games 24 70.6 10 29.4 34 100
Courseware 28 82.4 6 17.6 34 100
Assemblin Com uters 15 44.1 19 55.9 34 100

 

 

 

This group of respondents was also asked to indicate whether or not they owned a

computer. 30 (88.2%) respondents owned a computer, only four (11.8%) did not.

 

 

133
Section Two — Perceptions of Respondents in the Expert Rating Form

Group Regarding the Level of Importance of the Characteristics of
Computer Courseware in Promoting Effective Learning

This section answers the research questions relating to the perceptions of the
respondents in the Expert Rating Form Group regarding the level of importance of the
characteristics of computer courseware in promoting effective learning. The purpose was
to distill, from experts, knowledge of factors and characteristics considered to be
important in promoting effective instruction and effective learning via computer
courseware.

Data collected to answer the research question utilized the Computer Courseware
Evaluation Checklist Rating Form (see Appendix B). The questionnaire was based on
Gagné’s Events of Instruction with additional information gleaned from current research
and theories in educational psychology and academic uses of new technologies.

Two levels of analyses were applied to the data collected using the Rating
questionnaire. The first level of analysis used simple descriptive statistics to describe the
experts’ perceptions regarding the characteristics of courseware programs in promoting
effective learning. The second level of analysis, based on the results of simple descriptive
statistics, focused on determining whether the experts’ perceptions varied by their

demographic characteristics.

134

The major research question guiding this section of the report was
Research Question C:
What computer courseware characteristics do experts perceive as important
in promoting effective learning?
The subsidiary questions were:

Research Question C]: What are the experts’ perceptions regarding the level of
importance of computer courseware characteristics in promoting effective
learning?
Research Question C2: Do the experts’ perceptions regarding the level of
importance of computer courseware characteristics in promoting effective learning
vary by their demographic characteristics such as gender, age, professional
discipline, teaching experience, courseware development experience, experience
using computer productivity tools, and computer ownership?
Research Question C3: What other features do experts perceive as important to

the success of computer courseware in promoting effective learning?

Experts’ perceptions regarding the level of importance of computer courseware
characteristics in promoting effective learning

Respondents in the Expert Rating Form Group were asked to rate their perceptions of
the level of importance of computer courseware characteristics in promoting effective
learning. The mean importance ratings and standard deviations of the perceptions were
computed for each characteristic. The possible means rating ranged from the lowest, 1.00,
indicating low importance to the highest, 3.00, indicating high importance. Table 5-5
shows the means, standard deviations, and ranks of the perceptions regarding the
characteristics. From this information, the mean importance ratings observed ranged from

the lowest rating, 2.03 for the characteristic “tests the learner on restating lesson

135

information in his or her own words”, to the highest rating 2.84 for the characteristic
“presents complete, accurate, or current lesson materials.” All the characteristics were
rated as either high or medium in their level of importance, with each level having equal
number of characteristics, 40 (50%). No characteristic was considered to be of low
importance, that is, below 2.0 on the rating scale.

The mean importance ratings, standard deviations, and ranks were also computed for
the characteristics of computer courseware sections. The five sections were courseware
structure, screen layout and design, media use, prerequisite skills, and new lesson content.
Table 5-6 presents the results of the analysis. The results indicate that the characteristics
of both courseware structure and new lesson content were rated equal and at the top of
the level of importance. Third in the level of importance were the characteristics of
prerequisite skills, followed by the characteristics of screen layout and design. Media use

characteristics ranked last, but nonetheless were considered to be of medium importance.

136

 

«Goa—.00

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

0. co. SN Nm 00000. 8.0.000 0.0E 0. 0.08.0 w0...0.. 0. 80.00. 0... 00.0w 0. :0..0:..00. 800.0...
0. mm. eoN Nn 000.00. 0.. 0. 8.00000... 0:0 .8....0:.... £00008 00.0..000:.00 0:0 8....0000
m. vm. ”0N .m 00.000 .0 00:02.0 w:.E00. 0... :0 00:00.... 80.00. 80:00. 0000:0000.
m. Nm. N...N Nm 60.0 4x0. 00...... ..0.00 00.0.0.3 0.00.5.0 0:0 .00»... 000.00 0... .0 85.0.0000
00E00. 8.m0.0..0
m. Nm. N...N Nm 0>...:w00 00: .0 00:00... 0. 80.00. 0... .0. 80.03.0000 .0 >.0..0> 0 000.3...
N. 3.. SN .m .0208 00000. .0 02.5.0090 .00.w0. 0....
. . 00. SN Nm 0000.008... :0..0.E0.0. .0 0.0. 0... .96 .0..:00 .0E00..
0000.50.80 0.80.00.
0. 0... «EN Nm 0... :0 0.000000. 000.5... 0:0 000.00. 0.....0 0... 00000,... 0. .0500. 0... 02.000...
9.00.0.5. 0. .0500. 0... .0. 5.00... 0
N 0... .wN .m 0.0.8.00. 00000. 0... 5.3 00.. 80.00. 0... b.>..00..0.0. .0 00.80 .0 .08. 0....
N c... .mN NM .0208 00000. 000 83.00.30 0.0300500 0003.00 00.02
N o... .aN Nm 003.00.... .0 00.... 0:0 0.000000 .0 8.9008 00..0> 0.008...
N 5.. .wN Nm 00.0000... 0.00.00... 0. 00.5.8 0..0E00. .0 000500.000 0... 0.00.000
N c... .wN Nm 0.0300500 0... 00: 0. 30.. 00 00008.5
N o... .wN Nm 0.00.0380 00.0000 000 0.0.0.0 0. 00:00:85 0.008...
N n... .wN Nm 0.00. .0 0.00.0280 50:0 00:80.. 0. 0000.00. 0.80.00. .0 085000.00 0... 00....000
N 2.. SN Nm 8.... 0:0 0.000000 .0 :0..00......0 .0 00: 0.80.00. .0 0000.02.00 0... 0.00.000
. mm. 33 Nm 0.0.8.0.: 00000. .0250 .0 6.0.0000 0.0.9000 0.008...
0.00% a m. 0003. z 00.310.00.000

 

 

 

 

 

w:.:.00. 0>..00.c0 .0000w00 .0... 20300.58 8.2.0.00

.0 00..0..0.00.0..0 .0 0000000.... 0... .0 0.0.0. 000 .000..0.>00 0.00080 .0082

m6 030...

 

137

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

0m 0.... cmN Nm .0500. 0... 0. 00>.0>0. 0.. ...3 .0... 000000 000 .000000..0 .0.....0 .0 :88 8.0.0.00m
.00.:00 00000. 0... .0.00.:

mm N0. omN Nm 0. 000.0 .0 00000000 .0050 0... m:.30=0. 0. .0500. 0... 00.0» 0. 00005.00. 800.0...
0m 00. cmN NM 0.0. .0 5030000. 0... 0000000 0. 00....0 0.0. .0 000 0.. 0.
00000. 0... 0. 0.9.0.

3 .m. NON .m 00.00. .0 0.....0 30.0.30 :0 w0..:.0..00 00000.0 0. .0500. 0... .0. 3.0000000 000.5...
mm .m. mmN Nm 00.0 00.038: 00000... 0.00. .00000m.>00 00300.08 .0 00.. 0....
mm N0. mmN Nm 0>.00 0. .0500. 0... .0. 0.0030... .0>00 0.0000...
mm 3. mmN Nm .0030... 00.0.0 0. 0000.00 .0>00 0:... .0500. 0... 0.0.. 0. 0.0... 000 0.900... 80.5...
000.00.

0m 0m. mmN .m 0030.0 0... 0. 00.0.0. 000.0..0 000.00. 00000... 0. .0500. 0... .0. 3.0000000 00006....
on N0. mmN .m 00002.0 00.0.0E.0 .0 .00. 0 0. 00000 .0 00.000 90.0.: 00 .0500. 0... 0.00 0.
cm .3. RN .m 000.00. 0.....0 0... 00000.0 0. .0500. 0... .0. 80.03.0000 3000.000 000.>0.0
0N no. 0m.N Nm 00000000. .0.:000.0 0. 0.0030000
0N on. 0m.N Nm .0500. 0... .00.. 00.00000 00000.00 .0030... .0 0.0.00 .0.000w 0... 8000.0
330.000.: 80800.0

0N N0. 0m.N Nm .0 00.00. w0.:.00. 0... 00.0.0 ...3 .0... 0.0.000 000 0.00. .00>0.0. .0 ..000. 8.0.0.0....
0N 0m. 0m.N Nm ..:0 00000. ..000 0. .0 000.00 0000 00 03008.0 0000.503. .0 9.0.0.0. .00050 0.. ..
mN N0. me .m 00.0.0000 .0 000000..0 30: 0. 00.0. 000 0.000000 w:........0 :0 .0500. 0... 0.00 ..
.N N... .cN .m 0000.000 00000.0. .0 0000000000.
NN .0. N0.N Nm 0.0.8.0.: 00.000 0... .0 w0.500. w0.0:0..00 0. 000.00% .0 000 0.. ...
NN 00. N0.N Nm 00500. 0.. 0. 0.....0 0:0 00.00. 0... 00 00000.... .0500. 80000. 000000000.
0N .0. SN Nm . 00.0. .0 8.900.... 000 0.000000 8.0.3.2250 =08. 00.0.0.0..m
ON 0... meN Nm 00.0 00...... ..0.00 00300.3 0.80.0.0 000 .0000. 000.00 0... .0 b.0..00..m
0. mm. ooN Nm 003.0030 .0 .0050.0.@
000.000 0000030..

0. mm. SN Nm 8.00.000 00000000... 8.0.0000 0.0... .00. 8.00.000 300005.00. .0 000 0.. 0.

 

 

 

 

 

 

138

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

an . 0. RN Nm .0500. 0.0.0 00.0008 00......0 .0 .0.>0..0.. .0 00... 0... .0 0.00.000 800.0...

mm N0. me 0N 0000.000 00000.0. 0... .0. 0.0.05.0 0008. .0 0.0.0.0080 0....

803.00 .0 0.0.30.2. 0... 0. 00.0.0. 80000000000 000 000000 90000:...

00 N... 0mN .m .0 w0..000.0 b. 80.0.00 .0 0.0.>0..0.. 02.80 .00.... 00500. 0.. 0. 0000.50.0. 0.00020

00 N... 3N .m 83.00.30 00000. 000 0.00. 000050.80 0003.03 00.03.

00 .0. 0... .0 0.0020... 0058 2.. .o 00.68. 0:05....» a. 2.50 .o 3... 2..

80.0.00 .0 0.0.3.003 300 9.500.

mm on. .vN Nm 0. 0.5.0.. 000 .0500. 0... .0... 0000000... .0 M00005. .0.. 0.0.0 .0 ..002 8.0.00.0m

Nm 0.. .vN Nm 00000. 0... 0.9.9.0.... 8.00.8.0 .0 0.000.. 0 5.00.00. 030.8. 000000 .0 00.. 0....

.m S. .vN Nm 0.00350 00.0.00 .0 .0>00 .0 30...; 0 m0.>.00 00 .0500. 0... 0.00 ..

an N... NvN 2. 0.0.8.0.: 00.000 0... .0 w0.500. m0.000..00 0. 000.> .0 00.. 0.. ...

av 00. EN Nm 0.0. .0 3030002 0... 0000000 0. 80.0 0.0. .0 00.. 0....

0.0% 0... .00.. 0.03.003

000.0 0. .0030... 0... .003: 0. 30.000 0 00008.... .0050 M038 .0 30.5.0 .00.

0.. N0. va Nm 0.00320 0>.00 0. 80.00020 .0 8.8.0.0 .8000. .0 00.. 0... 03000 000 8....08n.

0.. N0. nvN .m 00000003. 08000 0. DZmS. .0 00.. 0....

0.....0 .0

m0 N0. SN Nm 0000.50.80 0.0500. 0.0.0 M0000 000 ~80.0000 .0 02w00 0... 00 0.003000. 8030...

m0 .0. SN N... 0980 00200 =0.0>0 0... .0 .00000 .0..0.> 3.0000

9. .0. SN Nm 00000080. 0>...0000 0.0.000 .0m 0. 00000... 0...”... .0 00.. 0....

00.0 0.00. .00000w.>00

m0 hm. SN Nm .00w00 .00..0.m20. 00:29.00... 800000.000 .0 0.00.020 00200 M0050 00020.30

.0 am. va .m 0.00.00... .0 M03000... .0 00.. 0....

.0 we. avN .m .0500. 0... .3 0000. 000.0.000 .0 02.0. 000000 .0000.00 .0 80000000000 .02.0 030..w
02.000 0000000..

mm 2.. RN cm _..0.0 .0.0.0 0... .0. 02.002 000.0 .0020 00.0..000500 000 00 00005.00. 0.00020

 

 

 

 

 

 

000.0.000 .0 000.000 00000.00 m00.0.0

 

 

139

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

on 5. SN cm 2.83 :38 8:3: .: 8888.8: :88. 2.8.8. .8 8:82 2: 28,—.
on 2.. SN .m 2.83 :38 852: .: 8:88.88: 8.. 8an— 2: 2.3.
E. 2.. _ _.N am 8:82 2: .8 “.2898 23% 288.33 88... .8 8.88.8.2— 8828888:
R 2. m . .N .m 5:25.... .2222... .o 2.5.... 2.58:. 02.6... 2 5.888 .c 8:52.“: 2:
S. 8. m _ .N on 8:82 2: 2 0288.2 2830288 2: .8 8:22.882. 8 .8828m
.882 2: .:

mm mm. EN 3 888.828 22.288 8.. 82.82: c. 8.... :5. 28280 :82. o. 8:82 2: 2.3.
E. 2.. x. _ .N am :28 .2828 8:28.. $58929: £2888... .8: 202.8 880% .8 02. 2:.
I. 3.. o _ .N mm 3.2.8 5:52.82 2: 82.2.8 8. 8.8 .8 22. 2:.
. N. mm. a _ .N .m .882 2: .8 :8 2: .8 8.22:8 :5. 28280 :88 2: 8 B28:
2. ms. 2d .m 8888.8: 2808 2 228.200 ".0 mama; 8 van—2:8 o2. 2:.
E :. and 2 =3.3. 2.. a. .83. 2....” .82.... ._.N 2...... .5 .253. 2.. 382
a: on. mNN mm 8:2. 882.8 8 9:82.. 2825.588 2 282 .8 02. 2:.
we no. wNN mm .882 2: .8 9:832. 2: 8 387. 28888.5 82.80 .5222 .8 328:
we 2.. «Wm mm 8288 882 2: @188 o. 28:88 328:
no 8. «Na mm :28 .28. .88: .880 £2823 2:288 v.8 88b: :28 2: .: 8.88m
- 2.2:... 8 882.2. 2: o. :22»... 8888888 :5. 28:2. 8:28.:

we 8. ”Nu mm 8 Iw.:.....:o=. 2:22:88 3 2.2:... 8 8828: 8:8: 2: 8.8.2.28: 8 8:88:
8.82.

m=_._m 88... 88. 2: .8 8:228:88 2: E8. o. .9888: 2.8883 03:82.2

8 2. com o... 2.333 2.823.... .88... 2.. Ba 2...... 5... 2.8.33... =82 3.228
No me. mad 7:. . A228 2:90 rwdv 28.8 m.:80..8 .8 028 2:.
no on. Nm.~ .m 22.22.. 2.88 2: .8 9.882 2:28.88 .: 88:88:... .8 02. 2:.
E _ n. mmN on :2888 .8288 o. 8:28 9.2.82 .8 58:98 25:80
8 2.. 34.. mm :88. 2: .28 o: 8 m8 2 2.8 2. EB 8:82 2: 8:3 8.2m
am 8. DH 7:. 02.22... 82.2.: 2: 8.. 28:22.. :88. .8 .2858 .8 .28— 2:.

 

 

 

 

 

 

140

Table 5-6

Means, standard deviations, and ranks of characteristics of computer courseware sections

Courseware Sections

 

Courseware Structure

 

Screen Layout and Design

 

Media Use

 

Prerequisite Skills

 

New Lesson Content

 

 

 

 

 

Further intensive analysis using the multidimensional scaling procedure (MDS) was
conducted on the data collected from respondents in the Expert Rating Form Group in
order to determine whether any specific or systematic patterns of relationships existed in
the perceptions of the experts regarding the level of importance of the characteristics of
computer courseware in promoting effective learning. The patterns of relationships
examined consisted of three-dimensional graphical illustrations of how the experts cluster
together (i.e., similar) or spread apart (i.e., dissimilar) in their perceptions regarding the
variables.

The MDS coordinates of all the variables were computed in SPSS. The MDS results
were then imported into Systat to generate a three-dimensional plot of the relationships

matrix. Figure 5-3 presents the three-dimensional plot of all the variables. The MDS

141

coordinates of the variables are presented in Appendix I. The meaning of the
abbreviations used to name the variables are explained on page xv.
Figure 5-3

MDS of perceptions of respondents in the Expert Rating Form Group regarding the level
of importance of the characteristics of computer courseware in promoting effective
learning (variable view of all the characteristics)

/\

 

Dimension 3

 

 

 

 

 

 

142

The plot was analyzed by visualizing it as a cube with the center of the cube as the
origin of the coordinates for the variables. A close inspection of the three-dimensional
plot revealed six distinct clusters of perceptions among the experts regarding the variables.

The distinct clusters were located in the upper, upper right, lower right, lower, lower left,

and upper left regions of the cube.

The following indicates the variables that were located in each of the regions.

Regions of the Clusters

Variables

Upper mid region CCSCRNL CLOGORG CHELP
CCSCRNB CICHUNK CAPNOVP
CMENU CINTERA
CGRAPH CNAVIGT

Upper right region CGASITU COBJ CON CILPROB
CSSCRNL CLGSKIL CNIVARY
CLGF ACT CUSEHOW CEPPROB
CERSKIL CCLAOBJ CPFF ACT
CNISIMP CISTRAT CPFCONC
CLBWREV CTESTOB CPFPROB
CAPCONC CF EEDBK CREVCON
CAPPSIM CGATASK CREVQUE
CERSREV CILCONC CEPBEHA
CNIACCU CILFACT

Lower right region CLCONTR CSRFACT CSRESUB
CSRCONC CSRSKIL CLGPROB
CNISKIL CINTAUD CAPSKIL
CPFCONS CPFSKIL CNIBEHA
CLGCONC CEPSKIL

Lower mid region CRCCONT CSRBEHA CLEVDIF
CLGBEHA CILSKIL CILBEHA

Lower lefi region CNICOGS CCOMPLX
CPFORIG CEPCONC

Upper lefi region CCERROR CTABCON CVIDEO
CVISUAL CGLOSSA CAUDIO
CBSCRN L CTXTSIZ CNIPROB
CSEFECT CTXTSTY CEPFACT
CTXTCOL CICONS CAPFACT
CHLINKS CANIMAT

143

Two major spheres were identified from the three-dimensional plot. One sphere
included a cluster of variables in the upper right region and another cluster of variables in
the upper left region of the cube. This sphere of cluster of variables consisted of
characteristics mostly related to principles of courseware structure, human interface
design (screen layout and design), learner control, and the use of media (such as video,
audio, and graphics). The second sphere included clusters of variables in the upper right,
lower right, lower, and lower left regions of the cube. This sphere of cluster of variables
were mostly concerned with components of instructional theories emerging from current
research in educational psychology triggered by the phenomenal incursion of new
computer technologies in education and training. The variables located in the regions
within the second sphere of the three-dimensional plot were related primarily to lesson
content characteristics as defined in Gagné’s events of instruction. A comparison among
the variables displayed within the two spheres and the results of the mean ratings of the
variables presented earlier in Table 5-5 revealed some close relationships. Most of the
variables within the first sphere of the three-dimensional plot were also the variables that
received high mean ratings. Similarly, variables that were located in the second sphere of
the three-dimensional plot constituted the majority of the characteristics that received
mean ratings of medium level of importance. As explained earlier, no variable received a
low mean rating.

The upper mid region contained variables that were primarily related to characteristics

of learner control over the pace, order, type, and amount of courseware lesson materials

144

to interact with (see Figure S-3a). The characteristics in this region included the use of
menu, screen design balance and coherence, navigational tools, amount of information
presented on each screen, help function, logical organization of lessons, and interactivity.
The use of graphics was perceived by the experts as at the same level of importance as
the characteristics relating to learner control. This perception may be due to the fact that
human interface design, which defines the nature of learner control, involves the use of a
lot of graphics. One other characteristic that was located in the upper region, although it
is seemingly unrelated in attributes and functions to human interface design or learner
control, was how well a courseware tests the learner on solving a variety of novel
problems. It was not clear from the pattern of perception represented in this region why
the experts perceived the latter characteristic as having the same level of importance as the

other characteristics displayed in the region.

145
Figure 5-3a

Enlarged view of the upper mid region of the three-dimensional plot of the variables
// /
\ \
L

cmovr V
C“ "' :t 'NAVIGI’
1 o ccs can]; u oconc
0 W“ n
on 4 RA
3i
\- q,
/
can!!! ‘
M

\

1.5

 

Dimension 3

 

O -
0.5
.\
,~95
*2.

 

 

 

/O/b D‘m°“s\09 ‘»

IN NO

 

Both the upper right region (Figure 5-3b) and lower right region (Figures S-3c)
contained the vast majority of the variables displayed in the three-dimensional plot.
Almost all the variables located in these two regions were related to lesson content

characteristics as defined by Gagné’s events of instruction. However, there were no

146

discernible patterns in the way the variables were displayed within the two regions. The
variables seemed organized as assortments of different components of the Events of
Instruction. The only variable that seemed different in nature from other variables located
within the two regions was identifying the audience of computer courseware. While the
characteristics may be somewhat related to lesson content, it was more closely akin to the
characteristics associated with courseware structure in this study. It was not clear from
the patterns of perception why the experts perceived audience identification as at the

same level of importance as other variables displayed in these two regions.

147

Figure S-3b

Enlarged view of the upper right region of the three-dimensional plot of the variables

 

Unms‘m 3

 

J'L

 

 

 

148
Figure 5-3c

Enlarged view of the lower right region of the three-dimensional plot of the variables

 

 

 

 

/\
1R \
\

O”
“i
5
a v

,1

O :2,

 

 

Q
[0 ,N /
1
/ O D\fl°n“o“

The lower mid region contained a cluster of variables that was mostly related to
characteristics of attitude and motor skills (see Figure 5-3d). An interesting observation
was the experts’ perception that the level of difficulty of lesson materials for the intended

audience was at the same level of importance as variables or characteristics of lesson

149
content on attitude and motor skills to be learned. It may be that the level of difficulty is

somehow related to the ease of learning achievement more especially in attitude and motor

skills domains than in other knowledge domains.

Figure 5-3d

Enlarged view of the lower mid region of the three-dimensional plot of the variables

/

 

\
-1 , y/

u
CRCCONT CILBEHA

M3
E
‘2

 

,C

’o‘(

"\
,N-O

/ n.

 

,'\
o

 

)3
I N 09

d’

\ ’Q$
)7,

/”\‘O W‘

,% ’I\§3

 

150

The variables that were located in the lower left region all seemed related to higher

cognitive processes (see Figure 5-3e). The variables were:

0 describing or showing the use of mental strategies or procedures to solve
problems

- complexity of lesson structure for the intended audience

- confirming originality of learner’s solution to problems presented

- eliciting performance by asking the learner to apply concepts and rules to

instances not previously encountered in the lesson

From the results of the mean ratings for these variables (as shown earlier in Table 5-5)
it appeared that the experts perceived the characteristics to be of only medium
importance in promoting effective learning via computer courseware. The mean ratings,
ranging from lowest, 1.0, to highest, 3.0, for the variables in the order listed above were

2.44, 2.38, 2.33, and 1.16.

FigL

Wat:

15 1
Figure 5-3e

Enlarged view of the lower left region of the three-dimensional plot of the variables

 

 

/ \
,09 <
mg: /\
11 F /
n , \. 1g
§’1'2 carco
E
5 /

 

 

 

 

The remaining region examined was the cluster of variables in the upper left region (see
Figure 5-30. As explained earlier, virtually all the variables located in this region were

related to characteristics of the use of media and human interface design. Overall, the

152

variables were perceived by experts to be of medium importance when cross examined

with the results of their mean ratings as presented earlier in Table 5-5.

Figure 5-3f
Enlarged view of the upper left region of the three-dimensional plot of the variables

/
// \

/

 

O
a i
W

 

Dimension 3

 

 

 

 

153

In sum, each region was examined closely for specific patterns of the perceptions of
the experts. It was observed that each region contained variables that were closely related
in attributes and/or functions. The variables located in each region were mostly linked to
some closely related events of instruction as specified by Gagné or sections of the
characteristics of computer courseware as specified for the purpose of this study.
However, each region also contained a minority of variables from other components or
sections of computer courseware characteristics as defined for the purpose of this study.
The results for each region indicated that the experts were fairly close in their opinion or
perception regarding the relative level of importance of the variables in each cluster. Each
variable in a cluster region was perceived as having fairly the same level of importance as

the other variables in the cluster.

 

154

Effects of demographic characteristics on experts’ perceptions regarding the level
of importance of computer courseware characteristics in promoting effective
learning

One way analysis of variance (AN OVA) was used to determine whether the
perceptions of respondents in the Expert Rating Form Group regarding the level of
importance of computer courseware characteristics to the success of the programs varied
according to their demographic characteristics such as: gender, age, professional discipline,
teaching experience, courseware development experience, experience with computers as
productivity tools, and computer ownership. The results are presented in Tables 5-7 and
5-8. As the results revealed, there were only significant differences in perceptions
regarding the characteristics of courseware structure among the experts by their gender (F-
value = 4.3) and age (P-value = .047). The result of a post hoc Tukey’s test (see Table 5-
8) indicates a significant difference in the perception of experts 36-45 years old compared
to those over 46 years old at the level .05.

Further analysis was conducted on the data using the stepwise regression procedure to
determine whether gender, age, or any other demographic characteristics, are significant
predictors of the experts’ perceptions regarding the level of importance of characteristics
of courseware programs in promoting effective learning. The results are presented in
Table 5-9. Age was the only factor observed to be a significant predictor of the experts’
perceptions at the .05 level. Compared to experts over 46 years old, those between 36
and 45 years of age are predicted to have higher perceptions regarding the level of

importance of the characteristics of the following sections of computer courseware:

155
courseware structure (B= 0.43, F=.Ol), prerequisite skills (B= 0.35, F=.05), and new

lesson content (B= 0.41, F=.03). In the case of screen layout and design, experts over 46

years old are predicted to have significantly lower perceptions regarding the level of

importance of that set of characteristics ([3: -O.36, F=.04). In general, it appears that

the older an expert is, the less likely he or she would consider the computer courseware
characteristics under study as important in promoting effective learning. The converse of
such a perception would be true for younger experts.

Although gender was a significant factor in explaining differences among experts in
their perceptions of the level of importance of the courseware characteristics, the
regression results indicate that it was not a significant predictor. Teaching experience,
previous experience using computer courseware and multimedia, and even professional
expertise in courseware development, all appeared not to be significant predictors of the
experts’ perceptions regarding the level of importance of the computer courseware

characteristics under study in promoting effective learning.

156

2. v .— 28555 .
5.2.3.5.. aha—Ea: m Gm {38:2 u z

 

 

 

 

 

 

 

 

 

 

 

 

 

Wm 033.

83. oeSN EN. SN 3 com. NYN c. 82:00 :83: 302
mug. 3.8.,” 3v. G.~ 3 new. 3:... 3 mid—m 833.32.— 93.53 ,
Raw. 3:. a3. Emu 2 En. Sm @— 9.5 «:82
23.. 2.3.. man. 23 2 av. inn 2 :33 v5 5931— 520m
.33. 323. new. SN 2 9mm. 28 2 833m 0.832560
, 2:27.— 0255— a m eaozlllmllllmw .-lm.rnlo.a. -l. m l. , i . l H; i t l l l -
9.25.,— o_a_z . uefieuomwoauafieu 9.3.38..an
gov—5w .3 wEEB. 03.8%.». 0026...“ :3 .9: 983858 659:3 mo «=3.on canon—Eco he 3330830 05 we

vocatoqé .3 .26. 2: wEanE 985 .Ecm wean: eon—xv... 05 5 35?...nt he 53882 no 3.3.2 355? he mam—SJ.

157

3. v ._ £85.55 .
5.38.. 23...... n :m :35: u z

 

«Eb. vaaaN Nam. Ned a an. GEN : awn. .v.N m— 33:00 :88; 302

 

 

 

 

$1. as: 8.“. com a 8.... a: : at RN 2 aim 9:352: 368..

 

 

 

253. 8: «=82

 

 

 

3:. 333 Sn. m_.N a one. nmN Z NVM. moan m— :mmmongfieasgom

 

 

 

 

 

 

 

 

  

«can?

vllll il illll. I: ,.i il. -l lil i .ll ll. ..l Illiiiilll l lli .. lli ill .l

    
    

 

33>-.— o:_:>....— Gm :02 Z Gm :92 Z Gm :9: Z

 

 
 

2.835
39.38:.90 8:309:30

  

 

 

 

 

 

 

02x3 6:: we

 

3.3: 6:: mm

r'lllliilll Ill ll 3 lli.

so» :_ em: .3
33.30. oéoobo 3:33 :8 35 2:39:30 339:8 .3 8338 0:89:30 .3 8333830 05 .3
853395 .3 33. 2: 33.3%: 935 53"— wfifia 9.me 2: :_ $532.8. .3 33022. .3 8.32 3:35, .3 aim—8"

w-m 23,—.

 

158

Table 5-9

Regression results of the perceptions of respondents in Expert Rating form Group
regarding the level of importance of the characteristics of courseware
in promoting effective learning

A _e in cars the cm redictor entered

W... Inn
sections value value

Illlfl-I
_ _ value 7 value value *

Courseware 0.43

 

 

 

 

 

 

 

 

 

 

 

 

 

* Significant at 0.05 level.

Other features experts perceived as important to the success of computer
courseware in promoting effective learning

Only a few comments were provided. The comments were related essentially to the
type and scope of learning that can be mediated via computer courseware. The comments
are documented and analyzed below.

One factor an expert thought might be crucial to the success of computer courseware in

promoting effective learning is designing computer courseware in ways that augment the

159

classroom needs of a teacher and also facilitate learning processes that promote the
learner’s cognitive processes. In the words of the expert:

I think a good computer courseware is not only (in your evaluation cases)
dependent on the software itself, but also how the instructor uses it in the
classroom. I feel that in your evaluation form, it seems that the sofiware will do
everything the instructor (teacher) will do. For example, it seems to me that it will
be difficult to implement “eliciting the performance” in the “normal” computer
courseware, except the so called “intelligent tutoring system.” I am also not sure
that using the computer to give precise and quick feedback is really good for the
learner’s cognitive thinking and learning.

Another expert expressed the same concern slightly differently. The expert remarked
that while some characteristics may be important in some type of educational software it
may not be so important in others. As the expert noted:

Generalization is difficult if a courseware is just a tutorial. Some parts are not
crucial, while if it is at simulation level or problem-solving, those not-so-important
parts will become a necessity. So depending on the level of courseware, different
items gain importance.

The same concern was expressed by another expert who thought that a “one-size fits
all” assessment instrument for computer courseware might be difficult to construct, if not
impossible. The expert noted that:

Without knowing the purpose of a specific courseware module, it is impossible to
rate the characteristics as to their importance for the module’s effectiveness.

Some modules teach by repetitive practice (e.g. typing); others present complex
and expect collaborative discussion. What makes them effective examples of this
group will differ.

The utility of the computer courseware is a factor that an expert considered paramount

to the success of courseware programs. According to the expert:

160

The most important (consideration with courseware) is ease of use for both
students and instructors. No long introduction. Each step should supply concise
context of learning.

The way computer courseware reacts to a user response is thought to be crucial to the
success of educational software by an expert. For instance, computer courseware should
offer more than just stimulus-response learning modality. In the words of the expert:

‘Correctness’ implies only one possible solution or answer to problem; in many
cases this is true, however, regardless of the range of possible ‘correct’ answers,
developers should not discourage participants who may not be grasping the
materials. Instead of returning an error, the courseware should take the student
through a review of the material they did not grasp. ‘Grasping’ is not black and

white and the developers must anticipate numerous ‘correct’ answers in many
situations.

Section Three -— Perceptions of Respondents in the
Expert Courseware Evaluation Group Regarding the Quality
of Computer Courseware in Promoting Effective Learning.

This section answers the research questions relating to the perceptions of the
respondents in the Expert Courseware Evaluation Group regarding the quality of the
characteristics of computer courseware programs used in this study. The experts were
asked to rate the quality of the characteristics of the courseware programs in promoting
effective learning. The research questions for this section were constructed to fulfill a
two-fold purpose. First, data collected from experts in response to the questions will
help in determining the characteristics of the computer courseware programs that experts

consider salient to engendering effective learning of the lesson materials. Second, the data

 

161

will provide information useful in constructing an evaluation framework for an authentic
evaluation of computer courseware.

Data collected to answer the research questions about the experts’ perception
regarding the quality of the courseware under study utilized the Computer Courseware
Evaluation Checklist Form (see Appendix C). The questionnaire was a variant of the
other two questionnaire instruments used in this study.

Like the analyses conducted in section two, two levels of analyses were applied to the
data collected using the Computer Courseware Evaluation Checklist Form. The first level
involved simple data analysis to determine relationships among variables. The second
level was based on the results of the simple data analysis and was more intensive. The
intensive data analysis focused on the differences and similarities among experts who
participated in the study in terms of their perception about the degree to which the
quality of the computer courseware programs influenced effective learning of the lesson
material.

The major question guiding this section of the report was

Research Question D:
What are the experts’ perceptions regarding the quality of the
characteristics of computer courseware programs used in this study?

The subsidiary questions were:

Research Question D1: What are the experts’ perceptions regarding the quality of

the characteristics of the computer courseware programs used in this study in
promoting effective learning?

162

Research Question D2: Do the experts’ perceptions regarding the quality of the
computer courseware characteristics vary by their demographic characteristics
such as gender, age, professional discipline, teaching experience, courseware
development experience, experience using computer productivity tools, and
computer ownership?

Research Question D3: What other features of the courseware programs used in

this study do experts perceive as unique and helpful in promoting effective
learning?

Experts’ perceptions regarding the quality of computer courseware characteristics
in promoting effective learning

The respondents in the Expert Courseware Evaluation Group were asked to rate how
they perceived the quality of each courseware program under study as helpful in
promoting effective learning. The mean of the quality ratings (importance ratings) and
standard deviations were computed for the characteristics of each composite section of
the courseware programs. The possible mean ratings ranged from the lowest, 1.00,
indicating that the set of composite characteristics is rated very poor, to the highest, 5.0,
indicating that the set of composite characteristics is rated very good. Table 5-10
presents the means, standard deviations, and ranks of the experts’ perceptions regarding
the quality of the characteristics of the courseware sections.

From the information on the table, two sections of courseware 2 top the list of rank of
mean ratings. First in rank was courseware 2 structure (mean = 3.83). The prerequisite
skills section of courseware 2 ranked second (mean =3.75). Third in rank was the

characteristics of courseware 3 structure. Four of the five sections of courseware l were

163

in the last four ranks on the list of mean ratings. Overall, courseware 2 had the best mean
ratings, followed by courseware 3, with courseware 1 having the lowest mean ratings. In
general, the experts had a more positive perception of the quality of courseware 2
characteristics than courseware 3 characteristics which in turn were perceived better than
the characteristics of courseware 1.

Almost all of the courseware sections were rated as fair in quality as measured by the
mean ratings which ranged from the lowest, 3.28,.to the highest, 3.83. The only exception

was the prerequisite section of courseware 1 which was rated as poor (mean = 2.97).

Table 5-10

Means, standard deviations, and ranks of experts’ perceptions regarding the quality of the
characteristics of sections of the courseware programs under study

i R32
34

i
i
~ Courseware 1 .61 12

     
  

Courseware Structure r Courseware 2 34 .53 1

i Courseware 3 34 .67 3

5 * Courseware 1 34 .74
l Screen Layout and Design i Courseware 2 34 .72
l Courseware 3 34 .70

 

 

 

 

l Courseware 1 34 .81
I Media Use i Courseware 2 34

I Courseware 3 34 .96
l Courseware 1 34 .73
i Prerequisite Skills Courseware 2 34 .82
. * Courseware 3 34 .68
i ‘ Courseware 1 34 .60
i New Lesson Content Courseware 2 34 .59

 

 

 

 

Courseware 3 34 .90

164

Effects of demographic characteristics on experts’ perceptions regarding the
quality of computer courseware characteristics

One way analysis of variance (ANOVA) was used to determine the extent to which
the perceptions of the respondents in the Expert Courseware Evaluation Group regarding
the quality of computer courseware characteristics varied by their demographic
characteristics such as gender, age, professional discipline, teaching experience,
courseware development experience, experience with computers as productivity tools,
and computer ownership. The results are presented in Tables 5-11 through 5-20.

As Table 5-17 indicates, a significant difference in perception regarding the quality of
courseware 1 new lesson content characteristics was observed among experts who had
developed some courseware programs and those who had not. Experts who had not
developed any programs had a more positive perception about the quality of the
characteristics than others who had developed some programs (F=3.35, P=.049).
However, no significant difference in the perception was observed between experts who
had developed between one and three programs (mean = 3.16) and those who had
developed at least four courseware programs (mean = 3.19).

As shown in Table 5-15, a significant difference was observed in the perceptions
regarding the quality of courseware 2 structure between experts without content expertise
and those with expertise in the content area covered by the courseware programs under
study (F=4.22, P=.048). The experts without content expertise had a more positive
perception (mean = 3.89) of the quality of the characteristics of courseware structure in

courseware 2 than experts with content expertise (mean = 3.35). Table 5-17 indicates

 

165

that there was also a significant difference in the perception regarding the quality of
courseware 3 prerequisite skills characteristics between experts without content expertise
and those with content expertise (F =6.5, P=.02). The former had a more positive
perception (mean = 3.72) of the quality of courseware 3 prerequisite skills characteristics
than experts with content expertise (mean = 2.87).

No significant differences were observed in the experts’ perceptions by gender, age,
teaching experience, experience using computers as productivity tools, or computer
ownership. Also, no significant differences in perceptions regarding the quality of the
following characteristics of the courseware programs were observed among experts by
any demographic characteristics:

Courseware 1: courseware structure, screen layout and design, media use,
and prerequisite skills

Courseware 2: screen layout and design, media use, prerequisite skills, and
new lesson content

Courseware 3: courseware structure, screen layout and design, media use,

and new lesson content

166

 

5.3.5.. 95.35: H Gm 2.2.3.2 u Z

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

mmcm. ego.— v...\.. 2.6.. N. NfiNm 93 NN 32:50 :88. 302 i m 2:38:50
mmwo. N53. 3. ac... N. .5. an... NN 2.2m 2.8.522: i M 2:38:50
08m. mmmm. 3. aN.m N. co. av... NN 8: :33). i m 2:38:50
3%. ex. . . N... .3... N. we. mm.m NN 5.80355. :23 i m 2:38:50
Em .. mvmmN em. .55 N. .5. mm... NN 2:336 2:38:50 i n 2:38:50
2.3. NN 5. mo. .06 N. on. S... NN . 32:50 :83. 352 i N 2:38:50
. .3. 2...”. .3. 3..“ N. a... Na... NN m...u.m 2.2—522.. i N 2:38.50
v.5. ooNc. oN. .0... N. ca. 5.”. NN 8D :30: i N 2:38:50
Nwa. mmmc. .w. R... N. ac. Nm.m NN 3.30:5»... :25 i N 2:38:50
”an. meoN. mm. 8.». N. Nn. ohm NN 2:3.5m 2:38.50 i N 2:38:50
mmw .. com... VNd an... N. .3. mN.m NN 32:50 :88. 302 i . 2:38:50
.33. ommm. 3. SN N. w... No.” NN m...v.m 2.8.529... i . 2:38:50
Now... 33. .w. mm... N. m... 92 NN 8: :32). i . 2:38:50
m.vm. mmma. v... .n.m N. o... nN.m NN 3.800353%. 52% i . 2:38:50
3.9.. 3.0... nm. S...” N. no. .m.m NN 25.02% 2:38:50 i . 2:38:50
33>-.. 33>-.. Gm :32 Z am :33. Z
2:th 2:2 32.398.32.300 2:392:50

 

 

 

 

 

 

3.3% .3 333:8 .9 353.83....
.323. :88. 2.. :30. 3.93.83: 5. 32.. 33.2. :. 82.338.85.38 2:38.50 0... .3 8.83.0220 2.. .3
3:25 o... 3.332 95.0 :o..:=.:>m. 2:38:50 .396. 2.. :. 3:03:82 .3 85.3022. 2.. .3 2.82 03:13.3 29.35.
..-m 03:...

167

3.3.3.. ...-.38: n Gm ...0..E..Z u Z

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Sam. Nmaa. wNm a. .e ... N.NN an .. N.aN N.a.. a 30.50 :83. 302 - m 2:38:50
0 ...... 3:. we. vm... v. w... am...” .. ...... no... a ml=...m 2.83.0223 - m 2:38:50
voao. coon. mm. .m.m v. w... .m.m .. .a. a... a 8D .....03. - m 2:38:50
.vav. EN... S. «Wm v. No. 2.... .. o... mm... a 3:800:53». :020w - m 2:38:50
own... at .. ..v. we...” .... va. 3... .. S. a...” a 230.....m 2:38:50 - m 2:38:50
mama. ..Noo. em. and ... mm. 3... . .. o... no... a 32:50 :83‘. 302 - N 2:38:50
mem. 2.8. .6. ..m... v. we. ma... .. ac. a...” a 8.....m 2.8.3223 - N 2:38:50
SaN. oveN. 0... ea... v. a... 9m... .. .. ... .0... a 8D :30: - N 2:38:50
mom»: SN... 5... um... v. ..n. E... .. .... 0N... a 3.30 35.3.. :020m - N 2:38.50
emoa. maao. av. a...m ... mm. 3.... .. .... an... a 23.03% 2:38.350 - N 2:38:50
. ...N. mam... No. On... v. .m. NN... .. no.0 nm.m a 32:50 :83. 302 - . 2:302:50
.Nma. mom... .a. 8... ... on. Na.N .. m3. va.N a 2...... 2.83.0223 - . 2:38:50
..aS. ocwv. an. no... ... .0. we... .. ..ca. cm... a 8D :50: - . 2:38:50
2...... 3. 5.. ma. :0». v. ..n. onm .. 3. ac... a 3.3m. :5»... :020m - . 2:38:50
:85. 39:: av. an...“ V. cm. av... .. ..w. ...... a 230.....m 2:38:50 - . 2:38:50
033... 023-... am :32 Z am :35. Z Gm :33. Z
...: 3.3..
23.. 0.5:. .... 3. .0... 23.. 3-0m .3: 23.. mm n:5..00m 3:23:50 03302.50

 

0w: .... 338:8 .nv ~85.5.9.3...
.:..0.::. :83. 0... E3. 3.02.0030 5. 30... 33.0.. :. 3.83383588 2:38.58 0... ..5 8083.038 0... ..5
...—...... 0.... 3:230. 3.5.0 :5..::.:>m. 2:38:50 ..me 0... :. 33.3582 ..5 35.3020: 0... ..5 8.82 003...? ..5 £3.92.

N.-w 03:...

 

168

52.30.. 1.3—2.3. u Gm 3.2.552 u z

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

$5 $8. SNN N36 N 8.: 8% 2 22:5 :83 52 l N 2558.80
22. 28. .N. a: «N 3. RN 2 2.3m ”@320... l N £38.80
2%. eat a. WNN N 8; SN 2 8: «=5: l N 9338.80
88. SN“. 2. .2 .N an :N 2 ..NENSBNS 520m | N 33858
..NWN. 8%. an. :N «N B. .2 2 eaofim 9338.80 | N 293980
53. N _ 8. 8. 8N N X. ‘ a: 2 22:8 :88; 52 I N $38.30
NNS. 8... 28. N2 «N 2: N 2. 2 2.2m 2:33.»... I N 033858
38. £3. 2: N2 N N3 23 2 a: 28: | N 0338.80
28. 82R NN. ....N ..N 3. SN 2 cwaonsaifi 82% | N $33.58
N 3N. an: 3. N; «N no. «3 2 228% $38.80 I N €858
$8. NNNN. m: S. ..N 3. N2 2 22:8 :88; >32 I _ uaafisoo
:8. N .8. NN. 3N «N :2 3N 2 2.5m 3333.2; I _ E3858
:8. :3. NN. NE“ ”N 8. n2 2 3.: «=52 I _ $38.80
8:. 8 _ n. 3. ”Na «N No. SN 2 $50233 52% l _ $38.80
83. 2%. a. N2 «N NN. SN 2 2202a $38.80 | _ 295358
2:57.— o=_a>-~_ Gm :32 Z Gm 532 Z
a Z 3> 233993339590 3:59.30

 

 

 

 

 

 

 

Even—£05“. 2958.30 E 85.598 :33 .3...qu .3 Amino—.8 .9 32393.:
353E 520— 05 ER.— boéootu 8 E05 mat—o: E mo._mcano\m:c_.oom Egon—=8 05 mo sawtouofiaso 05 .3
5:56 2: 9:25?! 95.5 cogs—gm 2958.590 toaxm 2: £ 952.88; ho 22.320; 05 be 9:56.. 85?; ..o £9222

m _ -n 03:.

3% 02.; - £i...:=>%. «Nabacax

 

 

169

 

00:030.. ...—00..... ..u :m .00.-=5Z u Z

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

”03.. v08. afiwN nose 2 Ndm $.00 2 50500 0080.— 302 I. n 0538500
535. 008. 2.. $6 3 00. and m. 0:05 0385020...— I m 0.538500
05?. avg. 20.— vmd 2 K. ”Na 2 8D 0:002 I n 0.838500
Nnvw. the. 2.. mm.m 2 Nb. and w _ 0500900055 000.0w I m 0.538500
ova. 33. 2.. 00». 3 N0. and n. 0500s.:m 0538500 I n 0538500
002. .03..— 0m. :6. 0. 3. 3mm 2 50500 00.80; 302 I N 0538500
See. 2006 ax. Man a. 00. 36 w. .0.—Em 007500.05 I N 0538500
00 _ m. 008.. o._ 56 N0— 2: med n _ 8D 0:002 I N 0.538500
000m. 500; mu. m0.m 2 so. and 2 090000005.— 0020m I N 0.538.500
000N. me _ ._ em. N06 2 0w. Qum 2 05.03% 0538.500 I N 0538500
Emm. Nag. 0m. gum 2 Nhé Nvé m _ 50500 0080.— 302 I _ 0.538500
Nam. moms. mp. nod 0. Nb. 3N n _ 2.05 0.500205 I _ 0538500
33. C 5. Na. NWM 2 mm. 3.». n. 8D 0:002 I _ 0538500
3.2.. 2.00. Nu. ”ma 2 3.. «NE W _ 030000005.— 00000m I _ 0538500
E _N. :00. 3. avg” 2 mm. NN.m n _ 05.03% 0.5300500 I _ 0538.500
02:..— 0=_a>-..— Q m .502 Z a w .502 Z
0 Z 80> 50200900500500 0.538.500

 

 

 

 

 

 

 

.00.—50.050 0.038500 5 80.098 .23 has .3 3000000 .9. 30.0.0530
.0085. 0080— 05 E00— >_0>=00.=0 0. E05 9.5.0.. 5 803800500300008 0538500 05 .00 0000:8085? 05 .00
3:50 0.: 55.50.. 50.0 gag—SE 00038500 009...; 05 5 300000000. ..0 002500000 05 .00 8500.. 005:5 ..0 mam—00¢.
:6 030,—.

170

n... v 0 0.80.5.0 .
00.5.5.0 .05—.00.» n 0% £00502 u Z

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Nomw. N000. mN.0N 00.00 on 00.2.N 2.. .2. 0 .00.000 0080.. 302 I m 0.038.500
000.0. 0.2.00. .0. N2...“ 0m .0. SN 0 0.....m 00.05.00.000 I n 0538500
N02. mnemN 2.0. Nm.m on 00. 2..N v 8D 0:002 I m 0.538.500
N0 . w. 0000. N0. 00.... cm N0. 0v... 0 E80055; 000.0% I m 0.538500
000m. Non... 00. .0..” 0m 3.. and v 05.50% 0538.500 I n 0538500
002... $00.. 00. 00..” em nm. N...“ v 50.000 0080.. 302 I N 0538500
.wNv. .30. 00. KM 0m 00.. 00... v 0.5% 2.0.0.0000... I N 0538500
.0000. 00.0 .5. 00.m on 2.. an... 0 8D 0.00.2 I N 0538500
2.0.00. .500. 3.. 00..” on m0. 000 9 0900000050.. 000.0% I N 0538500
..va0. X. .N... .m. 0....” 0m 3.. nm.m 0 05.05% 0538500 I N 0538500
0000. van. mm.m m0.m om .m0N. N0.m 0 .00.000 0080.. 302 I . 0.538500
SAN. 000m. 2.. No.” on .0. SN 0 050% 2.8.0.0000... I . 0538500
002.. 2.00. 2.. 0.0.... cm .... m0.m v 8D 0.00.2 I . 0538500
0000. 0000. 02.. 00m 0m 00. mm... v 55000005.. 000.0% I . 0538.500
0.0N. N32... .0. N06 0m 00. 00.N v 05.05% 0538500 I . 0538500
00.03-.. 00.0.»-..— nm 000.). Z am 0005. Z

02 00> 00020023050500 0.538.500

 

 

 

 

 

 

 

005 .0208 0. 8.00008 .3 5.000000 .9 30.0.0.3...
3.0.0.0 0080. 0... E00. 0.02.0000 0. 50... 00.0.00 0. 00000050000008 0538500 0... ..0 00.800.00.500 0... ..0
5:00.. 00. 00.05000 000.0 0000203.. 0538500 .5050 0... 0. 3000000000 ..0 0000000000 0... ..0 9.000. 0000.5> ..0 m.m>.00<

0 .-m 0.00...

00.5.3.0 0.00.5.0 u Gm ..0050Z u Z

 

 

 

 

 

 

 

 

 

171

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

£8. 88. .50 ~00 a 3.0 o... 0 00. 0.00 0 0.2.50 :83... .52 | ... 205280
0 . .a. 0.. .... 00. 8.0 0 . ... a... m .0. 8.0 0 0...... 2.0.0.22. | 0 2038.80
23. $8. 8.. 00.0 0 00.. 00.0 m .a.. 00.0 0 a: 0.8:. | 0 203858
8.5. 0.8. .... 00.0 0 a. 0....“ m 8. 00.0 0 5.800803 =0200 I 0 0538.80
0 . 8. 0...... 00. $0 a a... 00.0 m 8. 2.0 0 23250 2033.58 | 0 29.8.8.0
.8... 08... R. on. 0 ... 2.0 m .0. ...... 0 0.2.50 :80... .32 | N 2038.80
0.0... 05... E. 00.0 a 0.... 00.0 m 00.. 5... 0 0...... 2.0.0.22. I 0 2038.80
0000. 0:0. ...... 3... a 8.. 0.0 0 .... $0 0 a: 0.82 I N 203858
.a.... 080. a. $0 0 .0.. 2.0 m 00. 3.0 0 .0000 0.50... 5200 | N 0338.80
80.. 32. NW. ...... 0 8. 3.0 m 00. 00.0 0 23250 238.80 I N 20:00.80
800. 8 .... ... .2 ... 3.. 0.0.. m 2. 2.0 0 0.2.50 :80... 302 I . 2038.80
.8... ..0. .. .a. 8.0 0 00. ....N m .0. 8.0 0 . 0...... 2.0.0.22. | . 2038.80
83. .03. .0.. $0 0 0.. 00.0 m 00. 00.0 0 I a: «.32 | . 2033.80
.0.... a... .. ...... .2 o 3.. a... m 00. 00.0 0 5.80 0.5.3 :88 | . 0533.80
E .. 0.8.0 .... .2 o ...... 8.0 m .... a: 0 23020 203858 I . 2038.80

00.0.3-.. 00.0.7... 9% 0005. Z 0% 0005. Z 0% 000.). Z I

0.500 000 .N ..N... .0000 .000 ... 000.30% 08.000050 0.0305000

 

 

 

 

 

 

 

050.. 0. 0000.505 00.0050. .3 0.000000 .9. 30.0.0000.
5005:. 00000. 00. 050. 0.0.0.0000 0. 0.00. 00.200 0. 00000058000008 05300500 00. ..0 02.000.00.500 00. ..0
0.000.. 00. 00.05%. 000.0 00.50.90 0538.000 009m. 00. 0. 0.00000000. ..0 000..000.00 00. ..0 00000. 35.5., ..0 0.00.000.
0.6 0.00...

 

3. v .— ...8555 .
55:3: tun—Ea: u Gm ...2.:=.Z u Z

 

 

 

 

 

 

 

 

 

172

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3:. 8R. . a: 2K 5 SN «.3 e ”.2 3: m 2280 :83 52 l m $38.80
3%. as. S. a: h 8. ”on e a. 8a n max“ 2:532: l m 2.38.80
NEW. 8:. 3. 3m 5 3.. N2 2 N: 2 n a: 282 l n 2.?358
59... :2. S. S: b E. a; e a. Q: m gméaifi scam I m “.3558
CE. 32. 3 2a 5 K. 02 2 x. v: m 228% 83858 I m 2958.80
_ E. Na.N. 8. 3a 5 3. 3m 2 3. gm m 2.2.50 :88; 52 | N 2.?858
3:. m _ a. S. «2 h 8. a: e x. :2 m 2.3.». 2:352: | N $5280
8. m. SS. 8. v2 5 z: :6. 2 a. 8:. m a: 282 | N 22,858
2?. $3. 3. 3m 5 8 8a 2 S. 8.,” m :98: 25:3 52% | N enafiaoo
:3. $8. an m: 5 mm. Ed 2 a. 8+ m 232:5 033280 I N aaafisoo
:95. $5” 3 2 .m h 3. 24°. a 8.» 8.: n E280 :88: 302 I _ 2.28.80
3:. :5... S. ”.2 5 fl. Na.N a Q. :2 m 23: 2:322: | _ engage
3:. ”Sm. 3. 3m 5 8. Rd 2 9.. com m a: :82 l _ 253858
c: _. 88.“ 3. SN 5 8. 2A 2 am. 3% n :38: 323 520m l _ $38.80
G _ MN. 8%.. 8. :2 5 mm. 8: e 2.. a; n 222:5 253858 | _ assuage
2:5»-.— o=_a>-...— Q m :32 Z a m :32 Z Gm :32 Z
95:. .5 v n; c Esteem 30:33:50 9:35»..ch

 

 

 

 

 

 

 

 

3:205: 2:8on 033858 mo .Bfiz: 22535:: .3 330:8 .nv QOEVDEV
3.22: :88. 05 E8. bo>=8bo 9 E2: win—u: :_ mo._mon_:8\m:o_.oom 2:33.58 05 ..o £65682“. 2: we
5:26 2.. 9:8an 9.90 guns—gm 2950250 9.me 2: :_ 2532?: ..o 22:69.0: 2: no $.82 85:: Ho £3.93.
2%. 05a...

173

00.00.30 0:00:30 u an .00.—:52 M Z

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

0.8m. moaN. ..m.mN 8.3. w N_.0N .000 0N 0:00:00 :0000‘. 302 I m 0.038500
0. .00. 0mg. 3. 00.». a no. 00.n 0N 0:05 00.050220 I m 0.0300500
000.. 800.. m ... .w.m w on. .m.m 0N 8: 0:002 I m 0.0300500
Nmo... 0000. N... M06 x K. mwd 0N 5.000000501— :00.0m I m 0.038500
Nwmv. No.0. 00. .m.m w an. N0...“ 0N 05.03% 0.0300500 I n 0..0300500
0N3. 000m. .300. flan a N0. and 0N 50500 000001. 302 I N 0.0300500
30.. 2.0..N m... N..v x 00. 00..” 0N 0...a.m 00.05020... I N 0.038500
50w. onN. N. m...” w 00. and 0N 8: 0.002 I N 0.0300500
ER. . .NN.. .... 05m x N... 50m 0N 5.0000000... :00.0m I N 0.038500
00mm. 0000. 0m. 00.m a ..n. and 0N 05.03% 0.0300500 I N 0.038500
003. N20... 2.0 86 a 0m. 0N.m 0N 0:00:00 :00004 302 I _ 0.0300500
SoN. 33.. mm. 0N.m 0 Eu wwN 0N 0:0.m 00.05020... I _ 0.038500
8N0. 0N3. 00. 00m 0 0m. 00..” 0N 8D 0.002 I . 0.0300500
80m. 0.00. 0.0. 0w.m w 00. ”Na 0N 5.0002553 :00.0m I . 0.0300500
0NNo. 2. :0 00. 06m x 0m. «Nd 0N 0.5.00.5 0.0300500 I . 0.038500
05.0.3-0 05.0.3-0 Gm :002 Z Gm :002 Z
0 Z 00 > 00020500350500 0.038.. :00

 

 

 

 

 

 

 

05050.0 0.005055 0:0 0.038500 .050500 5.05 000000000 00030.0 .3 300500 50 30.0.0300
3.0.0:. :0000. 0... E00. 38:00.00 0. 50:. 5.0.0.. :0 00000050200008 0.0300500 05 .00 000000.0950 0:. 00
5:00: 0... w:.0.0w0. 050.0 5:05.05 0.038500 0.00.0".— 0... :. 0500:0000. 00 050000.00 05 00 00.0009 00:010.» ..0 0.0.0.05.

a .-n 030,—.

174

5.5.2.0 ...—00:80 N Gm .00.—5.2 u Z

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

200. $8. 0.0N 0.00 N. 0. . 0 0.00 NN .550 808.. .52 l 0 2038.80
.0N0. 0N... N0. 0:. N. 0.. 00.0 NN 0:20 2.05.022. l 0 2050.80
2 ... 0.00.. 3. .00 N. a. 00.0 NN 3.: 0.82 l 0 2038.80
N50. 000.. 00. :0 N. .... 90 NN 5.800803 =880 | 0 203858
NNR. 000... 0.. 00.0 N. 00. 00.0 NN 22250 203858 | 0 2033.58
0...... :03. 00. 00.0 N. 00. 30 NN 2.2.50 808.. >82 I N 20:00.80
00.5. $8. .0. ...... N. 8. .00 NN 2.0.0 2.0.520... | N 2030058
0N8. 000.. 00. 0...... N. 8.. 00.0 NN 00: 0.82 | N 2038.80
8.0. 00.0.. 00. E0 N. 00. a... NN 58008.3 =8.00 .l N 203858
.NN.. eNmN N... N3. N. 00. 02 NN 22250 2038.58 1 N 2033.58
0 . 00. 0.00. N... a... N. .00 00... NN 2.2.50 :88.. 302 l . 20300.80
00%. MNVN. 00. 00.N N. N... No.0 NN 2.0.0 2.00.8.0... l . 20300.80
0:... 000NN 00. 0: N. 0.. 00.0 NN 00: 0.8: l . 20300.80
N80. 0 .00. 00. 00.0 N. 0... 0N... NN 5800800.. .8080 l . 203850
N02. ...N .. 00. N: N. 3. .0.... NN 22250 2030058 I . 2038.80
00.0.0-.. 00.0.»-.. cm :00: Z am :00: Z .
0 Z 00 > 05200900350500 0.0300500

 

 

 

 

 

 

 

5.55.5.0 .0.:0500 ....3 005.505 05.5.0 .3 0.00050 .9. 30.0.0.2...
.0..0.0:. :0000. 0... E00. 0.03.0000 0. 50... 5.0.0.. :. 00000050205008 0.038500 0... ..0 00..0..0.00.0..0 0... ..0
.....05 0:. 5.0.05. 05.0 :0..0:.0>m. 0.0300500 000...... 0... :. 0.505000. ..0 050000.00 0... ..0 0.500. 00:0..0> ..0 0.0.0.50.

07m 030,—.

175

00:00.50 .05—.050 u Gm .00.—052 n z

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

m _ No. 0wVN. aodm co. K 0. SEN 0.8 2 0000000 000004 302 I m 05300500
0. _ 2.. 02:. 2.. 0m.m 0_ m0. 0.9m 2 0:05 0005000000 I m 05300500
00mm. mam. __._ Nm.m 0. E. 0nd 2 003 0:002 I m 05300500
300. Nmmz. we. 3...“ 0_ 2.. _m.m 2 5000000003 00000m I m 05300500
EN. $00.— 00. mm.m 0. 3. mad 2 05.03% 05300500 I m 05300500
RP. :8. Nn. and 0. we. 35 2 0000000 000004 302 I N 05300500
ova. 2.0 _ ._ mm. No.0. 0. 00; N06 2 053m 0005000000 I N 05300500
33.. mas. 0: 004m 0. 00. 0m.m 2 00: 0:002 I N 05300500
0. me. 008. N5. 0n.m 0_ 00.. «van 2 0900000003 00000m I N 05300500
m _ N. mva._ 3.. Eu». 0. cm. 00d 2 0050:0m 05300500 I N 05300500
N Sm. ”Nov. 0:. 2.0 0. no. 006 2 0000000 000004 302 I _ 05300500
ENN. mNnN. In mwN 2 2.. 2.0:. 2 0:05 0005000000 I _ 05300500
0N0». 2:0. an. N2” 0. 2.. ”WM 2 00: 0:002 I _ 05300500
03». 2. S. 00. 0.0.5 0. ma. Nm.m 2 500000054 00000m I 0 05300500
mooN. oNcNg hm. 0N0 2 me. 006 2 0500006 05300500 I _ 05300500
00.0.70 00.0.»-..— Gw 0002 Z Gm 000: Z .
0 Z 00 > 00330208000500 05300500

 

 

 

 

 

 

 

00050—000 MES—00000 0000:0006 0003000 .3 0.000000 .9 30.300000
50055 00000. 00. 050. 303000.00 00 500. 050.00 0_ 00000058000080 05300500 05 00 00005000500 05 00
5:50 0.0 5:500. 0:000 00_.5_0>m 05300500 00005 00. 0_ 0000000000.. 00 0000000000 05 00 0.502 000.050, 00 £00.53.

0N-m 030,—.

176

Further analysis was conducted on the data using the stepwise regression procedure to
determine demographic factors that might be significant predictors of the experts’
perceptions regarding the quality of the characteristics of courseware programs under
study. The results are presented in Table 5-21. As the table shows, only expertise in
content area and expertise in courseware development were significant predictors of
perception regarding the quality of computer courseware characteristics. Expertise in
content area was a significant predictor of the quality of the characteristics of the

programs under study: Courseware 1 new lesson content (IF-.979, P=.OO4), Courseware
2 structure (B=-.879, P=.049), and Courseware 2 new lesson content
(B=-.926, P=.024). As a predictor, content expertise accounted for approximately 90%

of the variance in perceptions regarding the quality of the characteristics of new lesson
content in both courseware l and courseware 2 (see Table 5-21). This predictor also
accounted for approximately 80% of the variance in perceptions regarding the quality of
the characteristics of courseware 2 structure (see Table 5-21). In general, the results
revealed that expertise in content area was more likely to predict a negative perception of
the quality of the characteristics of the computer courseware programs.

Another significant predictor of the quality of the courseware characteristics was
expertise in courseware development. The factor significantly predicted the quality of

courseware 1 screen layout and design (B=.964, P=.OO8), and courseware l prerequisite

skills characteristics ($0957, P=.011). As a predictor, courseware development expertise

177

explained over 90% of the variance in the experts’ perceptions regarding the quality of the
characteristics of both screen design and prerequisite skills (see Table 5-21). The results
revealed that experts in courseware development are more likely than other groups to
have more positive perceptions regarding the quality of the characteristics of the screen

design and prerequisite skills sections of the courseware programs under study.

178

..25. 0...... .. 58.0.0000 .

             
             

  

- 1 50.000

00003 302 ..N 05300500

05.0.05 - N 05300500
50.000

000.00.. 302 .. . 05300500
0=0_m

00000000000 - . 05300500
aw... .5

50.03 00000m - . 05300500

005 000.000 0. 0000005..
005 50.000 .0 000000000

 

 

005 50.000 0. 000000..."—

 

50.002030
05300500 0. 000.0000
52000—050
05300500 0. 000000000

0000...”.— 000.200...—

 

 

 

 

 

 

 

 

0...:
0.. 0.0I -.p

 

5.050. 03.00.00 500.0000 0. 000.0 0000.. 00500000 05300500 0.0 ..0 0000.00.00.20 0.0 ..0
0.0.5.. 0.0 5.0500... 000.0 000.5030 05300500 0000.0 0.0 0. 0500000000 ..0 0000000000 0.0 ..0 0.0.000 00.00050..—

 
 

000_.000 0003000000

.N-m 030,—.

 

179

Other features of the courseware programs used in this study that the experts
perceived as unique and helpful in promoting effective learning

The respondents in the Expert Courseware Evaluation Group were solicited for general
comments on other features or characteristics not listed on the evaluation questionnaire,
which they perceived as helpful in or inhibiting to promoting effective learning via
computer courseware. The experts’ responses were compiled and grouped together
according to the respective courseware programs in cases where a specific comment
applied to a particular program, or according to themes in cases where comments applied
to all three programs. The themes under which the comments are classified were derived
from the instructional and learning principles expressed in the comments themselves.

Comments that were nebulous or did not provide enough information about how a
mentioned feature helped or did not help learning processes were not included in the
report. For instance, a statement like “the programs did not present objectives” was not
included since a statement concerning objectives was already on the evaluation
questionnaire. Also, the statement would not be included in the reporting since it lacked
specificity in terms of how the absence of the feature from a program affects learning
processes. The following were the categories or classification schemes used:

Overall quality of the courseware programs
Chunk of information presented

Language style

Interactivity and learner control

Questions and feedback

Lesson review

User interface design

$999939?"

180

8. Visual appeal
9. Media use
10. Environmental factors
1 1. Evaluating programs and quality improvement
12. Specific recommendations for improvement of the three programs
Where appropriate, an explanation is provided to contextualize a comment. The

classification schemes, comments, and discussions are presented next.

Comments common to all three programs

Overall quality of the courseware programs

There was a progression in the lesson format so that courseware 3 had more functions and
better computer design.

Overall, this (courseware 2) is the best of the three programs.
This courseware (courseware 2) was much better, in relation to the first (courseware 1).
It made good use of graphics (face, etc.) and language to communicate with the student.

Good amount of info per screen. Good balance of info and text.

Chunk of Information Presented
Too much text per screen and poorly formatted.
The amount of text displayed at once was overwhelming.

Text screens long, could be broken up, layered or shortened.

Langggge s_tyle

A number of experts remarked that “profanity” or “demeaning” phrases were used in

feedback statements to the user when responding to questions presented by the

bal

Pa!

Vex
d0n

I fin
and

181

courseware programs. The issue, as seen by most of the experts, is that users might be
offended by the use of “profanity words” with the consequence that learning processes
would be inhibited. The following statements represent some of the experts comments on
the issue.

Profanity (e.g., ‘Damn’) and demeaning comments (e.g., ‘You can not be serious! ’)
will turn off some people and interfere with their learning from the lesson.

Do not say a question was too easy, then whoever misses it is insulted.

Some of the feedback is condescending, such as answering a question correctly but
having it dismissed as ‘an easy one.’ The use of the words ‘ridiculously easy’ is
guilty of this attitude, also. I think the idea here is to make the learning ‘informal’
and ‘relaxed,’ but I think a little more sophistication would help.

Interactivity and learner control

Not better than a book. Too hard to go backward.

Very ‘book like.’ Little in this unit could not be achieved in a book (save responses to
students’ answers to questions and buttons to elicit help. Many screens are filled with
text, and, again, the charts and pictures used could easily be part of a book.

Limited in branching, interactivity, and testing.

I could not go back to review previous questions.

Not enough branching opportunities; no reviews either. But most importantly, no way to
back-up to previous screen to ‘re-do’ material.

Page controls (previous and next) will be helpful.
Frustrating that I couldn’t exit from middle or return to start of the programs.

Very linear design. Students benefit from the ability to review previous screens and they
don’t have that opportunity here.

I find all of these programs to be very linear. Users have to use the “Continue” buttons
and are not allowed to skip sections.

182

Questions and Feedback

Part 1 of this courseware (courseware 3) provides some questions and feedback which
provide some form of review for the users.

More questions, feedback, and student self-testing will make the programs more effective.
Longer quiz with some form of feedback.

Lesson Review

Summary page at end to review coverage and things learned would be helpful in all the
programs.

Summary at the end of what should have been learned and possible suggestions to
students about more info should be included.

User Interface Desig

The content (of courseware 3) is rich and complete, but the application seems a little
fragmented in interface design. The use of table of contents would have allowed me to go
back and forth through, the programs.

The design issues (interface design) that seem relevant to all the modules relate (for me) to
navigation control. I would like to see some fixed controls repeating on every screen in
the same place: Forward, Backward, etc. I would also like more feedback or notification
of where I am in the courseware (Page No., or Unit No., for example). There is some
sense of disorientation when these elements are not present.

Since we live in a button happy world, the pages need to be designed in such a way that it
is always clear where to click. This is often unclear in these courses. I think many people
tend to click whenever they see a box. I also think there needs to be more consistency in
the placement of the navigational buttons. There should be a portion of the screen that
contains forward, backward, etc. At one place I was unable to return to a previous page
when I wanted to review a concept.

Sometimes too many elements on screen or too many visual cues, i.e., text box with
border, text box without border, text on screen with no background color (the easiest or
most aesthetic tone), buttons, arrows, etc.

183

Visual Appeal

Very nice visual appearance.

Polished appearance.

Black text on yellow screen was hard for me to read.

Some text fonts/styles are difficult to read, e.g., italics.

Media Use

Use of video and/or sound would make each a better program.

Environmental factors — equipment and systems

Monitor screen size affects program size - on my Pentium system with lS-inch monitor
program windows appear in the center of the monitor about half size. On my 14-inch
monitor, I get full screen!

On Windows systems running in 1024x768 mode, text is often misaligned.

Windows is a horrible platform for courseware. My experience is typical. It took a
whole hour before I could view any of the courseware.

Some expanded introduction needed - defining what course(s) this services, who is the
intended audience, what is the goal (study aid - test prep, etc.) of the sofiware. We jump
into content abruptly without some context for the material. Structure is too linear, not
enough opportunities to reverse, quit, start over.

Evaluating the prams and Quality improvement

The text needs proofreading/editing for spelling/typos/grammatical errors, that distract
and interrupt the flow of reading for understanding.

Difficult to evaluate without content knowledge or knowledge of (the target) students.

184

Comments Specific to Courseware 1

This courseware was like a book transferred to a computer screen. It would have been
much easier to work with in book form. The text lines were too long for the eyes to
follow comfortably. Also, the “Next Question” button appeared on every page, but there
was only one question to start with and a couple at the end. The last couple of questions
did not seem to be directly related to the main content of the courseware.

Screen titled ‘nerve cell action potential’ is excellent. However, screens are crammed
with black and white text.

There are a lot of full pages of text in this tutorial -— for that, it’s better to use a book.

No animation in spots where there were obvious ways to use it. No ‘worked’ problems,
except for a hint. No important links to definitions of ‘new’ terms. Does not seem to
take advantage of the unique features of a computer, compared to say, a book.

Too easy and oversimplified for medical students, but okay for undergraduates.

If this is new information, it seems to be too much and could be broken into smaller steps.
If it is review, the text screens could be shorter with not so much to read.

The program referred to ‘Next Question’ when it actually meant another section.

The user can get stuck in the first example. The buttons only let you go forward. The
‘go forward’ button should be in the same place on every screen so that the user doesn’t
have to move the mouse to a different place on the screen at every new page. Instead,
just press the mouse to go forward.

I think some questions put in during the long lesson would‘ help the user instead of the
user reading forever and then getting the transmembrane potential -70 mV question. Plus,
you can’t go back to read any part of the section which precedes this question.

I think the background section which is on paper should be in the programs.

The title deals with one subject — the action potential. The program deals with defming
the various parts of action potential and what issues are involved, also propagation in
repolarization and unrepolarization nerves and finally with a reflex arc, delay times at a
receptor and effector and propagation time in a reflex arc. My impression is that the last
part, although related to the action potential, could be a separate program with more
examples of the calculations of conduction time, etc. The title did not reflect the content

185

of the tutorial and it was not clear where the user was being led. Some answers to wrong
responses could have been more ‘friendly’, less disparaging or belittling. Ability to go
forward or backward is only used in one place in the program.

Content is good, however overall package was poorly organized and designed.

Comments Specific to Courseware 2
Best of the three.

Nice screen on ‘How this tutorial works.’ ‘Five element model’ is a good organizer. One
example ‘A Simple Application’ is mostly text.

The content of this lesson was stated clearly and followed in a logical order. There were
only a few questions to test the student. But, it did point to another source of review
questions at the end of the presentation. It seemed that this lesson could use more
interactive examples to illustrate control processes.

Control systems (i.e., courseware 2) in particular can exploit animation.
Biological example #2 may be oversimplified and misleading.

I’m told here that the tutorial reviews chapter five — but I’m not told the name of the
text, or number of the course. Again, as in courseware 1, I feel a need for introductory
contextualization. One improvement is the incorporation of an ‘EXIT’ button early on.
The ‘background’ screen is helpful and an improvement with regard to setting up the
context. Use of graphics improved over courseware 1.

l. I like the word definitions.

2. It is difficult to determine if boxes are buttons or just design.

3. Testing is better than the first program.

4. Needs objectives, table of contents, or menu bar so you can navigate better or know
where you are.

5. Nice graphics.

1. Typing is weird on “many physiological reflexes operate on the basis...” page. The
blue section.

2. Some text is difficult to read, font size too small.

3. Good use of examples.

186
Comments Specific to Courseware 3
The graphics, especially the diagrams, were not tied to the text clearly to illustrate a
point. The quiz was poor. For example, there were two questions where you were

presented with true or false response option and neither was correct.

Much too easy. Most of the material is about classic reflexes, not synaptic mechanisms
per se. One conceptual error (use of term conductance when meaning current).

Choices (navigation) sometimes overwhelming. Transitions are great except for dissolve,
too slow activating.

It was confusing to navigate at times — options are presented in menu form, but it is
assumed that they will be selected in order. If the last option is chosen, the user can’t go
back.
Part 1 (Synaptic Mechanisms 1):
Question 5 (page 30) is confusing. Both foils are wrong (true or false).
The blur on yellow frame telling how to control the program is messed up — text
is in lower case, button names are in upper case. Text book readings at the end of

Part I is useful.

Under definitions - suggestion section, the words are messed up - on top of each
other.

At ‘the plan’ page (with the frog) you are limiting the user’s ability to navigate as

there is no forward button. Those who already know this are still forced to go
through the plan.

Part II (Synaptic Mechanisms 11):
Just information presentation with weak interactivity.
It is difficult to read text in yellow box on the ‘So what?’ page.

This part was harder to grasp due to several links.

187

In the second section of courseware, screen dissolves and wipes are used in a more
or less random fashion. These tend to slow down the computer and really don’t
add value.

Specific recommendations for improvement

1.

2.

10.

11.

12.

13.

Check spelling and syntax.

Provide menu (especially in courseware 3) to allow for better organization so user
can choose parts of the program to review so that he/she doesn’t have to start
from the beginning to access a certain part.

Go button should be placed in standard location.

Comments for incorrect answers are inappropriate.

No swearing

Include Table of Contents.

Include well defined statement of objectives.

When students give an incorrect response, they should be directed back to
appropriate material for review.

Include opportunities to practice skills learned; apply things learned and used in
other programs

Present questions that not only test recall of facts.

Present feedback, rating % scores and suggestions to better scores.

Make the programs more interactive (either through questions, not just multiple
choice, or navigations).

Change buttons colors when button is activated.

188

Summary

In summary, comments made by the experts regarding special features of the computer
courseware programs were compiled and classified under four major sections:
1. Comments common to all the three courseware programs
2. Comments specific to courseware 1
3. Comments specific to courseware 2
4. Comments specific to courseware 3
In general, two main observations were made regarding the substance of the comments
made by the experts. First, the comments relate more to features that were lacking in the
programs rather than unique features or characteristics of the programs that they
perceived as helpful in effectively learning the lesson content. The second major

observation was that the comments provided concrete suggestions and specific

recommendations on ways to improve the quality of the courseware programs.

CHAPTERVI

SUNIMARY, DISCUSSION, CONCLUSIONS, AND

RECOMMENDATIONS

This study investigated factors and characteristics that determine the success or
efficacy of computer courseware programs designed to support students’ learning of
concepts in physiology. There were four purposes for conducting the study. The first
purpose was to determine the effects of using computer courseware programs designed to
teach physiology concepts on students’ learning achievement. The second purpose was
to assess the level of importance of characteristics, as perceived by the experts, that
promote effective learning via computer courseware. The third purpose was to assess the
impact of the quality of computer courseware characteristics in promoting effective
learning as perceived by both students and experts. The fourth purpose was to construct
an evaluation framework that courseware developers, instructional designers, researchers,
project managers, curriculum designers, educators, administrators, program evaluators,
and manufacturers of software programs can use to analyze and assess computer

courseware programs for the characteristics of instructional quality or efficacy.

189

190

Summary of the Study

Population

Three groups of subjects participated in this study. The first group consisted of
students in the Physiology 431 class of Fall semester 1995, which was further divided
into three sub groups. One subgroup consisted of 55 students who used the courseware
programs under study and responded to the Computer Courseware Evaluation Form that
accompanied the programs. The second subgroup consisted of students who used the
programs, but did not respond to the evaluation questionnaire. The third subgroup
consisted of students who neither used the programs nor responded to the evaluation
questionnaire.

The second major population group consisted of 32 experts with expertise in
physiology, classroom teaching, courseware development, educational psychology, or
educational research. This group responded to the Computer Courseware Evaluation
Checklist Rating Form and was called the Expert Rating Form Group.

The third major group consisted of 34 experts with similar professional experience as
these experts in the second group. The experts in the third group responded to the
Computer Courseware Evaluation Checklist Form and were called the Expert Courseware

Evaluation Group.

191

Data Collection

Four sets of data were collected in order to answer the research questions. The first
set of data consisted of exam grades for all students in the Physiology 431 class. The
second set of data was comprised of the students’ demographic characteristics and their
perceptions regarding the quality of the characteristics of the courseware programs used
in this study. The data was collected using the Computer Courseware Evaluation Form
(Appendix A). The third set of data was comprised of the demographic characteristics of
respondents in the Expert Rating Form Group, as well as their perceptions regarding the
level of importance of the characteristics of computer courseware in promoting effective
learning. The fourth set of data was collected fi'om respondents in the Expert Courseware
Evaluation Group. The data consisted of the respondents’ demographic characteristics,
as well as perceptions regarding the quality of the characteristics of the courseware
programs in promoting effective learning. All three evaluation questionnaires used in
collecting data for the study were constructed by the researcher based on Gagné’s
instructional theory, as well as findings from current research on educational psychology
and academic uses of new technologies. The evaluation instruments were reviewed by
members of the doctoral committee and pilot tested on sample subjects from both the

students’ and experts’ population groups.

192
Data Analysis

Data collected to answer the research questions examined in this study were analyzed
under two chapters. While the analyses of data collected from students were presented in
Chapter IV, the analyses of data collected from the two groups of experts were presented
in Chapter V.

In examining data collected from students, one way analysis of variance (ANOVA)
was used to compare adjusted raw score differences among students who used all the
courseware programs, those who used one or two programs, and those who did not use
any of the programs. Regression analysis was then used to determine the extent to which
learning achievement depended on students’ demographic factors, rather than the use of
the computer courseware programs. Means, standard deviations, and ranks were used to
report the students’ perceptions regarding the quality of the characteristics of sections of
the courseware programs under study. Also, one way analysis of variance (ANOVA)
was used to determine whether students’ perceptions regarding the quality of computer
courseware characteristics varied by their demographic characteristics such as: gender,
academic discipline, class level, and computer ownership. Regression procedure was then
used to determine demographic factors that were significant predictors of the students’
perceptions.

As had been done for the students’ group, two levels of analyses were conducted in
examining data collected on the perceptions of respondents in both the Expert Rating
Form Group and the Expert Courseware Evaluation Group. The first level of analysis

used descriptive statistics including means, standard deviations, and ranks to report the

193

experts’ perceptions regarding the level of importance of the characteristics of courseware
programs used in the study and their perceptions regarding the extent to which the quality
of the characteristics helped in promoting effective learning. The second level analysis
used inferential statistics including one way analysis of variance (AN OVA) to determine
whether the experts’ perceptions varied by their demographic characteristics such as
gender, age, professional discipline, previous experience in using computer productivity
tools, and computer ownership. Regression procedure was then used to determine the
demographic factors that were significant predictors of the experts’ perceptions. A
separate, intensive analysis, using the MDS procedure, was conducted to determine
whether specific patterns of perceptions regarding the level of importance of courseware
characteristics in promoting effective learning existed among the respondents in the Expert

Rating Form Group.

Findings and Discussions

Research Question A

Research Question A was concerned about the effects of computer courseware
programs on students’ learning achievement. Two specific subsidiary questions were
examined to delimit the scope of the question. The findings regarding these questions are

presented and discussed below.

194

Research Question A]

Does courseware usage predict students’ level of learning achievements?

The results of AN OVA and Tukey’s post-hoe analyses revealed that students who
used all three programs significantly performed better than those who used just one or
two programs, who, in turn, performed significantly better than those who did not use
any of the programs. It appeared that the more courseware programs used, the better the

students’ performance, albeit an additive effect.

Discussion

Any additive effect resulting fi'om interacting with courseware programs several times
is not necessarily undesirable. While efficiency is an important concern in any learning
situation, the ability to engage learners in learning activities, including interaction with
courseware programs, in an enthusiastic and a repetitive manner, holds promise for some
beneficial and meaningful learning outcomes. One only needs to recall young adults’
experience with computer and video games. Young adults not only love to play
computer-video games, they often play the same set of games for several hours at a time
without becoming bored. Moreover, young adults often demonstrate an amazing depth of
retention and understanding of the content of games they interact with, a learning outcome
unmatched by most of their classroom learning experiences as demonstrated by their
abilities to recount detailed plots and themes of numerous games. This phenomenon

poses a fundamental question: what characteristics or features of computer and video

195

games produce such “edutainment” effects? Computer courseware programs can learn

from the “edutainment” properties of computer-video games.

Questions for Further Stuafv

1. Why are young adults addicted to playing computer and video games that require
an amazing depth of knowledge of the content unmatched by most classroom
learning experiences, as demonstrated by their abilities to recount detailed plots
and themes of numerous games?

2. What, if any, elements or characteristics of such computer or video games,
particularly Nintendo, can influence the way we develop computer courseware
programs to better engage students’ attention and sustain interest in learning
processes?

3. Why have courseware authors and developers failed to incorporate “edutainment”

properties of computer video games in academic courseware programs?

Research Question A2

To what extent is the students’ level of learning achievement dependent on factors
other than the use of the computer courseware, for instance, the respondents’
demographic characteristics (gender, age, GPA, academic discipline, class level,
previous experience with computers, and computer ownership)?

Out of the three courseware programs used in the study, the amount of usage of

courseware 2 (i.e., Control Systems) alone was a significant predictor of learning

achievement as measured by adjusted raw score. When the students’ demographic

196

characteristics were used as covariates in the analyses conducted, however, gender and
GPA emerged as the only significant predictors of learning achievement. Consequently,

courseware usage had little or no effect on learning achievement.

Discussion

The finding that, in general, computer courseware usage had little or no significant
effect on students’ learning achievement, confirms the findings of a number of studies on
media effects (Clark, 1983; Creswell, 1986; McNeil and Nelson, 1991; Reeves, 1991;
Thompson et al., 1992; Schlosser, 1994). It appears, though, that the use of computer
courseware had other beneficial effects that may translate to increased learning
achievement. For instance, almost all (96.4%) of the students who used the programs and
responded to the evaluation questionnaire indicated that the courseware programs helped
them to effectively learn the lesson materials. The only two exceptions were missing
data. Also noteworthy, is the finding that nearly all (96.2%) the students who used the
programs and responded to the evaluation questionnaire indicated that they would like to
use similar courseware programs in other college courses. Again, the only exceptions
were missing data. It may be, therefore, that a main effect of using the programs was an
increase in the level of students’ motivation to learn. This conclusion is consistent with
the result found by Askar, Yavuz and Kbksal (1992) in their study of fifth-graders’
perceptions of computer assisted instruction environment and their attitudes towards
computer assisted learning. However, while only 63 percent of the students in that

study stated interest in using computers in all other courses (Askar, é: al., 1992, p. 138),

197
virtually all the college students in the present study expressed such an attitude. In

addition, the student respondents in this study expressed a clear preference for computer
courseware usage in the sciences.

Since attitude, motivation and learning processes are intricately linked (Johnston,
1987; Askar, et al., 1992) , it may be that stating or demonstrating the benefits of a
particular computer courseware for the target audience will positively impact on the
learner’s attitude toward, and level of motivation in, using the program. Under such
conditions, the use of the program may not only result in more courseware usage, but also
in better learning achievement. For instance, positive statements regarding the usefulness
of a particular courseware, made by sample subjects drawn from the target population
during formative evaluation of the courseware, may be embedded as part of the
introduction or overview of the final courseware product. Indeed, the inclusion of review
statements as a standard “sneak preview” feature of computer courseware may be
beneficial in motivating target learners to use courseware programs. In this regard, the box

office experience of the movie industry may be instructive.

Questions for Further Study

1. What are the effects of changes in the learners’ levels of motivation as a result of
interaction with the programs, on their attitudes toward computer courseware?

2. What are the effects of changes in the learners’ levels of motivation, as a result of

interaction with the programs, on learning achievement?

198

3. Does the inclusion of testimonial or review statements about the usefulness of
courseware programs influence how much learners use, or benefit from, the
programs?

4. Under what conditions would the amount of computer courseware usage influence

learning achievement?

Research Question B

Research Question B assessed the student respondents’ perceptions regarding the
quality of the characteristics of computer courseware programs used in this study. The
following section presents and discusses the findings relating to three specific subsidiary

questions that examined the perceptions.

Research Question B1

What are the student respondents’ perceptions regarding the quality of the
characteristics of the computer courseware programs used in this study in
promoting effective learning?

In general, students perceived courseware 3 (Synaptic Mechanisms I and II) as the
best, in terms of the quality of courseware characteristics in promoting effective learning,
followed by courseware 1 (Action Potential), and then courseware 2 (Control Systems).
In terms of specific courseware components across the three programs, the students rated

the characteristics of instructional procedures better in quality than courseware structure

characteristics, which, in turn, were rated better than media characteristics. The results

199

applied across all three courseware programs. Interestingly, no significant relationship
was found between the students’ perceptions regarding the quality of courseware
characteristics and their learning achievement. Also, there was no significant relationship
between students’ perception regarding the quality of computer courseware and the

amount of usage of the programs.

Discussion

When the findings of this research question is compared with the findings observed
under Research Question A2, concerning the extent to which learning achievement
depended on courseware usage, it is interesting to observe that it was not courseware 3
(Synaptic Mechanisms I and II) that the students rated as best in quality that enhanced
their learning achievements, but courseware 2 which they rated last. However, as
indicated in the discussion under'Research Question D1 below, the observation regarding
the effect of courseware 2 (Control Systems) on Ieaming achievement makes sense when
it is realized that it was this program that the experts rated as best in quality. Further
discussion about the disparity in the results are presented under Research Question D1
below. Suffice it to say here, that a more tightly controlled experiment comparing
students and experts in how and what they evaluate about computer courseware may
help clarify the apparent disparities.

It is noteworthy that the students perceived the characteristics of instructional
procedures and lesson content to be better than courseware structure characteristics and

the characteristics of media. It is not clear, however, whether the students thought that

200

these categories of courseware characteristics had any relative level of importance to the
success of the programs in promoting learning achievement. The knowledge of what
courseware end-users consider to be important may be an important factor to the success

of the programs.

Questions for Further Study

1. Do learners’ perceptions regarding courseware quality vary by levels of
motivation to interact with computer courseware?

2. Do learners’ perceptions regarding computer courseware quality and their levels of
motivation change over the course of interaction with the courseware? If they do
change, what factors are involved?

3. What are the perceptions of students regarding the level of importance of the
characteristics of computer courseware compared to the success of the programs

in promoting effective learning?

Research Question B2

Do the student respondents’ perceptions regarding the quality of the computer
courseware characteristics vary by their demographic characteristics such as
gender, class level, academic discipline, previous experience using computer
productivity tools, previous experience using computer courseware, previous
experience with computer programming, or computer ownership?

Among the demographic variables examined, only students’ class level and previous

experience using computer courseware had a significant relationship to their perceptions

201

regarding the quality of the computer courseware in promoting effective learning. In all
three courseware programs, the junior students were more positive in their perceptions
regarding the quality of instructional procedures characteristics than the senior students.
However, both groups were similar in their perceptions regarding the quality of other
dimensions of the programs. Another finding relating to the effect of demographic
characteristics on students’ perceptions was that students who had no previous
experience using computer courseware had more positive perceptions of the quality of
courseware characteristics. Gender, academic discipline, computer ownership, and,
surprisingly, previous experience with computer programming, had no significant

relationship to the students’ perceptions.

Discussion

It is not clear from the results of the study why junior students were more positive in
their perceptions regarding the quality of the courseware used in the study compared to
senior students. One conjecture behind the differences in perceptions may be the element
of novelty. It may be that, in general, junior students, compared to senior students, were
less exposed to computers, in general, and computer courseware, in particular. If this
conjecture is true, then it is plausible to surmise that their relatively less exposure to
computer courseware made the use of the computer courseware in this study a novel
experience for the junior students more so than for the senior students. Thus the gap in
their familiarity with computers may have led junior students to be more accepting of the

quality of the courseware programs.

202

Similar explanations may be advanced for students with or without experience in
computer programming. In this instance, students with no computer programming
experience did not have the essential computer skills to adequately evaluate the
courseware programs, and so, were less critical than students with programming
experience.

One clear implication of the differences in perceptions regarding the quality of the
courseware between different groups who participated in the study is the need to
determine specific constructs that separate one group from another in their perceptions.
Answers to this need will be helpful in developing courseware with wider appeal,
acceptance, utility, and thus greater success in promoting effective learning. It may be
that the differences in perceptions observed in this study, reveal a fundamental need for

literacy in the use of computer courseware.

Questions for Further Stuajz

1. What constructs differentiate different class levels of students in their perceptions
of the quality of computer courseware?

2. Are differences in perceptions regarding the quality of computer courseware due

to differences in attitudes toward computers?

203

Research Question B3

What other features of the courseware programs do the student respondents

perceive as unique and helpfiil in promoting effective learning?

Ten main features of the courseware programs used in the study were identified by the

students as helpful in enabling them to effectively learn the lesson materials. The

1.

students remarked that the programs:

Reviewed and synthesized the Physiology 431 course materials in a way that
adequately prepared the students who used the programs for the class exams.
Presented lesson concepts in a clear and concise manner. As a student noted, the
programs “clarified a lot of points that were confusing in the book (i.e., the
textbook)”. Another student remarked: “The material was clear, focused and
relevant to the class. It helped out in that it simplified things. It made it much
easier to understand a concept.”

Presented the lesson concepts in a systematic fashion.

Used concrete examples, especially in feedback statements, that helped in
sustaining students’ interest in learning the lessons presented and in facilitating
better understanding of physiology concepts. As one student remarked, “The
courseware kept my interest longer than reading a textbook. It helped me to
visualize concepts more clearly.”

Used humor in feedback statements that sustained students’ learning interest.

Provided feedback that facilitated meaningful learning.

204
7. Applied theory to real life situations.

8. Helped in developing problem solving and thinking skills.
9. Individualized lesson instructions.
10. Presented review questions that enabled the students to apply concepts and to

practice skills learned.

Discussion

It is interesting that virtually all the features listed as helpful in learning the content of
the programs used in this study were related to characteristics of the subject matter and
instructional procedures rather than glitzy effects. This is an important finding, given
that, as the literature in educational psychology indicates, the nature and structure of
subject matter content as well as the characteristics of instructional procedures are
essential components of factors that promote effective learning (Hannum, 1988; West, et
al., 1991; and Jonassen and Wang, 1993).

As the students indicated, however, the use of humor as a vehicle for conveying lesson
content was as a factor that made interaction with the programs fun and interesting for
them. Virtually all the student respondents indicated that they liked the way humor was
used in the courseware programs, particularly the way it was used to provide feedback
for answers that go with questions at the end of each lesson unit. Although humor falls
' under the rubric of motivation, and a large body of literature exists on the use of
motivation in instruction, little or no research has been conducted on the use and effects

of humor as a pedagogical strategy in computer courseware. Clearly, the emphasis that

205

the majority of the students placed on the motivational influence of the humor used in the
courseware programs, underscores the need for fiirther study in this area.

Most of the students perceived the courseware programs under study to be a useful
supplement to classroom instruction. The programs were considered useful in promoting
effective learning particularly when perceived to be helpful in adequately preparing the
student for exams.

A closely related finding is the perception that the courseware programs embodied the
essence of the Physiology 431 course. While the perception may be helpful in focusing
students’ attention to important aspects of the course content, it may, however, be
potentially counterproductive. For instance, while some students may have used the
programs only as a supplement to and a review of the course materials, other students
may have used the programs exclusively, totally ignoring other course materials. The risk
is clearly implied in a student’s remark cited earlier. According to the student:

I heard from students who had taken the class earlier that the courseware was all
that was needed. It seems as though it became the center of the material to be
tested rather than merely a learning supplement.
Unless courseware programs are really designed, in the words of one of the students, as
“complete tutorial” instructions and as a full substitute for classroom instruction,
students who rely solely on the programs may be doomed to failure. It is necessary,
therefore, for instructors using courseware programs for an academic instruction to
explain clearly the relative importance of the programs in relation to other course

materials. It must be pointed out that the programs under study were clearly explained as

206

only supplements to classroom instruction. Nonetheless, some students perceived the
programs as the component of the course materials that was most important to the
instructor.

The students’ comments reveal that the level of use of courseware programs depends
largely on the degree to which they perceived the programs to be relevant to their learning
needs, particularly the need to perform well in exams. Since, as the society dictates,
performance in exams is the prime indicator of students’ intellectual skills and learning
achievements, courseware programs must be designed to enable the student to process
information in a way that fosters meaningful learning and, consequently, successful
performance in exams. Of course, the potential of a particular courseware program to
facilitate learning outcomes that ensure success in exams must be made clear.

The students were strongly in favor of courseware programs that facilitate the use of
students’ learning styles. It must be point out, however, that while learning styles-
centered programs may be desirable, they are usually very difficult and expensive to
produce.

A salient feature that the students liked about the courseware programs under study
was the nature of feedback presented for review questions at the end of each lesson unit.
The students particularly liked how the feedback presentations explained why answer
selections were correct or wrong. They claimed that the use of feedback to explain
answer choices helped to clarify misconceptions they might have had about the lesson
concepts and principles presented and thereby helped them to better understand the

Physiology 431 course contents. The claim agrees with previous findings in research

207

literature that the use of feedback as a learning strategy is one way in which courseware
programs can foster effective learning (Lillie et al., 1989).

Clearly, the majority of the students would like to use computer courseware for
academic instruction. However, the students differed on the subject areas and topics for
which computer courseware will be appropriate. It may be that the different viewpoints
actually indicate underlying differences in learning needs among the students. Further
study should examine the variety of learning needs among college students in a particular
subject discipline and how computer courseware can be designed to provide learning
experiences that fulfill the needs.

The chief concern students expressed about availability of the courseware programs
relates to the need to make similar programs available in other subject areas where they do
not currently exist. University administration must listen to this perceived needs by
providing necessary support and incentives to encourage faculty to develop and integrate
computer courseware into their academic‘courses.

Closely related to availability is the issue of accessibility. The students’ comments
revealed that those who had limited access to adequate computer systems to use the
courseware programs felt disadvantaged and, as such, were opposed to academic uses of
computer courseware. For the students who complained about limited access to the
programs, their real problem was failure to successfully follow procedures required to use
the courseware programs as set up in the microcomputer labs on campus. One solution
to the problem is to provide a short introductory tutorial session in which a teaching

assistant would show the students how to access and use the programs in the

208

microcomputer labs. Alternatively, the class instructor could use an overhead projection
system to present a short demonstration of how to log into the microcomputer lab
systems and use the programs. The demonstration could be provided during a class

session just before asking the students to use the programs in the labs.

Questions for Further Study
1. What are the effects of the use of humor as a vehicle of conveying instructional

lessons in computer courseware?

2. What type of humor can be utilized in computer courseware to foster effective
learning ?
3. What learning needs exist among college students in particular subject disciplines

where computer courseware can provide meaningful learning experiences?
4. How can the factors identified by the learners as essential to their ability to
effectively learn the computer courseware lessons be combined for optimum

learning effectiveness?

Research Question C

Research Question C assessed the perceptions of the respondents in the Expert Rating
Form Group regarding the level of importance of the characteristics of computer
courseware in promoting effective learning. The following section presents and discusses
the findings relating to three specific subsidiary questions that examined these

perceptions.

209

Research Question C1:
What are the experts’ perceptions regarding the level of importance of computer
courseware characteristics in promoting effective learning?

Presenting complete, accurate, or current lesson content, was the characteristic rated
highest in terms of level of importance in promoting effective learning via computer
courseware. Rated last in the order of level of importance, though still considered of
medium importance, was the characteristic of testing the learner on restating lesson
information in his or her own words. All the characteristics were rated as either high or
medium in their level of importance, with each level having an equal number of .
characteristics. No characteristic was considered to be of low importance.

In terms of courseware sections, the characteristics of both courseware structure and
new lesson content were rated equal and at the top of the level of importance. Third in
the level of importance were the characteristics of prerequisite skills, followed by the
characteristics of screen layout and design. Media use characteristics ranked last, but

nonetheless was considered to be of medium importance in promoting effective learning.

Discussion
Even though the students used a different evaluation instrument , the pattern of ratings

of the level of importance of courseware characteristics by the experts was essentially

similar to the pattern of characteristics that the student group identified as helpful to

them in effectively learning the courseware programs in the study (see the discussion

210

under Research Question 38 above). For both groups, the characteristics related to
knowledge (verbal information, intellectual skills, and cognitive strategy skills) to be
mediated, and the appropriate pedagogical strategies to convey the knowledge were the
most important factors in determining the success of computer courseware. It remains to
be seen whether specific courseware characteristics are actually related to learning
processes and/or learning achievements.

It is noteworthy that of the 40 characteristics rated to be of high level of importance
to the success of courseware in promoting effective learning, the ratings of the items relate

to Gagné’s events of instruction as follow:

0 9 (22.5%) items clearly relate to presenting new information

0 8 (20%) items relate to providing learning guidance

0 6 (15%) items relate to eliciting performance

- 4 (10%) items relate to providing feedback

0 3 (7.5%) items relate to gaining learner attention

0 3 (7.5%) items relate to informing the learner of the instructional objectives
- 3 (7.5%) items relate to stimulating recall of prerequisite skills

0 2 (5%) items relate to assessing performance

0 1 (2.5%) item relates to enhancing retention and transfer.

The proportion of items related to each event of instruction may be indicative of the
magnitude of features to look for, when considering the quality of courseware programs or
assessing the potential of the programs to engender effective learning. However, the
strength of this conclusion should be held with caution since, to start with, each category
of Gagné’s Events of Instruction did not have an equal number of characteristics on the
evaluation instrument. Besides, there were other characteristics on the instrument that

the experts rated as high in level of importance, which although were derived fi'om the

211

literature on current research in educational psychology and academic computing, were
not directly related to Gagné’s events of instruction. These characteristics are primarily
related to unique cognitive capabilities made possible by unique properties of the new
information technologies. One challenge posed by these findings is the prospect of

quantifying courseware characteristics for the purpose of evaluating courseware quality.

Questions for F wither Stud)!

1. In what ways are the specific characteristics that the experts identified as
important to the success of computer courseware related to learning processes
and/or learning achievements?

2. What criteria should be used to formulate standard measures for quantifying the

instructional effectiveness potential of computer courseware?

Research Question C2:

Do the experts’ perceptions regarding the level of importance of computer
courseware characteristics in promoting effective learning vary by their
demographic characteristics such as gender, age, professional discipline, teaching
experience, courseware development experience, experience using computer
productivity tools, and computer ownership?
Courseware structure was the only courseware component in which there was a
significant difference in the perception of experts regarding the level of importance of

computer courseware characteristics. Gender and age were the only two demographic

factors that accounted for the significant difference.

212

Age was the only significant predictor of perception regarding the level of importance
of courseware structure characteristics in promoting effective learning. Compared to
experts over 46 years old, those between 36 and 45 years of age were predicted to have
higher perceptions regarding the level of importance of the characteristics of the following
sections of computer courseware: courseware structure, prerequisite skills, and new
lesson content. In the case of screen layout and design, experts over 46 years old were
predicted to have significantly lower perceptions regarding the level of importance of the
characteristics associated with that section. In sum, it appears that the older an expert is,
the less likely he or she would consider the computer courseware characteristics under
study as important in promoting effective learning. Gender, teaching experience, previous
experience using computer courseware and multimedia, and even professional expertise in
courseware development all appeared not to be significant predictors of the experts’
perceptions regarding the level of importance of the computer courseware characteristics

under study in promoting effective learning.

Discussion

The observed differences in perceptions due to age may be related to a generational gap
in background experience related to computer-related technologies. For instance, experts
who were below 36 years of age compared to those over 36 years of age, may have been
exposed more to computers and computer-related technologies, such as television, video
games, and computer games, including Nintendo. Perhaps the relatively younger experts

have had greater exposure to new technologies, and as such, greater identification with,

213

and/or appreciation for, the computer courseware characteristics evaluated in this study.
However, the observed differences are meaningful only as they shed light on how
courseware characteristics relate to learning processes and achievements. It is important,
therefore, to establish a clear understanding of the relationship between the ratings of the
level of importance of courseware characteristics and learning processes, as well as
learning achievements. The first step in establishing this relationship is to determine the
reliability of the evaluation instrument. The overall reliability index of the Computer
Courseware Evaluation Checklist Rating Form used in this study was 0.9592.

It is plausible, therefore, to conclude that experts who were below 36 years of age
provide a more adequate view of the level of importance of the courseware characteristics
under study. This conclusion should be considered tentative, however. The finding about
differences in perception due to age is tempered by the limited number of experts who
participated in the study. Further study on the issue involving greater number and

diversity sample subjects is warranted.

Research Question C3

What other features do experts perceive as important to the success of computer
courseware in promoting effective learning?
In addition to the characteristics specified and examined in this study, the experts
indicated the following features as necessary for computer courseware to be successful in

promoting effective learning.

214

1. Features that serve the instructor’s classroom teaching goals

2. Features that match teaching goals with students’ learning needs

3. Modularization of program lessons to meet specific instructional and learning
goals.

4. Providing feedback that explains why answer choices are correct or incorrect as a

way of facilitating meaningful learning.

Questions for Further Study

1. What computer courseware features can empower instructors to effectively meet
their teaching goals in support of their students’ learning needs?

2. How can instructors effectively utilize computer courseware to support their

teaching goals and student’s learning needs?

Research Question D

Research Question D assessed the perceptions of the respondents in the Expert
Courseware Evaluation Group regarding the quality of the characteristics of computer
courseware programs used in this study. The following section presents and discusses
the findings relating to three specific subsidiary questions that examined these

perceptions.

215

Research Question Dl

What are the experts’ perceptions regarding the quality of the characteristics of
the computer courseware programs used in this study in promoting effective
learning?

Overall, courseware 2 (Control Systems) was rated best, followed by courseware 3
(Synaptic Mechanisms I and II), with courseware 1 (Action Potential) having the lowest
rating. In general, the experts had a more positive perception of the quality of courseware
2 characteristics than courseware 3 characteristics which, in turn, were perceived better
than the characteristics of courseware 1.

In terms of quality, almost all the courseware sections were rated as fair. The only

exception was the prerequisite section of courseware 1 which was rated as poor in

quality.

Discussion

In comparing the qualities of characteristics of the three courseware programs used in
the study, whereas the students perceived courseware 3 as the best, the experts rated
courseware 2 as the best. The difference in perception regarding courseware quality
observed between the student evaluation group and the expert evaluation group was quite
an interesting finding in light of the students’ learning achievements. It was courseware 2,

which the experts rated as the best in quality, and not courseware 3 which the students

216

rated as the best, that had significant effect on students’ learning achievement. Two
plausible reasons may account for the disparity. One possible reason may be the
separate instrument that each of the subject groups used for the courseware evaluation.
Perhaps the evaluation instrument used by the experts, compared to the one used by the
students, yielded a more dependable measure of the courseware quality. The second
plausible reason may be that since the experts were generally more experienced than the
students, then the thoughts, views, and perceptions of the experts were a more valid
measure of the courseware quality. To the extent that this second assumption was true,
one may conclude that the experts were likely to be more dependable in providing a more

valid, and perhaps more reliable, measure of computer courseware quality.

Questions for Further Study

1. When students and experts use the same instrument to evaluate computer
courseware, do they differ in their perceptions regarding the quality of the
program characteristics?

2. When students and experts use the same instrument to evaluate computer
courseware, would the experts group provide a more reliable measure of

courseware quality than would the student group?

217

Research Question D2
Do the experts’ perceptions regarding the quality of the computer courseware
characteristics vary by their demographic characteristics such as: gender, age,
professional discipline, teaching experience, courseware development experience,
experience using computer productivity tools, and computer ownership?
Significant differences in perceptions regarding courseware characteristics existed
among the experts by expertise in courseware development and content. Experts who had
developed no programs had a more positive perception about the quality of courseware 1
new lesson content characteristics than those who had developed some programs. The
experts without content expertise had a more positive perception of the quality of

courseware 2 structure characteristics, as well as courseware 3 prerequisite skills

characteristics, than experts with content expertise.

Discussion

The observed differences among the experts in their ratings of courseware quality
revealed clear bias or preference for characteristics that were related to their own area of
expertise. A closer look at the pattern of responses to the evaluation instrument revealed
that, in general, the experts tended to rate the courseware characteristics that are related to
their own expertise, while skipping, by indicating on the evaluation instrument lack of
expertise to evaluate, the characteristics that did not relate to their expertise.

A clear implication of this pattern of response is the need to have experts with

expertise in diverse areas or disciplines, or a team of experts with diverse expertise, rather

218

than a single expert or a monolithic area of expertise, to be involved in courseware
evaluation. In this scenario, an expert will only review or evaluate aspects of courseware
programs that relate to their own expertise. Together, the team of experts, except in the
rare instance where an expert has all the necessary expertise to conduct all aspects of
courseware evaluation, will provide a composite of cross-sectional evaluation of
courseware programs. It is hypothesized that such an approach will yield a more reliable

measure of courseware quality.

Research Question D3

What other features of the courseware programs do experts perceive as helpful in
promoting effective learning?

In general, the experts did not indicate that the programs offer unique features that are
helpful in promoting effective learning. However, they did provide useful suggestions on
how to improve the quality of the programs.

One important suggestion the experts made was to restructure Part B of the Computer
Courseware Evaluation Checklist Form, dealing with lesson content characteristics, in
such a way that the characteristics are grouped together in terms of Gagné’s learning
outcomes categories rather than the events of instructions. The suggestion was given on
the grounds that the classification of lesson content characteristics by events of
instruction as presented in the evaluation instrument that the experts used may

inadvertently cause the impression that courseware programs are expected to exhibit all

219

the characteristics listed on the instrument. However, courseware programs need not
exhibit all the characteristics, though at times they may do so where the purpose and
objectives of the programs involve all the learning outcomes. Stated differently, a
courseware program may contain all, or only some, of the characteristics of the learning
outcomes. Thus if the purpose of a program is to teach intellectual and cognitive skills
learning outcomes, it would be wrong to evaluate the program for characteristics of
attitude and/or motor skills outcomes. However, if the objectives of a program include all
the learning outcomes, it would be wrong not to evaluate the content in terms of the
characteristics of all the learning outcomes. In sum, restructuring the lesson content
characteristics by learning outcome categories and including statements that clearly
explain the nature, firnctions, and how to use the instrument, will help improve the utility
of the evaluation instrument.

The experts’ suggestions, along with the important findings presented above form the

core of the framework for evaluating computer courseware constructed below.

Limitations of the Study

The scope of this study was limited in three important ways. First, as a case study,
the research project was conducted only on a set of programs that were developed and
used by the faculty-developer in one college-level semester course offered only at

Michigan State University. Moreover, the set of programs were used only as

220

supplements to classroom instruction. Secondly, the student subjects who participated
in the study were self-selected. Thus the real effects of using the courseware programs on
students’ leaming processes and achievements may be difficult to isolate from the effects
of the idiosyncratic realities of the particular class and courseware programs under study.
Any generalizations about the effects of computer courseware on students’ learning
processes and achievements drawn from this study must necessarily be limited to this
student population.

Thirdly, the number of experts readily available to participate in the study were very
limited, thus making random sampling of subjects impractical. Besides, all but one of the
experts were drawn from Michigan State University. Thus the views expressed by the
experts who participated in the study may not be representative of the views of other
experts with similar skills. Any generalizations drawn from the study on the perceptions
of experts regarding the characteristics of computer courseware in promoting effective

learning must necessarily be limited to this population of experts.

221

Conclusions and Implications

Although the computer courseware programs helped in enhancing the students’
learning achievements, GPA and gender were the significant predictors of learning
achievement.

The courseware programs enhanced learning achievement more for higher achieving
students (i.e., those with higher GPA scores) than for lower achieving students
(i.e., those with lower GPA scores).

In general, the amount of courseware usage had little or no effect on learning
achievement. The amount of usage of Courseware 2 (Control Systems) was,
however, a significant exception to the conclusion, at least with regards to this
study. It is therefore imperative to determine the conditions under which the
amount of courseware usage will predict learning achievement. Such information
will provide a very useful addition to the knowledge base on academic uses of
computer courseware.

As the students indicated, the computer courseware programs made the lesson
content clearly understandable and fun to learn.

One important reason the students gave as to why the courseware programs
helped them to learn the lesson content better, was that the programs were
developed by the same instructor who taught the course. It appeared that content

expertise combined with knowledge of both students’ characteristics and specific

222

class situations have decisive influence on the quality and acceptance of computer
courseware.

A closely related matter is the claim by most of the student respondents that
they perceived the courseware programs used in the study as representing the
topics that were important to the class instructor. The “same instructor”
perception could provide the motivation that the students need to pay careful
attention to the lesson information that the courseware programs present, and
thereby promote effective learning. Instructors should, therefore, be encouraged
to be savvy in using new technologies and integrating computer courseware into
their academic curricultun.

Students with basic computer literacy skills favor and support full integration of
computer courseware into academic curriculum. On the contrary, students with
no computer literacy skills opposed academic uses of computer courseware.

Two main implications can be deduced from the different viewpoints. One
implication relates to pedagogical practices in the age of information technology.
The pervasiveness and versatility of new information technologies in almost every
aspect of life today as well as the widespread societal demands for academic uses
of computer-related technologies suggest that the integration of computers in
academic curriculum will continue to grow by leaps and bounds. If computers
become an integral part of academic instruction, then all students must be required .
to have basic computer literacy skills. The prerequisite will help to eliminate one

source of frustration with the use of computer courseware and therefore increase

223

the likelihood that courseware programs will succeed in promoting effective
learning. The second implication of the students’ attitude concerning the
integration of computer courseware in academic curriculum for administrative
policy is that schools and colleges must, at least in the interim, provide students
who lack the prerequisite skills the opportunity to acquire the skill. Failure to
provide such an opportunity is a recipe for failure of any attempt to promote
effective learning via computer courseware.

The majority of the students liked and wanted computer courseware programs as
supplements to classroom instructions, but not a replacement for lecture and
textbook materials.

Instructors must educate their students that although courseware programs used
as a supplement to classroom instruction can help in preparing for exams, the
programs should be viewed and used only as a supplement, and not as the sole
source of information for exams.

Most of the students expressed the need for similar programs in other academic
disciplines, particularly in the sciences (e.g., physics, biology, and biochemistry)

9,

where there is the need to “relate various concepts and apply them. However, a
number of the students indicated that computer courseware programs are only
appropriate in the sciences and not in non-science subject areas. It must be noted
that most of the students who participated in the study were either pre-medical or

natural science students. It is only fair to assume that their view about subject

areas that are appropriate for computer courseware development was biased in

IO.

ll.

12.

224

favor of science disciplines. It is critically important, therefore, to determine
subject areas that students think are appropriate for computer courseware. Such
information may help provide the knowledge base to deal with issues relating to
adoption and acceptance of technology innovations in education. Further study is
needed to determine if students in social sciences and humanities have a similar or
different view about academic uses of computer courseware.

Wherever possible courseware programs should be developed to be multi-platform
compatible as a way to make the programs extensible and thus solve potential
problems of limited availability and accessibility. One emerging technology that
offer great potential for developing multi-platform compatible courseware is the
Internet, particularly the World Wide Web (WWW). As some student
respondents suggested, the Web technology should be promoted and utilized more
for the purpose of providing academic instruction. Although more technology
does not necessarily mean better performance, nonetheless, the more options that
can be made available to leamers, the better the chances for the learner to achieve
optimum effective learning.

As some students indicated, developing computer courseware is a way for faculty
who are disorganized in their lectures and have never developed educational
software, to “bring some organization to a seemingly structureless lecture.”
Learners have different learning styles and learn through different learning
modalities. Some students learn best by reading books or other printed materials,

while some learn more by using any combination of different media of instruction,

13.

14.

225

including books, television, and computer-based instructional materials.

Therefore, the traditional concern of educators regarding whether or not certain

media are more effective in promoting learning achievement than traditional

classroom instruction is a moot question. The proper questions to ask are:

- What specific courseware characteristics will promote effective learning
processes and achievements?

- How should the characteristics of intellectual and cognitive skills, learning
styles, instructional strategies, and media be combined and represented in
computer courseware to facilitate meaningful learning and enhance learning
achievement?

0 How can the variety of computer-related technologies available today be
effectively utilized to empower learners to “learn how to learn”, solve
problems, conduct research, and generate new knowledge?

In general, most students thought that the feedback statements used in the

courseware programs were humorous. On the contrary, most of the experts

expressed that the use of what they perceived as, ‘profanity words’ in the
feedback statements can be insulting or offensive to the user. Clearly, further
study is needed to understand the underlying cause of this difference and to
determine the effects of the use of humor as a vehicle of providing instruction via
computer courseware.

A minority of the student respondents expressed concern that the contents of the

computer courseware programs were simply a repetition of the same materials

15.

I6.

17.

226

presented both in the classrooms and the assigned readings from the textbooks and
study guides. As one student noted “the materiel on the courseware should not be
too repetitive of information found in course readings and lectures. Sometimes I
found the courseware a bit too repetitive of what we had already learned.” This
concern is critically important, given that the experts felt that the most important
characteristic of computer courseware is presenting “complete, accurate, or
current lesson materials.” The issue then: can courseware programs designed to
supplement classroom instruction be “complete, accurate, and current” without
being “wordy” as both the students and the experts critiqued with respect to the
programs under study? What levels of information should be included and how
should the information be organized in supplementary courseware programs to
achieve optimum information requirements, and at same time, conciseness and
clarity. These and similar concerns can be easily identified and corrected through
formative evaluation. Another way the problem can be corrected is to provide
ample opportunities through hyperlinks to allow the learner to control how they
interact with and explore courseware programs.

One major criticism of the courseware programs under study was the wordiness of
the presentation of information. Courseware developers need to present lesson
information using simple and concise statements.

Computer courseware must not take too much time for the learner to complete.
The computer courseware lesson content must be easy to understand and

manipulate.

18.

19.

20.

21.

22.

23.

227

Courseware programs should be designed to allow users easy navigation, control
over the pace, and dynamic interaction with the programs in ways that are
intuitive, meaningful and relevant to the learner.

Visual or special effects must not hurt the eyes.

Use text styles, fonts, and sizes that do not strain the eyes.

The students alleged that the programs had some spelling errors. Courseware
developers can improve the quality of courseware programs by using a spell
checker to correct misspelled words and a grammar checker to correct spelling,
grammatical and punctuation errors.

Whenever possible, computer courseware should be usable across different
computer systems (multiplatforrn compatibility).

The kinds of subject matter and topics that computers are pedagogically suitable
for was a concern to some students. One common suggestion was that computer
courseware should be designed to provide learning experiences that allow abstract
concepts to come alive or turned into virtual reality using simulations and
multimedia elements such as video, audio, and graphics animation. The students
insisted that computer courseware should be developed primarily in subject areas
that are highly abstract, difficult to visualize, or very expensive and/or dangerous
to present otherwise. As a student remarked, “use the computer to convey
information in non-traditional ways, using graphics and case histories.” So, in

addition to other forms of instruction, computer courseware should be used

24.

25.

26.

228

primarily to present new forms of information, knowledge, or learning which
could not be easily accomplished through traditional classroom lecture or reading
assignments. It should avoid presenting same information available otherwise,
and, thereby prevent boredom and fatigue that occur from redundancies,
information overload, and excessive reading which adds little or nothing to learning
and knowledge gained.

Use animation to portray complex systems and processes (such as control
systems) in a way to facilitate learners’ understanding of the processes and
sustain their interests. Both the students and the experts concurred on the need.
While the experts were more stringent in rating the quality of courseware
characteristics, the students were more detailed as to the specific ways the
programs enhanced learning. Both provide perspectives that are useful for
obtaining necessary information for meaningful formative evaluation.

Part B of the Computer Courseware Evaluation Checklist Form, as suggested by
some experts, will be restructured by reclassifying lesson content characteristics
under distinct lesson outcome categories instead of the nomenclature of events of
instruction. Other changes are warranted on the evaluation instrument by the
research findings discussed above. The new evaluation instrument, presented
below, will be structured as follows:

A. Part A deals with human user interface design

B. Part B deals with characteristics of media use

229

C. Part C deals with lesson content by the following categories:

Verbal information
Intellectual skills
Cognitive strategy
Attitude skills

Motor skills

Recommendations

One goal of this study was to construct a framework for evaluating computer

courseware based on the findings of the study in concert with findings from the literature

on current research in educational psychology and the new information technologies. The

final evaluation instrument, presented below, should serve as a comprehensive framework

for a valid and reliable evaluation of all aspects of computer-based multimedia courseware

programs including the level of importance and the degree of quality of instructional and

learning characteristics presented or represented. The evaluation instrument is designed

to allow flexibility in the scope or aspects of courseware evaluation. For instance, the

instrument can be used to conduct courseware evaluation at five levels:

1. Evaluation of only the general features of courseware that uses no media elements

(audio, sound, and video).

230

2. Evaluation of the general features of courseware and the use of media elements

(audio, sound, and video).

3. A comprehensive and detailed evaluation of only the lesson content.

4. A comprehensive and detailed evaluation of the lesson content and the use of
media.

5. A comprehensive and detailed evaluation of various aspects of computer-based
multimedia courseware.

The evaluation instrument can be used for both formative and summative evaluation.
As a framework for formative evaluation, the instrument can be used as a means of
prescribing the components or features of a computer courseware development project.
The careful use of the instrument will allow early and continuous detection of any errors
and defects in courseware programs during the development cycle. As such, courseware
authors, developers and publishers can utilize the instrument as a checklist to ensure the
adequacy and quality of courseware programs under development. As a framework for
summative evaluation, the instrument can be used by schools, educators, administrators,
and others in charge of academic curriculum, to assess the worth or determine the
suitability of computer-based multimedia programs for curriculum adoption. The
instrument will allow evaluators to determine the potential of computer-based multimedia

courseware programs to promote effective learning.

 

__ 2530280 «62:53:. 825-8558 .8 0.02.2. 28...... .o :o..2..a>0 8:80.. ...... 0222.29.80 <

o ...a m .< 2.... .

 

__ 2.8:. .0 0m: 2.. E... .:0.:8 :88. 0... ..o 8.2.2.20 8:80... :5 0352.02.68 <

o as m 2...... _

 

__ .:0.:8 :88. 2.. ...:o ..o 8.23.30 8:80.. .2... 0>.m:2.0...E8 <

.25.... u ...... _

 

= 30:3 .2... 6:38 6.2.... 2880.0 «62: .o 02. 2.. ...... 2:302:00 ..o 8.5.8.. .22.0w 2.. ..o 8.629.“.

m. ...a < 2.... __

 

__ 30v; .2... 6:28 6:26. 2850.0 2.8:. o: 02. ...... 2:38.28 ..o 8.5.8.. .....0:0m 2.. ...:o ..o :o..a...m>m.

 

.253 < E... __

 

”.82. 2. :8 E2552. 2.. ..o 2.2. .o 2..:0 2.. .2...0 ...... was? 8.8%.; 2:80.... 0...... w:.3o=o.. 2. ..

 

2.2—=52.— 0... ... 0n: 2.. ...... Eats—505E83—

.Am0.m0.2.m 3:28.52. 2. :03 2. 8.2.5... £23.08... 6.0.205... $2.058 6.2... :0... .:0.:8 :88. 2.. A0 :5 .02. «.8...
AN. 298.. 00.....0.:.-:s:.=.. 2.63.8. 2325.. 22902.60 2.. CV .0.:0conE8 02... 3.30:0. 0... ..0 08.0. :. 8.5.0.022...
2530258 «62:22: ..85..0.=..E8 2.. ..o 8.580222 .o 8.6280... ..e a 2.. w:..a=.a>0 .a.. 2.03059... a

mu m0>.0m E2552. 2.. ..o 82.8 2.80.. 0.. h .80: 2. ...? .o 8:98.. 0.. Same... 2:38.58 2.. :2...» .a.. 022...... 2.. ..o 88.
:. 2.2.2022... 50.. 8...... ..000 .0 00:5... E. .... .95. 2.. ....3 28.. 80%.. .2... 2.... 52:52. ...... o. 200%.. 95. 0... 22....

6330238 8.2.88 .3 208...... .o 2002....

2.. o. .....820 2.. w:.Ew0. 0500....0 2... 8.253. 0500...... :o 8.88. 322.022. 3:22.260 So... 0.9.5... .5. 60.2.2.8

... 30.05.00. :o 8.8.0. So... $.55... .:o..2...m:. .o .08... 0%:qu .0 202.2. 50...? o. 023.. 2.. 0:...820 o. 8.26:8 83
2...... 2. ... 8.2.88 .0 >2.qu 8 20025 2.. m:..2....>0 :o 2.2.... 2.. ..o $5.5... 2.. :e :02... 8.2538 2...» E2553. 2E.

 

SE2...”— 9.35.93.—
A: .e 25.7. ...—85.32..—

 

gammmDOU sag—FADE Gaga—J—EHEOU
GEH<DA<>H 97.4.. 079547.72 m0..— v—MCBHEE <

 

5.335....—

231

232

.0..0..0.00.0..0 8.20.82. 0 0.02.08 0. 00.82.00 2.. 02... .0.. 0.. =0.» .0... 2.008 Hagan—...... O7.—
mz <7. .— .. U U> H7. 0.0..0 :0... fia—hflfig Q7.

.0.0300.200 2.. ..0 002.02.... 2.. 0. ~........ .0.. 00.... 0.5.0.0880 2.. .0... 2.008 HAG<U.A....< .57..
Hz <7. .. ... U U> ............... <7. 0.2.0 :0... ddfiU—Am: 87.
H7. <7. .. a U U> .— 0.0..0 :0... EGO.—

 

 

 

 

 

H7. <7. .— .. U U.» .— 0.0..0..0.... 12..
”.7. <7. .— ..u U Q.» U 0.0...0 :0... .9660
H7. <7. .— ... U U> ... U> 0.0..0 :2... .980 >~.n..>

.0. A< 2.0.8.0825 >00. 2.2.0.0880 8.8.8.... 0 .0
5.0.... 0... .0... 2...... 20.. .. 6.88.3.0 .0.. 80800200. .0 008002.. 0. 0..0..0.00.0..0 8.20.5... 0 ..03 30... .0 ..0...3 0. 80.20
H. .280 20.8.0 .20.. 00.000 w:..0. 282.00 ...w.. 0... :0 200...... 0000... .8080... 20300.28 2.. ..0 5.0.8.0 02mg 0... .0....

 

 

 

no" on. d ”—0.50 :05 0:.—
-- 0 0.2.. 22.. €22.02
0 0.2.. 22.. ...»...
v 0.2.0 22.. .285...» ...... 2..... ......»

 

 

 

.
.
.
.

NNNN
”WNW
V???

.0. A< 2.0.8.0825 ~9.0. 0..0..0.00..0..0 8.20.20.— 0 .0 0000......8. .0 .900. 2.. .0... ...—....
20» .. .0.:8000 .0 .— 20300.28 0... ..0 2000.....0 .0 0000020 2.. 0. .0....0000 0. 0..0..0.00.0..0 8.20.22. 0 ..0...3 0. 028.. 0... .20....
20......0 .20... 00.000 250. 8.0.-.. 282.00 ...0. 2.. :0 0.00.02. 0000... .00..0..0.00.0..0 20300.28 2.. ..0 0000238. ..0 .26. 0... .0...

 

 

2.0.80.5

233

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

$3.83..

.2 <2 ..2 .5 .. .. o 5 =88. ...... ...... 885.88.. :82... 8...: z . N N ..

m2 <2 ..2 .5 .. .. o 5 .8...8 =88. 2.. N......0 ... 82.8.... 8.8.. N. . N N ..
:83. o... .3 use 0....

.2 <2 ..2 .5 .. .. c .5 .. 8....8... ..8 982.8 .8... o... ... 8.8.. N. . N N ..
ABE—=5
.3538 $030.93 $55.53.: .3358...

.2 <2 ..2 .5 .. .. c .5 8...... ...... 8.8.2... .8...8........... 8.. 2F .. . N N v
3:: Emmo— p-OQO E .5 cop—um :08 GO

m2 <2 ..2 .5 .. .. o .5 828...... 888...... ... ......8 ... .888 ...: ... . N N v
35%.... 32.8...

m2 <2 ..2 .5 .. . o .5 2.. .... 222.... :88. ... b.8358 ...: N . N N ..

.2 <2 ..2 .5 .. .. o .5 .8...8 =88. ... 8.8.8? 8...... 2... N . N N v
00:06.8 39.8.: 05

.2 <2 ..2 .5 .. .. o .5 .... 2828. :88. ... N=85... .... .8... 2.... N . N N ..
.250— 0... 8 00:50.2

.2 <2 ..2 .5 .. .. o .5 83858 2..... 8.8.258... ... .8523 e . N N e
53:00 Gammo—

mz <2 ..2 .5 .. .. 0 .5 ...... 828...... 2.3858 8.3.... 8...: N . N N N
:Ommo_ 05 ac mama—Ewan 05 «a

m2 <2 ..2 .5 .. .. o .5 2..... 08.82.... ... .8...8 .88.... ... 8.8.. e . N N N

.2 <2 ..2 .5 .. . o .5 288...... 2.. 8.. ... 3.... .... 2.2.8.5 N . N N N

.2 <2 ..2 .5 .. .. o .5 88...... 8.8.5.... ......8............ N . N N v

.2 <2 ..2 .5 .. .. o .5 8.2.8.3.. ... .8823 . . N N ..
9.32:5 03599.30

8559.2.

.5435 ...o .5... ES... 45.22.... 255.58 .< ...... ...o 4.5....

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

M22 <2 .22 .2> .2 O O> 5288.525 3000.. 2 2.22.22 .2: 0m: 0:2. ”N N 2.
5288.525 308.. 2

m2 <2 .22 .5 .2 O O> 2.2222200 .20 mam—<2. .o XmQZCo 0m: 0:2. N N v
2.0.0 £295.22 .2953

M22 <2 .22 .5 .2 O O> £8. 2..:0220m50: 0.058582: 8: 0.22. eN N v
. 680 ...—cc. 20:22:92.:
£0922 4.55.352 6529—032 9:05.288

m2 <2 .2 2 .2> A2 0 O> .0 9:02:20 :02“... w:o:.u 00:20:00 N N v
60.0 ....0. £225 .828 £02283

m2 <2 .22 .3 .2 O O> 855020 2.5. So»... 50.0... 0.2. :2 00:02.5 N N v
40.0 ....0. 6:22.... .8200 .m0222N2...mv

M22 <2 .2 2 .2> .2 O O> 2:02:20 :5. Bob.— :00.0m 0.: .20 3:063:00 MN N v
60.0 ....0. .2825 ..28 602.2283

.2 <2 ..2 .5 .. o 5 28.8.... ...... ......N. 828 2..... 2.2.925 NN N N
:mfio:

2 <2 ..2 .5 .. o 5 828 :22... 2.. ... .82... .22: .880 .N N N
‘ 00:22.8:— .0.:2E:U-:5E==

.2 <2 ..2 .5 .. o 5 8.... 8.28. 22.2.... .... 2.2.8.2250 ..N N N
‘ $8.20

M. 2 <2 A2 2 A5 .2 O O> wcfib ..mdv 30:0 32.00.80 20 080 0.2.2. o. N v
20.82.02.322 0. ..0Ea02 0.2.
.02 22:352qu ...—02.0.02: :88. 0:. .2225 m0.—

m2 <2 .22 .5 .2 O 0> .250. 0.2. $3880.52: 00.32. .o 250. 0.22. 2: N v
8225:0822

m22 <2 .22 .5 .2 O O> :o_.a=:o.2:_ 2o 0:... 0.2. .50 .2200 6:802 52 N 2.

m2 <2 .22 .5 .2 O O> ...—5.3.2.3 .0 320:9... .20 0m: 0.2.2. o. N v

m 2 < 2 .2 2 .5 .2 O O> $20.23. .3525... 2 2032.002 2 N v

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

235

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

.Ii ...... .98.... ......8... .. .... H._ ....
. 02 <2 ..2 .5 .. .5 ease? ...... 8.8:. .28.... .... 8.. 2F
. 0.02.0.5: :8002
_ 02 <2 ..2 .5 .. .5 o...... m......3. 0585.8 ... .....a. ... 8.. 2:
“ 02020.0... :88.
_ mz <2 ..2 ..5 .. o> 2..... 0......8. 8.2.9.5 ... ......> ... 8.. 2....
. £320.62: 3.58 05
. mz <2 ..2 .5 .. .5 ... m............ 058......» ... Ease...“ ... 8.. 2:
_ . ...—020.2: 00.38
. 02 <2 ..2 .5 .. .5 2..... 0558. 0582.... ... 82.50... 8.. 2:.
_ E03... .5 .50.. 0050205525 00: <50... .0 5..
§ 1 \ iii- 52.00 2.0 . 5W 1 EL
02 <2 ..2 .5 .. o> ... 9.28... 38.5.5.8 ... 88.... 8.. 2... 3.
.20. 2o
02 <2 ..2 .5 .. .5 .....838 2.. 85...... ... 8.... ...»... 8.. 2F m.
. .20. ..o
.2 <2 ..2 .5 .. .5 ......acs. 2.. 85...... ... 88.... 0.2... 8.. 2... N. ,
.20.
02 <2 ..2 .5 .. .5 ... .....882 2.. 85...... ... ....8 ... 8.. 2: ..
52.0.8025 20:02:02.0 .o 05.0. ..c 32:02:
02 <2 ..2 .5 .. .5 8.2.... ... 520030 ... 3.22.3... 2: ... ,
52.05.02... 03.220... ,
.2 <2 ..2 .5 .. .5 0.2:... .0.. ... 8.6:... .50: ... 8.. 2... ..N

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

236

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

00.03 :30 .0.. .0 0... :. 80088.8.

<2 ..2 .5 .. .5 :88. 05.88 .... 8.08. 2.. 08. a. . N n v
0.00.. ..0 08080.30 .030 80000.. 0.

<2 ..2 . .5 .. .5. 808.8888... 88.808 2.. 8:080 0.. . N ... e
00.03

<2 ..2 ...> 2 0> :30 .060... :. 02.08.020.82 .0800. 0....0..0< .... . N +n 0
80.000 .8000. 00.2800 0.08 0. 0.08.0 m:...:..

<2 ..2 ..> 2 O> 0. .0800. 0... 00.0w 0. 00000.8. 000.8... 0.. . N n v
0.0.8.08

<2 ..2 .3 .2 O> 00000. 80800 8 6.0.0000 6.0.0800 0.0000... 00 . N n 0
08080.30

<2 ..2 ..> .. O> 00.0000 0:0 0.28.0 0. 8.88.0.8. 0.0000... 3 . N m 0
80:08:.
0000000... .0 00.00. 00.0.00. 0... 00.08 ...3 .0...

<2 ..2 ..> .2 O> 0.0.000 0:0 0.00.. 80.6.0. ..0 ..000. 00.0.08..m m0 . N m 0
00000. 0... .0...0

<2 ..2 .5 .2 O> 00 .0 .00 0. 0.000.. ...B .0800. 0... .0....» 00.0.m N0 . N m 0
8.888.:— .0....0>
00:.00. 0.. 0. 0.....0 0:0 00.00.

<2 ..2 ..> 2 O> 0... :0 00.8000 .0800. 000000.. 000000882 ... . N n 0
0.0800 .0 00.80.... w:.:.00.

<2 ..2 ..>, .. O> 0... :0 00.8000 .0800. 000000.. :0..0000..:. 0.. . N n 0

L. ii ; ---- I V 8.80.3. .0080.— “0.0.09
”2072.52922—
EA<DO .29 Ag”:

 

 
 

0200200 20000: .0 ...... ‘

 
 

..8 .0500

237

 

<2

02

.5

O>

000.0.000 .0 000.000 8:00.00

0:00... .058. 2.. ... 02.3... 2. =5 8..
08.80 0:0 08.8.0.0 .0.....0 ..0 ..000. 00.0.08..m

 

 

<2

 

.22

 

0>

 

 

 

O>

.0800. 0... 80... 00.00000
08.8.00 80.8.0 .8 0.0.00 8.0000 0... 00.0.8.0

 

30.0.00 3.0.000

 

 

<2

.22

.5

O>

8000. 0... 800%.... 00.08008 ..0
bo..0> 0 8.00.00. 030.8. 000000 ..0 000 00.2.

 

<2

.22

.5

O>

00.08008 .0 00288.0 30: 0. 00.0..
0:0 0.000000 003.000 00 .0800. 0... 0.00 .2.

 

<2

02

.2>

O>

00.0. 000 0.000000 ..0 00..00..000
.0 000 0.8800. ..0 0000.00.80 0... 08.2800

 

<2

.22

.5

O>

00000.
0... 0. 00.0.:00000 0.08.8.0 .0: 00000.00.
0.00.0. 000 0.00080 0.000 0. .0800. 0... 00.0<

 

<2

.22

0>

O>

.00.:00 00000. 0... 8.008
0. 000.0 ..0 000002000 .0080 0... 8.30.80
0. .0800. 0... 00.0» 0. 00.88.00. 000.8..2

 

<2

.22

0>

D>

00.0.0000 .0
00.0. 000 0.00080 ..0 00.08000 00..0> 08000.0

 

<2

..2

.2>

O>

00.0. .0 00.0.0000
000 0.000000 2.0.80.0... ..0 2.000. 00.0.08..m

 

 

<2

 

.22

 

.2>

 

 

 

 

O>

00800. 0.. 0. 00.000080 0:0 .00..0.00..0
0.000000 00.0..000800 000 00.0.0009

 

 

 

0:0—m .0500.—0.02

 

 

 

 

 

 

238

 

<2

..2

n.>

00.03.30 .5 08.22.09 05

9 3.0.0.. 0080:7023 v5 0:380 gangs...

8 3:03.: 3 00.5.50 .5 0839.2. .0050:

0> 530 c0500. 0: 8 3.35.0.2. 9:000:—

2.

 

<2

:2

m>

033...: .5 £032.09 30: $1.500.
:. 0.3.5. :8 3:30. 05 .05 60.353...
O> .8 m:._0v2: .3 .m...0.m .0 =82 8.0.255

aw

 

 

<2

 

..2

 

..>

 

 

 

 

5:30. 05 .3 3.0098 28.2.0 5.63 0.98.3:
O> .o Bra—.2. ..o 3.0. 05 ..0 0.9538 00.0.5...

 

we

 

0.03.:<

 

<2

.2

A;

3.032.. 2.2:: 8
O> .26: .3 but?» a m:.>.8 :o 3:80. 05 88...

he

 

<2

..2

m>

23:80.5 «€03.05
O> 9 5.5.8 3.0an. ..o 3.353.: 08.550

 

<2

..2

A.>

v0.58... 2.032: 9
O> 5.5.8 «L088. .0 «858:8 05 25...:00

no

 

<2

m2

m>

50:30.
0030.83. 0>...:woo 00: .5 00.88.. 9 .0500.

O> 05 8.. 00...:atoqao ..o .0033 0 0023.5

 

<2

.2

a;

A.

8032: 3.3m 2 5.3.8 .03: .5...
O> 3:80. 05 20.. o. 9:... :5 3:89: 8.032..

no

 

<2

m2

m>

6> 028 o. 3:30. 0... 5.. 2:03.05 .96: 3:82“.

 

 

 

<2

 

m2

 

m>

 

 

 

 

£83
62.. 2:30.03 30.0 :. 8039:. 05 0.33.. 2
30.05. 0 ”20.60:: 3:8: $38 .3 .3283
..wd. 08032.. 028 2 «05.0002: .8 00.3.8.0
O> .85.: ..o 0m: 0... 252.0 .05 03.605

 

S

.0

 

 

 

 

 

 

 

 

239

 

....Z

<2

..2

.>

O>

......m .8 00580.8: 8.8.50. 0... .8 w:.8.. :5
88.800 ..0 0080: 0... :0 83:00.. 8:30...

an

 

m2

<2

..2

...>

O>

:88. 0...
:. ...m5. 8.8. 8 8.....m 80.99... .... 8.88.8:
00.88.. 0. .050. 0... 8.. 5.880%... 8:.8...

an

 

m2

<2

..2

..>

O>

005.888..

A.....050. 0... :0 303:8. 8:.8... :5 :050.
8...... 0... 00.88.. 0. .050. 0... 8.8.8..

uh

 

m2

<2

..2

.5

O>

:0..8: 00588.8:
.....0 8.08 0... 8.. :0......0. 2.0.... .08...
8.8.8080: :5 :0 8.88.8. 3:08...

on

 

m2

<2

.2

 

..>

 

 

 

 

0>

.050. 0... ..o :0.00....0 0......
28.8%.... 8.08 .8 005888.. 8.888809

 

m5

—

 

.....m 8.0—).

 

m2

<2

..2

.>

O>

8.58.... 8.0.8.... 8 .8.
0 :. 8.80 ..0 00.2.0 w:...:8 :o 8.50. 0... «.8...

g

 

m2

<2

..2

..>

0>

.050. 0... .... 0:08 88.0.00: 8 :08.
80.80 8:88.. .8 805.8880 .08.: 8.2m

mu

 

m2

<2

.2

.5

0>

:088. 0:8.5 0... 0. 88.0. 80.98 8.58
00.88.. 0. .050. 0... 8.. 8.5.88.0 8:..6...

Nb

 

m2

 

<2

 

..2

 

n.>

 

 

 

 

O>

0:........ 8 8.5.8.. 0...
0. :0....0. 80:38:00 :5 80.8.. 8.8.8:...

8 .w:..0:08 8:588... .... 0:8..8 8
8.5.8.. :88: 0... 88.8080: 8 8....089

 

...

 

 

 

 

 

 

 

240

:08. 8...... 0... 00.80. 0. 8:80.
0... .0.. 8.38.8000 .::0...::0 8:.>0...

 

 

 

 

 

 

 

 

:88. 0... :. ...ws...
0...... 80.0.2... :0 8.8 :0 8:80. 0... 0.8...

 

 

 

 

 

 

 

APPENDIX A

 

<z_ __<m_><_<< _m_<z _ _ _<m—><_<< _ m—<z_ _ _<m _><_<<_m_ 8225.22.85.52

35:25
N

 

 

_<z _ <m >< << m <2 _ <m >< <<1 m;,<z, _ <m ->< << ms ...?zooséoafisofi _

 

<2
<2

<2

 

nigw... m . .9:an ..alzy:o.emf 5:134 P . -f
mlmaw‘Bmmmaoo ‘ : .mz<3mmz:cw. , ;

38.30 wanker—=90- i

    
  

 

 

 

<< m .2 0.2: =2: :2me .3 5303...? SZ
_ $— >< << m - \ 2.2: ..2: .78.. £232.85. «aim—...:—
<< m

 

 

 

 

 

_ <m >< <m 29:9 :2: 4332399359 93.33.53 2:

... 52.282? .325 2.. ... e8» 8 ...... mes—a: 33mm
_ $— >< << m >< 22.9 :2: 49.5.2323 8.52:8 2..... .832 mus—92
_ «a. >< << m - << 9.9.3 :0... 433239239 9.933....3

2.. ... 5:55:52. .82.: 2.. =2: 3:2: mus—m: H.522
_ é >< << m m 22.: .a.: .88» 2.2.3385. «@253

an 3 8:98.33an .95 85.3.9230 ...—592...:— a a: 5:2...

2: .55.... ......» 5 = 6.9533 .3..— .flofioficm 5m 2: .«o 28 mam—BB .3 3.320820 0.53858 .2353 05 53a 5:3
5% 038 E86 05 :o 2865 83:. .< :5 E Eu: sumo 8m .88.. nouns—«>0 05 2 S 5.— 98 < :5: 9.3; 8?:— 95
2a 22: .a.: 202 .32.». £5 5 com: .5» asp—wen §3§=8 3:5an 0;. .8 .58 523.96 933858 .35 :5 =: 33...

 

”2.3.3::

EEO..— ZO_P<DA<>H gammy—DOD flak—ZOO

 

.a.—:52 .51-5m

< Many—Hm:

241

242

 

<m

><

<<

<2

<m

><

<<

<2

<m

><

<<

83:29: 8.58

.2: .3 9:8. 98:28
:2 828:» ..o 8: 2:.

 

<m

><

<<

<2

<m

><

<<

<2

<m

><

<<

8an—
ofi 9 28:8 2:38:58
05 .3 855.2 2:.

 

<m

><

<<

<2

<m

><

<<

<2

<m

><

<<

.8285:— 93 380:8
£8: 05 .«o 3032
a 5? =38. 2.. 82m

 

<m

><

<<

<m

><

<<

<2

<m

><

<<

88805 2.80

4:258 :8 .828 :80
..wdv cocascefi a: 9.5.8
.3 8083388: 22.

 

<m

><

<<

<2

<2

<m

><

<<

<2

<m

><

<<

323“

89.25 05 8.: 8185::
6:: 380:8 :88.

no 80:88:88: 22.

 

<m

><

<<

<2

<m

><

<<

<2

<m

><

<<

38:8
:88. «o 3:20 2:.

 

<m

><

<<

<2

<m

><

<<

<2

<m

><

<<

22:8
.88—2 2: no 3032
a 5:5 :88. 05 m:_::_wom

 

<m

><

<<

<2

<m

><

<<

<2

<m

><

<<

.83858 2:
8: o. 26: 8 $8825

 

 

 

 

<m

 

><

 

<<

 

 

<2

 

 

<m

 

><

 

<<

 

 

<2

 

 

<m

 

><

 

<<

 

 

83.83: 858
838:2 :_ 33:8 :88.
.3 8083888: 22.

 

 

 

243

_<z

<m

><

<<

<2

<m

><

<<

<2

<m

><

<<

>m<mm040
5 325.:— =o=u§££
«o .3333 22.

«—

 

_<2

<m

><

<<

<2

<m

><

<<

<2

<m

><

<<

628:3
mam: 05 no .3333 2F

5—

 

”<2

<m

><

<<

<2

<m

><

<<

<2

<m

><

<<

:oungcmcafimmoooa 5
32m: he mans—Bum: 2F

@—

 

N<Z

<m

><

<<

<2

<m

><

<<

<2

<m

><

<<

539583 9.2938

5 Eu 3 ENPZOU

m0 man‘s—hams;
..o nous—.82; 22.

m—

 

w<z

<m

><

<<

<2

<m

><

<<

<2

<m

><

<<

«353:. :88—
05 .8 9:53. mfioSfi—S
:_ 396 mo 8: 2:.

v—

 

<m

><

<<

<2

<m

><

<<

<2

<m

><

<<

£3.83:— :88—
05 «o 9:58. 9:05.20
5 82> no 8: of.

«—

 

W<2

“<2

 

 

<m

 

><

 

<<

 

 

<2

 

 

<m

 

><

 

<<

 

 

<2

 

 

<m

 

><

 

<<

 

 

«3.83:. 858
05 mo wEan. @6525
:m assigns mo 8: 2:.

 

~—

 

 

<m

><

<<

m<2

<m

><

<<

<2

<m

><

<<

2.2.3.?

0.... .8. a. 22:8

:83. 3...... 225.0.
2222. ... 2.2.8.... 92>...»
82:9... .3 32.3.8: 2....

cm

 

<m

><

<<

m<2

<m

><

<<

<2

<m

><

<<

2.2.80 :88.
2.. 9.5.3.0 ... 2.2.8....
32>”... ..o «8238.. 2....

mm

 

<m

><

<<

m<2

<m

><

<<

<2

<m

><

<<

«c262...
32>». ..o 2.20 2....

Nu.

 

<m

><

<<

m<2

<m

><

<<

<2

<m

><

<<

..2.2..2...2.
...... 22.5.59...
5.3 22:8. 2.. 26:2.

.N

 

<m

><

<<

m<2

<m

><

<<

<2

<m

><

<<

€02.5o .mtasoac...
£23.28 .m..2.8.m.....
£29588 .32..

8 ..va 3.9.8.... 02253
2.2.253. ..o or... 2....

an.

 

.

 

 

<m

><

 

<<

 

 

 

 

=«:»m-
M H¢<>¢Hm¢DOU

 

.

 

 

<m

 

><

 

 

- 3.9... m ......wccu
N. mmwkummaou I

 

 

<2

   

 

.

 

<m

 

><

35.3.5.3...
.m¢<3mm¢:ou

 

<<

 

 

   

do...

.3832. .2523. 28.
3:22.22... Eggs
2.. ..o >233? 2....

 

a.

8.5.33...— 3:23:52..—

245

.II 82.5.25:

88.0w .........2

.Il Bee. H2......2..... 0......EE .3.

H<20 .mm

+2 I..- .m I12-.. la... '83.... ”28.50.50? .2

0.2.8... 2..—z ”.2286

..m

...?» :5... 5.2.8.8... 9.32.8 2.. 2.2.9.... 080...

 

5.2.8.3..— .anuuv II .— ......—

  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

. 8a.... .8.
<m >< << .3 808.2529... 2.... om_
8.3 .22
<2 <m >< << m <2 <m >< << .3 «85.2%.... 2... 8
<2 <m >< << m <2 <2 >< << 200.805.3283.... .2
22.888...
5.38.8... ..o 22 _
<2 <2 >< << m <2 <2 >< << o....o>o.§=8.eo§. Rm
85:28.2. ...o... _
<2 <m >< << m <2 <.. >< << 533252258... e“
. 88.2.8. _
3.8.2.... o. 82:80..
<2 <2 >< << m <2 <m >< << .8 $822.2...“ 2... mm,

 

 

 

 

 

 

 

 

 

 

 

 

 

 

246

 

 

 

 

 

22.06%. .a.—Eco. N 983358 :o 558 hon 26% .5» Eu 2:: :02: 32. Jaw—26 :0 .3.
33.58; 5:03 _ 295858 no .853 Ba 2.0% :9» Eu 2:: :03: 32— .03.26 :0 .9
a: a. _ :82 ataxia m 2.38.58 8: so» 2.. :83 a 3.2303 38: has. 32. 6?:on :0 . _ v
wmmfiofiam 35.58 N 0.838....8 om: so» Eu :3? a $5333 88: .98.: Be: .0953 :0 .cv

Eat—.88 cocoa; _ 033858 8: .5» Eu :8? a Amcofimomv «2:: >52: 26: .omBQE :0 .9“

92 Ill 8% ”.2383 a .56 so» an .wm

 

mzmhm>m mmPDmZOU.OZ_1—m2mmm<

c2025 18985392.»: $3 302‘ #533333 382552
$6 483; .295 ..an 83325 02325.85
Esmsz .a.: .55 ..m3 302 205522228

 

md<3mmeOU mmHDAZOU
mm2<0 MESA—ECU

 

 

 

 

 

fiasco.“ mm<m<H<Q
Eggs EmmIsz‘m—yfim
033:8 02—mmmUOMEDy—OB

 

 

.moEEa 35 =a “.85 83..— EES 3512.5 3: so» 32 wage—.a.“ 05 .3 523 Km

Siam

5m: 3 8&8 £85 .sfi a...

.25... Boga—Eon =oE=m2m H_o>e_ man—U mm

247

8505500

 

 

oz 80>
58858 owe—.8 .550 595 5 8523.5 255858 :08 8: o. 8.: so» 2.53 .3

 

 

oz 80>
505.98% 05538 5c» 5 £5.58 55o 2 AmVEEmoa 255853 05 5.05582 .55 2.83 .3

”9:05:80

 

 

oz 80>
$0582; £5.25: :88— 8 852 05 Ego. 533.250 2 so» Eon 2558.53 05 Be 535% 55o 5o» E .3

 

2: a. _ :82 2.955% m 255858 :0 5:88 Ba .5on .55 Eu 2:: :05: 5c: .8296 :0 .3

 

248

$.58 ...... :_ 552:3...5 .5. 8:22.89 .59» .5. £55 .a.—2' .5.» ...—2:.

 

.523 08% 2: 5 E Q _ :82 25.55% n 2558550 53: 05: .55 .55 850558 85: 5:: 255 88$ .3

 

.523 858 2: 5 758% .9558 N 2558.500 585 0.6: .35 .5» 88558 8:5 5:: 81.5 88:— .av

 

.523 858 on. 5 25.85.— 5503 _ 2558500 585 55: .85 .5.» $8558 85° .5: 2:5 88..— .m:

 

APPENDIX B

a. 2:58.58 85:58 ..0 508.....0 .0 8000:: 0... 0. 2 5.5.0.0250 >8. 5.5.0.0250
.58....2. : ..0 008.555 .0 .08. 0... ...... 0.5... :0.» .. .0.:558 .0.: 255858 85:58 ..0 8:050
..0 80008 0... 0. .5588 5 5.58. .5: 8.8585 ..0 5.5.0.0550 8.8.2:: : 50.55 0. 028.0 0... 0.:. 0. 5 0.8:. 50>

 

"2.5.00.5

2558500 0.... 5 .8582: 2: 8.0.5:: .5: £25802:

8.505.: .5858 .58.. .50.. 88:: 0. 50.. 50:: ....5 50505:... 0... .05.... 2558500 85:58 5 8.8.85

0.: .5. m.:0>0 5:08:85 .5: 850050 5.58. «588:: .0.. 50505:... : a: 8:58.. 5.0.. 5 .. A8582... .5: 50500
.::0..0=..m5 :0... 50500 :88. .5: 2505.: 2558500 85:58 m5.:=.:>0 .5: 555.5: .....5 58.. 5055.85 2....

..5:0...:: 5 8:05.85 50088.0 .5: .508m 5 30.9.08: 58.5058 5 52:82 :0 00:03.5 550.582: 0.. ..0
85:00: 5.5.: 5... .0.. 8.8.8 5 8:05.85 ..0 5.8... 8.08:0 2558500 85:58 ..0 50850 .0 8000:: 0... 0. .858
0.: 85:80. 0>..00....0 .5: 8:05.85 0>..8....0 ..0 8.8.8.0250 :0. 30.2.08: .55.:058 5 52:82 50... 35.05.. m:
:05 w: 8205.85 ..0 5.00... 8.880 ..0 58:8: 5055 0. 028.. 0... 0.:. 0. 50.. 50:: 0. 5 5055.85 5... ..0 80:5: 2....

 

5.262....

SEC... UZ—hé Hmsv—Um—IU 7.65351 >m gamszGU fiEEOU

 

A: .0 05:2 555:5”.—

m 592mm:

249

 

0050.55: 50.505. 0... .0..

 

 

. N m 8.5.0.5.: :080. ..0 5550......5 ..0 .0>0. 0.. H 5
8:50. 0... 0. 00:58.2 0558.500

. N m 0... ..0 :0..:.8:0.:05 .0 505055 5
50.500 :080.

. N m 5:: 803.0030 0558500 5005.05 505.). n

 

5080. 0... ..0 5.5.52. 0... .: 58:58

 

 

 

 

 

 

 

 

 

250

 

o . N m 0.8.5522... .50.:00 .::>0.0. ..0 5050”. v
o . N n 0558500 0... 85 0. 50.. :0 505025 N
o . N M 850.55: 50550.... ..0 50.50....505. N
o . N m 803.0030 ..0 505056 .
0.5.05...@ 5:... 0....
80:03.85? 0.558500 555.00 8.... 5. 505.5...
5.808: ... 0.8.8.8550 800:... 0... :. 8.8.8.5550

.0 00::...0...:. ..0 00505. 0... 0.580.. 8:0... 5.58000: ... HKDSDMHW Hz<>>mm~500 u< ......

 

 

 

 

8.8.8.8550 0... 5.88.0 0. 585 0.. 5:0 5... 8.05:. .0 85:5 .050 8035: 0. 00... .00.. 8:0...

.0.:8 0... 0. 5.5.88 50... 0.... 5:: 5088:: 0.. 5.50.8 5.... 505
0558500 85:58 ..0 80550.. .0 8.8.8.8550 85.0 0.5.0:. 080.: 50.88 ..80 :.....5 505.>0.: 808:: .55... 0... 5.

M ........3............:::.:: a 03:0 :05 dJaU—w—A—hd‘ .Fcz
: . 0.25 E... .58
n N 0.0..0 :0... .55.sz
m n 0.0..0 :0... .w—UE

 

 

6G9:
u—t—n—I—t
NNNN

 

251

 

Aaafiomtan
8 .5an— 05 .6.“ 35:25“:
$535: :83. 95 .33 «a: .6an—

 

 

 

 

 

 

_ N m 05 £36835 .3 «Bug .8 .0>0. 2: m—
:ccsgmoa 5:25.:

_ . N m mo 0:: 05 ~96 .82—8 .553.— 2

_ N m . aemcargzafi ho a: 2: 2

_ N m 330%”: .8593 2 x2313 m “—
mo>_.oo.3o canno—

_ N m .23 $8. oofigotoq 5233 £032 2
. 32:8

_ N m 59.2 05 meta—o 2 2262:. 3033— m—
:83. 2: .3 can 05 3 «0.305...—

_ N m can 330:8 £2: 2: .3 2639: N—

 

ana—£25 fits—03¢:
£03298 .a.—3.862: .moEEaxo .35:

 

 

 

 

 

 

 

 

_ N m ..mdv 8.9.85 .a.—£6.58. .3 8: 2E. _ _
=5.
:83. :03 5 8 528 :08 :c 12:32.“

_ N m 5:58.85 .3 25:3 E325 on... :—
oon—0:5.“ 35:25

_ N m . on. 8.“ 2:826 :83. mo bus—9:3 2:. a

_ N . m 22:8 :83. .3 =o_.§m§a§osh a

 

 

9.52:5 Ea.— ” c...—

 

mmo=u>=8=o 29:09.39
9.7.8me ... 85.83229
2. 353.59.: a: 3.50:

.-...E:_w.ow._m.=_ ENE?! ,, x
82:.» 2: 5 3:»:89935
2: 8599. 3.3..— ..Dannouu: =

     
    

 

 

252

 

@000 .3838: .8285

 

 

 

 

 

 

o _ N m 0.89 _0:o_.aw_>0: 08300880 .8 03 0E. nN
A20 .0.—02 Esau—09>“: .m0w0a
8.803008 8:38.989: «808888

c _ N m 8 08080—0 c028 9880 00:20:00 «N
60.0 .38. .823 .880 £02.53

o _ N m 08080.0 0:0 80.3. :028 05 E 00:0.am mN
.80 ..x2 .82—5 .880 .0029“th $580.0

0 _ N m 0:0 :8?— =020m 05 .8 5:030:00 NN
A000 89:5 .880 £029.23 08080—0

o _ N m can 28.?— :0080 05 No 58:90.5 _N
p.980

o _ N m c023. =E0>o 05 .8 .0250 .083 88:00 8
909:0

o _ N m wcib ..wdv 08:0 95000.80 .8 800 2:. @—

 

 

 

 

 

 

=w_n0: ...—a 39:3 50.8w

 

mm0..0>_.00=0 08:02:80
wing»? ... 030.20.00.85
.8 00:303.... .8 00.30:

...-5.00 :5 =_ 502.5...—
0002: 0... 5 00230080028
0.: 0.53.. 030—.— ..930000: .=

EDBDZHM gammy—DOD 3. In.—

 

N

m

 

 

 

 

 

 

COOOOO

_————v—I

 

 

NNNNN

 

MMMMM

 

 

 

 

 

253

 

 

 

 

 

 

 

 

 

 

 

 

 

 

.. _ N m
o _ N m
o _ N m
o _ N m
o _ N m
o _ N m
M 8:00 00.0098 8
o _ N m 8800:. 0.0080800 o. 080. 8 0m: 0:... mm
.00. 8 5:30:00.
o _ N m 0:. 00:080 0. 00:90 3.0. 8 00: 0:... Nm
.08. 8 380000.
0 _ N m 0... 00880 0. 000.0 .x0. 8 0m: 0:... .m
:8. 8
o _ N M 3:30:00. 0:. 00:080 0. 880 8 0m: 0.: cm
8:08.08.
E8280 8 0:8. 8 8.80.: 0288
o _ N m o. >M<mquO 8 0:80:08 0:... 0N
8:08.08. 03.80...
c _ N m 0.0.80 .0m 0. 8:08.. ...—m: 8 00.: 0:... ”N
o _ N M 820888. 30000 0. 3252 8 00.: 05. ..N
:0_.0:.8..:_ 000000 0. thH200
o _ N m ...O mama; 8 XmQZ. 8 00: 0:... 0N

 

 

 

 

 

5.00m 80 .884 80.0w

 

 

0008:8800 0.0300080
8.80000 :_ 0880.00.08
.8 008.088 8 0050:

 

5880 0.... :_ 828.:—
00008 0... :_ 00.310.00.08
0... 0.0.0.0.. 00.00... .b0mm000: .—

 

Hy—DFUDy—Fm gammy—DOD 3. t0.—

 

 

254

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

c . N m
o . N m
000.00. 0.. 0. 0.....0 .000 00.00. 0...
o . N m 00 02.0000 .00.00. 000000.. 02.00.0000. C».
0.0.000 .0 00..00..0 m0.E00. 0...
o . N m 00 02.00.... .0500. 000000.. 02.000000. 0m
00..:0..< 80.001. 3.0.09
00000>..00....0 0.0302000 00.0.00 0.... 0. 00.2.5.0
M20000... 0. 0..0..0.00.0..0 000000 0... 0. 00..0..0.00.0..0
..0 00008000.. ..0 00.9.. 0... 0.0.00. 0000.0 8.000000: 0. PZHHZCU ZOmmHA “m 20..
o . N m
o . N m
o . N m
60.0 .0.00....0 000.000.. .w0....w._..w...
o . N m .00..00..00 .00.. 0.00.8 .0.0000 ..0 000 0.. ... an
0.0..0.0.0 00.000 0...
o . N m ..0 m0..:00. 00.000000 0. 0.000 8 000 00... ..m
. 0.2.0.00. 00.000 0...
c . N m 8 m0.0.00. M0.000..00 0. 00.0..»8 000 0.5. on
0.0..0.0.0 00.000 00.8 3.0.00.
0 . N m w0.000..00 0. 002.08.00 .8 000 000. mm
0.0.8.00. 00.000 0... ..0
o . N m w0.0..00. M0.000..00 0. 000.00% 8 000 0.; 0m

 

 

 

 

 

8: 000:.

 

 

00000>..00...0 0.033.000
”0.000000 0. 0..0..0.00.0..0
..0 00008005. ..0 00.u0n

 

0.00.00 0.... 0. 00.2.5.0
000000 0... 0. 00..0..0.00.0..0
0... 0.0.00. 0000.0 2.0000000 ..

 

EDSDZPm HM<>>HW¢DOD .< ......

 

 

255

 

00.0. .0 00.202... 000
0 . N 0. 0.000000 020.0020... 8 ..000. 00.0.0.0..m 0..

30$.0005 0000000... .0
00.00. 00.0.00. 0... 00.05 ...... .0... 0.0.000
o . N m 000 0.00.. 50.6.0. 8 ..000. 00.0.0.0..m 0..
0......0 0005.020... ...0 ..000g..0_05...m

00000>..00...0 0.0300500 0.50.00 0.... 0. 00.2.5...
00.000000 0. 0..0..0.00.0._0 000000 0... 0. 00..0..0.00.0..0
..0 0005.005. ..0 00.00: 0... 0.0.00. .0000... .....0000000 ..

 

 

 

 

 

 

 

 

  

    

  

: .. 0.20.200 200000 0.. ......

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

_ c . N m

j 0 . N 0

_ 0 . N m

w .0500. 0... ..0 00.00020 0.220 2020.30.

V o . N m .905 ..0 00005.0...00 00.8.000500 we
.0500.

_ 0... 8 00.00020 @2020 02.00. 000200 .0

__ o . N m .2222. ..0 00.0. 058 0.05020 000000... 3.

w .0500. 0... 50... 00.00020 002.080 .

. 0 . N m 50.00... ..0 0.0.00 .8000w 0... 00.000—O me i

4. 000.00. 0.. 0. 00.00000... 000 000.200.... ,

_ c . N 0. 0.000000 00.0..000500 000 00.2.0000 Nv

h 00000. 0... .08 00 .0 ,

U o . N m .000 0. 0.00 0.. ...3 .0500. 0... .023 00.0% .v i

. 00.2.00 ...6 0.0300500 0... ..0 .00.-.04 02.03.020.0—

_ 00000>..00....0 0.0300500 00.0.00 0.... 0. 00.2.5.0

__ ”0.000000 0. 0..0..0.00.0..0 000000 0... 0. 02.23.00.020

2 ..0 0008.005. ..0 00.00.. 0... 0.0.00. 0000.0 00.000000: .. ..ZHHZOU ZCmmHA um .50

 

256

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

00000>..00...0 0.00.00.000
00.000000 0. 0..0..0.00.0..0
..0 0000200.... ..0 00.00:

 

 

 

 

000000 0... 0. 00..0..0.00.0..0
0... 0.0.00. 0000.0 5.0000000 .—

 

 

L Elm—0.7.0.0 7.60%.... R. ‘0...

 

00.0.0000 .0 00.0.
. N m 000 0.000000 ..0 00.0.0000 00..0> 0.0000... n...
0.00.0880 00.0000
. N m 000 0.08.0 0. 000000000. 0.0000... Nn
. 0.0.0.00. 00000.
. N n .0003 .0 6.0.0000 6.0.00.8 0.0000... .m
000.004 0.. 0. 08.0.0.0...— 307%.000000
. N m
. N m
. N m
02.000
0.00.0 .0.00. .0.0. 0... ..0 000.00.00.00 0...
0.00. 0. b00080: .00..00...00 03.00000.
0.00.0000 0000.020. ..0 00000... 0... 000
. N ... 3.0.0 ..0... 0.....0000 ..0 ..000. 00.0.0800 on
0003.00 .0 0.0.>0..0.. 300 00.0.00.
0. 0.0...0. 000 .0500. 0... .0... 0000000...
_ 0 . a 00:008. ... 0:00... =82 800500 a.
000.0.000 .0 000.0..0 .0000.W00m...0.0 .
.0800. 0... 0. 00>.0>0. 0.. ...3 .0... 000000
. N m 000 000000..0 .0....0 ..0 ..000. 00.0.0800 00

0.....0 8.0.0.020... ..0 ..000~. 00100—0050 .

257

 

80.80 8000. 0... .8008
0. 0:0.0 .8 00:03.00 .08.: 0... 8.328..
:. .050. 0... 0:...» 0. 8.88.8. 00:30.:

am

 

 

 

 

 

 

80.80
:0000. x0.:800 088 0. 0.:8.0 8.8..
:. .0800. 0... 02.88. 8:08.08 00:38:

 

an

005.080 8:804 ...—5.8...—

 

 

 

:0..00: 00588.8:
.....0 8.08 0... 8.. :0..:..0. 0:0.0 .0:0.:
008.8080: :5 :0 8.88.8. 08000.:

R

 

00::....: .0 0.0.22.0: 0... 0. 8.0.0.
000550080 :5 0:0..00 8.8.0:... 8

w:..0:88 .3 00:8...0 8 0838.0: :0..00:
.85 :050. 0.. 0. 8.58.0.8. 08000.:

 

0>.00
0. 8:50. 0... 8.. 08030.: .08: 08000.:

on

mm

 

 

 

 

 

 

€80 2.. ea. 2038..
0:30 :. 80.8.: 0... ..83: 0. .30.8.0 0

08:00:: .08.: m:...0: 8 30.8.0 rad.
080.30.: 0280 0. 00.8008: 8 00.3.8.0
.0808 .8 00: 0... 032.0 :5 009.8000

 

:m

 

 

 

000:0>..00....0 0830880..
”800000: :. 0.3.0.0888
.8 005288. .8 0050a—

 

 

 
    

..::.,.._0.01. 0.... ..., 838.. y
0000:... 0... :. 00.320.00.80

0... 0.0.0.0.. 0000.: .a.—000000: .—

 

.»...th 8.000.. "0 ......

8:80.— 0: 0. 8.88.8.8— .507. u:..:000..— ,

 
  

 

:0800. 00.w0.8.0
02.880 00: .0 00.88: 0. .0800.
o . N 0 2.. a. 8.05:8... ... but? a 88,2. 00

 

8000. 0... :. :80.::00:0
30:080.: .0: 0005.08 0. 00.8

 

 

 

 

 

 

 

258

 

 

 

 

 

 

 

 

 

 

   

c . . N m :5 0.:0080 2:5 0. .050. 0... 0..0< 3
0:83 :30
c . N m .050... :. 8.58.0.8. .0.. .050. 0... 0..0< no
00:08.88: 0... m:I..L.0..H
o . N m
o . N m
o . N m
00588.8: 0.850.
0... :0 83:00.. 00:.>0.: :5 :083.
o . N m 0.....0 0... 00.88: 0. .0800. 0... 00.8.03. N:
88...:
.0 8.23.2. 0... 0. :0.0.0. 000580080
:5 08:0: 8.8.0:... .0 88.088
£85880 .3 08...... 8 8.58..
o . N m :0..00: 0... 00.8.0880: .0 008.8009 8
80.8.: 0
:0.0.0 0. 8.880 .08: :8: .0800.
c . N . m 0... :.0.. 0. 08... :5 0880.: 0028.: 8
00:02:30.: 0
0005028....” 0.0300800 :8:.00 0......:.. @022... 1 i I 1 ; i.
8.000000 :. 2.0.8.0088 080:0 0... :. 8.3.8.0808
.8 005.85... .8 00.00: 0... 0.0.0.0. 0000.: $800000: .— hZHHZOU ZOmmHa “m .8:

 

 

n‘ "

 

259

 

0...0.0 8 0058.0...0:
0.80.00. 0... .8 8.8.. :5 ~80.0000
c . 0 0 .o 8.08205 .8008. 8032.. 0.
.0800.
0... .3 0:08 08.0.00: 8 000.0. 08.80
c . . N m .0808: .8 0000080080 .02.: 030.5 Nu _
:08000... 080.00.: 0.
0 . N 0 8.3.8 {3.8. .o 82.0.8 8:080 .0
:0.0002: 080.00.: 0. 8.8.00
0 . N m 0.850. 8 000800.80 0... 08....80 on
00.... :5 0.8080 .8 8.50.8.0 8
0 . N m 00.. 0.8.50. .8 000800.80 0... 088.800 0:
080..
.8 08080.80 .85 :0..00:: 0. 008:00.
0 . N m 0.850. .8 000800.80 0... 08.800 8

0.0.2.80..— ...—.80...—

 

 

 

 

 

 

 

 

 

 

 

 

 

 

8000. 0.... .... 8%... 00.00.
i 8 0.....0 80.0.3: ..0 8.8.0.8: 00.80.:
0 o . N m 0.850. 0... 8.. 5.8.880 00:80.: h:
, 00800. 000......
0... 0. :0.0.0. 00088 8.0.08 00.80.:
. c . N m 0. .0800. 0... .0.. b88880 00:30.... 8
00:08.88.— 0... a0...—

 

 

 

 

 

 

 

 

 

 

000=0>..w01....0l0...gal..5l0ilr! -lx, .050.—,8 0.8!... 000.8. ,.., ,
8.000000 ... 0..0...0.00..0..0 0008.0 0... ... 00..0...0.00..0..0
.8 0000.088. .8 00.80.— 0... 0.0.00. 0000.: 35.000000: .—

    

 

 

260

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

o _ N m
:88. 05 Boga—5:: 3.9.58 .8 but?
o _ N m a wins—2: m3£>2 38% mo 8: 2: cm
358. 2.3m 2: oozofln 3 5an—
o _ N m of 8m mutiny—950 728:6?” 3235 an
.83—.a.; 3.: 32.3.3— “52.3.5
o _ N m
o _ N m
:88— 05 E Ewan. .a.—Em
o _ N m Roam—E =a win: :0 .558. 05 93% an
mEoBoa 2.3:: .o _o>o=
o _ N m mo but? a 9:28 .8 SE3. 05 38% K
cousin cog—:8? 5 .52 a E cocoa
o _ N m mo 86:0 9:me co .053. 2: 88% on
moi—SW5 .o mcocgzm Bu: 2 8:: 28
o _ N m 230:8 9.31% :o 858. 05 28,—. mm
$83 :30 he: .5 E; E =o=-E£E
o _ N m :88. 9588. :o 553— os. 38,—. vs
355.8...3m ufinmonnt.
$233.3...» Bataan—.8 5:23 :5 :_ 32.3.:—
wfimmomma E acmtotahaso 822.» a... 5 323.3322?
ac 8:5..2—5. ..c 3..qu 2: 853.— 023..— .Dammouo: .= PZHHZOU ZOmwmw— um tam
o _ N m
o _ N m
o _ N m l

 

 

 

 

 

 

 

 

261

 

 

 

 

 

+_N omé. 2.: 26 n-—

”maoz E 02.3398 9.303. :8» ..o own». 95 286E 8.8—a ...BEoE .a.—=8.“ a 2m .8» .= .3

 

Esau 38:: 35o
Eng—Egon 295858 859:3 E 3:396 :85? ham
Even—Egon 033353 .2383 E 892—5 .23 ham
0.330858 05 3 @2969 no.8 23:3 05 E 85596 5223 bSm
III. 2958.58 05 E @2260 88 52:8 05 E 85.898 .23 ham

Ill Even—Egon 29508.80 5.2.88 E 82.698 :55?» b.35—
EoEn—Egon 253858 .3383 E 82.698 .33 5.28"—
Essofisoo 2: .3 v9.38 «Ba 28:3 05 E affix". =85? 532$
I 033853 05 3 @238 33 22:8 05 E 82.698 .23 .a.—scam

 

 

BEEBE .a.—Emmy“; .3

+3 9. - on mm - N .695 28 N
“Eng E van“. ow< .Nw

vacuum 032 ”not—~00 . — w

...—85?» Soon :EEE8E 9:325. 05 bang 3.on 2.3.99.5

ZOFEOKZ— gun—zany .l D t..—

:3...» E... E 52:98:!— eaa 5:22—93 ...—9» ...: 52:: .a.—o.» .5» :52;

3......ch .25 .3.

ii

02 89> 232.58 a .56 so» on .5

 

82388 «32.28 02.3289.
5.35828 ”.3228

825 x3228

.5325 ..8m_8.. 6.63%: ......3 8.88 58.338 582.532
.10 .82... .28.. .....3 89.5025 92525.8”...

Ema? .a.... .....e... 2...... 308 20:52:25.8

2.3.8 83.5.5

25...... ..mmzmnfixmm

eggs. 02.880550;

262

 

 

 

 

 

 

 

 

$39. 85 =a x85 83..— 223 35598 can :9» 26: 9.32.8 05 ..o :02? .em

 

 

 

 

 

+2 a - N. w - v m - _ o
Homo—26¢ 96: so» mEEwoa 9838.58 no .383: 8aEonnu 05 885E 833 .nm

APPENDIX C

 

<2 9? .— 5 my U> .— 32585 .55—
<Z m> .— ..u G U> U 0.25 :2: .9660
<Z m> .— h D U> U> 225:2: .980 >¢H>

 

 

"E 2 EEBBEEU .58: E38522: 5.8.5:: : .... 3:5... 2.. .a.: ..E... .5.“ a. 6.35:»: .5..— .:2:82:2 5 8582:
E 02:88:58 5.8?8 : :03 3o: 5 823 2 50:8 2: 2 20:2 5:50 288:8: Em 05 :o 0:: mE-Eo .3 0:888:58
2:3858 859:8 .5 >58: 2: 53: :EE8 59» 2:8 “E86 2: :c 2:E:E 8:2: .m 2:: :8 < 2:: E 50: :08 .5..—

 

.:2:8 888— :5. 0.5.255 8:302:90 ..22—590 "Egaégm :5. w:_§_::< I. m ...!— ::: < 2:.— 38:99::
.0 2:.— .8: :E858E 2288 : E 22:. .8882: 2:8 9:: m .5.— ::: < 2:.— 5o:
852:5 83858 2: 53: :E.:~E£E .208» 2 85.2 0 2:.—
.2:38.58 2: E 8582: 2: 8:55: :5: £25885 .mo—EoEa 580:8 68:: 3:: 88m: 8 :9»
32.: E3 “—53:52: 2:. 2:3858 2: E 80:55: 98>: 5:::58E :8 8585: wEE:o_ 2: mE88m: 823E
x85 2:. 685228 ::: 52:8 _::E8=.=mE rod 52:8 :88— 2:38.58 m:::=_:>o :8 55.55:: .23 E8: m .5:

.:8_::w8 b.2588 E 2:3858 3o: 2 222 258.5 .2525: 053858 859:8 wE.:E:>o .23 E8: < 2::

50:55.5: 82.5.96 2:38.58 859:8 E5 2 3.5: 025 2: 22:.

SEC:— hm—Av—Um—IU 20:58—55: gammy—DOD lumbar—ECU

 

A: .5 25:2 3.5-:3.—

U 5929—:

263

264

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

   
  

.280.
0... 0. 00:80.2 2:38.80 0.:
<2 A.> A. .. 0 O> <2 ..> A. .. O O> <2 A.> .. .. O 0> ..0 8888.8: .0 .:0:.2:.m :.
f 8280 880. :8 82.0030
<2 A.> A. ... O O> <2 A.> A. .. O 0> <2 A.> A. .. O O> 2:38.80 803.2. 22:5. m
880. 2:..0 w:.::.w0..
2...: 3....» 2.8.822...
<2 A.> .. ... O O> <2 ..> .. n. 0 O> <2 A.> .. n. O O> 82:00 .::>0.2 ..0 30.8.. v
. 2:38.58
A<2 ..> A. .. O O> <2 ..> A. .. O O> <2 A.> A. .. O G> 2: 8: 0.30.. :0 80:02.: m
_ 00:22.:
<2 ..> A. .. O 0> <2 A.> A. .. O O> <2 ..> A. n. O O> 8822.0 8:85.82 N
<2 A.> A. .. O O> <2 A.> A. .. O O> <2 A.> A. .. O O> 8280.308 .:2:0.:.m .
0.5.085 82m0..A.
_ E ...... ._ :82 3.32: . 25...: .828 - 13.53.. 5.3. ,w 8888 w -
Liam—.8888 N 8.4.3.28... .2888; 84.30.88 It... ,

.0.:8 088 E... 0. u:.:..000: E0... 0:... :8 :88»: 0.. :...0..: 22... 8.. 8:38.80
8.2.88 .0 80.38.. ..0 8.8.8.0280 ..0...0 0:...0:. 8:0... .8208 ..0:0 :.....3 :0:.>0.... 80:... x8... 0... :. .l 0.02

 

.... 0.8.80 0. 0...: .0: 0.... 8.. ...... ..0 .0...:0......: ..w8... :0>0 ...2Hm..< E ..
..0 .0.:30280 0... ..0 82.00.30 0... 0. 2...: .0: :00: 0.8.8.0280 0... ...... 8:2: ma..<0.a....< .....2.

<2 ..> A. .. U G.» <2 0.2.0 :2: ddfl<0.......< #02
<2 ..> .. .. U U> ..> 0.2.0 :2: 4.8.. >28,
<2 A.> A. .. U Q.» A. 0.0...0 :0... .~.OOA.

 

 

 

 

<2 m>

A;

O>

<2

O>

$9832 9536 3 38.68:

m—

 

<z .5

O>

O>

m>

O>

<2

.5

O>

mo>_.oo.3o :83— .Ea 38.
358822: 5253 :23,—

v—

 

<2 .5

O>

<2

m>

G>

<2

m>

O>

32:8 :83.
05 bra—o 9 82.8.6 3053:

2,

 

O>

<2

m>

O>

<2

.5

O>

:88—
2: :o 2.0 05 3 8333.5 28
330:8 £2: 2.. ..o 3053—

N—

 

.<2 .5

D>

<2

m>

O>

<2

m>

O>

$25.3 .92050:

.3323 .2233:
£03888 .85: rag 8328..."
anzoafifi .3 on: 2:.

:

 

.<2 .5

O>

<2

.5

O>

<2

m>

O>

E5 coau— goao 5 8 528
some .5 32.82.. cogs—8:5

:o 2533 250:8 2:.

@—

 

m<2 m>

O>

<2

m>

O>

<2

m>

G>

85:35

.8325 05 8: 2335.».
:88— ..o bio—9:8 2:.

 

_<2 .5

O>

<2

.3

O>

<2

m>

O>

28:8 :83.
mo 553.335 30%... 2:.

 

<2 .5

 

 

 

 

 

 

O> <2

 

m>

 

.—

 

m

 

O

 

O>

<2

 

m>

 

m

 

m

 

0

 

O>

35:25
332$ 05 .5: ...—area:
:88. :o b.3056 :o .0>0. 2:.

 

 

  

 

, ...—Elfin: 32.2% 1
, nflfl<>2flm¢3°0 i

2:396 .E.=eU
[m «mgalfi501

 
   

.a_.=u.¢.—1.me_.o<
_ mafifiéoo

 

9.52:5 Emma...— .

 

$553.5
i $25958 i E..—

 

h m

266

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Ago .36. £85.. £28
£2583 3:25... 98 .393.
<2 .5 .. .. O O> <2 .5 .. ... O O> <2 .5 .. .. O O> 52805.0556350 NN
~68 £23..
:28 £02.83 9:080.» can
<2 .5 .. n. .0 O> <2 .5 .. ... O O> <2 .5 .. n. O O> =55...50.805.332.965 .N
:9on .628 ..850
<2 .5 .. u o o> <2 .5 .. m o .5 <2 .5 ._ n. o o> £:.:§.§§._>_eu=oo an
:33: E... .5934 :33m
<2 .5.. .. OC><Z.5....OO><2.5....OO>
<Z.5.. u. OO><Z..>..,..OO><Z.5.....OO>
<Z.5.. n. OO><Z.5....OO><2.5.....OO>
<Z.5.. ... OO><Z.5....OO><2.5....OO>
arch—o wEmb ..wdv
<2 .5 .. h. 0 O> <2 .5 .. n. O O> <2 .5 .. .. O O> Eotowc..oot8..oomaoo..,.. a.
3.3.2:“:
o. SE3. 2.. 3.. 5.55933
25.5.9: :88. o... 5.3
m5. .oEno. a... 55.888...
<2 .5 .. ... 0 O5 <2 .5 .. r. O O> <2 .5 .. h. 0 O> ..ooewovho.o>o.o.:. ...
5:558... coca—5....
<2 .5 .. n. O O> <2 .5 .. ... O O> <2 .5 .. n. O O> 3282.250.o...=85E8.. ...
8.5.59...
<2 .5 .. n. O O> <2 .5 .. n. O O> <2 .5 .. ... O O> .om:.:o=8....oom=o..... c.
9.593% E2 ......

 

 

.= .5. = :82 2..—aim
n 5359500

mEofiam .a.—=50
N Hz<>>mmmDOU

3.5....— :o..o<
— §<5HWKDOU

Hméimhm
Hz<5mm~=~00 .< t...—

 

 

267

 

<2 .5

O>

<2

.5

O>

<2

O>

   

22... 2.2.32
05 00..—EGO Ow uO—OO .«O 0m: OFF Om

 

<2 .5

O>

<2

.5

O>

<2

..>

O>

 

828...... 882.28
.8 82 .8 8.80... 0.0.8.... 2

5.2208 ... 2.22.... 2:. a.

 

<2 .5

<2

.5

O>

<2

.5

O>

82.2.8.8.
0.5.2.0... .2280 .0m

2 88.5. .32. .... a: 2..

  

 

,<2 ..>

O>

<2

.5

O5

<2

..>

O>

8.88.8.8.
8002. 2 32m: .8 0m: 2:. nm

 

 

O>

<2

.5

0>

<2

.5

O>

8.8.50.8.
8000.. 8. ...th200 ...O
mam—5.. 8 XmQZ. .8 0.... 2:. em

 

_.<Z ..>

_<2 ..>

O>

<2

.5

O>

<2

.5

O>

2.20 .8838:
.288... ....8. 88.822...
0.850280 .8 02. 2:.

  

 

”<2 ..>

O>

<2

.5

O>

<2

.5

05

2.20 ......8. 88.8%
608.. .8888... .8828...
28:88.8 8 2880.0

.820... 98:... 02.20.80 em

  

 

 

 

_<2 .5

 

 

 

 

O>

<2

 

.5

 

..

 

...

 

O

 

O>

<2

 

..>

 

.—

 

 

O

 

O>

 

 

 

 

A. 20 ....2 .82....
.880 £0.82». 2880.0
.28 .88. ..020... 0... ... 028:5 mm

 

 

.. .3: ...... .8»...— 89.0w

 

  

a ... ...... ._ ...32 9.2.2.3
m m8<>>mm88

h

...—08.3 .8280
N nay—SSHEDOU

   

.8288.— 88¢.
— g5m95¢0

     

Hy—EDKhm
nay—<5m—WKDOU u< .....—

268

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

m......08... .880. 0... .... 8.2.80.
<2 ..> .. n. O O> <2 ..> .. ... O O> <2 ..> .. ... O O> m:.080=.o_..=....o8..0...r mm
28.88... .880. 0... ..o 9.....80.
<2 ..> .. .. O O> <2 ..> .. n. O O> <2 ..> .. n. O O> w=.0........0...80...>.88..0..h on
. ......08...
8.80 0... .8 9.2.80. 8.0.8.80
<2 ..> .. n. O O> <2 ..> .. .. O O> <2 ..> .. ... O O> ......8.........8..o8=0..... mm
8.8.88...
888 0... ..o 8.80. 8.08.80
<2 ..> .. .. c o> <2 ..> .. . o o> <2 ..> .. b. o o> 58.....83oaifi ...
8: ...—.0.).
<2..>.....OO><2..>....OO><2..>...,.OO>
<2..>....OO><2..>....OO><Z..>....OO>
<2..>....OO><2..>....OO><2..>.....OO>
<2..>....OO><2..>....OO><Z..>....OO>
<2..>....OO><Z..>....OO><2..>....OO>
8.8.. 8.0098
8 8.80... 0808388
<2 ..> .. n. O O> <2 ..> .. .. O O> <2 ..> .. n. O O> o.m=00...o8=0..... mm
...0. ..o b....8.80. 0... 00.880
<2 ..> .. n. O O> <2 ..> .. .. O O> <2 ..> .. .. O O> o...0........x0..88=0..... mm
...0. ..o b.....8..0. 0...
<2 ..> .. n. O O> <2 ..> .. n. O O> <2 ..> .. n. O O> 00:8808.80N.m.x0..o8=0..... .m

 

=w.n0n— ...... .8»...— 80.0w

 

 

m ”my—5.352300

:. ...... .. ...... 8.2%

8.808% ......80
N H¢<>>HmMDOU

.8288.— 8..0<
— nay—«$352300

amp—Uimhw
m¢<>>mm¢DOU n< ......—

 

 

269

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

<Z.2>.2200><Z2>2200><Z2>2200>
<Z2>2200><Z2>2200><22>2200>
<Z2>.2200><Z2>2200><Z2>22DO>
<22>.2200><22>2200><Z2>2200>
<Z2>.2200><22>2200><Z2>2200>
26.282222
£222... 228 923. 222 :o coucuza
<2 .2> .2 ,2 o o> <2 .2> .2 .2 o o> <2 .2> .2 2 o o> besiomae 5:88.22 9.
3.8.80 20 82622.0.
32.532 222 :o c2253..
<2 ..> .2 2 o o> <2 ..> .2 2 o o> <2 .2> .2 2 o o> 35.2.888 8832.5 a

 

52:523. .3531— 9.2530

 

2: ...... ._ ...:2 2..—22m

mEofixw 29.2.50

—a==0~°n— =¢=U<

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

m 22~2<>>2m~5OU N 22252529500 22¢<32m~5OU 2.222.260 20mg: "m ......—
<Z .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O O>
<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O O>
<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O O>
<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 G O>
<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O O>
<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O O>
2.80 .m2ooto 5222.282
$5232.23: £222.55
<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O O> 38: 98.2.20 382m 2o 3: 22.2. mm
on: «:32

 

 

:. ...... __ .282 2.22%
m 22352958

mESmmm .a.—2:90
N 22255225500

25253.2 =c_.u< ‘
2 ”223529590

2223202222
nay—55292300 n< ta.—

 

 

270

 

32:2 .8 2822222222 2222a 3.23:3

 

<2 .2> .2 2 O O> <2 .2> .2 2 D O> <2 .2> .2 2 O O> 222.22.22.23 :32 82.22.2825 2.2.
2 2.2.2128...
38392.2 .5 92.28 922.2282 222
8222.: :23 2222 22.2228 225 88.2
<2 .2> .2 2 .O O> <2 .2> .2 2 O O> <2 .2> .2 2 O .2> 25.622 2.2 :88 m8a2=E22m we

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

......m 32.2.2289... 2.. 22.82.25.235

 

_<Z .2> O> <2 .2> O O>

2
2
O
D-
2
O

O> <2 .2>

G-
Ll.

 

00>

2
2
O
D-
2
0

<2 .2> o> <2 .2> 22> <2 .2>

O-
L!-

 

.22522 05 2..
220200.226 222222? 2282.23.22 .822:

 

._ <2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O O> 2.2 35:52.62 mo282m=oEon2 we
_ 222.232 05 2.. 2288228 2002229
2202282 022322222 2.. 82.62222

<2 .2> .2 ,2 o o> <2 .2> .2 ..2 o o> <2 .2> .2 .2 o .2> 2°2222=2o22§§§2§2 3.

 

.20an— 222 82.2
2228.226 ...—2.2222228 8032.2
“<2 .2> .2 .2 o o> <2 .2> .2 .2 o o> <2 .2> .2 .2 o o> 2.2228280520582256 9

 

2222.282 0.2 8 2222222322
2252 .822222222 62.28223

<2 .2> .2 2 0 22> <2 .2> .2 2 22 22> <2 .2> .2 ,2 o o> 82.3.8535.ng a.

 

:88.
_ 222 23222 .222 2c .28 9 03¢
2 <2 .2> n2 ...— O C> <Z .2> .— 2 O .U> <2 .2> .2 n— O 20> on 2:3 2022.250— 05 3:3 m025m 2v ,
2 82232.20 9:359:30 2: 2.. 222.2234 0.2. .a.—242.48..—

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

_= 2:... 2_ .2232 9222.253 nEu2mwm 2.2...ch 22.22.202.22 E2229<
m Hm<3mm~500 N 2.22352522200‘ 2 , may—«$929500 2.2222550 2259:, 222 2.2.2.2

271

 

322222222 .8 3222.2 22222 9.28228

<2 .2> .2 2 o o> <2 .2> .2 .2 o 22> <2 .5 .2 .2 o o> 2o8_.2§58§s§2.2 R
mucus—233m 32250 222.3

<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O O> 02.22222» 22252322220259.2882 mm
2225222922 22832 222.225

<2 .2> .2 .2 .o o> <2 .2> .2 .2 o o> <2 .2> .2 .2 o 22> 8.28:8...032258maae2 2m
,_ . 2222.234 22 8 22.222.529.22: .392 «222.2208...—
0 o>
c o>

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

2
2
0

_<2 .2>
<2 .2> .2

0> <2 .2>
O> <2 .2>

O-
2
O

O> <2 .2>
O> <2 .2>

D-
‘1.

 

2
O
2
2
CD
2
2

 

2222262U «:22.» .222:
232 222 2c 352882.202 222
Eno— 2 22830022 20.222.22.226
322209.02 322223.222 $222892:
_ 2.. 9.82.2 2.. 225 2.22.2. 5.22
<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O 0> .a.—22.332.22.82 83.25% cm
822322222 2.2 £25.23 26:
9222.232 :2 8822222 :3 8:282
222 2222 2022823222 2.. 92220282:
<2 .2> .2 ..2 o o> <2 .2> .2 .2 o o> <2 .2> .2 2 o o> 2.2.2.223223822222m a.
. 83.22822
.8 .2822... 288.202 mac—5:
_ 22:32 222 :2 329222 0.2
_ :23 2922 8.2228 2252 8.222922%
<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O O> .m222xm2o2282mo2a222822m av
£226 82.22.229.92.— .29 =33— meta—2.22.25

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  
  
 

I 12.2253M. 52.2.3. ,1
\ 222.233.2228

 
   

2.22.528 2.2mm”: 22:..2

272

 

 

 

 

 

 

<2

.2>

O>

<2

.2>

O>

<2

.2>

22>

22022822 35222202222
:22? 2.22.222 222 2.2.2 2x22222222
£22m 20.20.22 32222220222022
222222 :0 22.222.222.222: 92282.2

2m

 

<2

.2>

22>

<2

.2>

O>

<2

.2>

O>

.8232... 2.2 £22223
222 8 22.22.2202 8.222222222922202
222222 20228 $222832:

20 922.022.2222 2.2 32232222 .8
«2.2222223 2222322 2220.222 220an—
o._ .22 22.222.222.225 £22822

om

 

<2

.2>

O>

<2

.2>

O>

<2

.2>

22>

.2>—cm .22 2.2an.
222 2.2.2 mace—2.22.2 2o>o= 32282.2

mm

 

<2

.2>

 

 

.2

 

2

 

O

 

22>

<2

 

.2>

 

 

.2

 

2

 

O

 

O>

<2

 

.2>

 

 

 

 

O

 

22>

2.2822

.2222 222.222 2222232203 802m :2
802.2922 0222 22.2.23: .22 282222222
22 22.222825 2.2.2.222 $222222 2.2
328% ..wdv 2:02.282 .2>—om o2
228323.222 2.2 323222.26 .8222:
2.2 8: .2222 95.22% 222.22 8.222800

 

 

vn

 

222—.2312 22 .2. 22.222.222.282: 392 “2222:8922

 

 

m 22522552300

:2 2:... 2_ 2282 222.22%

__

...—22.23% 22.22280
N 2223522922200

2.222.292.22 22.222u<
2 223522922200

_ PZHHZOU ZCmmHA 2: 2.2.2.2

 

 

273

 

 

 

 

 

 

 

 

 

<2 .52 2 OC><2 2>2200><22>2200>
<22>.2 2 OO><2 2>2200><22>2200>
<2.2>.2 2 OO><2 2>2200><22>2200>
<22>.2 2 OO><2 .2>.2200><2.2>.2200>
822.252.22.202
9.222.232 05 22.2 22232282
82232.2 225 2225... .2222. 22.
<2 .2> .2 2 22 22> <2 .2> .2 2 22 22> <2 .2> .2 2 22 22> 82203222058205.2382 8
02222222222 2.2 2.22.6223
222 .22 2222222 30220222292228
225 2.22.8 22222.22.22:
.6 $520222: 22222222222223
2.. 02222222222 2.2 2.2222222 22222622
<2 .2> .2 2 O O> <2 .2> .2 2 2.2 O> <2 .2> .2 2 O O> 02222883580222.2232285 3
802.2922 22282 2 22.22.2228
2022: 22222.2 .20an- 222 .2222
<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O O> 9.22222222253282282222222 8
2:22:28 22.28.22 222 2222222
.22 .2222 2.2 3222222228 22222
222 323.222.... :2 225.22 02.
<2 .2> .2 2 22 o> <2 .2> .2 2 22 22> <2 .2> .2 2 22 22> .2222:2 282222.222 82232.2 2.2
25222.20 2.2882 282.288 22.2.2222
. o2 2.2—Em 922222222 222 222232 0222
<2 .2> .2 2 22 22> <2 .2> .2 2 22 22> <2 .2> .2 2 22 22> 22.22 28283282822322 222

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

352222.29 222.2234 2.2—22.5.2.2

 

2: 2:: 22 .2822 22:22.22
32232292222222 .

2:92.23 2.2.2.520
. .. 22.222232222222522

—I==0—GA— =639<
- 2 2.25.32.22.58

- 222222.222 282222 .22 22.2 x

274

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

<Z2>.2 2 OO><2 2>2200><22>22GO>
<2 .52 2 OO><2 2>2200><22>2200>
<22>2 2 OO><2 2>2200><22>2200>
<Z.2>.2 2 OO><2 2>2200><22>2200>
<2 .52 2 OO><2 2>2200><22>2200>
<2 .52 2 OO><2 2>2200><22>2200>
228.202 .2222 222 2.292222
82.28 2.2 m2=22m 2220292222 2222
922.222.22.202 322222222 .22 .222an—
<Z .2> .2 2 O 22> <2 .2> .2 2 O O> <2 .2> .2 2 O 22> 222 2.2.2 22222222229220 8223.222 2.2
229E8—
0222222222 222 .22 22.2222 302.2220
9222222222 02282.2 .22 2.222.222.2—
<z .2> .2 2 o 22> <2 .2> .2 2 22 22> <2 .2> .2 2 22 22> 2228.2 22225222223222.2922 22
220222220. moment...
o>222=w8 822 2.2 82282.2
.22 222.222.2— 222 2.2.2 3222223292.?
<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O 22> 2.2220223223222522 3
:88. 222
222 22020222228220 222252522 2.2:
32222232 .22 82.: 22:“ 22.23228
<2 .2> .2 2 22 22> <2 .2> .2 2 22 0> <2 .2> .2 2 22 o> 22.2.2.2o220E8_222922< 3
22253 223.2 22.2222 :2
<2 .2> .2 2 0 22> <2 .2> .2 2 O 22> <2 .2> .2 2 O 22> 22.222222222022225220222232222mxm< mo

 

32:22:28.5.— .2222 9:229:22

 

 

_= 2:... 2_ 228:2 222.2222
.2 22223522222228

mESnzw 2.222250
N 223522922200

22.22.292.22 22.2223.
2 222352922290

— Elm—2.260 ZOmmHA "m 2.2.2.2

 

 

275

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O O>
<2 .2> .2 2 O 22> <2 .2> .2 2 22 22> <2 .2> .2 2 O O>
<2 .2> .2 2 O 22> <2 .2> .2 2 22 22> <2 .2> .2 2 22 22>
<2 .2> .2 2 O 22> <2 .2> .2 2 O 22> <2 .2> .2 2 O O>
<2 .2> .2 2 O m2> <2 .2> .2 2 O O> <2 .2> .2 2 O 22>
<2 .2> .2 2 .0 22> <2 .2> .2 2 O O> <2 .2> .2 2 O O>
<2 .2> .2 2 O O> <2 .2> .2 2 O 22> <2 .2> .2 2 O 22>
2:222 .20 82222222202202 2.2.2.2232
222 .20 92222222 2222.2 22222222822 2.2
<2 .2> .2 2 O O> <2 .2> .2 2 O 22> <2 .2> .2 2 O 22> 82322 .2222 22.2 22232232 8223.22 Mb
222.23. .2222 2.2 02222222 2222.22.22.23
2.2 22825 2.22.2228 2.2228202
<2 .2> .2 2 0 O> <2 .2> .2 2 O 22> <2 .2> .2 2 O 22> 2o 8.222222222222228 28.2222 macaw N2.
222222282222 222203.222 .22 22.22228
<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O 22> 2.2.2an220 22:22:23.5 22.222.222.20 22.
228223.222
222203.222 2 22.222.22.22 P222232
<2 .2> .2 2 O 22> <2 .2> .2 2 O O> <2 .2> .2 2 O 22> 2.2 922228228 .2222 2:222:00 on
3:: 222222 22.28228
.20 22.222.222.292 2.2 om: 22.2.2an—
<2 .2> .2 2 O 22> <2 .2> .2 2 0 22> <2 .2> .2 2 O 22> 2.2 2282228228 222 22.222.222.20 20
9.2.2.2 2.2 2.2222222222222222 2.2.2.22
22.222.222.222 2 22.222.23.22 22.20an—
<2 .2> .2 2 O O> <2 .2> .2 2 O O> <2 .2> .2 2 O 22> 2.2 2282228228 222 222.22.222.20 we

 

2222222282 922222.522

 

 

:2 22:22 : 228—>2 2222a=2m
n 2255292300

2

22222292m .a.—222.20
N 22235229500

222.222.22.22 22.222o<
2 2225529500

E 2.2222200 ZmemJ 22 2.2.2.2

 

 

276

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

<2 .2> .2 2 O> <2 .2> 22> <2 .2> 22>
<2 .2> .2 2 22> <2 .2> 22> <2 .2> O>
22882 .2222 222.223.82.22
82.222288 2.2 but? 2 22222222222
<2 .2> .2 2 22> <2 .2> 22> <2 .2> O> 23.22.62 2208.22.20 om: 2:. 2222
220222222 m=222m 222 822822
.22 22222232 .2222 2.2.2 2222222222223
<2 .2> .2 2 22> <2 .2> O> <2 .2> 22> 2228222222222 8222.622 22.
2922:2222. 222:2 22.222.29.32 «2222222222222
<2 .2> .2 2 22> <2 .2> 22> <2 .2> 22>
228.22
.2222 222 22292222 3222? 28292222
<2 .2> .2 2 O> <2 .2> 22> <2 .2> O> :22 mc2m==o222282222m2moh 222
222.2022
222222222 2.2 23.222 2.2 22022222,
<2 .2> .2 2 O> <2 .2> 22> <2 .2> 22> 2292328 22.222222222222282. 2.2.
252222.22? 222.22.222.22
2.2 282 22 .22 22.22222 .20 82.2222
<2 .2> .2 2 O> <2 .2> 22> <2 .2> 22> was—22E 22.222228292222282 02.
82.22285 2.2 222.222.2222?
3022 .22 3.22.2 2228 9.28228
<2 .2> .2 2 22> <2 .2> 22> <2 .2> 22> 25.2.2.2.w 820582222282. 22
222.23 223.2
2.222 2.2 .2222 :2 22.222.222.825 22822.2.
<2 .2> .2 2 O> <2 .2> O> <2 .2> 22> 92222282 22.2 2022822222282. 2.2.

 

 

coca—2.2.2.2292 n=2nnenn<

 

 

277

 

 

 

 

 

+2N 862 3.22 22.6 n - 2

2223.2 222 325220.228 922220.82 2.2.2.2 2.2 ems: 0222 023222222 832.2 .2382: .22—228.2 a .2222 22.2.2 .22 .2.»

222222220522 922230222228 222222.288 222 822222.226 2.2.2.2223 223m
2222222022622 0222382228 2022.22.22.20 222 822.20.22.22 .2223 23m
22382228 222 2.2 222260 2222 222222.222 222 222 82222228 2.2.2.2223 228m
ll. 222382.28 0.22 2.2 220.2260 3.8 22202228 0222 :2 82222228 .2223 23m
222222222622 823822.222 2222.25.20 222 82220226 2.2.2.2223 2222.28 2
222.282.220.622 2.2382223 22.222223 222 82222228 .2223 .22—228 2
222382.28 0.22 22.2 220.2260 no.8 222222.20 0.22 222 9222.20.22.» 2.2.2.2223 .22—2222 2
222382.23 222 2.2 222260 no.3 25222.20 05 :2 9.22.20.22.22 22223 .22—.28 2

 

 

 

 

202222228222 252.229.202.222 .3

I- +222. 3 - on mm - 0N 2022:: 22:22 mN
2.2.28.2 :2 0928 ow< .N22

2an 2 02.22 220222200 .22

228222.22 2.2.2.222 22.222.252.222: 323.2228 222 22.2.2.2... 832.2 222.22.22.22:

22.222.52.222: 29.22.29 I... 0 2.2.2.2

278

 

2 03382on
.3222 322.22 0222 :2 mEEmca 222382.28 2222.22.23 222 2.2.2.222 9:222 .2922 22.2.2 3:222:28 2222.2 .228 2223 82202.2

22.20.225.20 2.2.2.2522 .3

.22 22> 2.2222288 22 223.2 .292 .222 .2.»

 

22.22226 2225.22.22 22222222222
223222.28 222222.228

222222 225.228

2.28.2.2 22822282 222.62.22.22 .223 2.2.2.22 2222222022222 5.22.22.25.22
2+6 .2832 628.2 .2262 2222222225 2222225222922

2228222 .8222 222.22 .23 22.2.2.2 2.222.522.2228

2.3228 mm<22<2<n

2.228 2222222322222

2.23228 22222222092292.2222

 

 

 

 

 

 

 

 

22.2.2.2 2.2222 2222 2.22220 832.2 2.22223 3220222228 222222 :92 o>a22 mega—2.2.2 222 2.2 22.22223 .3

 

 

 

 

 

+222 2 - 2. c - v m - 2 22
2229202032 262.2 25.2 2228.23.22 222.238.222.222 2.2 20.28:: 0222222232222 0222 028222222 832.2 .3

279

$.53 35 E 5::—.925!— ega 5:82—93 ...-9» ..8 :2...— Eu.» :95 3:2:

 

m Essufisoo

 

N 083280

APPENDIX D

APPENDIX D

Letter requesting Dr. Thomas Adams’ permission to conduct the
study using his courseware programs and Physiology 431 Class

September 20, 1995

Dr. Thomas Adams
Department of Physiology
Michigan State University
East Lansing, MI. 48824

Dear Dr. Adams

Letter Requesting Your Permission to Use Your Class for Doctoral Study

I am doing a study for my doctoral dissertation about evaluating the success of computer
courseware in a university setting. I am aware of the several CAI programs you’ve
developed for your classes over the years. I would like, therefore, to request your
permission to do my study on the use of the programs in your Physiology class 431
during the Fall semester, 1995.

My study will focus on determining factors that influence the success of computer
courseware programs when students use such programs as supplement to classroom
instruction. The study will involve collection of three types of data. One form of data
will be students’ perceptions about the usefulness and success of using the programs.
The second data will be collected from you as the class instructor and will relate to
records of students’ academic achievement due to the use of the programs. The third
data type will be experts’ review of the programs using evaluation criteria based on
research findings on effective teaching and effective leaming.

This study is strictly formative evaluation in nature with the sole purpose of collecting
data that will better inform computer courseware authors on factors to consider in
developing successful programs. My central interest is to find out what works and why.
In doing so, data collected from subjects involved in the study will be kept strictly
confidential by not associating subjects with specific responses. Subjects will not be
required to indicate their

280

28 1
APPENDIX D

names on the questionnaire-type checklists that will be used to review the courseware
programs.

In my analysis, I may include excerpts from open-ended responses from subjects
involved in the study in presenting the report of this study. I will use pseudonyms and
disguise any personal identifiers when writing the report and in any publications.

Thank you in advance for your time.
Sincerely

James Obielodan
Computer Laboratory
Michigan State University
East Lansing, MI. 48824

 

Consent Form

I, Dr.

 

(Name)
have read and understood the attached letter informing me of the requirements of the
study and of my rights if I choose to participate.

I to participate in the study.
(volunteer/decline)

 

 

(signature) (date)

APPENDIX E

APPENDIX E

Letter requesting students in the Physiology 431 class
to participate in the study

November 1, 1995

Dear Mr./Ms.

 

Letter Requesting Your Participation In My Doctoral Study

I am doing a study for my doctoral dissertation about evaluating the success of computer
courseware in a university setting. I will like to request your participation in the study
which will evaluate three courseware programs used in your Physiology 431 class.

If you choose to participate, the study will require you to interact with the programs and
fill out a questionnaire on each program. Also, if you indicate your availability for a
follow-up on the attached consent form, you may be requested to participate in a focus-
group discussion on the programs. Your participation in this study is expected to take a
total of about 2 - 3 hours.

Students’ course grades will be collected from your class instructor only for the purpose
of correlating academic performance with the use of the programs. The instructor will be
required to provide the students’ course grades to the researcher using only students’
identification numbers. The instructor will not have access to the questionnaire that
students fill out and therefore will not be able to associate any student’s name or
identification number with any specific response.

You may decline to answer or discuss any question on the questionnaire. You may decide
to withdraw your participation at any time during this study.

This study is strictly formative evaluation in nature with the sole purpose of collecting
data that will better inform computer courseware authors on factors to consider in
developing successful programs. My central interest is to find out what works and why.
In doing so, data collected from subjects involved in the study will be kept strictly
confidential by not associating subjects with specific responses. Subjects will not be
required to indicate their

282

283

APPENDIX E

names on the questionnaire-type checklists that will be used to review the courseware
programs.

In my analysis and presentation of the report of this study, I may include excerpts from

your response to open-ended questions on the research questionnaire. In doing so, I will
use pseudonyms and disguise any personal identifiers when writing the report and in any
publications.

Attached is a copy of the questionnaire that you and other students will use in evaluating
the courseware programs. In case you have any question regarding this study please call
me at 355-4500 ext. 137 or email me at 13501 jbo@msu.edu.

Kindly complete and sign the attached consent form and the questionnaire and then mail
the two items in the self-addressed stamped envelope provided.

Thank you very much for your time and cooperation.
Sincerely,

James Obielodan

Computer Laboratory

Michigan State University
East Lansing, MI. 48824

284

APPENDIX E

Consent Form

1, Mr. Ms.

 

(N me)

have read and understood the attached letter informing me of the requirements of the
study and of my rights if I choose to participate.

 

 

I to participate in the study.
(volunteer/decline)
I be available to participate in a follow-up group discussion.
(will/will not)

If you can help participate in a follow-up group discussion on the computer courseware
programs, please provide the following information:

Name

 

Address:

 

 

 

Phone:

 

 

 

(signature) (date)

285

APPENDIX E

Questionnaire to Evaluate the

Computer-Aided Instructional Programs
Used in PSL 431, Fall Semester 1995

Attached is a copy of the questionnaire to evaluate the computer-aided instructional
programs used in your PSL 431 class this Fall semester. Kindly complete the
questionnaire and the accompanying consent form and then mail the two items in the self-
addressed stamped envelope provided.

If you have any question regarding this study, please contact:

James Obielodan

Michigan State University

Faculty Facility for Creative Computing
Computer Laboratory

Room 114 Computer Center

East Lansing, MI. 48824

Phone —> 517-355-4500 ext. 137
E-mail —> 13501 jbo@msu.edu

Thank you very much for your time and cooperation.

APPENDDK F

APPENDIX F

Letter, email or phone messages to experts
requesting them to participate in the study

Dear

I would like to request you to participate in my dissertation study which
will involve experts like you to respond to a questionnaire on evaluating
computer courseware. The evaluation instrument constructed for that
purpose will require you to rate the level of importance of each
questionnaire item to the success or effectiveness of courseware. The
questionnaire will take about l’lz hours to complete.

Please let me know of your availability to participate in this study. You
can reach me by phone at 517-355-4500 ext. 137 or by email at

13501 jbo@msu.edu.

Your cooperation and support will be deeply appreciated.

Sincerely

 

James Obielodan

Faculty Facility for Creative Computing
Room 114

Computer Laboratory

Michigan State University

East Lansing, MI 48824

286

287

APPENDIX F

Dear

I would like to request your participation in my dissertation study which will involve
experts like you to interact with three computer courseware programs and then respond
to a questionaire to evaluate the programs. The process is estimated to take about 2
hours.

Please let me know of your availability to participate in this study. You
can reach me by phone at 517-355-4500 ext. 137 or by email at

13501 jbo@msu.edu.

Your cooperation and support will be deeply appreciated.

Sincerely

 

James Obielodan

Faculty Facility for Creative Computing
Room 114

Computer Laboratory

Michigan State University

East Lansing, MI 48824

288

APPENDIX F

November 9, 1995

Address

 

 

 

Dear __

I am doing a study for my doctoral dissertation about evaluating the success of computer
courseware in a university setting. I will like to request you — as an expert courseware
developer, content expert, researcher, or university faculty — to participate in the study
which will evaluate three courseware programs used in the Physiology 431 class during
this Fall semester. Your participation in this study will greatly help me in completing my
doctoral dissertation in Educational Systems Development at the College of Education
here at Michigan State University.

The study will require you to interact with the programs and to fill out a questionnaire on
each program. I will provide the questionnaire instrument when you and the other
experts convene for the evaluation. The second phase of this evaluation process involves
a focus group discussion among the experts participating in this study. 'The meeting will
be held on November 17 at 3:15 PM in Room 500, Computer Center. Your participation
in the group discussion is crucial to the success of the study. In preparation for the
focus group discussion, please respond, as best as you can, to the questions in Part C of
the evaluation instrument. Your participation in this study is expected to take about 3 - 4
hours.

I will not use your real name in any conversation, analysis, and reports of the study. I
may include excerpts from your response to open-ended questions on the research
questionnaire. In doing so, I will use pseudonyms and disguise any personal identifiers
when writing the report and in any publications.

289

APPENDIX F

You may decline to answer or discuss any question on the questionnaire. Also, you may
decide to withdraw your participation at any time during this study.

Kindly complete and sign the attached consent form and mail in the self-addressed
stamped envelope provided.

In case you have any question regarding this study please call me at 355-4500 ext. 137 or
send me an email at 13501 jbo@msu.edu.

Thank you in advance for your time and cooperation.

Sincerely,

James Obielodan

Faculty Facility for Creative Computing
Room 114

Computer Laboratory

Michigan State University

East Lansing, MI. 48824

Phone —> 355-4500 ext. 137
Email —> 13501jbo@msu.edu

290

APPENDIX F

Consent Form

1, Mr. Ms.

 

(Name)
have read and understood the attached letter informing me of the requirements of the
study and of my rights if I choose to participate.

I to participate in the study.
(volunteer/decline)

 

 

(signature) (date)

APPENDIX G

APPENDIX C

Cover letters that accompanied the evaluation instruments
for the experts

January 3, 1996

Address

 

 

 

Dear

 

Enclosed with this letter is an evaluation instrument related to my dissertation study. As
previously requested or discussed with you, I need your help in responding to the
questionnaire.

The Computer Courseware Evaluation Checklist Rating instrument is designed to allow
you to rate the importance of each questionnaire item in determining the success or
effectiveness of courseware. Instruction on how to complete the questionnaire
instrument is included. The questionnaire will take about 2 hours to complete.

Please return the completed evaluation questionnaire in the self-addressed stamped
envelope provided or you can call me to pick it up.

In case you have any question regarding the instrument please call me at 355-4500 ext.
137 or send me an email at 13501jbo@msu.edu.

Your immediate response to this urgent matter will be deeply appreciated. Thank
you very much for your time and support.

Sincerely,

James Obielodan
Room 114, Computer Center, MSU
East Lansing, MI. 48824

291

292
APPENDIX C

January 8, 1996

Address

 

 

 

Dear

 

Enclosed with this letter is an evaluation instrument and three courseware programs
related to my dissertation study. As previously requested, I need your help as an expert
to interact with them.

The Computer Courseware Evaluation Checklist instrument is designed to allow you to
evaluate the quality of the characteristics of computer courseware programs in promoting
effective learning. The three computer courseware programs to be evaluated are: Action
Potential, Control Systems, and Synaptic Mechanisms (Parts I and II). The programs are
available on PC-based Microcomputer Labs around MSU Campus. If you own a PC and
prefer to run the programs on the system, a set of four disks containing the programs are
enclosed with this letter. Your task is to interact with the programs and then use the
checklist to evaluate them.

Please return the completed evaluation questionnaire in the self-addressed stamped
envelope provided or you can call me to pick it up.

In case you have any question regarding the instrument please call me at 355-4500 ext.
137 or send me an email at 13501 jbo@msu.edu.

Thank you very much for your time and support.

Sincerely,

James Obielodan

Room 114

Computer Center
Michigan State University
East Lansing, MI. 48824

APPENDIX H

294

APPENDIX H

F ollow-up messages to the experts

The second email message

Dear Expert/Research Volunteer

This is just a reminder concerning the evaluation instrument for my
dissertation study that I sent you about three weeks ago. It is critical for
the study to complete all data collection latest this week Friday.

I will be very grateful for your help in completing and mailing the
questionnaire back to me. You can also send an email or call me to pick it
up when completed. Please ignore this message if you have already mailed
or sent the questionnaire.

Again, thank you very much for your support and help.

 

James Obielodan

Faculty Facility for Creative Computing
Room 114

Computer Center

Michigan State Unviversity

East Lansing, MI 48824

Phone -->517-355-4500 ext. 137
Email «>13501jbo@msu.edu

APPENDIX I

APPENDIX I

MDS coordinates of the experts’ perceptions regarding the level of
importance of the characteristics of computer courseware

in promoting effective learning

1 CCLAOBJ 1.6465 -.3 342 .6808
2 CINTAUD 1.3269 .0965 -1.3391
3 CUSEHOW 1.2328 .7025 .3790
4 CLBWREV .0928 -1.1796 1.4553
5 COBJCON 1.3415 1.1065 .1197
6 CRCCONT -1 .4194 -1.7009 -.5380
7 CLEVDIF -.2764 -.3104 -2.4139
8 CLOGORG .8879 1.3531 .4862
9 CCOMPLX -1.5071 -.3997 -1.0686
10 CICHUNK .6437 1.2353 .4594
11 CISTRAT 1.3182 -.3155 .1575
12 CREVCON -.1687 -1.5557 1.2610
13 CREVQUE .4502 -1.3600 1.3757
14 CTESTOB .4917 -1.4966 .8314
15 CF EEDBK .8413 -.9349 1.2850
16 CHLINKS -l .1307 1.2676 -.6895
17 CLCONTR 1.0806 .5724 -1.3252
18 CINTERA .7302 1.5951 .1730
19 CCERROR -.6925 1.5385 -.9163
20 CVISUAL -.8170 1.0523 -1.21 16
21 CSSCRNL .9674 1.0246 .0659
22 CCSCRNL .1264 . 1.6149 .7489
23 CBSCRNL -1.8125 -.1203 -.1967
24 CCSCRNB -.6937 .5157 1.461 1
25 CNAVIGT .0339 .3253 1.2961
26 CTABCON -1.8281 -.2189 1.4752
27 CMENU -.8363 .4599 1.4750
28 CHELP -.6968 -.5803 1.8549
29 CGLOSSA -1.5631 -.7621 1.2417
30 CTXTCOL -2.0545 .5234 -.25 88
31 CTXTSIZ -1.5115 -.0924 1.1164
32 CTXTSTY -.8250 1.4088 -.3702
33 CICONS -1.6540 1.3850 .1139
34 CGRAPH -.1379 1.3598 .3237
35 CANIMAT -l.l491 1.0107 .5723
36 CVIDEO -1.3276 1.5151 .2351
37 CAUDIO -1.6352 1.091 1 .4140

295

38
39
4o
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63

65
66
67
68
69
70
71
72
73
74
75
76

CSEFECT
CGASITU

CGATASK

CILFACT
CILCONC
CILPROB
CILBEHA
CILSKIL
CSRFACT
CSRCONC
CSRSKIL
CSRBEHA
CSRESUB
CNIACCU
CNISIMP
CNIVARY
CNICOGS
CNIPROB
CNIBEHA
CNISKIL
CLGFACT
CLGCONC
CLGPROB
CLGBEHA
CLGSKIL
CEPF ACT
CEPCONC
CEPPROB
CEPBEI—IA
CEPSKIL
CPF F ACT
CPFCONC
CPFPROB
CPFORIG
CPF CONS
CPFSKIL
CAPF ACT
CAPCONC
CAPPSIM

296

APPENDIX I

-1.9000
.6765
1.2981
1.0184
.9459
1.0029
-.4398
-.8942
.7834
.9844
.5322
-.3530
-. 1421
1.5509
1.0715
1.2501
-.7431
-1.0786
.5332
.8966
.9587
.2257
.3066
-.6662
1 . 1380
-1.8152
-2.0748
.9467
.31 10
.3373
1.4532
1.4533
1.3875
-.8810
1.0573
.4941
-1.9095
.5250
.4038

-.6999
1.0266
-.6194
-1.1834
-.9826
-.O951
-1.5684
-2.2072
-.7673
-.6723
-.5859
-1.1734
-1.4863
.9165
.6949
.6445
-.0203
1.4043
-.6375
-1.1271
1.0801
-1.1950
-.4247
-.5749
.5732
.8352
-.3512
.8000
.8264
-.6923
.4645
.4645
.0594
.1910
.0020
-.2964
-.6304
-.5132
-.4756

.4399
-.3286
1.0210

.0210

.2365
-.0584

-1.4016
-1.0046
-.2410
-.7580
-.8605
-1.2635
-.6062

.2595

.4390
-.4991

-1.0758
-.l946
-1.4482
-.8145
-.0452
.1274
-l .4218
-1.3821

.5091

.51 12
—.9615
-.6251
-.4555

-1.0697
-.1356
-.1356

.2951

-l.4098

-1.0972

-1 . 1612
1.3790
1.0296
1.0640

77
78
79
80

CAPNOVP
CAPSKIL
CERSKIL
CERSREV

297

APPENDIX I

-.2794
-.3706
-.0017

.5328

-.2684
-1.9855
-.6537
-.6971

1.7290
-.5392
1.3731
1.0382

LIST OF REFERENCES

LIST OF REFERENCES

Alkin, M. and F itz-Gibbon, C. (1975). Methods and theories of evaluation programs.
Journal of Research and Development in Education, 8(3), 2-15.

Alkin, M. C. (1969). Evaluation Theory Development. Evaluation Comment. Vol. 2, 2-
7. °

Alkin, M. C. and Ellett Jr., F. S. (1994). Development of Evaluation Models. In Husen,
T. and Postlethwaite, T. N. (eds.) (1994). The International Encyclopedia of
Education 2nd ed. Oxford, England: Pergamon.

Aronson D.T. and Briggs, L. J. (1983). Contributions of Gagné and Briggs to a
prescriptive model of instruction. In Reigeluth, C. (ed.). Instructional-design
theories and models: An overview of their current status. Hillsdale, New Jersey:
Lawrence Erlbaum Associates, Publishers.

Ary D., Jacobs, L. C., and Razavieh, A. (1990). Introduction to Research in Education.
Fort Worth: Holt, Rinehart and Winston, Inc.

Askar, P. Yavuz, H. and Kbksal, M. (1992). Students’ perceptions of computer assisted
instruction environment and their attitudes towards computer assisted learning.
Educational Research 34, 2, 133-139.

Babbie, E. R. (1973). Survey Research Methods. Belmont, California: Wadsworth
Publishing Company, Inc.

Babbie, E. R. (1979). The Practice of Social Research. Belmont, California: Wadsworth
Publishing Company, Inc.

Beeby, C. E. (1977). The meaning of evaluation. In Current Issues in Education. No. 4:
Evaluation. Department of Education, Wellington. pp. 68-78.

Boettcher, J. V. (1993). 101 Success Stories of Information Technology in Higher
Education: The Joe Wyatt Challenge. McGraw-Hill, Inc.

Borg, W. R. and Gall, M. D. (1983). Educational Research: An Introduction 4th ed. New
York: Longmans, Inc.

298

299

Borsook, T. K. and Higginbotham-Wheat, N. (1992). The Psychology of Hypermedia:
A Conceptual Framework for R & D. In Simonson, M and Jurasek, K.(eds.), 14th
Annual Proceedings of Selected Research and Development Presentations at the
1992 Convention of the Association for Educational Communications and
Technology. Sponsored by the Research and Theory Division in Washington,
D.C.: AECT.

Bosco, J. (1986). An analysis of evaluations of interactive video. Educational
Technology, 26(5), 7-17.

Briggs L. J., Gustafson, K. L., and Tillman, M. H. (eds.) (1991). Instructional Design
Principles and Applications: Second Edition. Englewood Clifi‘s, New Jersey:
Educational Technology Publications.

Brinkerhoff, R. O. et al., (1983). Program Evaluation: A Practitioners Guide for Trainers
and Educators. Boston, Massachusetts: Kluwer-Nijhoff Publishing.

Carney, T. F . (1972). Content Analysis: A Technique for Systematic Inference From
Communications. London: B. T. Batsford Ltd.

Clark, D. J. (1984). How do interactive videodiscs rate against other media? Instructional
Innovator, 29(6), 12-16.

Clark, R E. (1983) Reconsidering research on learning from media. Review of
Educational Research, 53 (4): 445-460.

Clark, R. E. (1985a) Confounding in educational computing research. Journal of
Educational Computing Research, 1 (2): 137-148.

Clark, R. E. (1985b). The importance of treatment explication: a reply to J. Kulik, C. L.
Kulik, and R. Bangert-Drowns. Journal of Educational Computing Research, 1
(4): 389-394.

Cohen, J. (1977). Statistical Power Analysis for the Behavioral Sciences. New York:
Academic Press.

Creswell, K. W. (1986). Does Instructional TV Make the Grade? Journal of Educational
Television, Vol. 12, No. 1.

Criswell, E. L. and Swezey, R. W. (1984). Behavioral Learning Theory-Based Computer
Courseware Evaluation. Educational Technology, November, 1984.

 

 

300

Cronin, M. W. and Cronin, K. A. (Spring 1992). Recent Empirical Studies of the
Pedagogical Effects of Interactive Video Instruction in "Soft Skill" Areas. Journal
of Computing in Higher Education. Vol. 3(2), 53-85.

DeBloois, M. (1988). Use and effectiveness of videodisc training: A status report. Falls
Church, VA: Future Systems, Inc.

Dick. W. and Carey, L. M. (1991). Formative Evalutation. In Briggs etal. (eds.)
Instructional Design Principles and Applications: Second Edition. Englewood
Cliffs, New Jersey: Educational Technology Publications.

Dick. W. and Carey, L. M. (1991). Surnmative Evalutation. In Briggs et al. (eds.)
Instructional Design Principles and Applications: Second Edition. Englewood
Cliffs, New Jersey: Educational Technology Publications.

Donthu, N. and Rust, R. T. (1989). Appendix: Review of multidimensional scaling. In
R. Batra and R. G1azer(eds.), Cable TV advertising: In search of the right formula.
New York: Quorum Books.

Driscoll, M. P. (1994). Psychology of Learning for Instruction. Boston: Allyn and
Bacon.

Duffy, T. M. and Jonassen (eds.) (1992). Constructivism and the Technology of
Instruction: A Conversation. Hillsdale, New Jersey: Lawrence Erlbaum
Associates, Publishers.

Duchastel, P. (1983). Text illustrations. Performance and Instruction Journal, 22(4) 3-5.

Eggen, P. D. and Kauchak, D. (1992). Educational Psychology: Classroom Connections.
New York: Macmillan Publishing Company.

Eisner, E. (1979). The Educational Imagination. new York: Macmillan.

F lagg, B. N. (Ed.) (1990). Formative evaluation for educational technologies. Hillsdale,
NJ: Erlbaum. '

Fletcher, J. (1990, July). Effectiveness and cost of interactive videodisc instruction in
defense, training and education (IDA paper P-2372). Alexandria, VA: Institute for
Defense Analyses.

Gagné, R. M. (1985). The conditions of learning (4th ed.). New York: Holt, Rinehart &
Winston.

301

Gagne', R. M., Wager, W. and Rojas, A. (1991). Planning and Authoring Computer-
Assisted Instruction Lessons. In Briggs et al. (eds.) Instructional Design
Principles and Applications: Second Edition. Englewood Cliffs, New Jersey:
Educational Technology Publications.

Gayeski, D. and Williams, D. (1985). Interactive Media. Englewood Cliffs, NJ: Prentice
Hall.

Glaser, R. (1976). Components of a psychology of instruction: Toward a science of
design. Review of Educational Research, 46, 1-24.

Golas, K. (1982). The effectiveness and cost of alternate models of formative evaluation
for printed instructional materials (Doctoral dissertation, Florida State
University). Dissertation Abstracts International, 43, 2873A.

Gold, P. (1989). A powerful solution meets an overwhelming problem. Instructional
Delivery Systems, 3(5), 6-7.

Good, T. L. and Brophy, J. (1995). Contemporary Educational Psychology. New York:
Longrnan Publishers USA.

Hannafin, M. J. (1985). Empirical issues in the study of computer-assisted interactive
video. Educational Communication and Technology Journal, 33 (4), 235-247.

Hannum, W. (1988) Designing courseware to fit subject matter structure. In Jonassen,
1). (ed.). Instructional Designs for Microcomputer Courseware. Hillsdale, New
Jersey: Lawrence Erlbaum Associates, Publishers.

Hartley, J. (1981). Eighty ways of improving instructional text. IEEE Transactions on
Professional Communications, PC 24, 1, 17-27..

Heinich, R., Molenda, M., Russell, J. D., and Smaldino, S. E. (1996). Instructional and
Technologies for Learning 5th Edition. Englewood Cliffs, New Jersey: Prentice-
Hall, Inc.

Holsti, O. R. (1969). Content Analysis for the Social Sciences and Humanities. Reading,
Massachusetts: Addison-Wesley Publishing Company.

Horn, R. E. (1989). Mapping Hypertext: The Analysis, Organization, and Display of
Knowledge for the Next Generation of On-Line Text and Graphics. Lexington,
MA.: The Lexington Institute.

 

302

Hooper, S. & Hannafin, M. (1991). Psychological Perspectives on Emerging Instructional
Technologies: A Critical Analysis. Educational Psychologist, 26(1), 69-95.

Jonassen, D. H. and Wang, S. (1993). Acquiring Structural Knowledge from Semantically
Structured Hypertext. Journal of Computer-Based Instruction, 20(1), 1-8.

Joint Committee on Standards for Educational Evaluation (1981). Standards for
Evaluation of Educational Programs, Projects and Materials. New York:
McGraw-Hill. .

Johnston, V. M. (1987). Attitudes towards microcomputers in learning, 1: pupils and
software for language development. Educational Research, 29, 1, 47-55.

Jonassen, D. H. (ed.) (1988). Instructional Designs for Microcomputer Courseware.
Hillsdale, New Jersey: Lawrence Erlbaum Associates, Publishers.

Jones, L. E. and Koehly, L. M. (1993). Multidimensional scaling. In G. Keren and C.
Lewis (eds.), A handbook for data analysis in the behavioral sciences:
Methodological issues. Hillsdale, New Jersey: Lawrence Erlbaum Associates,
Publishers.

Kalowski, N. (1987). Videodisc developers want more flexibility. Instructional Delivery
Systems, 1(2), 19-21.

Kearsley, G. & Frost, J. (1985). Design factors for successful videodisc-based
instruction. Educational Technology, 25(3), 7-13.

Kemp, J. E. Morrison, G. R. and Ross, S. M. (1994). Designing Effective Instruction.
New York: Macmillan College Publishing Company.

Kleinman, G. (1984, February). Learning with computers: potentials and limitations.
Compute! 6(2), 120-121.

Kozma, R. B. (1991, Summer). Learning with media. Review of Educational Research,
61(2), 71-211.

Kress, G. and Synder, J. (1994). Forecasting and market analysis techniques: A practical
approach. Westport, Connecticut: Quorum Books.

Krippendorff, K. (1980). Content Analysis: An Introduction to Its Methodology.
London: Sage Publications.

303

Kruskal, J. B. and Wish, M. (1978). Multidimensional Scaling. Beverly Hills: Sage
Publications.

Kulik, J. A., Kulik, C. L., and Bangert-Drowns, R. L. (1985) The importance of outcome
studies: a reply to Clark. Journal of Educational Computing Research, 1 (4): 381-
387.

Kulik, J. A., Kulik, C. L., and Cohen, P. A. (1980) Effectiveness of computer-based
college teaching: a meta-analysis of findings. Review of Educational Research, 50
(4): 525-544.

Lezotte, L. (1994). The Effective Schools Research Assistant: An interactive database of
Effective Schools Research Abstracts. Okemos, Michigan: Efi'ective Schools
Productivity Systems.

Lillie, D. L., Hannum, W. H. and Stuck, G. B. (1989). Computers and Effective
Instruction: Using Computers and Software in the Classroom. New York:

Longrnan

Martindale, C. (1991). Cognitive Psychology: A Neural-Network Approach. Pacific
Grove, California: Brooks/Cole Publishing Company.

Mastroprieri, M. A. and Scruggs, T. E. (1987). Effective instruction for special
education. Boston: Little, Brown.

McNeil, B.J. and Nelson, KR. (1991). Meta-analysis of interactive video instruction: A
ten-year review of achievement effects. Journal of Computer-Based Instruction,
Vol. 18, No. 1, pp. 1-6.

McQuillan, Patrick (1994). Computers and pedagogy: the invisible presence. Journal of
Curriculum Studies, 26(6): 631-653. -

Mehrens, W. A. and Lehmann, I. J. (1984). Measurement and evaluation in education and
psychology. New York: Holt, Rinehart and Winston.

Merrill, M., Reigeluth, C, and Faust, G. (1979). The instructional quality profile: a
curriculum design tool. In H. O'Neil, Jr.(Ed.), Procedures for instructional
systems development (pp. 165-204). New York: Academic.

Office of Technology Assessment (1991). Federal funded research: Decisions for a
decade. Congress of the U. S., Washington, D. C. Ofiice of Technology
Assessment.

304

Office of Technology Assessment (1988). Power on! New tools for teaching and
learning (OTA-SET -379). U. S. Government Printing Office.

Okey, J. R. (1991). Procedures of Lesson Design. In Briggs et al. (eds.) Instructional
Design Principles and Applications: Second Edition. Englewood Cliffs, New ,
Jersey: Educational Technology Publications.

Popham, W. J. (1975). Educational Evaluation. Englewood Cliffs, new Jersey: Prentice
Hall.

Provus, M. (1971). Discrepancy Evaluation. Berkeley, CA.: McCutchan

Reeves, T. C. (1991). Ten commandments for the evaluation of interactive multimedia in
higher education. Journal of Computing in Higher Education, 2(2), 84-113.

Reigeluth, C. M. (ed.) (1983). Instructional-Design Theories and Models: An Overview
of their Current Status. Hillsdale, New Jersey: Lawrence Erlbaum Associates,
Publishers.

Rowland, G. (1992). What do instructional designers actually do? An initial investigation
of expert practices. Performance Improvement Quarterly, 5(2), 65-86.

Saroyan, A. (1989). The review process of formative evaluation: the role of the content
expert and the instructional designer. Unpublished doctoral dissertation, McGill
University, Montreal, Canada.

Saroyan, Alenoush (1993). Differences in expert practice: a case from formative
evaluation. Instructional Science 21 :451, pp. 451-472.

Schlosser, C. A. (1994). Distance Education: Review of the Literature. Ames: Iowa
State University of Science and Technology, Ames Research Institute for Studies
in Education.

Schvaneveldt, R. W.(Ed.) (1990). Pathfinder Associative Networks: Studies in
Knowledge Organization. Norwood, New Jersey: Ablex Publishing Corporation.

Scriven, M. (1967). The Methodology of Evaluation.” In Perspectives of Curriculum
Evaluation. AERA Monograph Series on Evaluation No. 1. Edited by R. E.
Stake. Chicago: Rand McNally.

Semrau, P. and Boyer, B. (1994). Using interactive video in education. Boston: Allyn
and Bacon.

305

Shuell, T. (1992). Designing Instructional Computing Systems for Meaningful Learning.
In M. Jones & P. Winne (Eds.), Adaptive Learning Environments: Foundations
and Frontiers. New York: Springer-Verlag.

Smith, E. (1987). Interactive video: An examination of use and effectiveness. Journal of
Instructional Development, 10(2), 2-10.

Smith, M. L. and Glass, G. V. (1987). Research and Evaluation in Education and the
Social Sciences. New Jersey: Prenctice-Hall, Inc.

Stake, R. E. (1967). The Countenance of Educational Evaluation. Teachers College
Record. 68, 523-540.

Steinberg, Esther R. (1990). The Centrality of Learner Characteristics In Computer-
Assisted Instruction. Journal of Computing in Higher Education. Winter 1990.
Vol. 1(2), 49-58.

Stufflebeam, D. L. et a1. (1971). Educational Evaluation and Decision-Making. Itasca, IL:
Peacock.

Stufflebeam, D. L. et a1. (1971). The relevance of the CIPP Evaluation Model for
Educational Accountability. Jom'nal of Research and Development in Education,
5(1), 19-25.

Tessmer, M. (1993). Planning and conducting formative evaluations: Improving the
quality of education and training. London: Kogan Page Limited.

Thompson, A. D., Simonson, M. R., and Hargrave, C. P. (1992). Educational technology:
A review of the research. Association for Educational Communications and
Technology.

Travers, J. F ., S. N. Elliot, and T. R. Kratochwill (1993). Educational Psychology:
Effective Teaching, Effective Learning. Madison, Wisconsin: WCB Brown &
Benchmark Publishers.

Tyler, R. W. (1950). Basic Principles of Curriculum and Instruction. Chicago, IL: The
University of Chicago Press.

Wager, W. and Gagné, R M. (1988). Designing computer-aided instruction. In Jonassen,
D. H. (ed.). Instructional designs for microcomputer courseware. Hillsdale, New
Jersey: Lawrence Erlbaum Associates, Publishers.

306

Warries, E. (1990). Theory and the Systematic Design of Instruction. In Sanne Dijkstra
et. al. (Eds.). Research on Instruction :Design and Effects. Englewood Cliffs, New
Jersey: Educational Technology Publications.

Weber, R. P. (1990). Basic Content Analysis. London: Sage Publications.

West, C. K., Farmer, J. A. and Wolff, P. M. (1991). Instructional Design: Implications
from Cognitive Science. Englewood Cliffs, New Jersey: Prentice Hall.

Wilkinson, 1... Hill, M., Welna, J. P., and Birkenbeuel, G. K. (1992). Systat: Statistics. ’
Evanston, IL: Systat, Inc.

Wolf, R. M. (1984). Evaluation in Education. New York: Praeger Publishers.
Woolfolk, A. E. (1995). Educational Psychology. Boston: Allyn and Bacon.
Worthen, B. R. (1990). Program Evaluation. In Walberg, H. J. and Haertel, G. D. (eds.),

the International Encyclopedia of Educational Evaluation. New York: Pergamon
Press.

 

”111111111111111111111