4». |: ‘1‘ :21. ,..,
32!! v9: Vl.r'.t| vrft.‘rlol .
.Iv . ....§11.le.lof
f. ‘l':'¢.2 VII!

I
I... . ... 1...!71: 9
ii"? iii-.5311. bi
: . lilirfll; » Iiii‘ltli‘- I

 

ll!!.rtl i .
I. 8!... )I§;.:5v§.e.v .
f? t.;..|3«.-nuin. £171.14?
.ngul ‘1‘ l

‘llvlvt ti.‘ |OL
:v‘l-Osl.‘

|A 5|)...I
1. : 1. (.11.
.)..v.n!a. 3...

 

.xxx.J§l-:vll§ulx. .‘,41\l:s)
. i}!!i€lu.¢ {)llo‘li.’
. £33.! (In! .. fenllDA

... \.| 111.00" 'ptroé luuullov’llllal’ ,VIIli'Ill
‘ .uv,r'll ‘1'!!- .0lvn .‘J‘Ii‘é‘

 

. 191:1}; u 1.9.x..- in. a . . , afiafiuuqvir. at;

ill-If
t: .I ‘

Ii! .56.. .
. . 011! 1.0?! ’ V
_ . ,V .. llol 4rit.».;.v.v|..

 

{HESIS

GAN STATEU UN VE

llllllllllllllll ll lllllllllll‘llllllllllllllll

1293 00881 0057

 

 

 

 

 

 

 

 

 

 

This is to certify that the

thesis entitled
THE ROLE OF EVALUABILITY ASSESSMENT IN SUGGESTING

PROGRAM EVALUATION STRATEGIES FOR EXPERIENTIAL
LEARNING PROGRAMS:
A CASE STUDY OF THE INTERNATIONAL STUDENTS VOLUNTEER CORPS

presented by

Shawn Thomas Lock

has been accepted towards fulfillment
of the requirements for

MasterSJf Sciencedcgree in Resource Development

     

v/(g Am

1
‘ Major professor

Date May 18, 1992

 

0-7639 MS U is an Affirmative Action/Equal Opportunity Institution

 

 

 

LIBRARY
Mlchlgan State
Unlvorslty

 

 

 

PLACE IN RETURN BOX to romovo this checkout from your record.

TO AVOID FINES Mun on or bdom dds due.

ll DATE DUE DATE DUE DATE DUE “

 

 

ll

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

_MSU Is An Aflirmativo Action/Equal Opportunity Institution

THE ROLE OF EVALUABILITY ASSESSMENT IN SUGGESTING
PROGRAM EVALUATION STRATEGIES FOR EXPERIENTIAL
LEARNING PROGRAMS:

A CASE STUDY OF THE INTERNATIONAL STUDENTS VOLUNTEER CORPS

BY
Shawn Thomas Lock

A THESIS

Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of

MASTER OF SCIENCE
Department of Resource Development

1992

ABSTRACT

THE ROLE OF EVALUABILITY ASSESSMENT IN SUGGESTING
PROGRAM EVALUATION STRATEGIES FOR EXPERIENTIAL
LEARNING PROGRAMS:

A CASE STUDY OF THE INTERNATIONAL STUDENTS VOLUNTEER CORPS

BY
Shawn Thomas Lock

This thesis investigates the use of evaluability
assessment (EA) as an evaluation tool for experiential
learning programs, using the International Students Volunteer
Corps (ISVC) as a case study. The study consisted of focus
group interviews using a modified version of EA, and a review
of project documents and previous participant interviews.
Outcomes were a program logic model that specified the
important decisions, activities, and implications of the ISVC,
and a list of potential items for further research and
evaluation.

The results of the study indicate a need to add a
goals/outcomes and assumptions components to the program logic
model to enhance its utility. EA as an evaluation tool was
seen as particularly useful to resource-constrained program
staff. Recommendations include the need to develop a primer
on EA for program staff with little evaluation experience, and
the need to develop appropriate training mechanisms to

instruct program staff on the use of EA.

To my mother,
Mary E. Lock,
for her endless support through all my journeys,
and
in memory of

Joseph W. Lock

iii

ACKNOWLEDGEMENTS

I would like to thank.my friends and family for their support
throughout my whole academic career. Their companionship
helped to pull me through what sometimes seemed an endless

process.

I would like to thank my thesis committee, Frank A. Fear,
David Horner, and George Rowan, for their participation,
encouragement, and good humor throughout the writing process.
I am especially indebted to Frank Fear, who, as a mentor and
friend, has always been there to help me in my personal,

academic, and professional growth.
Finally, I would like to express my gratitude to my parents,

who have always allowed all of their children the freedom to

be themselves.

iv

TABLE OF CONTENTS

LIST OF FIGURES O O O O O O O O O O O O O O O O O O O O v11

CHAPTER 1
INTRODUCTION . . . . . . . . . . .
The Contribution of Evaluability Assessment
to Program Evaluation . . . . .

Evaluability Assessment in Experiential Education

Programs . . . . . . .
Study Purpose . . . . . .
Overview of the Case Study .
Organization of the Study .

mourns: H H

CHAPTER 2

REVIEW OF THE LITERATURE . . . . . . . . . . . . . . . 10
Evaluability Assessment . . . . . . . . . . . . . 11
Utilizing Evaluation Results . . . . . . . . . . . 16

Using Evaluability Assessment in
Cross-Cultural Learning Programs . . . . . . 19

CHAPTER 3

CASE STUDY DESCRIPTION . . . . . . . . . . . . . . . . 23
Phase I: Program Proposal . . . . . . . . . . . . 23
Phase II: Program Development . . . . . . . . . . 26
Phase III: Program Implementation . . . . . . . . 28

CHAPTER 4

METHODOLOGY . . . . . . . . . . . . . . . . . . . . . 34
Evaluability Assessment . . . . . . . . . . . . . 34
Link to the Study Objectives . . . . . . . . . . . 45

CHAPTER 5

STUDY RESULTS . . . . . . . . . . . . . . . 47
Overview of the Methodology . . . . . . . . . . . 47
Focus Groups . . . . . . . . . . . . . . . . . . . 47
Focus Group Results . . . . . . . . . . . . . . . 48
Document Analysis . . . . . . . . . . . . . . . . 59
Stakeholder Interviews . . . . . . . . . . . . . . 61
Conclusion . . . . . . . . . . . . . . . . . . . . 62

vi

CHAPTER 6

SUMMARY CONCLUSIONS, AND RECOMMENDATIONS . . . . . . . 63
Summary of Study . . . . . . . . . . . . . . . . . 63
Main Findings . . . . . . . . . . . . . . . . . . 66
Recommendations . . . . . . . . . . . . . . . . . 71

APPENDIX ONE: OUTLINE OF ACTIVITIES . . . . . . . . . . 74
APPENDIX TWO: IN-FIELD SEMINAR OUTLINES . . . . . . . . 76
APPENDIX THREE: VOLUNTEER PLACEMENTS . . . . . . . . . 85

APPENDIX FOUR : TABLE OF CONTENTS : INTERNATIONAL
STUDENTS: SHARING THE WEALTH WITH THE COMMUNITY . 87

APPENDIX FIVE: MSU FACULTY QUESTIONNAIRE . . . . . . . 89

APPENDIX SIX: EVALUABILITY ASSESSMENT: A PRIMER FOR
SMALL-SCALE PROJECT STAFF . . . . . . . . . . . . 92

LI ST OF REFERENCES 0 O O O O O O O O O O O O O O O O O 1 o 3

Figure 5.1
Figure 5.2
Figure A.1

Figure A.2

LIST OF FIGURES

ISVC Program Logic Model: Session I .

ISVC Program Logic Model: Session II .

Sample Overheads for the First EA Session

Sample Program Logic Model: The ISVC .

vii

49

53

97

CHAPTER 1

INTRODUCTION

The Contribution of Evaluability Assessment
to Program Evaluation

Program evaluation involves the measurement of program
performance indicators, and the testing of the logic behind a
program that links resources and activities with expected
outcomes (Wholey, 1987). Program staff and policymakers can
use program evaluation results to improve program performance
in future efforts.

Wholey (1987, pp. 77-78) identified four problems that

prevent the utilization of program evaluation:

1. Lack of definition of the problem
addressed, the program intervention, the
expected outcomes of the program, or the
expected.impact.on the problem addressed.

2. Lack of a clear logic of testable assumptions
linking expenditure of program resources, the
implementation of the program, the outcomes to
be caused by that program, and the resulting
impact.

3. Lack of agreement on evaluation priorities and
intended uses of evaluation.

4. Inability or unwillingness to act on the basis
of evaluation information.

Evaluability assessment (EA) is a tool that helps clarify
a program's structure, establish the plausibility of its

1

2

activities in reaching intended goals, determine the
evaluability of the program, and enhance the utilization of
the evaluation results (Smith, 1989). EA was first developed
by Joseph Wholey and his associates at the Urban Institute in
Washington, D.C. (Nay & Kay, 1982, Smith, 1989). It was
developed to address the perceived inadequacies of program
evaluation. First used as a summative evaluation tool, EA is
now used to assess all aspects of the program process.

In brief, EA involves gathering an evaluability
assessment team that can help gather materials and develop a
program logic model that outlines the resources and activities
necessary to reach the program goal. Throughout the process,
individuals who have a vested interest in the program --
stakeholders -- are identified and interviewed to produce
additional information. After developing the program logic
model, a decision is made on what aspects of the program can
be evaluated, and. which. aspects are :most important. for
evaluation. Developing an evaluation strategy that results in
findings that can and will be used by stakeholders is a key

component of EA.

Evaluability Assessment in Experiential Education Programs

Program evaluations conducted with experiential programs

in higher education appear to have the same deficiencies that
prevent utilization of evaluation findings. One significant

set of experiential activities in higher education is cross-

3

cultural learning programs. Cross-cultural experiences occur
in a number of ways. Many academic programs require
undergraduates students to participate in a cross-cultural
experience as part of a liberal arts education. Given the
growth in worldwide communication and interaction, cross-
cultural skills will most likely become increasingly
important. Consequently, it is important to have effective
evaluation tools to assess such programs.

Evaluation of experiential programs have concentrated
more on assessment techniques, rather than on overall program
evaluation. Such evaluations have focused on the importance
of developing clear objectives, preparing the student for
placement, monitoring the experience, and then judging some
final product (Duley & Gordon, 1977). Program evaluations
have often been implemented in the form of checklists
describing the steps involved from initial student intake to
final assessment, in which the evaluator then matches the
program against the ideal checklist (Knapp & Sharon, 1975,
Permaul & Miko, 1977) . The EA process helps to examine
implementation questions that answer issues related to the
plausibility of program goals and implementation activities.
EA represents an effective method for looking at broader

programmatic issues to assess program performance.

4
Study Purpose

Evaluability assessment (EA) has been used successfully
in many Cooperative Extension Service and other governmental
programs. Many of these programs involve adults in learning
programs. Given the limited attention paid to program
evaluation efforts in adult experiential learning programs in
higher education, and the narrow focus of existing research,
there is a need to utilize existing evaluation methods to
improve program performance review.

The objectives of this study are to:

1. Determine the usefulness of the evaluability
assessment process in suggesting appropriate
evaluation methods for experiential learning
programs.

2. Suggest guidelines that program staff can use to
adapt evaluability assessment to fit individual
needs/programs.

3 . Highlight the utilization-enhancing characteristics
inherent to evaluability assessment, and suggest
additional guidelines.

This study uses the International Students Volunteer
Corps (ISVC) as a case study in which to apply the EA.process.
The ISVC was a cross-cultural volunteer program conducted
twice during two academic years, 1989-1991. This study will
focus on the first year of the ISVC. 'This choice is necessary
due to several changes that occurred during the second year:

1. Although the recruitment phase of the second year
was quite successful, retention was low. Out of 80
students that originally registered for the

program, only five students continued on a regular
basis.

5

2 . The students that remained in the program had
limited english speaking skills. The majority of
students had just started their first year at MSU,
and several were still attempting to pass their
english competency test, before being granted full
admittance into the university. The time necessary
to master english skills and fulfill the program
requirements at the same time proved to be too
difficult, and students were unable to start the
actual volunteer phase of the program.

3. Due to other commitments, a reduction in program
responsibilities, and fewer staff meetings, faculty
staff members played a less prominent role in the
project. As a result, there is less information
available on the program valuable to others, both
from an experiential aspect on the part of the
staff, and in the form of written feedback from the
students.

4. The program was terminated early when attendance
continued to drop even further, and students were
unable to complete the program requirements.

The first year of the program was successfully completed,
closely following the program process as outlined in the
funding proposal. Several evaluative activities occurred
during the first year, and staff met on a weekly basis to
reflect on the program's current activities and on its long
term implementation. The program staff's high level of
participation during the first year resulted in more
information on which to conduct a more complete evaluation,
and provide information relevant to the study objectives
listed above.

As a case study, the ISVC offers several additional
opportunities. Increasing world communication and travel

bring peoples of diverse backgrounds in contact with each

other on a regular basis. Developing peoples' ability to

6

communicate effectively in other cultures is essential. The
ISVC represents one program that helps students, and local
volunteer agency workers, develop a broader understanding of
other cultures. Applying EA procedures to the ISVC will
provide an example for other cross-cultural programs.

As an experiential program, the ISVC offers an
opportunity to test the EA process. The lessons learned in
this application could be adapted for other programs needing

a similar evaluation mechanism.

Overview of the Case Study

The International Students Volunteer Corps (ISVC) was a
cross-cultural volunteer program conducted at Michigan State
University (MSU) during the 1989-90 and 1990-91 academic
years. The ISVC was a collaborative effort among several
academic and administrative units at ‘MSU; The primary
objective of the ISVC was to enhance international students'
conceptual and practical understanding of American society
through American volunteerism. Secondary objectives included:

1. strengthening interdepartmental networking’ among
the academic units at MSU;

2. expanding upon those units’ linkages with local
public and private agencies:

3. supplementing the personnel resources available to
local agencies: and

4. exposing local agencies and volunteers to the
cultures represented by MSU students.

7

The ISVC was comprised of three phases. £h§§g_gng. The
first phase involved recruitment, project planning, and the
pre-field orientation, The pro-field orientation occurred as
a series of two-hour seminars held over a five week period.
Topics in the seminars included the concept of volunteerism in
the U.S. and its development in the local community, elements
in a successful volunteer experience, such as writing learning
objectives and job descriptions, and aspects of cross-cultural
communication in the workplace.

PnQQQJTwQ. In phase two, students began their volunteer
experience in agencies they identified during phase one.
Students' placements included such organizations as the
Salvation Army, the American Red Cross, the Voluntary Action
Center, the Capitol Area Literacy Coalition, and the Council
Against Domestic Assault. During this phase, students
attended bi-monthly reflection seminars.

£hg§g_1hzgg. Phase three consisted of students efforts
to evaluate the ISVC project and their experience. Students
participated in group sessions focusing on their experiences,
and a group of nine students presented papers at a closing
symposium. "MW
Mutiny”-

Participants in the ISVC came from a range of countries,
including: Nigeria, South Africa, Malawi, Trinidad & Tobago,
Malaysia, India, China, and Taiwan. Their academic

disciplines were equally diverse, ranging from Agriculture

8
Economics, Education, and Soil Sciences, to Sociology and
Resource Development.

During the project, several evaluative activities were
conducted. At the end of phase one, students filled an
evaluation form covering the recruitment and volunteer
placement process, and the pre-field curriculum. At the end
of the program, students filled out a final program evaluation
form. During this time, a graduate assistant assigned to the
project also conducted interviews with each student or her/his

academic advisor or her/his volunteer supervisor.

Organization of the Study

A review of literature sources relevant to evaluability
assessment (EA), as it pertains to program performance, occurs
in Chapter Two. The EA process is described, along with
critical elements associated with conducting a successful EA.
Elements of an evaluation that enhance utilization of the
results are discussed, as well. The ISVC is described in
detail in Chapter Three. The methodology for the study is
outlined in Chapter Four. The methodology details the steps
for conducting a preliminary EA that is suitable for cross-
cultural, experiential programs. The study results are
presented in Chapter Five. Summaries of the program logic
model process, interviews, and document analysis are used to
suggest an ”utilizable" evaluation process. In Chapter Six,

the suitability of EA for cross-cultural programs is

9
discussed, along with a consideration of questions and issues

associated with further research.

CHAPTER 2

REVIEW OF THE LITERATURE

In. the first chapter; a. brief’ overview’ of’ program
evaluation was presented, along with a description of the
International Students Volunteer Corps (ISVC) . Though several
programs have systematically introduced students into new
cultures, evaluation and assessment have focused on awarding
credit, rather than looking at programming aspects (Duley 8
Gordon, 1977). Evaluability assessment (EA) was proposed as
an appropriate method to develop evaluation strategies for
cross-cultural learning programs.

Chapter two describes those aspects of evaluability
assessment (EA) that relate to developing evaluation
procedures of cross-cultural experiential programs. The first
section outlines the actual process in EA. The second section
links EA to the notion of utilizing evaluation results for
program improvement. The final section provides a general
overview of cross-cultural learning programs in higher

education.

10

11
Evaluability Assessment

WW

Evaluability assessment (EA) has evolved over time from
a mostly summative technique, to one that can be used during
program development or program monitoring (Smith, 1989, pp.
13-14). Joseph Wholey and his associates are credited with
developing EA at the Urban Institute in Washington, D.C. (Nay
& Kay, 1982, Smith, 1989) . The EA process arose from a desire
by researchers at the Institute to match program rhetoric and
practice, and whether or not program rhetoric could actually
be measured in an evaluation (Nay & Kay, 1982, p. 225). In
1984, EA was adopted in five states in a practical effort to
define the EA process, largely from a summative viewpoint
(Smith, 1989). It is now frequently used as part of the
program development process in many Cooperative Extension

System efforts (Mayeske, 1991).

As stated above, EA is a process of matching program
rhetoric to reality, and then devising a strategy to measure
program rhetoric. The EA team is responsible for developing
the program model, described below, that links program
rhetoric to realityu They are involved. throughout ‘the
process, from identifying the EA strategies to analyzing

results, drawing conclusions, and. making recommendations

12

(Smith, 1989, p. 33). The EA team is composed of program
staff, policy makers, and administrators. Individuals on the
EA team should have subject-matter expertise related to the
program, and experience with the clientele (Mayeske, 1991).
Members of the EA team are stakeholders, persons with a vested
interest in the program, but they are only a subset of the
stakeholder group. Further discussion on stakeholders in the
EA process occurs later in the chapter.

A program is a set of resources and activities used to
accomplish one or more goals (Mayeske, 1991, Rutmam, 1980,
Smith, 1989) . Boundaries need to be established for a program
in order to link evaluation findings to an easily identified
intervention and its results (Rutman, p. 45). Otherwise,
evaluation results are tied to an imprecise set of activities,
and contribute little to further program development. Smith
(1989) suggests two criteria for determining program
boundaries. First, EA should focus on.major programs. Major
programs represent a significant commitment in resources.
Second, the nature of the questions to be answered determine
the size of the program boundaries. Those questions that are
most important determine the direction and extent of EA

activities.

MW
After developing an EA team and defining program

boundaries, the next step is to develop the program theory or

13

program logic model. Building a program logic model is a
process of: 1) determining means-ends, cause-effect
relationships in a program as defined by the program staff
(Smith, 1989, Mayeske, 1991): 2) identifying clearly defined
program components that can be implemented in a prescribed
manner: and 3) specifying plausible goals and effects (Rutman,
1980). Wholey (1987, pp. 78-79) describes program theory as
an if-then expression:

If the following program resources are available,

then the following program activities will be

undertaken... If these program activities are to

occur, then the following program outcomes will be
produced... If these activities and outcomes occur,

then progress will be made toward the following

program goals.

The program logic model is developed as a flow chart by
the EA.teamm The model developed.by the EA team.is refined.by
reviewing project documents, site visits, and further
interviews (Wholey, 1987). Depending on only one source may
lead to an overemphasis on a particular component of the
model, such as outcomes. Gathering information for a variety
of resources and individuals will help to bring out most
components of the program (Nay & Kay, 1982).

The final model should contain information on key
components, or intermediate objectives, that precede the main
goal(s), the activities and resources necessary to accomplish

each component, and the indicators of success (Smith, p. 53).

Mayeske (1991) also emphasizes listing the potential barriers

14

at each component, methods to reduce barriers, intervening
events that the staff will have no control over, and spin-offs
from each component - good or bad. The model can be presented
in any format that the EA team desires, but it should be
easily understood by all stakeholders, and should break the
program down into distinct components for final analysis
(Smith, p. 53).

Program theory is essential to the EA process, because it
outlines the assumptions and activities that lead to the
program goals. The evaluator can look at the program theory
and actual implementation to see if there is a fit. In other
words, did actual activities follow what was intended in the
program theory (Scheirer, 1987)? Before the actual evaluation
process begins, analyzing the degree of implementation and
resulting outcomes, a comparison of the program logic model
and implementation can determine whether or not the program,
as originally proposed, is evaluable. Thus, the program
theory stage gives valuable information in whether an
evaluation effort should continue.

Finally, the process of building the program logic model
requires that key actors, or stakeholders, come together and
agree on the program's theory. This process builds consensus
among the EA team, and gives each participant similar views

and expectations of the program (Smith, 1989, Mayeske, 1991).

15

W

W are anyone who have a vested interest in the
program or evaluation results (Patton, 1986) . Stakeholders
have valuable information about the program, and can make the
program logic model more reflective of the program's original
design (Mayeske, 1991) . Some stakeholders play a role on the
EA team, while others may be targeted for interviews to
further define the program theory and/or provide information
during the actual evaluation process as it arises out of
evaluability assessment. Thus, stakeholders not only utilize
evaluation results, but are affected by them: their own
interests may depend on the evaluation results (Smith, 1989) .

Both Smith (1989) and Mayeske (1991) suggest that the two
most important items to determine about stakeholders is their
awareness of the program, and their interest in it. Questions
generally include asking a stakeholder about her/his knowledge
of the program, its goals and activities, the perceived
benefits, and any general comments related to the
stakeholder's personal observation on the efficacies of such
programs. Stakeholders provide information on what
information may be most important to collect for utilization.
Consequently, the less a person knows about the program, the
less likely that person is a true stakeholder (Mayeske, 1991,

p. 6-4).

16
Utilizing Evaluation Results
WW1;

Wholey (1991) suggests that EA is a process for achieving
consensus on how to utilize evaluation results. The EA
process specifies evaluation priorities and intended uses,
such as stimulating improved program performance ,
communicating the value of the program to stakeholders, and
efficiently allocating program resources. The process of
building a program logic model should result in a hierarchical
list of evaluation priorities. The EA team, and additional
stakeholders, can choose the most important priorities on
which to focus the evaluation.

EA is an evaluation needs assessment that helps determine
if an evaluation is appropriate and whether the program is
evaluable (Patton, 1991) . If the necessary information is not
available to make that decision, the evaluator and/or the EA
team may need to collect preliminary information. Wholey
(1991) outlines a rapid-feedback evaluation process, similar
to EA, to help gather preliminary results to determine
potential evaluability and utilization. Rapid-feedback
evaluation involves reviewing available information,
collecting new data if necessary, conducting a preliminary
evaluation, and determining the potential for a full-scale
evaluation. The potential evaluability is highly dependent on
its likely uses.

17

The results of the EA process should indicate the
potential data that can.be produced, and whether stakeholders
are willing and/or able to utilize the evaluation results.
Smith (1989, p. 136) suggests five choices in determining
utilization:

1. Decide to evaluate the program

2. Decide to change the program

3. Decide to take no further action

4. Decide to stop the program

5. Do not decide and ignore the EA.
The option(s) selected often depends on the purpose of the EA,
i.e. , whether the EA is for program development or evaluation.
When the main purpose is program performance evaluation, the
EA process, especially the program logic model and stakeholder
interviews, may reveal that there is little desire to utilize
the results, and an evaluation may be an inefficient use of
resources. Deciding to stop the program, or to not take
further action, would be appropriate at this time. However,

the EA.process may reveal a need to change the program, rather

than evaluate, if the program was not designed properly.

,1, ,. ,- a . _.. ,;;;:..;, - . -;. ,. ;

Determining utilization before the evaluation can help
save valuable resources. Patton (1991) suggests several steps
to determine utilization. An evaluator can conduct a mock
utilization session in which stakeholders and/or the EA team
are asked to show how they plan to use the results. Major

findings could be anticipated, and participants would have to

18

show how those findings would be used in the program or with
key policy makers. Having stakeholders write down what
findings they anticipate, and keeping those statements for
comparison at the end of the evaluation, helps them to pay
more attention to the final results. Finally, involving major
stakeholders at all stages of the evaluation raises their
investment in the process and the likelihood that they will
use evaluation results.

The evaluator should take a proactive role in promoting
utilization (Patton, 1990) . Patton recommends that the
evaluator negotiate with stakeholders-- to facilitate an
agreement on which questions are most important, and then be
held accountable for delivering the information on those
questions. In addition, the evaluator may need to uncover any
fears or stereotypes stakeholders may have about evaluation
before she is able to negotiate the process. Uncovering
stakeholder fears or stereotypes can be done through small
group exercises and answering questions anonymously submitted
on paper (Patton, 1991).

In a series of discussions and debates on evaluation
(Alkin, 1990) , panelists agreed that an evaluation is not
effective unless there is proof of utilization by the intended
users. EA has the potential to bring stakeholders together in
agreement on a program’s direction, and the priority
evaluation issues, but it is effective only if the results can

be utilized by stakeholders. Utilization, then, is an

19
essential product of EA, but it is also an ongoing focus

throughout the process.

Using Evaluability Assessment in
Cross-Cultural Learning Programs

mm

Experiential learning can.be viewed from the perspective
of learning from life experiences, such as skills learned on-
the-job or self-directed learning from a hobby or sponsored
experiential learning occurring in an guided, academic
program. This study is concerned mainly with sponsored
learning. Experiential learning involves not only observing
and reading about an activity, but also directly participating
in the activity through volunteerism and internships (Keeton,
1978). Kolb’s work on the learning cycle is often used as a
theoretical framework for experiential learning programs
(Kolb, 1976a, cited in Doherty, et a1. , 1978) . In this
framework, students reflect on concrete experiences to help
build concepts and generalizations for testing through further
experiences.

Experiential learning takes place in several forms with
several goals. The forms range from independent study, to
group projects conducted as a class requirement, to graduation
requirements and service learning centers. The goals of the
programs include career exploration and development,
institutional analysis, personal growth and development, field

research, social/political action, and cross-cultural

20

experiences (Duley 8 Gordon, 1977). Skills often found most
helpful include: observing and recording information,
gathering information orally, reflecting on activities,
problem solving, and cultural analysis (Duley, n.d.).

Cross-cultural experiences occur in a number of ways.
Programs such as the now defunct Justin Morrill program at
Michigan State University required undergraduates students to
participate in a cross-cultural experience, whether in the
U.S. or abroad, as part of a liberal arts education. The
Experiment for International Living and its School for
International Training represent institutions whose primary
mission is the training of individuals for international
careers (Batchelder & Warner, 1977). A great deal of their
training, such as language and field research, is devoted to

cross-cultural learning.

WWW
To justify the validity of experiential learning in

traditional academic institutions, researchers have
concentrated more on assessment techniques, rather than on
overall program.evaluation. .Assessment activities stress the
importance of developing clear objectives, preparing the
student for placement, monitoring the experience, and then
judging some final product (Duley & Gordon, 1977) . Some
programs may attempt a fol low-up interview to gauge the extent

to which students have integrated the experience. Techniques

21
used to assess the end product include: simulations,
performance tests, essay examinations, interviews, self-
assessment and product assessment (Knapp & Sharon, 1975) .
Diaries or journals and periodic questionnaires represent the
main method of monitoring the experience in process (Permaul
& Miko, 1977).

The overwhelming focus is on creating competencies.
While literature does exist for program evaluations, these are
often presented in the form of checklists describing the steps
involved from initial student intake to final assessment. The
evaluator then matches the program against the ideal checklist
(Knapp & Sharon, 1975, Permaul & Miko, 1977). In addition,
there appears to be a gap in the literature dealing with
experiential learning. Recent literature addresses the
experiential aspects of teaching english as a second language,
designing training programs for foreign teaching assistants,
and developing international education programs.

EA offers a process to determine effective evaluation
strategies for cross-cultural programs. Evaluation efforts
that monitor the program while it is in progress, such as
those methods mentioned in this chapter, have an important
role in overall program assessment. Patton (1986) classifies
these efforts as implementation evaluation-- activities that
answer what actually happened in the program. EA, however,
allows the evaluator and the EA team to go one step beyond

implementation questions to answer issues related to the

22
plausibility of program goals and implementation activities.
EA also assists stakeholders in developing evaluation
procedures to determine program performance (Wholey, 1991),
including both expected and unexpected outcomes (Patton,
1991).

Because evaluation efforts have concentrated on
implementation issues, there is a need to look at overall
programmatic issues. Looking at cross-cultural programs from
a programmatic perspective can provide information on the
worth of programs in a student’s educational experience. The
International Students Volunteer Corps (ISVC), described in
the next chapter, provides a rich setting in which to answer
the question of worth. As a completed program, the ISVC
represents a case study to apply the program performance

assessment function of evaluability assessment.

CHAPTER 3

CASE STUDY DESCRIPTION

The purpose of this chapter is to provide the reader with
background on the International Students Volunteer Corps
(ISVC) . The program description follows a sequential format,
from the program proposal through the first year and into the
second year. Special emphasis is given to program development
and implementation. The evaluation process is briefly
described. However, a more detailed discussion is presented in
the next chapter, and the results from the evaluation are

discussed in Chapter 5.

Phase I: Program Proposal

The program proposal to the National Association for
Foreign Student Affairs (NAFSA) resulted from previous efforts
undertaken by two faculty members in the Department of
Resource Development at Michigan State University. Their
efforts grew out of a desire to help international students
integrate their academic experience, with practical experience
in the local community. They observed that international
students often completed their education without experiencing

the local culture:

23

24

Traditional Instructional methods -- exposure to

pertinent literature, site visits to agencies, and

guest lectures by members from the surrounding
communities -- typically provide international
students with only a glimpse into the dynamics of

American culture... Significant problems can

emerge if a graduate later transfers elements of a

misunderstood system into her or his country.

(Fear 8 Axinn, 1989, p.1)

The proposal emanated from a desire to provide a
comprehensive educational experience for international
students. Because the Department of Resource Development had
a tradition of volunteer work in the community, volunteerism
became the vehicle by which students would gain a greater
understanding of the local culture. The objectives of the
program were designed to benefit both the students and the
community. The original project objectives were to:

1. enhance international students’ conceptual and

practical understanding of American.society through
American volunteerism,

2. strengthen interdepartmental networking among the
academic units at MSU,

3. expand upon those units' linkages with local public
and private agencies,

4. supplement the personnel resources available to
local agencies, and

5. expose local agencies and volunteers to the
cultures represented by MSU students.

Incorporating international students into volunteer
activities was a new'priority for NAFSA.during 1989. The ISVC
proposal was the only program selected for funding during that
year. NAFSA staff chose the ISVC as a prototype effort -- a

potential model for other educational institutions to adapt in

25
years to come. The ISVC was funded for $14,100.00 for the
academic year 1989-1990. A no-cost extension was later sought
and granted, so that the project timeline was extended to
December 1990. Additional funding came from MSU's Urban
Affairs Program for a half-time graduate assistantship.
Faculty members time was an in-kind contribution, and amounted
to $22,500.00. The total cost for the first year (funds

raised and contributed) amounted to $49,600.00.

W

The core staff represented an effort to fulfill one of
the program objectives -- strengthen interdepartmental
networking -- in that it brought staff together from several
different units across campus. Members of the core staff
represented the Department of Resource Development (College of
Agriculture and Natural Resources), the Center for Urban of
Affairs (Urban Affairs Program), the Office for International
Students and Scholars (International Studies and Programs),
and the Service Learning Center (Student Affairs and
Services). In addition, the core staff included l/4-time and
1/2-time graduate assistants responsible for student
recruitment and program management, respectively.

The core staff was responsible for overall program
development, implementation, and evaluation. They met on a
weekly basis to review current activities and plan future

events. Each member was responsible for one or more of the

26
seminars conducted for participants (described later in this
chapter) as well as coordinating a learning group of three to
five participants throughout their volunteer experience. The
first year of the ISVC required an intensive time commitment
from the core staff in the form of planning, implementation,

and supervision.

Phase II: Program Development
Wins:

In the process of developing the ISVC, the core staff
conducted two focus group sessions during August, 1989. The
first focus group involved several volunteer supervisors from
local volunteer agencies, such as the Salvation Army, the
American Red Cross, and the Voluntary Action Center. The
meeting concentrated on several issues, including:
incorporating international students into a volunteer
organization, determining a minimum time necessary for an
effective volunteer experience, and discussing the cost-
benefit issues related to extra training/orientation times for
international students.

Overall, focus group members were highly supportive of
the ISVC. Their support arose from.positive experiences with
the Center for Urban Affairs and the Department of Resource
Development, which had historically involved international
students in community projects through a field study class.

Several participants at the meeting became members of the

27
advisory committee, formed during the project implementation
phase.

The second focus group meeting involved international
students. The students were able to provide their perspective
on issues related to developing the ISVC to meet the
particular needs and circumstances of international students.
Issues discussed included: presenting the concept of
volunteerism to individuals from another culture, adapting
program requirements to meet students' constraints, developing
links with international student groups, and motivating
students to participate in the ISVC.

During the meeting, one student expressed the frustration
that she was unable to participate in the United States Peace
Corps due to her international status. A faculty member
suggested that this program could be an opportunity for
interested students to experience American culture --a
”reverse" Peace Corps of sorts-- and the name of the

"International Students Volunteer Corps" was coined.

99W

Both focus groups provided valuable information that was
incorporated into the curriculum and program development
process. The original program proposal outlined a three-phase
effort, pre-field, in-field, and post-field. The curriculum
for the in-field was initially drafted by one graduate

assistant, and then modified on an ongoing basis by the core

28
staff. Refinements to the process outlined in the original
proposal occurred during the recruitment process. Similarly,
throughout the life of the program, changes were made in
response the needs of students and the demands of program
implementation. A sequence of project activities is listed in

Appendix One.

Phase III: Program Implementation
Recruitment

Several strategies were used to recruit students for the
program. During an open house for new international students,
an information booth was set up to distribute brochures and
answer questions. The university's Office for International
Students and Scholars, whose director was on the core staff,
had an information. display: Notices were sent to the
presidents of each international student group, and an article
was written in the university newspaper.

In addition to these formal attempts, informal networks
were used by the core staff. They contacted students they
knew whom could benefit from the program. The graduate
assistant in charge of recruitment relied extensively on
networks she had developed in the graduate student population.
Informal contacts were often the most successful recruitment
tool. However, each method helped to attract a unique group

of students.

29

Twenty-five students participated in the program at
different levels. Twelve students were able to participate in
all three phases of the program, the remainder participated in
varying degrees, but were unable to complete the full program.
Participants in the ISVC came rom a range of countries,
including, Nigeria, South Africa, Malawi, Trinidad 8 Tobago,
Malaysia, India, China, and Taiwan. Their academic
disciplines were equally diverse including agriculture,
economics, education, soil sciences, sociology, and resource
development. All participants were graduate students, with

the majority at the doctoral level.

W

In the first phase, students were to attend 10 sessions
on American volunteerism, the American human service network
system, and American and human social problems. During the
program development process, a cross-cultural communication
emphasis was added to acquaint students with possible
challenges in their volunteer experience. In addition, a
component on learning objectives was inserted to help students
establish personal goals for their volunteer placement.
Although the original proposal called for students to develop
work plans on a particular aspect of volunteerism, it was not
emphasized during actual implementation. Instead, students
were encouraged to reflect on their experience in preparation

for a year-end symposium.

30

Because initial recruitment efforts resulted in a low
turnout, an orientation seminar was developed to explain the
program and make it easier for students to join the program.
In addition, a social get together was planned while students
were on Winter Term class break in order to developra sense of
community within the participants. A small meal was given at
each seminar to encourage a social atmosphere, and to entice
students to come to meetings on a regular basis.

Several changes were made in the curriculum throughout
the pre-field to adjust to the students’ needs. More time was
spent on explaining the volunteer system in America, due to
students’ unfamiliarity with the system. As a result, less
time was devoted to cross-cultural communication. The three
major topics covered were the American volunteer system,
designing an effective learning experience, and cross-cultural
communication. Additional assistance was provided by local
volunteer agency supervisors. They presented material through
guest lectures and provided real-life examples during each
session. Students also spent a significant amount of time
contrasting and comparing volunteerismito social institutions
in their own.countries. .A full description of the seminars is

in Appendix Two.

111W
The second.phase consisted of students’ field placement.

The proposal called for a minimum of 5 hours of volunteer time

31

per week. Actual volunteer time varied, depending on the
students’ and the particular agencies’ needs. The placement
process began during the in-field phase. Students were asked
to identify interest areas (e.g., work with the elderly), and
the graduate assistant responsible for placement then
suggested possible volunteer agencies. Many students were
able to begin their placement by the end of the in-field
phase. A full list of placements is in Appendix Three.

Several students attended the majority of the seminars,
during both the pre-field and in-field phases, but did not
find a volunteer placement. This occurred either from
dissatisfaction with their first placement, an inability to
work out a placement in their interest area, or from
scheduling conflicts. During the in-field phase, students
attended bi-monthly sessions to discuss activities at their
placement site. Concepts discussed during the pre-field were
reviewed in light of their new experiences. Volunteer agency
supervisors provided feedback and suggestions to students
concerning their experiences. The supervisors also gained a
greater understanding of an the students’ goals and the
challenges of volunteering in a cross-cultural situation.
Students were encouraged to keep in mind a possible writing
topic -- related to their placement, concepts in the pre-
field, and similar activities in their own country -- for
presentation at a program-closing symposium. Core staff also

kept in contact with students in their learning group to

32

monitor students’ volunteer experience.

W
The post-field phase focused on a closing symposium,
! ' 1'. 0,: Ao‘! : o _.l :2 9.9-. ,. ,o ,- ‘z. 9
with_thg_§gmmun1§y which was held in August of 1990. Nine
students presented.papers, or summarized sessions, describing
their volunteer experiences, and comparing them to aspects of
volunteerism in their own country. Volunteer supervisors from
local agencies presented their impressions of the experience,
and concurrent sessions were conducted on issues of
international students and volunteerism (e.g., cross-cultural
communication and recruitment). A symposium proceedings was
produced, along with a short video on the ISVC with footage
from the symposium. The Table of Contents for the symposium

proceedings can be found in Appendix Four.

12W

Dissemination of the ISVC program to other practitioners
included presentations at conferences and written articles.
The symposium proceedings were distributed nationwide via two
networks: the National Association for Foreign Student
Affairs, and the Michigan Campus Compact. The Michigan Campus
Compact funded several ISVC activities including the symposium
and symposium proceedings. Core staff members gave

presentations at conferences in Michigan, Washington, D.C.,

33
and Canada. Articles appearing in NAFSA’s newsletter drew
inquiries from several schools across the U.S.

Program evaluation consisted of a participant
questionnaire pertaining to the pre-field seminar format, the
placement process, and recommendations for future years. In
addition, interviews were conducted with selected groups of
students, their volunteer supervisors, and their academic
supervisors. Details on both the program evaluation
questionnaire, and the interview format, are discussed in

Chapter Four, and the results are presented in Chapter Five.

CHAPTER 4

METHODOLOGY

The literature review in Chapter Two focused on the
importance of evaluability assessment (EA) and the steps
necessary to conduct an EA. Experiential learning programs
were discussed as one programming activity that might benefit
from the EA process. Chapter Three described in detail the
International Students Volunteer Corps (ISVC), a cross-
cultural experiential learning program at Michigan State
University. In this chapter, the EA process is used to
develop a methodological framework to propose an evaluation
strategy for the ISVC. The first section discusses the steps
that will be taken.to conduct the EA, along with the rationale
for each step. The second section reviews the EA process in

light of the research questions stated in Chapter One.

EvaluabilityAssessment
,,. ,; ~3 ;.. ,,. .p- , ,. ,- - .g a, ;p_,., ;.
As stated in Chapter Two, The EA team is responsible for
developing the program model that links program rhetoric to
reality. They are involved throughout the process. The EA

team can be composed of program staff, policy makers, and

34

35
administrators. Members of the EA team should have subject-
matter expertise related to the program and the clientele
(Mayeske, 1991).

The EA team in this study includes program staff members
and selected ‘volunteer administrators from ‘participating
agencies. The EA team represents a diverse set of
individuals, including a chairperson of an academic
department, a director of an international students office, a
director of an urban outreach center, and various volunteer
coordinators in charge of programming and volunteer training.
Each of the members contributed to the ISVC throughout its
implementation in such activities as program planning,
conducting seminars, and evaluating activities.

The purpose of the study is to use the EA process to
suggest evaluation strategies, and to assess the applicability
of EA to experiential programs, using the ISVC as a case
study. The EA team’s primary responsibility is to develop an
accurate program logic model. Consequently, the members of
the EA team need to be familiar with the ISVC from start to
finish. Although student participants and other faculty
members were involved in the program, they were not involved
in conceptualizing, planning, and.monitoring the program. In
terms of conducting the EA, the students and faculty members
represent stakeholders whose views and experiences can help to
validate the program logic model, rather than help to

construct it.

36

In addition to selecting the EA team, it is important to
define the program boundaries. Boundaries are necessary to
link.evaluation findings to an easily identified intervention
(Rutman, p.45) . Smith (1989) suggests that boundaries be
determined by the nature of the questions asked about the
program, by focusing on major programs, or by those that
represent a significant commitment in resources. This study
establishes as its boundaries the first year of the ISVC. The
reasons for this decision can be summarized in two points:

1. The first year of the ISVC was the only year in
which the program was conducted as proposed in the
funding proposal. The program. was terminated
before completion during the second year, and thus
there is insufficient information to answer the
research questions in that year.

2. The first year represents the more significant
commitment of resources in terms of funding, staff,
and evaluation effort.

The EA team will focus solely on the first year of the

ISVC for the purpose of building the program logic model.
Each member was involved on a regular basis in program
activities, and participated in an evaluation focus group at
the end of the first year. Their input, and information from

other sources, will be used to develop the program logic model

in order to suggest possible evaluation strategies.

W
Building a program logic model is a process of
determining means-ends, cause-effect relationships in a

program as defined by the program staff (Smith, 1989). This

37
study will take a multi-method or triangulation approach
(Gorden, 1980) and use three separate sources to build the
program logic model. The three methods are: 1) a modified
focus group process: 2) document analysis: and 3) interviews
with stakeholders and program participants.

W. A focus group is an interview with a group
of people on a specific topic. The length of the interview
generally lasts up to two hours (Patton, 1990) . One advantage
of focus groups is that each participant, or interviewee,
hears other interviewees’ comments, and so can make additional
comments or revisions to their initial response. However,
because of the large size of the group, fewer questions can be
asked (Patton, 1990) . Two focus groups will be conducted
within a one-month period in which the program logic model
will be built.

Krueger (1988) states that focus groups are normally
composed of individuals who do not know each other. Close
associations between participants can result in information
that arises from past experiences that may cause people to
*withhold. some knowledge. In addition, the focus group
moderator, or interviewer, is often a stranger to the
participants, and not affiliated with the subject under
discussion. Building a program logic model with a program
staff requires modifications of these basic characteristics of
a focus group. The EA process specifically brings program

staff together to reflect on a program in ways they may never

38
have before. Krueger emphasizes that, if the participants do
know each other well, they should be reminded that the focus
group is a reflective activity, and not that it is not an
actual planning or evaluation meeting.

The focus group moderator was the graduate assistant
responsible for day-to-day program management. While a focus
group moderator usually does not have as close an association
with the topic in question, in this effort it was not abnormal
for him to fulfill such a role in the course of his duties as
program manager. Previously, he had conducted an evaluation
focus group with volunteer administrators and program staff.
Focus group participants were accustomed to seeing him in a
similar role, and so it was concluded that the results of the
focus group would not be negatively impacted by his taking the
moderator role. However, as moderator, there was no
substantive input from him regarding the building of the
program model .

The purpose of the first session was to build the program
logic model. It consisted of a three-stage process that
focused on slowly refining the model. Participants were asked
to provide the following information:

1. Please list the key components of the ISVC program-

- from the first program activity by staff to the
final closing meeting. Your responses are not
confined to any particular source of information.
You can draw from knowledge of the funding proposal
and/or your experiences. You are encouraged to
confer with each other. Total agreement is not

necessary. Any disagreements will be noted and
indicated in the final model.

39

2 . For each key component just mentioned, list the
activities and resources necessary to accomplish
the activity. Indicate what event determines that
the component has been successfully completed.

3. For each key component, list the potential
barriers, methods to reduce barriers, possible
events that the staff had no control over, and any
spin-offs.

The entire session involved refining' the requested
information, and the final results were developed into a
program model. The model was then distributed to participants
for their personal review prior to the second session.

The second session involved the same group of
participants, and consisted of two separate discussions: 1) a
review of the program logic model: and 2) a mock utilization
exercise. The review part of the session was an opportunity
for participants to suggest changes in the model as a result
of their reflection process conducted outside of the group.
Participants were also asked to respond to any items found in
the document analysis and interviews, as presented by the
moderator, that suggested additional interpretations of the
program than found in the model. Finally, participants were
reminded of the third review session in which they would have
an opportunity to respond to the evaluation strategies
proposed based on the program model, and to critique the EA
process as a whole.

DQQQEQDI__AEQIY§1§- Patton (1990) regards program

documents as an invaluable evaluation tool. He describes two

main roles for document analysis: 1) as a basic source of

40

information about program activities and decisions: and 2) as
a source of questions that the evaluator can pursue in
interviews with.participants and stakeholders (p.233). While
documents can be subject to interpretational bias, they are
not as likely to be influenced by the evaluator’s bias as are
observational and interview data (Smith, 1989). While access
to program documents is often a concern (Smith, 1989), this
did not happen in the current study. As the program.manager,
the writer was responsible for managing all documents
associated with the projects.

Document analysis was used to review the program logic
model designed by the ISVC staff. The documents reviewed
include the program proposal, staff meeting minutes and
agendas, seminar outlines and notes, evaluation reports, and
correspondences with the program funder. Any discrepancies
that occurred between the documents and the model were brought
to the attention of the staff during the second focus group
session. Participants were asked to reflect on the
discrepancies and determine if there was a need to modify the
model based on the document analysis.

W- The
primary reason for interviewing stakeholders and.participants
is to obtain program information that can be used at a later
date to influence decisions related to the program (Smith,
1989) . Interviews help to bring out common themes relevant to

the program of which the program staff may be unaware, and to

41
help accurately construct the program logic model (Mayeske,
1991).

Interviews for the ISVC were originally conducted in the
summer of 1990. The original interviews were developed and
conducted with a different research question in mind, one that
pertained to the effects of the ISVC experience on students’
cross-cultural knowledge. Consequently, the interviews may be
considered a subset of the document analysis process with
information specific to students’ and stakeholders’ perception
of the program.

The original set of questionnaires was developed by the
writer and the program director, a faculty member at MSU. A
student, her/his volunteer supervisor, or her/his academic
advisor, were interviewed using three separate interview
forms, one for each specific group of interviewees. The
questions relevant to this study were used to determine the
interviewees’ knowledge of the program, for the volunteer
supervisor and the academic advisor, and the interviewees’
recommendations on program design. The questionnaires are
presented in Appendix Five.

Sixteen students were selected as possible interviewees.
Students had to match three criteria to be considered for
interviews. First, the student must have attended at least
one pre-field seminar, and continued to express interest in
the volunteer experience. Second, a student must have

attended at least one in-field seminar, and maintained contact

42
with the ISVC staff. Third, a student must have volunteered
at least four times over a time period of at least one month.

The sixteen students who possessed the necessary
attributes were ordered alphabetically and assigned numbers
one through sixteen for the purpose of a simple random
selection. Using a table of random numbers, the students were
re-ordered into a list of sixteen. The list was divided into
three sets. The academic-adviser and agency-supervisor
interview groups had five persons. The student group had six
persons. The extra person was assigned to the student group
due to the special emphasis of original study’s intent, i.e.,
the impact of the experience on the students’ cross-cultural
knowledge.

Interviews generally ranged from 1/2 hour to 1 1/2 hours
in length. Interviewees’ responses were written down by the
interviewer during the interview itself. Questions were read
directly from the interview guide, and interviewees were
prompted, when necessary, to explain statements not understood
by the interviewer.

Although Smith (1989) recommends that the interviewer not
be associated with the program, the interviews were conducted
by the graduate assistant working with the ISVC. One initial
intent of the interviews was to generate information for
program planning. Because programming activities were a.main
responsibility of the graduate assistant, the decision was

made to have him conduct the interviews. As the interviews

43
are used as a subset of document analysis in this study, any
interviewer bias regarding program design issues in the
interviews can be compensated for by the focus group meetings

and other document analysis results.

71L!1,77!o ,; ;_ o, o ,1 3‘. _:_o i:?‘:_:'.'.3!

Several steps were taken to enhance the utilization of
the EA process. The purpose of this study was to use EA to
suggest possible evaluation strategies and to critique the
process. The efforts taken to enhance utilization can be
critiqued only in the sense that they facilitated the EA
process. It will not be possible to determine their
effectiveness in actually promoting utilization.

The first step taken to enhance utilization was a mock
utilization session conducted during the second focus group
meeting. The purpose of the mock utilization session was to
determine what the stakeholders-- the EA team --intend to do
with the results of any evaluation efforts, and what
information was most relevant to them in their current work.
The following questions were posed to the group:

1. If the ISVC, or a similar program, were to
continue, what information would you be most
interested in learning about to help you with
future programming tasks?

2. What information about the ISVC ‘would. be :most
helpful to you in other work in which you are

currently engaged?

3. In what ways will you use the information, (e.g.,
for future funding proposals, student recruitment)?

4.

44

In what form would the information be most useful
for you, (e.g., reports or graphs)?

The second step taken to enhance utilization was a review

session at the end of the EA process with the EA team. The

purpose of this session was to review the entire EA process,

present the results of the EA, and solicit feedback from the

EA team regarding their impressions of the process and the

results.

At this time, the EA team was given an opportunity

to rethink any issues related to utilization. The following

questions were posed to the EA team:

1.

Now that you have seen the proposed evaluation
strategies, do you think that these methods will
produce information that you can use?

Are there additional issues related to utilization
that were not apparent during the mock utilization
session?

In what ways has the EA process affected your
thinking about the ISVC?

Now that we have reviewed the EA process, please
critique the value of this process in evaluating
experiential programs such as the ISVC.

As discussed earlier, Smith (1989) suggests five
choices for determining utilization. Three of
these choices are relevant in this study:

1. Decide to evaluate the program
2. Decide to take no further action
3. Do not decide and ignore the EA.

Which choice do you feel is appropriate for this
study?

45
Link to the Study Objectives

As stated in Chapter One, the objectives for this study
are:

1. Determine the usefulness of the evaluability
assessment process in suggesting appropriate
evaluation methods for experiential learning
programs.

2. Suggest guidelines program staff can use to adapt
evaluability assessment to fit individual
needs/programs.

3 . Highlight the utilization enhancing characteristics
inherent to evaluability assessment, and suggest
additional guidelines.

Although each activity described in this chapter
addresses each research objective, the study can be divided
into two phases that answer different aspects of the
objectives. The first phase is the actual conduct of the EA.
This involves the first two focus sessions in which the
program logic model is developed and utilization needs are
determined, and the analysis of program documents and
interviews is conducted. During this phase, the evaluation
methods are proposed. Completion of this stage provides
information for research objectives 1 and 3.

The research objectives are reflective in nature. Their
purpose is to facilitate review of the EA process, and its
results, as a mechanism for evaluating experiential learning
programs. The appropriateness of the evaluation methods are

dependent on the EA.team’s satisfaction that the methods will

produce utilizable results. The objectives help to critique

46
the process of developing evaluation methods acceptable to the
appropriate stakeholders-- in this case the EA team. The
objectives are not meant to critique the actual evaluation
methods per se.

The second phase of the study is the actual reflection
process. It consists of the review session in which the EA
team critiques the appropriateness of the evaluation methods
and the EA process in general. This phase also consists of
the writer’s review of the EA.process, and a final assessment
of the use of EA with experiential learning programs. This
phase addresses each research objective, and forms the

majority of Chapter Five.

 

 

47

CHAPTER 5

STUDY RESULTS

Overview of the Methodology
As described in Chapter Four, the study methodology
consisted of three main activities: focus groups, document
analysis, and stakeholder interviews. The results of the
study are presented in this chapter, and are discussed in

light of the research questions in Chapter Six.

Focus Groups

W

Three focus groups were conducted over a three-week
period. Each session was two hours in length, and was
organized in three sections: an opening presentation, a
description of the purpose of the session, and a group
discussion regarding the main topic. The results of the first
and second sessions were summarized and presented back to the
group at the following session for further discussion.

Participants in the focus groups consisted of university
faculty and administrators, and local volunteer agency
representatives. Due to scheduling conflicts among
participants, attendance varied throughout the three sessions.

Those who were unable to attend the first session were sent a

written summary of that meeting.

Discussions throughout the first two sessions were
dominated by the participants with the focus group leader
participating only when necessary. The final session followed
a review format.innwhich.the focus.group leader summarized.the

two previous sessions for final feedback from participants.

Focus Group Results
EQQQ§_§£QQR_I

The purpose of the first focus group was to introduce the
main concepts of evaluability assessment, and to build a
program logic model of the International Students Volunteer
Corps (ISVC). The first draft of the program logic model is
shown in Figure 5.1.

Participants were asked to describe the key components
associated with the events, activities, resources, indicators,
and barriers of the ISVC. The program logic model, as
presented in Figure 5.1, summarizes the discussion. The first
row of events largely follows the program proposal with the
addition of the first two stages-- develop program proposal,
and develop program with stakeholders. The activities,
resources, indicators, and barriers connected to each event
help to explain its role in the program.

The activities and resources rows indicate what was
necessary to complete an event. Constructing these two rows

required a fairly straightforward recall of project history.

48

. 3.8.5 8...: on... use... .96. ..m 2...... _

 

 

58.8.5382

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3158...... Eggs-.33.... 32.8.... 8.3.3..
get—use! .gggg 9.8.881..- Eegaag. (mu—(2.0.5.;
axis-2.8.8 gains-.3855 8.8.8.3 5.5.8.8.... 5.2.333...
81.88.35 . 3.8.38.3. 5.8933131333. 3...... East...

8335:... «2.25... 8%..185888341 25:68.... €33.53]... se.-8......
38.33 89.1.51... 33:51:88.3... 318...; .326. 8381.3...
Ila-la; gal...— bialco grass-3 8.5.8..

gain—Boo haw-05.38.... «Quasi 88.858; 38.518.39.81 Julio—.8538. 3.3.8333!
858.83.... 88.5.8.3 Eli-.0 3.1.3.28... plane-.83.... sue-sis... , 45.1.8.3.

. 38...! 8......- .38....

. 53...... 5.26.8.8... 3.5.5338 .....B..e 893388.33 32.61..
5.2.8.... 835.39.... .88....9... 8.2.8.5.... assaiag 33.3.5335. 5.33.3
633.888.: gut mains—26.. b.5883! 3.88.3380 . e932 58:026.}..—
.5383... .838 Sign... 58...... 8.... 35:38... .38.... a: a... .838 a... 888. 33
828...... 3...... 8...... 8...... 5.35... 18......
5.8.8.. .2680 22...... .938 22...... .888 32...... .828 38...... se.... as»... 8.2.8 3...... 8.26..

 

 

 

 

 

 

 

 

50
For instance, in order to develop the program proposal, it was
necessary to use previous experiences from similar programs to
develop a program plan and a set of objectives. The resources
needed to conduct the symposium included outside funding and
the input of those involved in the ISVC.

The indicators and barriers rows involved more reflection
on the implications of each event for subsequent events, and
an actual evaluation of how successfully an event was
completed. Completing the row of indicators required the most
prompting from the focus group leader. The resulting list of
indicators include both the goals of the program, as in the
first event's indicator of better linkages on- and off-campus,
as well as the completion of intermediate steps, such as the
number of students recruited during the recruitment stage.

The information in the last row--barriers--provides
information on the difficulty of moving forward from one event
to another. Participants spent a significant amount of time
discussing the barriers associated with student recruitment
and its effect on several events following the recruitment
stage. A similar amount of time was spent discussing the
problems encountered in matching the students with the correct
volunteer placement, and preparing agency supervisors for
international student volunteers.

Figure 5.1 displays that, for each event, there were
associated activities, resources, indicators, and barriers

that affected not only the particular event, but also

51

subsequent events. A barrier for one event could affect a
later event, as well. For example, the event of developing
the program with stakeholders had implications for conducting
the in-field sessions, and the potential barrier of volunteer
retention. Thus, the program logic model reveals how events,
resources, etc., are linked throughout the project.

Participants focused mainly on the first half of the
project's lifespan: those events starting from«developing the
program proposal to conducting the pre-field sessions.
Although the session focused on the first year of the ISVC,
participants used the experiences from the second year to
contrast the first year and bring out the unique elements in
it. They used similar experiences from other projects to help
emphasize different aspects of the ISVC, as well.
Participants drew upon their particular professional and
academic backgrounds to highlight different aspects of the

project.

EQ§B§_§IQBR_II

After the first session, the focus group leader
summarized the discussion in a chart, and then distributed it
to the participants for their review before the second
session.

The purpose of the second session was to review the
program logic model developed previously, make any necessary

changes, and conduct a mock evaluation utilization session.

52

The mock evaluation exercise was the primary objective of the
session. However, the participants felt that an analysis of
the program logic model was crucial to any further activities.
The participants proposed several additions to the model. The
revised model, Figure 5.2, includes two additional rows for
goals/outcomes and assumptions. None of the other rows were
modified from the original program logic model.

A row for'goals and desired outcomes was added at the top
of the model. As the events and activities flow directly
from the goals, the participants felt it was necessary to add
the row to explicitly state them. The goals arose mostly from
the program proposal, but also revealed additional goals that
developed throughout project implementation. Thus, the row
results from a review of project history and a reflection on
the evolving nature of the program over time.

The participants felt that the model focused on process
to the exclusion of the underlying assumptions, thinking, and
rationale that guided the choice of program events and
activities. The assumptions form an important part of the
cause-effect thinking that influenced the choice of events,
and link the rows of events and activities with the
goals/outcomes row. The assumptions were also seen as
revealing more about which indicators were chosen to show the
completion of an event. The barriers often reflect where the

assumptions were incorrect.

53

.. 8.38 8.8:. 9.8.. 2.32.. 96. um 2...»...

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

i 8 325.38. ,
.3888 2.... , 8.18.5... .82.. .83 838...... 6.... 8.5.. 83......
. . r .. .1 .ggig 9.2.33.6...- Eognilifi $3.5
32.1 .2. 582.6 .82... 9.23.... $32.28... :5... 88: .2. 2o .1 8...... .8 2o 2 3.82.. 8..
- a V _. _ .
.8822... , 8.6.8.
5.... 68.8.5 28. .38.... 88... 8.18 2 8.... 2 9....- ..282 5.8.59
80.5.5. .8. .822... .1 8.1.9.5.. .e .8...» 4.82.8 22.8... ... .35.... .o>m. 2 82.8.... 5.8.9. 2......8 =8 2...
33.2.. 3825 8 .32... 2.28.. .o 8.. 4.3.8... .8... a... 88......» 3.5.8. .8 2.8.8.. .28.. o>m. 8 89...... 8.8..
_ __ __ __ __ .—
353! in 35.8... >3...»— bEEEoo .223an b.5553 8.5. .65..
.8335 Juan—Eco =3. U>m. 5.5.... 2......8- 60.9.8»- .ooeog E- 32. 3.2.3.8.... .8533 6.3.8. at... 5&3 2.3....
3.95 8.29.2 8088...... .2825 5.8868 2.. .82... 5.8.... 2...... 2.. 88.5 23...... 5.8... 4.52.... 8......
_ __ __ _ _ __ __
.359.
3.3.559... 95...... snug 3.8.8....» 88.5.9» 2.5.1. .538
.56 an... .o nave-.3892. . 080.25 3.50:9 9.60....0 .8 13:35.... 4.552 8.39...
8. 88.8....» o>m. 8...... 2.... 2 8 2 38.35.. , 8. :82... 9.1.8.. .._o.. 283 as... 02.8..» 8 8.26.. o>m. u... 8.
52. 22.3 2...... 88.8....» 382.. , o. 22.2.. 2....) 2.82.5. , $288.... 2.8.. 2:8 9.8.88.3. . .389... . 8.8...
832...... 8.8 2:3 «.825 , 8.. 283 3828 , .9893: 28... 5825 .o .8... 3.2.... 8.3.8 32>»... ,
u _ _ — _ - __
3.8!... 2.89 8.830- 83.3..
5.3%...»- 8...2 in! 3.25:. .5358. =- 2 g d 83.353 9:83! 5.3,
8.8.2.8.... 8.83.3318 8888......- §§szuiato§8¢ 53.8.8.3. 5.38125...
6.33355... $858....— nfiieog ail-8.3918. €353.38 93.385 suntan-2.6....
.8289... .3680 825.55.... «.82.: 8.... 8.5.5., 18...... 8: as... §u8o .8... 338. 31h
— _ . _ _ _ — u
8288 2.. .838. .818. .818. 98.88.... .2... 39......
5.8.8.. .888 22.-.»... §e8o 22.... .2680 22.5... .888 :82... .28... 552.. 8.25.. 352.. 8.26..
= _ _ __ __ _ _
3.8.... 28. Base... .8123»... am: 52...... 2..
805.898, a. 5.88.5.3 £633 20.. 2.5353 55.3 one...» 9.2....
2.2.2.1.... n.don ack— ofifi 603.895, 3.332 Eu... .20.... E... 30.3.... 52.. 3:92.. .2329... 893.... 838....
8...... 2 5.22... 22...... u... 9.82.. 822...: 2.. 888...... 598.. 2...... a... 2...... .822... $8.8. 53.8.5
can 826.598 Babe... u 0.0353 .2825 868... .52.... a Save...» 8.....3 5... =.. .o 38.. u... 38... .. 3.2.3233...
9.3.5.550 bingougm .2.- o.a....ua... .0. 8.5.5.. «in... .o 9.05 a .353. .2. .589... 3.33. .286...» 3:15....

 

 

 

 

 

 

 

 

 

n50.333—

82-88.

in

Slog-63.00

54

The second activity of session two was to conduct a mock
evaluation utilization session. The purpose of this exercise
was to have participants recommend questions that they felt,
if answered, would provide information valuable to not only
the program itself, but also to subsequent programs with a
similar focus and audience. The recommendations were grouped
into the three categories:

1. Issues in a Project's Lifespan?
- What is the link between objectives and
decision-making?
- What are the link of objectives to
activities?
- Which activities contributed most/least to
project outcomes?
- What, if anything, happened to agencies
positively or negatively?
- What, if anything, did the students learn
from each other?

2. Characteristics of Participants
- What role does motivation play?
- What motivates students to stay/leave?
- What role does money play (what role would
it play) to begin participation? to maintain
participation?
- What elements/criteria make up the decision
to join and stay for different groups?
- How will participation differ among
different student groups?
- What will a social market study of
international students reveal? What effect
would that have on recruitment and retention?

3. Programming Issues
- How does one use the results of a social
market study in future programs?
- How can American students fit into similar
programs?
- What will a social market study of agency
groups reveal?
- What is the difference between agencies
having experience with international students
or clientele and those without such
experience?

55
- Is volunteering an effective mechanism to
enhance knowledge of American culture?
- What other mechanisms are effective?
- What impact does volunteerism in the
international context have on students'
experience in an ISVC?
- The ISVC is an American process: how can it
be developed as a cross-cultural process?
- What are the effects of cross-cultural
dynamics on learning?

One issue transcended the three categories: the need to
know more about the audiences that would participate in
projects similar to the ISVC. Focus group participants were
most interested in knowing more about students and agency
personnel. Information on students was needed to better
conduct recruitment efforts and increase retention.
Information on agency personnel was needed to better match
students to a volunteer site and orient both the student and
agency personnel about the cross-cultural implications of the
experience.

A second issue arising from the questions was: How could
the information or lessons learned from the project be used to
improve other programs? Several questions arose regarding
whether a different mechanism, other than the ISVC, would help

international students learn about American culture.

EQ§2§_§IQBD_III

The purpose of the third session was to review the
program logic model for any final revisions, and to review'the
suggested evaluation strategies for the evaluation needs

suggested in the second session. The final review of the

56

model provided a last opportunity to reflect on the ISVC.
Although no changes were made to the model, participants
reiterated the importance of having a row placed in the model
that explicitly stated the assumptions behind goals, events,
and activities. Participants stressed that assumptions lead
to the choice of the particular events. The discussion also
reinforced the importance of selecting agencies capable of
hosting an international student volunteer.

Participants raised long-term programming issues, such as
the need to determine if volunteerism was the most effective
mechanism to accomplish the program goal of helping students
learn more about American culture. Finally, they discussed
the issue of program sustainability, and the importance of
support from the University and the community for such
experiential programs.

The focus group leader then summarized the evaluation
needs/questions posed by the participants in the second
session, and presented several examples of the types of
activities that could be taken to look at those questions.
The recommendations are summarized below:

1. Issues in a Program's Lifespan

Design and process questions:
- Conduct a more extensive evaluability
assessment process through a consultant or
dissertation study

Outcome questions:
- Initial interviews with students and agency
supervisors in the ISVC suggest insufficient
information for further study. Project
activities that contributed mostly to project

outcomes would be studied in an extensive EA
as recommended above

57

2. Characteristics of Participants
- Encourage thesis or dissertation research
- Develop a sample from.a social market survey
possibly conducted by an advertising,
business, or psychology student
- Conduct additional focus groups or surveys,
or collect information from other accessible
university records
- Compare students across categories (i.e. ,
years in school, country) for recruitment and
retention data
- Administer pre-post tests during a program
3. Programming Issues
- Utilize many of the same activities from
above
- Use results of above to conduct EA as a
program design tool
- Stakeholder interviews should target
individuals who can address issues of
experiential learning settings
- Many issues of adapting an American
tradition within a cross-cultural experience
can be done through applied research
- Conduct a "lessons learned conference" on
experiential programming for international
students and cross-cultural learning

Several factors influenced the recommendations. One,
funding available to conduct further evaluation efforts likely
would be limited. Consequently, any recommended evaluation
strategies should recognize this limitation (e.g., using
doctoral students to conduct evaluations to fulfill their
research requirements). Two, the ISVC had been finished for
over a year at the time of the focus groups. Thus, the
results of any evaluation activity should have relevance for
similar activities in the future that could benefit from the
lessons learned in the ISVC.

The focus group discussions indicated that a more in-

depth evaluability assessment process could explore issues

58

such as student recruitment in further detail with benefit to
several units at the University. The discussions also brought
up the need for increased networking among units during
program development. Conducting applied research with
participants during an actual program could provide
information on retention and the constraints encountered by
participants and staff.

The final segment of the session focused on a review of
the evaluability assessment process. Participants suggested
that a primer be developed for use by other groups constrained
by resources and/or evaluation expertise. The EA process
provided an opportunity to reflect on a project outside of the
normal day-to-day activities. Participants felt that staff
considering similar projects would benefit not only from the
results of the study, such as the program logic model, but
also from knowing about the reflection process that was
necessary to develop the model. It could be particularly
helpful during program development. Finally, most of the
participants felt that the EA process was a helpful reflection
process that provided valuable information for future
activities, and emphasized the importance of developing a
module that other program staff could use to review their own

programs.

59
Document Analysis
The document analysis process consisted of a review of
the program proposal, reports to the funding agency, notes
taken at meetings, and articles written by staff and students.
The purpose of the document analysis process was to determine
the level of consistency between what was written about the

project and the program logic model developed by the focus

group participants.

 

The program logic model parallels the program proposal.
The reports to the program funder, NAFSA, elaborated on each
stage of the project and described any variations from the
proposal. For example, a major variation occurred during
student recruitment. The original program brochure required
students to’submit a letter of interest along with a letter of
support from their advisor. Students were then to go through
a series of interviews before being accepted into the ISVC.
However, a low student turnout convinced the program staff
that the requirements were to strict and recruitment efforts
changed to open participation up to anyone who was interested
in the program. Changes, such of the type described here,
were communicated to the funder to inform them of lessons

learned throughout the project.

60
W

Notes from the staff meetings generally parallel the
discussions on the activities, assumptions, indicators, and
barriers listed in the program logic model. A major part of
each staff meeting, conducted nearly every Friday throughout
the project, was devoted to planning ongoing activities, but
also involved discussions on barriers, such as student
recruitment. Towards the end of the project, the meetings
more on planning towards the future.

During the focus groups, participants remarked that while
the Friday meetings afforded an opportunity to reflect on the
project, the meetings may not have allowed the quality time
that was necessary to look at the program "from a distance."
They suggested that the EA process would have helped program
staff had it been conducted during the program as a formative

evaluation tool.

es W Stud t

The articles ‘written about ‘the ISVC fall into» two
categories, descriptions of the program and student, and staff
reflections. Descriptions of the program were written.mostly
by the graduate assistant in charge of daily programming and
closely follow the program proposal.

Articles written by the staff often discuss the
implications of the ISVC based on both years of the project.

Thus, the articles go beyond the time frame of the program

61
logic model, but reinforce the importance of recruitment,
volunteer .placement, and close working relations with
community agencies. Staff articles more closely reflect the
evaluation questions that arose during the mock evaluation
utilization exercise during the second session.

Student articles provide information regarding the
barriers encountered in volunteering, as well as indicators of
the program's goals. Each student stated what they had
learned from both the classroom sessions and the actual
volunteer experience. The experience provided them an
opportunity to reflect on their own countries, and the
possible impact of American-style volunteerism, as well.
While the students' articles provide an account of their
experiences, they provide little in the form of analysis on
the program process that could be used to modify the program

logic model .

Stakeholder Interviews

The stakeholder interviews were originally conducted with
a different set of research questions. Consequently, they
provide little information relevant to the program logic
model. The interviews are more outcome- than process-
oriented. However, during the final focus group session,
participants recommended that any further study of the ISVC
include interviews with those stakeholders not interviewed

during the initial interview process, which would include

62

mostly agency supervisors.

Conclusion

The evaluability assessment process conducted for this
study consisted of three focus sessions, document analysis,
and stakeholder interviews. Participants in the EA study were
able to develop a program logic model which they felt
accurately described the ISVC, and suggested possible
research/evaluation questions for further study. The
participants felt that the EA process was an effective means
of reflecting on a program, and suggested that it would be
valuable both as a formative evaluation tool and as a program
planning tool for others starting similar projects. Given
these conclusions, the results of the EA process in light of
the research questions, and the implications for the future

use of EA in similar programs, are considered in Chapter Six.

CHAPTER 6
SUMMARY CONCLUSIONS, AND RECOMMENDATIONS
Summary of Study

WM

Program evaluation involves the measurement of program
performance indicators, and the testing of the logic behind a
program that links resources and activities with expected
outcomes (Wholey, 1987). Wholey (1987, pp. 77-78) identified
four problems that prevent the utilization of program
evaluation:

1. Lack of definition of the problem
addressed, the program intervention, the
expected outcomes of the program, or the
expected impact on the problem addressed.

2. Lack of a clear logic of testable assumptions
linking expenditure of program resources, the
implementation of the program, the outcomes to
be caused by that program, and the resulting
impact.

3. Lack.of agreement on evaluation priorities and
intended uses of evaluation.

4. Inability or unwillingness to act on the basis
of evaluation information.

Evaluability assessment (EA) is a tool that may help
clarify a program's structure, establish the plausibility of
its activities in reaching intended goals, determine the
evaluability of the program, and enhance the utilization of
the evaluation results (Smith, 1989). EA was first developed

63

64
by Joseph Wholey and his associates at the Urban Institute in
Washington, D.C. (Nay & Kay, 1982, Smith, 1989).

EA involves gathering an evaluability assessment team
that can help gather materials and develop a program logic
model that outlines the resources and activities necessary to
reach the program goal. After developing the program logic
model, a decision is made as to what aspects of the program
can be evaluated, and which aspects are most important for

evaluation.

Wish:

Given the limited attention paid to program evaluation
efforts in adult experiential learning programs, there is a
need to explore existing evaluation tools in more detail.
This study used the International Students Volunteer Corps
(ISVC) as a case study in which to apply the evaluability
assessment process. Focusing on the first year of the ISVC,
the objectives of this study were to:

1. Determine the usefulness of the evaluability
assessment process in suggesting appropriate
evaluation methods for experiential learning
programs.

2. Suggest guidelines that program staff can use to
adapt evaluability assessment to fit individual
needs/programs.

3 . Highlight the utilization-enhancing characteristics

inherent to evaluability assessment, and suggest
additional guidelines.

65
W

This study took a multi-method or triangulation approach
(Gorden, 1980) and used three separate sources to build a
program logic model. A program logic model is a process of
determining means-ends, cause-effect relationships in a
program as defined by the program staff (Smith, 1989) . The
three methods were: 1) a modified focus group process: 2)
document analysis: and 3) interviews with stakeholders and
program participants.

Three focus groups were conducted over a three-week
period. Participants in the focus groups consisted of
faculty, university administrators, and local volunteer agency
representatives. The main purpose of the focus groups was to
develop the program logic model and determine the primary
evaluation and research concerns of the participants.

The purpose of the document analysis was to determine the
level of consistency between what was written about the
proj ect--program proposal, reports to funding agency, and
meeting notes--and the program logic model developed by the
focus group participants. The stakeholder interviews were
originally conducted with a separate set of research
questions, and so provided little information valuable to the
program logic model.

The following sections present the main findings of the
study by each research question. Important issues are raised

in using EA as a program review tool. An EA primer is

66
presented as an aid to help other program staff adapt the

process to their activities, as well.

Main Findings
1. Determine the usefulness of the evaluability
assessment process in suggesting appropriate
evaluation methods for experiential learning
programs.

Because the ISVC had been completed for over a year, the
focus group participants focused.more on research issues that
would provide information for future and similar programs,
rather than on evaluation methods. Nevertheless, the EA did
provide participants with an opportunity to reflect on the
ISVC and to suggest critical program activities that require
further research. The research questions were listed in
Chapter Five, and included questions pertaining to student
characteristics and motivation, student and agency supervisor
preparation, and overall program structure.

EA acts as a discussion-promoting tool in that it can
bring staff together to reflect on a project. The process of
building a program logic model requires that program staff
reflect on a program differently from that which takes place
at regular staff planning meetings. It provides an
opportunity to step outside day-to-day programming issues and
look at the program from a holistic viewpoint, and to analyze
the assumptions that guide program goals and activities.

Programs staffed by individuals from diverse academic

backgrounds benefit from learning how their backgrounds

67
influence their assumptions and program decision-making
styles.

The focus group participants felt that the EA process
offered a program evaluation/reflection process that could be
conducted by program staff with little expertise in program
evaluation. The EA process requires very few resources, and
so represents a valuable option for resource-constrained
projects. The last section of this chapter summarizes the EA

process so that it can be used by a variety of project staffs.

2. Suggest guidelines that program staff can use

to adapt evaluability assessment to fit
individual needs/programs.

The EA process conducted in this study represents an
adaptation of EA as proposed by Smith (1989) and Mayeske
(1991). Changes were necessary to fit the needs and
constraints of the study, i.e., time and funding. Many
project staff will face similar constraints. Thus, it is
important to assess the available expertise and resources that
will be available to conduct an evaluation. EA offers a
process that can be adapted to meet the evaluation needs of a
project staff. In this study, the process was condensed into
three focus group sessions that permitted the program
reflection process important in an EA. It also allowed for a
quicker process--one that provided information on the program

to the staff. Program staff interested in using EA should

determine their goals and information needs, and adapt the EA

68
process accordingly.

This study used EA as a program reflection tool for a
completed program. The stage of a program's lifespan in which
a staff use EA will influence how they adapt EA to fit their
needs. An EA conducted while a program is in process will
affect the information and the resulting activities needed.by
program staff. For example, the EA conducted in this study
resulted in researdh questions important to broader issues
involved in designing and implementing future programs. An EA
conducted during a proj ect's implementation may result in
recommendations for specific evaluation tools and activities
necessary to design upcoming events. Program staff should
adapt the EA to fit the stage of a program's lifespan and
their evaluation needs.

An important part of the EA process is choosing the EA
team. The choice of who should be on the team is affected by
the stage of the project’s lifespan when the EA is conducted
and the goals of the EA. The EA team should contain program
staff and stakeholders who know about the project, who have an
interest in the program's success, and/or who wish to conduct
similar programs in the future. An EA conducted halfway
through a program's lifespan may have an EA team composed
mostly of staff and stakeholders who know about a project. An
EA conducted at the end of a project may have an equal mix of
program staff, stakeholders interested in a program's success,

and those interested in conducting a similar program or with

69
expertise in similar programs. The composition of the EA team
should reflect the goals of the EA. Finally, while EA team
members may differ on various aspects of the program, each
team member should be committed to making the EA a positive
process that.benefits the program.staff and.current and.future
stakeholders.

The choice of the EA coordinator should reflect the
information. needs of’ the jprogram staff and any' program
constraints, as well. A focus group leader familiar with the
project may have biases that affect decisions made throughout
the EA. A external focus group leader may be more objective
and able to point out inconsistencies to the EA 'team.
However, an internal EA coordinator may have greater access to
program documentation and be able to develop more trust and
cooperation among the EA team. The decision of whether to use
an internal EA coordinator, as was used in this study, or an
external person will depend on the resources available to the
program staff and their assessment of what will help them

achieve their goals for the EA.

3. Highlight the utilization-enhancing
characteristics inherent. to’ evaluability
assessment, and suggest additional guidelines.

The findings for the two research questions listed above

point to the ease with which EA can be adapted to fit a

particular program staff's needs. As an evaluation approach,

EA offers an alternative that individuals can adapt to their

70
information needs, rather than using a method which is
available, but does not directly address program needs. Using
a method that can directly address program needs helps to
insure the relevance of the information and its utilization in
future activities.

The importance of the EA team was discussed above.
Carefully selecting team members can enhance the utilization
of the EA results. Stakeholders who have a vested interest in
continuing the same or similar programs are more likely to
participate fully in the EA process, and use the results.

Activities such as mock utilization sessions focus the
evaluation on those questions that will provide the most
useful information for stakeholders. The process of building
the program logic model may provide useful programming
information that will require no further investigation. After
developing the program logic model and reflecting on the final
product, the EA team may be able to list those questions that
are most important given constraints, such as time, personnel,
and funding.

Finally, the way in which the EA results are packaged can
influence their utilization. Long written reports may not be
read by those who have limited time, while information
presented in charts allow a potential user to quickly review
key points. Evaluation summaries can be packaged to provide
information to potential users conducting similar projects.

Handbooks that highlight essential points that cut across

71
programming areas provide resources that can be used beyond
the life of the original project. The EA guidelines presented

in the last section of this chapter provide one example.

Recommendations
Four practice recommendations arise from the evaluability
assessment (EA) process conducted in 'this study; Each
recommendation is geared towards enhancing the use of EA.

1. A goals/outcomes row and an assumptions row should
be added to the program logic model. The addition
of these two project descriptors will help the EA
team to more clearly understand the thinking that
guided.program.activities, and possibly contributed
to program barriers. If a primary goal of the EA
is to list the assumptions behind goals and
activities, an external EA coordinator may be the
most helpful in walking the EA team through the
program logic model.

2. A resource guide explaining the EA process is
needed for the resource-constrained projects
mentioned above. A resource guide should be brief,
but provide sufficient description of EA and
indicate the steps necessary to conduct a
successful EA. .Appendix Six contains an example of
an EA primer for small-scale projects.

The primer in Appendix Six will be distributed to a
group of potential users for their review. The
targeted group will include a faculty member
involved in evaluation, university administrators
involved in small-scale projects, and individuals
working in community agencies. The purpose of
distributing the primer is to have the group
recommend modifications in the primer that will
increase its ‘usefulness to resource-constrained
project staff.

After revising the primer, it will be distributed
as part of a publication series through the
Department of Resource Development. The
publication series reaches extension staff
throughout the state of Michigan, who work often
with small-scale projects. The publication will be

72

distributed to the National Association for Foreign
Students Affairs (NAFSA), as well.

The ease with which EA can be adapted to fit the
particular' needs of a program staff makes it
especially’ useful for application in resource-
constrained environments. Programs with minimal
funding may be hard pressed to reserve funding for
evaluation efforts. EA represents an effective
method of evaluation for projects with personnel
and monetary constraints.

Recommendation 2 listed a process for revising the
primer in Appendix Six to be used by small-scale
project staff. In addition, a training seminar
will be developed to help project staff use EA with
their programming efforts. Often, project
personnel may avoid program evaluation because some
evaluation methods involve a strong reliance on
quantitative tools. Evaluation is often seen as a
task requiring an individual with special skills.

The training seminar will consist of three
components. The first component will explain the
process of using EA as outlined in the primer. In
the second component, the facilitator will lead the
group through a practice EA of a hypothetical case
study. The practice session will highlight the
major phases of EA. The third component will be a
review of EA in light of the practice session to
discuss any questions or concerns, and talk about
the use of EA in the participants' projects.

The training seminar could be conducted as a half-
day program, or conducted over several sessions,
depending on participants’ time constraints.
Individuals who participate in the sessions could
train others in their organizations. In this way
the seminars can be viewed as a training of
trainers.

While the training seminars will help to inform a
wide range of program staffs about EA, it is
important to develop training materials that
program staff can use to train new staff members
and review periodically. Several educational
technologies exist that can serve this purpose.

The primer can easily be packaged with a video that
contains the most important aspects of the training
seminar. The video would summarize EA, show
highlights of a group conducting a practice EA.with

73

a hypothetical case study, and review the most
important parts of the practice EA as a final
review. Increasingly, hardware is being developed
for the personal computer that allows the editing
and production of videos. This technology can be
used on a regular basis to update the training
video as more experience is gained with EA.

More and more college and university campuses are
being equipped with satellite uplinks that allow
them to broadcast classes, seminars, and
conferences to other locations on a regular basis.
This form of distance learning could be used to
train groups in the process of EA. A project staff
or community group interested in using EA could
arrange to have an EA trainer present the material
through the satellite system at a designated time,
and use a conference call process to conduct a
question and answer period while still on camera.

Expert systems represent another avenue to extend
EA to small-scale projects. An expert system is a
computer program written to provide training or
information on a particular topic. Users respond
to a series of questions posed by the program. The
program then applies the answers to a database of
information.and.provides users with recommendations
for next steps. An EA expert system would help
users learn about EA and recommend a process to
follow in conducting the EA.

The technology chosen to present EA depends on the
circumstances of the program staff requesting
assistance. An important characteristic of EA is
its ease of use by staff without extensive
evaluation experience. The educational technology
chosen to present EA should not interfere with the
primary purpose of helping program staff to better
design, modify, and learn from ongoing programs
with their clientele.

APPENDICES

APPENDIX ONE
OUTLINE OF ACTIVITIES

August 1989
Program Planning
- focus group meetings with agency supervisors and
students
- development of contact agencies
- development of pre-field seminars
- development of recruitment plan

September 1989
Program Planning
- pre-field seminar design continued
- contact agencies continued

Recruitment
- Welcome Week fair in the International Center
- article in campus newspaper, State News
- brochure developed and distributed
- phone and personal contact of prospective students

October 1989
Recruitment
- information meeting for interested students
- phone and personal contact continued

November 1989
Recruitment
- membership/information meeting

December 1989
Program Activities
- first meeting with advisory committee
- evaluation of recruitment phase
- program introduction meeting with students
- Christmas get-together with students

January 1990
Pre-field Seminars

- introductory seminars on volunteerism and cross-
cultural communication

- beginning work on volunteer placement

74

75

February - June 1990
In-field Phase
- volunteers at placement site
- monthly seminars

Program Activities
- evaluation of pre-field seminar
- second meeting with advisory committee
- planning process for symposium

July - September 1990
Program Activities
- conduct symposium
- evaluate first cycle of ISVC

ISVC Cycle II
- design and conduct recruitment
- revamp and organize new'weekend.pre-field seminar

APPENDIX TWO

IN-FIELD SEMINAR OUTLINES

SEMINAR TITLE: PRE-SEMINAR ORIENTATION

DATE/TIME: NOV. 22, 5:00-7:00, Rm. 201 International
Center

PACILITATOR: STA?!

OBJECTIVE: By the end of this session participants will
be able to describe, in group discussion, the
ISVC, their roles and activities as members of
the group and the timeline for activities.

ACTIVITIES:

I. W
1. Get to know each other
2. Learn details of the ISVC
3. Learn of timeline
4. Express any misgivings/hopes
5. Identify next meeting time

II. ’ ' n
Each participant identifies his/herself
stating: 1) Name 2) Major 3) Nationality 4)
Why they decided to participate. 5) What
previous volunteer experiences have they had.
6) What are their interests. 7) What do you
hope to get out of this.

III.

W
W - By this point the
participants may have attended the October
social and/or read the proposal. during this
process of clarification, it may help to have
a participant explain the program as he/she
knows it. The staff and other participants
can fill in the detail. Having a participant
sum it up will help us to know if the group
has really got it.

76

IV.

VI.

VII.

77

- Summarize role of
committee and point out members on roster
sheet.

- s
1. Program Benefits
2. The Reader and Curriculum/Timeline
3. Staff Expectations - seminar attendance....

4. W: a one page description
of the ideal internship. Areas to focus on
might be:

a. job and agency description

b. the skills a person would learn

c. the skills a person would bring

d. the benefits they would get personally
e. the benefits they would bring back to
their country or future work

f. set-up next meeting for processing
exercise.

We should make specific references to the
country/region presentation. In the
presentation they may wish to cover:
1. Short description of country and people
2. Typical trouble spots or ease which
people encounter when coming to the U.S.
3. What are some of the typical reactions
people get from U.S. residents
4. Reference to the day's class as it
bears on the particular country
5. Map - refer to map of the world,
indicate logical groups and identify
group for the December 8 meeting.

W
1. Looking EgrgardZApprehgnsign List
Participants make their own list of
looking forward/ apprehension ideas ,

compare them with others in small groups
and then share them ‘with the larger
group. The exercise helps to get out
misunderstandings or concerns which the
staff can work with right then or
throughout the course of the seminars.
Bee attachment.

W
1. Identify next meeting time - December?

78

SEMINAR TITLE: WINTER BREAK

DATE/TIMI:

PURPOSE:

ACTIVITIES:

I.

II.

III.

IV.

DECEMBER 18, 5:00-7:00, RM. 201 INTERNATIONAL
CENTER

By the end of this meeting the staff and
students should have a better idea of the
students' goals for the program, and what
modifications may be needed in the curriculum.
In addition, each activity should begin to
promote greater communication between all
involved.

1. Discuss the results of the internship
exercise.

2 . Conduct the Looking Forward/Apprehension
Exercise.

3. Solicit ideas on a group T-shirt.

8 ns i r
The results of the internship exercise serve
three functions: 1) Gives each other a

sense of what the other participants want to
get out of the ISVC, and to see where they fit
in based on their replies. 2) The items they
stated as important to learn can be compared
with our curriculum to see if they want to
suggest changes. 3) Allows them to think
about their ideas on their placements and may
give Bassey more information.

d

Participants make their own list of looking
forward/apprehension ideas, compare them with
others in small groups and then share them
with the larger group. The exercise helps to
get out misunderstandings or concerns which
the staff can work with right then or
throughout the course of the seminars.

1. There's gotta be an artist somewhere in the
group.

s
1. First Seminar - Monday, January 8, 5:00-
7:00p.m., Rm. 201.

2. Country/Region Presentations by:
Malaysia/Mainland China

79
SEMINAR TITLE: THE CONCEPT OF VOLUNTEERISM
DATE/TIME: WEEK OF JAN 8, 5:00-7:00, Rm 201
FACILITATORS: Staff
OBJECTIVES: Participants will be able to:
1. define and contrast the related terms
voluntary' action, 'volunteering, and

voluntary association.

2. describe the role of the United Way in
the Lansing Area.

3. describe a case example of how one
organization administers its volunteer
program.

4. compare and contrast the role of
volunteerism in the U . S . with similar
institutions in their own country.

acuvmzss :
I . WW:
II - MW

III. The QQDQQQL of Volunteerism

l. The Terms
a. Each. participant should take five
minutes to write their own definition of
what a volunteer is. They can use the
definition as a comparison to the
following discussion, and the staff can
use it to draw them into dialogue.

b. Next. the facilitator (Frank/Shawn)
discusses the 'terms ‘voluntary’ action,
volunteering, and voluntary association.
As each term is discussed, the students
should compare their definition to see
where it fit in with the above terms.

2. The Characteristics of Volunteers
a. Income
b. Education
c. Gender
d. Marital Status

80

e. Predominant volunteer activities

IV. '

l. The United Way

2. The Role of Local Social Service Agencies
3. Case Example - how one agency administers
its volunteer program

V. W

The students should form into groups of three
as they occur along the tables. They should
take about fifteen minutes to discuss the
three questions listed below. We can discuss
their responses for twenty minutes to see how
well they have understood the basic concepts
and how it fits with their expectations. The
discussion should flow easily into the country
overview with the presenters describing how
the concepts fit into their own societies.

1. How does the first hour's discussion fit
with your idea of volunteering in the U.S.?
2. What issues would you like the panelist to
touch on next week?

VI. Wants

1. Choose next week's country/region presenters

81

SEMINAR TITLE: VOLUNTEER PROGRAMS IN THE LANSING AREA

DATE/TIME: WEEK OF JAN. 15, 5:00-7:00, Rm 201
FACILITATORB: negereger: John: Ageney_3ep§: Sharon Radtke,
Mary Joseph, Mary Edens

OBJECTIVES: Participants will be able to:
1. describe and contrast the different
methods by which agencies administer
their volunteer programs.
2. describe a case example of how one
organization administers its volunteer
program.
3. compare and contrast the role of
volunteer administration ix: local
agencies ‘with similar institutions in
their own country.

ACTIVITIES:

1. WW

II. 923W

III. WW

1. Each Agency Rep should give a short
description.of‘how they utilize volunteers and
their importance to the agency's everyday
activities. Topics to include:

a. Volunteer Recruitment

b. Job Designing and Contracting

c. Volunteer Training

d. Volunteer Management

e. Volunteer Evaluation

f. Student Concerns

g. role of agency in community

2. The students should then interact with the
agency reps about their concerns and.guestions
based on the agency reps’ descriptions and
personal items of interest. We should have the
list of concerns ready and possible questions
to get the ball rolling if the students are
slow in jumping into discussion.

82

1. student status with agency placements
2. committee to set up agency reps seminar
3. how's the food?

83

SEMINAR TITLE: LINKING THE VOLUNTEER CONCEPTS TO ISVC

ones/11x3:
racznxmamons:

OBJECTIVES:

ACTIVITIES:

ACTIVITIES

JANUARY 22, 5:00-7:00p.m., Rm 201
FRANK, DAVID, AND STAFF

The Participants will be able to:

1. Summarize the ‘major concepts from
seminars 1 and 2.

2. describe the process of establishing
their volunteer placement with the agency
contact person.

3. list.the rights, responsibilities, and
ethics of volunteers.

I. Intr9dnstienzznrngse_9f_§eminar
11- Q2unLIxLBegiQn_Ere§en§atien
III. Linking_fhe_angenf§_9f_yglnnteeri§m

a. A summary of the concepts from seminars 1
and 2. This section also includes a
distinction of the formal and informal
volunteer sectors. If Sharon Radtke is
present, she can help to draw examples from
the community.

IV- .Establishing_the_yglunteer_21asemenf

a. Using the overhead, establish a uniform
process for contacting the agency
representative. ZList the information students
should be collecting.

b. Reinforces the timeline for the ISVC.

c. Volunteer Ethics and Responsibilities.

d. Go over with each student where they are in
the process of contacting their placement site
so far.

v. Final_Anngunsement§

84
SEMINAR TITLE: STRUCTURING THE VOLUNTEER LEARNING EXPERIENCE
ones/11x3: JANUARY 29, 5:00-7:00p.m., Rm 201
PACILITATORS: Frank/Staff
OBJECTIVES: Participants will be able to:

1. describe the elements of a well-
written job description.

2. describe and compose a learning
agreement.

3. compare the existence of learning
contracts, formal and informal, with that
of their own country.

ACTIVITIES:

I. Infrgdu2fi2n_and_£nrnese_gf_§eminar

II- 92nntrzzzegign_£re§entatign

III. yglnnteer_zla2ement§

IV- Elemenfs_9f_a_Hell:flriffen_lgb_ne§§rinfien

v. Q2nstrn2Ling_A_Learning_Agreemenf
1. Distinguish between the job description
form and the learning agreement.
2. Identify important aspects of the learning
agreement.
3. Set up timeline to write learning
agreement.

v1. §£§IIL§IBQQDL_§IQEEiDQ§

v. Final_Ann92ngenen§§

APPENDIX THREE

VOLUNTEER PLACEMENTS

CHILD ABUSE PREVENTION SERVICES
- child care for children 6 wks - 6 yrs.

W INGEAM COUNTY COOPERATIVE EETENS ION
SERVICES
- nutrition assistant
- help low-income families with children to eat better
with the resources they have available.

W INGRAM COUNTY COOPERATIVE EXTENSION
SERVICES
- community 4-H coordinator
- assist in implementation of the 4-H youth program in
Ingham County
- responsible for planning, supervision and evaluation of
the community education program at Walnut St. School.

AMERICAN RED CROSS
- clerical duties

W ECONOMIC CRISIS CENTER 5 The SALVATION
ARMY

provide temporary shelter and food to homeless families

computer work

stack up food

clerical duties

LANSING NEIGHBORHOOD COUNCIL
- coordinate neighborhood concerns
- attend neighborhood meeting/activities
- public relations
- environmental issues

Ii§g§_fiinge LANSING COMMUNITY COLLEGE
- assistant ballroom dancing instructor

- IOLDUMAR NATURE CENTER
- environmental education
- computer work
- gardening
- sales

85

86

BQ§D§D_H§§_D§h§n 833108 DAYTIME CENTER

- provide recreational activities to recovering seniors

Hgih§§_Zh§ng NATIONAL ORGANIZATION OF NOMEN 8 COUNCIL
AGAINST DOMESTIC ASSAULT
- civil rights organization to advance the rights of
women
- demonstrations at the capitol
- talk to legislators
- educating women about their rights

MW GATE“! COWI'I'Y SERVICES

- diagnosis and treatment of a wide range of health care
needs
- counseling

A;ggg_LiQ EAST LANSING PUBLIC SCHOOL
- enrichment program for families

Sunefhra_xarnnaratne BT- LA'RBNCB HOSPITAL

figphe_3em§n§n§ EAST LANSING MAYOR'S OPPICE
- environmental task force

Li:flei_fleng NESTMINISTER ELEMENTARY SCHOOL

Anne;§e_pnneen LITERACY COALITION

APPENDIX FOUR

TABLE OF CONTENTS:
INTERNATIONAL STUDENTS: SHARING THE WEALTH
WITH THE COMMUNITY

Preface . . . . . . . . . . . . . . . . . . . . . . . . . i

The International Students Volunteer Corps:
Sharing Skills and Experiences with the Community . . . iii

Section 1: Student Reflections

The ISVC Experience:
Personal and Professional Development . . . . . . . . . . 1
Pavalavalli Govindasany

Doing Volunteer Work with the
Council Against Domestic Assault . . . . . . . . . . . . 5
Naihua Ehang

Linking Scholarship with Reality: A Volunteer's
Experience at the Voluntary Action Center . . . . . . . 10
San Acquaah

Removing Stumbling Blocks to Cross-Cultural Communication 14
Jeyaratnam Sundre-

International Volunteerism: A Global Perspective . . . . 20
Titus Tshiduhulwana Singo

Implications for Personal and Professional Development . 25
Titus Tshiduhulwana Singo

International Volunteerism: Reflections on

Experiences with a US Organization from a Sri Lankan
Perspective . . . . . . . . . . . . . . . . . . . . . . . 30
Sunethra Earunaratne

International Cross-Cultural Communication in the Workplace:
A Personal Experience . . . . . . . . . . . . . . . . . . 37
Surasena Batagoda

87

88
Section 2: Agency Reflections

International Volunteers . . . . . . . . . . . . . . . . 45
Sharon Radtke

Agency Panel: Reflections on the ISVC Experience . . . . 46
Annette Duncan

Incorporating the ISVC Volunteer: A Gateway Perspective 51
Karen Casey

Section 3: Staff Reflections
The International Students Volunteer Corps:
An International Student Advisor's Perspective . . . . . 55
David Horner
The International Students Volunteer Corps:
A Perspective from an Academic Department . . . . . . . . 58
Frank Fear
ISVC: A University Community Development Outreach
PerspeCt ive O O O O O O O O O O O O O O C O O O O O O C O 6 3
John Nelcher
International Student Volunteer Corps:
Reaching Out to Serve the Needs of Michigan’s Communities 66
Rex LaMore

Placement I O O O O O O 0 O O O 0 O O O O O O O O O O O O 69
Mary Edens and Lana Dar

Reflection On Recruitment of International Students . . . 74
Bassey Iban Asuquo Ebiana

Issues in Program Management and Evaluation . . . . . . . 79
Shawn Lock

Section 4: Appendices
Appendix One: Listing of Events . . . . . . . . . . . . 87
Appendix Two: Seminar Outlines . . . . . . . . . . . . 90
Appendix Three: Evaluation Instruments . . . . . . . . 110

Appendix Four: Contact Addresses for
Further Information . . . . . . . . 123

APPENDIX FIVE
MSU FACULTY QUESTIONNAIRE

Are you aware of the ISVC? How did you find out about
the program?

To what 'extent were you aware of your advisee's
participation in the ISVC?

Do you feel programs like the ISVC are an important part
of an international student's academic experience?

Have you noticed any impact this program has had on your
advisee?

To what extent have you and the student interacted about
the ISVC, its potential benefits, and its relationship to
the student's program of study or academic experience?

Has your association with the ISVC resulted in increased
linkages with or awareness of any public and private
agencies in the East Lansing/Lansing area?

Do you have any recommendations for the ISVC staff as it
designs next year's program?

89

90
MSU Student Questionnaire
Why did you decide to participate in the ISVC?

What do you hope to get out of the ISVC in terms of
skills and knowledge in the short and long term? Did
your goals change as you learned more about your
placement/had more experience?

Do you feel your placement experience was successful in
meeting your goals? Why or why not?

To what extent was there a cultural exchange at your
placement between you and your supervisor/co-
workers/clients?

Have you discovered any new aspects of American culture
as a result of your experience with the ISVC? Has the
volunteer experience helped you in other cross-cultural
situations?

Has the ISVC experience prompted any thoughts about your
own country? Comparisons and/or contrasts?

Have you benefited from the ISVC?
- professionally?
- personally?
- academically?
- other?

Do you think this experience will give further benefit
after it is over?

Have you learned. more about the ‘ways in 'which. an
organization functions? Have you compared or contrasted
that with organizations back home?

What recommendations would you make for the ISVC next
year?

91
Agency Supervisors Questionnaire
How did you learn about the ISVC?

To what extent have you utilized international
persons/students as volunteers in the past in your
agency?

What role has the student played as a volunteer in your
agency?

What considerations affect your decision to use an
international student volunteer that are different from
those used in utilizing American volunteers?

Has his/her being an international student brought any
unique or new ways to analyze a problem or suggest new
ways to do work at the agency?

To what extent has a sharing of cultures occurred in this
experience?

Has your agency expanded its linkages with departments
within MSU as a result of your participation in the ISVC?

What recommendations do you have for the ISVC staff for
the next year's program? (i.e. , student preparation,
agency supervisor preparation, or placement)

APPENDIX SIX

EVALUABILITY ASSESSMENT:
A PRIMER FOR SMALL-SCALE PROJECT STAFF

WW
Evaluability assessment (EA) is a tool that helps program

staff clearly describe their program’s structure (i.e., its
goals and activities), establish the plausibility of its
activities in reaching intended goals, determine the whether
the program is evaluable, and enhance the utilization of the
evaluation results (Smith, 1989). The EA process helps a
program staff to step away from the project, describe what
actually happened, determine whether the program can be
evaluated and how, and suggest*ways to best use the results of
the evaluation.

EA was first developed by Joseph Wholey and his
associates at the Urban Institute in Washington, D.C. (Nay &
Kay, 1982, Smith, 1989). It was developed to address the
perceived inadequacies of program evaluation. First used as
a summative evaluation tool, EA is now used to assess all
aspects of the program process. This primer will focus
specifically on the use of EA as a summative and reflective
evaluation tool. The list of selected references at the end
of the report contain in depth discussions on EA, and describe

92

93
the additional use of EA as a program design tool.

In brief, EA involves gathering a group of individuals
knowledgeable about the program, called an evaluability
assessment team, that can.help gather materials and develop a
model of the program that describes the resources and
activities necessary to reach the program goal. Throughout
the process, individuals who have a vested interest in the
program -- stakeholders -- are identified and interviewed to
produce additional information. After developing the model of
the project, a decision is made as to what aspects of the
program can be evaluated, and which aspects are most important
for evaluation. Developing an evaluation strategy that
results in findings that can and will be used by stakeholders

is a key component of EA.

WWW

EA has several characteristics that make it easy to use
for small-scale projects. The process used to conduct an EA,
described below, can be adapted to fit the individual
circumstances of any project. For example, EA is normally
conducted over several day-long workshops. However, if time
is a limited resource, the sessions can be adapted into
shorter focus group sessions. When adapting the EA process,
staff members need to be aware of any compromises in the
process that may affect gathering information pertinent to
their goals for the EA.

94

EA does not require significant experience with program
evaluation. The action steps described below, and the
references listed in listed at the end of this report, should
provide most program staff with the information they need to
conduct an EA. If the results of the EA show the need for
further evaluation activities that require specific expertise,
the program staff can decide which alternatives to pursue
based on the available resources.

EA provides several levels of information. First, it
results in a program description that explains the cause-
effect relationships in a program from project design, to
project implementation, to project completion. Second, it
provides immediate information on the plausibility of a
program and where it might be modified to achieve its goals.
Third, it results in a series of evaluation questions and
potential evaluation methods that can be carried out if so
desired. Program staff benefit from one or all of the levels
of information. The decision on which information to use will
depend on the available resources. For example, some
suggested evaluation needs may be beyond program funding at
the time of the EA, but may be pursued later on as funding
becomes available. Thus, EA provides a range of information
for program staff to use according to their needs and

resources 0

95

W

This section presents an outline of the basic steps

involved in conducting an EA. The selected references listed

at the end of this report can be used to adapt the outline to

specific needs.

1.

Choose to conduct an evaluability assessment.

EA is an evaluation option appropriate for
resource-constrained programs, and/or when a
program staff is uncertain as to which aspects of
the project should be evaluated and how. However,
if the program staff has already identified a
specific area in need of evaluation, an EA may not
represent an efficient use of program resources.

Choose an EA coordinator.

The EA coordinator is responsible for conducting
the focus groups, overseeing any stakeholder
interviews that are necessary, arranging for an
analysis of program documents, and suggesting next
step evaluation methods.

Whether or not an internal or external EA
coordinator is chosen depends on the needs of the
program staff. An external coordinator may be
better for programs whose staff have had strained
working relationships, or are divided on which
aspects of the program are most important for
evaluation. An internal coordinator may be better
when the program staff are most concerned with
reflecting on the lessons learned in a program for
use 'with. similar’ programs in the future. .An
internal coordinator may be more cost effective:
however, depending' on the scope of the. EA, a
program staff may be able to find an external
coordinator within the host institution who has not
been involved in the program.

Choose an EA team.

The EA team should be comprised of program staff
and those individuals who have a vested interest in
the program, or stakeholders. Examples of
stakeholders include upper-level administrators,

96

representatives of organizations that benefit from
the program, and individuals familiar with the
program's target audience. Stakeholders should
have knowledge of the program from its inception.
If a stakeholder continually needs to be updated
about the program during the EA, then s/he may not
have information to contribute to the EA process.

Besides participating in the focus group to build a
model of the project, the EA team may have
additional responsibilities such as conducting
interviews with other stakeholders. Thus, they
should be able to commit the necessary time to
complete the EA process.

Conduct Phase One: Explain the EA process and
develop a program logic model.

The first stage in the EA is to explain the EA
process to the EA team, so that each person knows
how the process will be conducted and what the
goals are for the effort. Next, the EA coordinator
explains the use of the program logic model and its
role in the EA. This phase may conducted in one
session, or over several sessions, depending on the
time constraints of those involved. Figure A.1
lists an example outline for this session that can
be used as overheads.

Building a program logic model is a process of
describing the cause and effect relationships in a
project, identifying the program components that
bring about the cause and effect relationships, and
specifying the intermediate and final goals and
effects of the program. Wholey (1987, pp. 78-79)
describes program theory as an if-then expression:

If the following program resources are
available, then the following program
activities will be undertaken. . . If these
program activities are to occur, then the
following program outcomes will be
produced. . . If these activities and
outcomes occur, then progress will be
made toward the following program goals.

Building the program logic model can be the most
significant part of the EA process, because the
process may alert program staff about important
issues that they were unaware of while conducting
the program. The model may point out next steps

97

 

 

Figure A.1
Sample Overheads for the First EA Session

Overview of the Sessions
- Session 1: Building a Program Logic Model
- Session 2: Reviewing the Model & Conducting'a.Mock
Evaluation Session
- Session 3: Review Proposed Evaluation Process &
Critique EA

Overview of Evaluability Assessment
- As a Program Planning Tool
- As a Formative Evaluation Tool
- As a Summative Evaluation Tool
- These focus groups will focus on EA as a summative
evaluation tool

Key Points in an Evaluability Assessment
- Program Staff

Program Logic Model

Stakeholder Interviews

Document Analysis

Evaluation Proposal

Evaluation Decision-Making

Program Logic Model
- Specifies cause-effect relationships
- Specifies outcomes
- Specifies resources
- Specifies obstacles and alternatives
- Specifies how our activities will help us reach
the outcomes we have targeted in our program
proposal

Steps in Building a Program Logic Model
- Outline major phases
- List two to three most significant activities to
accomplish each phase
- List resources needed to successfully conduct each

 

 

98

that make any formal evaluation process
unnecessary.

The program logic model contains seven rows: goals
and outcomes, program events, program activities,
assumptions guiding the first three rows, resources
needed to conduct events, indicators of successful
completion of an event or goal, and barriers to the
completion of a goal or event. The rows are linked
to one another through the stage in which they
occur in a program. For example, each event
requires certain activities and resources, is
designed to bring about a particular goal or set of
goals, and will have indicators of success and
potential barriers. Thus, the program logic model
not only indicates the sequence of events
throughout a program, but also the associated
outcomes, activities, etc. An example of a program
logic model can be found in figure A.2 of this
report.

If the discussions indicate that there are other
stakeholders who need to be interviewed, the EA
coordinator should arrange for those before the
second phase of the EA. The EA coordinator may
also conduct an analysis of program documents for
additional information about the program. ‘The main
purpose for the stakeholder interviews and document
analysis is to generate information from other
sources that can confirm the program logic model,
or suggest additional aspects that the EA team
should consider.

Conduct Phase Two: Review the program logic model
and conduct a mock evaluation-utilization session.

In the second meeting, or set of meetings, the EA
team reviews the program logic model that the EA
coordinator has developed from the first session.
At this point the EA team can make any changes to
the model and discuss any further information the
model indicates relevant to the evaluation. The
results of the stakeholder interviews and document
analysis should be discussed to determine if any
additions are needed to the model, or if there is
any conflict between information in the model and
that found in the additional research. If changes
are made to the model, the EA coordinator will
incorporate them and present the model at the next
session.

99

. 0.6. 2F .082 as... 58...... sass... a... 25......

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

S... 2 303.382
Silence... 0.5., coal—canal. J2»: v.33 93.85 82.. 3.8... 32.8.3...
. p .. .1 giant-82.88.. 8188...... 88.95%, 2.229....)
2 o...- .o.. 2835. .83.. 2.5.2.. .bafiagfiae cam .88.. .2. 2o .1 02.... .3 so a .822... 3a..
_ — u _ _
~U§>_o>5 . 30%
.3... .8853 as. 38...... 88». 8.8.. a. 8.... 2 2...... 3.28. 538.....9
3.5.5. .8. .822... E 82.8.2.1... .58 x888 22.2.. .o .33.... .o>m. a. 83.3.... 885.2. 3.58 E. .3.
336... 3.825 as .89.... 02:3; .3 8.. 5.5.9.... .33 5.3 8.84%.; . 3:52 5.. i=3 .03!— U>m~ ac census: 8.8m
= _ __ __ __ a
.838 .989. 358... >6..."— bisfifiou .2338.— bisflficu 3.5. .85..
583m .8838 be.» U>m_ 5.6.... 25:3. .868? 88.598 >89 .83.- .qca..e:.8.c. 58...... {ESE-n- 52... :39: 3.3.5
3.5.. 3.8.: 882.2...» 28.35 3.5338 8.8.3.1238... 23.... .a. 88>... 33...... 5.3... 65.3... 8......
__ _ a = _ = . .
. .352
3.8.5.89... 9.3.3 8.85 88.89... 88.5.2. 2.6.1 .83.
5... so... .3 3.3.5.8.... 8.88.8 8:8»... 958.... c- .3335... £38.. 3.59...
.8 83.598 U>w_ 8:3 Sen 2 :- ... 38.8.3.— 2... 3.8!..- efleol 93 2....) base.— . 953:» s- ..226... . U>m_ o... B.
as. 582. 2...... 88.899 .88... s 22.2.. 2.53 82.89.... iii... 2.8.. 2...... 8.235. , 3882.. .. 8.3...
285...... .25 ....§.. 38.5.... 3.. use, :83... .8833 3...... 38.38 .0 as...» 88>... 8:38.. .38... ,
- __ _ — — _ _
8.8!... 3.3 8.83- :88.
5.89..»- 818 608.898 29>... 5.9.333 .1 a. .89. d 8:383 9:88. .8233
88.5.»... 3.95.33...» 888.83.. 8.08.}?! 33.353.358.52. 5.3.8.339
£088.55... {986851. 9:58.25... has—.3818. Sees—.8988... 9.3.96.3 .gheél,
Sims... fies... gig". .38.... 84... 5.5.3.. . 18.8.. a: 38. 83.80 5.... 838.31..
_ — u . _ _ _ _
«3.2.3 a... 28.8. 82.8. 3.18. 5.3.1:. 5.... 189....
5.33... .325 22.18.. .388 29...... .895 232.. .835 28...... .383.“ 3.5.8.. 9.26.. an... 9.26..
_ _ __ — _ _ —
8.8a... :3. 55...... 89.2.2, 3...: 39...... .3.
88.598 3 5.835.?! 2638 93 bans—Eco, 5....) .935 98....
5.2.2.3... Ewan 8.58.. 231 68.898 4.53... .5... auto ec- toqol Be... 8.8-...— 8838 8min: 8885
use 3 85...... is... o... 958... 83.5.2 . a... 885.8, 583 3...... a... 2.3 .822... $8.8. 55......
a... «88.89.! 8939... - 829:3 .382... 382.. 1.530.826 - 3.8.5: 8...... as... =- ._o :83 05 58... .o 3.3.383...
38.5.2... o 2.888% . a... 3......8.. 3. 38...... 21...... .o 938 . 52...... a... 8.5.2.. 9.2.8 38...... 83......

 

 

 

 

 

 

 

 

 

28.81--

8.583.

a!"

183.6338

100

The second activity in phase two is to conduct a
mock evaluation- utilization session. The purpose
of this activity is to have the EA team suggest
possible evaluation needs. The team lists
information. they’ need, in light of the issues
raised by the program logic model, to modify the
program under review, or to use in future
programming efforts. The value of the exercise is
to help the EA coordinator suggest evaluation
activities that will produce results the EA team
can use in current and future tasks.

Conduct Phase Three: Propose evaluation activities
and decide on next steps.

If changes were made to the program logic model in
the second phase, they are reviewed now and the
final model agreed upon by the EA team. Any last
comments about the model can be discussed at this
time.

The EA coordinator then presents and reviews
her/his suggestions for evaluation.activities.based
on the mock evaluation-utilization session
conducted in phase two, observations of the EA
team’s discussion, and her/his own past experience
in program evaluation. EA team members can review
the coordinator's proposal, ask any questions,
and/or suggest modifications.

Once the EA team has reviewed the proposed
evaluation activities, they are ready to decide on
how to proceed next. Smith (1989) suggests several
options:

a. Decide to egalgatg the program: the EA team
can use one or all of the EA coordinator’ s
proposed evaluation activities, suggest
further review' of evaluation, alternatives,
and/or suggest additional evaluation
activities from their own activities.

b. Decide to change the program: the process of
building the program logic model may indicate
immediate changes that the EA team can make to
improve the program. The EA team can
implement the changes and then conduct a later
EA to determine if the changes produced the
intended effects.

c. Decide to take no further action: the EA team

101

may decide that any evaluation activity would
not produce results significant enough to
justify the resources expended, or that the
program is proceeding as desired so that no
changes or evaluation is needed at the present
time.

d. Decide to m the program: the EA may
indicate that the events, activities, and/or
resources are insufficient to bring about the
intended goals and outcomes, and that further
resources are not available to change the
program. In this case, the EA team may decide
to stop the program and use present resources
in other ways.

Whether or not the EA team decides to proceed with an
evaluation, the EA process gives program staff and
stakeholders an opportunity to build consensus about the
program’ 3 goals and activities. It can help to increase
communication among different units within an organization and

develop new ideas for present and future programs.

Selected References for Evaluability Assessment

Hickman. 1». (ed ) (1987) WWW.

. .I . . .A. 11'. _v -1 _°! N. _° I.
1251. San Francisco: Jossey-Bass.

Gorden, R. (1980). ' '
. Homewood, Illinois: The Dorsey Press.
Krueger, R. (1988). °
applisg_zsssszsh. Newbury Park, California: Sage.
Mayeske. G- W- (1991)- W
Washington, D.C.: Extension Service. 'U.S. Department of
Agriculture.
Nay, J., Kay, P. (1982). ve s'
ssssssmsnt. Lexington, Massachusetts: LexingtonBooks.
Patton. 14- (1986). Wong Newbury
Park, California: Sage.
Rutman, L. (1980). e e ° 5°
ssssssmsnt. Beverly Hills: Sage Publications.
Smith, M. F. (1989). b' ' s :

sppxsssh. Boston: Kluwer Academic Publishers.

102

LIST OF REFERENCES

LIST OF REFERENCES

Alkin. 11- (66-) (1990). Wu- Newbury Park.
California: Sage.

Batchelder. Du Warner. E- (1977)- W

’A" ,:1 a :99 0: 2 0 0“- - ra ‘2- - on.
Brattleboro, Vermont: The Experiment Press.

Doherty, A., et al. (1978). Toward a theory of undergraduate
experiential learning. In M. Keeton, Tate (Ed.) , W

- W O .
experiential_learning (1) (pp- 23-36)-

Duleyy J. (n.d.). Basic skills.for experiential learning: What
skills do students need to make the most of experiential
learning opportunities [Papers on Learning and Teaching].
East Lansing, MI: Michigan State University.

Duley, G., Gordon . (1977). l e-s o o e
- . Princeton, N.J.: Educational
Testing Service.

Fear, F., Axinn, G. (1989) ”Linking Scholarship with Reality:
Helping Graduate Students Understand the Dynamics of
American Volunteerism." A proposal to the National
Association for Foreign Student Affairs.

Garden. R- (1980)- W
tastiss. Homewood, Illinois: The Dorsey Press.

Keeton, M. (1978). introduction. In M. Keeton, Tate (Ed.),

Lsszning by EXEQILQDQE - gust, whyI how: st gisectigns
WW (1).

Knapp. Ju Sharon- (1975) - AWL
W. Princeton, N.J.: Educational Testing Service.

Krueger. R. (1988). W
W

. Newbury Park, California: Sage.

Mayeske. 6- w. (1991). W
Washington, D.C.: Extension Service. U.S. Department of
Agriculture.

Nay. J0: Kay. P- (”Ba-Win):
ssssssmsng. Lexington, Massachusetts: LexingtonBooks.

103

104
Patton. 11- (1991) - WWW- Prat-session

workshop at the 1991 American Evaluation Association
Annual Conference.

----- (1990) - WWW- Newbury
Park, California: Sage.

----- (1990). The evaluator’s responsibility for utilization.

In Alkin. M- (ed-h WW- Newbury Park.
California: Sage.

----- (1986)- nt1l1z3tignzfgsused_9xaluatign§- Newbury Park.
California: Sage.

Permaul J01 M1k0- (1977) W
W- UCLA: CABL-

Rutman, L. (1980). ' s v u '0 5° Ev
ssssssmsng. Beverly Hills: Sage Publications.

Scheirer, M. (1987). Program theory and implementation theory:
Implications for evaluators. In L. Bickman (Ed. ), using

WWW (pp 77- 92) San Francisco:

Jossey-Bass, Inc.

Smith. M. F- (1989)- W193].
spprsssh. Boston: Kluwer Academic Publishers.

Wholey, J. S. (1991). ' s ev ua ' °
sxslssgisns__gssg. WOrkshop at the 1991 American
Evaluation Association Annual Conference.

----- (1987). Evaluability assessment: Developing program

theory. In L. Bickman (Ed.) , gsing prggzgm thssn 1n
syslgstisn (pp. 77-92). San Francisco: Jossey-Bass, Inc.

 

 

lawman
9:239 005mm
Auvaan

1

fiifi

 

 

"Iilflfliliflfljfliflifli'lmlflmwflmfir