.5E:E_:EEEEEEE:ESE mmw

 

AtfifihRY

:, ;;§;_;;an State

;'.- uprCJ?
‘ _‘._.|\Nr'"y

 

 

PLACE IN RETURN BOX to remove this checkout from your record.
TO AVOID FINES return on or before date due.
MAY BE RECALLED with earlier due date if requested.

 

DATE DUE DATE DUE DATE DUE

{APB :21 z 2011

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

2/05 c:/CIFIC/DateDue.indd-p.15

 

.—

Burkholder, Harry A. 2004

CITIZEN PLANNER TRAINING PROGRAM EVALUATION: METHODS OF
EVALUATION IN SUPPORT OF MORE EFFECTIVE PLANNING OFFICIALS

By

Harry A. Burkholder

A Thesis
Submitted to
Michigan State University
in partial fialfillment of the requirements
for the degree of
Masters of Urban and Regional Planning
Department of Urban and Regional Planning

2004

fie
m

RETURNING MATERIALS ;
““e in book "'

TABLE OF CONTENTS

LIST OF TABLES ............................................................................ iii

LIST OF FIGURES ........................................................................... iv

INTRODUCTION ............................................................................. 1

SECTION 1

JOB OF THE PLANNING OFFICIAL ..................................................... 3
1.1 Planning Official Training Trends .............................................. 4
1.2 Initial Training Programs ........................................................ 5
1.3 Continuing Education ............................................................ 7
1.4 Type I. Continuing Education That Adds Skills

To A Base Set Of Initial Skills ................................................. 8
1.5 Type II. Continuing Education That Reflects An
On-Going Process Of Learning ................................................ 9

1.6 Training and Continuing Education State Mandates ....................... 10
1.7 American Planning Association Support for Planning Officials ......... 10
1.8 Michigan State University Citizen Planner Program ....................... 11
1.9 Value of Training ............................................................... 12
1.10 Value of Continuing Education .............................................. 13

SECTION 11

CURRENT EVALUATION METHODS ................................................. 15
2.1 The purpose of training evaluation ............................................ 15
2.2 Training evaluation methods (Business Field) .............................. 19
2.3 Training evaluations methods (Planning Field) ............................. 24
2.4 Michigan State University Citizen Planner Program. . . . . . . . . . . .2.......4
2.5 Hamilton County Regional Planning Commission. . .. . . . . . . . . . . . . .- ........ 29
2.6 Main Street Program ............................................................ 30

SECTION IH

METHODS OF EVALUATION ........................................................... 33
3.1 Testing ........................................................................... 34
3.2 Cost Analysis .................................................................... 39
3.3 Performance Criteria ........................................................... 42

ii

LIST OF TABLES

1.1 Participants’ Evaluation of 2003 Citizen Planner courses,
Statewide Average, Percent Responding, and Mean Score
on a 1-5 Scale .................................................................................... 25

1.2 Selected Examples of Pre- and Post-test Mean Scores
and Changes in 2003 Citizen Planner Program Participants’
Knowledge of Concepts and Tools, Using Point Scale of l to 3 .......................... 26

1.3 Results of 2003 Survey Conducted with 2002 Citizen
Planner Program Participants, Percent Responding ......................................... 27

1.4 Comparison of 2002 Citizen Planner Program Participants’
Responses to Questions Related to Preparedness for and
Confidence with Membership on Board or Commission .................................. 28

iii

LIST OF FIGURES

1.1 The Training Cycle ........................................................................ 18
1.2 Kirkpatrick Method of Training Evaluation ............................................ 23
1.3 Citizen Planner Training Program Essay Test ........................................... 38
1.4 ROI Formula ................................................................................ 41
1.5 ROI (Training Program Evaluation) ..................................................... 41

iv

I know of no safe depository of the ultimate powers of society but the

people themselves; and if we think them not enlightened enough to

exercise their control with a wholesome discretion, the remedy is not to

take it from them but to inform their discretion by education.1

- Thomas Jefferson

Current development trends in both urban and rural environments have made land
use an important social and political issue. As deve10pment decisions continue into the
21st century, individuals responsible for planning will face an increasingly complex array
of planning issues. These issues will demand “that the planning profession comprise
groups of people well versed in contemporary philosophy, social work, law, the social
sciences, and civic design.”2

However, land use planning is not determined solely by professionals. Land use
planning is unique in that important land-use decisions often are made by local citizens. It
is the Planning Commissioners, Zoning Boards, and Zoning Board of Appeals Members,
all either appointed or elected citizens, who make the primary land-use decisions guiding
land use decisions in their community. How can communities know that their planning
officials are making effective land use decisions? “Every year across America, thousands
of citizens are appointed or elected to county or municipal planning commissions, zoning
boards of adjustment, or related planning authorities. Except in large cities, they are
unpaid volunteers whose motives for seeking the posts range fi'om civic activism to
professional exposure to individual concerns. What many share, however, is a lack of

good preparation for the duties their new positions entail.”3

 

' Solnit, Albert. :13; 19b of the BM’ g Commissioner. Washington DC: Planners Press, American

Planning Association 1987.

2 Davidoff, Paul. Ream gs in Planning Thgog. Washington DC: Planners Press, American Planning
Association 1989

3 Schwab, Jim. “Training Citizen Planning Officials,” zgning News June 1991: 1

Many communities have recognized that citizens often do not have the knowledge
or tools to be effective planning officials. This recognition has led many communities to
seek training opportunities that will provide planning officials with the tools needed to
become more proficient at their assigned duties. Evidence of this recognition was
confirmed in a 2000 survey for the American Planning Association which identified 67
training programs primarily targeted to new planning commissioners.4

Current trends in planning official training programs demonstrate both initial and
continuing education training opportunities. The primary purpose of most citizen planner
programs is to provide planning officials with the skills and knowledge to be more
effective planning officials. However, one question remains unclear. How do we know
whether the training provided by citizen planner training programs is producing more
effective planning officials? Currently, many training programs measure the effectiveness
of training through participant surveys. While surveys provide an excellent measure of
training effectiveness, most data is based on personal opinion. This trend is indicative of
the lack of quantitative measures indicating training produces more effective planning
officials. The purpose of this paper will be to determine whether there are any measurable
impacts, or quantitative evaluation methods, that measure whether citizen planner
programs are producing more effective planning officials, and if so, in what manner
would the impact or evaluation be represented. To help guide this discussion this paper
will be organized into three sections. Section One will describe the current scope, and
value of planning official training programs in Michigan and the United States. Section

Two will discuss the purpose of training evaluation, specifically looking at the value of

 

‘ Chandler, Michael. “Training Programs for Citizen Planners,” W 2000
Online. 2. pag online. American Planning Association <http://www.planning.org/thecommissioner>

proving, improving, learning, feedback, and controlling. This section will also discuss
how to measure training effectiveness by examining the current state of training
evaluation methods in both the business field, specifically examining the measurement of
reaction, learning, behavior, and results, and the planning field, specifically looking at the
National Main Street Program, and training courses offered by the Hamilton County
Regional Planning Commissioner Training Program and the Michigan State University
Citizen Planner Program. Finally, Section Three, based upon the evaluation methods
found in the business field and the planning field, will discuss and describe three methods
of evaluation that would be effective in determining whether citizen planning training
programs are producing more effective planning officials.

Section 1.
Planning Official Training: A Scope

To understand why training and continuing education for planning officials has
become such a significant issue for communities, it is valuable to understand the
responsibilities of the local planning official. The term “Planning Official” was created
by the American Planning Association to identify any appointed or elected official
involved in planning decisions for the betterment of a community, region, county, state,
or county.5 The term “Planning Official” can be organized into three specific types of

planning capacities.6

 

5 “Planning Official” Wig: 2004. American Planning Association.
12 March 2004. < http;//www.planning.org/commissioners/index.html>

a “Planning Official” < http;//www.planning.org/commissioners/index.html>

0 Planning Commissioners- Appointed volunteer or, less commonly, elected
officials who serve on local or regional planning commissions. Commissioners
oversee the work of pubic planning agencies. They initiate and guide long-range
planning efforts, conduct public meetings on proposed plans and projects, review
development proposals for conformance with local plans and development
regulations, and develop new planning programs

0 Zoning Board of Appeals Member- Appointed volunteer officials who serve on
a board that hears appeals on zoning and land-use matters. The work of the board
is generally limited to review of applications for zoning changes, conditional use
permits, sign permits, variances, and other appeals. In some communities, the
functions of a planning board and a zoning board of appeals are performed by a
joint planning and zoning commission.

0 Planning Board Member— Appointed volunteer officials who serve on the
planning commission, the zoning board of appeals, or any other commission or
board that oversees planning related issues. This may include historic
preservation commissions, design review boards, park boards, or other board and
commissions that address the well being of the community.

Planning official responsibilities may vary from community to community.
However, most follow this general outline established by the APA. Planning official
responsibilities are generally outlined within the community charter and can be accessed
on most local government websites.

Section 1.1
Planning Official Training Trends

Currently, training programs for planning officials exist throughout the country.
Programs are administered through state, regional, county, and local government,
planning associations, university extension services, and professional training
associations. Each individual training program has unique organizational structure,

content material, and program requirements. A description of every training program that

currently exists is beyond the scope of this paper. This paper will discuss and describe a

few training programs that are representative of the current trends in planning official
training.

Section 1.2

Initial Training Programs:

One of the primary trends in planning official training is programs that focus on
“initial” training. Initial training programs are primarily characterized by their core set of
training courses covering a basic skill set. The core set of training courses is presented on
an annual basis and provides planning officials with the fundamental skills with which to
make informed land use decisions. A 2000 APA survey on planning official training
programs found the most common subjects covered during training include the powers
and duties of the planning commission, zoning practices, the principles and practices of
planning, and planning and zoning law.7 These subjects are representative of the core set
of training courses found in many initial training programs. Most initial training
programs provide only a limited number of training hours. Upon the completion of the
initial training program, there are no opportunities for continuing or advanced training.
The following examples represent training programs which provide initial training to

planning officials.

Hamilton County (OHIO) Regional Planning Commission “Certified Planning

Commissioners ’ Program " 8 The Hamilton County (Ohio) Regional Planning

 

7 Chandler, Michael. “Training Programs for Citizen Planners,” W 2000
Online. 2. pag Online. American Planning Association <http://www.planning.org/thecommissioner>

Commission Certified Planning Commissioner Program is a 16-hour workshop designed
to provide initial training for planning commissioners. The workshop is designed to
improve the participant’s ability to: perform the essential duties of a planning
commissioner; interpret and analyze technical and legal information; recognize the need
for information and obtain it; reach decisions fairly and efficiently; understand laws that
affect planning commissioners; understand the unique role of the planning commission;
understand the process of comprehensive planning; and conduct efficient pubic meetings.
The program is administered through a class presentation and discussion with hands-on
exercises. Participants of the program attest to its value. The program website provides
several participant comments, including one, who noted the value of the training and
believed that classes should be required for planning commissioners.

T raining for Planning Ofiicials9 - The Atlanta Regional Commission, under the
direction of the Community Planning Academy, provides the “Training for Planning
Officials” course designed to equip citizen planners and community leaders with the
technical knowledge and leadership skills necessary to successfully plan their
community. The course is divided into three one-day long sessions. Participants are given
four weeks between each session to allow time for independent study. A certificate of
completion is awarded to participants who attend all three sessions and complete the
independent study.

Part 1. introduces the foundation of planning in Georgia, specifically within the

Atlanta region. Topics include a historical basis of land use planning and regulation, and

 

8 “Certified Planning Commissioners Program” Pl 'n P rshi H '1 on 0
W93, 2004. n. pag. Online. 12 Nov. 2003 <http://planningpartnership.org/cpcp/>

9 “Training for Planning Officials Community Planning Academy.” AW
2004. n. pag. Online. 15 Nov. 2003. <http://www.atlreg.com/comunitybuilding/cpa.html>

an overview of the legal environment in which planning is conducted. Part 11. reviews the
practical tools of planning from the local, state, regional, and specific use perspectives.
Part III. focuses on bringing theory to practice and teaches participants how to put
principles learned in other sessions to work in their communities.

Section 1.3

Continuing Education

The most prominent trend in planning official training is programs that offer
continuing education opportunities. Continuing education is an integral aspect of the
education process because it provides planning officials with the additional skills needed
to make better informed land use decisions. As communities evolve, the breadth of
knowledge and skills needed to make informed land use decisions will continue to grow.

Current trends in training programs indicate that continuing education can be
categorized into two types. The first type of continuing education reflects training that
can be traditionally identified as continuing education. There is a clear point at which
initial training ends and a continuing education program begins. These continuing
education programs are designed to add supplemental skills to a base set of core
competencies. The additional skills may include more in-depth training in a particular
basic area, or it could include an entirely new type of skill.

The second type of continuing education makes no clear formal distinction
between initial and supplemental skills. There is no clear point at which initial training
and continuing education is either identified or started. These training programs reflect an
on-going, continual process of learning. This is not to say there is no recognition between

what would be considered basic topics and more advanced topics. The titles of many of

the training courses are identified as introductory; however there is no indication that
these courses must be taken prior to other topics or even at all. Participants may select

from a wide choice of subjects that include both introductory and advanced topics.

Section 1.4
Type 1. Continuing Education That Adds Skills To A Base Set Of Initial Skills

At this time, the majority of planning official training programs provide both
initial and continuing education training opportunities. The following programs represent
training programs that offer continuing education training that is specifically designed to
add skills to a base set of initial skills.

Citizen Planner Training Collaborative ( CPT C) '0 The University of
Massachusetts Extension, in collaboration with the Massachusetts Municipal Association,
offers eight courses covering both basic and complex planning issues. The eight courses
are divided into two sections, Level One and Level Two, each consisting of four
modules. Upon completion of the first four modules, the participant receives a Level One
Certificate. Level One modules include introductory topics: Introduction to the Zoning
Act; Introduction to the Subdivision Control Law; Introduction to the Roles and
Responsibilities of Planning and Appeals Boards; and The Basics of Reading a
Subdivision Plan. Level Two modules include intermediate topics: Special Permits and
Variances; Nonconfonning Structures; Uses and Grandfathering; Making Plans Work;
and Zoning Exemptions. Upon completion of all eight courses, participants may continue

to take advanced training courses. Advanced course topics include: How to Hold a

 

'0 “Citizen Planner Training Collaborative.” Universig of Mggsachusetts Extensign 2004 n. pag. Online.
2 Nov. 2003. <http://www.umas.edu/masscptc/corecurric.html>

Perfect Public Hearing; Writing Reasonable and Defensible Decisions; and Preserving
Community Character. The CPTC Program offers all courses in both the classroom
setting and on-line.

Section 1.5

Type II. Continuing Education That Reflects An On-Going Process Of Learning

Several training programs offer continuing education that is specifically designed
as an on-going process of learning rather than a continuation of an initial education
program. Examples of such programs include.

The North Carolina Citizen Planner Program11 The University of North Carolina
and the North Carolina Chapter of the American Planning Association have developed
the North Carolina Citizen Planner Training Program. The training program is designed
for planning commissioners but is also used as a resource for planning professionals,
elected officials, and members of other boards and commissions. The program consists of
10 training modules which can be ordered by a community which then trains the planning
official locally. Each module is provided in 25 pages of outline form which includes
several sample situations and discussions. The first five modules are organized as a core
curriculum, including: Working Together; An Overview of Planning in North Carolina;
Comprehensive and Strategic Planning; Zoning; and Subdivision Management. The
remaining five modules are organized as electives, including: Landmark Legal Cases in
North Carolina; Coastal Area Planning; Aesthetic Regulations; Environmental
Management-Watershed Planning; and Transportation Planning. Communities have the

option to mix and match modules depending upon their own needs.

 

" “North Carolina Citizen Planner Training Program" North Carolina University/ Noah lem’ finger
M 2004 n. pag. Online. 11 Nov. 2003 <http://www.nc-apa.org/Citizen_Planner1.htm>

Section 1.6
Training and Continuing Education State Mandates

Each of the previous training programs previously discussed, including initial and
continuing education (Type I. & Type 11.), are training programs that are completely
voluntary. A local planning official is not required to attend any of the described training
programs in order to serve on his or her local planning board. Training simply provides
an opportunity for planning officials to increase their skills and knowledge. However, as
the value of training and continuing education becomes more widely recognized, several
states have taken pro-active steps to mandate training and continuing education for
planning officials. In the last three years three states: Kentucky, Tennessee, and South
Carolina have mandated training and continuing education for planning officials.
Section 1.7
American Planning Association Support for Planning Officials

As previously mentioned, planning officials share a unique responsibility. While
many planning officials often have little to no formal planning knowledge or experience
they are frequently called upon to make important land-use decisions. Because the
consequences of their decisions can be far reaching, their job is one of great
responsibility. The American Planning Association (APA) has recognized this
responsibility and has actively supported planning official training and continuing
education at both the national and state level.

The APA’s primary support network for planning officials is the Planner
Commissioner Service (PCS). The goal of the PCS is to provide training, through

products and services, to planning commissioners and other elected officials who are

10

actively involved in planning. Membership in the APA is not required to benefit from the
PCS services. PCS products include: “Best of Contemporary Community Planning” an
online webocast or CD-ROM training program comprised of nine planning sessions from
the 2002 National Planning Conference; a live audio conference series; training rational
in ethics; audio and video packages; and the Commissioner newsletter. PCS services also
include: the planning commissioner training resource center; the planners book service;
ethics information; national and local APA workshops; and a calendar of events.12
Section 1.8
Michigan State University Citizen Planner Program

Michigan State University Extension “Citizen Planner Program " 13 The Michigan
State University Extension Citizen Planner Program, in collaboration with MSP, is a non-
credit training series leading to an optional certificate of competency awarded by
Michigan State University Extension. The Citizen Planner Program offers six basic
training sessions including: Basics of Planning Part I: Introduction to Planning-the
Comprehensive Plan- Site Plan Review; Basics of Planning Part H: Introduction to
Zoning- Zoning Process-Zoning Board of Appeals; Legal Foundations of Planning and
Zoning; Plan Implementation- Tools and Techniques Best Practices for Innovative
Planning and Zoning; and the Art of Community Planning. A certificate is awarded to
participants who complete the six basic courses and complete 30 hours of community
service in a related land use capacity. Program participants may also receive training

from a selection of other planning topics. These planning topics may vary but are

 

'2 “Planning Commissioner Service.” Aggricg Planning Association 2004 n. pag. Online. 11 Nov. 2003
<http;// www.planning.org/commissioners/index.html>

'3 “Citizen Planner Program.” Mighigg State Univgrsig Extension 2003 n. pag. Online. 12 Jan. 2004
<http://www.msue.msu.edu/cplanner>

11

primarily geared as local hot topics. Past topics have included: private property and the
takings issue; farmland preservation tools and techniques; conflict management; planning
for water quality; wetlands; principles of new urbanism; riaparian rights; recreational
planning; right to farm act; and community and economic development. The program is
provided regionally through classroom presentations and hands-on activities.

Section 1.9

Value of Training:

Three primary components contribute to the overall value of planning official
training. First, by definition, the planning official holds a wide array of planning and
municipal responsibilities. Second, these responsibilities often have significant impact
upon the development and organization of a community. Finally, most planning officials
are either elected or appointed citizens and often have limited, if any, formal experience
or knowledge of the activities and responsibilities associated with the role of the plamring
official. Training for planning officials responds to these issues by educating the citizen
on the technical and administrative intricacies of the position while reducing the level of
on the job training.

Training is also fundamental to the overall competency of the planning official.
Training can provide the planning official with the knowledge, tools, and techniques
needed to make informed decisions on technical land use and planning issues. Training
can also prepare the planning official for the administrative responsibilities associated
with the position. The American Planning Association’s “Growing Smart Legislative

Guidebook,” a guidebook containing model planning statues for communities, supports

12

planning official competency by including mandatory training in part of their model
statute recommendations. Author Stuart Meck explains,
“The purpose of the program (training) is to familiarize members with the
commission’s procedures, applicable laws of the local government, state
laws and administrative rules, plans and related technical aspects of
planning. This will ensure that each commission member understands the
broad policy and regulatory context in which the commission functions as
well as follows appropriate procedures in conducting hearings and
meetings and in decision making.” '4

Section 1.10
Value of Continuing Education:

The most prominent trend in planning official training is programs that offer
continuing education opportunities. Continuing education is an integral aspect of the
education process because it provides planning officials with the additional skills needed
to make better informed land use decisions. As communities evolve, the breadth of
knowledge and skills needed to make informed land use decisions will continue to grow.

Marshall Slagle, the former President of the Kentucky Chapter of the American
Planning Association, who spearheaded training and continuing education requirements
for planning officials in Kentucky, noted that the position of the Kentucky chapter of the
APA was that, “if we were going to have ‘smart growth’ in Kentucky, then we must have
‘smart peOple.’ '5 Slagle noted tremendous statewide support for continuing education
legislation from important stakeholders like the Kentucky League of Cities, Kentucky
Association of Counties, and Kentucky Farm Bureau. Slagle also noted support for

continuing education legislation from those he thought would be most opposed.

 

"Meek, Stuart. 1. t: it; . ' 1 'v 'dcmlt- Mn 1 ta tes rt . .i 1 r . M : n-i
of Change,, 7-36 2002 ed. American Planning Institute Publications vol. 1
' Slagle, Marshall. “Kentucky Enacts Continuing Education Requirements for Planning Officials: The

Inside Story,” Land Use gig Zoning Digest vol.53 no.9 (2001): 1

13

“HBAK (Home Builder Association of Kentucky) realized that

more informed planning commissions, boards of adjustment, and

professional planners would provide a better understanding of the

responsibilities and provide for a more level playing field — a problem

many home builders and developers perceive that they face on a statewide

basis. Further, HBAK (as well as others) understood that most persons

appointed to these boards have little or no orientation or continuing

education regarding their purpose and duties; generally knowledge about

their position was gained in the line of duty.”1

Continuing education is also important because it can address topics that tend to
be very site specific or detailed in orientation. For example, rural planning officials may
receive training in watershed management. This topic may be very important in a rural
community but not as important in more urban communities. Continuing education also
allows planning officials to receive training on the “hot issues” that may be currently
affecting their communities. Therefore, continuing education can be responsive to the
needs of the community.

Continuing education also serves as recognition that planning issues and the skills
and knowledge needed by planning officials continually change and evolve. Continuing
education is flexible and can reflect the newest and most pressing planning issues. For
example, cell-phone tower regulation has become an important topic in continuing

education curriculum. This type of issue reflects the continual change in planning issues

over time.

 

'6 Slagle 2.

l4

Section 11. Current Evaluation Methods

Section 2.1
The purpose of training evaluation

The primary components that contribute to the overall value of training and
continuing education are also fundamental to the purpose of training evaluation.
Evaluation is the tool by which the ultimate success of the training program can be
measured. According to author A.C. Hamblin,l7

“Evaluation means, literally, the assessment of value or worth.

Strictly speaking, the act of evaluating training is simply the act of judging

whether or not it was worthwhile in terms of some criterion of value, in

the light of the information available. However, in the training field,

evaluation has traditionally been taken to include, not only the assessment

of value, but also the collection and analysis of the information on the

basis of which assessment is made.”

As this paper will later discuss, the training field uses a number of evaluation
methods and techniques. General consensus as to the best training evaluation method is
not clear. However, the primary purpose behind training evaluation has been well
documented. Training evaluation literature identifies several key components that support
to the purpose of training evaluation.

Proving
Improving
Learning

Feedback
Controlling

 

‘7 Hamblin, A.C. Evalgag'gn gag control of Training. London: McGraw Hill Publishing, 1974.

15

Proving: The purpose of “Proving” derives from the need to measuring the direct
effects of training. The direct effects of training are understood by analyzing statistics
such as cost analysis, but direct effects also helps evaluate the entire training process.
“Proving aims to demonstrate conclusively that something has happened as a result of
training or developmental activities, and that this may also be linked to the judgments
about the value of the activity: whether the right thing was done, whether it was well

done, whether it was worth the cost, and so on.”'8

Improving: The purpose of “Improving” derives from the need to measure the
quality of training. “Improving implies an emphasis on trying to ensure that either the
current, or future programs and activities become better than they are at present time.”19
Therefore, the ultimate purpose of improving is to recognize which training components
require the most work. “The greatest service evaluation can perform is to identify aspects

of the course where revision is desirable.”20 This recognition will ultimately lend to better

trainers and curriculum.

Learning: The purpose of “Learning” derives from the need to include evaluation
within the overall context of the training process. “Evaluation cannot with case be
divorced fi'om the process upon which it concentrates, and therefore this slight problem

might well be turned to advantage by regarding evaluation as an integral part of the

 

'8 Smith, Easterby, Evahragn’ g Mmggment Development, Training and Education. London: Gower
Publishing, 1986.

'9 lbid., 18.

2° Ibid., 14.

16

learning and development process itself.”2| In 1970, researchers Warr, Bird, and
Rackham found the positive impact on the learning of supervisors attending an accident
prevention course was in large part due to a pre-course questionnaire administered as part
of the evaluation study. This example, where the attempt to observe something actually
changed the thing that one was observing, is better known as the Hawthorne effect.
“Alternatively, this may be used to advantage as part of the training process; the
knowledge that one’s success at learning is likely to be assessed at the end of the day

tends to concentrate the mind wonderfully.”22

Feedback: The purpose of “Feedback” derives from the need to provide valuable
information that can be used to evaluate whether training objectives are being met and
what aspects of training need improvement. “The main purpose of what we are calling
feedback evaluation is the development of learning situations and training programs to
improve what is being offered. There is a secondary aspect, as identifying what is good
and what is not so good improves the professional ability of members of the training
department. 23

Feedback evaluation also provides an opportunity to influence the facilitation of
training activities. “Timely feedback to the trainers about the effectiveness of particular
methods and about the achievement of the objectives set for the program will help in the

development of the program currently being run and those planned for future

 

2' Ibid., 14.
2’ Ibid., 18.
’3 Bramley, Peter. Evaluating Tmining Effectiveness. London: The ~1chme Training Series, 1996.

17

occasions ”2"

training cycle in figure 1.1

Figure 1.1 The training cycle25

 

Identification of
Training needs

 

 

 

 

 

Setting of
Objectives

 

 

 

I /

 

Evaluative feedback
loops

 

 

 

 

\ Running the

 

Training

 

 

The facilitation of feedback into training activities is illustrated in the

 

Selection of
' media or methods

/

 

 

 

Controlling: The purpose of “Controlling” derives from the need to maintain a

standard of training that meets the predetermined goals of the training program.

“Controlling, is a very common activity for evaluation and involves using evaluation data

to ensure that individual trainers are performing to standard, or that subsidiary training

establishments are meeting targets according to some centrally determined plan.” This

 

2‘ lbid.

2’ Ibid., 3.

l8

type of evaluation is often used when there is question the training activity is meeting its
primary objectives.

Section 2.2

Training evaluation methods (Business Field)

The process of constructing the most effective methods of citizen planner
program training evaluation will require examining the current scope of training
evaluation methods. To help in this process, this section will discuss and describe the
current training evaluation methods in both the business and planning field.

Training evaluation methods within the business field have been established for
many years. The most widely excepted methods of evaluation were created nearly 40
years ago by Donald Kirkpatrick of the University of Wisconsin. Kirkpatrick created an
evaluation framework through a series of four levels: Reaction; Learning; behavior; and
Results; from which training can be evaluated. (See figure 1.2)

The first level of evaluation is defined as the “reaction” level. The reaction level
provides information on what the training participants thought of the particular training
program. Evaluation data is focused on training program materials, instructors, facilities,
teaching methodology, delivery mechanism, and content.26 The reaction level does not
include a measure of the learning that took place. The reaction level is the most
frequently used method of evaluation. A 1986 survey of fortune 500 companies indicated
that most evaluations (86%) consist of trainee reactions that are written at the end of a

course.”

 

2‘ Basarab, David and Darrell Root. Th9 Training Evaluation Process. Boston: Kluwer Academic
Publishers, 1992.

27 Goldstein, Irwin. W Pacific Grove: Brooks/Cole Publishing Company,1993.

l9

“Reaction data reveals what the target population thinks of the
program- the participants’ reaction to and/or satisfaction with the training
program and the trainers. It may also measure another dimension: the
participants’ planned actions as a result of the training. i.e. how the
participants will implement a new requirement, program, or process, or
how they will use their new capabilities. Reaction data should be used to
adjust or refine the training content, design, or delivery. Planned action
data can be used to determine the focal point for follow-up evaluations and
to compare actual results to planned results. These finding may lead to
program improvements.28
The primary tool in evaluating reaction is the questionnaire or rating sheet. The
questionnaire can provide revealing answers about the program and provides the
participant with direct access to the evaluation process. Several guidelines should be used
in creating the questionnaire and capturing participant reaction including:29
0 Design a questionnaire based on the information obtained during the needs
assessment phase. The questionnaire should be validated by carefirlly
standardizing procedures to ensure that the responses reflect the opinions of the
participants
0 Design a instrument so that the responses can be tabulated and quantified
0 To obtain honest opinions, provide for the anonymity of the participants. Often,
it is best to provide for anonymity with a coding procedure that protects the
individual participant but permits the data to be related to other criteria, like

learning measures and performance on the job.

 

2" Phillips, Jack and Ron Stone. How to Measure Training Measures. New York: McGraw-Hill, 2002.

2” Goldstein, Irwin. Training In Qgganizatiogs. Pacific Grove: Brooks/Cole Publishing Company, 1993.

20

0 Provide space for opinions about items that are not covered in the questionnaire.
This procedure often leads to the collection of important information that is
useful in the redesign of the questionnaire.

0 Pretest the questionnaire on a sample of participants to determine it

completeness, the time necessary for completion, and participant reactions.

The second level of evaluation is defined as the “learning” level. The learning
level of evaluation determines the ability to show attainment of the principles, facts,
techniques, and skills presented in the training program.30 The learning level of
evaluation determines how well the course achieved its learning strategies by examining
learning indicators of the participant

“The evaluation of learning is concerned with measuring the extent to

which desired attitudes, principles, knowledge, facts, processes,

procedures, or skills that are presented in training have been learned by the

participants. Measures of learning should be objective, with quantifiable

indicators of how new requirements are understood and absorbed. This

data is used to confirm that participant learning has occurred as a result of

the training initiative. This data also is used to make adjustments in the

program content, design, and delivery.31

The evaluation of learning indicators is useful in measuring cognitive learning
and behavioral skills. Cognitive learning can be measured by calculating the difference
between a pre-test, administered before training begins, and a similar context post-test,

administering at the conclusion of training. The difference between the pre-test and the

post-test provides evidence as to the learning gained in the training. Behavioral skills can

 

3° Basarab, David and Darrell Root. flhg Training Evaluag'on Process. Bgston: Kluwer Academic
Publishers, 1992.

3' Phillips, Jack and Ron Stone. Ron. How to Measgrg Tgining Measures. New York: McGraw-Hill, 2002.

21

be measured using a performance—based check list administered during a workshop or
practical exercise. The check list depicts all tasks a participant must demonstrate to show
knowledge of a particular skill.32

The third level of evaluation is the “behavior” level. The behavior level of
evaluation measures how well the participants training skills or behaviors translate to job
performance. Evaluation data focuses on the participant, the work setting, and the support
systems for applying learning.

“This evaluation measures behavioral change on the job. It may

include specific application of the special knowledge, skills, etc., learned

in the training. It is measured after the training has been implemented in

the work setting. It may provide data that indicate the frequency and

effectiveness of on the job application. It also addresses why the

application is or is not working as intended. If it is working, we want to

know why, so we can replicate the supporting influences in other

situations. If it is not working, we want to know what prevented it from

working so that we can correct the situation in order to facilitate other

implementations.”

The behavior level of evaluation can assess job performance through a large
number of measuring tools including questionnaires, interviews, and observation
however, it is important to note that on the job measures should be related to the over all
objectives of the training program.

The fourth level of evaluation is the “results” level. The results level of evaluation

measures the time-span between realizing training results as to the training objectives of

the organization. Results that may be analyzed include costs, turnover, absenteeism,

 

32 Basarab, David and Darrell Root. The Taming Evaluation Process. Boston: Kluwer Academic
Publishers, 1992.

33 Phillips, Jack and Ron Stone. Ron. How t9 Measge Tam ' g Measures. New York: McGraw-Hill, 2002.

22

grievances, and morale. The most common result analyzed is cost or return of investment
(R01). R01 data is primarily focused on the monetary benefits of training.

“This is an evaluation of the monetary value of the business impact of the
training, compared with the costs of the training. The business impact data
is converted to a monetary value in order to apply it to a formula to
calculate return on investment. This shows the true value of the program
in terms of its contribution to the organization’s objectives. It is presented
as an ROI value or cost benefit ratio, usually expressed as a percentage.“

 

 

 

 

 

 

 

 

 

Figure 1.2

Kirkpatrick method of training evaluation35

Level Issue Question Answered Tool

1 Reaction How well did they like the course? Rating Sheets,
Questionnaires

2 Learning How much did they learn? Tests, Simulations

3 Behavior How well did they apply it at work? Performance Measures

4 Results What return did the training investment yield Cost-Benefit Analysis

 

Authors Jack J. Phillips and Ron Stone also consider “Intangible Benefits”
to be another important method of training evaluation that complements the purpose of
the Kirkpatrick methods. Intangible benefits focus data on the added value of the training
in non-monetary terms. “Intangible data is data that either cannot or should not be
converted to monetary values. This definition has nothing to do with the importance of
the data; it addresses the lack of objectivity of the data and the inability to convert the
data to monetary values. Subjective data that emerge in evaluation of business impact
may fall into this category (e. g., increases in customer satisfaction, customer retention,

improvements in response time to customers) Other benefits that are potentially

 

3‘ Ibid., 7.

35 Parry, Scott. Evglnag'ng the Impact 9f Training. Alexandria: American Society For Training And
Development, 1997.

23

 

intangible are increased organizational commitment, improved teamwork, improved
customer service, reduced conflicts, and reduced stress.36

Section 2.3
Training evaluations methods (Planning Field)

As outlined earlier in this paper, there are several training programs throughout
the United States specifically designed for planning officials. Training evaluation
methods often vary from program to program however, participant surveys remain the
predominant training evaluation tool.

In Michigan, the Michigan State University Citizen Planner Program with support
from a People and Land (PAL) Grant has used extensive participant surveys as the
primary method of evaluation since the programs inception in 2002. The surveys are
designed to evaluate both the process and impact of the program through a series of steps
including: the numbers of persons reached, participants’ perceptions of usefulness of
program, changes in knowledge and skills, and change in behavior or practice.”

The number of persons reached by the Citizen Planner Program was calculated
through the registration process. Data found that a total of 521 people participated in one
or more course during the thirteen program series conducted in 2003. Of those, 354
(68%) attended all six of the core sessions. For the eleven program series conducted in
2002, 54% of participants attended all six of the core sessions. 38

Participant perception of the usefulness and effectiveness of the program were

determined through surveys conducted at the end of each course and at the end of the six-

 

” Phillips, Jack and Ron Stone. Ron. 39w to Mew Training Mgggges. New York: McGraw-Hill, 2002.

37 Norris, Patricia and Wayne Beyea, Marisa Trapp, and Mike Klepinger. Final Report submitted to People
and Land PAL Project. Michigan State University Citizen Planner Program, 2003. #321

3‘ Ibid.

24

course core. The statewide averages to four specific questions measuring the

effectiveness and usefulness of the program are presented in table 1.1 for all courses held.

The opportunity for written comments on the evaluation forms produced valuable

feedback including:

:1 “Nice job, well-delivered and appropriate for my needs. Thanks!” (Clinton)

1:1 “Kept my attention for two and a half hours after a nine-hour work day -

pleasantly surprising.” (Grand Traverse)

1:1 “MSU Extension provided great handouts and a wealth of information. I will

look for future learning opportunities offered by MSU Extension.” (Kalamazoo)

:1 “Very worthwhile. Thank you.” (Lapeer)

D “Good information - topics are seldom discussed.” (Livingston)

1:1 “Very nice job - professional instruction complemented by organized materials.”

(Otsego)

Table 1.1 Participants’ Evaluation of 2003 Citizen Planner courses, Statewide Average,
Percent Responding, and Mean Score on a 1-5 Scale (with 2002 values in parentheses for
comparison) N=172039

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Poor Fair Good Very Good Excellent Mean
(%) (°/o) (%) (%) (%) Score
(out of 5)

Handouts and 0.5 1.8 5.6 45.2 46.9 4.4 (3.8)
other materials
Instructor 0.2 2 4.9 31.7 61.1 4.5 (4.1)
Overall 0.2 1.2 5.1 41.6 51.9 4.4 (3.8)
Organization
Met participant 0.6 1.6 6.4 40.6 50.8 4.4 (na)
expectations
3’ Ibid., 6.

25

 

Participant changes in both knowledge and skills were determined through a
series of pre-tests and post-tests presented at the beginning and end of each course and at
the end of the six-course core. The pre-test includes material that will be covered in the
training, and asks participants’ knowledge of, familiarity with, and use of specific tools
and concepts. To ease participant anxiety on test taking, the pre and post test were
identified as questionnaires. Upon completion of the sixth course, a post-test is
administered over the same material covered in the pre-test. The results from the
questionnaires allow for a comparison of pre- and post-test scores and analysis of
changes in knowledge or skills. The testing analysis provides information about the
increase in participants’ knowledge, but also provides guidance on where adjustments
may need to be made to Citizen Planner Program curriculum, materials, or presentation.40
Table 1.2 offers a brief summary of specific instances of increased familiarity with
concepts.

Table 1.2 Selected Examples of Pre- and Post-test Mean Scores and Changes in 2003

Citizen Planner Program Participants’ Knowledge of Concepts and Tools, Using Point
Scale of 1 to 3 (1=Never heard of this, 3=Very familiar). N=138"l

 

 

 

 

 

 

 

 

 

 

 

Pre—test Post-test
Tool/Concept Mean Mean Change
Build-out Analysis 1-47 2.28 0.8
Bundle of Property Rights 1-67 2-46 0.78
Conservation Design 1-68 2.30 0.62
Conservation Design Community Audit 1.20 1.94 0.75
Process
‘° Ibid., 7.
" Ibid.

26

 

 

 

 

 

 

Exempt Division 1.57 2.22 0.66

Overlay Zoning 1.68 2.33 0.65

Traditional vs. Participatory Planning 137 2.13 0.76
Tirnelines

You vs. I Messages 1.86 2.46 0.61

 

 

 

 

Participant changes in behavior or practice is measured through a series of

surveys, interviews, and field observations with program participants and counties of the

previous year. The changes in behavior and practice become evident over a long period

of time, therefore long-term evaluation will be required. Questions were directed to the

County Extension Directors and program participants. Questions were designed based on

specific pre-program data collected to target clearly measurable changes, as well as to

ascertain evidence of changes by individuals, boards and commissions, and communities.

During the long-term evaluation of the 2002 programs, 143 participants and 11 county

Extension contacts were reached, yielding a 95% confidence level with a +/- 6%

sampling error.42 Table 1.5 provides a brief summary of participant survey results.

Table 1.3 Results of 2003 Survey Conducted with 2002 Citizen Planner Program

Participants, Percent Responding. N=14

343

 

 

 

 

 

 

 

 

 

 

Question Response.
Since the Citizen Planner Program ended: Yes (%) No (%)

Positive change in how commission/ board meetings are 48 35
conducted
Positive changes in how meeting minutes are kept 24 57
Board/commission improved the way site visits are 23 25
conducted

‘1 Ibid.

‘3 Ibid., 8.

27

 

 

 

Positive changes in process for reviewing development
proposals

43

13

 

Board/commission paid closer attention to the legal basis
for its activities or decisions

64

18

 

Positive change in the general civility of meetings

20

59

 

Specific practices of commission/board have been corrected
because of a member’s participation in the Citizen Planner
Program

34

61

 

Changes proposed or made to comprehensive plan or
zoning ordinance

70

23

 

Participation in Citizen Planner Program improved
commturication/interaction with planning commission
members in neighboring communities

69

25

 

Changes in the way decisions are made by
board/commission regarding community’s economic
development or environmental protection policies

43

52

 

 

Would recommend Citizen Planner Program to new
members of board/commission

 

96

 

 

 

'Some participants chose not to answer questions or responded N/A

In some instances, 2003 survey results were combined with information fi'om the

pre— and post-program questionnaires to provide evidence of change. (Table 1.6)

Table 1.4. Comparison of 2002 Citizen Planner Program Participants’ Responses to
Questions Related to Preparedness For and Confidence With Membership on Board or

 

 

 

 

 

 

 

 

 

 

Commission.44
2002 P t-
Example of Preparedness or 2002 Pre- OS 2003 Follow-up
Confidence program program (% responding)
("/0 responding) (% responding)
Have read some, most, or all of 70 87 1 91 6
comprehensive plan ' '
Have read some, most, or all of 89 2 92 3 93 7
zoning ordinance ° ' °
Prepared for job as commission 50 63 7 60 8
or board member ' '
“ Ibid., 9.

28

 

Beyond the formal evaluation of the Citizen Planner Program and its impact on
knowledge and practice, evidence of impact is increasingly available fi'om program
participants.

Manistee County

Jerry Mathieu, program participant, is applying the tools and information
he gained from the program in a campaign to install a sewer system
around Manistee County’s Bear Lake. The lake is surrounded by SO-foot
lots containing septic tanks and wells that may leak nitrates and other
substances that threaten water quality. Mathieu has worked tirelessly to
build support for the sewer initiative, despite its substantial monetary cost.
“Citizen Planner was very helpful in [teaching me] how to deal with
people’s reactions to controversial issues such as this,” he said. “It also
increased my awareness of the implications that growth and land use
practices can have on a community, even rural conununities.” Time and
effort are still needed to complete the project, and as he awaits news
regarding project funding, he is soliciting support from each municipality
involved.

Monroe County

In Northfield Township, Ken Dignan, Zoning Board of Appeals member
and program participant, feels that the Citizen Planner Program was an
exceptionally valuable experience. According to Mr. Dignan, “As a
member of our ZBA and a very active community member, it has allowed
me to lend droughts to our board members and planning commissioners as
we face significant challenges with a great deal of potential growth and
change in our community.” Mr. Dignan also noted, “With my
conversations with fellow officials and community members, there is a
great deal of interest in future programs from many in my community.”

Section 2.4
Hamilton County Regional Planning Commission

The Hamilton County (Ohio) Regional Planning Commission “Making Great
Communities Certified Planning Commissioners Program” is another citizen planning
program utilizes the survey evaluation. Surveys are distributed at particular stages of the
training program, including a participant background survey at the time of enrolhnent, a

participant statement of intent prior to the beginning of the workshop, an evaluation at the

29

conclusion of each workshop, a follow up survey of each participant, and an annual
survey for commissioners. Program evaluation forms included sections for criticism and
suggestions. The program evaluation forms also provided participants the opportunity to
rate the training included‘criticism and suggestions, and rate the program on several
important questions. 2003 program evaluation forms found that, 94% of the attendees felt
the value of the content presented was excellent or good, and 90% of attendees thought
the quality of the speakers was at the same level. The evaluation forms also found 80% of
the respondents said they would take the workshop again, and 100% said they would
recommend it to a friend. The data gathered from the 20003 surveys was used to make
program improvements. 45

According to Hamilton County Regional Planning Commission senior planner
program coordinator Catalina Landivar-Simon, the training program has also actively
acted upon other evaluation methods including, examining how many program trainees
ask for advanced training, and examining how many jurisdictions have established
training within their annual budget.46
Section 2.5
Main Street Program:

While the fundamental purpose of this paper is to identify and analyze training
evaluation methods, the planning field provides an excellent opportunity to examine how

other programs, un-related to training, are evaluated. For example, the National Main

 

’5 Vondrell, James. Making Great Communities ’ Certified Planning Commissioner 's Program External
Evaluation. Prepared for the Hamilton County Regional Planning Commission- Planning Partnership, 2003

‘6 Landivar-Simon, Catalina . Personal interview. 12 April. 2004

30

Street Program uses a series of criteria evaluation that measure the effectiveness of their
programs.

The Main Street Program is facilitated through the National Main Street Center of
the National Trust for Historic Preservation, and is designed to improve all aspects of the
downtown or central business district, producing both tangible and intangible benefits.
Improving economic management, strengthening public participation, and making
downtown a fun place to visit are as critical to Main Street's future as recruiting new
businesses, rehabilitating buildings, and expanding parking. Building on downtown's
inherent assets -- rich architecture, personal service, and traditional values and most of
all, a sense of place -— the Main Street approach has rekindled entrepreneurship,
downtown cooperation and civic concern. It has earned national recognition as a practical
strategy appropriately scaled to a community's local resources and conditions. And
because it is a locally driven program, all initiative stems from local issues and
concerns."7

Main Street Programs can be formed at the local level and are primarily already
established commercial district revitalization programs. In order to receive National Main
Street Programs status, local main street programs must meet 10 performance standards.

The Ten Standards include:

0 Has broad-based community support for the commercial district revitalization

process, with strong support fi'om both the public and private sectors

 

‘7 “About the main Street Progranr,” atiogl Main Street Center 2004 n. pag. Online. 1 May, 2004
<www.mainstreet.org>

31

0 Has developed vision and mission statements relevant to community conditions
and to the local main street programs organizational stage

0 Has a comprehensive Main Street Work Plan

0 Possesses an historic preservation ethic

0 Has an active board of directors and committees

0 Has an adequate operating budget

0 Has a paid professional Program Manager

0 Conducts a program of ongoing training for staff and volunteers

0 Reports key statistics

0 Is a current member of the National Trust’s Main Street Network membership

Program

The ten standards provide benchmarks and guidelines on how the organization
should be function including providing incentives to organizations to perform better and
be more effective. The benefits of being recognized as a National Main Street community
include national recognition and promotion, web-site listing, and the fact that the
organization is a much stronger and better functioning organization.

The ninth performance standard, reporting key statistics, establishes the criteria
evaluation. Statistics determine tangible measurement of the Main Street Programs
success and is crucial to garnering financial and program support. 48 Statistical data is
collected on a regular ongoing basis and can include:

0 Community Population

0 Net of all gains and losses in jobs

 

’8 Guzman, Tom. Personal interview. 21 April. 2004

32

0 Net of all gains and losses in new business

0 Number of building rehabilitation projects

0 Number of public improvement projects

0 Number of new construction projects

0 Number of housing units created: upper floor or other

0 $ Value of private investment spent in above projects

0 $ Value of public investment spent in above projects

0 $ Value total of all investment add public and private investment

0 Ground-floor vacancy rate when your program started

0 Ground-floor vacancy rate now

0 Rental rate per square foot when program started

0 Rental rate per square foot now

0 Your program’ annual operating budget

The performance criteria established by the Main Street Program provides an

excellent measure of evaluation. By looking at criteria of the Main Street Program,
training programs may be able to better develop criteria evaluation.

Section III.
Methods of Evaluation

The evaluation tools used within the business and planning community provide a
strong foundation fiom which other evaluation methods can be determines. The following
section will discuss and demonstrate several additional methods that would effectively
measure whether citizen planner training programs produce more effective planning

officials.

33

Section 3.1
Testing

One method of evaluation that would effectively measure whether citizen planner
training programs produce more effective planning officials is testing. Based upon the
Kirkpatrick “learning” level of evaluation, testing would measure cognitive learning and
behavior skills. Cognitive learning can be measured by administering a pre-test given
before training begins, and a post test at the conclusion of training. The Michigan State
University Citizen Planner Program already uses this method in their evaluation.
However, test results recognize the participants’ perception of knowledge. A more
effective test would measure knowledge of specific principles, facts, techniques, and
skills.

Testing has unpopular sentiment in both the business and planning field. In the
business field, critics argue that employees resent or are at least suspicious of being asked
to take tests.49 In the planning field, due to the voluntary nature of most training
programs, and popular distain most adults feel for testing, facilitators are reluctant to
include testing. This reluctance was supported earlier in this paper in the description of
the Michigan State University Citizen Planner Program which found that participants
accepted pre and post testing when they are identified as questionnaires.

Regardless of the sentiment of testing, testing can be more effective than
subjective judgment on decisions regarding the value of training. For example, testing is
valuable to the instructor because they supply one of the most important sources of

information as to how well the instructor is meeting the objectives of the unit of

instruction. Testing is important to the trainee because they indicate what progress the

 

’9 Denova, Charles. [gs]; gggmflgn for training gvalggg'on. New York: Van Nostrand Reinhold
Company, 1979.

34

trainee is making, assist in the diagnosis of the areas of difficulty, helps distinguish

between the relevant and the irrelevant, and can provide incentives toward greater

effort."0

The ultimate purpose of testing should be to measure the degree to which

individuals have attained the desired outcomes of the training activities. Therefore, test

construction should follow several key steps.S '

Determine the scope of the test. Does the test cover a lesson, a unit, a phase, a
specific job, or some other measurable part of the training activities?
Determine what is to be measured. What was the objective of the training
program? Design a test that measures attitudes, abilities, skills, or mastery of
principles and! or facts

Select the test items. Write items for each topic and/or subtopic without regard to
the number of test items that will be used in the final draft.

Select a technique. Try to select the training technique most suitable for the
purpose of the test.

Fix the length of the test. Choose the number of items that will cover the
instructional material adequately.

Select the final items. Choose those items that treat the most essential and
significant portions of instruction. Never use a test item to measure material not

covered by the instructional activities.

 

”Ibid., 11.

5' Ibid.

35

0 Arrange the items in final form. Groups similar items together and arrange them
in an approximate order of their difficulty
0 Prepare directions for the test. The instructor is obligated to make it perfectly
clear what the trainee is to do and how the trainee is to do it.
0 Prepare a scoring device. Scoring devices aid in the speed and accuracy of
evaluating the trainees.
0 Question the questions:
a. Does the question cover the important or useful aspect of the training lesson,
course , and/or program?
b. Is the question stated in the language of the learner?
c. Does the wording give away the answer?
d. Does this question give the answer to another question?
e. Is the question phrased in the negative? If so, change it
f. Is this test item just on memory? Items should measure application, not just the
memorizing of facts
g. Is the test item of a catch or leading type? The trick question should be avoided
Selecting a testing technique is an important component in successfully meeting
the objectives of a test. The in-depth knowledge provided by most citizen planner
programs would indicate an essay question format would be the most effective testing
technique. Essay questions are used to measure objectives dealing with understanding,
attitudes, interests, creativity, and verbal expression. The strengths of the essay format
also include (1) Freedom of expression and creativity & (2) Emphasis on a participant’s

depth and scope of knowledge of the subject manner.

36

Essay questions allow the trainee the opportunity to answer the test question in
the manner they feel is best. A well constructed essay question will also measure the
learner’s ability to deal with subject manner at their level of understanding, to organize

their thoughts, and to express themselves in writing. To help ensure this takes place, a

few measures need to taken into consideration.52

1. Design a test with many short answer essays instead of a few long answer essays.

2. Essay questions must be written in the language of the participants and should be
precise in meaning and unambiguous and enough detail must be provided in the question
to channel the trainees toward the correct response.

The construction of an essay test should also keep in rrrind the following

suggestions when writing and using essay-type test items.53

1. Use essay test items only for those functions for which they are best suited

2. Design essay items so that they measure the trainee’s ability to apply the principles that
have been taught

3. Employ a relatively large number of short answer items rather than a relatively smaller
number of long-answer items.

4. Ask for specific Answers

5. provide enough detail in the essay question so that the trainee has an outline to follow
or is accurately aimed toward the correct formulation of a response

6. Require all individuals to answer the same questions

7. All essay items must mean essentially the same thing to everyone who knows the
material

8. Suggest a limit (space, words, and time) for each essay item.

9. Do not use essay items that are linked to the solution of a problem stated or that
respond to another question

10. Phrase essay questions so that they encourage the demonstration of high levels of
understanding

 

’2 Ibid., 31.

’3 Ibid

37

For the purpose of this paper, I have constructed a sample essay test (Figure 1.6)
based on the Michigan State University Citizen Planner Training Program training

material. Essay questions are based on the training programs six core modules

Figure 1.3 Citizen Planner Training Program Essay Test

J

Basic Training Part I:

(1) Describe the primary purpose of a comprehensive plan, and
describe at least three components found in most comprehensive
plans?

(2) Describe the relationship between the Comprehensive Plan and the Zoning
Ordinance considering: (a) adoption authority, (b) function, (c) and what it
describes?

Basic Training Part II:

(1) You are the chair of a planning commission. At the next the planning commission
meeting, a small number citizens, concerned about a new golf course development
proposal on nearby wetlands, become heated and begin to monopolize the meeting. As
the chair of the commission how would you ensure all sides are able to give their
opinion?

(2) Describe at least three of the seven guidelines used when making ordinance
interpretations?

Legal Foundations of Planning and Zoning:

(1) Define what is meant as a “bundle” of rights, and how does it relate to personal
property interests?

(2) A local artist in a sleepy Midwestern village applies to convert the up-stairs of his
home into an art studio. He plans to use the studio for painting lessons, and as a
showroom where art dealers can purchase his work. The current zoning ordinance
classifies the home as single family, non-commercial. As a planning commissioner,
what mechanism would expect the artist to apply for to allow for the studio?

Plan Implementation— Tools and Techniques:

(1) Describe one disadvantage and two advantages of a Planned Unit Development

38

(PUD) from the perspective of the Developer?
Municipality?

(2) Discuss the function of overlay zones and how they might be effective in your
community?

Best Practices for Innovative Planning and Zoning:

(1) Describe the three ways in which “Conservation Design” is different than “clustering”
as it pertains to density and open space?

(2) Describe the five characteristics of Traditional Neighborhood Design and explain the

social benefits of each characteristic?

The Art of Community Planning:

(1) Describe at least three techniques used to conduct better planning commission
meetings considering: (a) Preparation, (b) Starting/Conducting Meetings, (c) End of
Meeting, and ((1) After the meeting

(2) You are the chair of a planning commission. At the next planning commission
meeting a neighborhood block group, angry over a special assessment tax to construct

new sewer lines on their street, become angry and begin yelling and disrupting the
meeting. Describe at least three ways in which you would manage the conflict?

Section 3.2
Cost Analysis

A second method of evaluation that would effectively measure whether citizen
planner training programs produce more effective planning officials is cost analysis.
Based upon the Kirkpatrick “results” level of evaluation, cost analysis would measure the
value of training in monetary terms and calculate a return-on-investment (ROI)

The monetary value of citizen planner training can be difficult to determine based
on the fact that training effectiveness is predominately based on planning decisions which

does not necessarily have often have monetary value. However, citizen planner training

39

 

has monetary impacts on one of the most significant issues facing planning officials and
commtmities; land use liability.

Many communities and municipalities understand the significance of training in
relation to their legal liability. The ramifications of planning official decisions can
potentially expose communities to legal risk. “The Association of Washington Cities
(the state of Washington’s equivalent organization to the Michigan’s Municipal League)
believes their training, very formal review of municipal codes and availability of a second
opinion for that review and dispute resolution process has reduced land use liability.”4

In Michigan, the connection between training and municipal liability has also
been well documented. In a 1991 Zoning News article, Mark Wycoff believed the
investment in training is seen as a “good buy, considering their potential legal liability for
questionable zoning decisions and the seriousness of the boards land use powers.”55 The
Michigan Municipal League identifies liability as the primary influence behind its
training for officials it supports, and cites the 1989 US Supreme Court case Geraldine
Harris v. City of Canton, Ohio. In that case the Court held

“Under certain circumstances, a municipality can be held liable in

civil rights action under 42 USC 1983 for constitutional violations

resulting from its failure to train municipal employees. This failure to train

amounts to deliberate indifference to the rights of persons with whom the

municipality comes into contact. Focus must be on adequacy of the

training program in relation to the tasks particular officers must
perform.”56

 

“Chasco, Paul. Assistant Director for Insurance Services for the Association of Washington Cities. E-mail
to Kurt Schindler. Michigan State Extension Agent. July. 2002.

’5 Schwab, Jim. “Training Citizen Planning Officials,” MM June 1991: 3

56 “Education Programs.” MW (2004): n. pag. Online.
6 Dec. 2004. <http://www.mrnl.org/education/education_programs.htm>

40

Marshall Slagle noted that when the Kentucky legislation for continuing
education requirements was introduced, “it was supported by the KLC (Kentucky League
of Cities) as they saw this as a way to improve upon liability insurance that they offered
to their member cities- if you adopted good regulations and had good continuing
education for your people you became a better [less] risk?7

“Lawsuits arising out of planning commission decisions make up a small portion
of the litigation that is defended by the LLM Insurance Pool. On average the figure is less
than 5%.” However, “the average cost of a land use lawsuit to a municipality is about
$40,000 with ranges generally from $25,000 to $75,000.”58 With typical training
programs not exceeding $400 the monetary value fi'om potentially reducing land use
liability is cost effective. “The extent to which training helps planning officials make
better decisions, [as individuals and as a group], this would show up in the absence of
valid claims and damages paid. This would result in lower premiums?”9

Return on investment (ROI) originates from the finance and accounting field and
usually refers to the pre-tax contribution measured against controllable assets. It measures
the anticipated profitability of an investment and is used as a standard measure of the
performance of divisions for profit centers within a business. For training program

evaluation, the investment portion of the formula represents capital expenditures such as

development and delivery costs.(’0 The calculation of the return for a training program is

 

‘7 Slagle, Marshall. E-mail to Harry Burkholder. 10 Feb. 2004

58 Foster, Michael. Assistant Director, Risk Management Services, MML
E-mail to Harry Burkholder, May 5"I 2004

‘9 Murphy, Kevin Pool Administrator, MML
E-mail to Harry Burkholder May 3"I 2004

41

not always feasible, however placing a lower premium rate in place of the net program

value in formula 1.5 produces a cost effective equation.

 

 

Figure 1.4 ROI Formula: Figure 1.5 ROI (Training Program Evaluation)
Pre-tax earnings Net program value (or savings)

ROI= X 1000 ROI= X 1000
Average Investment Program Costs (or investments)

Section 3.3

Performance Criteria
The final method of evaluation that would effectively measure whether citizen

planner training programs produce more effective planning officials is performance
criteria. Based upon the Kirkpatrick “behavior” and “results” methods, and the Baseline
Data of the Main Street Program, the establishment of performance criteria would
measure the effect of training on job performance, specifically analyzing variables such
as turnover, absenteeism, grievances and morale. For the purpose of this paper I have
outlined five measures of performance criteria: officials requesting advanced training;
planning official turnover rate; number of new planning officials requesting training;
length of commission meetings; and number of planning decision made. It is important to
note that before any criteria can be established, a consensus as to how the criteria should
be measured will have to be addressed.

The first evaluation criteria method will measure how many planning officials,
have since participated in, or have requested some form of advanced training, after
participating in some form of initial training. The evaluation process would include

distributing surveys asking participants if they have been involved in additional training

 

6° Basarab, David and Darrell Root. WW Kluwer Academic
Publishers, 1992.

42

outside or within the program in which they received their initial training. Repeating the
same training or continuing participation in a training program indicates a level of
satisfaction with the initial form of training. Additional training will increase the
knowledge and skills of the participant, therefore increasing the effectiveness of training.
Therefore, the request for additional training speaks to the quality of the training program
and demonstrates the value of training objectives. In 2003, over 100 participants
attended the Michigan State University Citizen Planner “Advanced Academy.” The
Academy provides former program participants the opportunity to continue training at the
advanced level. The Hamilton County Regional Planning Commission found that 80% of
survey respondents said they would also take the training workshop again. The high
number of participants who attended the Academy, the high percentage of respondents
who said they would repeat the training, further supports the notion that additional
training produces more effective planning officials.

The second evaluation criteria method will measure the turnover rate of planning
officials who participate in training. The most common reason for planning official
turnover is a lack of understanding for the duties and responsibilities of the position.
Many new officials become overwhelmed by the position and either quit or do not seek
another term. Most training programs provide courses on the role of the planning official
and provide tools to help the new planning official cope with their responsibilities. As
training helps acclimate the planning official into the position, the participant will
become more comfortable with their roles and responsibilities. As the participant
becomes more comfortable, the likelihood of remaining in the planning capacity

increases. If planning the boards are cohesive and maintain consistency, they are more

43

prone to be effective planning officials. The evaluation process would include the
distribution of surveys to former training participants asking them if they still reside on a
planning commission.

The third evaluation criteria method will measure the number of new planning
officials who request training. This would be accomplished through a question during the
registration process that would identify new planning officials. The request for training
by new planning officials resides from either a personal desire to become a more
effective planning official, or from recommendations by other planning officials. %96 of
past program participants said they would recommend the Michigan State University
Citizen Planner Program to new commission members, and %100 of past program
participants said they would recommend the Hamilton County Regional Planning
Commission training program to new commission members. Citizen planner training
programs have the ability to produce effective planning commissions or boards. If one
member leaves the board, the new official’s knowledge and skills will most likely not be
up to par with the experienced planning board. To catch up to the current board level, the
new official will either seek training or be asked to receive training by the other board
members. The level of excellence, due to training, established by the existing members
influenced the decision of the new member to acquire the same training. Therefore,
training programs that effectively promote former participants to recommend training to
new planning officials is effectively producing better planning officials.

The fourth criteria evaluation method will be to measure the length of planning
commission meetings. A number of citizen planner programs focus curriculum on the

process of conducting more effective planning commission meetings. Training sessions

44

include course material on argument facilitation, avoiding conflict, meeting facilitation,
and meeting timelines. All of these courses relate to the length of planning commission
meetings. Therefore, if training was effective, there should be a noticeable difference in
the length of the most planning meetings. Measurement of this criteria, would involve
logging the length of each planning commission meeting over a period of time beginning
at the first meeting following the training.

The last criteria evaluation method will be to measure the number of decisions
made per planning commission meeting. Much like the previous criteria, planning
decisions are often tied to the efficiency of planning meetings. In many instances,
decisions are tabled and placed on the next meetings agenda. This is often in response to
the lack of efficiency of most planning meetings. Therefore, if training is effective, there
should also be a noticeable difference, due to meeting efficiency training, in the number
of planning decisions reached.

Citizen planner training programs provide a valuable service to planning officials
and active citizens in land use. The goal of most training programs is to produce more
effective planning officials, however the methods of evaluating this process have not
been clearly demonstrated. Survey methods remain the predominant method of program
evaluation. It is my contention that citizen planning programs can also be effectively
evaluated to measure their effectiveness through testing, examining cost analysis, and

meeting established criteria.

45

llllllllllllIlllllllllllllllIllllllIllIlllllllllllllllllllll

 

31293 02637 8145