 

b. J
t... .4
:2..an3

 

it... .4 II 5 .
Stir. ) ‘ 1.. t

Vﬁu. 3v. :

. H!!!

r‘l En. L

1 5“. M...
00

3 q!
.13.”...

n :3 7...
331.32?
.. 1.... .3 : .

($.13: . o. 1

9 ..
. 3.5:.
.. 1.5%.
{x .

:l
.25
.II. 3!

lﬁ

.«z,

 

THESIS

’/
’)

.-

"" Hg]

/: \-'\/\a’

|ll|llll||\tHIlllllllHlWNWHHIHHHIHUIHIHW

3 1293 021061

 

Michigan State

 

LIBRARY
University

 

 

This is to certify that the

dissertation entitled

The Effects of Practice Variability and Velocity Feedback
on the Development of Basic & Strategic Training Skills

presented by

Morell E. Mullins, Jr.

has been accepted towards fulﬁllment
of the requirements for

Ph. D . degree in Psychology

 

 

MWA

Date 3/”? 24”

MSU is an Afﬁrmative Action/Equal Opportunity Institution 0-12771

PLACE IN RETURN BOX to remove this checkout from your record.
TO AVOID FINES retum on or before date due.
MAY BE RECALLED with earlier due date if requested.

 

DATE DUE DATE DUE DATE DUE

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

11/00 CJCIRCIDatoOuepﬁs-p.“

THE EFFECTS OF PRACTICE VARIABILITY AND VELOCITY FEEDBACK
ON THE DEVELOPMENT OF BASIC AND STRATEGIC TRAINING SKILLS

By

Morell E. Mullins, Jr.

A DISSERTATION

Submitted to
Michigan State University
in partial ﬁJlﬁllment of the requirements
for the degree of

DOCTOR OF PHILOSOPHY
Department of Psychology

2000

ABSTRACT

THE EFFECTS OF PRACTICE VARIABILITY AND VELOCITY FEEDBACK
ON THE DEVELOPMENT OF BASIC AND STRATEGIC TRAINING SKILLS

By

Morell E. Mullins, Jr.

A study is presented in which two training manipulations (practice variability and the
provision of velocity feedback) were hypothesized to bring about increased self-
regulation on the part of trainees. This increased self-regulation was then hypothesized to
lead to the enhancement of both basic and strategic performance during training, which
along with declarative and adaptive knowledge about the task should inﬂuence
performance on a more difﬁcult generalization version of the training task. It was
suggested that situational judgment testing provided an ideal assessment methodology for
adaptive performance, and a situational judgment test was developed for this study.

Contributions of the study are discussed.

For Cindy... Without you, I wouldn’t be here

iii

ACKNOWLEDGMENTS

As with my Master’s Thesis, I ﬁnd myself intimidated, if only slightly, by the
idea that I have something for which writing acknowledgments is worthwhile. I put a
great deal of time and effort into this dissertation, but none of it would have been possible
without the continued assistance, guidance, and input from many individuals.

First and foremost, I must acknowledge the inﬂuence of my advisor, Steve
Kozlowski. Throughout my graduate career, Steve encouraged me to be my best, placing
challenge alter challenge in front of me to help me grow into a stronger scientist-
practitioner. While we didn’t always agree, I learned as much from those times when we
clashed as any other, and I leave Michigan State counting Steve not only as my mentor,
but as my ﬁiend. I would be remiss if I did not also thank my committee members —
Kevin Ford, Rick DeShon, and Ann Marie Ryan. My dissertation provided special
challenges, and they all handled it (and me!) with great aplomb, always willing to provide
assistance when I was struggling. Their advice throughout the process was invaluable,
and I feel extremely fortunate to have worked with each of them.

On a more personal level, I developed a substantial support network of friends
and family during graduate school, all of whom kept me going when things got rough.
The most important such individual is my wife, Cindy, who busted her butt to keep me
going, and whose love inspired me more than I think even she could know. My mom and
dad provided constant support from back home in Arkansas, and knowing how proud
they were of me kept me going, as did the encouragement I received ﬁ‘om my sister Carol

and brother Jeff.

iv

My friends... I’m fortunate, because there are almost too many to name. My
graduate student mentors Ken Brown and Stan Gully showed me what it took to be
successful at MSU, and move on into the academic world, and the students I went
through the process with — Mike, Ann, Lori,.KD, Karen, and everyone else — constantly
reminded me that I wasn’t alone. Outside of school, the support and encouragement
offered by the wonderﬁil group of people I wish I’d met three years earlier helped me
keep things in perspective. Christy, Jason, Jae, Anne - I love you all.

I think that every student ﬁnds, at the end of graduate school, that what has been
accomplished is a teameﬂ‘ort for which only the person with the diploma has anything
tangible to take away from it. To everyone who has helped me — faculty, students, .
family, and friends — you have my undying gratitude. The only letters I can offer to put
aﬁer your names are F.o.M. You’re all Friends (or Family) of Morrie. I could not have

done it without you all, and would not have wanted to.

TABLE OF CONTENTS

LIST OF TABLES ................................................................................................ ix
LIST OF FIGURES ............................................................................................... xi
INTRODUCTION ................................................................................................. 01
TRAINING OUTCOMES ..................................................................................... O4
SELF-REGULATION ........................................................................................... O9
Self-Regulation and Goals .......................................................................... ll
Self-Regulation and Feedback .................................................................... l4
Manifestations of Self-Regulation ............................................................... 17
The Self-Regulatory System ....................................................................... I9
TRAINING MANIPULATIONS ........................................................................... 26
Practice ...................................................................................................... 26
Feedback .................................................................................................... 32
SITUATIONAL JUDGMENTS TESTS: MEASURING
ADAPTIVE KNOWLEDGE ...................................................................... 38
CONSTRUCT DEFINITIONS .............................................................................. 42
Self-Regulatory Skill .................................................................................. 42
Cognitive Ability ........................................................................................ 43
Mastery Goal Orientation ........................................................................... 43
Performance Goal Orientation .................................................................... 43
Practice Variability ..................................................................................... 43
Velocity Feedback ...................................................................................... 44
Self-Efﬁcacy .............................................................................................. 44
Self-Regulatory Activity ............................................................................. 45
Declarative Knowledge .............................................................................. 46
Adaptive Knowledge .................................................................................. 46
. Basic Performance ..................................................................................... 46
Strategic Performance ................................................................................ 47
HYPOTHESES ..................................................................................................... 47
Practice Variability - Self-Efﬁcacy ............................................................. 48
Practice Variability - Self-Regulation ......................................................... 48
Velocity Feedback - Self-Efficacy .............................................................. 49
Velocity Feedback - Self-Regulation .......................................................... SO
- Self-Efﬁcacy — Proximal Training Outcomes .............................................. 50
Self-Regulation — Adaptive Knowledge ...................................................... 52

Self-Regulation — Strategic Training Performance ...................................... 53

Declarative Knowledge - Adaptive Knowledge .......................................... 53
Declarative Knowledge —- Basic Generalization Performance ....................... 53
Adaptive Knowledge — Strategic Generalization Performance ..................... 54
Training Performance - Generalization Performance .................................. 55
Mediational Hypotheses ............................................................................. 56
METHOD ............................................................................................................. 57
Design ....................................................................................................... 57
Overview ........................................................................................ 57
Simulation ...................................................................................... 58

Skill Components ........................................................................... 60
Performance Dimensions ................................................................ 63
Procedure .................................................................................................. 63
Participants ..................................................................................... 63
Informed Consent ........................................................................... 64
Experimental Training/Demonstration ............................................. 64
Experimental Protocol .................................................................... 66
Training Scenarios .......................................................................... 66
Generalization Scenario .................................................................. 66

Goals .............................................................................................. 67
Incentives ....................................................................................... 68
Manipulations ............................................................................................ 68
Practice .......................................................................................... 69
Feedback ........................................................................................ 70
Control Measures ....................................................................................... 7O
Demographic Information ............................................................... 7O
Cognitive Ability ............................................................................. 70

Goal Orientation ............................................................................. 70

Goal Commitment .......................................................................... 7O
Self-Regulatory Skill ....................................................................... 70
Process and Outcome Measures ................................................................. 71
Self-Efﬁcacy ................................................................................... 71

Basic Task Knowledge ................................................................... 71
Adaptive Task Knowledge .............................................................. 71
Self-Regulatory Activity ................................................................. 72
Post-Experimental Screening Measures ...................................................... 73
Motivation ...................................................................................... 74
Feedback Satisfaction ..................................................................... 74
Analyses ..................................................................................................... 74
RESULTS ............................................................................................................. 79
Hypothesis Tests ........................................................................................ 87
Manipulation Effects .................................................................................. 101
Decomposition of Model Linkages ............................................................. 105
Summary .................................................................................................... 113

vii

Supplemental Analyses ............................................................................... 114

DISCUSSION ....................................................................................................... 1 16
Overview of Results ................................................................................... 117
Contributions ............................................................................................. 128
Limitations and Implications ....................................................................... 134
Directions for Future Research ................................................................... 140
Conclusion ................................................................................................. 144

REFERENCES ...................................................................................................... 145

APPENDICES ...................................................................................................... 159
Appendix A: Consent Form ....................................................................... 160
Appendix B: Debrieﬁng Form ................................................................... 161
Appendix C: Award Instructions ............................................................... 162
Appendix D: Feedback Screens ................................................................. 163
Appendix E: Measures .............................................................................. 168

Demographic Information ............................................................... 169
Goal Orientation ............................................................................. 170
Goal Commitment .......................................................................... 171
Self-Efﬁcacy ................................................................................... 172
Declarative Knowledge Test ........................................................... 173
Adaptive Knowledge Test ............................................................... 176
Self-Regulation “Diary” Entry ......................................................... 180
Self-Regulatory Activity Items ........................................................ 181
Self-Regulatory Skill Questionnaire ................................................ 182
Motivation Questionnaire ............................................................... 183
Feedback Satisfaction Questionnaire ............................................... 184
Appendix F: Figures .................................................................................. 185

viii

Table 1:

Table 2:

Table 3:

Table 4:
Table 5:
Table 6:
Table 7:
Table 8:
Table 9:

Table 10:

Table 11:
Table 12:

Table 13:
Table 14:
Table 15:

Table 16:

LIST OF TABLES

Hypotheses, Constructs, Measures, and Analysis Plan ............................. 75

Means, Standard Deviations, Intercorrelations, and
Internal Consistency Reliability Estimates ................................................ 80

Multivariate Effects of Control Variables and Manipulations
on Self-Efficacy and Self-Regulation at End of Trial Blocks

3, 6, and 9 .............................................................................................. 88
Predicting Declarative Knowledge at the End of Training (HS) ................ 91
Predicting Adaptive Knowledge at the End of Training ............................ 92
Predicting Basic Performance at the End of Training (H7) ....................... 93
Rotated Factor Matrix, Trial 9 Performance Dimensions ......................... 94
Predicting Strategic Performance at the End of Training (H8, H10) ......... 95
Predicting Basic Performance During Generalization (HI2, H14) ............ 97

Predicting Strategic Performance During Generalization
with Adaptive Knowledge and Strategic Performance at

End of Training (Hl3, H15) .................................................................. 98
Manipulation Effects on Self-Efficacy at End of Training ....................... 99
Manipulation Effects on Self-Regulation at End of Training ................... 100

Effects of Manipulations on Basic Performance at
End of Training ..................................................................................... 102

Effects of Manipulations on Strategic Performance at
End of Training ..................................................................................... 103

Effects of Manipulations on Basic Performance During
Generalization ....................................................................................... 103

Effects of Manipulations on Strategic Performance During
Generalization ....................................................................................... 104

ix

Table 17:

Table 18:

Table 19:
Table 20:
Table 21:
Table 22:

Table 23:

Effects of Process, Knowledge, and Performance Variables
on Strategic Generalization Performance ............................................... 107

Effects of Process, Knowledge, and Performance Variables

on Basic Generalization Performance .................................................... 108
Effects on Basic Performance at End of Training ................................... 110
Effects on Strategic Performance at End of Training .............................. 110
Predicting Declarative Knowledge ......................................................... 111
Predicting Adaptive Knowledge ............................................................ 112
Manipulation Effects on Self-Regulation Diary at End of Training ......... 115

Figure 1:
Figure 2:

Figure 3:

Figure 4:

Figure 5:

Figure 6:

LIST OF FIGURES

The Self-Regulatory System (Kozlowski et al., 1997) ............................. 186
Experimental Model (Theoretical) .......................................................... 187
Proﬁle Plot of Means for Self-Efficacy Following

Trials 3, 6, and 9 .................................................................................... 188
Proﬁle Plot of Means for Self-Regulation Following

Trials 3, 6, and 9 .................................................................................... 189
Revised Model, Post-Analytic ................................................................ 190
Trainee Score Over Nine Training Trials ................................................ I91

INTRODUCTION

It has become increasingly clear in the last two decades that training design
should concentrate on moving beyond "fads" and instead look for those options, available
to trainers, which are most likely to result in the greatest learning, skill generation, and
ultimately, skill transfer and retention. The advancement of training as a ﬁeld of study
within scientiﬁc psychology requires that questions be asked not only about how we can
best generate the kinds of results organizations value, but also why the manipulations we
employ can and should be effective. The literature on training design offers many
potential avenues to be explored. Design elements that have received attention in the
literature include: sequencing of materials (Dansereau, Brooks, Holley, & Collins, 1983);
provision of feedback (Earley, Northcraﬁ, Lee, & Lituchy, 1990; Waldersee & Luthans,
1994); amount of control given to learners over their learning experiences (Tennyson,
1980, 1981; Gay, 1986; Nelson, Dunlosky, Graf, & Narens, 1994); variability of training
experiences (Schmidt & Bjork, 1992; Schmidt, 1975); and provision of goals (Locke &
Latham, 1990; Latham & Locke, 1991). While these do not constitute a closed set of
training manipulations, they should serve to illustrate the kinds of things that can be done
within training programs to assist trainees.

In this study, I have examined the effects of two speciﬁc training manipulations,
based around the nature of the practice in which trainees engage and the feedback they
receive about their practice. These general areas are two that should receive increased
attention as training design efforts become more complex and begin to ﬁilly utilize the
technologies available. Kozlowski, Toney, Mullins, Weissbein, Brown, and Bell (in

press) argue that the capabilities to (I) generate various types of skill-developing practice

scenarios and (2) provide detailed feedback information regarding those scenarios, are
likely to be central components of many future training systems, particularly those used
to train individuals to perform complex tasks. As such, the manipulations utilized for this
study represent the cutting edge of training theory and practice, making the further
understanding of their effects offered by this study all the more crucial to progress in
training design. A

“Practice" and "feedback," however, are exceedingly broad in their scope.
Questions of appropriate practice during training may range from how much practice, to
the extent to which the practice experiences of trainees vary substantially over the course
of training. Feedback is perhaps an even broader term, and can range from the purely
descriptive (how accurate was the trainee, how frequently were certain actions
performed, how consistent was the trainee's performance with regard to the goals of
training) to the highly evaluative (being referenced to others who are going through the
training or have done so in the past, to the trainee him- or herself, or to a general
positive/negative label) (Kozlowski et al., in press). Moreover, feedback can be provided
that combines both descriptive and evaluative components.

This study begins the task of understanding how feedback and practice work
together by isolating elements of each and looking at their effects as well as their
interactions with one another. It makes sense, as we will see, to argue that the
manipulations employed should have an eﬁ‘ect on valued training outcomes. However,
little empirical support exists for such contentions in the training literature itself, and this

study seeks to ﬁll that void.

In the opening section of this paper, I will accomplish several goals. First, to
frame the study, I will review the literature on training, focusing on the kinds of
outcomes that are valued in training settings. With an understanding of our desired end-
states established, I will then turn my attention to the question of what kind of cognitive
processes we should be attempting to stimulate in trainees to reach our goals. I will argue
that a critical element of any successﬁil training program may be the successﬁil
stimulation of trainee self-regulatory activity. Trainees should be active in the learning
process, should be considering how to use the information and skills they are gaining, and
should be attempting to make the information presented in training make sense in light of
the overarching goals not only of the training program, but of the job to which they must
transfer the knowledge and skills from training.

Once my review of the literature on self-regulation is completed, I will discuss the
literature surrounding my two focal manipulations. Both the practice and feedback
literatures are extensive, so I will discuss them ﬁrst at a broad macro level, and then
move to a more speciﬁc consideration of the potential ways to operationalize them. I will
discuss the options available to trainers with respect to the types of feedback and how
practice may be. utilized before discussing my speciﬁc manipulations, along with a
rationale based on training and other literatures for why they should produce the desired
outcomes, with arguments here centering on the stimulation of self-regulatory activity.
Here, as with the section on self-regulation, I will focus not only on the training literature,
but also on what can be gained from examinations of literatures outside the traditional
purview of Industrial/Organizational Psychology -- speciﬁcally, the cognitive and

educational ﬁelds.

Each section of the literature review, then, will build on the previous. In
considering the manipulations, I will argue that the two manipulations I have selected
will enhance self-regulation on the part of trainees, which I will have already argued as
the appropriate method for reaching the kinds of outcomes valued by organizations in
today‘s marketplace. The next section, then, will discuss those training outcomes, and the
research and theory available to guide our understanding of them.

Training Outcomes

The literature on training is expanding with every new journal issue, and a
comprehensive review of the literature on training is becoming prohibitive for anything
less than a series of books. Such a review is not necessary for the purposes of this study,
however, because a great deal of the most critical information is contained in the
literature on training outcomes.

Depending on the goals of the speciﬁc training program, the knowledge and skills
to be gained will vary. It may be that the training was commissioned solely to teach
trainees appropriate behaviors in the workplace (e.g., sensitivity training), in which case
the goals would involve the capability of trainees, following the program, to be able to
identify those behaviors which are appropriate or inappropriate in the workplace. Or the
training may have been commissioned to teach a new skill, like the use of a new piece of
software, in which case the goal would involve trainee capacity to use the program
following completion of the training. In either case, however, it will be hoped that
trainees will retain what they have learned beyond the ﬁnal training session and will
actually apply what they have learned on the job. This is most commonly referred to as

"transfer," and issues of transfer have become increasingly salient to training researchers

in recent years (e.g., Baldwin & Ford, 1988; Ford, 1997). Transfer can occur in terms of
either simple retention (do trainees do the same things they were taught in training on the
job?) or more complex adaptation (do trainees take what they have learned and attempt to
apply it to their job context, which will be non-identical to the context of training and
may complicate the usage of their knowledge/skills?) The more interesting theoretical
and practical question is the one involving the adaptation of knowledge or skills, since in
most cases, the job context and the training context are markedly different. This presents
problems of a practical nature, as trainees must be able to adapt their skills to the
different environment if the training is to have any value. The theoretical problems are
interesting as well, as we ﬁnd ourselves faced repeatedly with the question of how
training can be designed in order to ensure that the skills gained should be adaptable,
from a theoretical standpoint.

Retention-based transfer is the simpler of the two general types. Here, our goal is
simply that trainees retain as much of the information or skills from training as possible,
in a form identical (or nearly so) to what they were taught during the training program. It
is easiest to think about retention-based transfer from both knowledge and skill
perspectives.

The knowledge gained in training is the element most likely to be retained with
minor modiﬁcations. Consider training on a new computer program. An example of the
kind of knowledge that would simply need to be retained ﬁ'om such a training program
might be the meaning assigned to the function keys (F 1-F12) on the keyboard. If the

program's "Help" function is activated by pressing F3, this is something trainees will

need to learn in training, and will need to remember when they begin using the program
on the job.

The speciﬁc skills gained in training on such a program should also be retained.
For example, assume the program in question is an upgrade of a spreadsheet program.
The process for entering data and doing computations is one which should not change
from the training environment to the job, because the program itself is going to remain
constant. Hence, once the skill is gained, it should not be lost. Note that the distinction
between this example and the previous example is intended to be one of declarative
versus procedural knowledge; in the former example, we simply want the trainee to know
that if the F3 key is the "Help" key, while in this example we want to trainee to know
how to do more complicated things with the program, such as data entry. There are
certainly declarative components to data entry tasks, but there are also procedures to be
followed with respect to frequency of saves, backup procedures, and the like which are a
concern while data are being entered as well.

The real difﬁculties come not in getting trainees to retain what they have learned
in training (though statistics do show that only a small percentage of what is learned may
be retained), but in getting trainees to adapt what they have learned in training. Learning
by rote is one thing; developing adaptive skills is another altogether. This problem has
been discussed for over a decade now in I/O journals, with one of the articles that shaped
the ﬁeld being a paper by Baldwin and Ford (1988) in which the authors pointed out that
both the maintenance of skills and the generalization of those skills to the job were

critical elements of transfer.

Recently, the broader psychological literature has begun to pay attention to issues
of adaptation and adaptability. The act of learning, and more broadly, the use of
knowledge and skills gained through the learning process, is not static. Rather, the bulk
of what we learn may be speciﬁc to the context in which it was originally learned,
meaning that in order to use it in other contexts, we must adapt or change it in some way.
The adaptation required may be subtle (the keyboard layout on the computers used in
training may be different ﬁom the keyboard layout of the computers used on the job,
requiring an adaptation to the new layout) or much more complex (different tasks may
need to be performed with the new program once it is implemented on the job, for which
trainees were not speciﬁcally trained). Training researchers (e.g., Hesketh, 1997) have
begun discussing such topics as “adaptive expertise,” but the notion of adaptability has
long been inherent in training research. This is evidenced by Kelly’s (1982) discussion
of generalization across time, settings, and situations.

We are therefore concerned about the retention/maintenance of knowledge and
skills gained in training, as well as their potential adaptation to different contexts. A
reasonable next question is what factors we might expect to have an impact on the
likelihood that transfer, in its various forms, may occur. In general, the literature
discusses three sets of factors that will affect whether or not transfer occurs. These are
individual factors, situational factors, and design factors. Individual factors are those
things trainees bring with them into the training situation. Warr and Bunce (1995) listed
several potentially important individual factors, including demographics such as age,
educational qualiﬁcations, job tenure, and speciﬁc experiences on the job, as well as

more internal factors like attitude, motivation, anxiety, self-efficacy, and the use of

strategies. While not all of the individual factors were ultimately shown to be critical to
obtaining desired outcomes ﬁ'om training, the study nonetheless showed that individual
factors are something we must be cognizant of, and at the very least be able to control for
or predict in our training programs. Tesluk, Farr, Mathieu, and Vance (1995) also
considered individual factors, including organizational commitment and cynicism, but
went beyond the individual and considered situational factors. Tesluk et al. considered
two levels of situational factors, the suborganization, and the organization, and at both
levels considered such situational variables as managers’ attitudes towards the material to
be trained and the presence of an appropriate climate. Multiple climates exist within an
organization, reﬂecting the organization’s norms and values in many different areas (e.g.,
the climate for cooperation deﬁning the norms for employees cooperating with one
another, or the more general interpersonal climate deﬁning the norms for how employees
interact with one another around the office; Mullins, Kozlowski, Schmitt, & Howell, in
preparation). The norms for how work gets done and how knowledge and skills from
training may be implemented into day-to-day activities will clearly inﬂuence the extent to
which retention and adaptation of skills occurs. Hence, climate may be a primary
situational constraint.

Studies involving training transfer are therefore made complicated by both the
characteristics trainees bring with them into the training program, and by the environment
to which they must ultimately return in order to utilize their new knowledge and skills.
Both of these factors make it difficult for us to determine the unique effects our training
interventions — that is, how we design our training - may have had on the likelihood of

transfer. A solution to this problem, at least in the early stages of attempting to

understand transfer, is the use of lab studies in which individual differences and transfer
environments can be controlled. This study attempts to do precisely that. Information on
individual characteristics is collected before, during, and at the conclusion of the
experiment. Moreover, the transfer task is identical for all participants, lending us the
opportunity to observe the capability of individuals to generalize their skills based
primarilyion the training manipulations to which they have been exposed. Ideally, this
means that we will be able to make statements about the efﬁcacy of the speciﬁc training
design elements utilized.

As was noted at the beginning of this paper, the literature on training design offers
many potential avenues by which transfer (in terms of maintenance and
adaptation/generalization) might be encouraged. Rather than begin a detailed discussion
of the manipulation at this point, though, it is reasonable to ask what cognitive processes
might allow our manipulations to actually have an effect on the outcomes of interest, as
discussed in the previous pages. One domain which has begun to be more heavily
researched in this regard is self-regulation.

Self-Regulation

Self-regulation came to prominence with Flavell's (1979) discussion of
metacognition as an individual's awareness of and control over his or her own thought
processes. A more speciﬁc form of metacognitive activity, self-regulation, has received
increased attention in both educational (e.g., Ames & Archer, 1988; Bouffard, Boisvert,
Vezeau, & Larouche, 1995) and I/O (e.g., Kanfer & Ackerman, 1989) psychology. Self-
regulation is the act of monitoring the differences between goals and current states

(Kozlowski, Toney, Weissbein, Mullins, Brown, & Bell, 1998), (and involves cognitive,

behavioral, and affective components (Kozlowski, Toney, Weissbein, Brown, & Mullins,
1997). The centrality of self-regulation to cognitive processes has been forceﬁilly argued
by Bandura (1991; Bandura & Wood, 1989), who posited a system in which individuals
engage in self-monitoring, judgment, and self-reaction while attempting to regulate their
own behavior. While the labels applied to the subcomponents of self-regulation may
differ across theoretical approaches, several aspects of self-regulation are relatively
constant. First, individuals engaging in self-regulatory activities should, based on their
monitoring of their behavior, note discrepancies between their desired performance and
actual performance and act to minimize those discrepancies. Moreover, by actively
thinking about how they are approaching the task and how they can best reach their goals,
individuals should be forced to not only develop a hill representation of the problem
space, but also to develop a ﬂexible representation, which can be adapted as new
information is discovered. Constant self-regulation allows them to better adapt what they
have learned without sacriﬁcing anything they have gained from prior experiences. From
an affective or self-reactive standpoint, it is also clear that how people feel about their
own behavior (often operationalized in terms of self-eﬂicacy, or beliefs regarding their
capabilities to perform the task) will inﬂuence future behavior as well. Hence, while
slightly different perspectives may be taken to understanding what self-regulation is, the
correspondence across these perspectives is high with respect to both the basic
functioning and the theoretical importance of self-regulatory mechanisms.

In this section, I will review recent research on self-regulation that is relevant to
topics within training. First, I will discuss the linkage between self-regulation and goals,

with particular attention paid to a type of goals (mastery/leaming; Ames & Archer, 1988)

10

that is likely to enhance the quality of self-regulation during training. I will then consider
the role of feedback relative to goals, with respect to self-regulatory activities. While
conventional wisdom seems to indicate that feedback in general should be optimized by
simple concurrence with goals (Latham & Locke, 1991), the fact that different
mechanisms may be operating in determining the amount, quality, or type of self-
regulatory activity arising from goals as opposed to feedback may cast doubt on this
assumption. The distinction between goals and feedback is made at this time simply to
demonstrate the explanatory capacity cf self-regulation as a guiding frame, but will be
returned to in later discussions of substantive issues as well. Once the discussion of
feedback is concluded, I will brieﬂy discuss other manifestations of self-regulation, to
provide the reader with an overview of other domains in which the concept has been
applied, and then I will conclude this section by presenting a model of the self-regulatory
system developed by Kozlowski, Weissbein, Brown, Toney, and Mullins (1997).
Self-Regulation and Goals

One understanding of self-regulation, advanced by Ridley, Schutz, Glanz, and
Weinstein (1992) puts central importance on the role of goals. The authors treat goals as
potentially motivating via their actions on the self-regulatory process, speciﬁcally
through their impact on self-evaluation and self-efficacy judgments. Such judgments are
based on the perceived ﬁt between the goals adopted by the individual and the feedback
. individuals receive about how well they are meeting their goals. Mitchell, Hopper,
Daniels, George-Falvy, and James (1994) echoed Ridley et al.'s (1992) ﬁndings, arguing
that when resources were available, goals should bring about increased, beneﬁcial self-

' regulatory activity. These papers, however, discussed goals and goal setting in very

ll

general ways and do not provide a great deal of guidance with respect to what type(s) of
goals might best enhance self-regulatory processes in trainees. That some goals should
do so is not in doubt, but it is not unreasonable to ask what speciﬁc type of goals might
be applied if we wish to maximize the extent to which our trainees engage in self-
regulatory activities.

One recommendation arises from a distinction originally drawn in the educational
literature based on goal orientation. Goal orientation (Elliot & Dweck, 1988; Dweck &
Leggett, 1988) refers to the extent that individuals tend to adopt or prefer either mastery
or performance goals. Mastery-oriented persons tend to adopt goals that focus on
improving skills and increasing competence (Duda & Nicholls, 1992), whereas
performance-oriented persons adopt goals designed to help them perform well and to
look good in comparison to others (Meece, 1994). The cognitive processes surrounding
. the adoption of these two types of goals are likely very distinct. Because of the focus
mastery orientation applies on skills and competencies, individuals with such an
orientation might reasonably be expected to engage in high levels of self-regulatory
activity in pursuit of their goals. Performance orientations, on the other hand, might be
expected to require somewhat less self-regulatory activity, as the capacity to perform on a
task may be something which is optimized when automatization of the skill set has been
achieved. This distinction is borne out in the literature, where research has consistently
shown that mastery-oriented individuals do engage in more active self-monitoring and
other self-regulatory activities than do performance-oriented individuals (Bouffard et al.,
1995; Ames & Archer, 1988; Meece, Blumenfeld, & Hoyle, 1988; Miller, Behrens,

Greene, & Newman, 1993).

12

Goals obviously provide critical information to trainees, which can then be used
to help the trainees make judgments about their progress throughout the training program.
Such judgments are central elements of the self-regulation that trainees engage in during
training. While they may be stimulated by any type of goal, research seems to indicate
that providing mastery goals, which focus trainees on gaining skills and competencies
rather than on simply performing a new set of skills (Kozlowski et al., 1998), may offer
one of the best goal-based manipulations of self-regulation. Hence, if one of our training
goals is high levels of self-regulation among our trainees, we might be well-advised to
utilize mastery goals as a training manipulation.

While substantial evidence exists to demonstrate that mastery (vs. performance)
goals are superior in enhancing self-regulatory activity, I am interested in moving beyond
simple considerations of such goals to determine how we can further amplify their effects
on self-regulation in training. It is possible that other manipulations may possess additive
effects in combination with the presentation of mastery goals. Some of these were
mentioned in the introduction to this paper. For instance, the extent to which trainees
who are learning a task are given variable versions of the task on which they may
practice, as opposed to practicing an identical task every time, might impact self-
regulatory activity by providing trainees with a broader experiential base from which to
draw (Schmidt & Bjork, 1992; Schmidt, 1975). Similarly, it has recently been suggested
that errors be used as training tools; sometimes they are built in to training scenarios to
encourage alterations in the way trainees practice and think about the task, while in other
situations trainees are encouraged to make errors while practicing and observe the

consequences (Lord & Levy, 1994; F rese & Altmann, 1989; Ivancic & Hesketh, 1995).

13

Using errors as practice manipulations can be particularly beneﬁcial, Ivancic and Hesketh
(1995) point out, when problem solving and hypothesis testing skills are desired. In
either case, however, errors provide information which is likely to stimulate self-
regulatory activity on the part of trainees. As a ﬁnal example, consider the sequencing of
materials presented in training (Dansereau et al., 1983). The order in which trainees are
exposed to elements of the material to be learned can be used to guide their relevant
cognitions. This capacity, in turn, can also be used to shape the self-regulatory activities
in which trainees engage by forcing them to develop their understanding of the training
domain in a speciﬁc manner. Any or all of these manipulations have the potential to
augment the self-regulatory effects of mastery goals.

In the beginning of this section, I mentioned that the self-regulatory effects of
goals may arise from a comparator process between the goals provided and the feedback
received. While this may be the case, more theoretical work has been done
understanding the effects of feedback on self-regulation than has been done
understanding the general linkage between goals and self-regulation. The comparator
argument, however, may be insufﬁcient. Assuming that nothing more occurs in
determining self-regulation than a comparison of training goals with feedback is overly
simplistic; both empirical and theoretical work on the role of feedback, particularly
relevant to self-regulation, push us to a broader consideration of feedback within training.
Self-Regulation and Feedback

In considering feedback, we must ﬁrst think about relevant theoretical
frameworks ﬁ'om which the effects of feedback on self-regulation can be considered.

One obvious theoretical framework we can use for this purpose is control theory (Carver

l4

& Scheier, 1982, 1990). Control theory posits a negative feedback loop in which
individuals notice a discrepancy between how they believe they should be performing
and how they actually are performing, leading them to engage self-regulatory
mechanisms to assist in reducing that discrepancy. The negative feedback loop reduces
deviations ﬁ'om a compared value beginning with an input function involving a
perception of the present state. This perception is then compared to a reference value by
a comparator mechanism. If the individual perceives a discrepancy between the present
state and the reference value, an output function is engaged (a behavior is performed) to
reduce the discrepancy. This causes a change in the perception of the present state.
Other external forces (disturbances) may also impact the environment and cause a present
state change. While most oﬁen disturbances increase the discrepancy between the
present state and reference value, they may also act to reduce the discrepancy. Under
control theory, then, the process brieﬂy described in the previous section occurs.
Individuals (trainees) receive information about their performance during training,
compare that information with their initial goal state, and alter their behavior in such a
way as to maximize the likelihood that they will meet their goals. The view of control as
a central element of self-regulation is echoed in the work of other authors (e. g., Nelson,
1996)

'~ . Control theory also aligns itself well with what Karoly (1993) referred to as the
"directive function" of self-regulation. That is, self-regulation helps to direct future
behavior, in this instance by comparing current behavioral states with desired goal states.
However, recent work has called into question the assumption that a simple matching of

goals to feedback may allow control theory to operate. Mullins, Kozlowski, Toney, Bell,

15

Brown, and Weissbein (1999) tested the effects of matching, or mismatching feedback to
goals which were designed around either mastery or performance goal orientations. In
this study, the authors provided trainees with either mastery ("Learn and master these
critical skills”) or performance ("Obtain a score of [x] on the task") goals, then provided
either feedback that matched the goal (mastery feedback for mastery goals, etc),
feedback that did not match the goal (performance feedback for mastery goals), or a
combination of mastery and performance feedback, along with extra information used in
calculating the overall performance feedback information. Because the task was heavily
cognitively-loaded and required substantial self-regulation and strategy use in order to
generalize the skills gained in training to a more difﬁcult environment, the authors
hypothesized that providing mastery goals (consistent with the literature, e. g., Ames &
Archer, 1988) and mastery feedback (was mastery demonstrated?) would optimize
generalization of the material ﬁ'om training. However, the authors found that simply
matching feedback to goals did not maximize generalization/transfer performance.
Instead, they found that the best generalization occurred when individuals were given
mastery goals and both performance and mastery feedback. A further examination of the
feedback literature provides one potential explanation for this ﬁnding.

Karoly's call for self-regulation to serve a directive function is critical to our
explanation of Mullins et al.’s (1999) ﬁndings. Some forms of feedback may be better
suited to direct or guide trainee attentions than are others. Mastery feedback, while it
offers descriptive information about the behaviors in which trainees engaged, may not
ultimately be directive in the sense that it can motivate trainees and encourage them to

continue moving forward. The kind of feedback which is more apt to do so has been

16

referred to in the literature as "velocity" feedback, which is information that allows
trainees to make judgments about their progress on a task (Carver & Scheier, 1990; Hsee
& Abelson, 1991; Kluger & DeNisi, 1996). The capacity to observe improvement over
the course of a training program is likely central to individuals’ beliefs regarding their
potential to deal with the material from training and any changes which might arise in its
generalization, and is also likely to provide more of a directive inﬂuence. Thus, it may
be that because it provided a way for trainees to observe their improvement during
training, the performance feedback served as velocity information and actually increased
the amount of self-regulation in which trainees engaged during the Mullins et al. (1999)
experiment. 1

In total, then, theoretical explanations exist for why feedback should impact self-
regulation. Adopting a control theory perspective (Carver & Scheier, 1990) allows us to
consider discrepancies between goals and feedback as one potential avenue through
which self-regulation may be accomplished. However, recent research has pointed out a
potential ﬂaw in this logic, as in some cases additional feedback will need to be offered
to provide velocity information, which can provide a directive force to trainee cognitions.
Manifestations of Self-Regulation

An impressive array of issues have been considered by researchers concerned
with self-regulation. In this section I will highlight several such issues which are relevant
to the discussions in the previous portions of this review, and which serve to better deﬁne
what is meant by self-regulation before moving on to a theoretical model of the self-
regulatory system that can serve as a guiding framework. Two particular manifestations

of self-regulation are of interest at this point. First, I will consider self-regulation as the

17

basis for at least some forms of higher-order skill development. This is important
because such skill development is one of the goals of most training programs. Second, I
will consider self-regulation as manifested in the use of strategies. Both of these
manifestations, I will argue, are central to any consideration of the role of self-regulation
in training.

Etelapelto (1993) argued that on many tasks, those individuals who perform best
are those who are more aware of their own cognitive processes and engage in higher
levels of self-monitoring. That is, people who think about their own thinking (are
possessed of high metacognitive awareness) are likely to develop more complex
cognitive representations of a domain, and be better able to access the elements of such a
domain. The suggestions made by Etelapelto (1993) are important because they allow us
to expand our consideration of the role of self-regulation during training beyond the
bounds of the formal training program itself, and consider its long-term outcomes. If
self-regulatory activity is, as Etelapelto suggests, related to higher-order skill
development within a given domain, it does not require a large inferential leap to suggest
thatwe should encourage self-regulatory activity during training.

The centrality of strategies to learning has been noted by many authors (e. g.,
Ames & Archer, 1988). One mechanism by which strategies may operate in learning
environments is via self-regulation, as developing and implementing strategies is
inherently a self-regulatory activity. Individuals must monitor their environment, come
to an understanding of the task, and make decisions about how best to approach the
challenges presented by the task with which they are faced. A multitude of studies have

discussed strategy use during training or other learning activities, including Garcia and

18

Pintrich (1994), Meece (1994), Pintrich and DeGroot (1990), Schunk (1994), Bandura
and Wood (1989), Curtis (1992), Barley, Lee, and Hanson (1990), and others. The
contexts in which strategies have been applied makes it clear that they are critical to a
variety of complex learning activities, and the nature of strategies makes the conclusion
that they are manifestations of heightened self-regulatory activity a reasonable one to
reach. conceptualizing strategies as one manifestation of self-regulatory activity again
allows us to better see the role self-regulation plays in learning, this time in terms of the
processes in which learners or trainees are likely to engage.

The relevance of considering both higher-order skill development and strategy use
as indicators or manifestations of self-regulatory activity should be clear. While the
manner in which the two phenomena manifest themselves are different, the argument that
self-regulation underlies both of them is interesting, and should ﬁrrther encourage us to
consider self-regulation as a central element of training design efforts. In the next
section, I will review a model of the self-regulatory system proposed by Kozlowski et al.
(1997). This model serves to break down self-regulation into cognitive, behavioral, and
affective components, and may be used to guide us in identifying indicators for each of
the aspects of self-regulation.

The self-Regulatog System

Kozlowski et al. (1997) developed a model of training with its origins in the work
of Bandura (1991), suggesting that self-regulation is actually a system composed of three
interlocking elements. The most common element of self-regulation is self-monitoring,

the extent to which individuals engage in cognitive analysis of what they have done and

what they need to do to meet their goals. Kozlowski et al. argue that the actual practice

19

in which individuals engage is also reﬂective of self-regulation, as trainees may change
what they are doing to ensure that they meet their goals, and that the self-evaluations
made about performance (in terms of both attributions and self-efﬁcacy) are critical as
well. All three of the elements of what Kozlowski et al. (1997) term the "self-regulation
system" are hypothesized to work together in determining the extent to which critical
outcomes may be reached. See Figure 1 for a graphical display of the three elements of
the system.

The distinction Kozlowski et al. (1997) draw among cognitive, behavioral, and
affective domains is not pure, however; the separation of the elements is in some senses
heuristic. What we do affects what we feel, what we feel affects what we think, what we
think affects what we do, and so forth. The system, the authors propose, is more than the
sum of its parts, but it is by studying the parts — thinking, feeling, and doing — that we can
come to the most meaningful understanding of self-regulation as a whole. Hence, there
are reciprocal links among the three domains such that inﬂuences on any one will likely
affect the others; the distinctions are useful, primarily, for conceptual and measurement
reasons.

The simplest place to begin is with the cognitive component of self-regulation,
self-monitoring. Self-monitoring is the allocation and application of attentional resources
to the task (Kanfer & Ackerman, 1989), monitoring goal discrepancies and the ongoing
process of questioning whether the current strategic approach to the task is optimal
(Kozlowski et al., 1997). The cognitions that are triggered during the learning process

are among the most salient and forceﬁil manifestations of the individual’s experiences,

and may be what stays with the trainee the most strongly because of the effort involved in

20

their production. The requirement that trainees actively consider what they are doing as
they perform the task forces them to learn at a deeper level than would otherwise occur,
and enhances the quality of the conclusions they reach by having preliminary conclusions
questioned as a regular part of the process. Clearly, when trainees are encouraged or
required to reﬂect on what they are doing on the task, and how they can improve their
performance, it is unlikely that they will lose motivation, stagnate, and eventually quit the
task. Instead, they will develop more complex cognitive representations of the task,
which should enable them to better adapt what they have learned to other environments.

The affective, self-evaluative elements of self-regulation may be almost as salient
as cognitions to trainees, and are certainly afforded the same importance in Kozlowski et
al.’s (1997) model. Self-evaluation, to the authors, includes both self-efficacy resulting
from training and attributions made about training performance.

Self-efﬁcacy (Gist & Mitchell, 1992) is the belief on the part of an individual that
he or she is capable of accomplishing a speciﬁc task. The task can be as simple as
walking a mile, or as complex as landing an F-14 on an aircraft carrier. Self-efﬁcacy
arises primarily from past experiences, though it can also arise from vicarious experience
and trainees' knowledge of their skill in a given area, and forms the basis for later trainee
performance. Recent research has begun to suggest that the effects of self-efﬁcacy may
occur in part because of its importance to self-regulation. Kanfer and Ackerrnan (1989)
found that task-relevant self-efficacy was absolutely essential if self-regulatory activities

were to occur, a (ﬁnding supported by Miller et al. (1993) as well as Bandura (1991).
That is, if people do not believe they are able to accomplish the task (they have low self-

eﬁicacy), then they are unlikely to engage any self-regulatory mechanisms and therefore

21

create a self-fulﬁlling prophecy in which low efﬁcacy leads to low performance (see
Lindsley, Brass, & Thomas, 1995, for a discussion of efﬁcacy-perforrnance spirals).

Attributions about past experiences and about the training experience itself serve
a similar ﬁmction to self-efficacy. Trainee expectations, whether of success or failure,
are based in part on the attributions made about past performance. Trainees who attribute
past failures to external causes may be more likely to work hard at a task and actively
self-regulate because they believe themselves capable of succeeding, while those who
attribute past performance to internal causes are less likely to see themselves as capable
and therefore less likely to engage the other elements of the self-regulatory system, self-
monitoring and practice (Martocchio & Dulebohn, 1994). Similarly, within training the
attributions that are made by trainees about their experiences are likely to impact their
processing and retention of information, which will ultimately affect the entire self-
regulation process (Hastie, 1984). Trainees with low self-efﬁcacy, or those who make
poor attributions, are likely to become demotivated and not try as hard at the task.

The third domain of the self-regulation system as discussed by Kozlowski et al.
(1997) is the behavioral component, practice. The literature presents multiple beneﬁts of
practice, including increased retention of information from training to transfer (Hastie,
1984), increased persistence (F orsterling, 1985), and increased positive affect (Carver &
Scheier, 1990).

Multiple methods are available to utilize practice to encourage the activation of

other components of the self-regulation system. First, Frese and Alth (1989) suggest
that trainees be instructed to engage in active hypothesis testing. This activity will

invoke cognitive monitoring elements, as well as eﬁicacy judgments based on the quality

22

of the hypotheses tested, and will force the trainee to practice different elements of the
task as hypotheses are chosen, tried, and discarded. While some studies have not
supported the importance of hypothesis testing activities to adaptive transfer (e.g., Smith,
1996), the issue remains open and the rationale to expect activities such as hypothesis
testing to be important to transfer remains solid. In addition, errors have seen increasing
use as training tools. Sometimes they are built into training scenarios to encourage
alterations in the way trainees practice and think about the task, while in other situations
trainees are encouraged to make errors while practicing and observe the consequences
(Lord & Levy, 1994; Frese & Altmann, 1989; Ivancic & Hesketh, 1995). Using errors as
practice manipulations can be particularly useful, Ivancic and Hesketh (1995) point out,
when problem solving and hypothesis testing skills are desired. It should therefore be
clear that what gets practiced, and how, has an impact on the likelihood that trainees will
be able to solve novel problems when the task is altered.

The three elements of the self-regulation system do not work in isolation from one
another. Appropriate practice can stimulate trainees to think about different elements of
the task and can shape how they feel about their performance. Appropriate self-
monitoring can direct trainees toward appropriate elements of the task to practice, and
can give them accurate information on which to base their self-evaluations. And
appropriate self-evaluations increase the likelihood that trainees will continue to practice
and monitor the task to the ﬁillest extent of their ability and to develop the resilience to
deal with setbacks. In order to adapt, trainees must monitor the appropriate elements of
the task, practice them in a manner that allows them the greatest comprehension, and

make self-evaluations about their progress and performance that enhance later

23

adaptability of what they have learned. What kinds of indicators might we then look at,
to determine the extent to which individuals are activating the various elements of the
self-regulatory system?

In considering behavioral elements of self-regulation, the most direct approach to
assessment involves either observing the behaviors of trainees who are asked to self-
regulate, or to ask them to report the activities in which they engaged that reﬂect self-
regulatory activity. Because many training instances do not allow for substantial one-on-
one contact between trainer and trainee (i.e., they are done in large group, classroom-type
settings), the second option may be better. That is, in determining the type of self-
regulatory behaviors in which trainees engaged, we might be advised to ask them
questions about what they were thinking about while they were learning the material, and
what strategies they employed to help them learn and master the material.

The affective elements of self-regulation are also relatively straightforward. As a
result of their experiences during training, individuals will begin to evaluate their own
capacities with respect to the training material. This is clearly parallel, as was discussed
above, to self-efﬁcacy (Gist & Mitchell, 1992). Hence, when we are concerned about
whether the affective elements of the self-regulatory system have been activated for
trainees, the easiest way to assess this should be-a measurement of their task-relevant
self-efﬁcacy.

The extent to which individuals engage in self-monitoring may be inferred from
the extent to which they have learned the material from training. Controlling for the
effect of cognitive ability on learning declarative information, it is reasonable to expect

that individuals who engage in greater self-monitoring of their own progress learning the

24

material should, in fact, learn the material better than those who do not monitor their
learning. Hence, we may expect that the learning trainees engage in over the course of
training should allow us to draw inferences about the extent to which they were
monitoring their own progress on the cognitive components of the training task.

In this section, I have attempted to accomplish several goals. First, I reviewed
self-regulation in a general sense, deﬁning the construct and talking about its applications
in the literature. Then I turned to a consideration of those elements of the training
environment which might be manipulated to enhance self-regulatory activity among
trainees. Goals, speciﬁcally mastery/learning goals (Ames & Archer, 1988) were
presented as a foundation on which training could be built to encourage self-regulation.
Other manipulations, such as variability of practice, were discussed as possessing the
potential to enhance self-regulation as well. The potential non-redundancy of feedback
and goals was then considered, and the argument was put forth that velocity feedback
might provide an additional "push" to trainees that might enhance self-regulatory activity
beyond simply matching the feedback directly to the mastery goals provided. Moving to
the other end of the model (so to speak), I then considered potential manifestations of
self-regulatory activity, including strategy use and skill development. I concluded this
section with a discussion of the self-regulatory system model proposed by Kozlowski et
al. (1997), including a consideration of how the various elements of that system might
manifest themselves in ways that could be assessed within the bounds of a training
program. In the next section of this paper, I will take up a more detailed discussion of the

Speciﬁc manipulations selected for this study, including their theoretical underpinnings,

25

prior research and theory involving the constructs, and how this research allows us to
make predictions of what may happen in training programs.
Training Manipulations

Thus far, I have brieﬂy reviewed the literature on training transfer, to allow the
reader an idea of what kinds of goals are desired, and I have reviewed the literature on
self-regulation, to provide one explanation as to how we may best be able to reach those
end-states. If the logical progression is followed in this manner, the next question should
be, “How can we stimulate trainee self-regulation?” The best way to accomplish this is
through appropriate design of our training programs.

It was mentioned at the outset that any number of manipulations/design
components are available to trainers. However, Kozlowski et al. (in press) note two
speciﬁc training design elements that should receive increased attention, practice and
feedback. In this section, I will examine each of these literatures in order to develop an
understanding of what we know, as well as what we still need to know about how such
manipulations operate.

Practice

Psychologists have long recognized that what (and how) people practice should
have an inﬂuence on both learning and skill development (e.g., Kerr, 1982a, 1982b).
Recently, Kozlowski et al. (in press) have postulated that. practice is one of the two
elements of the training environment which can most easily be leveraged in order to
obtain desired outcomes. That is, what (and how) trainees practice during training is

almost completely under the control of those individuals who design the training

26

program. As such, speciﬁc alterations of the practice opportunities can have pronounced
effects on what trainees take away from training.

The inﬂuence of practice has been examined in a variety of contexts. Kerr
(1982a, 1982b) examined the effects practice had in relatively basic motor-skill learning
tasks. Catalano and Kleiner(1984) utilized a task in which trainees were asked to light-
based stimuli presented at either set intervals or random intervals, with the ultimate
criterion being the speed with which they were able to respond to a similar stimulus
presented at novel intervals during a generalization trial. Practice has also been examined
for its effects on geometrical problem-solving (Paas & van Merrienboer, 1994), the
ability of trainees to make judgments about digital logic gates (Carlson, Sullivan, &
Schneider, 1989), children (rather than adult) motor skill learning (Cliﬂon, 1985) and
various sport-related skills (Green, Whitehead, & Sugden, 1995; Kozar, Vaughn, Lord, &
Whitﬁeld, 1996). The bulk of the manipulations utilized in the literature cited above
involve the distinction between variable and constant practice, a reasonable place to start.
There are potential beneﬁts to providing either variable or constant practice opportunities
for trainees, depending on the speciﬁc goals.

If the training program is designed to teach people a single task, which is unlikely
to vary substantially (for example, a motor skill‘task such as putting chips into a circuit
board on an assembly line), it makes sense to train people on a single version of the task,
which they repeat ad nauseum until they can perform it to near automaticity. This type of
task is one which involves relatively little self-regulation.

However, modern jobs are increasingly prone to variation. People must adjust to

changes in their environments and be prepared for the unexpected. The individual

27

mechanically putting chips into a circuit board will end up creating no end of problems if
the wrong bin of chips is put down beside him, or if his supervisor accidentally
requisitions the wrong circuit board, or if someone up the line rotates a board the wrong
way while installing a separate chip, or if the production scheme changes markedly for
the line on which he works. To prevent breakdowns in post-training performance, it may
be useful to provide trainees with variable practice opportunities. That is, instead of
having trainees put a single chip in a single circuit board, provide them with a variety of
chips, which require either a separate placement in the same board, or a different board
altogether. This should allow for a more complex cognitive representation of the task to
be established, the speciﬁc mechanisms for which will be discussed shortly.

The utility of variable practice becomes more clear when we move beyond
(relatively) simple motor tasks. In tasks where cognitive effort must be expended (e.g.,
tasks requiring vigilance), we do not want our trainees to fall into a “routine” in which
they are tempted to do the same thing every time they perform their task. Rather, we
want them constantly thinking about the best way to approach their task, about what
elements of the environment need to be monitored, and so forth. In other words,
providing variable practice opportunities should encourage trainee self-regulatory
activity.

This contention ﬁnds support in recent writings by Schmidt and Bjork (1992).
The authors argue that, "variable practice alters the practice context to force a change in
behavior from trial to trial, encouraging additional information processing activities about

the lawﬁrl relationships among the task variants" (p. 214). The increased information

28

processing requirements of a task which is practiced in a variable, rather than constant,
fashion, should be expected to stimulate additional self-regulation on the part of trainees.

The relevance of the constant vs. variable practice argument to applied domains
should be clear based on the examples provided above. One of the main concerns of
trainers, as noted repeatedly, is the transfer of knowledge and skills from training back to
the job. The type of transfer desired should help to deﬁne what type of practice trainees
should be engaging in.

If trainees simply need to retain what they have learned, in an identical form to
what they were presented with in training, then constant practice should suffice.
However, if trainees must be able to adapt their skills, to generalize what they have
learned in training to the job, and beyond, then variable practice should be more
beneﬁcial. Because there is an apparent push in many organizations for increasingly
adaptive workers, variable practice opportunities may be increasingly required, and their
utility will be explored in this study.

An important question must be considered if we are to recommend variable
practice is this: From a theoretical standpoint, why should variable practice advantage
trainees, relative to constant practice? Increased self-regulation is one important element
of the answer, but the argument can be made that such self-regulation is actually an effect
of yet another psychological process, and not the ultimate cause of the observed effects.
Examining the literature on practice points us to another, much more foundational theory,
that involving schemas. A

Johnson and McCabe (1982), drawing onthe work of Schmidt (1975, 1976)

suggested that schema theory might serve to explain observed ﬁndings regarding the

29

efﬁcacy of variable (as opposed to constant) practice in the development of generalizable
skills. Though their discussion centers primarily around motor learning, it is reasonable

to translate their logic into cognitive terms. For example, the authors argue that what gets
stored in memory as a result of practice is not speciﬁc movements, but rather the
abstracted relationships among different types of information, including the initial
conditions, the response speciﬁcations, the sensory consequences, and the actual
outcomes of the movement. It does not require a great deal of effort to translate their
discussion from motor to cognitive terms.

In any work environment, certain conditions are going to exist which require a
response. These initial conditions, in motor terms, are simply the conditions under which
movement occurred, but generalize to the conditions under which any response is made
to a given stimulus, or set of stimuli. The response speciﬁcations in motor terms involve
the organization of the movement generated in response to the initial conditions. From a
cognitive perspective, this is simply the organization of potential mental and physical
(depending on the nature of the task) responses to the stimulus. In both the motor and
cognitive cases, the sensom consequences are the sensory feedback received by the
trainee as a result of the response produced, and the actual consequences consist of
information regarding the result/outcome of whatever action was taken.

The schema an individual has about how to complete a task is a set of
relationships (which can range from relatively basic, to extremely complex) which allows
them to generate response speciﬁcations and expected sensory consequences when
presented with a set of initial conditions. In cases where trainees are provided with

constant practice opportunities, their schemas will be relatively basic. They will know

30

the one appropriate response to one set of initial conditions. However, if they are
provided with variable practice opportunities (which translate roughly to variations on the
same initial condition), then they are going to develop more complex schemas. These
more complex cognitive representations of their environments should allow them to
better deal with changes in their tasks, as well as ambiguities that are introduced into their
work environments. In order to make maximally effective use of these schemas,
however, trainees have to be aware of the different potential response options to various
initial conditions, and be able to select the appropriate way to approach a given task,
based on what they know (i.e., they must self-regulate).

The logic of variable practice, then, is as follows. Providing trainees with
variable practice opportunities should result in the development of more complex
cognitive representations (schemas) of the task domain. Because their representations of
the task domain are more complex, trainees will necessarily engage in greater self-
regulation when performing their tasks. This will enable them to identify contingencies
which might alter how they should perform their jobs, and allow them to better adapt
their skills beyond the training environment, to the job environment. Hence, because of
the complexities of the schemas that should. be encouraged by providing variable practice
opportunities, trainees should be more likely to develop into the adaptive learners that
organizations increasingly require.

In the next section, I will conduct a similar review of feedback. I will begin with
a basic discussion of what feedback is, and the role it plays in training design. I will
discuss different types of feedback, and then discuss one whose effects should be similar

in some respects to those of variable practice. I will discuss how and why such feedback

31

is likely to provide us with more adaptive learners, again with emphasis on potential
increases in self-regulatory activities among trainees, and demonstrate why the
manipulation has been selected for this study.

Feedback

For practice to have value for trainees, it is necessary that its effects be translated
into meaningful terms. Without some indication that their practice is allowing for the
enhancement of knowledge and skills, trainees may become frustrated and reduce the
effort they are willing to expend learning the task. As such, feedback - information given
to trainees about their progress toward whatever goal the training ultimately pursues - is
invaluable as a tool for trainers. It is easily conﬁgured, and can present information that
varies in terms of amount, quality, and level of detail. It can be purely descriptive ("You
did [x]. You did not do [y]."). It can be evaluative ("You did well while attempting to
[2]."). It can also provide guidance for the trainee ("You might want to focus on [q] in
the future, to help you better succeed at the task"). It is one of the most ﬂexible tools
available to trainers in how they design their training programs. It is also the second of
the two training design elements Kozlowski et al. (in press) argue are likely to be central
to future training design initiatives.

It is not unreasonable to argue that feedback is m for learning and skill
development to occur. Practice, in a vacuum, will simply turn into repetition. Without
some idea how they are doing, learners/trainees will not be able to focus their attentional
resources appropriately, nor will they be able to learn from their mistakes (because they
will not know, for the most part, that mistakes have been made). Feedback can be made

available to trainees in a variety of formats, and stimulate a variety of cognitive,

32

affective, or behavioral responses depending on how it is structured. A recent review of
the literature on feedback (Kluger & DeNisi, 1996) makes clear how much work has been
done in this area, and the variety of approaches that have been taken to providing
feedback to trainees.

Kluger and DeNisi (1996) point out that the element of feedback that most people
have come to expect is knowledge of results. That is, how did I do on the task? But such
knowledge, they argue, is only one element of the broader process by which feedback has
its effects. They discuss a more general theory of feedback interventions which, they
would claim, possesses a great deal more explanatory capacity than the studies focusing
on knowledge of results that have dominated the feedback literature since the ﬁrst
published studies nearly a century ago.

According to their feedback intervention theory, feedback is useful in the
regulation of behavior because it allows trainees to compare their current state to
whatever goals or standards they have been given. For our purposes, this involves a
comparison between how trainees are told they are doing in training, and what goals they
have been given to pursue. The notion of a comparator process operating between goals
and feedback, and directing behavior, is one which is not new to the literature. Carver
and Scheier (1982, 1990, 1998) developed their control theory of behavior around a
similar notion. This theory will be discussed in more detail shortly.

Beyond the notion of the comparator process, Kluger and DeNisi’s (1996)
feedback intervention theory has several other interesting components. It assumes that
attention is limited, and that therefore only those discrepancies between feedback and

goals which receive attention will inﬂuence the regulation of behavior. This is a safe

33

assumption to make, and is interesting on several levels. We already understand that
human learners possess limited cognitive resources (Kanfer & Ackerman, 1989).
Therefore, not every element of the environment will be attended to, nor will all of the
information that is available to trainees be utilized. What we need to be able to do is
structure our feedback, relative to the goals of the training program, in such a way that
trainees will attend to the areas where the most important discrepancies are likely to arise,
and thereby guide their self-regulatory activities. As Kluger and DeNisi (1996) note, the
direction of attention is unpredictable if left to trainees, and any feedback intervention
allows trainers to more tightly focus the elements of the environment on which the
trainees are focusing.

Hence, while questions about feedback can center around how much feedback, or
what kind of feedback, it is probably of greater utility to ask questions about how we can
best utilize feedback to encourage trainees to think about the appropriate elements of the
training environment, to perform the critical behaviors that will lead to successful skill
development, and to feel as though they are making progress on the task. This is
particularly important when we want trainees to develop skills that are adaptable to other
environments. Trainees need to be engaging in appropriate cognitive, behavioral, and
affective practices during training in order to develop generalizable skills for the
workplace. Feedback can do this by pointing out to trainees the appropriate elements of
the task on which to focus. One type of feedback which has recently begun to gain
prominence in the literature, and which seems especially suited to the task described

above, is velocity feedback.

34

Velocity feedback is that which allows trainees to track their progress over the
course of training (Kluger & DeNisi, 1996; Hsee & Abelson, 1991). The capacity to
observe improvement over the course of a training program is likely central to
individuals’ beliefs regarding their potential to deal with the material from training and
any changes which might arise in its generalization, and is also likely to provide more of
a directive inﬂuence. Hsee and Abelson (1991) discuss the effects of such feedback in
terms of displacement - that is, the directional difference between the current level of
performance and some reference level. In our terms, that would be the difference
between the feedback received and the goals of the training program. When trainees are
able to observe the change in this discrepancy over time, then we have provided them
with access to information about the velocity (in terms of change over time) of their
improvement on the task.

The mechanism by which velocity feedback operates is thus a familiar one to
those familiar with the feedback literature. Hsee and Abelson (1991) discuss the effects
of velocity information in terms of displacement between the current and reference
levels, which they expect to have an affect on satisfaction. We may derive from this that
' those individuals who perceive favorable velocities - who see favorable distinctions
between their current and reference levels — will likely be more motivated during
training, and will work harder at achieving their goals.

More generally, however, the framework established by Carver and Scheier
(1982, 1992, 1998) with their Control Theory provides substantial explanatory power.
According to Control Theory, behavior is controlled through a process of comparison, in

Which current states are compared to desired states. An example often used to describe

35

control theory is the thermostat, which is provided with a setting that deﬁnes its
comparator, the desired end state. The thermostat then activates the heater (if the sensed
temperature is too low) or the air conditioner (if the sensed temperature is too high) to
bring the ambient temperature to the appropriate level. The temperature is re-checked
periodically, and the “behavior” of the thermostat, in terms of whether it leaves the air
conditioner or heater on, is adjusted accordingly based on the current temperature, until
the current temperature reaches the desired end state. A similar mechanism, the authors
argue, is in place in human learners; given a desired end state (a goal), we modulate our
behavior (learning activities) to attempt to reach that end state, observe our results (in the
form of feedback), and continue observing and modulating until the goal is reached.

Based on this understanding of how goals and feedback may operate, it is clear
that velocity feedback provides one fairly straight-forward application of their ideas.
Given a goal, trainees who are provided with sufﬁcient information to determine whether
they are progressing toward that goal should, over time, approach it more easily.
Moreover, based on Hsee and Abelson’s (1991) work, as well as further speculation by
Carver and Scheier (1998), they should have more positive affective reactions to the
experience, provided they are able to actually observe the change over time in their own
behavior.

It may be noted that based on the literature, we might expect the primary effects
of velocity-type feedback to be affective. However, it is impossible to fully disentangle
trainees’ cognitive, behavioral, and affective reactions to the training environment, since
one will necessarily inﬂuence the other two (Kozlowski et al., 1997). We should also

expect trainees who receive velocity feedback to be more satisﬁed . However, because

36

they will be more actively considering their learning and performance on a task, those
individuals should also show increased cognitive activity (manifesting in such things as
increased leaming and self-regulation) as well as an increased likelihood of appropriate
behaviors to enable better learning. Velocity feedback should thus be beneﬁcial to
trainees not only because of the increased satisfaction that other authors have discussed at
length, but also because of the concomitant increases in self-regulatory (cognitive and
behavioral) activities that should accompany increased satisfaction with their progress
during training.

The implications for training design are fairly straightforward. If one of our
goals, within the conﬁnes of a training program, is to increase trainee self-regulation
around the training topic, then velocity feedback (or feedback with velocity components)
is almost certainly an appropriate direction to go. Not only can it lead to increases in
satisfaction, but it can help guide and shape the cognitions and behaviors trainees engage
in during the training process. Feedback in general has the capacity to direct a great deal
of the trainee’s focus, and velocity feedback provides a ﬁner form of the manipulation,
allowing more precise direction of trainee attention than might otherwise be the case.

The two focal manipulations for this study, then, center around the variability of
practice experiences provided to trainees, and the feedback trainees are given following
each practice session. Each of these manipulations has the potential to increase trainee
self-regulatory activity, which should allow for the development of trainees who are
better able to adapt what they have learned in training to other environments.

A question that may be asked, however, is whether we can actually measure the

potential adaptability that has been discussed throughout the opening sections of this

37

paper. It is relatively easy to infer, from standard end-of—training assessments, whether
trainees have learned what they were supposed to in training, and whether they can use
the skills they have gained. It is more difficult to infer whether they will be able to adapt
their skills when they return to the job. One measurement technology that has recently
gained attention in the selection literature, situational judgment testing, holds the
potential to remedy this deﬁciency. Because situational judgment tests provide
respondents with hypothetical situations of the type they might encounter on the job, their
framework provides an ideal mechanism by which the capacity of trainees to adapt what
they have learned to other contexts may be tested. The speciﬁcs of situational judgment
testing will be discussed in detail in the following section.

Situational Judgment Tests: Measuring Adaptive Knowledge

At the end of training, trainees should have learned something about the training
material. Such learning, at its most basic level, may simply take the form of declarative
knowledge. Do trainees remember facts about what they learned, or how to do things?
However, as we have repeatedly noted, we want our trainees to be adaptive. We are thus
interested in whether trainees possess adaptive knowledge - that knowledge base which
allows trainees to generalize their skills ﬁom one domain to another.

If adaptability is based on knowledge of and experience with the content domain,
situational judgment tests are a promising assessment device. Within a relatively limited
paper-and-pencil framework, SJTs allow for the presentation of detailed information
regarding potential situations, as well as requiring respondents to discriminate among a
number .of potentially valid responses. Scenarios can easily be constructed which require

the use of the knowledge and skill components ﬁ'om training, but which alter key

38

elements of the situation in ways that require respondents to adapt what they have learned
about the task. For example, in this task two defensive perimeters are present. However,
there is no reason why a third perimeter might not be deﬁned, and trainees forced to
make decisions about how to deal with threats approach three, rather than two,
perimeters. This is a step up in complexity from the original task, but is a logical
extension of what has been learned. Similarly, trainees might be presented with scenarios
in which certain key pieces of strategic information (e. g., speed) are absent, and be forced
to make decisions about what information available within the task constitutes the best
substitute for the lost information. The ﬂexibility of the SJT, and its capacity to alter the
presentation of the task in subtle but meaningful ways that require trainees to adapt what
they know to slightly different situations, makes it an ideal mechanism for assessing
adaptive knowledge. The relevance of SJTs should become more clear as we consider in
greater detail how they are typically constructed.

The procedure utilized by Motowidlo, Dunnette, and Carter (1990) in the
development of their low-ﬁdelity simulation is in many ways one with the potential to
generalize beyond their content domain. The ﬁrst step of their development process
involves a review of previous job analyses to determine the core skills necessary for the
performance of the task(s) to be trained. Such prior documentation is useful, but it is not
necessary if time and money are available to conduct a job analysis for the express
purpose of developing an SJT. In a training context, the core skills will have been
identiﬁed in the training goals and built into the training program itself; hence, while a
job analysis may have been conducted at some point prior to the training program to

assist in program development, it is not strictly necessary as the trainer will certainly

39

possess a great deal of knowledge about the goals and intent of training. For our
purposes, consider the "job analysis" to have focused on the kinds of self-regulatory
activities central to Kozlowski et al.’s (1998) ﬁamework. Hence, we are interested in
trainees being able to (1) engage in active thought about the task, (2) use strategies and
comprehend the complexities of the task, and (3) adapt to changes within the rules or
difficulty of the task.

Once the core skill dimensions have been identiﬁed, the next step in the
development process involves developing critical incidents of performance based on the
training task(s). Depending on the expertise level of the trainer, this step may or may not
require outside assistance. Again, if the goals of the training program are clear, then it
should not be unrealistic to expect the trainer to be able to generate critical incidents of
good and poor performance of the skills that are focal to training. The kinds of critical
incidents designed for the SJT utilized here focused on combining the three general
competencies abstracted above, when considered in the context of speciﬁc mastery goals
relevant to the task of interest.

With critical incidents collected, the next step in the development of a situational
judgment test involves writing descriptions of task situations. These situations should
reﬂect the behavioral categories determined by the job analysis. In general terms,
trainers should consider the contexts in which the elements of training may need to be
applied, and write descriptions of situations corresponding to those types of contexts.
Plausible alternative responses should then be considered, and framed as response options

to the situations.

40

After the response options have been selected, the next step in such an effort
involves asking subject matter experts (SMEs) to evaluate the relative effectiveness of the
alternate strategies for each task situation. Any individual with expertise in the subject
matter may serve this role for the trainer. If a speciﬁc task is being used during training,
for example a word processing task, then individuals who have a great deal of experience
with the task should be recruited to assess the quality of the potential responses. The best
and worst alternative strategies must be identiﬁed, and all options are rated as to their
effectiveness. From these data the most and least effective response options are
determined.

The scoring system is dualistic, in the sense that respondents are asked to rate the
response option they would most likely choose as well as that which they would least
likely choose. The option determined to be the "best" via the responses of the SMEs is
assigned a score of +1 if it is the "most likely" response for an individual, and a score of -
1 if it is the "least likely" response. Similarly, the "worst" item is assigned a score of +1
if it is the "least likely" response, and -1 if it is the “most likely. " All other responses are
coded as zero, so the score on any given item ranges from -2 to +2.

The procedure and scoring system outlined above, while described in Motowidlo
et al. (1990) can obviously generalize to a variety of situations. It is not content-driven,
and as such has the potential for broad applicability.

That self-regulation is critical to the development of adaptability should be clear
based on the literature review presented above. Potential behaviors in which trainees
may engage are considered and rated by SMEs, such that appropriate responding to the

SJT indicates that trainees engage in the same basic processes surrounding the self-

41

regulation of their own behaviors that more experienced performers do. Trainees must
also be able to monitor their cognitions appropriately, and determine the important
information that must be abstracted from a scenario and how to use it, and they must
believe themselves to be capable of performing well on the task if they are to perform
well on a simulation of it. The inferences we wish to make about the importance of self-
regulation to appropriate responding on the SJT are reinforced to a large extent by the
role self-regulation must play in the development and assessment of the questions that
make up the test itself.

To this point, I have mentioned a large number of constructs. In the next section I
am going to review the critical constructs for this research, and then move on to a
discussion of my speciﬁc research hypotheses, which should follow naturally from the
preceding review.

Construct Deﬁnitions

Self-Regulatory Skill

Individual differences should exist in the capacity to self-regulate appropriately
(Pintrich, Smith, Garcia, & McKeachie, 1993). How these manifest depends on the
operationalization of self-regulatory activity. In this study, self-regulatory activity is
deﬁned in terms of the extent to which trainees self-monitor their goals, feedback,
learning, performance, and progress on the task. Self-regulatory skill, for the purposes of
this study, is therefore the extent to which trainees generally tend to monitor goals,

feedback, learning, performance, and progress, independent of context.

42

Cogpitive Abilig

The general mental capacity of individuals has consistently been shown to be
predictive in a variety of domains. While cognitive ability is not a focal IV or DV in this
study, it is reasonable to expect it to have an impact on task performance due to the
complex, cognitively-loaded nature of task performance. As such, cognitive ability will
be assessed for use as a covariate in this study.

Masteg Goal Orientation

The literature on educational psychology draws a distinction between two primary
types of goal orientation. Individuals with a mastery goal orientation tend to adopt goals
which focus on improving skills and increasing competence (Duda & Nicholls, 1992).
They also tend to prefer challenging tasks, which require them to learn new information
or skills (Button, Mathieu, & Zajac, 1996).

Performance Goal Oriemtion

Individuals who adopt a performance goal orientation, by contrast, then to focus
on goals designed to help them perform well and to look good in comparison to others
(Meece, 1994). They are more concerned with demonstrating competence, rather than
developing competence. They tend to prefer tasks where they know they will be able to
perform well, and where they will receive praise for their efforts (Button et al., 1996).
Practice Variabilig

The literature supports the contention that providing trainees with variable
practice results in the development of more generalizable skills than having them practice
the same version of a task every time (e.g., Hatano & Inagaki, 1986; Holyoak &

Spellman, 1993). Practice variability will be accomplished in this experiment by altering

43

the position and meaning of targets across 9 training scenarios. While the difficulty will
not be altered as a result, the nature of the practice experience will vary as trainees
attempt to map task features onto the different scenarios.
Velocity Feedback

Velocity feedback is that which allows trainees to track their progress over the
course of training (Kluger & DeNisi, 1996; Hsee & Abelson, 1991). In this study,
velocity feedback will be based on feedback provided to trainees following each training
session which should allow them to gauge their progress in a variety of areas. Trainees
will be provided with score information, where score is computed by adding 100 points
for every target prosecuted correctly and subtracting 100 points for every target
prosecuted incorrectly. 10 additional points will be subtracted for every target crossing a
defensive perimeter during a training trial. Information on the three components of score,
as discussed above, will also be provided. Trainees will also be able to track the number
of targets engaged (total, correct, and incorrect), the number of times they zoomed, and
the number of pop-up targets engaged. This information will allow trainees several
metrics they can use to track progress over the course of training. Moreover, trainees
provided with such information will be provided with a “progress record sheet” on which
they are asked to make estimates of their progress on the task (perceived percent
improvement), further strengthening the velocity manipulation.
Self—Eﬂicag

Self-efﬁcacy (as popularized by Bandura (1991) and others) is a trainee’s task-
speciﬁc belief in his or her capacity to perform. For the purposes of this study, it

includes task-focused self-perceptions centering on trainees' capability to cope and

44

develop methods for dealing effectively with the challenges, information, and decisions
involved in the task (Kozlowski, Gully, Smith, Brown, Mullins, & Williams, 1996).
Self-Regplatory Activity

A review of the recent literature on self-regulation reveals similarities across
approaches to its measurement which allow us to derive a general assessment of self-
regulatory. activity based on what trainees self-monitor. Kanfer and Ackerrnan (1989)
assessed self-regulation in part via trainee monitoring of goals, feedback, performance,
and progress. Smith (1996) developed a similar measure, in which the monitoring of
goals, progress, performance, learning, and feedback were all assessed. Brown, Mullins,
Weissbein, Toney, and Kozlowski (1997) utilized a measure of self-regulation in which
items assessed the extent to which trainees monitored their goals, their progress, and their
strategies. Finally, Ford, Smith, Weissbein, Gully, and Salas (1998) measured self-
regulation in part by assessing trainee monitoring of progress, goals, learning, and
performance.

In all four cases, trainee self-monitoring around goals and progress (in terms of
where mistakes were made, where practice was needed, or both) were considered
important aspects of self-regulatory activity. Actual monitoring of performance was
assessed in three of the scales, while the monitoring of learning and feedback were
assessed in two of the four scales. Based on the literature, then, self-regulatory activity
as a construct will be treated as the amount of self-monitoring trainees do with respect to

their goals, feedback, learning, performance, and progress on the task.

45

Declarative Knowledga

For this study, foundational knowledge about target cue values and the structure
of decisions on the task must precede any other form of knowledge or skilled
performance. Without a thorough understanding of how decisions can be made, and what
the pieces of information that can be gathered to make these decisions mean, it is
impossible for trainees to proceed. The declarative knowledge construct in this study
therefore centers around knowledge of target cue values and knowledge of how decisions
may be made in TEAMS/TANDEM.

Adaptive Knowledge

Trainees may be able to adapt what they have learned during training to other
contexts. The literature suggests that such alterations may be in the objectives trainees
must pursue, or the constraints placed on their existing knowledge base (Ren & Sheridan,
1995). Changing objectives may focus trainees on facets of the task that were previously
not central. Constraints may involve the removal or alteration in meaning of information
available within the task. Trainees will be forced to demonstrate knowledge of how to
adapt to both changing objectives and knowledge constraints.

Basic Performance

Basic performance is the trainee’s capacity to prosecute targets accurately. This
is based on a thorough knowledge of the cue values and an understanding of how
decisions are made in TEAMS/TANDEM. Basic performance is reﬂected by score on
the task, excluding defensive perimeter intrusion penalties. Such penalties are not

considered part of basic performance because they reﬂect attention to strategic

46

components of the task. Basic performance is therefore equal to the number of targets
prosecuted correctly, minus the number prosecuted incorrectly, times 100.
Mm

Strategic performance is reﬂected in the trainee’s attention to elements of the task
which allow him/her to make decisions about which targets are most important to
prosecute. The number of times participants zoomed out, the number of marker targets
hooked, the number of target speed queries at both inner and outer defensive perimeters,
the number of inner and outer defensive perimeter intrusions, and the number of “high
priority” targets engaged all reﬂect the use of strategies related to target prioritization and
situational assessment.

Hypotheses

The hypotheses will be discussed in an order corresponding to their appearance in
the theoretical model (see Figure 2). The model is intended to serve as a guide to the
hypotheses. All of the elements of the model are represented in the hypotheses that
follow. The training manipulations (practice variability and velocity feedback) are ﬁrst
hypothesized to inﬂuence the process variables (self-regulatory activity and self-
efficacy). These process variables are then hypothesized to affect proximal training
outcomes, which in turn affect distal training outcomes. The process variables are also
hypothesized to mediate the relationship between the training manipulations and the
proximal training outcomes, as the proximal training outcomes should mediate the
relationship between the process variables and the distal training outcomes. Each of

these hypotheses will be discussed in turn.

47

Practice Variability - Self-Efﬁcagy. Self-efﬁcacy is the trainee’s assessment of
his or her task-speciﬁc competencies (Bandura & Wood, 1989; Gist & Mitchell, 1992).
Because trainees in variable practice conditions are going to be exposed to a broader
array of tasks, it is reasonable to suggest that they will develop a greater degree of
conﬁdence that they are capable of meeting the challenges of the simulation. The
presentation of challenges in variable training episodes is more likely to give the
appearance of skill-building than will the presentation of an identical training experience
with different goal sets applied over the course of the training program.

The argument might also reasonably be made that the consistency inherent in
practicing the task in exactly the same way every time might increase trainee efficacy
because consistency increases their familiarity and comfort with the task. In training
situations we want our trainees to be adaptable, and capable of generalizing their skills,
so we are more concerned that they develop dynamic self-efficacy (i.e., "I feel conﬁdent
that I can meet any challenges that might arise from changes in this task"). It is therefore
reasonable to hypothesize a positive relationship between practice variability and self-
efﬁcacy.

Hypothesis 1: Exposure to variable practice will be predictive of high self-

efficacy among trainees.

Practice Variability - Self-Regalatory Activig. Recent training literature has
begun to consider self-regulatory activity as an important element of training (e. g.,
Kanfer & Ackerman, 1989; Ford, Smith, Weissbein, Gully, & Salas, 1998; Smith, 1996;
Brown, Mullins, Weissbein, Toney, & Kozlowski, 1997). This literature suggests that

such activity is reﬂected in trainees' monitoring of several elements of training. They

48

must monitor (a) the goals they are given, (b) the feedback they receive, (c) the learning
in which they engage, ((1) their performance, and (e) their progress on the task. These
elements are all present in at least two of the studies cited above.

Schmidt and Bjork (1992) suggest that, "variable practice alters the practice
context to force a change in behavior from trial to trial, encouraging additional
information processing activities about the lawful relationships among the task variants"
(p. 214). It is expected that processing about the relationships among scenarios centers
around the self-monitoring activities listed above. Trainees should be monitoring their
goals, feedback, learning, performance, and progress across scenarios. Individuals who
do are exposed to constant practice scenarios will engage in less of the appropriate self-
monitoring than will variable-practice trainees, as Schmidt and Bjork's (1992) argument
would suggest.

Hypothesis 2: Exposure to variable practice will be predictive of increased

self-regulatory activity among trainees relative to constant practice

scenarios.

Velocity Feedback - Self-Efﬁcm. Velocity feedback, as discussed above, is
information which allows individuals to make judgments about their progress on a task
(Carver & Scheier, 1990; Hsee & Abelson, 1991; Kluger & DeNisi, 1996). The capacity
to observe improvement over the course of a training program is likely central to
individuals’ beliefs regarding their potential to deal with the material from training and
any changes which might arise in its generalization. Hence, we might expect that

velocity feedback would predict self-efﬁcacy.

49

Hypothesis 3: Exposure to velocity feedback will be predictive of high self-

eﬂicacy among trainees.

Velocity Feedback - Self-Regplatog Activity. The feedback individuals receive is
linked to the self-regulatory activities in which they engage (Carver & Scheier, 1982,
1990). Feedback is also one of the elements of the training environment included as a
focal point of selfémonitoring in measurements of selferegulation (e.g., Kanfer &
Ackerman, 1989; Smith, 1996). In some situations, feedback may reﬂect trainee
performance relevant to the goals of training. Self-regulatory activity may be enhanced,
however, by providing feedback which augr_nen1§ the goals of the training program. By
providing velocity feedback in addition to mastery goals and sequenced mastery feedback
we might expect that trainee self-regulatory activity should be focused on the changes
that occur in their behavior and cognitions over time. Speciﬁcally, they should be
focusing on what they are doing differently with respect to meeting their goals, utilizing
their feedback, learning the task, performing on the task, and improving in all of those
domains. This should occur in part because of the presence of information directly
related to performance that velocity feedback provides. Their self-regulatory activity
should thus be inereased beyond those who do not receive velocity feedback, based on
the increased focus provided on understanding how their performance and mastery of the
material has developed over time.

' Hypothesis 4: Exposure to velocity feedback will be predictive of increased
self-regulatory activity among trainees;
Self-Efﬁcacy - Proximal Training Outcomes. The literature contains substantial

evidence of a link between self-efficacy and performance (e.g., Gist, Schwoerer, &

50

Rosen, 1989). Hypotheses 5 through 8 break down performance into two aspects of
cognitive performance and two aspects of behavioral performance.

In terms of cognitive performance, two types of knowledge should be gained by
trainees who are high in self-efﬁcacy. First, trainees who believe themselves to be
capable of performing the task should learn more basic declarative information (the basic
information components and decision structure underlying training performance) than
should individuals who are low in self-efﬁcacy. Individuals who do not believe
themselves to be capable of performing on the task will pay less attention to basic
declarative information, and will therefore not learn it as well.

Hypothesis 5: Self-efficacy for the training task will be predictive of trainee

declarative knowledge.

The second aspect of cognitive performance is represented by trainees' capacity to
demonstrate adaptive knowledge. This is the capacity to take what was learned in
training and apply it to situations in which the task has been altered. The literature
suggests that such alterations may be in the objectives trainees must pursue, or the
constraints placed on their existing knowledge base (Ren & Sheridan, 1995). In either
case, a different context is created which should cause trainees to reconsider how to best
proceed in order to reach their goals. Only trainees who believe themSelves to be capable
of performing the task will be able to adapt their knowledge to questions posed in a
manner which requires adaptation.

Hypothesis 6: Higher self-efficacy for the training task will be predictive of

high trainee adaptive knowledge.

51

Parallel predictions can be made about the behavioral components of task
performance as well. First, basic task performance should be more likely to develop in
individuals who believe themselves to be capable of performing the task. Basic
performance is rooted in trainee information processing. It reﬂects the trainee’s capacity
to gather information and to make appropriate decisions based on that information.
Trainees who possess low self-efﬁcacy are unlikely to engage in the substantial
information processing required for basic task performance.

Hypothesis 7: Higher self-efficacy for the training task will be predictive of

high basic training performance.

Strategic training performance is the collection of behaviors in which trainees
engage which allow us to infer their comprehension and utilization of advanced task
strategies. Such advanced strategies are necessarily complex. A sense of personal
competence arising from high self-efﬁcacy is likely to promote greater exploration of
such strategies relative to individuals with lower efficacy.

Hypothesis 8: Higher self-efficacy for the training task will be predictive of

high strategic training performance.

Self-RegulatoLv Activity - Adaptive Knowledga. The self-regulatory activity
hypothesized to result from the manipulations emphasized monitoring several aspects of
the training scenarios. Trainees should, as a result of the manipulations, monitor their
learning, performance, goals, feedback, and progress. If they monitor these things, we
may expect that they will be better equipped to adapt what they have learned, based on
the development of a deeper understanding of the training task and their skills in relation

to it. Trainee self-regulatory activity should thus predict adaptive knowledge.

52

Hypothesis 9: Greater self-regulatory activity during training will be

predictive of higher trainee adaptive knowledge. .

Self-Regplatogg Activig - Strategic Training Performm. A similar argument
can be made for strategic performance as predicted by self-regulatory activity.
Individuals who engage in the types of self-monitoring described above are more likely
to have seen the necessity for adopting a strategic approach to the task, in order to have a
general method for approaching the task as presented across scenarios. Individuals who
actively monitor their goals, feedback, learning, performance, and improvement on the
training task should come to a deeper understanding of the strategic elements of the task
(these are reﬂected particularly in the goals and feedback), and should therefore have
their strategic training performance enhanced.

Hypothesis 10: Greater self-regulatory activity during training will be

predictive of higher strategic training performance.

 

Declarative Knowledge - Adaptive Knowledga. In order to develop adaptive
knowledge about the training task, trainees must ﬁrst possess basic declarative
information about the task. That is, before trainees can adapt what they know, they must
ﬁrst know. something. It is reasonable to expect that the more they know, the more
advantaged they will be in this regard.

Hypothesis 11: Greater declarative knowledge about the training task

should be predictive of higher adaptive knowledge about the training task.

Deche Knowledge - Basic Generalization Performance. Declarative
knowledge about the task should also be predictive of basic task performance during the

generalization scenario. Recall that basic task performance is rooted in trainee

53

information processing. Declarative knowledge about the task forms the basis for such
information processing activities, and should thus be predictive of basic performance
during the generalization scenario.

Hypothesis 12: High declarative knowledge about the training task should

be predictive of high basic performance during the generalization scenario.
This hypothesis will be examined both in terms of the simple effect of declarative
knowledge on basic generalization performance, and in terms of its effect when
controlling for basic performance at the end of training.

Hypothesis 12a: High declarative knowledge about the training task should

be predictive of high basic performance during the generalization scenario,

. controlling for basic performance at the end of training.

Adaptiva Knowladge - Strategic Generalization Performance. Individuals who
possess adaptive knowledge (the capacity to utilize what they have learned about the
training domain when objectives change or constraints are introduced) will be more likely
to demonstrate high strategic task performance during the generalization scenario. They
have already shown that they are able to adapt what they know in a cognitive sense. This
is one of the primary requirements for performing well during generalization. It should
also be predictive of those behaviors which indicate a strategic approach to the task has
been adopted.

. Hypothesis 13: High adaptive knowledge about the training task should be
predictive of high strategic performance during the generalization scenario.

An interesting question arises when we consider the prediction of strategic

performance during generalization with adaptive knowledge. Is the adaptive knowledge

54

test capable of predicting such performance better than it could be predicted with
strategic training performance alone? lfthe measure of adaptive knowledge is to have
any value, we must assume such to be the case. Utilizing the adaptive knowledge
measure, we should be capable of predicting strategic generalization performance even
when we control for strategic training performance. Hence this hypothesis will be tested
as hypothesis 12 was, considering both the simple effects of adaptive knowledge on
strategic generalization performance, as well as its ability to increment the variance
explained by prior strategic performance.

Hypothesis 13a: High adaptive knowledge about the training task should be

predictive of high strategic performance during the generalization scenario,

controlling for strategic performance at the end of training.

Training Performanca - Generalization Performance. The model suggests that if
we want to predict generalization performance, we should ﬁrst consider with trainee
performance during training. This is consistent with long-standing theory in I/O
psychology (e. g., Wernimont & Campbell, 1968).

Hypothesis 14: Basic performance at the end of training will be predictive of

basic generalization performance.

Hypothesis 15: Strategic performance at the end of training will be

predictive of strategic generalization performance.

The more interesting aspects of these linkages deal with the potential of the two
knowledge measures for predicting generalization performance beyond training

performance (hypotheses 12a and 13a).

55

Mediatipnal Hyppthasas. As a result of the manipulations, two primary processes
are triggered in trainees which allow them to perform well on the various aspects of the
task. First, trainee self-efﬁcacy is increased, due to the trainees' exposure to different
elements of the task and the presence of feedback that allows them to track their progress
over time. Second, trainee self-regulatory activity is enhanced, with respect to the critical
elements of the training task identiﬁed from the literature. These intervening
mechanisms allow us to better understand how training manipulations lead to proximal
training outcomes. Training manipulations such as these succeed because trainees believe
themselves to be capable of performing on the task, and self-regulate an appropriate
amount. Consistent with this logic, I propose that self-efﬁcacy and self-regulatory
activity mediate the relations between the training manipulations and the proximal
outcomes of training.

Hypothesis [6: Self-efﬁcacy and self-regulatory activity will mediate the

relationship between the training manipulations (practice variability and

velocity feedback) and the proximal training outcomes (declarative
knowledge, adaptive knowledge, basic training performance, and strategic
training performance).

Similarly, self-regulatory activity and self-efﬁcacy should be expected to have
effects on generalization performance. Individuals who self-regulate more, and are more
efficacious, should be expected to perform better when the task changes on them. As
above, though, these effects should occur primarily through the mediating mechanisms of

the proximal training outcomes.

56

Hypothesis 17: The proximal training outcomes (declarative knowledge,
adaptive knowledge, basic training performance, and strategic training
performance) will mediate the relationship between the process variables
(self-efficacy and self-regulatory activity) and the distal training outcomes

(basic generalization performance and strategic generalization performance).

1 Method
Mgr.)

Overview. The experiment utilized a 2 (practice; variable vs. constant scenarios)
x 2 (feedback; mastery vs. mastery and velocity) ﬁllly-crossed between subjects design.
All participants received sequenced mastery training goals onto which manipulated
parameters (practice and velocity feedback) were applied. The experiment took place in
a single session lasting approximately three and one-half hours. Participants engaged in
nine practice scenarios spread across three blocks, with each practice scenario lasting
four minutes. Within-subjects factors therefore included blocks (3) and scenarios (9).
Following the training blocks, participants took part in a single four-minute post-training
trial which was identical for all conditions, followed by a generalization scenario in
which the workload was increased and the rules underlying the task were modiﬁed. The
‘ generalization scenario was also identical for all conditions.

Simulation. The PC-based naval radar simulation TEAMS/TANDEM
(Kozlowski, 1996; Weaver, Bowers, Salas, & Cannon-Bowers, 1995; Weaver, Morgan,
Hall, & Compton, 1993) was used as an experimental platform for this research. This
simulation is an ideal platform for training research because of its capacity to examine

learning and skill acquisition in complex, dynamic environments. Participants must

57

gather information ﬁom their radar screens by selecting ("hooking") a target, gathering
information about that target, and making a decision about the disposition of that target.
They must also make strategic decisions about which targets to engage. Some targets
will prove more threatening than others and must therefore be dealt with ﬁrst. These
aspects of the task make it possible to examine two dimensions of performance, one
based on information processing, and the other based on strategic understanding of the
task.

Four decisions must be made about each target, in the appropriate order. First,
trainees must make a decision about whether the target is an Aircraft, Submarine, or
Surface vessel. This is the ”Type" decision, and is made using target speed, target
communication time, and target altitude as the critical cues. Second, trainees must decide
whether a target is civilian or military (the "Class" decision). Here, the critical cues are
direction of origin, maneuvering pattern, and intelligence. Third, trainees must decide
whether the target is peaceful or hostile (the "Intent" decision) using countermeasures,
threat level, and response cues. Once the Type, Class, and Intent decisions have been
made, trainees must make the ﬁnal engagement decision by shooting hostile targets or
clearing peaceful targets.

Skill components. The comprehensive manual available to all participants
throughout training documents three skill sets: basic functionality, declarative
knowledge, and strategic knowledge.

Basic functionality on the task involves developing an understanding of the
features of the program (hardware and software) that make it possible to perform on the

task. These features include hooking targets, accessing cue menus, and "zooming” to

58

alter sensor ranges. PC features such as the appropriate use of a mouse and keyboard also
fall under "basic functionality." All elements of basic functionality must be developed
early in training, to ensure trainees have the opportunity to learn and practice the more
complex task components.

The declarative knowledge component is required if trainees are to process
information and render appropriate decisions. It includes (a) cue information for target
decisions and (b) the structure of the underlying decision model. The cue information is
accessed ﬁom pull-down menus, and three cues are available for each of three component
decisions. Knowledge of the meaning of these cues is essential for later skill
development because the cues form the basis for the appropriate prosecution of targets.
The TEAMS/'1' ANDEM decision model involves understanding those component
decisions and the order in which they must be made. Trainees are instructed to make the
decisions in a certain order: ﬁrst target Type (air, surface, submarine); then target Class
(civilian or military); and last, target Intent (peaceful or hostile). The ﬁnal decision
depends solely on the target's Intent. The ﬁnal decision (shoot or clear the target) can
only be made after all three of the subdecisions are made in the appropriate order. The
structure of the decision sequence is therefore relevant to the order in which the
subdecisions are made.

Five skills are relevant to strategic performance on the task. Participants need to
use the zoom function in order to "zoom out" to assess the situation beyond the radius of
their initial radar screen. This is critical because of the presence of two defensive
perimeters with which trainees must concern themselves. The inner defensive perimeter

is located at 10 nautical miles, and is visible as a cross-hatched region at the center of the

59

initial radar display. The starting radius of the radar display is only 32 nautical miles, so
trainees have no way of knowing how many targets are approaching their outer defensive
perimeter at 256 nautical miles. Because targets crossing either of the two defensive
perimeters (at 10 or 256 NM) result in penalties (a deduction of 10 points/crossing during
the training trials), it is essential trainees learn where these perimeters are. While the
inner perimeter is clearly visible, there is no marking for the outer defensive perimeter.
Stationary "marker" targets are just inside the invisible outer perimeter. Trainees can
hook these targets and then zoom out to 512 NM to determine where on their screen the
outer perimeter is located. In this way, they are better able to identify those targets likely
to cross the outer perimeter. Trainees must also decide which targets constitute the
greatest threats to their defensive perimeters, and which perimeter is more critical to
monitor. Faster targets are of higher priority, since they are more likely to penetrate a
perimeter than slow-moving targets. Trainees generally need to be able to "trade-oft"
targets approaching their inner and outer defenSive perimeters. Here, strategic decisions
revolve around the number of targets near each perimeter, their priority, and their "cost"
in points if they penetrate.

Performance Dimensions. Two types of performance can be assessed using data
collected with TEAMS/TANDEM. These performance dimensions parallel the two types
of knowledge (declarative and strategic) discussed above. The ﬁrst is basic performanaa.
This is reﬂected in trainee score based on targets prosecuted correctly and incorrectly.
This variable is non-identical to the score trainees in velocity feedback conditions receive
as part of their feedback, because it is computed without attention to penalty circle

intrusions. Hence, basic performance is simply the number of targets prosecuted

60

correctly, minus the number of targets prosecuted incorrectly, times 100. In order to
obtain high basic performance, trainees must learn the cue values associated with the
decisions to be made, along with the decision structure for the task.

Strateg'c perfprmancp deals with the behaviors in which trainees engage during
training that indicate strategy use. For the purposes of this research, ﬁve types of
strategic performance are relevant. Each was discussed in the previous section in terms
of strategic knowledge. My discussion here will expand them to demonstrate how
strategic performance can be assessed using information available within
TEAMS/TANDEM.

The number of times participants zoom out is an indicator of their attention to
elements of the task beyond the scope of their initial radar display. Potentially
threatening targets may appear at any point on the radar screen. Trainees must learn to
monitor their entire radar screen if they are to prosecute such targets before they penetrate
a defensive perimeter. The importance of preventing perimeter intrusions is stressed in
the operating manual given to all trainees. A high number of times a trainee zooms out
indicates a strategy focusing on understanding the scenario as a whole, and preventing
targets from crossing either defensive perimeter.

Second, the number of marker targets hooked is indicative of trainee attention to
the outer defensive perimeter. As with zooming out, hooking marker targets (stationary
targets located just inside the outer defensive perimeter) indicates a trainee strategic focus
on assessing the situation. A high number of hooked marker targets allows us to infer
that trainees have adopted a strategy in which they are focusing on stopping targets from

crossing defensive perimeters. The hooking of marker targets allows trainees to make

61

strategic decisions about which targets are approaching the outer defensive perimeter, and
must therefore be dealt with.

The number of target speed queries at both inner and outer defensive perimeters is
also of strategic importance. Speed queries are a critical strategic element of the task.
They reﬂect trainee attention to target prioritization. Targets which move fast are more
likely to penetrate a defensive perimeter, and should therefore be assigned a higher
priority than targets that move slowly. A high number of speed queries is indicative of a '
strategy focusing on identifying and prosecuting those targets most likely to penetrate a
defensive perimeter.

The number of inner and outer defensive perimeter intrusions allowed is an
indicator of strategy effectiveness. Allowing few defensive perimeter intrusions indicates
that trainees have adopted a strategy in which they are focusing on preventing targets
ﬁom crossing their defensive perimeters, and are selecting targets based on the likelihood
that they will do so.

Finally, the number of “high priority” targets the trainee engages is an indicator of
strategy use. Targets are of higher priority if they are (a) moving fast, and (b) close to a
defensive perimeter. .A total of 7 “high priority” targets were present in each scenario,

' with 4 such targets near the inner defensive perimeter and 3 such targets near the outer
defensive perimeter. Identifying and prosecuting high priority targets rather than low
priority targets is an indicator of high-level strategy use and game comprehension on the
part of trainees.

Thus, four indicators are relevant to our assessments of strategic performance.

The number of times participants zoomed out, the number of marker targets hooked, the

62

number of target speed queries at both inner and outer defensive perimeters, and the
number of inner and outer defensive perimeter intrusions all reﬂect the application of
strategies.
Procedure

Participants. Undergraduate students at Michigan State University were recruited
from the psychology subject pool and through visits to classes. 332 individuals took part
in the study; as pre-study power analyses based on Cohen's (1988) power tables provided
an estimate of statistical power between .96 and .99 for 200 participants, it is reasonable
to expect that statistical power was ensured by collecting data from additional
experimental participants. 14 participants were discarded from all analyses for one of
several reasons. Participants were removed from the dataset if they (1) indicated they
had attended an experiment in that lab prior to the current experiment, (2) did not
complete the experiment, or (3) demonstrated motivational or other problems during the
course of the experiment, which were noted in the experimenter’s log and were deemed
severe enough to warrant excluding the participant ﬁ'om the dataset. Of the 317
remaining participants, 79 were in the variable practice/velocity feedback condition, 79
were in the variable practice/no velocity condition, 78 were in the constant
practice/velocity feedback condition, and 82 were in the constant practice/no velocity
condition.

Informed consent. Consistent with UCRIHS guidelines, prior to the beginning of
the experiment all subjects were required to read, sign, and date a consent form providing
a general description of the study, as well as its risks and beneﬁts. Following the study,

participants were debriefed. The consent form is presented in Appendix A, and the

63

debrieﬁng sheet is in Appendix B. Permission was granted by UCRIHS for the conduct
of this experiment.

Experimental trainingldemonstration. Following completion of the initial
questionnaires, trainees were provided with a brief (ﬁve minutes) training session in
which the experimenter demonstrated basic task features. The experimenter
demonstrated use of the mouse for hooking targets and using the pull-down menus, and
explained the basic decision structure of the game. The feedback screens were explained,
as was the manner in which trainees progress through the experiment. They were told
that they would go through three blocks of study, practice, and feedback cycles, followed
by an opportunity to demonstrate how much they learned on a more difﬁcult and
challenging version of the task. .

Experimental protocol. When all participants had arrived (or at ﬁve minutes past
the scheduled start time), introductory materials were read and they were asked to read
and sign a consent form. Once the consent form was completed, the initial scales were
distributed. These scales included demographic, self-regulatory skill, and goal
orientation items. After trainees completed these scales, the Wonderlic Personnel Test
was administered. When all measures were completed, a demonstration of the task was
conducted. Manuals were distributed, and participants were given ﬁve minutes to
familiarize themselves with the manual.

After the ﬁve minutes were up, participants were given the mastery goal sheets
for session 1 and given two minutes to study the manual to determine how to best meet
the goals for the ﬁrst session. After two minutes elapsed, participants were instructed to

begin the ﬁrst scenario. The training scenarios each lasted four minutes. Three training

64

scenarios (i.e., trials) was conducted in each of three training sessions, for a total of nine
scenarios. Each scenario was preceded by a two-minute study period, and followed by a
two-minute period in which participants had the opportunity to review feedback from that
scenario. An experimental diary, in which trainees were asked to report the goals they
sought to achieve during the previous session, was completed prior to the presentation of
feedback. This diary served as a supplemental assessment of self-regulatory activity, and
was collected prior to feedback to ensure that the feedback did not serve to prime diary
entries.

After the end of the ﬁrst block and the end of the third block, trainees were given
questionnaires to answer, including a declarative knowledge test and the situational
judgment test assessing adaptive knowledge. They were also given measures designed to
assess their self-regulatory activities, with respect to the self-monitoring of their goals,
feedback, leaming, performance, and progress. Their self-efﬁcacy was also assessed.
Once these tests were completed following the ﬁrst and third blocks, trainees were given
a short (ﬁve minutes or less) break.

After the break following the third block, participants were told that they had one
ﬁnal opportunity to practice their skills before moving to the generalization trial. All
participants then took part in an identical post-training trial. This trial was included to

allow the end-of—training performance measures to be endogenous relative to the paper-
and-pencil measures collected following the ninth scenario. All study materials were
removed for the purposes ofthis post-training trial. Once the post-training trial was
completed, trainees were given instructions regarding the generalization scenario; they

were told that the ﬁnal scenario is more difficult, that the rules of engagement have

65

changed and that there are many more targets to deal with. They were then given two
minutes to look over a sheet detailing the changes in the task and develop a strategy.

Training scenarios. Each training scenario contained 22 targets. Of these targets,
15 ﬁrst appeared in the region between the inner and outer defensive perimeters, and 7
initially appeared outside the outer defensive perimeter. The scenarios were scripted so
that 4 targets would penetrate the inner defensive perimeter and 3 targets would penetrate
the outer defensive perimeter, if no action were taken against any target. The number of
targets, their placement relative to the perimeters, and the proportion that cross each
defensive perimeter were constant across all scenarios and conditions.

Generalization scenario. Following the nine training scenarios, trainees were
given detailed information regarding the generalization scenario. In the generalization
scenario, the number of targets increased ﬁ'om 22 to 60 (a 172% increase), and the rules
of engagement were modiﬁed. The inner defensive perimeter became worth 175 points,
and the outer defensive perimeter became worth 125 points (per target that crossed the
perimeter). In training, each target that crossed either defensive perimeter resulted in a
deduction of 10 points ﬁ'om the trainee's score. In addition, more "pop-up" targets, which
appear suddenly on the screen, were included, more targets attempted to cross the
defensive perimeters, many of the "pop-ups" appeared very close to the defensive
perimeters, and perimeter intrusions were differentially distributed, with many more
targets approaching the cuter than the inner perimeter. Increasing the difﬁculty and
modifying the rules enables us to determine the extent to which trainees are able to adapt
the skills they gained during training to more complex situations.

Goals. All trainees were provided sequenced mastery goals. A set of goals were

 

66

given to trainees at the beginning of each of the three blocks of scenarios. The ﬁrst goal
set (scenarios 1-3) instructed trainees to learn target cue values, learn how to make
correct engagement decisions, and explore the game and get familiar with their
equipment. The second goal set (scenarios 4-6) instructed trainees to learn how to
regularly ”zoom-out" to see the "big picture,” learn to ﬁnd "markers" to check targets just
outside their outer perimeter, and learn to defend their inner and outer perimeters. The
ﬁnal goal set (scenarios 7-9) instructed trainees to learn prioritization strategies and learn
how to "trade-oft" targets approaching their inner and outer perimeters. Learning advice
on how to best accomplish these goals was also provided to all trainees.

Incentives. Participants were informed of awards offered for two categories of
task proﬁciency at the beginning of the experiment. They were told that the ﬁrst award
would be made for the players who did the best on the ﬁnal, more difﬁcult version of the
task at the end of the practice sessions. They were told that the second award would be
made for the players who were most adept at answering the questions presented
throughout the task. This award was based on a composite score computed for each
participant on the declarative and adaptive knowledge measures by adding together
scores on the ﬁnal administrations of each test; For each award, monetary incentives
were offered for the top four participants, meaning that eight total monetary awards were
given out. Written instructions detailing the incentives were provided at the beginning of
the experiment, and participants were reminded of the incentives prior to the

generalization scenario. The instructions are presented in Appendix C.

67

Manipulations

Practice. The literature suggests that the extent to which individuals are exposed
to variable, as opposed to constant, practice opportunities, will affect the development of
knowledge and skills (Hatano & Inagaki, 1986). Individuals receiving variable practice
scenarios should develop skills that are more capable of generalizing than should
individuals who practice the same scenario every time. One condition (constant practice)
practiced the same scenario every time; this scenario presented targets at the inner and
outer defensive perimeters and provided the opportunity for trainees to practice all of the
critical skills as they learned them.

Participants in the variable practice condition practiced a scenario that looked
different every time. The scenarios were presented in the variable practice condition to
allow trainees to attend to less complicated elements of the task ﬁrst. The apparent
complexity of the task increased in a manner consistent with trainees' experience of the
task domain. The number of targets initially visible changed, as did the location and
meaning of targets on the radar screen. The absolute number of targets did not change,
however, nor did the proportion of targets that began inside and outside of the defensive
perimeters.~ The number of targets that crossed the defensive perimeters did not change,
though these crossings may be more salient in some scenarios than in others, again
reﬂecting changes in trainee focus that will naturally take place across training. The
difﬁculty of the scenarios was therefore constant, though the necessity to engage in
variable practice should enhance trainee self-regulatory activity by allowing a broader

array of experiences. The ﬁnal training scenario was identical for all participants, as was

68

the generalization scenario. This allows statements to be made about relative
performance across conditions at the conclusion of training and during generalization.

Feedback. Feedback was provided using the FASTBACK program developed for
use with this research paradigm. All participants received feedback relevant to the
mastery goals. Half of the participants received additional velocity feedback reﬂecting
their score on the task. Score was computed by the program by adding 100 points for
every target prosecuted correctly, subtracting 100 points for every target prosecuted
incorrectly, and subtracting an additional 10 points (during training) for each target that
crossed the defensive perimeter. Both score and the constituent elements of score were
provided to participants in the velocity feedback conditions, as was other information
which should allow them a sufficient basis to track their progress over the course of
training. Individuals receiving velocity feedback were also asked to complete a
“Progress Record Sheet,” on which they were instructed to make an estimate of how
much progress they’d seen in their game-play on TAG from one trial to the next. This
was included to strengthen the velocity manipulation, and to ensure that trainees were
thinking of the feedback they received in “velocity” terms (i.e., change over time).
Mastery feedback was sequenced to parallel the goals for both the velocity and no-
velocity conditions, to ensure that the only velocity information trainees receive comes
ﬁ'om the score information and its components. A listing of all potential feedback

information is contained in Appendix D.

69

Control Measures

Demoggaphic infgrmatipn. Age, sex, computer experience, and other relevant
demographic information were collected at the beginning of training to ensure the
comparability of conditions and for potential use as covariates.

ngnitive abilig. The Wonderlic Personnel Test was used as an index of
cognitive ability. The test reﬂects multiple domains of cognitive functioning, and the
user's manual for the test (W onderlic, 1992) offers acceptable levels of validity (.63) and
reliability (ranging from .73 to .95 based on the type of reliability estimated). This test
was administered at the beginning of training.

Goal orientation. In order to control for dispositional inﬂuences on the
manipulations, I gathered information on trainee goal orientation using two 8-item scales
developed by Button, Mathieu, and Zajac (1996). Responses to these scales were made
on ﬁve-point Likert scales ranging from "Strongly Agree" (5) to "Strongly Disagree" (1).
Coefﬁcient alpha reliability estimates for this and all other scales are included in the
correlation matrix in Table 2. These scales, and all others, are included in Appendix E.

Goal commitment. Seven-item measures of goal commitment (Hollenbeck,
Williams, & Klein, 1989) were collected at the conclusion of each pre-training goal
manipulation. Responses to this scale were made using a 5-point response format
identical to the format utilized for the Button et al. (1996) goal orientation measure
described above. This scale is included in Appendix E.

Self-regplatogy skill. A set of items were written to parallel the self-regulatory
activity items at a more general level. Eleven items were written based on their relevance

to self-regulatory skills important to learning. A sample item is, "When I'm practicing a

70

new skill, 1 monitor how well I'm learning its requirements." Responses to this scale
were made using a 5-point response format ranging from "Not At All True of Me" to
"Very True of Me." This scale is included in Appendix E.

Prpcess and Outcome Measures

Salfﬁﬁgcy Self-efﬁcacy was assessed following scenarios three, six, and nine,
utilizing an 8-item self-report measure developed speciﬁcally for this research paradigm
(see Kozlowski et al., 1996). This measure uses a Likert-type scale rather than
conﬁdence ratings, with self-efficacy operationalized in terms of task-focused self-
perceptions with item content centering on trainees' capability to cope and develop
methods for dealing effectively with the ,challenges, information, and decisions involved
in the task. Previous research has estimated the internal consistency reliability of the
scale to be between .84 and .95 (Mullins, Brown, Toney, Weissbein, & Kozlowski, 1998;
Kozlowski et al., 1996). This scale is included in Appendix E, and was collected at the
end of each scenario block (following scenarios 3, 6, and 9).

Basic task knowledge. Basic declarative knowledge regarding the task was also
assessed following scenario 3 and scenario 9. Nineteen multiple-choice items focusing
on the extent to which participants had learned basic declarative information necessary to
performance on the task were utilized, focusing on target cue values and the structure of
the decision sequence required by the task. This measure is presented in Appendix E.

Adaptive task knowledge. A measure of adaptive knowledge was developed for
use in this study, utilizing situational judgment questions. These questions presented
respondents with game-play scenarios and asked for the identiﬁcation of the most and

least effective potential responses. Questions were developed around the training goals

71

that focused on the types of strategic skills necessary for adaptive performance to
develop. This measure was structured in such a way as to require trainees to move
beyond their direct experience with the task. It forced them to adapt their knowledge and
skills with TEAMS/1‘ ANDEM to new situations or task features. A pilot test of this
measure prior to data collection for the full experiment ensured the alarm keying of the
items corresponded to independent expert judgments of the correct adaptive responses to
the scenarios. Four experts were provided copies of the measure, and asked for their
responses to the items. Any item on which at least three of the four experts did not agree
with the author’s a priori coding of the item was rewritten to clarify the item intent. Nine
items were used. These items are presented in Appendix E.

Self-regalatopy activity. A review of the recent literature on self-regulation
reveals similarities across approaches to its measurement which allow us to derive a
general assessment of self-regulatory activity based on what trainees self-monitor. Kanfer
and Ackerrnan (1989) assessed self-regulation in part via trainee monitoring of goals,
feedback, performance, and progress. Smith (1996) developed a similar measure, in
which the monitoring of goals, progress, performance, learning, and feedback were all
assessed. Brown et al. (1997) utilized a measure of self-regulation in which items
assessed the extent to which trainees monitored their goals, their progress, and their
strategies. Finally, Ford et al. (1998) measured self-regulation in part by assessing trainee
monitoring of progress, goals, learning, and performance.

In all four cases, trainee self-monitoring around goals and progress (in terms of
where mistakes were made, where practice was needed, or both) were considered

important aspects of self-regulatory activity. Actual monitoring of performance was

72

assessed in three of the scales, while the monitoring of learning and feedback were
assessed in two of the four scales. Consonant with the literature, then, a scale was written
which tapped these ﬁve domains within self-regulatory activity. Because the scale is
constructed in such a manner as to reﬂect what the literature treats as indicative of self-
regulatory activity, higher scores on the scale may be taken to mean that trainees are self-
regulating more. This scale is included in Appendix E.

A supplemental assessment of the relevance of trainee self-regulation was
collected following each training scenario. Trainees were asked to record, in an
“experimental diary,” the goal or goals they focused on during the previous session. It is
reasonable to expect that trainees who reported focusing on the goals they had been
assigned for a speciﬁc scenario block, rather than other goals, are engaging in self-
regulatory activities that are more relevant to the task. Any reported goal corresponding
to one of the mastery goals traineeswere assigned for that session was scored +2.
Reported goals corresponding to a mastery goal from any other scenario will be scored
+1. All other responses will be scored 0. A second coder was trained to ensure reliable
coding of responses. This supplemental assessment Was purely exploratory, and was
included to determine if this represented a valid approach to identifying the relevance of
trainee self-regulatory activity. As such, it was not included as an explicit part of the

experimental model or hypotheses.

Post-Experimental Screening Measures

Two scales were included to allow the identiﬁcation of trainees whose data might
be of questionable utility. Trainee motivation is critical in this regard. Trainees who

report being demotivated at the end of the experiment may have provided questionable

73

data. The same may be said of trainee satisfaction with the feedback provided. The data
of participants whose motivation or feedback scores fell more than three standard
deviations below the mean were discarded from all analyses.

Motivation. Trainee motivation for the experiment was assessed using four items
administered following the generalization scenario. A sample item is, "I tried to make
good choices in determining the best and worst options’on the sample scenarios I was
presented." This scale is included in Appendix E.

Feedback satisfaction. Trainee satisfaction with the feedback provided was
assessed using ten items administered following the generalization scenario. This scale is
included in Appendix E.

Analyses

The analysis plan for this study is detailed in Table 1. For each hypothesis, the

speciﬁc analysis relevant to understanding the relevant effect is detailed. All analyses

centered around the use of regression or MANCOVA-based analyses.

74

$25.80 035

 

 

8.86.35 ESE—cm? 2a 53 .5238
2.58588 23.: macaw—:an—e .963 m< ”33:3. beﬁBmomﬁom 30338 .0623 o: a o. 9522
wEEE. 53 55 33:8 .93.“? boa—smom .056“ boa—ameﬁom 8:25
eossmaéa masses eeﬁswom .98.. 2 28.88 £8; $8 6388 e_8_o> =5 838.... £8.05 4:

.263 2 “USEméom
.eoUEBQ ESE—awn .5228 moan—coco
2a 53 288583 2.5.7. 63236 8: .8 03256 .220 .9623 o: a 2 9622 .6859
Sousa—ES: 9.3:»: 53 £5 . n: sous—Eng x0330.“ 8303 38.3 ..=8 Co Eon—3.26“. 65 8:25
reeves masses Enema. . Basses: uses—.8". £8; 385928 88.8.. 38:5 :9. 838a e_8_o> a:

602.63 38.68.. 05 23 .9558.

:2: .8enzcoton :2: 18.8 2: co

2.qung :2: .28» :05 3.9an

£2662“. Enact—ma 2a 53 80:3,: :02», 2 68.8 2: .5 9:38.
888.88% 2.55 macaw—=35:— o_w8 82.6 53:64. boa—smoméom 60:qu Soﬁa—.8 EB :9:
mega: 53 .23 .558 32.3. boa—swam .326.“ be.a1&8..:8 538m
5.23253 9.385 Serums. 26% 2 §=er> 888.. $8 .b___os_a> 882.2 2 as. as. 8:85 ossa> a:

.33— .3553

a. 8:3: 9.2m .528 5.5

ﬁgs—~36 woman—.20 9:35:88

2.. 5.... as. as 88 e. E388

8.85.5 2: he 3383928 @838

ass .28. see-» ”E88928

586:8.“ 2:85:38
98 53 28.9.8.3 2:93 .835 888 member .8235 508350
macaw—3&5:— wEEE. 53 55 mots—o8 he .25. :68 8.5258 €2.52 . 2 9522 .685058 com—88$
snags. nausea senoaom 1 8.2352 55829 8:8... 588:8 .b__seE> 888.. 9 so. as. 8.82.. osecS a:
ﬁagaeﬁeo 889m 8.382 8328 .889:

 

:2.— 3392 98 .8882 £85200 683m 3 03a...

75

«082:8 030.

 

80.0200...

885:2... 0.0 2.3 0.8.8080:
2.520. 22.500 2.30.5228 0:0
$0850.28 53 :23 8:08.020:
0.00. 080.88 $582.05 :o_80.$0m

$558. 2. 0:0 02. .0 00:08.50:
0.08 0.002 80200.: 22:85:20.
2085028 20.23 5 3.8053—

..s... was... .8... 2..

$5.50 00:00:00 0:20.55 8.08:2—
0>.m:0.«0o 8.50:0 0:0 £02000 000%
008: .3080. .0208: 00200: 02:00..
.50 8000.. 0... :8: 8058. 00:5 .20
.0280: 02... ”00:08.02?— 2002. 030.86

0.6.... 9.. “88.00.28

20.5 $550.:

.8... o... 8.5.. 8.8.30 28:8...
00$0$:0 .080. 2.0.6 00.. 0.58

2: $:2085:0 0:0 5.02.50 8$0$:0
.030. .0>0 .8 0.08 0058. o. .250:
cc. $5200 .3 00.00800 .00 £020.25...
02. :0 0.80m ”00:08.50“. 0.00:. 0.00m

0.620 04 $085523

3:582 00:08:20..
0:8 2028.... 88800.8

85:20.2 00:05:50.2
8.8 as: 88.00.28

.20....

$558. .05 05 E 8:08.020:
.8. ease: 8.8... :5 05:8.
.20 0:0 02. .0 $0850.23 ”w:

a... 055.5 .8...

o... e. 2...: 050.05 8085.8
80. 0:8 8.8... =3. 0......8.

20 0:0 05 .0 $0850.23 K:

 

8302.05 _ 85080.0

885:9... 20 :0 2020:0800 02.0.305— 2m0. 080.88 0:0 8.20000...

2:020 53200 890.50. 2:. :0. e. 00.0.5800 .00. 25.80538.
.28 0:0 0300.305. 02.80.00: 3:02.035 ”ow—00.305. 0.2303. $558. .20 0:0 05 .0 030.305.
$0850.28 :23 030.305. 03.0365. 025000 002.02. :3» $550.:
0.23000 $500200... 5.8050... 0.620 3. ”>085m20m 0.2.003. $085523 20 0:0 02. .0 20850.28 6:

02:08.... 8.200:

0:0 00.20., 0:0 2.29.2.2. 2. 0300—305—

0058. 880.0 0. 00.0.5800 02800:.
$550.: .20 0:0 02. .0 0200.305. 802-E ”0300.305. 0580.000 $558. 2. 0:0 05 .0 0300.305.
0>280_000 0.02.0... 22:85:20 03.0.305— 0>280_00: .0200..— ...>» $558.
2085028 20.23 :_ 8.8030... 0.600 2 $085523 0.680.02— .b85m.20m 20 0:0 05 .0 50050.23 .2.

088.00 00.0095 00.8002 808.800 0.850%:

 

82.38 .. oz...

76

0005500 0.20.

 

0.0.0.02: .:8....:$.0

0.0 2.02 20.8553 0.0020 $550..
$5.00 8:05.050: 200. 0.002

0:0 0$00.>.05. 05.0.0.000 2.02 2......
:0..0~..0.0:0$ $5.00 8:05.050:
200. 0.002 $5.502: :0.002$0.~.

0.0.0.02:

5855.0 0.0 ..0 0500:0500
00.020 5.5.00 50.5.52

.28 0:0 .0$00.>.05. 05.0.0.000
$0850.28 2...; 0$00.>65.
05.:000 $58.00.: :0.002$0m

0.0.0.00...

.:85:$.0 20 2.02 0.0..0:0500
0.0020 5550.. ..0 0:0 02. .0 =0.
8:05.050: 200. 0.002 0:0 5.5.00
50.5.5228 2.02 22>. 8:05.050:
200. 0.$0.0..0 $58.00.: :0.002$0..

0.0.0.02:

50055.0 20 :0 0.0..0:0500
0.0020 5.5.00 50.5.52

.28 0:0 .0$00....05. 05.0.0.000
$0850.28 2...: 0$00.>.05.
05.:000 $56.00.: :0.000.$0.:

.0... :0..0N..0.0:0$ 02. $5.00 00.00200

.02 .0>020 0< ”8:05.050: 200... 0.00m

.2620 0... ”500.505. 05.0.0.00D

.0>020 0< ”500.505. 05.:00...

.0>020 0.. “500.505. 05.0.5000

0.0020 0.... ”8:05.050: 200... 050.53

.08.... 0.. 00.50... 50.00.000.200

..0>020 0... ”500.065. 05.500...

0.620 0.... ”2.2.0.... 50.00.000.200

55.05.5059
8:05.050: 200...
0.00m .0$00....0:v. 05.0.5000

500.505.
05.:00< .0$00.>.0:v. 05.0.5009

.0550...
8:05.050: 200... 050.23
.05.... 020.53....00

0$00.>.0:v. 05.:00<
.250... 220303....8

.:0..0~..0.0:0$

$550 8:05.050:

200. 0.002 .0 05.0.02:

3 .5. 0.5.0.. .o 0.... o...

.0 0$00....05. 05.0.0.00n. ”N...

$550.. .0 0:0 02. .0
0$00....05. 05.:000 .0 05.0.02:
2. .5. 05...... .o 0.... o...

.0 0$00.>.05. 05.0.5005 .. .I

.0...

05...... .5... 2.. ... 2.0.. 0......0...
00:05.050: 200. 0.$0.0..0

.28... .5. 0.5.0.. .0 0.... 2..

.0 5.5.00 50.5.5223 .0...

05......

..0 0:0 0... .0 50500 $555.

02. ..0 0$00.>.05. 05.:000

0...»... .5. 0:25.... 0.0

02. .0 5.5.00 50.5.5223 .0:

 

0500.00 00.00:..m.

02:000.).

0.05.0000

5002.09...

03......8 .. 0......

7‘7

 

.2333355 05 Ban

33.3 93 53:8 boa—$2.38
23 3850.28 :2? 585335:
on :5 88850 05 :o 3330..
goo—u.) v5 3:325» 838..

.8 38,3 05 523 5 £593
532%: 032::— 339505

#8556 2a =a 383583
2:97. cog—Eaton 533—805»
2mg 98 .8558: 85::otoa
is ammoabm 630365. 9533
5.5 Bag—Eton 333—803»
03286 9523:— 5323:

.2206th E836

2a 53 286553 2.55 9.23.:
9:26 Baa—Eaton is 063

can cave—>65— o>2§23v 53 .33
couﬁzﬂocom mat—6 8538.8

.33“ a .2

.couaﬁ—anow
Ea 9.38. 53 96% 38:8
.969“ m< ”conga—om 3h awe—Em

.couaﬁﬁbaom
2a ”558. 53 9E8 38:8

£3. 9:58:

Bagotum v3... 335$
.53 9:53: Bangotom
xmﬂ. 03m 632305.
9533‘ date—>65—
o>=8£oon 5333‘ boa—awe:
.aom 683.2% £0380...
£8; .bEnaES 8:05

Accumﬁ—Eoﬁmv cog—Eaton
v.5 oasgm .9552;
connccotom 3H omwoﬁbm

33328033 coca—Eaton—
xmﬂ 28m .3558:

.5395 9:58. on. .8 358.8

3&on 05 v5 Egg—=25:—
MEEE. 2: 5953 @2223?
05 833:. F? =23.sz

$3 as Eaavaom é:

douﬁzﬁoaom

mats 85::th

x8. uﬁoﬁbm .3 9633.5

3 :5 was»: he as 2: 5
8538.8 was uﬁoﬁbm ”2:

dogma—82.0w

mat—6 nag—Eaton

is 223 .6 256%

3 =5 9.35. he as 2..

 

xms £93 95335 canny—mom .969“ m< ”negate.“— 3h 03m coca—Stun— xma... 03m 8 cogs—Brun— me 23m ”3:
435:3... 2a 53 8.52083 .35
2:2: 9:58. $53. 00:55th 523303» 05 waist 33:8 :5 .533:anan 9.96
:3. 03326 28 onto—>55— u>=n~3 _ .269“ m< ”oogcuotom v3... awe—Sm 333:8“...qu 8538.3 “3. name-Em mo
53 55 Baa—Eaton 558.33% cogs—Bron— xmah 9623.:— 3 E? 9:58“ .3 98
£3.83 $533.5 532w“:— .u>onn m< ”$3.320— o>un€< omwoﬁbm .3338! gun—«3‘ 2: as ow3_>65_ 3334‘ ”2:
2:850 339m— 35332 925980 «352$:

 

839.8 ._ 2%...

78

Results

The model (Figure 2) speciﬁed that two manipulations (the presentation of
variable vs. constant practice scenarios, and the provision of velocity feedback in addition
to mastery feedback) would inﬂuence trainee self-regulatory activity and self-efﬁcacy,
both of which were treated as indicators of psychological processes for the purposes of
this study. These process indicators were then hypothesized to inﬂuence proximal
training outcomes (knowledge' and skills at the end of training) which, in turn, were
hypothesized to inﬂuence more distal training outcomes (skill demonstration during a
generalization scenario).

Means, standard deviations, intercorrelations, and internal consistency reliabilities
for all variables are presented in Table 2. Examination of the intercorrelations and prior
work suggested that several variables be treated as covariates in all analyses. Both
mastery and performance goal orientations have been shown in previous studies to be
related to learning and performance using the TEAMS/'1' ANDEM task, and will be used
as covariates in this study as well. Cognitive ability has also been used as a covariate in
previous studies, because of the complex nature of the task and the high cognitive
requirements for learning the elements of the task. Examining the intercorrelation matrix
suggested that participant gender might need to be controlled for as well. One-way
ANOVAs indicated that signiﬁcant gender effects were present during the ﬁrst six
training trials for basic performance, as well as during the generalization trial (F1315 =
8.266, p < .01). Moreover, every training trial demonstrated signiﬁcant gender effects on
strategic performance, with a signiﬁcant effect during generalization as well (F 1,31 5 =

25.86, p < .01). These effects uniformly favored males over females. As such, gender

79

Gus—5:8 038v

 

 

:3 8. 8. 2. z.- 8. 8.- 8. 8.- 8.- 8. 8.- 8. 8.- 8. 8:. 6038382.:
T-v :.- 8.- 2... 2. 8. 2. 2. 2. 2. 2. 8. 8. -- -- 8328055883.:
T-v 8.- 2.. 8. 8. E. 8. 8. R. R. 2. 2. -- -. <8 .2
T-V 8.- 2.. 2.. 2.. 2.- 8.- 8.- 8.- 8. 8.- -- .- om< .:
TV 2.. 8.- 8.. 8.- 2.. 2.. 8.. 8.- 8.- ..- -- .850 .2
TV 2. 8. m. 2. 2.. 8. 8. 8. c8 :2 8:853“— 28m 885 .8
T-v 3.. 8. 8. 8. E. S. 2. 8n 2% use 8.3 885. as". a
T-V 8. 8. 8. E. 2.. 8. ~8_ «.87 28m 5.85 s
T-V 8. R. 8. 2. 2. 8.. 38 28m was“; .8: .8
TV 8. 2.. 8. sm. F... 8.8 33¢. 0828.0 .n
33 a. 8. 8. :8 2.: 8 a»... 83:65. .8
:3 2m. 8. 2 .n 8.2 E 43. uwBESM .m
:3 2.. .3. 8.2. 2. 455823.885 .N
G3 2... 8.8 :- éoameaaaouggm ._
z 2 2 : 2 a a h e n a. m N _ am :82

 

magnum E338“ 356650 .9285 EB gown—“E828:— .m=o=m_>oo 33m .2832

N 035.

80

Amos—.230 038v

 

 

ac. :. ac. wcr ch aN. Mc. QM. cc. cM. MN. cM. «N . a~ . w. ._ Mc._ POM ﬂow-3H beta—2 .3
NM. wc. ac. vc. Ncr MN. M ~ . cN. M ~ . ac. cM. n~ . cN. 2 . cc. aNé 623232 KN
cc. vc. Nc. 2.- mcr a~ . wc. aN. 2. Mc. M N . Mn. n N . 2. and cN.MN cocoa-«3am x2330...— .cN
aN. :- _c.- Ncr “N... VM. :- cM. cN. Mc. MN. cN. ac. cc. cwc Mc.wN a. .muauﬁmﬁom .MN
3.. aN. cc. Mcr cN.. N M . cN. cN. RN. ac. cN. QN. 5.. cc. amc waHN NH $0359.23 .3
VN. cM. :- wcr MM.. MM. :- cM. MN. NM. MN. NV. M~. ~N. cad vwgm :- ﬁuaoEmcﬁm MN
cM. c_. 8. ac. cc.- MN. ac. ‘N. c_. _c.- a~. wc. ac. c_. _a.n 3.3 C- Jaoggou :30 .NN
cM. ac. _c.- c. 3.. MN. cc. NN. ac. _c. «- c_. ac. c_. and MYMN NH Jae—5:256 300 Am
aM. cc. 3.. nc. ncr VN. N—. MN. 2. cc. VN. cc. wc. :. N~.m hmdm :- Jaoggou ~30 .cN
cw. ‘4. ac. Mc. 3.. 2. Mc- VN. cc. :. MN. "—- c~. 2. Ncé Nm.m~ a. docaapmumcﬁm .a—
«M. :- wc. Ncr wcr cN. a~. RN. cN. ac. VN. M N .. cc. 3. and a_.m~ NH £23337:an .w-
NM. MN. cc. mcr V5. 3. ac. «- hc. 3. Nu. «N. Nu. MN. acd 2.3 :. douaimomnzom .2
Mc. cc. ac. 2. 5.- ac. cc. ac. cc. 3. Nc. 3.. cc. Nc. M_.m awn-N =65 boa—awomﬁom .c-
M N .. vc. cc.. 8.. vc. mc. vc. Nc. cc. cc.. M N .. 2.. mcr 3.. MM. SM .50 :30 8553....0A .2
I 2 Na 2 c— a w a c n v m N _ Om :82

 

.38 .N 2%...

81

A8358 83.3

 

 

8. c. .- 2. S. 8. 2. :. 2. 2. a. R. 2. z: 2. .2 8.: 35 35559.... .3.
8.. 2. 2. 8.- 8.. 2. 8.- n... 8.- 3.. a: 8. t: ..N. ..- -.. 8... .5. .8288 .2.
8. 8. 2. 2.- 8.. 8. 2. s. z. 2.. 2.. 8. =.. a: ..- ..- hom. 5.. 8.2...8 .2.
8.- 8. o. .- 8.- 8. a. 8.. 2.. 8.. 2.. 2... e... a... a... 8. 8.8.. 8... ..8. 8...... 828.. .:.
8. 8. 8. 8. 8. 2.- 8.. 8. 8.. 8. 8. 2. a. 2. a..." 8.8 8.... .855 8.5 .88... .9.
8. 8. 2. 8.- 8.. R. 8.- a: 8.. 2.. =. 2.. 8... 8. 8.... 8... 8F .855 .8on .25 .8...
S. 2. 8. 8.. R.- 2. 8.- a: 8.. R. 8. 8. 8. a. 3.. 8.2 2..... ..5 8.58... .8
8. 2. 2. 8.. 8.. 3.. 8. a... 8. a: 2.. =. R. 2.. .N. 8.. 8F .98.... 8.5.2 .3
8. 8. 8.. 8. S. 2.. a... :.- 8m.- 8. 2. 8. 2. 2. 88 a... may .855 88m 85.. .8
8. 2. 2. 2.- 2.. 2.. 2. a. 8. 3.. 2.. a... =.. 8. a... .3 8... .385 85.... gm... .8
8. 2. 2. 8.. 2.. u .. 8. 8... 8. 8. an. N... 8. a. 88 8... 5m ..5 «5:53 .3
8. a. 2. 2.. «N. a... 8. N... a. R. N... =.. 2.. 8. 8.. 2... so... .98.... 85.... a»... .2
8. 2. 2. 8. 2.. k .. 8.- N... 8.. 2.. Q. 2.. R. a: s... on. .5... .855 88m 8.5 .2
8.- 8. 8. 8.. 2.. k... a. 2.. 8. 8. E. 2. 8. 8. Sn 8... 5m 82.8 98.2 .. m
8. 8. S. 8. 8. 2.. a. 8.. R.- 8. 2. 2. 8. 8. .2 8... Pom. .855 .89.... 85.. .8
8. 8. 2.. 8. 8.- a... a .- a... 2 .. 8.- 8.. R.- 2. . a. 8. :8: 5m 48.. 8...... 8......8 .8
z 2 N. .. 2 a a .- 8 n .. m N . am .82

 

88 .N 2.3

82

$3550 2%:

 

 

TV 2. 2. E. 8. a. 8. S. 8. 2. 2. 2. 8. 8. heméommaimzﬁ
a3 a: 2.. a: 3.. 2. 2.. 2.. a. 2.. £2 a: 8.- aeaéoz .2
at a: E. a. 2. z. 2. a: 2. 2. 2. 2.- 8:83am gosuoomﬁ
33 3. N... a». an. NM. a. a». a. R. 2.. 358833 .2
G3 a. 2.. mm. 3. an. 3.. a. a. 8.. 3.38533 .3
:3 R. :2 a. R. Nm. 2. a. S. ._h§8_.cm.._om.m~
- £3 a. R. as. $.- =. a: 8.- 2. ..SSEEB .80 .2
SS 3. n». 3.. m». R. 8. Egﬁéﬁeoeso .3
85 «V. 8.. mm. 2.. .5. E .Eoeaaeou .80 .8
83 a. R. R. 8.- ﬂ dogged-bum .2
33 2. R. 3.- S 523an$$ .M:
£3 «m. S :- aouaswoméom .2
$3 3.- =3- bssamomaom 2
a: .50 .80 Bangor»; 2
an R on a z 2 2 a 8 a M: S 2 2

 

.38 .N 038.

83

Anon—.58 22$

 

 

2. 2. 2. 8. 8. 8. 8. 8.- 8. 2. :. 8.- 8. 2.. 85 3:85:85 .2.
8. 2. 8. 8. 2. 8. 8. 8. 8. 8. 2. 2. 8. 8.- 28 28.2 oasgm .2.
8. 8. 8. 8. 2. 8. 2. :. 2. 8. 8. 2. 8. 8. Em .58 28.85 .2.
8.. 2.. 2 2.- 8.. 2.. 2.- 8.. 8.. 8.. 8.. 8.. 2.- 2 .. 2. 8:. .83 85.2 .5225 .8
2. 8. 2. 8. 8. 8. 8. 8. 8.- 8. 8. 8. 8.- 2.. 25 .855 3% :88 .8
2.. 8 2. 8. 8. 8. 8. :. 8. 8. 8. 2. 8. 8.- 8.- 88 .855 58m .25 .8
2.. 2. 2. 2. 8. 2. 8. :. 2. 2. 2. 2. 8. 8.- ”Eh ..5 858m .2
8. 2. 8. 8. 2. 8. 8. 8. 8. 8. 2. 2. 8.- 8. 8:. .522. 852 .8
8. 8. 8. 8. 8. 8. 8. 8.- 8.- 8.- 8.- 8.. 8.- 2.- 8:. .855 88m 88. .8
2.. 8. 2.. 2... 2. 8. 2. :. 2. 8. 2. 2. 8. 8.. 8:. .528 85:25: .8
8. 2. 8. 2. 8.. 2. 2. 8. 2. 8. 2. 2. 8. 8. 5m ..5 858». .8
2.. 2. 2.. 8. 2. 8. 2. 2. 2. 8. 8. 8. :. 8.- 5m .528 85.5 .5: .8
2.. 2. 8. 8. 8. 8. 2. 8. 2. 8. 2. 8. 8. 8.- 5m .855 Ram .25 .8
8. 2. 2. 2. 2. 2. 8. 8. 8. 8. 2. 8. 8. 2. 5m .8280 38.2. ._ n
8. 8. :. 8. 8. 8. 8. 8. 8.- 8. 8.- . 8. 8.- 2 .- 5m .855 825 85: .8
8.- 2.. 8.- 2.. 8.- 8.- 2.- 2.. 2.. 2.- 8.. 8.- 8.- 2. 2.8 .83 88.. 858 .8
2 8 8 8 8 8 8 8 8 2 2 2 2 2

 

.88 .N 938

84

 

 

.8. 8. 2. 2.. 8. 2. 2. 2. 8. 8. 8. 2. 8. 8. 8.. 2.. 55.555288
T. 8. 2..- 8. 8. 8. 8. 8. 8. 8. 8. 2. 2.- 2.. 8.- 8.2.5.. 0.8.822.
T. 8.. 2.. 2. 8. 2. 2.. 8. 2. 8. 2. :. 2. 8.- .5”...5.. 8.28m .8
T. 2.. 8.. 2... 8.. 8.- 2... 2... 8.. 2..- 8. 2.. 8. me.- ..8.. m... 8289 .8
.E. 8. 8. 2. 2. 2. .8. 8. 2.. 8.- 2. 8. 95 .855 25 .8... .8
.5. 8. 8. 8. 8. 2. 8. 8. 8.. 2. 8.- 88 .855 8w 8.5 .8
.i. 8. 8. 8. 2. 2.. 8. 2.- 2. 2.- 88 ..5 85888
.i. 8. 2. 2.. 2.. 8. 8. 2. 8.- 8.... .388... 8.5.2 .8
T. 8. 8. k 2. 2.. 8.- 2. 2. “Eh .855 Km 85.. .8
.i. 8. 2. 2. 2. 2. 8.- 8..... .82. 85.... ..w... .8
.i. 8. 2.. 8.. 8. 2 2.. 5.". ..5 8528 .8
T. 2. 2.. 2. 8.- pom. .3»... 85.... a»... .8
.5. 2.- 8. 8.- pom. 8:5 8w 8.5 .8
7.. 2... 2... 5m .850 82.8 .. m
T. 8. pom. .855 .8 85.. .8
T. .5... 8.. m: 8.2qu 8
2. 8 2. 8 8 8 8 8 8 8 8 8 8 .n 8 2

 

.8 .N 8.2.

85

Table 2 Notes:

Values in bold type are signiﬁcant at the .05 level; values in bold italics are signiﬁcant at
the .01 level. Values in parentheses are internal consistency reliabilities, except the
experimental diary, which is an interrater reliability coefﬁcient; Perf. = Performance;

Om. = Orientation; T1 = Trial 1; T2 = Trial 2; T3 = Trial 3; EOT = End of Training; Trns

= Transfer; Spd = Speed; Pts = Points; Tgts = Targets

86

was covan'ed in all analyses. Additionally, it appeared that video game experience was
potentially related to performance on the task; this was also veriﬁed through one-way
ANOVAs which demonstrated signiﬁcant effects for video game experience on both
basic and strategic performance throughout training and generalization. In this section,
the hypothesis tests will be discussed ﬁrst. Then we will consider the effects of the
manipulations on the outcomes of interest, to determine whether this will further inform
future research, and consider possible revisions to the model if such prove necessary.
Finally, we will consider a supplemental analyses to determine the utility of the
situational judgment methodology as a tool for training evaluation, and to examine the
self-regulation diary utilized in the study.
Hypothesis Tests

Hypothesis 1 postulated that variable practice would lead to increased self-
eﬁicacy, relative to constant practice. Hypothesis 3 predicted that the presentation of
velocity feedback would also lead to increased self-efﬁcacy, relative to individuals who
did not receive velocity feedback. Hypothesis 2 predicted that variable practice would
lead to greater self-regulatory activity, relative to individuals who received constant
practice opportunities, and hypothesis 4 suggested that the presentation of velocity
feedback would lead to increased self-regulatory activity, relative to no velocity
feedback. A repeated-measures MANCOVA was run, with the covariates listed above, to
examine the effects of the two manipulations on self-regulatory activity and self-efﬁcacy

at the end of training. The results of this analysis are reported in Table 3.

87

Table 3: Multivariate Effects of Control Variables and Manipulations on Self-Efﬁcacy
and Self-Regulation at End of Trial Blocks 3, 6, and 9

 

 

df F Pillai’s Trace
Cognitive Ability 4, 305 19.023" .200
Trait Mastery Orientation 4, 305 22.625" .229
Trait Performance Orientation 4, 305 .903 .012
Gender 4, 305 4.775" .059
Video Game Experience 4, 305 2.720* .034
Practice Manipulation 4, 305 3.938“ .049
Feedback Manipulation 4, 305 5.300" .065
Block x Practice 8, 301 2.725“ .068
Block x Feedback 8, 301 4.773" .113
Practice x Feedback 4, 305 .777 .010
Block x Practice x Feedback 8, 301 .989 .026

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation; Self-Efﬁcacy (BOT) = Self-Efﬁcacy at End of Training
*=p<.1o r=p<.05 **=g<.01

In examining the results of the overall RM-MANCOVA, it is clear that the
manipulations, as well as many of the covariates, did in fact affect the self-efﬁcacy and
self-regulation variables over the course of training. However, the main effects are
difﬁcult to interpret in the presence of the two signiﬁcant interactions noted above, the
block by practice and block by feedback interactions. Examining the univariate tests, we
ﬁnd that the block x practice eﬂ‘ect primarily acts on the self-regulation DV (Fmog = 3.27,
p < .05), with the univariate effect on self-efﬁcacy being non-signiﬁcant (F2303 = .532,
ns), while the block x feedback eﬁ‘ect acts on self-efﬁcacy (F 2 .30; = 5.03, p < .01), with
the univariate effect on self-regulation being non-signiﬁcant (F2303 = 1.17, ns).

Considering the proﬁle plots (Figures 3 and 4) for these interactions revealed
interesting trends. On self-regulation (Figure 4), trainees in the constant practice

condition began higher on self-regulation than trainees in the variable practice condition,

with the difference disappearing over time (this is supported by an examination of the

88

between-subjects effects in a MANCOVA designed to explore the difference across
conditions). Note that the results here are reversed from what was hypothesized (variable
practice was expected to advantage trainees, relative to constant practice).

Looking at the effects on self-efﬁcacy (the block x feedback interaction; Figure
3), virtually no difference was present between the feedback manipulations at time 1,
while larger differences emerged at times 2 and 3. The mean differences of the results
favored trainees who received velocity feedback over those who did not receive velocity
feedback.

For both self-regulation and self-efficacy, trainee means unexpectedly dropped
from time 1 to time 3. An examination of the decline in scores on self-regulation and
self-efﬁcacy demonstrated something interesting. The mean difference from time 1 to
time 3 on self-regulation was -.3625 for trainees in the variable practice condition (with
an SD of 3.58 units) while the mean difference for trainees in the constant practice
condition was -1.2975 units (SD of 4.03). The difference between these means is
signiﬁcant (t (316) = 2.188, p < .05), indicating that trainees in the constant practice
condition showed larger decrements in reported self-regulation than did trainees in the
variable practice condition. Hence, individuals in the variable practice evidenced less
overall decline in self-regulation.

Similarly, the mean difference from time 1 to time 3 on self-efﬁcacy was -4.0124
units for trainees in the no velocity condition (SD of 6.97) while the mean difference for
trainees in the velocity feedback condition was 233% (SD of 6.06). Again, these means
are signiﬁcantly different (t (316) = -2.284, p < .05), with larger decrements in self-

efﬁcacy being found for individuals who did not receive velocity feedback. Thus, for

89

both manipulations, trainees decreased their engagement in self-regulatory processes and
on self-efﬁcacy over time, but these effects were minimized for trainees who received the
manipulations which were hypothesized to be beneﬁcial.

The relationship among the manipulations and the process variables was thus
more complex than originally hypothesized. The manipulations signiﬁcantly affected the
processes, as shown by the multivariate analysis. Examination of the signiﬁcant trial
block by manipulation interactions indicated that over time, the feedback manipulation
resulted in two distinct groups, with individuals who received velocity feedback
demonstrating higher efﬁcacy, on average, than those who did not. Moreover, while
both groups declined overall from trial block one to trial block three, individuals
receiving velocity feedback declined less in self-efﬁcacy than did individuals who did not
receive velocity feedback. The practice manipulation interacted with trial block in such a
way that while both groups declined over time, the decline was more pronounced for
individuals who received constant practice than for those who received variable practice
opportunities. The somewhat counter-intuitive nature of these ﬁndings (the general
decline of both efficacy and self-regulation) will be addressed in more detail in the
discussion section.

Hypotheses 5 through 11 dealt with the prediction of the proximal training
outcomes (learning and end-of-training performance). For these, I will follow the
hypotheses through, as they were presented in the text, and brieﬂy discuss each of the

ﬁndings.

Hypothesis 5 predicted that self-efﬁcacy would be predictive of declarative
knowledge at the end of training. A regression was run testing this hypothesis, the results

of which are presented in Table 4.

Table 4: Predicting Declarative Knowledge at the End of Training (HS)

 

 

AR7 df , - F Beta
Covariates .206 5, 312 16.22"
Mastery Om. -.072
Performance Om. -.109*
Cognitive Ability .409“
Gender .022
Video Game Exp. .049
Predictor(s) .051 1,3 1 l 21.32"
Self-Efficacy (EOT) .244M

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation; Self-Efﬁcacy (BOT) = Self-Efﬁcacy at End of Training
*=g<.10 *=p_< .05 **=p<.01

The analysis demonstrated signiﬁcant effects for two of the covariates (trait
performance orientation and cognitive ability), as well as for self-efficacy, on declarative
knowledge at the end of training. This supports hypothesis 5. Consistent with the
literature, trainees who were more efﬁcacious did, in fact, learn more about the task than
those who lacked efﬁcacy.

Hypotheses 6, 9, and 11 dealt with the prediction of adaptive knowledge at the
end of training. For, the purposes of this study, adaptive knowledge was measured with a
nine-item situational judgment measure constructed for this research paradigm. For each
question, respondents were required to read a scenario which presented them with a
problem of the type they could encounter in the task (or a modiﬁed version of the task)

and select the most and least appropriate potential responses to the scenario.

91

Hypothesis 6 speciﬁed that self-efﬁcacy would predict adaptive knowledge. That
is, trainees who were more efﬁcacious should be better able to demonstrate the capability
to adapt what they knew. Hypothesis 9 speciﬁed that self-regulatory activity should
predict adaptive knowledge, since individuals who tend to be more active, from a self-
regulatory standpoint, should have engaged in the deeper reﬂection necessary to adapt
what they have learned. Hypothesis 11 speciﬁed that declarative knowledge should
predict adaptive knowledge, verifying that some degree of positive manifold exists
among the different types of knowledge. These three hypotheses were tested with a

single regression, per the model, the results of which are presented in Table 5.

Table 5: Predicting Adaptive Knowledge at the End of Training

 

 

AR2 df F Beta

Covariates .093 S, 312 6.42"

Mastery Om. -.001
Performance Om. .028
Cognitive Ability .130”
Gender -.033
Video Game Exp. -.263
Predictor(s) .124 3,309 16.37M

Self-Efﬁcacy (BOT) ..103T
Self-Regulation (BOT) .132*

, Dec. Knowledge (EOT) .374"

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation; Self-Efficacy (EOT) = Self-Efﬁcacy at End of Training; Self-Regulation (EOT) =
Self-Regulation at End of Training; Knowledge (EOT) = Declarative Knowledge at End of
Training
*=p<.10 *=p<.05 “=p_<.01

Of the covariates, only cognitive ability was predictive of adaptive knowledge at

the end of training. That it was predictive should not be surprising. Individuals who

possess higher cognitive ability should be better at most written tests, such as the

92

adaptive knowledge test, than individuals of lower cognitive ability. Hypothesis 6, that
self-efﬁcacy would predict adaptive knowledge at the end of training, received only
marginal support. This ﬁnding is also reﬂected in the correlation matrix; self-efficacy at
the end of training exhibited a low zero-order correlation (r = .07, ns) with situational
judgment performance at the end of training. Hypotheses 9 and 11, however (that self-
regulation and declarative knowledge, respectively, would predict adaptive knowledge)
were supported. It appears that efﬁcacy is not useﬁil in predicting the development or
presence of adaptive knowledge, whereas trainee self-regulation and declarative
knowledge may be quite useﬁrl in that regard.

Hypothesis 7 speciﬁed that self-efficacy would be predictive of basic task
performance at the end of training. The results of the regression run to test this

hypothesis are presented in Table 6.

Table 6: Predicting Basic Performance at the End of Training (H7)

 

 

AR2 df F Beta
Covariates .017 5, 312 1.07
Mastery Om. -.119*
Performance Om. .046
Cognitive Ability .038
Gender -.005
Video Game Exp. .053
Predictor(s) .042 1,3 1 1 13 .75 * *
Self-Efﬁcacy (EOT) .221"

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation; Self-Efﬁcacy (EOT) = Self-Efficacy at End of Training
*=p<.10 '=p<.05 "=p<.01

Trait mastery orientation was predictive (albeit negatively) of basic performance

at the end of training, as was self-efﬁcacy. Individuals who were higher on self-efficacy

93

at the end of training were more likely to have developed the skills they needed to
prosecute targets correctly. Because end-of-training efﬁcacy was predictive of basic
performance at the end of training, Hypothesis 7 was supported.

Hypotheses 8 and 10 dealt with the prediction of strategic performance at the end
of training. Strategic performance is a cluster of behaviors performed by trainees which
indicate the adoption or use of key strategies for succeeding at the game. These are:
zooming out; hooking marker targets; querying target speed; allowing perimeter
intrusions; and engaging high priority targets. These ﬁve behaviors were entered into an
exploratory factor analysis with the two elements of basic performance (points correct
and points incorrect) to ensure that the basic and strategic performance elements clustered
appropriately. At the end of training, a principal components factor analysis employing
varimax rotation indicated that the seven variables (ﬁve strategic and two basic
performance components) split cleanly into two appropriate factors. This provides
support that the performance dimensions selected represent distinct elements of the

overall performance domain.

Table 7: Rotated Factor Matrix, Trial 9 Performance Dimensions

 

Factor 1 Factor 2

 

Points Correct .008 .924

Points Incorrect -.133 .618
Zooming Out .613 -.101
Hooked Marker Targets .532 -.117
Speed Queries .765 -.269
Penalty Points Allowed .782 .436
High Priority Targets Engaged .841 .336

 

94

The ﬁve strategic performance indicators clustered together cleanly, as
demonstrated by their rotated factor loadings above, as did the two basic performance
indicators. To simpliﬂ the analysis and presentation of the data, a summary variable was
created for overall strategic performance by creating a standardized version of each
element of strategic performance and then forming a unit-weighted composite (with
penalty points assigned a negative weight, as more penalty points indicates less strategy
use). This single indicator of strategic performance was then used in this, and all other
analyses in which strategic performance is a dependent variable. A basic performance
composite was created as well. For the sake of clarity, the elements of strategic
performance are still presented in the correlation matrix found in Table 2.

Hypotheses 8 and 10 were tested with a regression in which the effects of self-
regulation and self-efficacy were examined on strategic performance at the end of

training.

Table 8: Predicting Strategic Performance at the End of Training (H8, H10)

 

 

AR2 df F Beta
Covariates .177 5,311 13.34"
Mastery Om. .005
Performance Om. .051
Cognitive Ability .312"
Gender -.100T
Video Game Exp. .074
Predictor(s) .060 2,309 12.05M
Self-Efﬁcacy (EOT) .229"
Self-Regulation (EOT) .062

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation; Self-Efﬁcacy (EOT) = Self-Efﬁcacy at End of Training; Self-Regulation (EOT) =
Self-Regulation at End of Training

*=p<.ro *=g<.05 "=p<.01

95

In addition to a signiﬁcant effect for cognitive ability, a marginal gender effect
was demonstrated in the prediction of strategic performance. More importantly, self-
eﬂicacy was signiﬁcantly predictive of strategic performance, indicating that trainees
who believed themselves to be more capable of performing the task were, as
hypothesized, better able to do so, thereby supporting hypothesis 8 . Hypothesis 10 (that
self-regulation would predict strategic performance) was not supported.

It is worth noting again at this point that discussions of performance at “end of
training” denote performance on the tenth training trial. This tenth trial was included in
order to ensure that all participants practiced an identical trial at the end of training, to
ensure comparability of scores from the variable and constant practice groups. The end-
of-training efﬁcacy, self-regulation, and knowledge measures were collected immediately
following trial nine, to ensure that they were temporally prior to the performance
measures, which were therefore endogenous for purposes of testing the model in Figure
2.

Hypotheses 12 through 15 dealt with the inﬂuence of proximal training outcomes
(declarative knowledge, adaptive knowledge, basic performance, and strategic
performance, all at the end of training) on distal training outcomes (basic and strategic
performance during generalization). Hypotheses 12 and 14 dealt with the prediction of
basic generalization performance with declarative knowledge and basic training
performance in trial 10, respectively. The results of the regression run to test these

hypotheses are presented in Table 9.

96

Table 9: Predicting Basic Performance During Generalization (1-112, H14)

 

 

AR2 df F Beta
Covariales .059 5,31 1 3.89M
Mastery Om. .049
Performance Om. .053
Cognitive Ability .046
Gender -.073T
Video Game Exp. - -.017
Predictor(s) .542 2,309 , 209.62"
Basic Perf. (EOT) .656“
Dec. Knowledge (EOT) .253 **

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation; Basic Perf. (EOT) = Basic Performance at End of Training; Dec. Knowledge (EOT)
= Declarative Knowledge at End of Training

*=g<.10 *=p<.05 "=Q<.Ol

Of the covariates, only gender affected basic generalization performance, and then
only marginally. However, both basic performance and declarative knowledge were
predictive of basic generalization performance, indicating that prior performance on the
task is predictive of basic generalization performance and supporting both hypotheses 12
and 14. Hypothesis 12a speciﬁed that even controlling for basic performance at the end
of training, declarative knowledge would be predictive of basic performance during
generalization. Separating basic performance at the end of training and declarative
knowledge into separate steps in the regression, we ﬁnd that )R2 is .048 (p < .01) when
declarative knowledge is entered aﬁer basic performance, indicating support for
hypothesis 12a as well.

Considering the strategic elements of performance, hypotheses 13 and 15
predicted that strategic performance and adaptive knowledge at the end of training would
inﬂuence strategic performance during generalization. The most stringent test of

hypothesis 13 (and the one which best reﬂects the model) involves testing it in

97

combination with strategic performance at the end of training. This analysis is presented
in Table 10.

In this analysis, adaptive knowledge was unable to predict signiﬁcant variance in
strategic generalization performance when strategic performance at the end of training
was included in the analysis. Thus, hypothesis 13a is not supported, although if a
regression is run predicting strategic performance during generalization with the
covariates and adaptive knowledge by itself, the beta weight for adaptive knowledge
becomes a signiﬁcant .283 (p < .01), thus supporting hypothesis 13. This is relatively
less important than hypothesis 13a, however, which better reﬂects the model. Adaptive
knowledge has some predictive power with respect to adaptation, but its effects are
minimized in the presence of prior strategic performance. While hypothesis 13a was not
supported, hypothesis 15, which predicted that strategic performance at the end of '
training would predict strategic generalization performance, was.

Table 10: Predicting Strategic Performance During Generalization with Adaptive
Knowledge and Strategic Performance at End of Training (1113, H15)

 

 

AR’ df F Beta
Covariates .167 5, 311 12.44“
Mastery Om. -.005
Performance Om. -.031
Cognitive Ability .045
Gender . -.099"‘
Video Game Exp. -.059
Predictor(s) .496 2,3 09 227.63"
Adpt. Knowledge (EOT) .038
Strat. Perf. (EOT) .762"

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation; Adpt. Knowledge (EOT) = Adaptive Knowledge at End of Training; Strat. Perf.
(EOT) = Strategic Performance at End of Training

'=p<.ro *=g<.05 ”=p<.01

98

Hypothesis 16 suggested that the process variables (self-efficacy and self-
regulation) would mediate the relationship between the manipulations and the proximal
training outcomes. Hypotheses 1-4 were tested using a multivariate approach, which was
deemed more appropriate because of the need to understand how the manipulations
affected the psychological processes of trainees over time. Adopting a static approach to
the analysis of those hypotheses involves testing a pair of regressions, predicting terminal
self-efﬁcacy and terminal self-regulation with the two manipulations. The results of

these regressions are presented in Tables 11 and 12.

Table 11: Manipulation Effects on Self-Efﬁcacy at End of Training

 

 

ARI df F Beta
Covariates .141 5, 312 10.26"
Mastery Om. .250"
Performance Om. -.061
Cognitive Ability .017
Gender -.215**
Video Game Exp. .031
Manipulations .011 2, 310 2.03
Feedback . 101T
Practice -.032

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation;
T=Q<JO t==p<,05 "=Q< .01

No signiﬁcant effects emerged from the regression in which our manipulations
were used to predict self-efﬁcacy. Thus, it is inappropriate to test an hypothesis which
treats efﬁcacy as a mediator of the relationship between the manipulations and the

proximal outcomes of training.

99

Table 12: Manipulation Effects on Self-Regulation at End of Training

 

 

AR2 df F Beta
Covariates .192 5, 312 14.86M
Mastery Om. .406"
Performance Om. .033
Cognitive Ability .104“
Gender -.018
Video Game Exp. .112T
Manipulations .001 2, 310 .206
Feedback -.029
Practice -.016

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation;
I=n<,10 ‘=Q< .05 "‘=Q< .01

Again, because the manipulations were not predictive of self-regulation in this
regression, it is inappropriate to test the mediation proposed in hypothesis 16. Thus,
hypothesis 16 was not supported. The process variables, while potentially quite
important (based on the analyses reported earlier), do not mediate the effects of the
manipulations on the proximal training outcomes.

Hypothesis 17 suggested that the proximal training outcomes would mediate the
relationship between the process variables (self-regulation and self-efﬁcacy) and the
distal training outcomes (basic and strategic generalization performance). Stande tests
of mediation were conducted to test this hypothesis; regressions were ﬁrst run predicting
basic performance or strategic performance with a process variable. Then a separate
regression was run to determine if the addition of a proximal training outcome altered the
relationship between the process variable and the distal training outcome. In only one
case (self-regulation no longer signiﬁcantly predicted basic transfer performance when

declarative knowledge was included in the regression) was any evidence of mediation

100

obtained. In all other regressions, the predictive status of the process variable was not
altered by the inclusion of the “mediating” proximal training outcome. Hypothesis 17
was therefore also rejected.

The hypothesis tests only represent the ﬁrst stage of analysis, however. Because
some elements of the model were not supported, or remain unclear, in the next section we
will examine a decomposition of the observed effects, looking ﬁrst at what the
manipulations were actually able to predict, and then at the overall results of the study
(what elements of the study’s nomological network were predictive of other elements?)
Manipulation Effects

In terms of looking at the “big picture” of the results, it is useful to ask several
other questions. First, did the manipulations affect the outcomes of the training program?
Regressions were run predicting basic and strategic skill both at the end of training and
during the generalization trial to examine this question. Tables 13 through 16 are
presented in the following pages. The regressions indicated, as will be seen, that the
manipulations did possess some effects on the outcomes of interest, though they did not
affect the process variables as the model predicted. Because of the lack of prediction of
the process variables by the manipulations, they were not controlled for in the hypothesis
tests presented in the preceding pages. However, the hypothesis-testing regressions were
run a second time, including an additional step in which the manipulations were entered
as control variables, andthe effects were such that the inclusion of the manipulations did
not alter the signiﬁcance tests in almost any case. The only exception to this is the

marginal effect’for self-efﬁcacy on adaptive knowledge, which dropped even ﬁrrther

101

when the manipulations were covaried. Because no difference aside from this emerged
when the manipulations were covaried, the results are not going to be re-reported.

While the manipulations accounted for signiﬁcant variance in basic performance
at the end of training, beyond what was accounted for by the covariates, this effect was
carried by the feedback manipulation. The effect favored velocity feedback trainees over
no-velocity trainees. It does not appear that the practice manipulation signiﬁcantly

affected basic performance at the end of training.

Table 13: Effects of Manipulations on Basic Performance At End of Training

 

 

AR2 df F Beta
Covariates .017 5, 311 1.07
Cognitive Ability .058
Mastery Om. -.061
Performance Om. .034
Gender -.052
Video Game Exp. .059
Manipulations .026 2, 309 422*
Practice .075

Feedback . 142*

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation;
*=g<.ro *=g<.05 “=p<.01

Beyond signiﬁcant cognitive ability and gender covariate effects, the feedback
manipulation was the only manipulation to signiﬁcantly affect performance at the end of
training. This time, however, the effect is reversed. The mean differences on the
components of strategic performance, as well as on the composite variable, favor trainees

who did not receive velocity feedback over those who did. Individuals who did not

102

receive velocity feedback tended to perform more strategic behaviors than individuals
who did receive such feedback.

Regressions were also run predicting basic and strategic generalization
performance with the manipulations. Tables 15 and 16 contain the results of these

analyses.

Table 14: Effects of Manipulations on Strategic Performance At End of Training

 

 

ART df F Beta
Covariates .177 5, 311 13.34"
Cognitive Ability .309"
Mastery Om. .088T
Performance Om. .038
Gender -. 150*
Video Game Exp. .088
Manipulations .015 2, 309 2.73T
Practice -.043
Feedback -. 1 12*

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation;
T=;;<.10 *=p<.05 “=p<.01

Table 15: Effects of Manipulations on Basic Performance During Generalization

 

 

. AR2 df F . Beta
Covariates .059 5, 311 3.89"
Cognitive Ability .196**
Mastery Om. .010
Performance Om. .045
Gender -.1 131
Video Game Exp. .036
Manipulations .033 2, 309 5.62"
Practice * ' .122*
Feedback .133*

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation;
T=;;<.10 *=p<.05 "=p<.01

103

Here, both manipulations affected the DV of interest (basic performance during
generalization), and both were in the predicted direction (favoring variable practice and
velocity feedback trainees, respectively). As with the previous analysis, a signiﬁcant

covariate effect for cognitive ability emerged, as well as a marginal gender effect.

Table 16: Effects of Manipulations on Strategic Performance During Generalization

 

 

AR2 df F Beta
Covariates .167 5, 311 12.44"
Cognitive Ability .282"
Mastery Om. .065
Performance Om. -.002
Gender -.213**
Video Game Exp. .009
Manipulations .048 2, 309 9.46M
Practice -.027
Feedback -.218**

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation;
*=p<.10 *=g<.05 **=p<.01

Cognitive ability and gender again accounted for most of the variance in the
covariates. , As with strategic performance at the end of training, strategic generalization
performance was also predicted by the feedback manipulation, with the mean difference
favoring no-velocity trainees over velocity trainees. Individuals who received velocity
feedback-engaged in fewer strategic activities than did individuals who did not receive
such feedback. Again, while the manipulations did have some effects on the outcomes of
interest, their inclusion in the regressions did not alter the effects of the other,

hypothesized variables on the terminal effects.

104

D com sition f Model Linka s x lor to

To further explore the results, regressions were run in which basic generalization
performance, then strategic generalization performance, were predicted with all the
cognitive, affective, and behavioral predictors available within the conﬁnes of the task
(that is: self-efficacy, self-regulation, declarative knowledge, adaptive knowledge,,and
end of training basic and strategic performance). The results of these regressions are
presented in tables 17 and 18. It should be noted that elements of tables 17 and 18 are
redundant with the tests of hypotheses 12 through 15, as those dealt with the effects of
various elements of the model on the two types of generalization performance. However,
the tables also present unique information in terms of what effects other, non-
hypothesized variables had on generalization performance.

In Table 17, betas are reported at each step in the regression, as well as in the ﬁnal
step. This is done to allow a better understanding of the potentially unique portion of
variance each element of the model can explain, as well as the extent to which
performance at the end of training may mediate the effects of the remainder of the model
on generalization performance. In addition to the feedback effect (noted above), strategic
generalization performance was predicted by both types of knowledge, self-efﬁcacy, and
both types of performance at the end of training. Hence, each of these has the potential to
predict strategic performance, although all of them but the feedback effect become non-
signiﬁcant in the ﬁnal step, when the performance variables are entered. Interestingly,
the gender effect remains signiﬁcant even in the presence of all the other variables in the

ﬁnal step of the model.

105

Note that the nature of the effects described in Table 17 does make it appear that
the end-of-training performance variables may be mediating the relationship between the
knowledge and process measures and strategic generalization performance.

In predicting basic generalization performance (Table 18), the practice and
feedback manipulations (as noted above) demonstrated utility. Additionally, declarative
knowledge, self-efﬁcacy, and both types of performance at the end of training were
predictive of basic generalization performance.

Betas are presented at step and in the ﬁnal version of the model for the variables
in tables 17 and 18. The results are somewhat more robust than those for strategic
generalization performance, as even in the presence of the two end-of-training
performance variables, signiﬁcant effects are present for the practice manipulation,
declarative knowledge, and self-efficacy. No evidence was found for the linkage
between the process and knowledge measures and basic generalization performance
being mediated by end-of-training performance.

The unexpected nature of some of the results led to a series of regressions,
designed to examine other potential structures for the model. The goal was to develop a
more parsimonious model, with the potential to explain some of the unexpected ﬁndings
in the study, and to offer a way to approach ﬁrture research in this vein.

Each potential DV was regressed on all of the potential IVs (that is, all of the
variables that came before it in the model, or were entered originally in the same step as
the DV). Anything from the same step was then dropped, and the potential effects were

again examined. This was done for each element of the original model.

106

Table 17: Effects of Process, Knowledge, and Performance Variables on Strategic
Generalization Performance

 

 

AR2 df F Beta (at step) Beta (Final)
Covariares .167 5, 311 12.44"
Cognitive Ability * .302" .041
Mastery Om. .063 -.023
Performance Om. -.002 -.020
Gender -.214** -.100**
Video Game Exp. ’ .144 -.049
Manipulations .048 2, 309 9.46"
Practice -.027 .019
Feedback -.218** -.109**
Knowledge Meas. .097 2, 307 21.53"
Declarative .220* * .01 7
Adaptive . 192* * .042
Process Indicators .020 2, 305 4.61*
Self-Efﬁcacy .163** .041
Self—Regulation -.006 .000
Performance .377 2, 303 195.63"
Basic (EOT) -.178** -.178**
Strategic (EOT) .746M .746"

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation;
*=p<.10 *=p<.05 "=p<.01

107

Table 18: Effects of Process, Knowledge, and Performance Variables on Basic
Generalization Performance

 

 

ARI df F Beta (at step) Beta (ﬁnal)
Covariates .059 5, 311 3.89"
Cognitive Ability .177** .041
Mastery Om. .005 .003
Performance Om. .044 .046
Gender -. 1 15T -‘-.029
Video Game Exp. .036 -.027
Manipulations .033 2, 309 5.62"
Practice . 122* .080*
Feedback .133* .050
Knowledge Meas. .142 2, 307 28.53"
Declarative .395" . l64**
Adaptive - .063 .023
Process Indicators .051 2, 305 10.93"
Self-Efﬁcacy .272" . 125 * *
Self-Regulation -.044 .006
Performance .352 2, 303 147.44"
Basic (EOT) .624" .624"
Strategic (EOT) .133** .133"

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation;
T=p<.10 *=p< .05 "=p<.01

108

The regressions run, and what they revealed, will be discussed below, beginning
with the terminal end of the model and considering revisions working toward the front.

Recall that table 17 summarized the effects of all the elements of the model on
strategic generalization performance. Another regression was run in which all of the
elements in table 17 were considered, with the addition of basic generalization
performance as a potential predictor of strategic generalization performance. Entered in a
ﬁnal step by itself, basic generalization performance possessed a beta weight of .090 (p >
.05), indicating that no good reason existed to separate the elements of performance in the
last step of the model.

Table 18 reviews the effects on basic performance during generalization, which
includes everything with potential causal priority that was included in the model. The
next step back in the model involves a more detailed consideration of the effects on basic
and strategic performance at the end of training.

Examining the Tables 19 and 20, it becomes apparent that the knowledge
measures are in fact predictive of performance at the end of training; hence, the model
probably needs to be revised to reﬂect the causal priority of this knowledge to end-of-
training performance. While the logic of the original model may still be argued
(knowledge and performance are both types of outcomes from training, and might
reasonably be considered in the same “step” for model deﬁnition), the data indicate that
the knowledge outcome is a necessary (though not necessarily sufﬁcient) condition for
the performance outcome to occur. This, then, represents one potential change to the

model from what was originally hypothesized. ‘

109

Table 19: Effects on Basic Performance at End of Training

 

 

 

 

 

AR2 df F Beta
Covariares .017 5, 311 1.07
Mastery Om. -.078
Performance Om. .074
Cognitive Ability -.035
Gender -.005
Video Game Exp. .050
Manipulations ' .026 2, 309 422*
Feedback . 126*
Practice .074
Process Variables .038 2, 307 6.37"
Self-Efﬁcacy . 192* *
Self-Regulation -.074
Knowledge Measures .037 2, 305 6.40"
Adaptive Know. .040
Declarative Know. .207**
Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation;
*=p<.ro r'=;;< .05 “=p< .01
Table 20: Effects on Strategic Performance at End of Training

AR2 df F Beta
Covariares .177 5, 311 13.34M
Mastery Om. .044
Performance Om. .086T
Cognitive Ability .136"
Gender -.0931
Video Game Exp. .066
Manipulations .015 2, 309 2.78T
Feedback ' -. 1 13 *
Practice -.044
Process Variables ' .065 '2, 307 13.32"
Self-Efficacy .209**
Self-Regulation -.026
Knowledge Measures .124 2, 305 30.53"
Adaptive Knowledge .220**
Declarative Knowledge .262**

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. =

Orientation;
*=p<.ro *=p_< .05 ”=p<.01

llO

Trait Performance

With this modiﬁcation in place, the next question that must be addressed is
whether this means the knowledge measures should be entered in the same step, for
model-testing purposes, as the process variables, or whether there is some causal ordering

we can infer from further analyses, using the knowledge measures as DVs.

Table 21: Predicting Declarative Knowledge

 

 

AR2 df F Beta
Covariates .206 5, 312 16.22"
Mastery Om. -. 107*
Performance Om. -.1 16*
Cognitive Ability .397"
Gender .013
Video Game Exp. .036
Manipulations .001 2, 310 .198
Feedback -.025
Practice .039
Process Variables .062 2, 308 13.14"
Self-Efﬁcacy . 187* *
Self-Regulation . 127*

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation;
t=g< .10 *=p<.05 **=p< .01

The fact that both self-efficacy and self-regulation are predictive of declarative
knowledge, combined with the fact that they are collected temporally prior to the
knowledge measure, might reasonably lead us to conclude that the process variables are,
in fact, causally prior to declarative knowledge, and hence do not belong in the same step
for model-testing purposes. Identical analyses were run for adaptive knowledge,
revealing that self-regulation (but not self-efﬁcacy) was predictive of this form of
knowledge (see Table 22). This provides further evidence that the process variables

should be considered causally prior to the knowledge variables.

lll

Table 22: Predicting Adaptive Knowledge

 

 

AR2 df F Beta
Covariates .093 5, 312 6.42"
Mastery Om. -.041
Performance Om. -.015
Cognitive Ability .273"
Gender -.025
Video Game Exp. -.001
Manipulations .007 2, 310 1.29
Feedback -.079
Practice .016
Process Variables .020 2, 308 3.58*
Self-Efﬁcacy -.027
Self-Regulation . 171 * *

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation;
*=g<.10 *=p<.05 **=p<.01

It is interesting to note, in Tables 21 and 22, that self-regulation is predictive of
both types of knowledge. This ﬁnding, in combination with the fact that self-regulation
was directly predictive of little else in the study, will be taken up further in the discussion
section.

Recalling from the earlier tables that there were no effects on self-efﬁcacy or self-
regulation at the end of training, from a regression sense, but that each was affected by
one of the manipulations when the effects of the manipulations were examined over time
(the block by manipulation interactions), there is no reason to alter the front end of the
model. SummariZing what elements of the model were predictive of others, then, we

have the following.

112

Summag

Practice variability affects only basic performance during generalization. It also
interacts with time to affect self-regulation. Velocity feedback interacted with time to
affect self-efﬁcacy. Additionally, it affected both basic and strategic performance at the
end of training, and strategic performance during generalization. This is the only variable
which had an effect on strategic generalization performance that was not mediated by
strategic training performance.

Self-efficacy affects declarative knowledge, basic and strategic performance at the
end of training, and basic generalization performance. Self-regulation affects both
declarative knowledge and adaptive knowledge, only one of which was hypothesized.
Declarative knowledge affected basic and strategic performance at the end of training, as
well as basic generalization performance. Adaptive knowledge affected strategic training
performance. Basic and strategic performance during training, and basic performance
during generalization, were predicted by much the same things; however, no justiﬁcation
existed for putting the three of them in a single step in the model, as basic generalization
performance was not predictive of strategic generalization performance.

Both basic and strategic performance at the end of training predicted both basic
and strategic performance during generalization. In addition, strategic training
performance mediated the relationship between the knowledge measures, efficacy, and
strategic generalization performance. Effects exist for all of these on strategic
generalization performance before strategic training performance is entered in the

regression, at which point the effects disappear. This is classic statistical mediation.

113

Based on these analyses, it appears that a revision of my original model would
include those relations depicted in Figure 5. The model is far from perfect, and will need
to be reﬁned through further research to determine the precise nature of the effects
observed. The model represents the causal ordering that emerged ﬁom more detailed
analyses of the data ﬁ'om this study. Post hoc analyses of the data thus offer potential
directions for reﬁnements to the model which, as always, are the purview of ﬁrture
research.

Supplemental Analyses

The situational judgment methodology used to measure adaptive knowledge
represents an important methodological contribution of this study. One of the key ways
to determine whether this new measurement technique will prove beneﬁcial is its
capability to explain variance in performance beyond what we can explain with other
tools already at our disposal. To test this, I entered the control variables, followed by
declarative knowledge, and ﬁnally situational judgment performance, in a hierarchical
regression predicting strategic generalization performance. Even with cognitive ability,
declarative knowledge, and the other variables in the mix, adaptive knowledge as
measured by the situational judgment test explained a signiﬁcant increment in variance
(AR2 = .03 6, p < .001), lending credence to the idea that this may be a useful tool for
predicting the more complex elements of training performance and, perhaps more
importantly, the-capacity to transfer those complex elements to other environments.

The experimental diary utilized in this study was designed to provide an alternate
measure of trainee self-regulation to the Likert-type scale used to assess self-regulation

following trials 3, 6, and 9. A second rater was trained to score the diaries, utilizing the

114

scoring rules described in the method section. Interrater reliability for overall diary score
was .96. In addition, the diary only correlated ; = .13 (p < .05) with the other measure of
self-regulation at the end of training, indicating that while some overlap exists across the
measures of self-regulation, the actual shared variance across the assessments is minimal.
Several of the analyses were re-run using the diary in place of the other self-
regulation measure, with surprising results. First, the tests of hypotheses 2 and 4 were
run again, using the diary as the measure of self-regulation to be predicted by the
manipulations. As can be seen from table 23, both of the manipulations were predictive

of diary score.

Table 23: Manipulation Effects on Self-Regulation Diary at End of Training

 

 

ARI df F Beta
Covariates .081 5, 312 5.530"
Mastery Om. .025
Performance Om. -. 128*
Cognitive Ability .233"
Gender .022
Video Game Exp. -. 120*
Manipulations .070 2, 310 12.761 **
Feedback -. 105 *
Practice .246"

 

Notes: Mastery Om. = Trait Mastery Orientation; Performance Om. = Trait Performance
Orientation;
*=p<.10 *=g< .05 **=p< .01

Both of the manipulations are signiﬁcantly predictive of self-regulation diary
score, although the feedback effect is, as has been noted throughout, inverted such that
individuals who received no velocity feedback scored higher on the measure of self-

regulation than individuals who received velocity feedback. This effect was

115

nonsignifrcant when predicting the Likert-type measure of self-regulation. More
importantly, the practice manipulation, which was also nonsignifrcant in the previous
analysis, signiﬁcantly affected self-regulation diary score, such that individuals who
received variable practice opportunities did in fact get higher scores on the self-regulation
diary (that is, they reported attempting to engage in more of the goals of the training
program than did individuals who received constant practice scenarios). This would
seem to indicate that participants who practiced variable versions of the task were more
likely to be aware of and focusing on the goals of training than individuals who practiced
the same task every time, supporting hypothesis 2.

In addition, controlling for the manipulations as well as the other covariates, diary
score was signiﬁcantly predictive of strategic performance at the end of training (beta =
.221, p < .01), with EOT strategic performance mediating the relationship between self-
regulation and strategic generalization performance. Diary score was not, however,
predictive of basic performance either at the end of training or during generalization. The
diary was also signiﬁcantly predictive of adaptive knowledge at the end of training, with
a stronger beta weight (.227, p < .01) than that found for the other measure of self-
regulation used in the study, and was predictive of declarative knowledge at the end of
training as well. In many respects, then, the diary measure of self-regulation performed
more in line with theoretical predictions than did the Likert-type measure, a ﬁnding
which will be taken up in the discussion section.

Discussion
This study was designed to enhance our understanding of how the presentation of

variable practice opportunities and velocity feedback inﬂuenced behavioral and cognitive

116

outcomes, both at the end of training and during generalization. A model was proposed
in which the effects of the two manipulations of interest were proposed to occur through
self-regulatory mechanisms, consistent with a growing body of literature which treats
self-regulation as central to learning and skill development (e.g., Kanfer & Ackerman,
1989). While variable practice and velocity feedback have both received attention in
elements of the psychological literature in the past, they had not been examined,
separately or in combination, in an experiment focused on training individuals on a
complex cognitive task like the one utilized for this study. Hence, the study sought to
increase our understanding of how these two manipulations might operate to produce the
kinds of outcomes desired by trainers in a simulated real-world environment.
Overview of Results

Ten of the seventeen hypothesized relationships emerged when the data were
analyzed, with support found for hypotheses 2 (variable practice predicting self-
regulatory activity), 3 (velocity feedback predicting self-efficacy), 5 (self-efﬁcacy
predicting declarative knowledge at the end of training), 7 (self-efﬁcacy predicting basic
training performance), 8 (self-efﬁcacy predicting strategic training performance), 9 (self-
regulatory activity predicting adaptive knowledge at the end of training), 11 (declarative
knowledge predicting adaptive knowledge), 12 (declarative knowledge predicting basic
task performance), 14 (basic training performance predicting basic generalization
performance), and 15 (strategic training performance predicting strategic generalization
performance). Hypotheses 1 (variable practice predicting self-efﬁcacy), 4 (velocity
feedback enhancing self-regulatory activity), 6 (self-efﬁcacy predicting adaptive

knowledge), 10 (self-regulatory activity predicting strategic training performance), 133

117

(adaptive knowledge predicting strategic generalization performance), 16 and 17 (the
mediational hypotheses) were not supported. These will be discussed below.

Because of the nature of the data, hypotheses 1 through 4 were examined
longitudinally. The support found for hypotheses 2 and 3 was based on signiﬁcant trial
block by manipulation interactions, which were then decomposed to determine what
occurred across conditions, over time. It is interesting that the only signiﬁcant effects
over time emerged for feedback on self-efﬁcacy, and for practice on self-regulation. If
we consider the nature of the manipulations, however, this may not be overly surprising.
For explanatory purposes, I’m going to consider hypotheses 3 and 4 first, and then move
back to hypotheses l and 2.

Feedback is, in many instances, designed to focus trainee attention on critical
elements of the task and to provide information about how trainees are doing on those
elements. Such information will inherently affect trainee self-perceptions; if the
information they receive indicates that they are accomplishing their goals, then their
efficacy should be increased, while if the information they receive indicates that they are
failing to accomplish their goals their efﬁcacy should be diminished. While the velocity
feedback is not inherently evaluative in nature, it is reasonable to expect that trainees
assigned evaluative meaning to the scores they received, representing their feedback, and
made further evaluations based on the progress they observed over time on the task.
Thus, it makes sense that hypothesis 3, which postulated a link between feedback and
self-efﬁcacy, should have been supported. The observed effect presented in an
interaction of the feedback manipulation with time, such that over the course of the

experiment, trainees who received velocity feedback demonstrated less decline in

118

efficacy than did trainees who did not receive velocity feedback. Thus, the velocity
manipulation was advantageous with respect to efﬁcacy.

By the logic outlined above, feedback should have also had an effect on self-
regulation, per hypothesis 4. It was expected that the addition of velocity feedback would
augment trainee cognitive activity, and allow them to self-regulate “better” (in terms of
doing more of the appropriate activities - monitoring goals, progress, performance,
learning, and feedback), yet this did not occur. Instead, the effect may have been one of
interference, rather than augmentation. Trainees were provided with velocity (score)
feedback that was not directly relevant to the goals of the training program. It is possible
that they were distracted from their established goals by the velocity feedback, and set
their own goals. This goal shift could easily result in the reallocation of their self-
regulatory activity to domains other than those covered by the scale used to assess self-
regulation in this study. To assess this possibility, I examined the amount of time
trainees spent reviewing the feedback screens, as well as the contents of their
experimental diaries. Interestingly, trainees receiving velocity feedback spent
signiﬁcantly we time reviewing the non-velocity (i.e., directly goal-relevant) feedback
screens than did trainees who received velocity feedback, according to a series of one-
way ANOVAs. This was true across all nine training trials.

In addition, examining the experimental diaries of the panicipants reveals that
individuals who received the variable practice manipulation tended to be able to self-
report more of the goals of training than did individuals who received the constant
practice manipulation (this difference was signiﬁcant), and the individuals who did p9_t_

receive velocity feedback tended to report attempting to reach more of the goals of

119

training in their diaries than did individuals who received velocity feedback. This
indicates that individuals who received the variable practice manipulation and who did
p91 receive the velocity feedback manipulation reported attempting to attain the goals of
the training program signiﬁcantly more than individuals who received constant practice
or velocity feedback. This suggests two things. First, that variable practice encourages
trainees to focus on the speciﬁc goals of training, and second, that the presence of the
velocity feedback may distract trainees from the goals of the training program. The fact
that individuals who looked at the feedback directly relevant to strategic performance
more ultimately did m on strategic performance, both at the end of training and
during generalization, with the only difference being that they also received
velocity/score information, might reasonably lead us to argue that the presence of the
velocity feedback modiﬁed how the non-velocity feedback was perceived and utilized by
trainees.

With respect to hypotheses 1 and 2, which dealt with the inﬂuence of practice
variability on self-efficacy and self-regulation, only hypothesis 2 was supported.
Trainees receiving variable practice demonstrated less decline in their self-regulatory
activity over time than did individuals who received constant practice (see Figure 4).
However, the mean difference across groups actually showed that overall, trainees who
received constant practice self-regulated more than trainees who received variable
practice.

If we examine trends in the data, it does appear that while the means overall
favored constant practice over variable practice trainees on self-regulation, this difference

primarily occurred early in training, and became minimal at the end of training. It is

120

possible that in a longer-term experiment trainees in the variable practice condition would
have demonstrated higher self-regulatory activity than trainees in the constant practice
condition, but the time parameters of this research, along with the complexity of the task
to be learned, kept them from engaging in the more advanced self-regulation the measure
was designed to assess.

There was no signiﬁcant effect (either main effects or interactions) for the
practice manipulation on self-efﬁcacy (Hypothesis 1). This may have been due in some
part to the manipulation itself. While it was emphasized early in the experiment that
trainees would receive variable practice opportunities (for individuals in the appropriate
condition), the nature of early portions of the task is fairly repetitive, in terms of having
to perform the same basic operations (hook a target, make decisions, ﬁnd another target),
and might not lead to efﬁcacy gains relative to individuals who have identical scenarios
to practice every time. If the differences across trials were more pronounced, or if the
potential beneﬁts of variable practice opportunities were stressed more, then larger
efﬁcacy gains might have resulted for individuals in the variable practice condition.

A further consideration of the declines in self-regulation and self-efﬁcacy may be
warranted, to explore the ﬁndings. Both self-efﬁcacy and self-regulation were at their
highest following the ﬁrst trial block. Mean self-efﬁcacy at that point was 31.8, and
mean self-regulation was 24.1. Both dropped following the second trial block, to 28.0
and 23.2, respectively, before recovering slightly (28.7, 23.3) following the ﬁnal trial
block. The declines occurred in spite of overall improvements in both knowledge and
skill domains. Mean declarative knowledge scores rose from 13.0 to 14.7 questions

correct, and mean adaptive knowledge scores rose from 6.0 to 7.0. In addition, a general

121

upward trend, reminiscent of traditional learning curves, was observed in data on task
performance, and is demonstrated in Figure 6.

Examining Figure 6, it becomes clear that within each 3-trial block, improvement
took place in how well trainees performed the task. The bulk of the improvement in
trainee score occurred early in training, as is typically the case with learning curves, and
the improvement became less pronounced as trainees moved into the more complex,
strategic goals of the latter portions of the training program.

It may be that the change in rate of improvement is responsible for the observed
decrements in the process variables. In the case of self-efﬁcacy, trainees make initial
estimates of their own competence based on what they’ve seen of their performance. If
they do not feel that they are doing as well (and there does appear to be a slight decline in
performance from the end of each trial block to the beginning of the next), then their
efﬁcacy may drop. Another possibility is the goals trainees are given. Early in the
training process, the goals are simple, but become more complex as training progresses.
The shift in apparent complexity of the goals may be responsible for declines in efﬁcacy,
as trainees may believe themselves to be less able to meet the more complex goals than
they were to meet the relatively simple goals of the ﬁrst trial block. This would account
for the initial decrease in efﬁcacy following trial block two, where the goals became
much more complex, and the slight increase in efﬁcacy beyond that which was observed
in trial block three, where the goals were still complex, but trainees had experience with
complex goals and felt better about how they’d done than during their ﬁrst exposure to
such goals. Hence, efﬁcacy would logically have been highest for the simplest goals, and

would have been lower for the more complex goals from trial .blocks two and three.

122

Declines in efﬁcacy are not unheard-of, from a training perspective. Hesketh
(1997) warned that one of the dilemmas inherent in the design of any training program is
when to introduce the more difﬁcult elements of training, to ensure that trainees have had
sufﬁcient time to learn the simple elements and become comfortable with them.
Introducing the difﬁcult elements too early, Hesketh warns, may “disrupt growing self-
efﬁcacy for the learning task” (p. 330). Based on the trends in the data, it does appear
that the decline in efﬁcacy began when the goals shifted from relatively simple goals, to
more complex goals. Saks (1997) points out that one of Bandura’s (1986) four
mechanisms by which self-efﬁcacy can be inﬂuenced is mastery experiences, in which
trainees have the opportunity to successfully perform training tasks; declines in efﬁcacy
may thus be due to a lack of closure for trainees on any of the skills they have learned. If
trainees do not have the opportunity to successfully perform the training task, or do not
receive feedback that they have been successful at the task (either of which might
constitute a “mastery experience”), then they may not develop appropriate levels of self-
efﬁcacy. They may need some form of closure before changing goals. Hence, the
declines in self-efﬁcacy, while not necessarily the norm for training research, might be
due to factors that have been noted previously in the literature, and which can therefore
be dealt with as further research is conducted.

A similar argument could be used to explain the decline in self-regulation. 1n the
early stages of training, it was relatively easy for trainees to monitor their goals, progress,
performance, learning, and feedback, because the focus was on simpler elements of the
task. However, as the goals become more complex, it is not unreasonable to expect that

trainees Will report less active monitoring of the ﬁve elements of self-regulation assessed

123

in this study, since their attention should be increasingly taken up by more complex
goals. This possibility could be addressed in future research by assessing beta and
gamma change in the scale over time. Such an exploration would allow us to determine
if the meaning of the scales changes over time.

In some respects, these ﬁndings mirror Kanfer and Ackerrnan’s (1989) resource
allocation perspective, in that the limited cognitive resources of trainees become an issue.
Moreover, Hesketh (1997) notes that when the attentional demands of a training task are
high, trainees may not possess enough spare cognitive capacity to engage in the kinds of
cognitive activities necessary for self-monitoring. The training program in this study was
designed in such a way as to minimize the cognitive requirements for the initial three-trial
block, by focusing on simple declarative information, but the attentional demands were
increased markedly when defensive perimeters and marker targets were introduced in the
second trial block. Now, in addition to simply making decisions about what targets are,
trainees were required to attend to the overall context of the scenario, to make decisions
about which targets were important to prosecute ﬁrst, and to locate an outer boundary
which was functionally invisible. With these new elements of the task to focus on, we
may argue that they did nothave the capacity left for self-monitoring. However, this
possibility cannot be uniquely addressed in this research.

A second explanation for the declines in self-regulation is found in potential
frustration with progress. Again, over time (Figure 6) trainee progress begins to slow.
The fact that they have plateaued is not lost on trainees, who may decide that the
experiment is no longer worth their full attention, and stop self-regulating to the same

level that they did in the ﬁrst trial block when they stop seeing the same levels of

124

improvement, or when the goals become more difﬁcult to achieve. Thus, we can explain
the declines over time in terms of either increased complexity taking up limited
resources, or increased complexity inducing frustration in trainees. It is also worth noting
that scale-based self-regulation ultimately predicted only cognitive (i.e., learning)
outcomes in this study.

Hypotheses six, nine, and eleven all dealt with the prediction of adaptive
knowledge, a relatively unstudied construct, in spite of its theoretical relevance to issues
of training transfer. Hypothesis eleven, that declarative knowledge would predict
adaptive knowledge, was supported. Because the basics of the game (which are
subsumed under declarative knowledge about the decision process) must be understood
before more complex strategic behaviors can be meaningfully implemented, it makes
sense that declarative knowledge should predict adaptive knowledge, which deals with
applying strategic information to novel situations.

Hypothesis nine, that self-regulatory activity would predict adaptive knowledge,
was also supported. When we note that self-regulatory activity was assessed (as above)
by asking trainees to report the extent to which they engaged in various types of
monitoring, it makes sense that this should be linked to cognitive outcomes. Also, recall
that Table 21 demonstrated that self-regulatory activity was predictive of declarative
knowledge. These two knowledge measures were the only things that the self-regulation
measure signiﬁcantly predicted. With its focus on monitoring, the measure may have
been tapping only cognitive elements of self-regulation, and thus only predicted cognitive
outcomes. This might explain why hypothesis 10, that self-regulatory activity would

predict strategic training performance, was not supported; the types of self-regulatory

125

activity tapped by the measurement were primarily cognitive, rather than behavioral, and
thus were not predictive of the behavioral outcomes. While it is certainly not the case
that strategic training performance was a purely behavioral construct (it did require
cognitive underpinnings), the performance constructs represented the most behavioral of
the criteria in the experiment. In support of the cognitive/behavioral outcome split, the
experimental diary (a more behavior-focused measure of self-regulatory activity) was
predictive of behavioral (i.e., performance) outcomes.

Hypothesis six, that self-efﬁcacy would predict adaptive knowledge, was not
supported. This may be due to the greater complexity of adaptive knowledge, relative to
declarative knowledge, which self-efﬁcacy successfully predicted. However, the lack of
an effect for self-efﬁcacy on adaptive knowledge is somewhat troubling, as the efﬁcacy
measure speciﬁcally includes items such as, “I believe I can develop methods to handle
changing aspects of this task.” The lack of effect may be due to the primarily affective
nature of the efﬁcacy judgment, as opposed to the cognitive nature of the adaptive
knowledge test; it is relatively easy for trainees to guess that they can adapt what they
have learned, but it is clearly another thing entirely for them to actually do so. If this
were not the case, training transfer would‘not be nearly the critical concern in the
literature that it is. Nonetheless, a better understanding of the link between efﬁcacy and
the adaptation of knowledge must be obtained. As has already been noted, however, the
efﬁcacy measure m able to predict strategic performance during generalization; hence,
although the cognitive version of adaptability was not predicted by the efﬁcacy measure,
the behavioral version was. The more basic version of task knowledge was, however,

predicted by self-efﬁcacy, thereby supporting hypothesis ﬁve. Hypotheses seven and

126

eight, which suggested that self-efﬁcacy would predict basic and strategic training
performance, were also supported.

The prediction of generalization performance (both basic and strategic) is one of
the areas in which interesting results emerged. One such ﬁnding was the mediation of the
effects on strategic generalization performance by strategic training performance.
Hypothesis 13a (that adaptive knowledge Would predict strategic generalization
performance when controlling for basic generalization performance) was not supported.
The revised model presented at the conclusion of the results section should serve to guide
future research attempting to understand how knowledge affects strategic generalization
performance, as it places knowledge assessments in a step of the model causally prior to
end-of-training performance.

The other hypothesized knowledge-performance link, that declarative knowledge
would predict basic task performance (hypothesis 12) was supported. In addition,
hypothesis 13, that adaptive knowledge would predict strategic generalization
performance in the absence of information on end-of-training strategic performance
information, was supported as well. Hypotheses l4 and 15, which predicted that end of
training performance (both basic and strategic) would predict generalization performance,
were supported as well. The remaining hypotheses, 16 and 17, were both mediational,
and did not ﬁnd support from the data.

Hypothesis 16 suggested that the process variables (self-efﬁcacy and self-
regulation) would mediate the relationship between the manipulations and the proximal
training outcomes, while hypothesis 17 predicted that the proximal training outcomes

would mediate the relationship between the process variables and the distal training

127

outcomes. For hypothesis 16 to be tested as written would have required direct main
effects on the terminal process outcomes by the manipulations, which did not emerge in
the data (recall that the effects were interactional in nature, involving manipulation by
time effects). It appears, based on the exploratory analyses conducted, that both direct
and indirect effects were present from the manipulations to the proximal training
outcomes, but that too much was going on in the model to uniquely identify those effects.
Some degree of decomposition has been accomplished, in terms of better specifying the
likely causal sequence and identifying which elements of the model are able to predict
other elements. However, it may require studies focused on smaller portions of the
model to identify the speciﬁc conditions under which the manipulations will lead to the
outcomes desired.

The mediation suggested by hypothesis 17 was also not fully supported, although
the only true mediation found in the study was through strategic performance at the end
of training, to strategic generalization performance. It appears that the relationships
among the elements of the model are more complex than can be accounted for by a
simple mediational system. With the causal ordering better established, a more thorough
decomposition of the elements of the model will need to be conducted through studies
focused around speciﬁc sections of the model.

The proposed model did not ultimately provide a good ﬁt to the data, but
revisions of the model are being considered, and further analyses are being conducted to
reveal trends which might inform future model-building activities. In the next few pages,
' I would like to address some of the contributions of the study. I will then address the

limitations, and conclude with a consideration of potential avenues for new research that

128

may open as a result of this study.
Contributions

The study offers several potentially important contributions to the literature.
These fall into measurement, theoretical, and practical domains.

On the measurement front, one of the things this study did was provide a
mechanism through which adaptive knowledge could be assessed. The situational
judgment test developed for this study, per the guidelines developed by Motowidlo et al.
(1990), offered a tool that was relatively distinct from traditional declarative knowledge,
and which explained signiﬁcantly more variance in strategic generalization performance
than declarative knowledge explained on its own. Further reﬁnement of the measurement
technology appears to be necessary if such tests are going to be able to out-predict prior
strategic performance. However, the delay between the end of training performance
measure and the generalization trial was relatively small, meaning that it is possible the
strength of the observed effects for prior performance generalization performance might
be inﬂated beyond what we would see in other training programs. With a longer delay
between the end of training performance measure and the assessment of skill
generalization, we might ﬁnd the link between the two types of performance diminished,
while the predictive power of the test itself did not. While the data from this study cannot
directly address this possibility, it remains an option for future research to consider.

The measurement of self-regulation is another area that was clariﬁed (in some
ways) by this research. Based on the results of the study, it appears that the assessment
of self-regulation that was used for this research may have been too heavily cognition-

oriented. That is, it may have ignored the more affective and behavioral components of

129

self-regulation, focusing solely on monitoring, and therefore been able to predict only
cognitive outcomes. The possibility that written measures of self-regulation may only tap
cognitive components of self-regulation is one that must be considered as further self-
regulatory measurement tools are devised.

An additional possibility regarding the Likert-type measure of self-regulation
used in this study arose when the data from the experimental diaries were examined. In
considering the diary data, it became clear that the diary behaved much more in the
manner the literature would suggest an assessment of self-regulation should behave than
did the scale-based measure. The difference between these two measures, in many
respects, is the extent to which they differentially require trainees to actively self-
regulate. The scale-based measure, while asking questions about self-regulation, may not
require a great deal of self-regulation to actually answer, since trainees need only bubble
in sections of a scantron sheet. The‘diaries, however, require that trainees actually reﬂect
on the task and their approach to the task, and report those goals which they treated as
focal. Hence, the measure of self-regulation that required more self-regulation, the diary,
seems to have performed better for the purposes of the study. It predicted both cognitive
and behavioral outcomes, and allowed a more detailed look at what trainees were
focusing on across conditions. While it is reasonable that the better measure of self-
regulation should be the one which actually requires increased self-regulation, this has
not been discussed in the literature, and the contrast between the two different measures
of self-regulatory activity is an interesting one.

With respect to self-regulation, this study also attempted to further explore the

role of self-regulation in training skill development. It is clear, from the cognitive

130

literature, that some level of self-regulatory activity may be necessary for appropriate
skill development to occur. While there were not direct scaled-based self-regulation -—
skill linkages in this study, self-regulation was deﬁnitely related to the development of
both declarative and adaptive knowledge, lending credence to the idea that such activity
is crucial to trainees gaining the necessary knowledge base from the training process.
The version of self-regulation tested by the experimental diary was related to the
development of strategic skill, indicating that the linkage between self-regulation and
training outcomes may depend in part on how self-regulation is assessed.

Beyond the issues above, the study increased our potential understanding of the
role of velocity feedback in training design, while simultaneously raising new, interesting
questions. Individuals who were given velocity feedback spent more time looking at the
other feedback information they were providedthan did individuals who were not given
velocity feedback. However, this did not result in the desired outcomes. Instead, it
seems to have created a situation in which trainees took the velocity feedback (which
may have oriented them much more toward performance) and examined the feedback
relevant to the sequenced mastery goals in that light, potentially creating a frame of
reference distinct from that which would have been provided by either mastery or
performance feedback on its own. Velocity information thus represents one mechanism
by which increased attention to all of the feedback provided in a training program may be
obtained.

This allows us to reconsider the results of Mullins et al. (1999), who found that
individuals who received both mastery and performance (velocity) feedback did best on

the generalization trial, as opposed to individuals who received either type of feedback

131

individually. Referring back to the results of that study, Mullins et al. (1999) found that
the best scores during generalization were obtained by individuals who received both
mastery and performance feedback. This effect was replicated in this study, with basic
performance during both training and generalization performance maximized by
individuals who received both sequenced mastery feedback as well as velocity feedback.
The additional information afforded about where trainee attention was actually focused in
these conditions (increased attention to the other elements of feedback in addition to the
velocity information) allows us to begin to speculate about theoretical mechanisms which
might be responsible for the ﬁndings of both studies. The construction of schemata (a
sense-making mechanism on the part of trainees) is one potential explanation. This
explanation is consistent with the observed effects of velocity feedback. One
interpretation of those data would suggest that trainees who received no velocity
feedback developed a cognitive representation of the training environment which
emphasized the training goals (ultimately in terms of strategy development and usage),
while trainees who received velocity feedback developed a separate cognitive
representation of the environment in which they considered everything in terms of the
velocity feedback, and modiﬁed their knowledge structures to allow for strategy use in a
velocity-heavy environment. The diary ﬁndings tend to support this assertion. However,
since they tell us that trainees who received velocity feedback reported fewer training
goals, the question of whether different schemata will be developed based on the
presence or absence of velocity feedback is one that must be taken up in future research.
Ultimately, it remains unclear what is actually happening with respect to trainee attention,

and this will be another focus for future research.

132

Another potential contribution of the study is its ability to offer an initial
explanation for the efﬁcacy declines noted in the Kozlowski, Mullins, Toney, and Bell (in
preparation) study. The fact that efﬁcacy declined while performance improved, overall,
is interesting. It appears that the efﬁcacy declines may be related to the shifts in goals
over the course of training, as each goal shift was related to a concomitant (if only
temporary) decline in performance, which might have led to efﬁcacy declines over the
course of training (see Figure 6). This will be taken up in more detail in my discussion of
the limitations of the study, since while the data offer a slightly better understanding of
what is going on with efﬁcacy, the efﬁcacy declines were not predicted and as such
represent both a positive and a negative element of the ﬁndings.

Having considered both measurement and theoretical contributions of the study,
we are left with the contributions the study makes to practitioners. First, the study may
have implications for training design. Providing trainees with variable, as opposed to
constant, practice opportunities was shown to be advantageous with respect to self-
regulatory activity. This is particularly true for the diary-based assessment of self-
regulation, where the practice manipulation had a signiﬁcant main effect on diary score,
rather than the interaction with time that was observed for the scale-based assessment of
self-regulation. If a goal of training is for trainees to be more aware of and focus on the
speciﬁc goals of the training program, we might utilize variable practice opportunities,
rather than constant. In addition, it is clear that feedback shapes the focus of trainee
attention. The addition of velocity feedback did increase the amount of time trainees
spent attending to other feedback. However, because the velocity feedback did not match

the focus of the mastery feedback, it may have distracted trainees or forced them to

133

refocus their attention. It is conceivable that trainees who received velocity feedback in
this study may have adopted more of a performance orientation than a mastery
orientation, although the data from this study do not allow this possibility to be uniquely
addressed.

In considering the declines in efﬁcacy and self-regulation observed in the study, it
may be that designing modular training programs, in which each set of goals is addressed
in a single module, may prevent such declines. The data indicate that within each block
of three trials, trainee performance rose. However, no Opportunity for closure was
offered to trainees, and this may have been discouraging (made them feel as though they
did not accomplish their goals before moving on, which could have affected efﬁcacy
judgments) and distracting (having to shift attention from one element of the task to
another without understanding whether the ﬁrst element has been understood or
completed properly could affect self-regulatory activity). Well-deﬁned modules might
allow for this closure, and thus prevent declines in efﬁcacy and self-regulation by
providing distinct experiences when different goals are being addressed over the course
of training.

The study also has implications for training evaluation. The situational judgment
test designed for this study, which assessed adaptive knowledge, is a step toward
developing a paper-and-pencil assessment measure which will allow us to approximate
what trainees are going to be required to do on the job without actually having to send
them into a different environment. Through further iterations of this technology, we
should be able to develop assessment tools that will accurately predict whether our

training programs will provide trainees with the ability to utilize the knowledge and skills

134

they have gained on the job, even when the task changes or becomes ambiguous. The
results of this study are encouraging; the situational judgment test of adaptive knowledge
was able to signiﬁcantly predict strategic generalization performance, even when
controlling for cognitive ability and declarative knowledge.

Limitations and Implications

In terms of limitations and problems with the study, these can be summarized
under several headings. First, the assessment of self-efﬁcacy may not have captured all
of the relevant variance in the construct, based on the model. More speciﬁcally, multiple
types of efﬁcacy might need to be assessed to adequately test the theoretical model.
Second, while time appears to have been a critical variable, it was not an hypothesized
portion of the model. Third, the two measures of self—regulation functioned differently
with respect to the model, leading to questions as to whether they measured the same
construct. Fourth, the velocity feedback manipulation seemed to distract trainees from
the training goals they were assigned. Fifth, the practice manipulation may have been too
weak to obtain the desired effects, or the time frame might have been too narrow.
Finally, the sequencing of the goals may have affected the development of self-efﬁcacy
and self-regulatory activity. Each of these issues will be discussed in turn.

An initial concern raised by an examination of the results and the model centers
around the assessment of self-efﬁcacy. The efﬁcacy assessment was designed to predict
performance on the task, with emphasis on generalization of knowledge and skills to the
more difﬁcult version of the task. In the theoretical model, only indirect links from self-
efﬁcacy to generalization performance were included. Moreover, direct links were

included from self-efﬁcacy to the two knowledge measures. While the knowledge

135

measures were critical variables (and efﬁcacy did in fact predict declarative knowledge),
a question that must be asked is why self-efﬁcacy did not predict adaptive knowledge as
well. In this case, I would argue that the motivational variable, self-efﬁcacy, may simply
have not been predictive of knowledge. Many things inﬂuence the development of
knowledge, particularly complex knowledge as assessed by the adaptive knowledge
measure, and while motivation may be important, it may not be the primary force that
allows such knowledge to develop. The implication to be drawn from this centers around
the possibility that we might need to expand our thinking to include other factors which
might be relevant to the development of knowledge, in terms of experience, learning
style, and other potentially important factors.

Second, the analyses of hypotheses 1-4 centered around time-based effects. In
any study of training that involves learning or skill development (which, ideally should
be m study of training), time may be an important variable. People M change, over
the course of training. However, while the analyses reﬂect attention to time, the
hypotheses for this study did not. To some extent, a focus on outcomes of training is
appropriate. However, the model for this study included process components, which
implied change over time. This should have been included in the hypotheses. The data
collected and analyses conducted did allow for the critical time-based questions to be
asked, however, so while the initial conceptualization of the study did not focus on time,
the ﬁnal product did take temporal factors into account. Considering implications of this
ﬁnding, it is clear that change over time is an area training research should consider in
more detail. Examining outcomes is critical, but if we do not understand how those

outcomes are obtained, if we cannot grasp the process, then we will fall short of

136

understanding how training operates.

A third critical point is that the two assessments of self-regulation — the scale-
based measure and the diary measure — functioned differently, with respect to the model.
The self-regulation diary, in which trainees were asked to self-report the goals they had
adopted during training, was predicted by both manipulations, although the feedback
effect was reversed here, as everywhere else, such thatthe effect favored individuals who
did not receive velocity feedback over those who did. No main effects for either
manipulation were obtained on the scale-based measure of self-regulation. The diary was
also predictive of strategic performance at the end of training, an hypothesized effect that
did not emerge with only the scale-based measure of self-regulation included in the
model.

While the inconsistency of measurement of what should ostensibly be the same
construct across two measures ﬁrst appears to be a limitation to the study, however, it
may be one of the more interesting ﬁndings to emerge. A critical difference between the
scale-based measure of self-regulation and the diary is the extent to which the two types
of assessments require differential amounts of reﬂection. Little reﬂection on the task is
required for the scale-based measure, which simply asks trainees to bubble in the
appropriate circle to represent how much they did or did not do certain things. Bubbling
is a low-cognition activity. The diary, however, asked trainees to reﬂect on the previous
trial and self-report what goals they were attempting to obtain during that trial. Clearly,
more reﬂection is required by one method of assessing self-regulation than the other, and
we should not be surprised that the diary, which required more reﬂection (and therefore

more self-regulation) provided an assessment of self-regulatory activity more in line with

137

what should, theoretically, have been obtained.

Fourth, the effects of the velocity feedback manipulation seem to have involved
interference with the learning goals, rather than augmentation. Based on the data
collected, it appears that the presentation of velocity feedback (1) encouraged trainees to
pay more attention to all of their feedback, and (2) created a cognitive framework for that
feedback Iwhich may have been distinct from the framework the training goals attempted
to create. The ﬁrst point is based on the ﬁnding that individuals who received velocity
feedback spent more time reviewing the non-velocity screens than did individuals who
did not receive velocity feedback. The second is based on the ﬁnding that while the
trainees spent more time reviewing the critical information, which should have helped
them perform better on the generalization trial (from a strategic standpoint), they did not
actually demonstrate this improved performance.

Again, this is a mixed blessing. The distraction effect was not hypothesized, but
the ﬁnding that providing velocity feedback increased trainee attention to the non-
velocity feedback is potentially critical. Thus, this ﬁnding has implications for training
design, since in scenarios where the velocity feedback is not of a nature which tends to
oppose the remainder of the feedback (e. g., performance-based velocity feedback
distracting trainees from feedback linked to mastery goals), the increased attention to the
remainder of the feedback screens should be valuable.

The practice manipulation appears to have suffered from a different problem, that
of being overly weak, or conducted in a time frame which did not allow its effects to
emerge. Examining the empirical model generated at the end of my data analyses (Figure

5), it is clear that the practice manipulation did almost nothing. It predicted basic

138

performance during generalization, but nothing else aside from the interaction with time
in predicting self-regulatory activity. Examining the effects of the practice manipulation
on self-regulation over time, however, demonstrates that the individuals receiving
variable practice were on the upswing, with respect to self-regulatory activity, at the end
of training. It is possible that given a longer experiment effects for the practice
manipulation would have emerged. The manipulation may have been too weak to
generate noticeable effects in the time allotted. The process discussed in the literature by
which variable practice should beneﬁt learners is one of abstraction of critical elements
of the task based on repeated exposure. With only ten practice trials, it may be that the
task was too complex for all of the elements to be meaningfully abstracted in three and
one-half hours.

In terms of implications to be drawn from this ﬁnding, the value of variable
practice may have been overstated in the literature, particularly with respect to complex
tasks. An experiment demonstrating that practice variability interacts with task
complexity to determine outcomes would go a long way toward clarifying whether the
recommendations of Schmidt and Bjork (1992) are meaningful when we consider the
kinds of complex cognitive tasks that increasingly occupy our training programs.

Finally, it is possible that the sequencing of goals in the study may have affected
the development of self-efﬁcacy and self-regulation. The goals began relatively simple,
then progressed to more complex goals. However, it may be that in this experiment, the
requirements for achieving the initial goals were overly simple, and made the later goals
appear more difﬁcult by comparison. Previous studies utilizing this paradigm may have

avoided this effect through the use of more cues (5 per decision) or ambiguous cues. The

139

current study, with 3 cues and no ambiguity, may have been overly simple, creating
larger goal-performance discrepancies than anticipated, and thereby leading to declines in
the self-efﬁcacy of trainees, as well as self-regulatory activity. This is not to say that
sequencing goals is inappropriate from a training standpoint. The lesson to be taken
away from this ﬁnding is one which was warned about by Hesketh (1997); if the initial
goals of training are too simple to achieve, then we should not be surprised if trainee
motivation or efﬁcacy decline when the goals become more complex.

With several limitations of the study (as well as related implications) discussed, 1
would now like to turn my attention to potential directions for future research that have
presented themselves, in addition to those I have noted throughout my discussion.
Directions for Future Research

Several opportunities for future research present themselves, based on the
ﬁndings of this study. These deal with further exploration of some of the unexpected
ﬁndings, as well as an expansion of the measurement techniques developed for this
research.

One of the most interesting elements of the research was the fact that while
performance increased over the course of training, in a manner which approximated a
standard learning curve, self-efﬁcacy decreased from the beginning of training to the end
of training. "The speciﬁc causal factors for the decline in efﬁcacy, however, cannot be
uniquely determined via the data collected for this study. Several possibilities presented
themselves, and were discussed in the previous section. However, they remain
speculation, and studies must be conducted to determine the circumstances under which

trainee self-efﬁcacy will decrease, in spite of overall gains in learning and skill

140

development. Such motivational declines are an enormous problem, and research is
necessary to allow us to predict and prevent their occurrence.

Related to the decline in self-efﬁcacy is the nature of the velocity feedback
manipulation utilized in this study. The manipulation included two components. One
was simply the presentation of score feedback, along with the components of score,
which allowed trainees a single piece of information they could use to track their progress
over the course of training. The second component was the estimate each participant was
asked to make with respect to how much progress s/he had seen on the game from one
trial to the next. The decline in efﬁcacy my have been tied more to one of these
components than to the other, and I’m interested in exploring their relative importance to
the observed declines in efﬁcacy in a study which is better able to decompose the effects.

Along the same lines, a reasonable question is what effect it has to make velocity
feedback explicit, rather than implicit. When trainees are given only score feedback, we
must conclude that they are attending to it and noting change over time (implicit velocity
feedback), whereas when they are asked to make an estimate of their progress following
every trial, the velocity (change over time) is made explicit. We might argue that the
effects of implicit vs.‘explicit velocity feedback may depend on the task and the nature of
the feedback itself. That is, in the current study, it might be the case that because of the
nature of the feedback and the learning curve involved, participants might have done
better had the velocity feedback been implicit, rather than being asked to estimate their
progress following every trial. As it stands, participants perceived a great deal of
improvement early in training, but less improvement later on, based on the data collected.

The complexity of the task seems like a potential moderator variable. A possible study

141

might be a 2 x 2, in which task complexity (simple vs. complex) was crossed with
implicit vs. explicit velocity feedback. In both conditions, a questionnaire would be
given at the conclusion of the experiment to probe the extent to which trainees believed
themselves to be improving over time. Complexity could be manipulated through
information processing requirements, with low complexity involving three-cue decisions
with no cue ambiguity, and high complexity involving ﬁve-cue decisions with ambiguous
information. One hypothesis might be that explicit velocity feedback would be harmful
in conditions where the task is relatively simple (the three cue, no ambiguity condition),
as trainees may plateau relatively quickly and cease to observe improvement, whereas
such explicit velocity feedback would be beneﬁcial throughout training to individuals in
the more complex condition, where improvement should be continuous and should not
plateau over the course of training because their learning would not reach its maximum
before training ended. The dual nature of the velocity manipulation must be decomposed
for a better understanding of the effects observed in this study to be obtained.

As was noted in the previous section, the practice variability manipulation did not
produce results isomorphic with those observed in the psychological literature. However,
the existing literature deals with relatively simple motor tasks, and has not dealt to a great
extent with complex cognitive tasks like that utilized in this study. Two possibilities
must by explored, at the very least, with respect to this ﬁnding. First, are the lack of
ﬁndings in this study the result of insufﬁcient variability in the task? This is relatively
unlikely, as the appearance and requirements of the task shifted markedly over the nine
training trials, with different numbers of targets initially visible at different stages in

training, and different arrangements of targets on the radar screen. Studies may need to

142

 

be conducted to determine if degree of variability in complex task practice is potentially
causal in schema development.

Perhaps more likely is the argument presented earlier in the discussion, that the
schema development that seems to underlie the observed effects in the literature takes
longer for more complex cognitive tasks, and may not be able to occur sufﬁciently in
relatively short experiments dealing with cOmple'x cognitive tasks like
TEAMS/TANDEM. To determine if it is a question of task complexity interacting with
experimental duration, simpler experiments should be designed within this paradigm,
which can be accomplished in a shorter time frame, to determine if results similar to
those obtained in the motor learning literature can be obtained for cognitive tasks; if so, it
may reasonable to assert that for more complex tasks, longer-term training programs
should lead to similar types of outcomes, in terms of schema development and,
ultimately, increased adaptability and expertise.

Finally, from a measurement perspective, it is important that further research be
conducted to explore the potential utility of situational judgment-type measures of
adaptive knowledge as a means to predict adaptability and generalization of skills from
training back to the job. The results from this study are encouraging, but in no means
ﬁnal. While the SJT was able to out-predict cognitive ability and declarative knowledge,
it was not able to signiﬁcantly predict strategic generalization performance beyond
strategic performance at the end of training. This is a long-term goal, and will require
reﬁnement of the measurement system as well as a more thorough understanding of how
adaptability at the end of training may differ from adaptability on the job. An

understanding of how we may best assess these differences, from a measurement

143

standpoint is also necessary.
Conclusion

While the results of the study were mixed, and the model did not prove to be the
best possible representation of the data, several interesting ﬁndings emerged. The
manipulations —- practice variability and velocity feedback — did affect the processes they
were hypothesized to affect, if not in precisely the manner predicted. Self~regulation
(speciﬁcally, self-monitoring multiple aspects of the training environment) was shown to
have its primary effects on cognitive/ learning outcomes, and both proximal and distal
outcomes of training were able to be predicted relatively well with the study variables.

However, a great deal of work remains to be done. I have outlined several
directions I believe represent fruitful lines of research. Task complexity and learning
issues make laboratory studies an ideal medium for continuing to explore the concerns
raised in this study, while not limiting my ability to utilize data from outside sources as
well, as it becomes available. The issues central to this dissertation — training design,
training evaluation, and a better understanding of the cognitive processes that form the
foundation for learning and skill development — are ones which provide a myriad of
opportunities for exploration, and can reasonably be expected to be more than sufﬁcient

to sustain a stream of research for many years to come.

144

REFERENCES

145

REFERENCES

Ackerman, P. L. (1992). Predicting individual differences in complex skill
acquisition: Dynamics of ability determinants. MW, 598-
614.

Acton, W. H., Johnson, P. J ., & Goldsmith, T. E. (1994). Structural knowledge
assessment: Comparison of referent structures. Journal of Educational Psychology, 86,
303-311.

Ames, C. & Archer, J. (1988). Achievement goals in the classroom: Students'
learning strategies and motivation processes. Journal of Educational Paychology, 80,
260-267.

Anderson, J. R. (1982). Acquisition of cognitive skill. Psychological Review,
82, 369-406. .

Anderson, J. R. (1983). The architecture of cogpition. Cambridge, MA:
Harvard University Press.

Anderson, J. R. (1987). Skill acquisition: Compilation of weak-method problem
solutions. Psychological Review. 94. 192-210.

Baldwin, T. T., & Ford, J. K. (1988). Transfer of training: A review and
directions for future research. Personnel Psychology, 41, 63-105.

Bandura, A. (1986). Social foundations of thought and action: A social
cognitive theory. Englewood Cliffs, NJ: Prentice-Hall.

Bandura, A. (1991). Social cognitive theory of self-regulation. Organizational

Behaviognd Human Decision Processes, 50, 248-287.

146

Bandura, A., & Wood, R. (1989). Effects of perceived controllability and
performance standards on self-regulation of complex decision making. Journal of
Personality and Social Psyghplogy, 56, 805-814.

Bouffard, T., Boisvert, J., Vezeau, C., & Larouche, C. (1995). The impact of goal
orientation on self-regulation and performance among college students. British Joumal of

Educational chhology, 65, 317-329.
Brown, K. G., Mullins, M., Weissbein, D., Toney, R., & Kozlowski, S. W. J.

(1997, April). Mastery goals and strategic reﬂection: Preliminag evidence for
interference. In S. W. J. Kozlowski (Chair) and M. Quinones and J. Martocchio
(Discussants) Symposium "MetacOgnition and Training," presented at the Twelfth
Annual Meeting of the Society of Industrial and Organizational Psychology, St. Louis,
MO.

Button, S. B., Mathieu, J. E., & Zajac, D. M. (1996). Goal orientation in
organizational research: A conceptual and empirical foundation. Organizational
Bahagior and Humyt Decision Processgﬂ, 26-48.

Carlson, R. A., Sullivan, M. A., & Schneider, W. (1989). Component ﬂuency in

a problem-solving Context. Human Factors 3], 489-502.

 

Carver, C. S., & Scheier, M. F. (1982). Control theory: A useﬁrl conceptual
framework for personality-social, clinical, and health psychology. Psychological Bulletip,
a, 111-135.

Carver, C. S., & Scheier, M. F. (1990). Origins and ﬁrnctions of positive and

negative affect: A control-process view. Psychological Bulletin, 97, 19-35.

147

Carver, C. S., & Scheier, M. F. (1998). On the self-regu_|ation of behayior. New
York: Cambridge University Press.

Catalano, J. F., & Kleiner, B. M. ( 1984). Distant transfer in coincident timing as
a function of variability of practice. Perceptual and Motor Skills, 58, 851-856.

Chan, D., & Schmitt, N. (1997). Video-based versus paper-and-pencil method of
assessment in situational judgment tests: Subgroup differences in test performance and
face validity perceptions. Journal of Applied Psychology, 82, 143-159.

Chi, M. T. H., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989).
Self-explanations: How students study and use examples in learning to solve problems.
Cognitive Science, 13, 145-182.

Clifton, M. A. (1985). Practice variability and children's motor behavior.
Perceptual and Motgr Skills, 60, 471-476.

Cohen, J. J. (1988). Statistigal ppwer aaalysis for tha behavioral agiences.
Hillsdale, NJ: LEA.

Collins, A. M., & Loﬁus, E. F. (1975). A spreading activation theory of semantic
processing; Psychological Review, 82, 407-428.

Curtis, K. A (1992). Altering beliefs about the importance of strategy: An

attributional intervention. Journal of Applied Social Paychology, 22, 953-972.

Dalessio, A. T. (1994). Predicting insurance agent turnover using a video-based

situational judgment test. Journal of Businass and Psychology, 9, 23-32.
Dansereau, D. F., Brooks, L. W., Holley, C. D., & Collins, K. W. (1983).

Learning strategies training: Effects of sequencing. qumal of Experimental Edagtiop,

5_1, 102—108.

148

Duda, J. L., & Nicholls, J. G. (1992). Dimensions of achievement motivation in
schoolwork and sport. WM 290-299.

Dweck, C. S., & Leggett, E. L. (1988). A social-cognitive approach to motivation
and personality. Psychological Review. 95. 256-273.

Earley, P. C., Lee, C., & Hanson, A. L. (1990). Joint moderating effects of job
experience and task component complexity: Relations among goal setting, task
strategies, and performance. Journal of Organizational Behavior, 11, 3-15.

Earley, P. C., Northcraﬁ, G. B., Lee, C., & Lituchy, T. R. (1990). Impact of
process and outcome feedback on the relation of goal setting to task performance.
Academy of Management Jouma_l,_};, 87-105.

Elliott, E. S., & Dweck, C. S. (1988). Goals: An approach to motivation and
achievement. Journal of Personalig and Social Pachology, 54, 5—12.

Etelapelto, A (1993). Metacognition and the expertise of computer program
comprehension. Scandinavian Jourpal of Educational Research. 31. 243-254.

Flavell, J. H. (1979). Metacognition and cognitive monitoring: A new area of
cognitive-developmental inquiry. American Psychology, 34, 906-911.

Ford, J. K. (1997). Advances in training research and practice: An historical
perspective. In J. K. Ford, S. W. J. Kozlowski, K. Kraiger, E. Salas, & M. Teachout
(Eds), Imprpving training effectiveness in work organizations (pp. 1-18). Mahwah, NJ:
LEA.

Ford, J. K., Smith, E. M., Weissbein, D. A., Gully, S. M., & Salas, E. (1998).
Relationships of goal orientation, metacognitive activity, and practice strategies with

leaming outcomes and transfer. Journal of Applied Psychology, 83, 218-233.

149

Forsterling, F. (1985). Attributional retraining: A review. Psychological Bulletin,
9_8,, 495-512.

Frese, M., & Altmann, A. (1989). The treatment of errors in learning and training.
In L. Bainbridge & S. A. R. Quintanilla (Eds), Developing skills with information
technology (pp. 658-682). New York: Wiley.

Garcia, T. & Pintrich, P. R (1994). Regulating motivation and cognition in the

classroom: The role of self-schemas and self-regulatory strategies. In D. H. Schunk & B. J.

 

Zimmerman (Eds), Self-regplation of learning and performance: Issues apd educational

applicatipns. New Jersey: Erlbaum.

Gay, G. (1986). Interaction of learner control and prior understanding in
computer-assisted video instruction. Journal of Educational Psychology, 78, 225-227.

Gist, M. E., & Mitchell, T. R. (1992). Self-Efficacy: A theoretical analysis of its
determinants and malleability. Academy of Management Review, 17, 183-211.

Gist, M. E., Schwoerer, C., & Rosen, B. (1989). Effects of alternative training
methods on self-efficacy and performance in computer software training. Journal of
Applied Psychology, 74, 884-891.

. Glaser, R., & Bassok, M. (1989). Learning theory and the study of instruction.

Annual Review of Psychology, 40, 631-666.

 

Goldsmith, T. E., & Kraiger, K. (1997). Application of structural knowledge
assessment to training evaluation. In J. K. Ford and associates (Eds), Improving training
effectiveness in work organizations (pp. 73-96). Mahwah, NJ: LEA.

Green, D. P., Whitehead, J., & Sugden, D. A. (1995). Practice variability and

transfer of a racket skill. Perceptual and Motor Skills. 81. 1275-1281.

 

150

Hastie, R. (1984). Causes and effects of causal attribution. Journal of Personality

and Social Psychology, 46, 44-56.

Hatano, G., & Inagaki, K. (1986). Two courses of expertise. In H. Stevenson, H.
Azuma, & K. Hakuta (Eds), Child development and education in Japan. New York: W.
H. Freeman and Company.

Hesketh, B. (1997). Dilemmas in training for transfer and retention. 59mg
Psychology: An Intemationﬂeview. 46. 317-386.

Hollenbeck, J. R., Williams, C. R., & Klein, H. J. (1989). An empirical
examination of the antecedents of commitment to difﬁcult goals. Journal of Applied
Psychology, 74, 18-23.

Holyoak, K. J. (1991). Symbolic connectionism: Toward third generation
theories. In K. A. Ericsson & J. Smith (Eds), Toward a general theogz of expertise:
Prospects and limits (pp. 301-336). New York: Cambridge University Press.

Holyoak, K. J., & Spellman, B. A. (1993). Thinking. Annual Review of
Psychology, 44, 265-315.

Hsee, C. K., & Abelson, R. P. (1991). Velocity relation: Satisfaction as a function
of the ﬁrst derivative of outcome over time. Journal of Personaligand Social
Psychology, 60, 341-347.

Huber, J. (1997). Differences in problem representation and procedural
knowledge between elite and nonelite springboard divers. The Sport Psychologjg, 11,
142-159.

Ivancic, K., & Hesketh, B. (1995). Making the best of errors during training.

Training Research Journal. 1. 103-125.

151

Johnson, R., & McCabe, J. (1982). Schema theory: A test of the hypothesis,
variation in practice. Perceptual and Motor Skills, 55, 231-234.

Kanfer, R., & Ackerman, P. L. (1989). Motivation and cognitive abilities: An
integrative/aptitude-treatment interaction approach to skill acquisition [Monograph].

Journal of Applied Psychology, 74, 657-690.

Karoly, P. (1993). Mechanisms of self-regulation: A systems view. Annual
Raviaw of Psychology, 44, 23-52.
Kelly, H. B. (1982). A primer on transfer of training. Training and Development

Journal 36 102-106.

 

Kerr, R. (1982a). Practice variability: Abstraction or interference. Pergptpal

and Motor Skills 54, 219-224.

 

Kerr, R. (1982b). Practice variability and longer and shorter retention intervals.
Pergeptual and Motor Skills, 54, 243-250.

Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on
performance: A historical review, a meta-analysis, and a preliminary feedback
intervention theory. Pachological Review. 119. 254-284.

Kozar, B., Vaughn, R. E., Lord, R. H., & Whitﬁeld, K. E. (1996). Basketball
free-throw performance: Practice implications. Journal of Sport Behavior, 18, 123-129.

Kozlowski, S. W. J. (1998). Training and developing adaptive teams: Theory,
principles, and research. In J. A. Cannon-Bowers & E. Salas (Eds), Decision making
under stress: Implications for training and simalation. Washington, D. C.: APA.

Kozlowski, S. W. J., Gully, S. M., Smith, E. M., Brown, K. G., Mullins, M. E., &

Williams, A. E. (1996, April). Sequenced masteg goals and advance orgrarizers:

152

 

Enhancing the effects pf practice. In K. Smith-Jentsch (Chair), When, how, and why
does practice make perfect? Symposium conducted at the 11th Annual Conference of the
Society for Industrial and Organizational Psychology, San Diego, CA.

Kozlowski, S. W. J., Mullins, M. E., Toney, R. J ., & Bell, B. S. (in preparation).

Adaptive Performance: Mastegy vs. Performgce Goals and Feedback Consistency.

 

Kozlowski, S. W .J., Toney, R. J., Mullins, M. E., Weissbein, D. A., Brown, K.
G., & Bell, B. S. (in press). Developing Adaptability: A Theory for the Design of
Integrated-Embedded Training Systems. In E. Salas (Ed), Human/technology interaction
in complex ystems (Vol. 10). Greenwich, CT: JAI Press.

Kozlowski, S. W. J ., Toney, R. J ., Weissbein,-D. A., Mullins, M. E., Brown,
K. G., & Bell, B. (1998, April). Training adaptive performance. In S. W. J. Kozlowski,
S. Parker, & M. Frese (Chairs) and S. J. Motowidlo (Discussant) symposium "Beyond
task performance: Proactivity and learning, " presented at the Thirteenth Annual Meeting
of the Society for Industrial and Organizational Psychology, Dallas, TX.

Kozlowski, S. W. J ., Weissbein, D. A., Brown, K. G., Toney, R. J ., & Mullins, M.

E. (1997, May). Guiding the development of deployable shipboard training systema:

Enhanping akill apguisitipn, adaptabilig, and effectiveness (Final Report 1.1; Contract
No. N61339-96-K-0005). Michigan State University, East Lansing, MI.

Latham, G. P., & Locke, E. A. (1991). Self-regulation through goal setting.

Organizational Behavior zmd Human Decision Processes 50 212-247.

 

Lindsley, D. H., Brass, D. J., & Thomas, J. B. (1995). Efﬁcacy-perforrnance
spirals: A multilevel perspective. Academy of Management Review. 20. 645-678.

Locke, E. A., & Latham, G. P. (1990). A theory of gpal setting & task

153

performance. Englewood Cliffs, NJ: Prentice-Hall.
Lord, R. G., & Levy, P. E. (1994). Moving from cognition to action: A control

theory perspective. Applied Psychology: M International Revieva 43. 335-367.
Martocchio, J. J ., & Dulebohn, J. (1994). Performance feedback effects in

 

training: The role of perceived controllability. Personnel Psychology, 47, 357-373.

Meece, J. L. (1994). The role of motivation in self-regulated learning. In D. H.
Schunk & B. J. Zimmerman (Eds), Self-regplation of learning and performance: Issues
and educational applications (pp. 25-44). New Jersey, Erlbaum.

Meece, J. L., Blumenfeld, P. C., & Hoyle, RH. (1988). Students' goal orientations
and cognitive engagement in classroom activities. Journal pf Educational Pachology, 80,
514-523.

Miller, R. B., Behrens, J. T., Greene, B. A., & Newman, D. (1993). Goals and
perceived ability: Impact on student valuing, self-regulation, and persistence.
Contemporag Educational Psychology, 18, 2-14.

Mitchell, T. R., Hopper, H., Daniels, D., George-Falvy, J ., & James, L. R.
(1994). Predicting self-efﬁcacy and performance during skill acquisition. Journal pf
Applied Psychology, 79, 506-517.

Motowidlo, S. J., Dunnette, M. D., & Carter, G. W. (1990). An alternative
selection procedure: The low-ﬁdelity simulation. Journal of Applied Paychology, 6,
640-647.

Motowidlo, S. J., & Tippins, N. ( 1993). Further studies of the low-ﬁdelity

simulation in the form of a situational inventory. qumal of Qcpupational and

.Qrganizational Psychology, 66, 337-344.

154

Mullins, M. E., Kozlowski, S. W. J ., Schmitt, N., & Howell, A. E. (in
preparation). Predicting the use of the Internet in businasL An appliaation of innomn
um. *

Mullins, M. E., Kozlowski, S. W. J., Toney, R. J., Bell, B. S., Brown, K. G., &
Weissbein, D. A. (1999, April). Adaptive performance: Maatery versus performgpa
goals and faedback ppnsistengy. Paper presented at the Fourteenth Annual Conference of
the Society for Industrial and Organizational Psychology, Atlanta, GA.

Nelson, TO. (1996). Consciousness and metacognition. American Pachologist,
5_l, 102-116.

Nelson, T. O., Dunlosky, J ., Graf, A., & Narens, L. (1994). Utilization of
metacognitive judgments in the allocation of study during multitrial learning.
Psychological Science. 5. 207-213.

Ostroff, C. (1991). Training effectiveness measures and scoring schemes: A
comparison. Personnel Psychology, 44, 353-374.

Paas, F. G. W. C., & van Merrienboer, J. G. (1994). Variability of worked
examples and transfer of problem-solving skills: A cognitive load approach. Journal of
Educational Psychology, 86, 122-133.

Pintrich, PR & DeGroot, EV. (1990). Motivational and self-regulated learning
components of classroom academic performance. Journal of Educational chhology, 82,
33-40.

Pintrich, ‘P. R, Smith, D. A., Garcia, T., & McKeachie, W. J. (1993). Reliability
and predictive validity of the Motivated Strategies for Learning Questionnaire (MSLQ).

Educzaional apd Psychologiaal Measurement. 53. 801-813.

155

Posner, M. I. (1988). Introduction: What is it to be an expert? In M. T. H. Chi,
R. Glaser, & M. J. Farr (Eds), Th9 namrg pf axpegise (pp. xxix-xxxvi). Hillsdale, NJ:
LEA.

Ren, J., & Sheridan, T. B. (1995). An adaptive decision aid for real
environments. IEEE Transactions on Syatems, Man, and Cybernetics, 25, 1384-1391.

Ridley, s. 1)., Schutz, P. A, Glanz, R s, & Weinstein, c. E. (1992). Self-regulated
learning: The interactive inﬂuence of metacognitive awareness and goal-setting. Journal of

ExperimentaJ Educatiop, 60, 293-306.
Saks, A. M. (1997). Transfer of training and self-efficacy: What is the dilemma?

Applied Psychology: An International Review. 46. 365-3 70.

 

Schmidt, R. A. (1975). A schema theory of discrete motor skill learning.
Psychological Review, 82, 225-260.

Schmidt, R. A. (1976). The schema as a solution to some persistent problems in
motor learning theory. In G. E. Stelmach (Ed.), Motor cpntrol: Issues and trends (pp. 41-
65). New York: Academic Press.

Schmidt, R. A., & Bjork, R. A. (1992). New conceptualizations of practice:
Common principles in three paradigms suggest new concepts for training. Psycholpgjcal

Science 3 207-217.

 

Schunk, D. H. (1994). Self-regulation of self-efﬁcacy and attributions in academic

settings. In D. H. Schunk & B. J. Zimmerman (Eds), Self-ragplation of leaming and

parfpnnance: Issues and educational applications (pp. 75-99). New Jersey: Erlbaum.
Smith, E. M. (1996). Unpublished Doctoral Dissertation, Michigan State

University, East Lansing, MI.

156

Smith, E. M., Ford, J. K., & Kozlowski, S. W. J. (1997). Building adaptive
expertise: Implications for training design strategies. In M. A. Quinones and A.
Ehrenstein (Eds), Training for a rapidly changing workplace: Applications of

paychological rasearch (pp. 89-118). Washington, DC: APA.

Tennyson, R. D. (1980). Instructional control strategies and content structures as
design variables in concept acquisition using computer-based instruction. Journal of
Educational Psychology, 72, 225-232.

Tennyson, R. D. (1981). Use of adaptive information for advisement in learning
concepts and rules using computer assisted instruction. American Educatipnal Research

Journal 18 425-438.

 

Tesluk, P. E., Farr, J. L., Mathieu, J. E., & Vance, R. J. (1995). Generalization of
employee involvement training to the job setting: Individual and situational effects.
Parsonnel Psychology, 48, 607-632.

Trotter, R. J. (1986, July). The mystery of mastery. Psychology Today, 32-3 8.

Waldersee, R., & Luthans, F. (1994). The impact of positive and corrective
feedback on customer service performance. Joumal of Organizational Behavior, 15, 83-
95.

Warr, P., & Bunce, D. (1995). Trainee characteristics and the outcomes of open
learning. Personnel Psychology, 48, 347-375.

Weaver, J. L., Bowers, C.A., Salas, E., & Cannon-Bowers, J .A. (1995).

Networked simulations: New paradigms for team performance research. Behavioral

Rasearch Methods, Instruments, & Computers, ﬂ, 12-24.

157

Weaver, J. L., Morgan, 3., B., Hall, J., Compton, D. (1993). Team decision

making in the command information penter: Development of a low-ﬁdelity team decisipp

 

making task for assessing the effects of teamwork stressors. Orlando, FL: Naval Training

Systems Center, Human Factors Division.

Wenger, E. (1987). Artiﬁcial intelligence and tutoring systems: Computational
m1 cognitive mches to the communicaﬂm of knowledga Los Altos, CA: Morgan
Kaufman.

Wernimont, P. F., & Campbell, J. P. (1968). Signs, samples, and criteria.
Journal of Applied Psychology, 52, 372-376.

Wood, R. E. (1986). Task complexity: Deﬁnition of the construct.

Organizational Behaviomd Human Decision Processes 37, 60-82.

 

158

APPENDICES

159

Appendix A

Participant #
Informed Consent Tactical Action Game (TAG)

The study in which you are about to participate investigates your performance on the
Tactical Action Game (TAG). TAG is a computer simulated, radar tracking task in
which you will measure the attributes of contacts that appear on your radar screenand
decide what action should be taken. You will also be asked to answer questionnaires
which will help us understand how you learn to perform the game. It is important that
you answer the questions as honestly and accurately as you can.

Your participation in this study requires three and one-half hours of your time. You will
receive 7 Psychology subject pool credits for this 3.5 hour time commitment. No
possible risks or discomforts are anticipated as a result of this study.

Awards available for this study are explained in detail in a separate handout. Winners
will be determined at the conclusion of this study. If you win, you will be contacted at
the address and phone number you indicate below within 3 weeks of the study's
conclusion. Instructions for claiming the award through the Department of Psychology
Business Ofﬁce will be provided when you are contacted.

Participation is this study is completely voluntary. You are free to discontinue the study
at any time for any reason without penalty. Simply inform the investigator if you wish to
withdraw. Your responses will be completely conﬁdential. During the study we will ask
you to put your name on a computer scoring form. This is only to be used to identify and
contact you should you win an award. Your name will not be associated with your
responses and will be kept conﬁdential.

You are free to ask any questions you might have about this study at any time. At the
end of your involvement, you will be provided with feedback explaining the purpose of
this research inmore detail. You may ask about the results of the study when it is
complete by contacting any of the investigators. Their names and phone numbers are
listed on the debrieﬁng form you will receive at the conclusion of your involvement. If
you have any questions about your participation in this study, ask the investigator before
you indicate your consent to voluntarily participate by signing below.

I have been fully informed about the above-described study and its possible risks. 1 give
permission for my participation in this study. I know that the investigator and his/her
associates will be available to answer any questions I may have. I understand that I am
ﬂee to withdraw this consent and discontinue participation in this study at any time

 

 

 

without penalty.

Date

Print Name Signature
Address Phone

 

 

I60

Appendix B

Debrief'rng Form
Tactical Action Game (TAG)

The study in which you just participated was designed how training design elements act
to produce the kinds of outcomes valued by organizations conducting training in the real
world. During this study, you operated the TAG radar simulation. TAG simulates the
complex physical performance, information processing, and decision-making demands of
real teams performing fast-paced, critical tasks. To perform the TAG simulation you
needed to learn how to operate the task and develop strategies for effective task
performance. TAG required you to gather information about objects on the screen, make
decisions, and take actions based on the information you gathered. We will use this
information gathered during the study to link your performance on the task to your
knowledge of the task.

If you have any questions about this study or would like to receive a copy of the results
when they are complete, please notify the investigator now. If, in the future, you have
any questions about the study or would like to receive the results when they are complete,
please call one of the investigators listed below. Finally, thank you for participating in
this study. We tried to make it as interesting for you as possible, and we are open to
suggestions. So if you think we can improve this study in any way, please feel free to
talk to us now or call us in the future.

Investigator; Morrie Mullins 353-9166

161

Appendix C: Award Instructions

In this experiment you will be operating a complex computer task and answering
questions about your experiences. We will award a total of eight cash prizes to
the best TAG players. Two categories of proﬁciency will be recognized, and four
awards will be offered in each category:

Two 1st place prizes at $50 each
Two 2nd place prizes at $40 each
Two 3rd place prizes at $30 each
Two 4th place prizes at $20 each

Each set of four awards will be made for different types of game proﬁciency:

One pair of awards will be made for the players who understand the game the best
during the three practice sessions. You will answer several questionnaires about
the game during and after the practice sessions. You need to think carefully about
your answers to the questions. The top players who answer these questions the
best will be awarded lst, 2nd, 3rd, and 4th Place Prizes.

The second pair of awards will be made for the players who do the best on the
ﬁnal TAG session that will take place at the end of the experiment. The top
players who do the best will be awarded 1st, 2nd, 3rd, and 4th Place Prizes. We
let you know when the ﬁnal session is about to begin.

The Prizes are independent of each other. That means its possible to be awarded
prizes in both categories of proﬁciency. Your chances of getting an award are
pretty good -- it's all up to you.

Open your folder and take out the Tactical Action Game Description and the
Consent Form. Please read both sheets. If you agree to participate in our study,
please sign and date the consent form and return it to the folder. I’ll be around to
collect the folders later in the experiment.

162

Appendix D: Feedback Screens

FEEDBACK: FIRST AND LAST SCREENS

You will now have an opportunity to review your activity during the
practice period you just completed. You should use the information
provided on the following screens to guide your study and practice.

Remember that once you leave a screen, you cannot go back to review it
again. Spend as much time as you need on each screen.

Now, advance to the next screen to begin reviewing your feedback.

(Press the space bar to advance to the next screen)

 

You should use the information you have gained from the feedback to
guide your study and your practice.

Now, exit the feedback program and wait for instructions from your lab

monitor.

(Press the space bar to exit the feedback program)

163

VELOCITY (SCORE) FEEDBACK

During this past practice period:

 

 

 

 

 

 

 

 

Average amount of time spent on a target: #
Your total score: #
Points gained for engaging targets correctly: #
Points lost for engaging targets incorrectly: #
Points lost for targets entering defensive perimeters: #
Number of targets engaged: #
Number of targets engaged correctly: #
Number of targets engaged incorrectly: #
Average amount of time spent on a target: #
Number of times you zoomed in: #
Number of pop-up targets you engaged: #

 

0 Pay attention to your score and its constituent information,
and use it to track your progress during the training.

(Press the space bar to advance to the next screen)

164

MASTERY FEEDBACK: FIRST TRIAL BLOCK

During this past practice period:

TYPE subdecisions: # correct out of # total ( % )
CLSS subdecisions: # correct out of # total ( % )
ITNT subdecisions: # correct out of # total ( 8 )

0 If you're getting subdecisions incorrect, you have not learned
the cues.

o If you are relying on the manual to get cue information, you
may not have fully memorized the cues.

0 If you are not making very many decisions, you are not
practicing your skills.

(Press the space bar to advance to the next screen)

 

During this past practice period:
FINAL_ENGAGE decisions: # correct out of # total ( 8 )

o If you're getting these decisions incorrect, you have not
learned the cues or you have not learned the correct decision
rule for peaceful and hostile targets.

0 If you are not making very many decisions, you are not
practicing your skills.

(Press the space bar to advance to the next screen)

 

During this past trial you did the following:

Hooked at least one contact: Y/N
Decided Type/Class/Intent: Y/N
Engaged at least one contact: Y/N
Used the zoom feature: Y/N

0 If you have not done these activities, you are not getting
familiar with the task.

(Press the space bar to advance to the next screen)

165

MASTERY FEEDBACK: SECOND TRIAL BLOCK

During this past practice period:

 

Times you zoomed out: #
Marker targets hooked: #
Targets hooked outside of outer perimeter: #

o If the numbers above are low, you are not practicing the skills
necessary to defend the outer perimeter.

(Press the space bar to advance to the next screen)

 

During this past practice period:

Inner perimeter intrusions: #
Outer perimeter intrusions: #

o If these numbers are greater than zero, then contacts are
crossing your defensive perimeters.

(Press the space bar to advance to the next screen)

 

During this past practice period:
Targets that popped-up: #

0 If you did not notice any of these pop-ups, you were not aware
of changes in the situation that were occurring.

(Press the space bar to advance to the next screen)

166

 

MASTERY FEEDBACK: THIRD TRIAL BLOCK
During this past practice period:

Speed queries of targets near inner defensive perimeters: #
Speed queries of targets near outer defensive perimeters: #

o If either of these numbers is low, you are not checking the
priority of targets near your perimeters.

(Press the space bar to advance to the next screen)

 

During this past practice period:

Inner intrusions: #
Inner intrusions prevented:

Outer intrusions:
Outer intrusions prevented:

#
A
#

 

o If you are not preventing intrusions on both perimeters, you
are not prioritizing before engaging targets.

0 If there are more intrusions than preventions on either
perimeter, you are not attending to the perimeters or you are
not engaging targets efficiently.

(Press the space bar to advance to the next screen)

 

During this past practice period:

High priority targets engaged: # out of # total ( % )
Low priority target engaged: # total

0 If you are engaging more low priority than high priority
targets, you are not prioritizing effectively.

0 If you are not engaging a high percentage of high priority
targets, you are not engaging targets efficiently.

(Press the space bar to advance to the next screen)

167

Appendix E: Measures

INSTRUCTIONS FOR ANSWERING QUESTIONS ON THE SCANTRON

Please write your last name in the area marked YOUR LAST NAME. Please
write in your first name and middle initial in the areas marked YOUR
FIRST NAME and MI. Do NOT bubble the letters since we will not be
using your name for data, and your responses will be kept anonymous.

Please DO fill in your PID and bubble the corresponding numbers.

After filling in your name and PID, please answer the following
questions by filling in the appropriate bubble, beginning with question
number 1.

168

Demographic Information

After filling in your name and subject number please answer the
following questions by filling in the appropriate bubble, beginning
with question number 1.
1. What is your sex?

(1) Male (2) Female
2. What is your age?

(1) < 18 yrs (2) 18-19 yrs (3) 20-21 yrs (4) 22-23 yrs (5) > 23
3. What is your overall grade point average?

(1) 0 - 1.0 (2) 1.1 - 2.0 (3) 2.1 - 3.0 (4) 3.1 - 4.0 (5) >4.0
4. Have you been to this particular lab before?

(1) Yes (2) No
5. Are you left or right handed?

(1) Left (2) Right
6. Do you play video games?

(1) Never (2) Rarely (3) Sometimes (4) Frequently (5) Always

169

Goal Orientation Measures

Button, Mathieu, & Zajac (1996)

This set of questions asks you to describe how you feel about each of
the following statements. Please use the scale shown below to make
your ratings.

Strongly Strongly
Disagree Disagree Neutral Agree Agree
<---l ---------- l ---------- I ---------- I ---------- I--->
(1) (2) (3) (4) (5)

Check to make sure you are beginning on question number [x].

1.

2.

10.

11.

12.

l3.

14.

15.

16.

The opportunity to do challenging work is important to me.
I do my best when I'm working on a fairly difficult task.
I try hard to improve on my past performance.

When I have difficulty solving a problem, I enjoy trying
different approaches to see which one will work.

The opportunity to learn new things is important to me.

The opportunity to extend the range of my abilities is important
to me.

I prefer to work on tasks that force me to learn new things.

When I fail to complete a difficult task, I plan to try harder
the next time I work on it.

The things I enjoy the most are the things I do the best.

I feel smart when I can do something better than most other
people.

I like to be fairly confident that I can successfully perform a
task before I attempt it.

I am happiest at work when I perform tasks on which I know that I
won't make any errors.

I feel smart when I do something without making any mistakes.

I prefer to do things that I can do well rather than things that
I do poorly.

The opinions others have about how well I can do certain things
are important to me.

I like to work on tasks that I have done well on in the past.

170

Goal Commitment Scale

(Hollenbeck, Williams, & Klein, 1989)

Strongly Strongly
Disagree Disagree Neutral Agree Agree
<---l ---------- l ---------- I ---------- l ---------- l--->
(1) (2) (3) (4) (5)

I take the goal(s) of this task session seriously.

I'm willing to put forth effort to work toward the presented
goal(s).

It is quite likely that the goal(s) of this session may need to
be revised, depending on how things go.

I care about the goal(s) of this session.
I am committed to pursuing the goal(s) of this session.

It wouldn't take much to make me abandon the goal(s) of the
session.

I think the goal(s) presented to me are worth pursuing.

171

Self-Efficacy

This set of questions asks you to describe how you feel about your
capabilities to play TAG. Please use the scale shown below to make
your ratings.

Strongly Strongly
Disagree Disagree Neutral Agree Agree
<---l ---------- I ---------- l ---------- l ---------- l--->
(1) (2) (3) (4) (5)

Check to make sure you are beginning on question number [x].
1. I can meet the challenges of this simulation.

2. I am confident in my understanding of how information cues are
related to decisions.

3. I can deal with decisions under ambiguous conditions.

4. I am certain that I can manage the requirements of this task.

5. I believe I will fare well in this task if the workload is
increased.

6. I am confident that I can cope with this simulation if it becomes

more complex.

7. I believe I can develop methods to handle changing aspects of
this task.

8. I am certain I can cope with task components competing for my
time.

172

Declarative Knowledge Test

The following is a knowledge test about TAG. Please use the Scantron
sheet to answer the following questions. Bubble in the correct letter
for each question, making sure the question numbers match.

Check to make sure you are beginning on question number [x].

1. If a Response is Given, what is the likely Intent of the target?
a. Military
b. Hostile
c. Civilian
d. Peaceful
2. If a target's Speed is 25 knots, its Communication Time is 85

seconds, and its Altitude/Depth is 0 feet, what does this suggest
about the target's Type?

a. The target is a Surface Vessel
b. The target is a Submarine
c. The target is Civilian
d. The target is Military
3. A Submarine has which of the following characteristics?
a. Speed 30 knots, Altitude/Depth -20, Communication
time 85 seconds.
b. Speed 30 knots, Altitude/Depth 0, Communication time
30 seconds.
c. Speed 20 knots, Altitude/Depth 0, Communication time
80 seconds.
d. Speed 20 knots, Altitude/Depth -20, Communication time
90 seconds.
4. A Maneuvering Pattern of Code Delta indicates which of the

following? The target is:

a. Air c. Surface
b. Military d. Civilian
5. A Blue Lagoon Direction of Origin indicates which of the

following? The target is:

a. Unknown c Civilian
b. Sub d. Military
6. A target that gives No Response indicates which Intent?
a. Peaceful c. Military
b. Hostile d. Unknown

173

10.

11.

12.

If a target’s Altitude/Depth is 10 feet, what is the Type of the
target?

Air
Surface
Submarine
. Unknown

Q—OU'ID

Your Outer Defensive Perimeter is located at:

64 nm

. 128 nm
256 nm
512 nm

O—OU‘W

If a target’s Intelligence is Unavailable, what Class does this
suggest for the target?

Air

Civilian
Military
. Unknown

0.0 0'91

What Threat Level would be assigned to a Hostile target?

0.06m
haurmta

. 0
If a target's characteristics are Communication Time = 20 seconds
and Speed = 50 knots, which of the following actions should you
take?

Choose Intent is Peaceful

Choose Type is Surface

Get another piece of information
. . Choose Type is Air

Q—OU’DJ

A Military target has which of the following sets of
characteristics?

a. Intelligence is Platform, Red Sea Direction of
Origin, Maneuvering Pattern is Code Delta.

b. Intelligence is Unavailable, Red Sea Direction of
Origin, Maneuvering Pattern is Code Echo.

c. Intelligence is Platform, Blue Lagoon Direction of
Origin, Maneuvering Pattern is Code Delta.

d. Intelligence is Unavailable, Blue Lagoon Direction of

Origin, Maneuvering Pattern is Code Echo.

174

13.

14.

15.

16.

17.

18.

19.

If a target's characteristics are Intelligence is Private and
Maneuvering Pattern is Code Foxtrot, which of the following
actions should you take?

Choose Class is Military
Choose Intent is Peaceful
Choose Class is Civilian
Choose Intent is Unknown

O-OU'W

A Peaceful target will demonstrate what form of countermeasures?

Jamming
None
Unknown

. Undetected

Q-OU‘W

A Communication Time of 52 seconds indicates that the target is
likely:

. Air
Surface
Submarine
Unknown

D-OU'N

If a target's characteristics are Response is Inaudible, Threat
Level = 3, and Countermeasures are Jamming, which of the
following actions should you take?

Choose Intent is Unknown
Choose Intent is Peaceful
Choose Intent is Hostile
Choose Intent is Military

900'”

If a target’s Maneuvering Pattern is Code Echo, this suggests
that the target falls into which category?

Class is Unknown
Class is Military
Class is Hostile
Class is Peaceful

900'”

A Peaceful target will demonstrate what Threat Level?
a. .0 c. 2
b. 1 d. 3
Your Inner Defensive Perimeter is located at:

a. 10 nm c. 64
b. 32 nm d. 128

175.

Adaptive Knowledge Measure

Response options in bold face are keyed as the "best" option, while
those in italics are keyed as the "worst" option, based on a priori
coding and expert responses to the questionnaire.

The following scenarios are designed to assess your expertise in
applying what you know about the TAG game to different situations.
Please read each of the situations presented carefully, and mark the
option you would app; likely choose and the item you would least likely
choose on the scantron. Be sure to begin with number [x] on your
scantron form.

Scenario 1

Four targets are approaching your OUTER DEFENSIVE PERIMETER. Target One
is moving at 22 knots, Target Two at 350 knots, Target Three at 305
knots, and Target Four at 35 knots. All targets are equally close to
the defensive perimeter.

A. Prosecute Target One first
8. Prosecute Target Two first
C. Prosecute Target Three first
D. Prosecute Target Four first

1. Which of these options would you MOST LIKELX choose?
2. Which of these options would you LEAST LIKELY choose?

Scenario 2

Five targets are approaching your INNER DEFENSIVE PERIMETER. Target One
is moving at 222 knots, Target Two at 355 knots, Target Three at 425
knots, and Target Four at 3 knots, and Target Five is moving at 294
knots. All targets are equally close to the defensive perimeter.

A. Prosecute Target One first

8. Prosecute Target Two first

C. Prosecute Target Three first

D. Prosecute Target Five first

3. Which of these options would you MOST LIKELX choose?
4. Which of these options would you LEAST LIKELY choose?

[Please continue to the next page]

176

Scenario 3

Your radar screen indicates that 5 fast targets are approaching your
INNER DEFENSIVE PERIMETER, and that 3 fast targets and 5 slow targets
are approaching your OUTER DEFENSIVE PERIMETER. All of the targets are
about the same distance from their respective perimeters.

A. Deal with the fast targets at the outer perimeter first,
then move to the inner

8. Deal with all the targets at the outer perimeter first, then
move to the inner

C. Deal with the targets at the inner perimeter first, then
move to the outer

D. It doesn't matter which order you deal with the targets

because the penalty values are the same at both perimeters

5. Which of these options would you MOST LIKELY choose?
6. Which of these options would you LEAST LIKELY choose?

Scenario 4

In a sudden burst of activity, the enemy has sent its entire force
against your fleet, hoping to wipe you out. Your radar screen is filled
with targets, and there is no way you can hope to deal with all of them
before the fleet is overwhelmed, so you must choose wisely. Because of
the massive number of targets, you only have time to look at a minimum
amount of information on each target.

A. Look at target Intent, and only prosecute Hostile targets
B. Look at target Class, and only prosecute Military targets
c. Look at target Type, and only prosecute Air targets

D. Look at target speed, and only prosecute fast targets

7. Which of these options would you MOST LIKELY choose?
8. Which of these options would you LEAST LI my choose?

Scenario 5

The Admiral decides to expand the fleet because of increasing political
tensions, and adds a destroyer group 500 nm from your station. He
instructs you that you now have a third defensive perimeter to monitor,
located at 512 nm. Because of the distance of the new group from the
remainder of the fleet, he instructs you that protecting this perimeter
is much more important than protecting the perimeter at 256 nm. While
monitoring your radar, you notice 6 targets rapidly approaching your
MIDDLE DEFENSIVE PERIMETER (256 nm) and 5 targets rapidly approaching
the OUTER DEFENSIVE PERIMETER (512 nm).

A. Deal with the military targets first

B. Deal with the middle perimeter targets first

C. Deal with the outer perimeter targets first

D. Deal with the hostile targets first, since peaceful targets

don't threaten your destroyer group

9. Which of these options would you MOST LIKELY choose?
10.Which of these options would you LEAST LIKELY choose?

I77

For the next two scenarios, assume the following: In game-play
terms, you have three penalty circles. INNER DEFENSIVE PERIMETER
intrusions result in a deduction of 50 points. MIDDLE DEFENSIVE
PERIMETER intrusions result in a deduction of 100 points. OUTER
DEFENSIVE PERIMETER intrusions result in a deduction of 75
points.

Scenario 6

12 targets are approaching your inner perimeter. 8 are fast, 4 are
slow, and of the fast targets, six are close to the perimeter. 15
targets are approaching the middle perimeter. 8 of these targets are
also fast, but only 3 fast targets are close to the perimeter. 4
targets are approaching your outer perimeter. All are fast, and all are
close to the perimeter.

A. Focus on the INNER DEFENSIVE PERINETER

B. Focus on the MIDDLE DEFENSIVE PERIMETER

C. Focus on the OUTER DEFENSIVE PERIMETER

D. Focus your attention equally across all fast targets, loving

back and forth across perimeters until you have dealt with
all targets successfully

11.Which of these options would you KQEEELLKELX choose?
12.Which of these options would you LEA§1_L1KELX choose?

Scenario 7

Now assume instead that you have 22 targets sitting almost on top of
your inner perimeter, but hardly moving at all. You still have 15
targets approaching your middle perimeter, with 8 moving fast and being
fairly close to the perimeter. At your outer perimeter, you have 12
targets, all of which are moving fast and are as close to the perimeter
as the targets approaching the middle perimeter.

A. Focus on the INNER DEFENSIVE PERINETER

B. Focus on the MIDDLE DEFENSIVE PERIMETER

C. Focus on the OUTER DEFENSIVE PERIMETER

D. Focus your attention equally across all fast targets, moving
back and forth across perimeters until you have dealt with all
targets successfully

13.Which of these options would you MOST LIKELX choose?
14.Which of these options would you LEK§2_L1KELK choose?

[Please continue to the next page]

178

Scenario 8

Your radar screen is malfunctioning and is incapable of giving you an
accurate readout of its range, making it difficult to locate your
defensive perimeters. You know that two of your submarines were
assigned to guard the perimeters, but without information on the current
range of your scope, you aren't sure where to find them. You must
determine how to defend your defensive perimeter.

A. Look for submarine targets and prosecute all of the other
targets that approach them,

B. Look for submarine targets, soon out, and prosecute all of
the other targets that approach that region.

C. Look for stationary targets and prosecute all of the other
targets that approach them.

D. Look for stationary targets, soon out, and prosecute all of

the other targets that approach that region.

lS.Which of these options would you HOST LIKELY choose?
16.Which of these options would you LEAST LIKELX choose?

Scenario 9

Your defensive perimeters are at 10 nautical miles and 256 nautical
miles. Your radar screen is clear, and then suddenly your sensors come
to life and clusters of targets pop up all around your screen. You need
to decide which cluster of pop-up targets to deal with first. All
targets are moving at the same speed.

A. The four-target cluster at 81 nm
E. The two-target cluster at 15 na
c. The ten-target cluster at 250 nm
D. The six-target cluster at 280 nm

l7.Which of these options would you MOST LIKELY choose?
lS.Which of these options would you LEAST LIKELE choose?

STOP !

179

Self-Regulation "Diary" Entry

Trainipg Trials

Please use the space below to describe the goals you focused on during
the previous trial. Please be specific with respect to what goals you
adopted during the past five minutes. When you finish writing, please
wait for further instructions.

 

 

 

 

Generalization Tpial

Please use the space below to describe the most important/critical
element of the task that you had to learn or perform during the training
session today. Please be specific, because it is important for the
purposes of future training design that we accurately understand what
you, as a trainee, had the most difficulty grasping based on how we
designed the training.

 

 

 

 

180

Self-Regulatory Activity Items

The following questions ask you to consider the kinds of things you
were thinking about over the course of the previous three trials.
Because different people may pay attention to different things in the
task, there are no right or wrong answers to these questions, so please
answer as honestly as possible with respect to how much ygp thought
about each of the things listed below.

Almost Moderate A Great
Not At All Amount Deal
<---l ---------- l ---------- I ---------- I ---------- l"->
(1) (2) (3) (4) (5)
1. I tried to monitor closely the portions of the task where I

needed the most practice.

2. I noticed where I made the most mistakes during practice and
focused on improving those areas.

3. I evaluated the feedback at the end of the practice trial to
determine how I would approach the next trial.

4. While practicing the scenarios, I monitored how well I was
learning their requirements.

5. I monitored how well I was performing on this task.

6. I tried to monitor those portions of the task that would give me
information about how well I was reaching my goals.

7. I actively planned how I should approach each trial.

8. I used the feedback from each trial to devise a plan for how to
approach the next trial.

9. I developed strategies that allowed me to improve my mastery of
TAG.
10. I focused on long-term goals for skill development rather than

short-term performance.

IL I set goals for myself in addition to the mastery goals I was
provided.

181

Self-Regulatory Skill Questionnaire

The next set of items asks you to think about how you generally approach
learning, in terms of learning strategies and study skills, in the
classroom environment. ere are no ht o wron answe to these
gpestiop . Use the scale below to indicate how true each statement is
of your approach to learning. If you think the statement is very true
of you, circle 5; if a statement is not at all true of you, circle 1.

If the statement is more or less true of you, find the number between 1
and S that best describes you.

 

 

Not at all Very
True of Me True of Me
<---| | | —— — I |--->
(1) (2) (3) (4) (5)
1. I pay attention to things I need to practice when I'm learning

something new.

2. I notice where I make mistakes and focus on improving those areas.

3. I evaluate feedback I receive to determine how to approach future
activities.

4. When I'm practicing a new skill, I monitor how well I'm learning

its requirements.
5. I constantly monitor how well I'm performing tasks I'm working on.

6. I try to monitor those aspects of tasks I'm working on that give
me information about how well I'm reaching my goals.

182

Motivation Questionnaire

This set of questions asks you to describe how motivated you were in
this experiment overall. Please use the scale shown below to make your

 

 

ratings.
Strongly Strongly
Disagree Disagree Neutral Agree Agree
<---l I I ---------- I |--->

(1) (2) (3) (4) (5)
I put forth effort to answer questions accurately and honestly.

I tried to make good choices in determining the best and worst
options on the sample scenarios I was presented.

I tried to do well when playing the TAG simulation.

Overall, I was motivated to do well on all parts of the
experiment.

183

10.

Feedback Satisfaction Questionnaire

 

 

 

Strongly Strongly
Disagree Disagree Neutral Agree Agree
<---|- — I I ---------- I |--->

(1) (2) (3) l4) (5)

The feedback helped me to know if I was reaching my goal(s).
In general, the feedback I received was useless to me.

In general, the feedback I received was helpful to me.

There was feedback I received that I didn't use.

There was just the right amount of feedback.

I wished I could have received more feedback.

The way the feedback was provided frustrated me.

I was satisfied with the manner in which I received feedback.
I was frustrated because I didn't get the feedback I needed.

I was frustrated with the amount of feedback I received.

184

Appendix F: Figures

The following pages contain the ﬁgures referenced in the text of this dissertation.

185

 

1

 

 

Practice

 

(Behavioral)

 

 

Self-Monitoring
(Cognitive)

V

l

 

 

 

 

 

 

l

 

Self-Efﬁcacy
(Affective)

 

1

Figure 1: The Self-Regulatory System (Kozlowski et al., 1997)

186

 

Manipulations Process Proximal Distal

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Indicators Outcomes Outcomes

(End of Training) (Generalization)
Declarative

Practice Knowledge

. . . ; Self-Efﬁcacy

Vanabrlrty \. Basic Gen.
Adaptive Performance
Knowledge

Strategic Gen

Velocity Performance

Feedback Self-Regulatory Basic ng.

(in addition to “MW Performance

Mastery)
Strategic ng
Performance

 

 

 

Note: Bdchng.MW-Bdchfmmmeﬁﬂuiﬁgtid;w:Tm Mm-wcmmumm
wishhsicGei Mm-BchmhhMaﬁmﬁd;WcG-L Mm-Mcwminthe
Maionuﬁ

Note: CogﬁtiveAbﬁy.McIay0rMPeform-tca0rimm0erdc.mdVidconExpaianammed-mmdwm.

Figure 2: Experimental Model (Theoretical)

187

Estimated Marginal Means of SEFF

 

 

 

Estimated Marginal Means

 

 

B LOCK

Figure 3: Proﬁle Plot of Means for Self-Efﬁcacy Following Trials 3, 6, and 9

188

Estimated Marginal Means

Figure 4:

Estimated Marginal Means of SELF REG

 

25.0
24.5 I
24.0 .

(
23.5 r

41

- "PPRACMANP

 

 

 

D Constant Practice

22.5 0 Variable Practice
3

 

 

 

.3
MI

BLOCK

Proﬁle Plot of Means for Self-Regulation Following Trials 3, 6, and 9

I89

Manipulations

 

 

  

 

Processes Learning End-of-Training Generalization
@’ .080 Basic
"I J .6
\“ Self. 192 A w
"t 4 Emmy 09 207
‘0‘ {I '187 -.178
“I"; .127. 36‘
e\
'l t‘ .133
'1 r. .220
-. ‘
- Strategi

Note: All beta weights reported are from the ﬁnal step in the appropriate regression analysis, and all are

signiﬁcant (p < .05). Dotted lines indicate interaction effects of manipulations with time.

Figure 5: Revised Model, Post-Analytic

190

1200

 

 

 

1000 . . ‘ . ° '
300
e
8 400 ‘ . .
(I) e 0 e e
200 e 9
0 ‘ .
0 1 2 3 4 5 C 7 O I
'2“ 0 T1181 0 -M1 SD
A can
.400 . +1 so

 

 

 

Figure 6: Trainee Score Over Nine Training Trials

191

"Illllllllllllllll“