FACULTY VERSUS STUDENT PERCEPTIONS OF THE QUALITY AND RELEVANCE
OF A MASTERʼS DEGREE IN EDUCATIONAL TECHNOLOGY
By
Leigh Graves Wolf

A DISSERTATION
Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of
DOCTOR OF PHILOSOPHY
Educational Psychology and Educational Technology
2011

ABSTRACT
FACULTY VERSUS STUDENT PERCEPTIONS OF THE QUALITY AND RELEVANCE
OF A MASTERʼS DEGREE IN EDUCATIONAL TECHNOLOGY
By
Leigh Graves Wolf

In this dissertation, I seek to develop a deeper understanding of the perception of
quality and relevance of a specific masterʼs degree program as seen through the eyes
of two key stakeholders—the faculty and the students (current and alumni).
This study focuses on faculty and students in the Masterʼs in Educational
Technology program currently being offered by the Educational Psychology and
Educational Technology program, which is housed in the Department of Counseling,
Educational Psychology and Special Education, College of Education, Michigan State
University. Though a systematic program evaluation is beyond the scope of this
dissertation study, the focus is on two key terms to frame the study: quality and
relevance. These two constructs are seen through the eyes of two key stakeholders: the
faculty and students (current and past).
A sense of community and collaboration was the point of strongest alignment
between student and faculty views. The strongest divergence between groups was on
the question of the primary motivation for obtaining a masterʼs degree. The strongest
recommendation for change comes from in increasing population of non-k12
practitioners within the MAET student body to create more relevant experiences for non

iii

K-12 practitioners. At the end of this inquiry, one thing was consistently clear: the
MAET program has been successful because of change. To remain successful,
marketable, and viable in the academic marketplace, programs must continue to take
into consideration stakeholder viewpoints and continue to define quality and relevance
within their contexts to inform programmatic and curricular changes.

Faculty versus Student Perceptions of the Quality and Relevance of a Masterʼs Degree
in Educational Technology by Leigh Graves Wolf is licensed under a Creative Commons
Attribution-NonCommercial-ShareAlike 3.0 Unported License.
Permissions beyond the scope of this license may be available at
http://www.leighgraveswolf.com

DEDICATION
This dissertation is dedicated to the alumni and students of the Master of Arts in
Educational Technology program. Your passions are my passions and I strive every day
to develop a program that reflects the excellence and dedication you bring to the table.

v

ACKNOWLEDGEMENTS
I would like to send sincere thanks to my dissertation committee Punya Mishra,
Matthew Koehler, Ann Austin and Dick Prawat. I am thankful for your insight, coaching
and especially the gift of your time. Punya, you of course receive special recognition for
guiding me through the PhD program and being a cherished mentor and friend.
I would not have been able to balance work and school responsibilities without the
administrative support of Sharon Anderson, Emily Fink, Heather Johnson, and Brook
Thompson. I am also sincerely appreciative of my MAET colleagues Carrie Albin, Jodi
Spicer, and Jean-Pierre Heussaff.
I would like to thank the following educators for having a profound impact on my
own teaching and learning: Cheryl Rosaen, Steve Weiland, Patrick Dickson, David
Wong, Nicole Ellison, Carrie Heeter, David Legg, Sandra Moulin, and Gary Reid.
I am eternally grateful to Andrea Zellner not only for her impeccable editing and
feedback but also for her sincere friendship and willingness to share her sweet babies
with me when I need a smile. I also could not have made it through this journey without
the friendship of Brandon Blinkenberg, Sean Leahy, Scott Westerman, Nick Sheltrown,
Megan Williams, Brad, Amanda, Declan and Harwell Davies, Clint and Emily Crook, the
Schramily and our frequent trips to “the Dog.”
Additionally, I have to thank my PLN on Twitter and Facebook for always sending
encouraging words and support when I needed it the most; I wish I could list every
single one of you. I am so lucky to have a global support system.

vi

Family is another crucial element to my success. I would like to thank my mom
Betty Graves, for always supporting me, my sister Lauren for being awesome, my
mother and father-in-law Bob and Bobbi Wolf and my beautiful nieces and nephew
Anna, Erica, Lilah, and Aidan. Though my father was not able to see the end of this
journey, I know he was waiting for the day when he could post “LVD” on my “Iʼm
finished” status update.
Last and of course not least, I would like to thank my husband Scott Wolf. I know
itʼs been a long, bumpy and less than enjoyable journey. Thank you so much for
sticking by me through the bad times, and Iʼm looking forward to better ones.

vii

TABLE OF CONTENTS
LIST OF TABLES ……………………………………….…………………............…………. x
LIST OF FIGURES ………………………………………………..……...........……………. xi
CHAPTER 1
INTRODUCTION …………………………………………………..………………….............1
Program Evaluation ……………………………………...……………………............. 3
Why this study? …………………………………………..………………............……. 5
CHAPTER 2
LITERATURE SURVEY ………………………………….……………............……………. 7
Key Stakeholders ……………………………………….............………………..……. 8
Previous Evaluations of Masterʼs Programs .............……………..………………… 9
Defining Quality & Relevance …………………………............……………….……. 12
Summary ………………………………………............………………………………. 14
CHAPTER 3
FRAMING THE CONTEXT, RESEARCH QUESTIONS & INSTRUMENT DESIGN .... 17
Context …………………………………………………………………………...…….. 17
The Masterʼs Program in Educational Technology ………………………….. 19
Understanding Program Faculty Perceptions ………………………………… 21
Understanding the Perceptions of Students and Alumni ………………….… 22
Instruments ……………………………………………………………………….……. 25
Faculty Interviews ………………………………………….……………………. 25
Student Survey ……………………………………….…………………………. 28
Mapping Faculty Interview Protocol to Student Survey ………………….…. 30
CHAPTER 4
RESULTS: FACULTY PERCEPTIONS ………………………………………...…………. 33
What do Faculty Perceive as the Goals & Intended Outcomes of the Program?...36
What is the Pedagogical Knowledge that Faculty Believe Derives their Design of
Program and Student Experiences? …………………………………………...……. 40
What types of Activities and Lessons do Faculty Create for Students to
Experience in the Program? ……………………………………………….…………. 42
What Evidence do Faculty have to Confirm that the Above Intentions are Being
Met? …………………………………………………………………………….………. 44
Summarizing Faculty Perceptions ……………………..……………………………. 47
CHAPTER 5
RESULTS: STUDENT AND ALUMNI PERCEPTIONS ……...……………………………48
General Descriptors ……………………………………………………...……………. 48
Goals and outcomes …………………………………………….…………………….. 51
viii

What do Students Perceive as the Goals and Outcomes of the Program?...51
Student Experience ………………………………………………………...…………. 54
What Do Students Experience In The Program? ……………………………. 55
What Difference Did The Program Have to the Professional Lives of
Students? …………………………………………………………………….…. 58
Value ……………………………………………………………………………………. 58
What is the Value that Students Derive from Participation in the Program?..59
What Evidence Do we have that Shows the Program Provides Value?....….62
Summarizing Student Perceptions …………………………………………..………. 63
CHAPTER 6
COMPARISONS, RECOMMENDATIONS AND CONCLUSION ……………….………..66
Comparisons …………………………………………………………………………… 66
What Similarities Exist Between Faculty and Student Perceptions? …….… 66
What Tensions Arise Between Student and Faculty Perceptions?..…......... 67
Recommendations ………..........................................................................……… 70
What Ideas or Recommendations for Future Directions of the Program
Emerge? …….................................................................................………… 70
Limitations ….......................................................................................…………… 72
Connecting to Deeper Issues of Program Evaluation ….....................…………… 74
Conclusion …………......................................................................................…… 76
APPENDICES …………….........................................................................................… 78
REFERENCES ……………........................................................................................… 92

ix

LIST OF TABLES
Table 1. Mapping the Research Questions .................................................................. 30
Table 2. Introduction to Faculty .................................................................................... 35
Table 3. Descriptive Statistics for Primary Motivation for Enrolling in the Program ...... 59
Table 4. Descriptive Statistics for Degree of Connection to Program ........................... 61

x

LIST OF FIGURES
Figure 1. Student Population ........................................................................................ 49
Figure 2. MAET Graduation Year ................................................................................. 50
Figure 3. Primary Motivation for Enrolling in the Program ............................................ 59
Figure 4. Indicators of Success After Graduation ......................................................... 60
Figure 5. Degree of Connection to Program ................................................................. 61

xi

CHAPTER 1
INTRODUCTION
Graduate education is the Detroit of higher learning… If American higher
education is to thrive in the 21st century, colleges and universities, like Wall Street
and Detroit, must be rigorously regulated and completely restructured. My hope is
that colleges and universities will be shaken out of their complacency and will open
academia to a future we cannot conceive--NY Times, April 2009

Higher education is often described as being in crisis. Some programs are being
asked to change the way they do things, and to engage in more rigorous assessment of
their practices; others are simply running status quo. In this context, there is a premium
on the innovation of methodologies to better understand how these competing and
conflicting goals are negotiated. Some organizations such as Knomads
(http://knowmads.nl) and ThinkGlobal School (http://thinkglobalschool.org) seek to
address these challenges through developing distinctly different institutional models.
These institutions put an emphasis on student driven curriculum, collaboration, and
sharing. Other programs struggle to work within traditional university systems.
Programs that exist within traditional universities need to negotiate a wide and
often conflicting set of goals, including: meeting student needs at a time of declining
enrollment; faculty values that may not reflect changes in broader society and the
workplace; increased competition from other organizations (private and public); new
technologies that provide alternative ways to present curricula (Ponnuru, 2010;
Postman, 1995; Johnson, Smith, Willis, Levine & Haywood, 2011.) The rapid rate of
change of technologies requires that graduate programs in these fields be even more
nimble and receptive to change (Wilson, 2010).
1

Explicitly, graduate programs offer students a degree. Nonetheless, there are the
implicit goals programs offer to their student body. Two of the implicit goals are quality
and relevance. These are difficult to define, and definitions vary depending on the
context. For instance, faculty members can have a very different way of speaking of the
quality (or relevance) of a program than the students.
Historically, masterʼs degree programs have been the predominant form of
professional growth for K12 teachers. Traditional masterʼs degrees, however, have often
failed to meet the needs of career teachers (Tom, 1999; Blackwell and Diez, 1998;
Little, 1993). Traditional masterʼs degrees have been criticized for their lack of (1)
prestige and quality; (2) clear definition and purpose; and (3) coherent structure and
organization (Conrad & Eagan, 1990; Conrad, Howarth & Millar, 1993). Sparks and
Hirsh (2000) suggested that effective masterʼs degree programs for practicing teachers
incorporate the following tenets of successful professional development: focused on
helping teachers become deeply immersed in subject matter and teaching methods;
sustained, rigorous, and cumulative coursework; curriculum-centered and standardsbased; and directly linked to what teachers do in their classrooms (p. 45).
If higher education organizations are to contribute to the professionalization of
teaching, then it is imperative that masterʼs degrees more closely meet these criteria
and the needs of practicing teachers (Tom, 1999; Blackwell & Diez, 1998; Shulman,
1993). Another way of thinking about this is by asking whether these programs ensure
quality and provide relevance to their students.

2

Program Evaluation
Broadly these issues have been in the domain of program evaluation. Program
evaluation is a term that means many different things, to many different people, in many
different contexts. In the broadest sense it can be defined as a systematic way to learn
from past experiences. Program evaluation is complex genre. For instance, Thayer
(2006) and Stuffelbeam (2001) have argued that there could be as many as 22 distinct
approaches to evaluation. At the heart of such research is a focus on key stakeholders
who influence the design, implementation, and accomplishments of the program. This
is of particular importance because there is a greater demand for accountability in
education fueled by economic and competitive challenges. There is a need to “prove”
that educational programs are indeed doing what they have claimed to do. Such
“evaluations” are often demanded and driven by accreditation agencies. (Burke, 2005)
This is an important process, however, whether or not an outside agency requires it,
since it can provide much needed information for monitoring and improving programs.
Weiss (1998) argues that this can help administrators and faculty members take
decisions that are driven by data. For instance, such investigations can lead to
suggestions for corrections to programs, policies, and processes. It can lead to
information that can help determine which parts of the program are successful and
which are not, the parts that should be continued or expanded, and the parts that should
be abandoned. At a time of limited resources, developing a better understanding of what
leads to programmatic decisions (the faculty perspective) and how it is experienced and
perceived by students (the student perspective) can help organizations take decisions

3

regarding funding or not funding certain aspects of the program by providing “data on
what the program accomplishes and fails to accomplish, and for whom, and thus
clarifies the trade-offs that decision makers have to make” (Weiss 1998, p. 26). This is
consistent with what the National Council for Accreditation of Teacher Education
(NCATE), though not directly involved in the area of educational technology, has argued
as being the driving force behind program revision:
The unit regularly and systematically used data, including candidate and
graduate performance information to evaluate the efficacy of its courses, programs
and clinical experiences. The unit analyzes program evaluation and performance
assessment data to initiate changes where indicated. Candidates and faculty
assessment data are regularly shared with candidates and faculty respectively to
help them reflect on their performance and improve it. (NCATE, 2008).
Another aspect of the program evaluation has to do with organizational learning
(Weiss, 1998). There are five kinds of organizational learning that takes place within
systems of evaluation. They include, (a) a recording of program history; (b) providing
concrete feedback to practitioners within the unit; (c) as a mechanism for highlighting
program goals; (d) developing a mechanism for accountability; and (e) an opportunity to
understand social intervention.
In summary, there are two key reasons for conducting a program evaluation. The
first is to inform programmatic decision-making, and the second is to create a climate for
organizational learning beyond this particular study.

4

Why this Study?
This specific research study resides within this broader context of higher education
programs and their evaluation. In that sense it is a case study of one masterʼs
program—of its quality and its relevance in order to inform decision-making and seek to
provide information for organizational learning (i.e. to help improve the program). This
study contextualizes this debate within the case study of one masterʼs program, the
Masterʼs Program in Educational Technology (MAET) at Michigan State University. It
does so by seeking to better understand how faculty and students articulate the quality
and relevance of the program – instantiated through the goals and outcomes, value,
experience, and evidence of success or failure.
At a personal level, these issues are a prime motivator for my interest in this
research study. As coordinator of the program for the past three years (and an instructor
for three years prior), I often say to potential students: “We have a successful program
that meets the needs of our students and alumni.” Currently, that statement is
conjecture, predicated solely upon my personal experience developed through
conversations with alumni, faculty, and students. None of those anecdotal statements,
however, hold water when trying to truly justify my statement. This dissertation study is
my step towards gaining a better understanding of the underlying issues so that I can
better represent the program, as well as providing information that can help me guide
the design of the program in the future.
This dissertation is structured as follows. Chapter 2 provides a survey of the
literature on assessment of masterʼs degree programs and a more in-depth discussion

5

of the research questions that emerge from this discussion. Chapter 3 presents the
context of the study, a framework for each set of research questions in this study, and a
description of the faculty interview and student survey instruments. Chapter 4 focuses
on the qualitative analysis of the faculty interviews, and points to emergent themes that
address the research questions. Chapter 5 focuses on the student data and the
research questions connected to student perceptions of the masterʼs program. Finally,
Chapter 6 compares and contrasts the faculty and student perceptions through finding
commonalities and contradictions between these perceptions, provides
recommendations for the programʼs future and a conclusion to the study. The resulting
analysis attempts to provide an interpretive stance towards how these can be
understood and resolved and also highlights emergent themes and contributions of the
study.

6

CHAPTER 2
LITERATURE SURVEY
Program evaluation can be seen as an example of a wicked problem (Rittel &
Webber, 1973). A wicked problem by definition is one where different stakeholders may
have different perceptions of the problem itself, and thus of the solutions as well. As
Mishra & Koehler (2008) wrote:
… wicked problems, in contrast to “tame” problems (such as those in
mathematics, chess etc.), have incomplete, contradictory, and changing
requirements. Solutions to wicked problems are often difficult to realize (and
maybe even recognize) because of complex interdependencies among a large
number of contextually bound variables. Wicked problems, they argue, cannot be
solved in a traditional linear fashion, because the problem definition itself evolves
as new solutions are considered and/or implemented. Rittel and Webber stated
that while attempting to solve a wicked problem, the solution of one of its aspects
may reveal or create another, even more complex problem. Moreover, wicked
problems have no stopping rule—and solutions to wicked problems are not right
or wrong, simply "better," "worse," "good enough," or "not good enough." Most
importantly, every wicked problem is essentially unique and novel. There are so
many factors and conditions—all dynamic—that no two wicked problems are
alike. Accordingly, solutions to wicked problems will always be custom-designed
(p 10-11).

7

There are two key consequences of seeing problem evaluation as a wicked
problem. First, a wicked problem is perceived differently by different stakeholders.
Therefore it is important to understand the perceptions of the key stakeholders. Second,
wicked problems are tightly tied to their specific contexts. This means that any research
on quality or relevance needs to be situated in the specific context. In this case, the
context is a masterʼs program in educational technology offered at Michigan State
University. This program and its context are described in greater detail in Chapter 3.

Key Stakeholders
"Customer loyalty" is not a commodity a company owns. Where it exists at all -and the cases in which it does are rare -- loyalty to a company is based on
respect. And that respect is based on how the company has conducted itself in
conversations with the market. Not conversing, participating, is not an option. If we
donʼt engage people inside and outside our organization in conversation, someone
else will. (Levine, Locke, Searls & Weinberger, 2000)

Conrad and Eagan (1990) identified six criteria for evaluating a higher-education
program: faculty, students, resources, learning environment, curriculum, and placement
of alumni. In particular, researchers also suggest paying greater attention to the faculty,
who drive the design and implementation of the program, and students, who are primary
“stakeholders” in this enterprise (e.g. Clark 1979, Kirkwood, 1985). For this reason, this
study focuses on two key stakeholders: the program faculty (those in administration as
well as faculty who teach in the program) and the students (who include both program
alumni, i.e. those who have graduated; as well as current students).

8

Stakeholder inclusion needs to be an essential component of the process in order
to gain a comprehensive view of a program's accomplishments, future challenges, and
value. Nonetheless, it is often unclear as to what exactly the goals of a particular
program are and how they are being met. Thus, in order to better understand the goals,
successes, and failures of the program, it becomes important to study the variety of
stakeholders involved in the design, implementation, and application of the program. For
instance, the NCATE accreditation standards specifically call for the inclusion of multiple
perspectives in the assessment plan. Additionally, collecting information from multiple
sources allows the program to triangulate the evidence to support any claims being
made from the research (Wiggins, 1998).
In the next section we look at previous evaluations of masterʼs degree programs
and how they focus on specific stakeholders.
Previous Evaluations of Masterʼs Degree Programs
A survey of the literature indicates that there have been few evaluations of
Masterʼs degree programs. These evaluations have utilized a variety of methods of
collecting data and a variety of data sources. Some studies have focused entirely on
students and graduates of various programs. Wagner (2004) in his evaluation of a
teacher preparation program, made a strong argument for the value of including
students and graduates of programs in evaluation studies and suggests that of all
groups of stakeholders, “one particular group of stakeholders—alumni of teacher
preparation programs—provides a unique combination of insight and hindsight that
program evaluators can draw from when making program-improvement decisions.”

9

Kooker, Itano and Okimoto (2000) used focus group meetings with graduates in their
evaluation of the impact of a masterʼs in nursing distance education program. Similarly,
Trott, Barker & Barker (1988) in their evaluation of a masterʼs program in the speech
communications surveyed over 600 members of the speech communication association
to determine ratings of different masterʼs programs. Stellman, Cohen and Rosenfield
(2008) evaluated a one-year masterʼs of public health program for medical students by
surveying current students about their perception of satisfaction with coursework,
attitude towards the degree, career choices and trajectory. Similarly, Clark (2006)
assessed a community college leadership doctoral program through having alumni and
current students complete an online survey.
In contrast to studies that focused on students (or graduates of the program), other
studies have looked specifically at faculty members and their perceptions. For instance,
Manning (2010) focused entirely on community college faculty who were teaching
blended or web-assisted courses on their teaching practices and how these influenced
student persistence. Faculty perceptions were collected both through participation of
faculty in an expert panel as well as through web-based surveys. Peruski (2003) applied
an activity theory framework to analyze three faculty membersʼ experience with
designing and teaching online courses for the first time.
Other evaluation studies did not restrict themselves to a single group of
stakeholders for their assessment—seeking to look at the perceptions of both students
or program alumni and faculty and administrators. For instance, Ribak, Notzer, &
Drezne (1995; see also Bergman, Shatzman, & Danon, 1982 for a previous iteration)

10

conducted an evaluation of a masterʼs program in occupational health by means of
structured personal interviews of class leaders, teachers and students. Their research
focused on studentsʼ and teachersʼ expectations of the program, the integration
between various aspects of the program, and the value that students received from
going through the program (i.e. employment). Erdogan (2008) conducted an evaluation
of web-based instruction from both studentsʼ and instructorsʼ perspectives. Data for this
study was collected through one-on-one semi-structured interviews with 10 students
and 10 faculty members and the analysis focused on the similarities and differences
between these two perspectives.
These previous studies focused on two key stakeholders in the process of
evaluation, either the students (or alumni) of the program or the faculty who design and
deliver the courses/program. In each case, there are two key constructs under
discussion—that of the quality of a program, and/or its relevance.
In other words, the two key questions that most program evaluations seek to
answer are: Is the program offering quality and is it relevant to the students/alumni? In
fact, the studies listed above often fall on one side or the other of this issue. It can be
argued that the studies that focus on faculty tend to emphasize issues of quality while
the ones that focus on students tend to emphasize the relevance of the program. That
said, the terms quality and relevance are often ambiguous and difficult to define. This
has, however, not prevented scholars from attempting to define them. As Roach and
Barker (1984) wrote:

11

All attempts at evaluating education programs attempt, in one way or another, to
assess “quality.” Unfortunately, quality is an elusive variable to define and to
measure with reliability and validity. Two primary approaches have been
employed to assess quality of programs in previous studies. These include socalled “objective” measures of quality (e.g. admission requirements, course
listings, degree requirements, relative publishing performance of staff, placement
of graduates, salaries of graduates and faculty, productivity of graduates, library
holdings, faculty teaching loads, and graduate student/faculty ratios), and
“subjective” measures of quality (e.g. peer ratings by colleagues or by “blue
ribbon” expert panels) (p. 69).

The next section looks at the literature on quality and relevance and seeks to
develop a better understanding of both in order to construct a viable set of measures
and instruments for these constructs.

Defining Quality & Relevance
There are many definitions of educational quality. Harvey and Green (1993)
specifically looked at the idea of quality in higher education. They argued that since
there are a variety of stakeholders in higher education (students, employers, teaching
and non-teaching staff, government, accreditors, etc.), there are multiple perspectives of
the term quality. Moreover, they argued that quality is often seen as being a relative
concept – depending on both the user of the term and the situation in which it is used.

12

Instead of imposing one view of quality, Harvey and Green (1993) argued that there
are “five discrete but interrelated ways of thinking about quality,” though they add that
that these five categories are interrelated. The five ways of thinking about quality in
higher education are:
1. The exceptional view of quality where quality is seen as being something special,
connected to notions of excellence.
2. The quality as perfection frame sees quality as consistency or flawlessness.
They argue that this is a “democratic” idea where consistency is seen as a
measure of quality.
3. Quality as fitness of purpose sees quality as fulfilling a customerʼs needs and
desires. In a higher education context fitness of purpose is the ability of the
organization to meet its broader goals and to fulfill the needs of its students.
4. Quality as value for money, or in business terms as return on investment. In this
frame, a quality product is one that is optimized in terms of costs. Governments
and university administrators often view quality in this value-for-money approach
while students may have the same view – but from the opposite direction.
5. Quality as transformation is the enrichment or empowerment of students or the
development of new knowledge.
The idea of relevance has always been a part of discussions about quality. In
particular, the idea of relevance connects with the idea of quality as fitness of purpose,
which can be seen as the need to meet generally accepted standards or meeting
specific objectives or goals and objectives. It also connects to the idea of quality as

13

value-for-money in that the focus is on efficiency. Relevance also connects to the idea
of quality as transformation, in that it focuses on students and the goal of empowering
them with skills and knowledge, the pedagogical value of the program, and the kinds of
experiences the faculty design for the students to meet these goals and values. Finally,
both faculty and students need to decide on what evidence (based on their experience)
they would use to determine whether the program and the experiences did meet the
goals and objectives initially laid out.
Thus the ideas of quality and relevance are operationalized through focusing on
goals and objectives; the pedagogical value; the nature of experiences designed; and
the kinds of evidence used to confirm if the original goals and objectives are indeed
being met.
Summary
The survey of the literature on program evaluation and previous studies of
masterʼs degree programs identified a few key ideas that can guide the design of this
specific study.
First, evaluation studies are deeply contextual, due to the wicked nature of the
original problem. This means that evaluation of such programs have to be uniquely
designed for the specific context. In the case of this study, the context is a Masterʼs
Program in Educational Technology (and is described in greater detail in the next
chapter).
Second, the evaluation of these programs needs to focus on issues of quality and
relevance. The previous section shows that these constructs are ambiguous and have

14

been defined differently by different scholars. That said, this study focuses on goals and
objects, the pedagogical value, the nature of experiences designed, and the kinds of
evidence used to confirm if the original goals and objectives are indeed met.
Third, it is important to include key stakeholders in the evaluation process. Two
key stakeholders identified are the faculty who design and implement the program and
students / alumni who go (or have gone) through the program. Different studies have
looked at these stakeholders either singularly (looking just at faculty or student
perceptions of quality and relevance) or at both groups. It is clear that these two
stakeholders will have different perceptions of the goals and objectives of the program,
the nature of the experience, and how they see the goals being (or not being) met. The
faculty defines the vision of the program, implements it through designing courses and
other learning experiences, and measures its effectiveness. Students, on the other
hand, choose to come to a program with their own perspectives and goals, which may
or may not align with those of the faculty. It is important to better understand how these
perceptions align or contradict each other if the program is to effectively meet the needs
of the two key groups. Contradictions between these perceptions can lead to frustration
on the part of students or alumni because the relevance of the program does not match
the needs of the students. Thus, this study identifies the perceptions of both groups of
stakeholders with the idea of developing a better understanding of how these two
groups share or do not share similar constructs.

15

Fourth, this process of comparing the perceptions of both groups leads back to
the fundamental reasons for conducting such studies: to inform programmatic decision
making, and to create a context for organizational learning and improvement.
In brief, in this dissertation, I seek to develop a deeper understanding of the
perception of quality and relevance of a specific masterʼs degree program as seen
through the eyes of two key stakeholders—the faculty and the students (current and
alumni). This study focuses on faculty and students in the Masterʼs in Educational
Technology program currently being offered by the Educational Psychology and
Educational Technology program, which is housed in the Department of Counseling,
Educational Psychology and Special Education, College of Education, Michigan State
University. Though a systematic program evaluation is beyond the scope of this
dissertation study, I focus on two key terms to frame the study: quality and relevance.
These two constructs (which are described in greater detail later in this chapter) are
seen through the eyes of two key stakeholders: the faculty and students (current and
past). Once an analysis of the faculty and student perceptions are undertaken, this
study further seeks to compare these perceptions, highlighting both commonalities and
differences. It is only through this process of identifying points of convergence and
divergence of perception that one can initiate a process of reflection and accountability.
The next chapter provides further details of the specific program being studied, in
order to highlight the context for the study. It also describes the research methodology
and instruments developed for this purpose.

16

CHAPTER 3
FRAMING THE CONTEXT, THE RESEARCH QUESTIONS & INSTRUMENT DESIGN
Context
Just as the advent of electronic media helped shape society at the beginning of
the twentieth century, the capabilities and connected nature of the Internet is changing
the way we work and learn in the twenty-first century. Substantial efforts have been
made by local, state, and federal agencies in the United States to ensure that schools
are “wired”. Nearly every school in the United States is now connected to the Internet
(U.S. Department of Education, 2010). It is no surprise then that educational technology
has become an integral component in teaching, teacher education, and professional
development programs, and has become more frequently adopted by higher education
faculty.
In December 2001, the State of Michigan began to require that all certified
teachers be assessed for their “ability to use information age learning and technology
operations and concepts to enhance learning and personal/professional productivity.”
(Michigan State Board of Education, 2001). In 2003, the state of Michigan introduced
the Educational Technology (NP) Endorsement, which was developed to align with the
International Society of Technology in Education (ISTE) technology standards (Michigan
State Board of Education, 2008). The recent publication of the National Educational
Technology Plan underscores this need by more clearly articulating the aims and goals
of educational technology in U.S. education. (U.S. Department of Education, Office of
Educational Technology, 2010).

17

A range of higher education institutions now offer graduate level programs in
educational technology primarily to meet the need for training teachers to meet these
standards. It is not clear, however, how successful these programs are in actually
developing teachers who can use technology in effective ways. For instance, a survey
by Milken Family Foundation and ISTE found that teacher-training programs, in general,
do not provide future teachers with the kinds of experiences necessary to prepare them
to use technology effectively in their classrooms. (Milken Exchange, 1999). Specifically,
they found that formal stand-alone IT coursework does not correlate well with
technology skills and the ability to integrate technology into teaching. They
recommended that teacher preparation programs should increase the level of
technology integration in their own academic programs. Despite this call to arms,
studies of K-12 teachers' instructional applications of educational technologies to date
show many to be pedagogically unsophisticated; limited in breadth, variety, and depth;
and not well integrated into curriculum-based teaching and learning (e.g., Cuban, 2001;
Earle, 2002; McCrory-Wallace, 2004; Zhao, Pugh, Sheldon & Byers, 2002). In a 20-year
retrospective on U.S. educational technology policy, Culp, Honey, and Mandinach
(2003), describe a mismatch between educational technology leadersʼ visions for
technology integration and how most practitioners use digital tools. Researchers
emphasize technology uses that support inquiry, collaboration, and reformed practice,
while many teachers tend to focus upon using presentation software, learner-friendly
Web sites, and management tools to enhance existing practice.

18

The Masterʼs Program in Educational Technology
It is within this context that the Counseling, Educational Psychology and Special
Education (CEPSE) department within the College of Education at Michigan State
University (MSU) began granting masterʼs degrees in Educational Technology during
the 1998-1999 academic year. The current Master of Arts in Educational Technology
(MAET) program consists of three “paths” or modes of delivery: face-to-face, hybrid, and
online. The face-to-face program currently takes place overseas and caters to the
international school teaching community. The program can be completed in three years
over four weeks of condensed summer study sessions. The hybrid option consists of a
mix of fully online courses coupled with two condensed summer study weeks on
campus in East Lansing. In Spring 2008, the program became available for delivery
completely online. It is now possible for students to complete the degree without visiting
the MSU campus.
In its 10-year history, there has never been a systematic study of the MAET
program. Changes (and there have been many) have been driven by ad-hoc decisions
prompted both by perceived demand, competition from other programs, and the advent
of new technologies for the delivery of instruction. Additionally, budget restructuring and
other financial pressures (having to do more with less) demand a better review of the
MAET program. Another significant reason to better understand the perceived goals and
outcomes is the increased national and institutional emphasis on assessment and
accountability. The MAET program is eligible to apply to accrediting agencies (such as
the National Council for Accreditation of Teacher Education (NCATE), International

19

Technology Education Association (ITEA), Council on Technology Teacher Education
(CTTE), and the Association for Educational Communications and Technology (AECT)).
These agencies require programs to conduct internal evaluations and include
stakeholders in their program evaluation activities. Finally, the MAET program exists in
direct competition with other programs (across the state and the nation) that offer similar
programs. The past few years have seen an incredible increase in masterʼs programs
being offered in a variety of formats (hybrid and online) from a range of public and
private institutions. Developing a better understanding of student needs and their
alignment with program goals becomes a necessity for survival of the MAET program. In
this regard, we see this study as being essentially formative in nature—particularly
given the fact that the MAET program has not been studied in its decade long history.
As such, this study is a much-needed first step to help guide decisions on program
revision and expansion and sets the stage for a deeper evaluation study.
In this dissertation I am asserting that quality and relevance in the MAET program
can often be seen as playing out in three areas: the goals and intended outcomes of the
program; the values the program espouses; and, finally, the nature of the experiences
that are designed in order to meet these goals and values. Clearly, faculty and students
can have very different perspectives of these terms as well. It is thus important, when
we are speaking of evaluating higher education programs, to find out more about how
these two key stakeholders of faculty and students perceive issues related to quality,
relevance, goals, values, and experiences.

20

The specific research questions for this study emerge from an interest in defining
quality and relevance in ways that can help us understand the perceptions of faculty and
students in the MAET program.
Understanding Program Faculty Perceptions
The structure of higher education institutions places a great deal of the
responsibilities of the administration and teaching of academic programs on the faculty.
They are ultimately responsible for key decisions about the program, including broad
issues such as goals, values and design, as well as more core programmatic issues
such as admission criteria, curriculum, course sequences, grading, and evaluation.
Decisions on these issues are taken in multiple contexts and in different ways, often
contingent on local and time dependent constraints. It is not surprising, if given the
complexity of the program management and design, that different faculty have very
differing opinions of the nature, value, and goals of a program. Thus, including program
faculty in the evaluation of the program is of critical importance. By including faculty in
the evaluation process we can ascertain what the overall goals of the program are along
with underlying principles or tensions that play out in specific instructional or
programmatic decision making. It is against these goals and values that one can then
evaluate the nature of student experience in the program. Thus, the first part of the
research study involves the design and implementation of an interview protocol with
current and former faculty members in order to determine the vision and expectations
from the program. This leads to the first set of research questions, for the faculty:
1.

How do faculty members perceive the quality and relevance of the program?

21

a. What do faculty perceive as the goals and intended outcomes of the
program? (This would include their perception of why students come to the
program).
b. What are the pedagogical values that faculty believe drive their design of
the program and the student experiences?
c. What types of activities and lessons do faculty create for students to
experience in the program? How is this relevant to the students?
d. What evidence do the faculty have to confirm that the above intentions are
being met?
The data for answering these questions comes from a series of interviews with
faculty members (more detail on that below).
Understanding the Perceptions of Students and Alumni
Students, both current and alumni, form the next vital link in the assessment of
the program. They are particularly important because they are the programʼs connection
to the K-12 system which the MAET program influences. The goals of the program, as
evidenced by program documents, websites, and faculty ideas, need to be tested
against the actual experience of the students in the program. Ideally, there should be
agreement across these various indicators and stakeholders. But if these are not in
agreement, these points of departure can help draw attention to issues and concerns
that need to be addressed in the ongoing design/redesign of the program. Specifically,
this research studies student attitudes towards educational technology, their reasons for
seeking these advanced degrees, their experience in the program, the perceived

22

benefits to them of the program, and, finally, their perceived proficiency in both using
technology and in integrating it in their teaching of subject matter. In brief, students (and
alumni) are the ones who actually undergo (or underwent) the experience designed by
the faculty. So in parallel to what the faculty perceive is what the students actually
experience as they move through the program.
This leads to a second set of research questions having to do with student
experience and perception. There is a clear mapping of the research questions for the
faculty to those focusing on the students.
2.

How do students and alumni perceive the quality and relevance of the
program?
a. What do students perceive as the goals and intended outcomes of the
program? Why do they come to the program?
b. What is the value that students drive their participation in the program?
c. What do students experience in the program? How is this program
relevant to the professional lives of students?

Finally, it is important to investigate how faculty and student perceptions match (or
do not match). This is in keeping with the ideas of quality and relevance as being fitness
of purpose. For instance, it is important to know whether or not faculty meet student
needs, or if students perceive faculty intentions as being aligned with their needs and
requirements. The quality of the program can be judged from how well this happens.
This leads to the third and final research question.

23

3.

What insights can be gained when comparing data between faculty and
students?
a. What similarities exist between faculty and student perceptions?
b. What tensions arise between student and faculty perceptions?
c. What ideas or suggestions for the future directions of the program emerge
from this analysis?

The first two research questions reflect the fact that faculty and students often
approach the program from opposite directions. Faculty members design the course
curriculum and student experiences based on their perception of goals and needs and
what students need to know and why they think students are coming to the program in
the first place. They design experiences that embody their values and seek evidence to
learn whether or not these goals and objectives have been met. Students choose to
come to a program to meet their own goals and perceive their experiences and value
they receive from their perspective. Finally, when comparing the perceptions of faculty
and students, it is important that the instruments map onto each other and be applicable
to both groups. Without this conscious mapping, it would be difficult to answer research
question 3, which has to do with finding commonalities and differences between the two
perceptions to determine productive tensions that can help develop thoughts and ideas
for the future directions of the program. Also important to note here is that there is a
significant contrast between the numbers and availability of the different stakeholders.
Faculty members are fewer in number and were easily accessible by the researcher.
Students and alumni were far more spread out, particularly given the online options that

24

the program now offers. This also helped determine the nature of the instruments
designed.
Instruments
Two different instruments were designed for this study. First an interview protocol
was developed to address (a) faculty perceptions of the masterʼs program and (b)
students in the program. Once the interviews were transcribed and an initial analysis
performed, a survey instrument was designed for distribution to students and alumni.
Faculty Interviews
The first is an interview protocol for the faculty members and focused on the following:
1. Program History: These questions focus on faculty perceptions of the history of
the program, its origin, conceptualization, and how it has changed over time.
2. Student Body: Another set of questions focus on faculty perceptions of the
student body, their needs and requirements, and how this has had an effect on
the design of the program.
3. Teaching Tensions: A review of the program documents and personal
experience indicate that there are some inherent tensions in the MAET program.
In particular, these tensions have to do with the relationship between student
needs and program design; the relationship between educational psychology and
educational technology; and keeping pace with the rapid rate of change of
educational technology.
4. Program Strengths and Weaknesses: This set of questions focuses on the
faculty perceptions of the strengths and weaknesses of the MAET program by

25

building on the what the faculty have already articulated about the history,
student body and teaching tensions.
5. Future Planning: The final set of questions focus on what the faculty see as the
future direction of the program based on their experience teaching and/or leading
the program.
Exploring each of these in greater detail:
Faculty Perception of Program History
As stated earlier, there has been no formal departmental evaluation of the MAET
program. In an attempt to gather data on program history and record institutional
memory, the following questions were asked:
•

What course do (did) you teach? In regard to this course, what skills or
knowledge do (did) students gain in your course?

•

If you had to describe this program to someone else, how would you do it?
What are some key challenges faced by the program in the past? Have these
led to changes? In what way?

•

How has the masterʼs program changed over the years?

Faculty Perception of the Student Body
To understand faculty attitudes and assumptions about the student body, a series
of questions targeted this domain. Assumptions on the part of faculty can influence
student experience and the student and alumni survey allowed for comparison and
contrast of the answers to these questions against student self-reports. The following
questions targeted this cluster:

26

•

What differences will this outcome make in the lives of those who complete the
program? Why is this important?

•

In practice, what are the outcomes students gain from being in the masterʼs
program—the knowledge, skills, understandings, beliefs, values, or career
aspirations that they gain?

•

What are any other benefits students may derive from participation in the
masterʼs program?

•

What kinds of evidence do we have regarding these outcomes?

•

How well do you think the program is in meeting studentsʼ interests and needs?

Faculty Perception of Teaching Tensions
As an educational technology faculty, several tensions arise over how much time
should be spent on technology tutorial type of work and how much should be spent on
the larger goals of the course or project. These tensions play out in many different ways
in course subject matter and faculty attitudes, behavior and preferences. The following
questions addressed this tension:
•

There is always the tension between teaching technology (technology) and
teaching how to integrate technology (pedagogy), what are your thoughts on
these tensions? Give an example or relate this to something – say teaching
Dreamweaver versus building webpages for teaching?

•

There is also a perceived tension between theory and practice. Do you perceive
such tensions in this program? If so, how do they play out? What are your
thoughts on these tensions?

27

Faculty Perception of Program Strengths and Weaknesses
This cluster of questions is focused on how faculty perceived the overall intentions
of the MAET program. With the tensions and change over the years, it is important to
capture the faculty perception of the overall program strengths and weaknesses.
•

What are the intended learning outcomes of the program? That is, what are the
goals of the program in terms of what students are to learn to do, understand,
believe, or value?

•

I want to focus a bit more on program goals, why do you think students come to
our program?

•

What are the strengths of the masterʼs program?

•

What are the weaknesses or shortcomings of the program?

Faculty Perceptions of Future Planning for the Program
This cluster of questions in the interview is intended to allow faculty the opportunity
to brainstorm ideas about the future direction of the MAET program.
•

What are your ideas for new directions for the masterʼs program?

•

Are you aware of any ways in which the program is currently assessed? If you
know of any processes, instruments, or methods, how effective have they been
in measuring outcomes from participation in the program?

Student Survey
The second instrument designed for the study was administered to current students and
alumni of the MAET program. The survey focused on the following broad questions:
• What do students perceive as the goals and intended outcomes of the program?

28

• What do students experience in the program?
• What is the value that students derive from their participation in the program?
Exploring each of these in greater detail:
What do students perceive as the goals and intended outcomes of the program?
This line of open-ended questions focused on the student perception of the goals of the
MAET degree. The answers to these questions are compared to faculty responses to
the same questions.
• If you had to describe this program to someone else, what would you say?
• How often were you asked by instructors to demonstrate that you could integrate
the use of technology into the lesson plans?
What do students experience in the program?
To get a better idea of the perception of the experience, students were asked a blend of
multiple choice and open-ended questions intended to solicit data that could be used to
highlight specific programmatic outcomes.
• How well did the MAET program meet your needs?
• What are the strengths of the masterʼs program?
• What are the weaknesses of the program?
• What difference did participation in the MAET program have to your professional
life?
• TPACK survey questions.
What is the value that students derive from their participation in the program?

29

To understand the nature of the value students place on the degree, students were
asked a series of questions that were used to measure degree of connectedness,
overall satisfaction of the program and personal motivators for obtaining the MAET
degree.
• Why did you come to the program?
• Some educators claim that access to new technology in the classroom forces
teachers to rethink the nature of what they do in the subject matter domain. How
often did you experience this in the course of the program?
• How well did the MAET program balance the teaching of technology?
• How likely are you to recommend the MAET program to colleague or friend?
• Do you still feel connected to the MAET program?
Mapping the Faculty Interview Protocol to the Student Survey
The following table maps questions from the faculty interview and student survey to the
research questions and sub-questions. The faculty interview protocol and modified
student survey are included in the appendix.
Table 1. Mapping the Research Questions
Research Question
Category

Goals & Outcomes

Faculty Interview

Student Survey

What are any other benefits
students may derive from
participation in the masterʼs
program?

After graduation did you (check
all that apply)
Receive a pay increase
Apply for a new position
Get a better position
Change fields
None of the above
Other:

30

Table 1 (contʼd)

Goals & Outcomes

Goals & Outcomes

Goals & Outcomes

How well do you think the
How well did the MAET program
program is in meeting
studentsʼ interests and needs? meet your needs? (open ended
text)
If you had to describe this
program to someone else,
how would you do it?

If you had to describe this
program to someone else, what
would you say? (open ended
text)

In practice, what are the
outcomes students gain from
How often were you asked by
being in the masterʼs
instructors to demonstrate that
program—the knowledge,
you could integrate the use of
skills, understandings, beliefs,
technology into the lesson
values, or career aspirations
plans?
that they gain?

Experience

What are the strengths of the
What are the strengths of the
masterʼs program? (open ended
masterʼs program ?
text)

Experience

What are the weaknesses or What are the weaknesses or
shortcomings of the program? shortcomings of the program?
(open ended text)

Experience

Experience

Value

What differences will this
What difference did participation
outcome make in the lives of
in the MAET program have to
those who complete the
your professional life?
program
What are the goals of the
program in terms of what
students are to learn to do,
understand, believe, or value?

Selected TPACK survey
questions.

Why do you think students
come to our program?

Why did you come to the
program? (multiple select with
option of “other” fill in response)

31

Table 1 (contʼd)

Value

Some educators claim that
access to new technology in the
There is also a perceived
classroom forces teachers to
tension between theory and rethink the nature of what they
practice. Do you perceive
do in the subject matter
such tensions in this program? domain. How often did you
experience this in course of the
program? (open ended text)

Value

There is always the tension
between teaching technology
How well did the MAET program
(technology) and teaching
balance the teaching of
how to integrate technology
technology?
(pedagogy), what are your
thoughts on these tensions?

Value

How likely are you to
recommend the MAET program
to a colleague or friend?
scale
1- Not at all likely
10 - Extremely Likely

Value

Do you still feel connected to
the MAET program?
Yes
No
Please explain why or why not?
(open ended text)

By mapping the student and faculty questions to the goals and outcomes,
experiences, and value I set the framework for comparing and contrasting these
elements after analysis of each subsection.

32

CHAPTER 4
RESULTS: FACULTY PERCEPTIONS
The structure of higher education institutions places a great deal of responsibility
for the administration and teaching of academic programs on the faculty. The faculty is
ultimately responsible for key decisions about the program. These issues include broad
matters such as goals, values, and design, as well as more core programmatic issues
such as admission criteria, curriculum, course sequences, grading, and evaluation.
Decisions on these issues are taken in multiple contexts and in different ways, often
contingent on local and time dependent constraints. It is not surprising, given the
complexity of the program management and design, that faculty have very differing
opinions of the nature, value, and goals of a program. Therefore, the act of including
program faculty in the evaluation of the program is of critical importance since this can
inform evaluators of what the overall goals of the program are (and were), along with
underlying principles or tensions that play out in specific instructional or programmatic
decision making. It is against these goals and values that we can then evaluate the
nature of student experience in the program.
The Seidman interviewing model (Seidman, 2005) and grounded theory approach
guided the interview inquiry. The grounded theory approach is defined as “a qualitative
research method that uses a systematic set of procedures to develop an inductively
derived grounded theory about a phenomenon” (Strauss & Corbin, 1990, p. 21).
Creswell (1994) explained how to use this inductive mode of thinking to build a new
theory. The steps were (a) researcher gathers information, (b) researcher asks

33

questions, (c) researcher forms categories, (d) researcher looks for patterns (theories),
and (e) researcher develops a theory or compares pattern with other theories. I
gathered archival data from the Academic programs catalogue (APPENDIX A) and
developed a series of questions that attempted to fill in additional program history. In
addition to program history, I was interested in understanding the faculty perspective on
student engagement and relevance of the program. Program faculty were interviewed
to gather intended programmatic and learning outcomes for MAET graduates, program
history and current tensions. Current and former MAET faculty members were solicited
for interviews. Nine faculty members were contacted for interviews and ultimately seven
faculty members were interviewed for this study. (N=7)
Interviews lasted 30-60 minutes, were recorded, transcribed, and then coded for
thematic analysis. (The entire interview protocol can be found in APPENDIX B.) The
interview protocol with current and former faculty members was designed and
implemented in order to determine the vision and expectations from the program. In the
section that follows, pseudonyms have been used when using direct quotes from faculty
members.

34

Table 2. Introduction to Faculty
Name

History with Program &
Responsibilities

Evan

Started teaching and developing
the certificate courses in 1994.
Has taught face-to-face,
overseas and online courses.
Has not taught with MAET
certificate program for 4 years.
(Currently has responsibilities
with PhD program.)

Ben

Started working with the
program in 1997. In his first few
years, he was the advisor for all
the masterʼs students in the
master's degree program then
went on to direct the certificate
program before moving to other
responsibilities two years ago.

Earl

Started working at the university
in 1987. The department was
oriented towards Educational
Psychology when he started and
then moved towards
Instructional and Educational
Technology in the early 90s.
Has taught face to face,
overseas and online courses
with MAET and is currently
teaching online in addition to
responsibilities with PhD
program.

35

Table 2 (contʼd)
Bill

Starting working with the
university in 2000. Has taught
face-to-face and online courses
with MAET and currently
teaches online courses in
addition to responsibilities with
PhD program.

Deepak

Started working with the
university 1998. Has taught
face-to-face, overseas and
online courses with MAET and is
currently teaching online and
directing the MAET program in
addition to responsibilities with
PhD program.

Derek

Started working with the
university in the fall of 2007. Has
only taught online with MAET in
addition to responsibilities with
PhD program.

Ian

Started working with the
university in 1998. Has taught
face to face, overseas and
online courses with MAET and is
currently teaching online in
addition to responsibilities with
PhD program.

What do Faculty Perceive as the Goals and Intended Outcomes of the Program?
The answer to this determines what faculty believed to be the end-result of the
MAET program. Analysis of responses clearly shows that early in the development of
the program there were two separate program goals - one for the certificate and the
other for the Masterʼs program. As one interviewee, Evan, who has been associated
36

with the program the longest time, said of the certificate, “it was initially seen as a
separate thing just meant to help teachers in the schools.” Prior to 2001, the certificate
program was a non-credit program that was offered for free to local school districts as
an outreach program.
In approximately 2001, this outreach morphed into what is now known as the forcredit certificate program. The curriculum in the certificate program consisted of an
introduction to the micro-computer (mouse skills, operating system proficiency, etc.) and
an introduction to word processing skills (mainly the Microsoft Office suite). The
certificate faculty consisted mainly of adjunct faculty who were K12 practitioners. As
Ben, one of the early directors of the certificate program, said, the aim was “helping the
people who had shifted from being avoiders to adopters of technology. So, aiming at
those people who were intimidated, who were scared by it, which was clearly a
certificate value.” This group of instructors did not interact with the MAET program
faculty, though the certificate course are 800-level courses, and are used as a feeder, or
“on-ramp,” to the Masterʼs program. The goal of the certificate program was to give
teachers a baseline for technology use.
The goals for the Masterʼs program at this time were not clearly articulated in the
history of the program. There were two versions of the Masterʼs program running, one
overseas and one on campus. The courses were not strongly tied together by
overarching goals or outcomes, with little consistency and coherence. That said, there
was a distinct emphasis on teaching and educational psychology in the Masterʼs
program (as opposed to the certificate courses). This shift between the Masterʼs and

37

certificate mindset can be seen in the following quote from a former certificate faculty
member, Evan, who said, (speaking of the certificate courses) that “there was not much
emphasis on shifting the strategy of teaching, it was more hereʼs how you use
technology to manage. We looked at the computer as the object, the medium, and the
manager of instruction.”
In the past few years there has been a concerted effort to bridge this historical
divide between the MAET program and the certificate program both from the faculty
perception point of view and pedagogical point of view. Additionally, the addition of the
NP (Educational Technology) endorsement option for Michigan teachers has
strengthened the consistency between course sites and delivery modes. Evidence of
successfully bridging this gap came through when one new faculty member (Derek) who
was interviewed was not aware that the programs were previously separate entities.
In its present form, in terms of high-level program goals, unanimously the focus of
faculty is to “make better teachers” or “to prepare the next generation of teachers to
teach well.” All interviewees focused on the teachers and how completing the MAET
program focuses on improving teaching first, technology second. Quotes that
exemplify this are Derekʼs observation “I think they gain knowledge about the way
students understand” and Deepakʼs input “the goal has been to emphasize teaching and
learning with technology.” Most of the faculty members teaching in the courses that
come after the certificate courses have formal backgrounds in educational psychology
and educational technology. This has meant that in the interviews the faculty have often
downplayed the emphasis on technology. For example, one faculty member, Bill, said

38

the program has “an emphasis on using technology thoughtfully, never with an
emphasis on the technology.”
An additional issue that faculty mentioned had to do with focusing on specific
technologies as opposed to ways of teaching and learning with various technologies.
This is in sharp contrast with the earlier certificate courses which often focused on
specific software tools (such as Microsoft Word or Excel). More recently, faculty indicate
an aversion of sorts to focusing on specific technologies, since as many of them said
either specifically, or words to the effect that “technologies change so frequently.”
With the divide between the MAET and Certificate programs addressed (by all but
one faculty member who was new to the university) a few other strands emerged after
analyses when asked what outcomes, knowledge and skills students gain by
participating in the program. While there was general agreement across all 7
professors, one faculty member, Ian, did say “Itʼs probably different goals for different
people. So, it sort of depends on the student.” A simple frequency analysis of the other
responses revealed two words that describe the program: practical and authentic.
Responses ranged from Evanʼs “practical introduction to computers” to Benʼs “a strong
emphasis on authentic materials” and “itʼs not just credits, students are able to apply
what they learn to their classroom.” Additionally in their own way, all faculty members
stressed that another one of the program goals, as Deepak says here, is to “help people
become a better teacher.” Derek sums up this feeling nicely by saying “I think in the
experience of gaining technology specific skills, in their own experience of that act of
learning it sort of changes their thinking about how their own students learn.”

39

What is the Pedagogical Knowledge that Faculty Believe Drives their Design of
the Program and the Student Experiences?
The faculty discussed the pedagogical knowledge they were trying to pass on to
the students, rather than their own personal pedagogy or philosophies of teaching. In
this query, pedagogical knowledge is defined as the processes and practices or
methods of teaching and learning (Koehler & Mishra, 2009). An analysis of responses
revealed three key strands driving faculty pedagogy. First, the faculty wants the
students to experience a shift in teaching; second, they want the materials students
create to be authentic; and third, they want to develop a sense of community among the
student body.
All of the respondents put a high value on shifting the teaching strategies of the
teachers in the program. Some even mentioned that technology was not necessarily a
part of their pedagogical decision making. As Bill said, the goal was that a student would
learn about technology “by playing with it.” Evan described his pedagogical decision
making was to ”affect how people teach, whether or not they use technology.” Deepak
described the program curriculum as being “something that would help people become
a better teacher, and how you could learn to use technology in that process.” He went
on to say, “itʼs that you develop a way of thinking and a way of being. That, to me, is
more important than learning specific theories of programs or software applications.”
No one, however, mentioned surveying existing practices or beliefs of their teachers.
One faculty member, Ian, did say his approach could be seen as a “way of either
enhancing what they already know or learning those things.” He added, “I think they

40

gain that [technical] knowledge, but that knowledge translates to a belief, too, that
teaching really has to be flexible and adaptive and have more than one approach to a
given outcome. I think they gain a belief that all people can learn things. That sounds
really cheesy, but I guess what I mean by that is that I think in the experience of gaining
technology‑specific skills in their own experience of that act of learning, it sort of
changes their thinking about how their own students learn.” This quote is representative
of the notion of changing the pedagogical beliefs of students and alumni.
In terms of materials created, faculty always put an emphasis on assignments
being practical and relevant to the teaching practice. Ben said, “that was explicitly
stated, that you could walk into an interview with this notebook and say here's how I use
technology in the classroom. “ A strong emphasis when the program started, as is now,
“is on authentic materials.” Ian recollected that the promise to teachers was that “you
were going to produce things that you could use or hopefully had used in your
classroom.” Two tensions arose in this area. Discussing program history clearly teased
out tensions among faculty as to what is a Masterʼs course in educational technology.
One faculty member, Evan said “it was a struggle on the master's side to decide what
constitutes a legitimate course in a master's degree and how much is the practical,
useful, appropriate thing to have taught in a master's program.” The pedagogical stance
on assessment of graduate student work is another topic of note. As Evan, one of the
people who started the certificate program went on to say, “In the certificate, grading is
not primarily an assessment activity, it is primarily a teaching activity.” A part of this
may be due to the historical nature of the certificate program which began as a non-

41

credit course, where completion was a measure of being successful in the program. Of
course once it because a credit program (both in the certificate and masterʼs versions),
this had to change.
The final pedagogical strand that emerged was the sense of imparting a spirit of
community building with and to the students. For the capstone course Bill “group(s)
people by subject matter because with portfolios, they just have so much more in
common.” In addition to creating community in subject matter and coursework, several
faculty members described the importance of giving the students the skills necessary to
become change agents. Deepak says, “What we should be driven by is, are they
prepared for being agents of change or being good teachers in the classroom that they
go back to.” Evan expressed that graduates are “going to be known as someone who
knows technology, you want to be able to help these other people.” Ian asserted, “you [a
graduate of the program] are going to be in positions of leadership so people will look to
you to be able to help with these things.” The goal, as Ben stated, is to “build a stronger
sense of community within their own schools, particularly teachers that take it as part of
a cohort where we may be offering the program in their local schools so they're learning
with their peers. Are they going to be prepared for that it is what we should be thinking
about.” This idea of community building was expressed strongly by all faculty members
interviewed.
What Types of Activities and Lessons do Faculty Create for Students to
Experience in the Program?

42

The types of activities and lessons have changed drastically over time. It is
important to note here that the nature of the program has changed over time as well. In
2009, the program became available completely online, which would explain some of
the shift in student experience. When asked about the early days (face to face) Ben
recalled assignments such as having a football coaches or art teachers inventory
equipment and supplies with Excel. Another important assignment was described by
Evan as “a notebook full of documents created specifically for use in the classroom”
such as “a letter to parents, a budget for a project that you would do in the classroom.
Again in the beginning it was just how do you use Word.” Ben explained, “They
[students] actually created a three inch notebook that they turned in at the end.” A focus
was more on the practical how-to over theoretical application. For example, Evan said
students had to “add two cells in Excel and save my worksheet.” So the initial history of
the program was quite driven by creating experiences for students where they could
learn how to use specific software programs – with less emphasis on how it could be
used for teaching.
There has been a shift in the past few years in better integrating technology for the
purposes of better teaching. The TPACK model, which argues for a contextual
integration between technology, pedagogy and content, developed by Koehler and
Mishra (2006) naturally fit into the MAET program. For example, in one core course
Derek explained “we do an audio interview using Audacity. The goal being, in that case,
looking at student understanding by using an audio recording and putting that into a
project. They do a digital story as well using pictures and PowerPoint slides and movie

43

clips to depict a learning experience.” In this example, the focus is not only on the
technology but also on the pedagogical and curricular ideas expressed through
technology.
Many of the faculty members put the focus on the learning process and product
rather than step-by-step “how-to” learn the technology. In one particular case, Deepak
explained, “we hardly ever spent time directly teaching about technology, on purpose.”
A specific example was shared: “So I would say your goal is to teach how to use a
camera and get some film about something. We would literally give them two‑minute
tutorials. Here's the record button, here's the rewind button, here's where the tape goes.
That's all you need to know. The rest you have to figure out.” He went on to say,
“theyʼre creating something to teach an idea. So I think just so long as we can keep the
emphasis on designing projects for educational purposes and then, you know, keeping
them thinking about what is that purpose and to what degree are you achieving it with
this technology.” Thus, the goal here is to expand the student understanding beyond
how to use an exact tool step-by-step, which sets them up for dealing with future
changes in technologies.
In brief, over time there has been a shift in the kinds of experiences students
undergo as they move through the program. Initially the focus had been mostly on the
acquisition of specific technology skills. Over time this has shifted to a broader focus on
teaching and learning with technology and the role that technology can play in that area.
What evidence do the faculty have to confirm that the above intentions are being
met?

44

When asked this question, Bill says, “I know I could make up a good answer to
that, but I really have no idea. I would love to know.” It was clear after speaking with
program faculty that no systematic evidence has been collected that can demonstrate
the success (or failure) of the Masterʼs program. As one senior faculty member, Earl,
noted, “weʼve not done research on the impact or outcomes of our program, and that's
been clearly a weak point. But that emphasis just changed over time, and we've just
gotten away from it [the evaluation].” The program collects historic graduation rates and
numbers, however, these numbers do not tell much of a story. While numbers have
increased and graduation rates are in line with college and department expectations,
simple enrollment and graduation rates are not a true measure of success.
As stated in the discussion of pedagogical reasoning among faculty, there was a
feeling by one faculty member, Ian, grading in many of the courses is viewed as a
teaching activity, not as an assessment activity. Evan highlighted this tension by saying,
the “concern has always been the certificate; too many people get four points. And so
it's helping instructors give the appropriate kind of feedback, and in effect, enable them
to give critical feedback. So, if somebody's worked hard and the instructors are tempted
to say, that's a good job, let me give you four point.” Because this approach is not
clearly articulated among the entire faculty, grades cannot be used as a measure to
indicate success or failure of the program.
All students must take the “capstone” course CEP 807. While the official schedule
of courses description reads “Perspectives on educational technology, current theories,
research findings, and methods of design and evaluation” this does not give much

45

insight into the nature of the course. The latest course description reads “our total focus
in this class is to help you each end the course with a web portfolio that shows in a rich,
thoughtful way the work you have done in your masterʼs program, and which is written
for your authentic audience, not just for us or for meeting the requirements of the
capstone portfolio course (Instructor, 2011).” Earl, who has experience with the CEP
807 course explained, “The portfolio requires them to put up examples of work from
every course. It is a terrific window on evidence of what the students report as the most
meaningful on the assignments they are most pleased to share. The capstone project
as a way of sort of pushing them, pushing them, pushing them to make their thinking
visible, to make their work visible,” Earl went on to say, “I do think that having all of our
students finished with defined web‑based portfolio is a huge plus. I don't think we've
marketed it well. So many programs you end up with nothing but a transcript.” Since
students do not produce a final thesis for the MAET program, the portfolio is the most
concrete example of student work and evidence of what they produce (and learn) in the
program. This emphasis on the professional portfolio is an important indicator of how
the program has changed over time—from measuring skills with specific technologies to
having graduates construct their own professional portfolio that showcases their creative
and professional work.
Throughout the interviews, faculty cited anecdotal evidence of success. As Deepak
says, “It's awesome because we hear back from so many people about projects that
they've done that they had fun there, they're doing it, and that is really something. But I
think, perhaps, maybe the best example of impact is the number of people who came

46

back and talked for us who went through the program.” In sum, after interviewing the
faculty and reviewing program records, there is no systematic benchmark or set of
criteria in place to assess program successes and failures.
Summarizing Faculty Perceptions
There have been significant changes in the program at multiple levels over time. In
the beginning the Certificate program was a non-credit certificate program and has
grown into a full-fledged masterʼs program offering courses in face-to-face, hybrid,
online, and off-campus formats. As the interviews with the faculty members indicate,
these changes are not only just in scale, but also of the goals and outcomes, the
experiences of the students and evaluation.
Initially the focus of the program was on having students learn how to use
technologies. Though there was an emphasis on classroom practice, the experiences
were typically designed to impart specific skills about specific technological (hardware
and software) tools. Over time this has shifted to a broader emphasis on ideas and
issues from educational psychology, student learning, and that of developing better
teaching practices. Technology now is a context within which broader and deeper
issues of teaching play out. This is also reflected in an emphasis on the program being
practical and authentic for the students in the program. This transition has, of course,
come with some inherent tensions: from learning specific technologies versus ways of
thinking with technology; from evaluating skills and completion of the courses as
indicators of success to a more professional portfolio model of student assessment.

47

CHAPTER 5
RESULTS: STUDENT AND ALUMNI PERCEPTIONS
Students (both current and alumni) are a vital source to consider in the
assessment of the program. They are particularly important because they are the
programʼs connection to the K12 system which the MAET program influences. The
goals of the program, as evidenced by program documents (websites etc.) and faculty
ideas, need to be tested against the actual experience of the students in the program.
Ideally, there should be agreement across these various indicators and stakeholders.
But if these stakeholders are not in agreement, the points of departure can help draw
attention to issues and concerns that need to be addressed in the ongoing
design/redesign of the program. Specifically, this research targets student attitudes
towards educational technology; their reasons for seeking these advanced degrees;
their experience in the program; and the perceived benefits to them of the program; and,
finally, their perceived proficiency in both using technology and in integrating it in their
teaching of subject matter.
General Descriptors
The online student survey was sent out via email to all current students and alumni
of the program. Over the past several years, the MAET program has maintained open
communication channels with their alumni through a variety of media, email newsletters,
Facebook, Twitter and the program website. Invitations to complete the survey were
sent by email to 400 individuals. Additionally, reminders to complete the survey were
sent via email, Twitter and Facebook. 147 individuals started the survey, 79% of whom

48

(117 individuals) completed it. Of the respondents, 64% were Female, (which is in
keeping with the gender distribution of students / alumni population). The average age
of the survey respondents was 37 years, ranging from the youngest at 24 years to the
oldest at 61. The data showed that 31% of respondents are currently enrolled in the
program (the highest population of responses) while the rest are alumni. Of all
respondents, the highest percentage experienced the program online (52%.) Of the
remaining responses, the next highest percentage of responses was from overseas
students with the on-campus hybrid students coming third. The majority (60%) of
respondents were K-12 teachers. The rest of the population that worked in conjunction
with the K12 population were administrators (2.8%), librarians (4.8%), and school
technology coordinators (9%) while 19.3% reported “other” as their profession. This
included Intermediate School District consultants, technicians, higher education support
staff, school psychologist, school nurse, and full time graduate students.
Figure 1. Student Population
School
Technology
Administrator/
Coordinator 13

University
Professor
3
Librarian
7

K-12 School
Administrator
4

K-8 Classroom
Teacher 44
Secondary/
High School
Teacher 43

49

For interpretation of the references to color in this and all other figures, the reader is
referred to the electronic version of this dissertation.

Figure 2. MAET Graduation Year
0

10

20

30

40

50

Have not graduated
2011
2010
2009
2008
2007
2006
2005
2004
2003
2002
2001
2000
never graduated

Besides general descriptors, this survey aimed to answer the main inquiry into how
do students and alumni perceive the quality and relevance of the program.
To answer this question, the following sub questions informed the survey:
a.

What do students perceive as the goals and intended outcomes of the
program?

b.

What do students experience in the program? How is this program relevant to
the professional lives of students?

c. What is the value that students derive from their participation in the program?
The analysis of responses to these questions is described in detail below.

50

Goals and Outcomes
What do students perceive as the goals and intended outcomes of the program?
Survey respondents were asked the open-ended question, “if you had to describe
this program to someone else, what would you say?” A first pass (N=117) was simply a
count of positive, negative, and neutral responses. Positive responses touched on
aspects of the curriculum and general enjoyment of the experience. One student said of
the program, “very cool, very useful, practical learning you can actually apply to
practice, rather than purely theoretical learning.” Another said that, “The MAET program
will enhance your teaching whether you teach kindergarten or high school. It puts the
use of technology into perspective in the classroom and gives practical applications for
everyday lessons.” 89% of student responses to this question were positive.
Statements such as “Learning to use technology in education” or “It allows
teachers to combine education and technology in a time where we revolve around being
technologically advanced,” were coded as neutral. These statements did not include
any qualifying terms that suggested the program was a positive or negative experience.
Negative responses centered on the lack of relevance to their teaching or on the
program not meeting the needs of non-teachers, and frustration with professors.
Specifically, one comment mentioned, “I would say it is a masterʼs program specifically
designed for teachers k-12. Enroll only if you are a teacher teaching k-12 classes.”
Statements such as “take the certificate overseas but do the rest of the courses online”
indicate a lack of satisfaction with the entire overseas experience indicating there is not
value in the entire overseas program. A statement by one overseas student stating that

51

the degree is a “Masterʼs in PowerPoint” reveals that the student felt the content was not
challenging or relevant. One respondent stated, “Donʼt go to the MAET program. I
have discouraged others from attending.”
A deeper analysis of the “how do you describe the program” question separated
online and overseas groups to see if there was a distinct difference in the experiences
of each group. Overseas student responses were heavily focused on a sense of
community. The following statement is a representative example of a positive response
from an overseas student, "Its[sic] an intense, 4-week summer program where you get
totally immersed in using, thinking about, creating and developing technology for
education. You come out with a sense of purpose and mission, and a basket of skills
that enable you to use technologies in your teaching an a whole new, empowered, way."
Overseas students explained the program as not only changing their practice, but
also their view of the world: “The MAET program is a fantastic, life changing experience.
Not only will you further your understanding of integrating technology into education, but
you'll experience people like you've never done before. I can't recommend it enough!?”
In contrast, comments from students / alumni who had experienced the
online/hybrid versions tended to focus on the ease of completing the program and its
flexibility. For example, one student explained the program as, “a masterʼs program you
can complete completely online. You have the freedom to complete your assignments
on your own schedule. There are very few quizzes which is nice and the projects you
complete are ones you can use in your classroom. It is a great program to further your
education. Plus, it is only 10 courses.” Though flexibility and ease of completion were

52

the primary descriptors for online students, respondents did make mention in their
descriptions that the online program influenced their practice. For instance, one student
commented that, “This is a program that develops creative thought and innovative
practice utilizing technology. It is a convenient program that can be done online.”
To further assess the perceived goals and outcomes students were asked to
explain ways they were asked by professors to demonstrate proficiency. At a curricular
level, respondents cited specific projects experienced in the program (i.e. STAIR
project, TPACK project, Action Research project), and all student references to projects
were MAET activities that took place across all versions of the program. Others
mentioned the creation of artifacts or tools such as websites, web quests, blogs, or
personal portfolios. It became evident, however, that a small subset of students (N=9)
felt as if they were not asked to demonstrate proficiency, as suggested by this comment
“Iʼm not sure we were asked to demonstrate integration: during the first two years of
program, the projects seemed as though they were designed more to prove proficiency
in a particular application.” Furthermore, the idea of achieving proficiency may have not
been communicated by the faculty, “during my time in the MAET program we never
really talked about in meaningful integration or what it looked like.” Of the nine students
who did not feel they were asked to demonstrate proficiency, six of the students were
not K12 teachers. These respondents argued that the program did not address their
needs and goals. As one participant said, “The program is too narrowly focused on
teachers and teaching and does not leave much room for other professionals to be able
to see a clear benefit.”

53

Student Experience
In order to determine what students perceived as program strong points
participants were asked: what are the strengths of the masterʼs program? Analysis
showed that students felt they had supportive instructors (N=17), they expressed the
relevance of the program to professional practice (N=19), and they emphasized the
value of peer support (N=18). For example, speaking of the kinds of faculty support
they received, one student said that, “I screwed up a lot, but my professors always
helped me to problem solve and learn why something worked or not. I always liked that
there was never a wrong answer per say (sic). We were given opportunities to creatively
connect technology to learning.” Speaking to the relevance of the program to their
professional practice, one student said that one of the strengths of the program was,
“Allowing each teacher to create a project specific to their own situation within the
framework of the MAET program. Much of what I created was thought provoking and
gave me real working lessons and ideas that I eventually implemented into my
teaching.” Furthermore, when asked program strengths, students expressed
appreciation of going beyond the technology, which helped them in their professional
practice, “the thing that I've always been most impressed with was the fusion of
technology and psychology. We didn't just learn what tools were available; we learned
WHY they would work.” Finally, the notion of community echoed strongly in student
comments. The following comment was typical of this line of thinking, “It helped plug
me into a huge network of educators who use technology and that are willing to share
their ideas.”

54

When asked if they experienced a shift in thinking by participating in the program
an overwhelming 91% (N=107) of students reported experiencing a shift in thinking.
One student said, “I now try to subvert existing technologies to render lessons more
engaging and fun.” The use of the word “subvert” indicates an inclusion of the TPACK
frame. Further inclusion of the TPACK model is explicitly stated here: “Well of course,
especially after the TPACK project. I think just the sheer exposure to so many
technologies makes you reevaluate your options and approach. Honestly, to me
technology was PowerPoint Jeopardy games before this program. Now, it's this
powerful tool with infinite potential.” Another student said, “I experienced this shift. I
actually feel guilty using lessons that I have not yet altered to include technology.”
Students who did not experience the shift cited having already experienced a shifting in
practice. For example one student said, “I don't think I experienced a shift in thinking as
I have always been someone, since my first days of teaching, that has integrated
technology into my lessons.” Another goes on to say, “I have not yet, but this is due to
the fact that I have always believed in putting technology in the classroom.” The other
students who did not experience a shift cited the fact that they were not K12 teachers.
What do students experience in the program?
When asked about the weaknesses or shortcomings of the program, an
overwhelming amount of responses expressed frustration with the faculty teaching the
courses. The primary concern was with the perception that the faculty teaching the
courses were often not sensitive to the actual teaching contexts and realities the
students were in and in the technologies being covered in the program. For instance

55

one student said that the, “Professors needed to be chosen more carefully: they need to
understand the realities of K-12 classrooms and be proficient in all the applications they
expect their students to learn and use.” Along the same lines, another student
suggested that the program ought to, “hire professors who are educators. Hire
professors who have good rapport with their students.” Specifically referring to the
online courses, one student wrote that, “The online courses were a hit or miss and the
majority of the students felt they were just going through the motions to get the units out
of the way but that they were not inspiring or beneficial to our teaching.”
Another area of concern was with the process of grading and process of providing
feedback to students on their work. Several respondents cited dissatisfaction with the
grading and feedback process experienced in the program. Citing a level of
inconsistency in this, one responded said that, ”Professors have also been very uneven
in providing feedback, with some providing regular, detailed feedback and others
seeming to be completely AWOL for long stretches of time.” This lack of regular and
consistent feedback was also expressed by another respondent, who said that, “I did
not get feedback in a timely manner.” Another went on to say, “The grading system
really needs improvement. My biggest gripe through the whole program was how each
professor either did or did not give feedback to students on their progress in the class.”
In addition to the lack of feedback was a concern with the uneven nature of the
feedback even when it was provided. There was a perception of some of the students
that the grading was somewhat lax and possibly inflated with (as one student wrote),

56

“Too many students all got the same grade even though the cohort knew that we were
not all of the same ability or put the same effort into it.”
There was also some frustration with the fact that different students have different
levels of technology expertise. Some felt that those who were highly proficient with
technology were often not challenged enough, while those who were less skilled were
often overly challenged. This frustration over the lack of a “level playing field” was
expressed by one student (clearly someone who had a high technology skill-set) as
follows: “Maybe [the program needs to] figure out some way for differentiation - so those
who have a stronger background knowledge of using technology are just as challenged
as those who are beginning to explore technology.”
As mentioned before, a clear concern of the small (but important) group of
students who were outside of the K12 classroom context was that the program was too
strongly connected to teaching and learning in K12 contexts. These students often
questioned the lack of attention or relevance of the program to their professional lives.
As one student said, “The program needs to 1) be limited to admitting only k-12
teachers or 2) be broaden [sic] so that other professionals can find ways to more easily
apply the skills learned to a wider range of scenarios jobs.”
A challenge specifically faced by students who took all or most of their classes
online had to do with a perceived lack of connection to the program and to the larger
student community. One online student did express, “Being an online student, I felt kind
of lost in the shuffle.” Another online student said, “I constantly felt like I was on an
"island" with regards to being part of the MAET program.” One student mentioned this

57

by specifically contrasting the online experience to the overseas experience, by saying,
“in the online courses in general there was much less of a sense of
community/cohesiveness amongst participants than in the overseas program.”
What difference did the program have to the professional lives of students?
Three main strands of positive responses emerged from this line of questioning:
leadership, confidence, and network of support. In terms of leadership, many report
taking on greater responsibilities in their current professional contexts, such as their
school districts or buildings. One student reported that due to their going through the
program, “I am often asked to give workshops on technology. I have a hand drawn sign
over my office door that says "tech support" since I help colleagues so much with
integration of technology in their work. The information technology staff sends people to
me.” Others reported a growth of confidence not only in technical skills, but also as
professionals. As one student reported, there had been a “complete paradigm shift” in
how she approached her work. This shift can be seen in the fact that “now I can say I
am not just surviving every day in my teaching, but I am consciously teaching and
engaging my students.” Finally, the theme of access to a community of learners or a
network of support emerged, “The MAET has introduced me to a whole new network or
faculty and peers that I hope to leverage in the future in my pursuit of future degrees
and job opportunities.”
Value
The survey solicited information in order to better understand what value students
derive from their participation in the program. To address this, the students were asked

58

the question, “why did you come to the program?” The choices for this multiple-choice
question were derived from the faculty interviews. Studentsʼ primary motivation for
coming to the program (N=93) was a desire for professional growth and development.
The remaining students chose certificate renewal (N=7), a pay increase (N=5) and
convenience (N=2).
Figure 3. Primary Motivation for Enrolling in the Program
Needed

Convenience 2
Word of mouth
2

Column1	
   credits for
certificate
renewal 7

Other 8

Needed
credits for pay
increase 5

Desire for
professional
growth and
development
93

Table 3. Descriptive Statistics for Primary Motivation for Enrolling in the Program
N
What was your
primary motivation
for enrolling in the
program?

116

Valid N (listwise)

Minimum
.00

Maximum
5.0

Mean
2.6983

Std.
Deviation
.93455

116

What is the value that students derive from their participation in the program?

59

Respondents were asked to select several “indicators of success” to determine
what difference their participation in the MAET program made in their professional life.
Respondents (N=117) were directed to select as many indicators as applied to their
individual situations. After graduation, 52% of students reported receiving a pay raise.
34% applied for new positions, and 20% received a new position. 12% of students
changed fields and the remaining 35% in the “other” category had not yet graduated
from the program, reported going on to higher education, or reported retiring from the
school system.
Figure 4. Indicators of Success After Graduation

Other 41
None of the
above 14

Change fields
14

Receive a pay
increase 61

Apply to for a
new position
40

Get a better
position 24

The survey went on to ask, “do you still feel connected to the MAET program?”
The question was posed on an 11-point Likert scale of “Not at All Connected (0)” to
“Extremely Connected (10)”. It was assumed that students who went through the

60

program in a face-to-face format would experience a much stronger connection to the
MAET program. Nonetheless, after running a comparison of means, no difference was
evident. In the descriptive analysis below, there is not a discernible difference between
ways students experienced the program. Also, students generally still feel somewhat
connected to the program.
Figure 5. Degree of Connection to Program

10.00

Degree

8.00

6.00

4.00

2.00
.00
online

hybrid
hybrid overseas
overseas domestic

domestic

Format of Program
Table 4. Descriptive Statistics for Degree of Connection to Program
N
Degree

116

Valid N (listwise)

Minimum
.00

116

61

Maximum
10.0

Mean
7.2155

Std.
Deviation
2.69559

The survey asked the following question, “How well did the MAET program
balance the learning of how to use specific technologies with how to teach with
technology in your practice?” This question elicited mixed responses from students.
Many reported a "good" balance, or that the program did “well” or “very well” but did not
provide supporting statements to clarify what they meant by "good." More reported a
focus on technology over pedagogy, "I felt like I learned more about how to use specific
technologies (online tools, collaboration tools, etc.) just a slight bit more than how to
actually teach with technology." Another respondent said, "I feel that the program is
probably a little heavy on the 'bells and whistles' of tech without addressing actual use
and application in the classroom." Similarly another student wrote that, "The balance is
more toward making sure we can use technology rather than teaching with technology."
Another felt “the MAET program did a poor job balancing learning new technologies and
how to use technology in our daily teaching. There was no focus on how to solve
technical problems, set up computers and other technologies for others.”
What evidence do we have that shows the program provides value?
In order to better understand and measure the value of the program, we piloted
and tested a measure (the Net Promoter Score or NPS) that has typically not been used
in educational contexts. The NPS has typically been used in business settings to
capture consumer perception of how well a business is doing by asking consumers if
they would recommend the business to other people. In essence it is a tool that
provides a gauge for the loyalty of a businessesʼ customer relationships (Reichheld,

62

2003). In brief, NPS is based on the fundamental perspective that every company's
customers can be divided into three categories: Promoters, Passives, and Detractors.
By asking one simple question -- How likely is it that you would recommend [Company
X] to a friend or colleague? Customers respond on a 0-to-10 point rating scale and are
categorized as follows:
•
•
•

Promoters (score 9-10) are loyal enthusiasts who will keep buying and refer
others, fueling growth.
Passives (score 7-8) are satisfied but unenthusiastic customers who are
vulnerable to competitive offerings.
Detractors (score 0-6) are unhappy customers who can damage your brand
and impede growth through negative word-of-mouth.

To calculate Net Promoter Score (NPS), take the percentage of customers who are
“Promoters” and subtract the percentage who are “Detractors.” The NPS can be as low
as -100 (which means everybody is a detractor) and as high as +100 (indicating
everybody is a promoter). Business research has shown that a positive score (i.e. any
score higher than zero) is good and an NPS of +50 is excellent.
For the context of study, respondents were asked one simple question: “How likely
are you to recommend the MAET program to a colleague or friend?” When calculating
MAETʼs Net Promoter Score it was found that 78% of respondents were promoters and
9% were detractors for a total net promoter score of +69. This is a very good sign since
scores around +50 are typically excellent. Of course, this is the first test of such a score
for the MAET program, however, it does indicate that overall the students see the
program as being valuable to them and that they would (on the whole) recommend the
program to others.

Summarizing Student Perceptions
63

Most students came to the program for learning more and achieving professional
growth (with smaller numbers coming to the program to maintain their certification,
salary increases and convenience). Given this motivation, it is good for the program to
know that, overall, the program seems to be meeting the needs of the students. For
instance, the Net Promoter Score indicates that there is much in the program that
students value. Some students described the program as being a “paradigm shift” and
that it had been a “life changing experience.” That said, the survey of the students also
indicated some fundamental issues and concerns. Broadly speaking these concerns fall
into three categories.
First, there was a feeling that the program did not do as good a job in terms of
being relevant to the professional context of the students. This was most strongly felt by
the students who were not K12 teachers as they often felt that the program was too
strongly targeted to K12 contexts of teaching and learning. These concerns were also
shared by some K12 teachers, particularly in their complaint that many of the faculty in
the program did not understand the issues at the heart of K12 teaching and wished for a
greater emphasis on those issues. There was a significant concern with the nature of
student-faculty interaction from many students, suggesting that the faculty did not model
the expectation of providing consistent and timely feedback.
Second, there were some important differences between the overseas student
experiences (which is primarily face to face) and those in the hybrid and online parts of
the program. Students in the hybrid and online versions of the program had a lower
level of connection to the program than those in the overseas version. Students in the

64

overseas version of the program were more likely to describe the community and
connection they developed both with each other and with the program. Students who
completed the program in the hybrid or online formats tended to instead speak to
convenience and flexibility.
Third, student statements indicate that they did not feel that the program balanced
the needs of using specific technologies with how to teach with technology. Student
responses indicate that they perceived a greater emphasis on the technology and skills
development than in teaching with these tools.

65

CHAPTER 6
COMPARISONS, RECOMMENDATIONS AND CONCLUSION
The previous two chapters presented the perceptions of the faculty and
students/alumni along these dimensions. This chapter focuses on the points of
alignment and difference between these perspectives. To reiterate, the third set of
research questions was as follows:
What insights can be gained when comparing data between faculty and students?
a.

What similarities exist between faculty and student perceptions?

b.

What tensions arise between student and faculty perceptions?

c.

What ideas or suggestions for the future directions of the program emerge?
Comparisons

As stated earlier in the analysis, there are many stakeholders in the MAET degree
program, however, the faculty and students/alumni are the cornerstones of the
programʼs success or failure. Comparing the viewpoints between these two
stakeholders is an important step in gaining a clearer picture of the quality and
relevance that the MAET degree provides.
What similarities exist between faculty and student perceptions?
Faculty asserted that they were changing or influencing the teaching practices of
their students, and this was confirmed by the survey. When asked if they experienced a
shift in thinking by participating in the program an overwhelming 91% (N=107) of
students reported experiencing a shift in thinking. One student said, “I now try to

66

subvert existing technologies to render lessons more engaging and fun.” The use of the
word “subvert” indicates an inclusion of the TPACK frame.
A sense of community and collaboration was another strong point of alignment
between student and faculty views. Both groups believe that the program is helping
students achieve professional growth. While both groups had positive descriptions of
the program, when comparing answers to how one would describe the program to
someone else, one difference was seen in the degree of quality descriptors. There was
a lack of overt enthusiasm on behalf of the faculty. While the faculty described the
program as doing a “good” job, many of the students used descriptors like “awesome”
“life changing” “amazing” whereas the faculty were much more reserved in their
responses. Both groups emphasized the strong K12 focus, practical preparation for
teaching, and sense of community.
What tensions arise between student and faculty perceptions?
The strongest divergence between groups was on the question of the primary
motivation for obtaining a masterʼs degree. While most faculty did mention that they
believed students came to the program to learn about educational technologies and
grow on a professional level, most faculty asserted that they believed the primary
reason students came to the program was for credit renewal and a salary increase.
Conversely, students and alumni responding to the same question put a higher
emphasis on desire for professional growth and development (N=93) with a salary
increase (N=5) and credit renewal (N=7) lower down the list. This divergence is
interesting because it means that the students are actually better prepared for the kinds

67

of transformational changes the faculty are interested in developing in their students
than the faculty actually think they are. If a majority of students are not in the program
for merely instrumental reasons it implies that the faculty can push further into the area
focusing on how technologies can change educational practice.
A second point of tension arose when faculty were asked about specific
assignments. During questioning, faculty did not mention how they provided feedback
to the students. This is clearly a tension among the student body. This issue of
receiving consistent and timely feedback was of great concern to the students, and
dominated the majority of negative comments and experiences in the program. One
student said, “The grading system really needs improvement. My biggest gripe through
the whole program was how each professor either did or did not give feedback to
students on their progress in the class. Waiting until the end of the class doesn't fit the
concept of learning to achieve and understand.” Another goes on to say “I'm the type of
student that yearns for instant feedback. I want to know right away whether or not I did
well on a project. It took a lot of patience for me to not hound my professors for their
opinion on my projects! I know one of the focal points of the program is to evaluate
students on their progress and doesn't center on grades, and I had a hard time waiting
for a response and being okay with comments and suggestions, rather than a grade. :)”
One can almost see a relationship between the misperception that faculty have
about why people came to the program and the lack of emphasis given to feedback. As
the interviews indicated, most faculty believe that students come to the program not for
their personal or professional development, but rather to maintain certification or to get

68

raises. In such a situation the program goals (at least as perceived by the faculty) would
be to get students through the program, since learning is not a perceived priority.
Students on the other hand emphasized the importance of professional development as
a key objective of joining the program and hence valued timely, reliable and consistent
feedback (over stressing over end of semester grades, which often became the only
way they received any input from the faculty on their work). Clearly the faculty need to
become more aware of the priorities of their students and how they can help them
develop professionally through timely and consistent feedback.
Another tension mentioned by faculty and students is the different ability levels of
incoming students. One student states, “Some of the students that wish to apply to the
program are not as advances [sic] as many of the current students. The speed of the
class does have to have challenges for each individual, yet student achievement and
student enrollment will necessitate differentiation.” Another goes on to say, “There was
no fear of failure and so the standard for an MAET degree is lessened. Some graduated
who were not proficient at technology.” Faculty expressed this tension as well, Evan
states, “if somebody's worked hard and the instructors are tempted to say, that's a good
job, let me give you four point.” Another student says, “I would hate for the program to
be seen as something that anyone can do and no one will fail- in other words, a giveaway degree that you can learn a incomparable amount from and grow professionally,
or, if you desired, could come "play" at for three summers and enjoy the benefits of the
MA at the end of your title. I feel the course should be as difficult and challenging as it is
interesting and informative.” This comment alludes to students of all ability levels who

69

can perform at any level and still receive a passing grade, watering down the degree.
This tension is one that resonates across all graduation years and online, overseas and
hybrid delivery formats.
Clearly this is an important issue for the faculty to consider when designing student
experiences. The design of the curriculum needs to take into account the differing skill
levels of students and aim to develop consistent evaluation standards that are
transparent and fair so that student concerns in this area can be addressed.
Recommendations
Though it appears that the students/alumni are satisfied with their experience in
the MAET program, this is not enough to ensure program success, particularly in the
highly competitive and changing world of higher education. Only 30% of the possible
population responded to this evaluation. It is clear that the majority of the students who
had a positive experience with the program responded to the survey; however, more
systematic evaluation of ALL students is needed to capture the true pulse of the
program. Ultimately, continuing with these informal assessments of the MAET program
does not set the program up for long-term success.
What ideas or recommendations for the future directions of the program
emerge?
The strongest recommendation for change comes from in increasing the
population of non-K12 practitioners within the MAET student body. These students
strongly suggest the program create more relevant experiences for non-K12
practitioners. As evidenced in the survey data, however, MAET is achieving success

70

with K12 teachers and does not want to disservice one population by catering to
another. Possible ways to address this problem are to track non-K12 educators in
specific sections or cohorts. Additionally, easy-to-implement, small-scale curriculum
changes can be made. For example, many opening activities in online classes include
introducing yourself followed by asking what class the student teaches. By removing
the “what do you teach” question and asking, “what do you do,” the course instantly
makes a better first impression and is inclusive of all students.
Another suggestion to faculty is that they explicitly state and describe their
feedback mechanisms and practices in their courses. If feedback channels are clearly
explained in online courses or on the syllabi, this dramatically decreases the fear and
frustrations of the students. (Rowe & Wood, 2008; Bonnel, Ludwig & Smith, 2005) This
does not require faculty to change their practice by increasing the amount of feedback,
but rather asks that they clearly communicate the ways students are evaluated.
Another recommendation to add to the depth of the evaluation is to perform a
study of the MAET capstone portfolios. Every MAET student creates an online portfolio
in CEP 807. The content of the portfolios presents a fascinating longitudinal study of
the MAET program. Analysis of these portfolios can provide insight into many issues
such as the depth of understanding a student displays in the MAET program, curricular
connections, and technical or curricular proficiency. A similar study conducted with
undergraduate teacher interns (Rosaen & Wolf, 2005) provides a stage for this work,
and could also be contrasted with understandings of novice teachers.

71

Moving forward, more thorough documentation on reasoning behind curriculum
changes and programmatic visions need be laid out. As a program, a statement of
purpose (for lack of a better term) needs to be articulated. There are many answers to
the question “what is the MAET program?” Formulating these into a concise statement
will be helpful not only for program promotion and marketing, but for the future growth
and success of the program. If more than one person can clearly articulate what the
program is and how the program came to be, then the base of program knowledge
spreads allowing more faculty and staff members to help the students understand how
their individual courses fit in the MAET framework.
Limitations
Only full time program faculty were solicited for interviews in this study.
Historically, full time tenure-stream faculty taught the bulk of the MAET program, while
the Certificate courses were taught by adjunct K12 practitioners who had completed the
MAET program. While the Certificate courses are still primarily taught by MAET alumni,
the remaining 800-level courses have moved into a training ground for Educational
Psychology and Educational Technology (EPET) PhD students who are mentored by
faculty. These students gain valuable teaching online experience while working with
content developed by the faculty. Additionally, with program enrollment at such high
levels, EPET alumni have been hired as adjuncts to lead the teaching and development
of courses. With this expanding model in place, it is imperative that the teaching
assistants and adjuncts understand not only the MAET program as a whole, but also the

72

goals and outcomes of each of the courses along with the intensions and goals of the
students in the course.
Select TPACK survey questions were piloted in the student survey; however, the
data did not provide usable results. The integration of TPACK into the MAET program
did not emerge until approximately 2008 and this could explain the lack of usable data.
Additionally to truly capture the complexity and understanding of the integration of
TPACK into the MAET, students should be given the TPACK survey on intake into the
program and then given the survey again at specific points in the MAET program. This
data can then be compared, contrasted, and then added to the existing body of TPACK
research.
While the degree to which students and alumni felt connected to the program in
general was measured, a missing measure is the degree to which students feel
connected to peers and professors in the program. This is an increasingly important
benchmark given the growth of social networks (i.e. Twitter and Facebook) where
informal and formal learning takes place. Gathering this data could be a unique and
important statistic that sets our program apart from other institutions offering similar
degrees.
I took precautions such as ensuring anonymous survey responses and stripping
faculty responses of identifying information, however, there is a clear connection
between my work as coordinator of the program and my objectivity towards the data.
Students attending the program from 2008 forward may know me and this relationship
could have contaminated the results. This inquiry is based on a small scale and makes
no claims of generalizability. With a response rate of N=117, only 30% of the potential
73

population (N=400) responded to the survey. This low response rate could mean that
data reported is not representative of the entire student and alumni body. Consequently,
it is highly unlikely that the results found in this study are representative of other
master’s degree programs.
Connecting to deeper issues of program evaluation
The first chapter of this dissertation presented the two key reasons for conducting
a study of this nature. The first was to inform programmatic decision-making, and the
second was to create a climate for organizational learning. This was framed within the
context of program evaluation. Further, it was argued that program evaluation was a
wicked problem with two key characteristics. These are that wicked problems are deeply
connected to context and that different stakeholders have differing perspectives of the
same problem. Thus, the focus of this research would be to situate it within a particular
context (the MAET program) and to study two key stakeholdersʼ perceptions of the
program. Previous studies of Masterʼs degree programs had focused on either one
group of stakeholders or the other with a few that had focused on both.
The two key constructs that were regarded worthy of study in previous studies of
Masterʼs programs were that of the quality of the program and its relevance to the
students. Though quality and relevance were seen to be difficult to define, the review of
the literature indicated that the ideas of quality and relevance could be operationalized
through focusing on goals and objectives; the pedagogical value; the nature of
experiences designed; and the kinds of evidence used to confirm if the original goals
and objectives are indeed being met.

74

To remain viable, programs must identify key criteria important to their student
population. While master's programs explicitly offer students a degree, implicitly they
offer students a quality experience and relevant curriculum. While the constructs of
quality and relevance are moving targets (especially in the field of educational
technology) it should not exclude programs from attempting to define these terms in
their own contexts. By focusing on these constructs programs can remain in-tune with
their audience, a skill that is necessary for survival in the changing academic
marketplace eluded to at the beginning of this dissertation.
If the ultimate goals are to inform programmatic decision-making and to develop a
climate for organizational change, the points of alignment and divergence between
student/alumni and faculty can be a good place to start. Clearly these differences can
be due to a variety of reasons. It could be a lack of communication of goals and
objectives, a lack of understanding between the stakeholders, a lack of knowledge of
who the students are and what their goals and objectives are for joining the program.
The recommendations laid out above can go a long way towards making the program
successful and to establish a better connection between faculty goals and objectives
and those of the students.
Finally, this particular research study can be seen as a case of a limited form of
program evaluation that (a) focuses on the context of one particular program; and (b)
systematically investigates the similarities and differences between faculty and student
perceptions of the programʼs goals, objectives, value, nature of experiences designed
the kinds of evidence used to confirm if the goals and objectives are being met. The

75

data and analysis show that though there may be some alignment between these
overarching goals between the faculty and students, there are the some key differences
in perception – having to do with (a) why students join the program; (b) how the
curriculum is designed and implemented. More specifically, most students appear to
have joined the program for professional development though most faculty do not see
this as being the primary reason. Additionally, the student experience is a lot more
divergent than the faculty are aware of – both for non-K12 teachers and in terms of
receiving consistent and timely feedback. There also appears to be differences between
the studentsʼ experience and value depending on where (online or hybrid or face to face
overseas) the program is offered. Students in the online and hybrid versions appear to
value flexibility and are concerned with a lack of community while students in the
overseas version emphasize the development of community and professional
relationships. Clearly more needs to be done to ensure a common and consistent
educational experience to all these groups. Finally, many of these issues have to do
with the program itself changing over time, starting with a non-credit certificate,
technology skill acquisition focus to a for-credit program being presented in multiple
locations. Though this is clearly a change for the better (and consistent with what
students/ alumni appear to value) it brings issues of clarity and consistency of the
program goals and objectives to the forefront.
Conclusion
The goals of this study were to inform programmatic decision-making, and to
create an example for organizational learning that could be useful to others beyond this
particular study. The intents of this inquiry at a local level were to gain insight into the
76

local program, and capture a baseline of the perception of the quality and relevance of
the Master of Arts in Educational Technology degree at Michigan State University.
Nonetheless, by using data to inform programmatic decision-making, precedence is set
that models data-driven decision-making. Focusing on the perception of faculty and
students operationalized the ideas of quality and relevance. By researching these two
perspectives we have a better understanding of how these perceptions align and
contradict each other to inform the program’s future development. At the end of this
inquiry, one thing was consistently clear: the MAET program has been successful
because of change. To remain successful, marketable, and viable in the academic
marketplace, programs must continue to take into consideration stakeholder viewpoints
and continue to define quality and relevance within their contexts to inform
programmatic and curricular changes.

77

APPENDICES

78

APPENDIX A
Master of Arts in Educational Technology Program History
APPENDIX A
HISTORICAL MAET PROGRAM DATA
The history of the MAET program reflects the conflicts mentioned in the educational
technology field mentioned earlier. This historical data will be critical when conducting
interviews as the perception of quality in meeting student implied needs may change
depending on graduation year.
2000-2002
EDUCATIONAL TECHNOLOGY AND INSTRUCTIONAL DESIGN
The Master of Arts degree program with a major in educational technology and
instructional design prepares teachers, administrators, and other educational
professionals for the thoughtful use of a variety of technologies to support teaching and
learning in today's educational environments. The program draws on current theories of
learning and development to understand the role of technology in the learning and
trans- forming of valued content. The goal is to support educators in their efforts to
prepare students who are technologically liter- ate problem solvers and avid learners. In
addition to meeting the requirements of the University and of the College of Education,
students must meet the requirements specified below.
2002-2004
TEACHING AND LEARNING WITH TECHNOLOGY
Master of Arts The Master of Arts degree program in teaching and learning with
technology prepares education professionals for the thoughtful use and design of a
variety of technologies and technology based environments to support teaching and
learning. The program draws on current theories of learning and development to understand the role of technology in learning and instruction. Students focus their studies in
one of the following emphasis areas: Teaching with Technology in School Settings. This
emphasis area is intended primarily for teachers interested in the use of technology to
deepen student understanding of subject matter and enhance student problem-solving
ability. with majors in measurement and quantitative methods are de- signed for
students who are interested in applying the skills and knowledge that they acquire in
school systems, colleges, universities, state and federal departments of education, civil
service departments, test companies, and research organizations. Master of Arts The
master ʼs degree program in measurement and quantitative methods provides
opportunities to study quantitative methods and techniques of program evaluation.
Students take course work in assessment, test construction, data analysis, and
psychological foundations. In addition to meeting the requirements of the University and
of the College of Education, students must meet the requirements specified below.
2005 – 2007
EDUCATIONAL TECHNOLOGY
79

Master of Arts The Master of Arts degree program in educational technology prepares
education professionals for the thoughtful use and de- sign of a variety of technologies
and technology based environments to support teaching and learning. The program
draws on current theories of learning and development to understand the role of
technology in learning and instruction. Students focus their studies in one of the
following emphasis areas: Teaching with Technology in School Settings. This emphasis
area is intended primarily for teachers interested in the use of technology to deepen
student understanding of subject matter and enhance student problem-solving ability.
Learning, Design and Technology. This emphasis area is in- tended for educational
professionals interested in the design and evaluation of technology-rich on-line and
desktop learning environments, including web-based and other multimedia tools.
2007-2009
EDUCATIONAL TECHNOLOGY
The Master of Arts degree program in educational technology prepares education
professionals for the thoughtful use and de- sign of a variety of technologies and
technology based environments to support teaching and learning. The program draws
on current theories of learning and development to understand the role of technology in
learning and instruction. Students focus their studies in one of the following emphasis
areas: Teaching with Technology in School Settings. This emphasis area is intended
primarily for teachers interested in the use of technology to deepen student
understanding of subject matter and enhance student problem-solving ability. Learning,
Design and Technology. This emphasis area is in- tended for educational professionals
interested in the design and evaluation of technology-rich on-line and desktop learning
environments, including web-based and other multimedia tools. An Educational
Technology (NP) endorsement can be added to either an elementary or secondary
certificate by completing the requirements for MSU's Master of Arts degree program in
Educaional Technology or can be obtained by completing a prescribed 18-credit
program in educational technology.
2009 – PRESENT
EDUCATIONAL TECHNOLOGY
The Master of Arts degree program in educational technology prepares students for the
thoughtful use and design of technology in various educational settings. The program
draws on current theories of learning and development to understand the role of
technology in learning and instruction.
The program is offered in several different formats and allows flexibility to accommodate
both full-time students and working professionals by offering accelerated summer study
programs, online and hybrid courses.
An Educational Technology (NP) endorsement can be added to either an elementary or
secondary certificate by completing the requirements for MSU's Master of Arts degree
80

program in Educational Technology. For more information on the NP endorsement, visit
www.edutech.msu.edu.

81

APPENDIX B
Faculty Interview Questions
What course do (did) you teach? In regard to this course, what skills or knowledge do
(did) students gain in your course?
<How does this course(es) fit in with the overall masterʼs program?>
If you had to describe this program to someone else, how would you do it? (Audience?
Goals? Purposes? Outcomes?)
What are the intended learning outcomes of the program? That is, what are the goals of
the program in terms of what students are to learn to do, understand, believe, or value?
I want to focus a bit more about program goals… Why do you think students come to
our program?
In practice, what are the outcomes students gain from being in the masterʼs program—
the knowledge, skills, understandings, beliefs, values, or career aspirations that they
gain?
What are any other benefits students may derive from participation in the masterʼs
program?
What kinds of evidence do we have regarding these outcomes?
What differences will this outcome make in the lives of those who complete the program
<Why is this important? >
How well do you think the program is in meeting studentsʼ interests and needs?
There is always the tension between teaching technology (technology) and teaching
how to integrate technology (pedagogy), what are your thoughts on these tensions?
Give an example of relate this to something – say teaching dreamweaver v.s. building
webpages for teaching? Or whatever?
There is also a perceived tension between theory and practice. Do you perceive such
tensions in this program? If so, how do they play out < What are your thoughts on these
tensions?>
What are the strengths of the masterʼs program ?
What are the weaknesses or shortcomings of the program?
What are some key challenges faced by the program in the past?
82

<Organizational, social, global etc.>
Have these led to changes? In what way?
How has the masterʼs program changed over the years?
Are you aware of any ways in which the program is currently assessed?
<If you know of any processes, instruments, or methods, how effective have they been
in measuring outcomes from participation in the program?>
What are some current challenges?
How do you perceive that these concerns have been addressed?
<Any ideas on how to meet them?>
What are your ideas for new directions for the masterʼs program.

83

APPENDIX C
Student Survey

<STUDENT SURVEY PAGE 1>
Gender (drop down list)
Age (drop down list)
MAET Graduation Year (drop down list)
Where did you take the MAET program (check all that apply)
* Online
* East Lansing
* School District in Michigan
* Traverse City
* France (Valbonne)
* France (Rouen)
* England (Plymouth)
* Mexico City
* Thailand
What is your profession
K-8 Classroom Teacher
Secondary/High School Teacher
K-12 School Administrator
Librarian
School Technology Administrator/Coordinator
Community College Instructor
University Professor
Retired
Other
After graduation did you (check all that apply)
Receive a pay increase
Apply to for a new position
Get a better position
Change fields
None of the above
Other:

84

<STUDENT SURVEY PAGE 2>
Following questions in matrix form with scale:
Strongly Disagree - Disagree - Neither Agree or Disagree - Agree - Strongly Agree
I know how to solve my own technical problems.
I can learn technology easily.
I keep up with important new technologies.
I frequently play around the technology.
I know about a lot of different technologies.
I have the technical skills I need to use technology.
I can choose technologies that enhance the teaching approaches for a lesson.
I can choose technologies that enhance students' learning for a lesson.
The MAET program has caused me to think more deeply about how technology could
influence the teaching approaches I use in my classroom.
I am thinking critically about how to use technology in my classroom.
I can adapt the use of the technologies that I am learning about to different teaching
activities.
I can select technologies to use in my classroom that enhance what I teach, how I teach
and what students learn.
I can use strategies that combine content, technologies and teaching approaches that I
learned about in my coursework in my classroom.
I can provide leadership in helping others to coordinate the use of content, technologies
and teaching approaches at my school and/or district.
I can choose technologies that enhance the content for a lesson.
I can teach lessons that appropriately combine my subject matter, technology and
teaching approaches.
My MAET professors appropriately modeled combining content, technologies and
teaching approaches in their teaching.
How often were you asked by instructors to demonstrate that you could integrate the
use of technology into the lesson plans? (open ended text)
Some educators claim that access to new technology in the classroom forces teachers
to rethink the nature of what they do in the subject matter domain. How often did you
experience this in course of the program? (open ended text)
How well did the MAET program balance the teaching of specific technologies with the
pedagogy of teaching with technology? (open ended text)

<STUDENT SURVEY PAGE 3>
What was your primary motivation for enrolling in the program?
85

Needed credits for certificate renewal
Needed credits for pay increase
Desire for professional growth and development
Word of mouth
Convenience
Other:
After graduation did you (check all that apply)
Receive a pay increase
Apply to for a new position
Get a better position
Change fields
None of the above
Other:

How well did the MAET program meet your needs? (open ended text)
What are the strengths of the masterʼs program? (open ended text)
What are the weaknesses or shortcomings of the program? (open ended text)
What difference did participation in the MAET program have to your professional life?
(open ended text)
What is one thing you thing the program did well? (open ended text)
Where can the program improve? What did we do poorly? (open ended text)

<STUDENT SURVEY PAGE 4>
If you had to describe this program to someone else, what would you say. (open ended
text)
Do you follow MAET on any of the following social media channels:
Twitter (@maet)
MAET Newsletter (http://edutech.msu.edu)
MAET Facebook Page
Do you still feel connected to the MAET program
Yes
No

86

Please explain why or why not (open ended text)
How likely are you to recommend the MAET program to a colleague or friend?
scale
1- Not at all likely
10 - Extremely Likely

87

APPENDIX D
Faculty Interview Informed Consent Form
You are being asked to participate in a study to gather data on the Master of Arts in
Educational Technology program at Michigan State University. In its 10-year history,
there has never been a systematic evaluation of the Master of Arts in Educational
Technology program. Given this context there is an urgent need for developing formal
and informal mechanisms for program evaluation in order to evaluate the effectiveness
of
the current MAET program. I will conduct interviews with past and current faculty in
order to learn the history of the MAET program and to develop a conceptual framework
that will led to clusters of intended learning outcomes that will be used to develop a
survey for MAET students and alumni. Only summary data about the pedagogical
development of the MAET program will be reported. Individual comments or exact
quotations from the interviews will not be reported and participants will not be identified
in the summary report.
Researchers are required to provide a consent form to inform you about the study, to
convey that participation is voluntary, to explain risks and benefits of participation, and
to empower you to make an informed decision. You should feel free to ask any
questions
you may have. You must be 18 years or older to participate in this research study.
STUDY TITLE: Master of Arts in Educational Technology - Historical Inquiry
POTENTIAL BENEFITS
The interviews/participation will help the MAET faculty understand the historical
context and development of the program over time.
POTENTIAL RISKS
There are no foreseeable risks associated with participation in this study.
PRIVACY AND CONFIDENTIALITY
The data collected for this project will be confidential. Any information about
you, including your name and email address and audio recording (if any) will be
kept confidential to the maximum extent allowable by law.
YOUR RIGHTS TO PARTICIPATE, SAY NO, OR WITHDRAW
Participation in this research project is completely voluntary. You have the right
to say no. You may change your mind at any time and withdraw. You may choose
88

not to answer specific questions or stop participating at any time.
COSTS AND COMPENSATION FOR BEING IN THIS STUDY
Participation in this study will not entail any costs to you (beyond the time you
spend during the interview).
If you have any questions about this study, please contact the researcher, Leigh Graves
Wolf.
Researcher Contact Details: Leigh Graves Wolf, PhD Student, Department of
Counseling, Educational Psychology and Special Education, Michigan State University
Address: 509D Erickson Hall, East Lansing, MI 48824. Phone: 517-432-7195. Email:
gravesle@msu.edu
If you have questions or concerns about your role and rights as a research participant,
would like to obtain information or offer input, or would like to register a complaint
about this study, you may contact, anonymously if you wish, the Michigan State
University's Human Research Protection Program at 517-355-2180, Fax 517-432-4503,
or e-mail irb@msu.edu or regular mail at 202 Olds Hall, MSU, East Lansing, MI 48824.
Your signature below indicates your voluntary agreement to participate in the interview
and also to have the interview audio taped.
This consent form was approved by the Social Science/Behavioral/Education
Institutional Review Board (SIRB) at Michigan State University.
Approved on 8/24/2009 – valid through 8/24/20102. IRB# x09-775
APPENDIX E
IRB Approval Documents
APPENDIX D
IRB APPROVAL DOCUMENTS
Amendment to IRB# x09-775/ APP# i033884
3 messages
leigh graves wolf <gravesle@msu.edu>
Sun, Nov 28, 2010 at 4:30 PM
Reply-To: gravesle@msu.edu
To: irb@msu.edu
Cc: Punya Mishra <punya@msu.edu>
I am writing to inform the IRB of an amendment to my project IRB#
89

x09-775 which was determined exempt on 8/24/09.
I am adding a web based survey and sending out, via email, to a
subject population who are alumni of our Master of Arts in Educational
Technology program. No personal data will be collected from the
survey and the risk factor remains minimal for participants.
The following answer would be amended on my IRB application
20a. Subject Population
Current: Master of Arts in Educational Technology faculty (past and present) at
Michigan State University. Addition: Alumni of the Master of Arts in Educational
Technology
program at Michigan State University.
The revised consent form for the new population and online survey are attached to this
email.
Thank you Leigh Wolf
--Leigh Graves Wolf, Program Coordinator
Master of Arts in Educational Technology
Michigan State University, School of Education
phone 517.432.7195 web: http://edutech.msu.edu
Was this interaction helpful?
Please take our 10-second survey! http://goo.gl/tUic

5 attachments
consent_revisedNov2010_wolf.doc
MAET Alumni Survey 2010_page4.pdf
MAET Alumni Survey 2010_page3.pdf
MAET Alumni Survey 2010_page2.pdf
MAET Alumni Survey 2010_page1.pdf

IRB <IRB@ora.msu.edu>
Fri, Dec 3, 2010 at 2:47 PM
To: gravesle@msu.edu
Greetings Investigator(s),

90

Thank you for your email in regards to your research amendment. With the review of
your proposed changes you may continue with your research as planned
(Exempt). With the information and materials disclosed, this research can still be in the
exempt category as it does not pose any real risk to subjects.
Thank you,
Steven Smith
IRB Staff
884.6019

91

REFERENCES

92

REFERENCES
Bergman, R., Shatzman, H., & Danon, A. (1982). Follow-up study of the Tel Aviv
University nursing graduates. International Journal of Nursing Studies, 19(2), 7988.
Blackwell, P. & Diez, M. (1998). Toward a new vision of masterʼs education for
teachers. Washington, DC: National Council for Accreditation of Teacher
Education.
Bonnel, W., Ludwig, C., & Smith, J. (2005). Providing Feedback in Online Courses:
What Do Students Want? How Do We Do That? Clinical Nursing Education, 205222.
Burke, J. C. (2005). Achieving accountability in higher education: balancing public,
academic, and market demands. San Francisco, CA: Jossey-Bass.
Clark, K. E. (1979). The issue of quality in master's degree programs. In the Council of
Graduate Schools Proceedings of the Conference on the Assessment of the
Quality of Master's Programs (pp. 44-51). College Park: University of Maryland.
Clark, M.M. (2006). An assessment of the community college leadership doctoral
program at Mississippi State University as perceived by former and current
students. Unpublished doctoral dissertation, Mississippi State University.
Conrad, C. & Eagan, D. (1990). Masterʼs degree programs in American higher
education. In John C. Smart (Ed.), Higher education: handbook of theory
and research, 6. New York: Agathon Press.
Conrad, C., Haworth, J., & Millar, S. (1993). A silent success: Masterʼs education
in the United States. Baltimore, MD: The Johns Hopkins University Press.
Cooperative Educational Service Agency 6. (2010). ISTE NETS Pre-Survey. Retrieved
from http://www.cesa6.k12.wi.us/products_services/elearning/netspreservey.cfm
Creswell, J. W. (1994). Research design: qualitative and quantitative approaches.
Thousand Oaks, CA: SAGE.
Cuban, L. (2001). Oversold and underused: reforming schools through technology,
1980- 2000. Cambridge MA: Harvard University Press.
Culp, K.M., Honey, M., & Mandinach, E. (Jan 2003). A retrospective on twenty years of
education technology policy. Commissioned by the American Institutes for
Research to inform the development of the National Education Technology Plan.

93

Washington, DC: U.S. Department of Education, Office of Educational
Technology.
Earle, R. S. (2002). The integration of instructional technology into public education:
Promises and challenges. Educational Technology, 42(1), 5-13.
Erdogan, Y. (2008). An evaluation of web based instruction in view of the tutorsʼ and
studentsʼ perspectives. The Turkish Online Journal of Distance Education, 9(2),
86-96.
Harvey, L., & Green, D. (1993). Defining Quality. (C. Pollitt, Ed.) Assessment &
Evaluation in Higher Education, 18(1), 9-34
International Society for Technology in Education ISTE (2008) NETS
Jackson, J.W. (1994). A model for evaluation of doctoral programs in education.
Unpublished doctoral dissertation, Tennessee State University.
Johnson, L., Smith, R., Willis, H., Levine, A., and Haywood, K., (2011). The 2011
Horizon Report. Austin, Texas: The New Media Consortium.
Kirkwood, R. (1985). The quest for quality in graduate education. Educational
Record 66(3): 5-8.
Koehler, M. J., & Mishra, P. (2008). Introducing Technological Pedagogical Knowledge.
In AACTE (Eds.). The handbook of technological pedagogical content knowledge
for educators. Routledge/Taylor & Francis Group for the American Association of
Colleges of Teacher Education. 3-29.
Koehler, M. J., & Mishra, P. (2009). What is Technological Pedagogical Content
Knowledge (TPACK)? Contemporary Issues in Technology and Teacher Educa
tion, 9(1), 60-70.
Kooker, B. M., Itano, J., & Okimoto, H. (2000). Using focus groups to evaluate the
impact of a masterʼs in nursing distance education program. The Journal of
Nursing Education, 39(7), 329-332.
Levine, R., Locke, C., Searls, D., & Weinberger, D. (2000). The cluetrain manifesto.
Retrieved from http://www.cluetrain.com
Little, J. (1993). Professional development in a climate of educational reform.
Educational Evaluation and Policy Analysis, 15 (2), 129–151.

94

Manning, K. (2010). A Delphi study: Exploring faculty perceptions of the best practices
influencing student persistence in blended courses (Unpublished doctoral
dissertation). Capella University.
McCrory Wallace, R. (2004). A framework for understanding teaching with the Internet.
American Educational Research Journal, 41(2), 447-488.
Michigan State Board of Education (2001). The Information Age: Report by the Task
Force of the Michigan State Board of Education. Retrieved from:
http://www.michigan.gov/documents/embracing_119435_7.pdf)
Michigan State Board of Education (2008). Standards for the Preparation of Teachers:
Educational Technology. Retrieved from: http://tinyrul.com.npendorsement
Milken Exchange on Education Technology. (1999). Will New Teachers Be Prepared To
Teach In A Digital Age? Retrieved from: http://www.mff.org/pubs/ME154.pdf
Mishra, P., & Koehler, M. J. (2006). Technological Pedagogical Content Knowledge: A
new framework for teacher knowledge. Teachers College Record 108 (6), 10171054.
Moore, D.R. (1986). A descriptive study of the Tennessee State University educational
administration doctoral program as perceived by present and former students.
Unpublished doctoral dissertation, Tennessee State University.
NCATE. (2008). Unit Standards in Effect 2008. Retrieved from
http://www.ncate.org/Standards/NCATEUnitStandards/UnitStandardsinEffect200
8/tabid/476/Default.aspx
Peruski, L. (2003). Contradictions, disturbances, and transformations: An activity
theoretical analysis of three faculty members' experience with designing and
teaching online courses. Unpublished doctoral dissertation, Michigan State
University.
Ponnuru, R. (2010, February 24). The Case Against College Education. TIME.
Retrieved November 22, 2011, from
http://www.time.com/time/nation/article/0,8599,1967580,00.html
Postman, N. (1995). The end of education: redefining the value of school. New York:
Knopf.
Reichheld, F. F. (2006). The ultimate question: driving good profits and true growth.
Harvard Business School Press. Boston.

95

Ribak J, Notzer N, Drezne E (1995) Evaluation of an integrative Masterʼs program in
occupational health at the Tel-Aviv University Medical School. Saf Sci 20 :343–
347
Rittel, H. J., & Webber, M. M. (1973). Dilemmas in a general theory of planning. Policy
Sciences, 4(2), 155-169.
Roach, D., & Barker, L. (1984b). A region by region evaluation of masterʼs programs in
speech communication. Association for Communication Administration Bulletin,
49, 57–61.
Rosaen, C. L. & Wolf, L. G. (2005). Using the internet to support teacher reflection: An
authentic medium? Paper presented at the Annual Meeting of the American
Educational Research Association, Montreal, Canada, April.
Rowe, A., & Wood, L. N. (2008). What feedback do students want? Association for
Research in Education AARE, (1998). The Australian Association for Research in
Education (AARE).
Seidman, I. (2006). Interviewing as qualitative research: a guide for researchers in
education and the social sciences. New York, NY: Teachers College Press.
Shulman, L. (1993). Teaching as community property. Princeton, NJ: Carnegie
Foundation for the Advancement of Teaching.
Sparks, D. & Hirsh, S. (2000, May 24). Strengthening professional development:
A national strategy. Education Week, pp. 42-45.
Strauss, A., & Corbin, J. (1990). Basics of qualitative research: grounded theory
procedures and techniques. Newbury Park, CA: Sage Publications, Inc.
Stellman, J. M., Cohen, S., & Rosenfield, A. (2008). Evaluation of a One-Year Masterʼs
of Public Health Program for Medical Students Between Their Third and Fourth
Years. Academic Medicine, 83(4), 365-370.
Stufflebeam, D.L. (2001). Evaluation models: new directions for program evaluation. 89,
7-98. San Francisco, CA: Jossey Bass.
Taylor, M.C. (2009, April 26). The end of the university as we know it. The New York
Times. Retrieved April 26, 2009 from
http://www.nytimes.com/2009/04/27/opinion/27taylor.html.
Thayer, C. (2006). Participatory evaluation in nonprofit organizations: Rhetoric or reality.
Unpublished doctoral dissertation, The George Washington University.
96

Tom, A. (1999). Reinventing masterʼs degree study for experienced teachers.
Journal of Teacher Education, 50(4), 245–254.
Trott, D., Barker, D. R., & Barker, L. L. (1988). Evaluation of masterʼs programs in
the speech communication discipline: 1988. Communication Education, 37(4),
257-262.
U.S. Department of Education, National Center for Education Statistics. (2000).
Teachers' tools for the 21st Century. A report on teachers' use of technology.
Retrieved from http://nces.ed.gov/spider/webspider/2000102.shtml
U.S. Department of Education, Office of Educational Technology. (2010). National
Education Technology Plan 2010. Retrieved from
http://nces.ed.gov/spider/webspider/2000102.shtml
Vroeijenstijn, T. (1992). External quality assessment, servant of two masterʼs? The
Netherlands University perspective. In A. Craft (Ed.), Quality assurance in higher
education. (pp. 109-131). UK: The Falmer Press.
Weiss, C. H. (1998). Evaluation: methods for studying programs and policies (2nd ed.).
Upper Saddle River, NJ: Prentice Hall.
Wiggins, G. (1998). Educative assessment. San Francisco, CA: Jossey-Bass.
Wilson, R. (2010, February 7). For-Profit Colleges Change Higher Education's
Landscape. The Chronicle of Higher Education. Retrieved November 22, 2011,
from http://chronicle.com/article/For-Profit-Colleges-Change/64012/
Zhao, Y., Pugh, K., Sheldon, S., & Byers, J.L. (2002). Conditions for classroom
technology innovations. Teachers College Record, 104(3), 482-515.

97