“gm M LlBFiAng
ichigan tate
2,90 8 University

 

 

This is to certify that the
dissertation entitled

INFORMATION SYSTEMS PORTFOLIO MANAGEMENT:
THE IMPACT OF PORTFOLIO MANAGEMENT PRACTICES

presented by

BRANDIS PHILLIPS

has been accepted towards fulfillment
of the requirements for the

PhD. degree in Accounting & Information

 

 

Systems

IMajor Ffrofessor’s Signature

5119 lox

Date

MSU is an affirmative-action, equal-opportunity employer

I-—-l-l--4-n-A-l-—v-l-l-.-Ah_‘_-—-‘

 

PLACE IN RETURN BOX to remove this checkout from your record.
TO AVOID FINES return on or before date due.
MAY BE RECALLED with earlier due date if requested.

 

DATE DUE DATE DUE DATE DUE

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

5/08 K‘IProleccatPres/ClRC/DateDue.indd

 

Information systems portfolio management:
The impact of portfolio management practices

By

Brandis Phillips

A DISSERTATION

Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of

DOCTOR OF PHILOSOPHY
Department of Accounting and Information Systems

2008

ABSTRACT

INFORMATION SYSTEMS PORTFOLIO MANAGEMENT:
THE IMPACT OF PORTFOLIO MANAGEMENT PRACTICES

By
Brandis Phillips

The purpose of this dissertation is to extend the body of literature on
information systems (IS) portfolio management. Previous research by McFarlin (1981)
encouraged organizations to embrace managing their IS functions as one portfolio of
software applications, hardware and human resources which is similar to that of a
portfolio of financial assets. This dissertation extends McFarlin’s initial research by 1)
examining management practices that support the day-to-day management of the IS
function within organizations; 2) identifying perceptions of IS value and IS risk at an
enterprise (i.e. portfolio) level; and 3) analyzing the impact of management practices on
IS effectiveness as mediated through the risk/value tradeoff at the portfolio level. This
research is grounded in the theoretical perspectives of 1) contingency theory — in that
there is not just one way to organize an IS portfolio to achieve positive results; 2)
portfolio theory — IS projects are similar to financial assets in that the aim is to bring a
positive return from organizational investments in information technology; 3) absorptive
capacity —— business managers and IS managers need to work together in the management
of organizational IS to ensure the IS portfolio properly supports the business. Results of
a survey of 101 IS professionals indicates that perceptions of management practices
hypothesized as a second order factor significantly impact both perceptions of IS value
and IS risk. IS value is found to mediate the relationship between IS management

practices by significantly impacting perceptions of IS effectiveness. However, IS risk

was found not to mediate the relationship between IS management practices and the
dependent variable IS effectiveness. Other findings include differences between IS
managers and, IS auditors. IS auditors were found to View risk as a significant contributor
to perceptions of IS effectiveness, whereas IS managers did not. Contributions to
research include the validation of constructs for management practices using the Control
Objectives for Information Technology (COBIT) framework. Portfolio level constructs
for measuring IS value as well as IS risk are also validated. Going forward, researchers
now also have a theoretical framework in which to view the portfolio management

phenomenon.

Copyright by
Brandis Phillips
2008

ACKNOWLEDGEMENTS

I would like to thank my advisor Dr. Cheri Speier, for guiding me through this process
and providing me with direction as well as her insightful comments in shaping my
research. I would also like to thank my family (Suzette, Brandi & Derrien) for their love
and. support throughout my educational journey.

‘7

TABLE OF CONTENTS

List of Tables ................................................................................... i
List of Figures .................................................................................. ii
CHAPTER 1: INTRODUCTION ........................................................ 1
1.1 The Problem and need for study ....................................................... 1
1.2 Research Questions ...................................................................... 3
1.3 Importance of the Research and Contributions ...................................... 5
1.4 Scope and Outline of the Study ........................................................ 6
1.5 Organization of the Dissertation ....................................................... 6
CHAPTER 2: THEORETICAL UNDERPINNINGS ................................ 8
2.1 Contingent Perspective .................................................................. 9
2.2 Financial Portfolio Perspective ......................................................... 10
2.3 Absorptive Capacity ...................................................................... 12
CHAPTER 3: Research Model and Hypotheses ..................................... 15
3.1 Overview of Conceptual Model ........................................................ 15
3.2 Areas of Inquiry .......................................................................... 16
3.2.1 IS Portfolio Management Practices .............................................. 17
3.2.1.1 Monitor and Evaluate Performance ................................. 19
3.2.1.2 Define a Strategic IS Plan ............................................ 20
3.2.1.3 Manage the IT Investment ........................................... 21
3.2.1.4 Manage IS Human Resources ....................................... 21
3.2.1.5 Assess and Manage IS Risks ........................................ 22
3.2.2 IS Portfolio Value Overview ...................................................... 22
3.2.3 IS Portfolio Risk Overview ........................................................ 24
3.2.4 IS Portfolio Effectiveness Overview ............................................. 27
3.3 Hypothesis 1 - Management Practices and IS Portfolio Value ...................... 28
3.4 Hypothesis 2 - Management Practices and IS Portfolio Risk ........................ 32
3.5 Hypothesis 3 - IS Portfolio Value and IS Effectiveness ............................. 35
3.6 Hypothesis 4 — IS Portfolio Risk and IS Effectiveness .............................. 36
CHAPTER 4: Research Methods ......................................................... 38
4.1 Research Methodology .................................................................... 38
4.2 Unit of Analysis ............................................................................ 38
4.3 Sampling Procedure ....................................................................... 39
4.4 Data Collection ............................................................................ 41
4.5 Data Collection Method .................................................................. 42
4.6 Instrumentation ............................................................................ 42
4.7 Survey ....................................................................................... 49

vi

CHAPTER 5: Analysis of Data ........................................................... 51

5.1 Introduction ............................................................................... 51
5.2 Qualitative Findings ......................................................................... 51
5.3 Preliminary Analysis — survey, sample size, sample characteristics ................ 53
5.4 Factor Analysis ............................................................................ 56
5.5 Model Evaluation .......................................................................... 60
5.5.1 Measurement Model ................................................................. 60
5.5.2 Structural Model ..................................................................... 63
5.5.2.1 Hypothesis 1A ............................................................. 64

5.5.2.2 Hypothesis 1B .............................................................. 64

5.5.2.3 Hypothesis 2A .............................................................. 64

5.5.2.4 Hypothesis 2B .............................................................. 65

5.5.2.5 Hypothesis 2C .............................................................. 65

5.5.2.6 Hypothesis 3A .............................................................. 65

5.5.2.7 Hypothesis 3B .............................................................. 66

5.5.2.8 Hypothesis 4 ................................................................. 66

5.5.2.9 Control Variables .......................................................... 66

5.6 Evaluation of the Use of a Second Order Structure ................................... 67
5.7 Test for Mediation ........................................................................ 67
5.8 Analysis between Auditors and IS Professionals ..................................... 68
CHAPTER 6: Discussion and Conclusion ............................................... 72
6.1 Introduction ............................................................................... 72
6.2 Research Implications ........................................................................ 72
6.3 Managerial Implications .................................................................. 79
6.4 Recommendations ......................................................................... 83
6.5 Limitations ................................................................................. 83
6.6 Future Research ........................................................................... 85
APPENDIX A: Open Ended Interview Questions ..................................... 87
APPENDIX B: Responses to Open Ended Interview Questions .................. 88
BIBLIOGRAPHY ............................................................................ 98

vii

LIST OF TABLES

Table 1 Financial & Project Portfolio Comparison ...................................... 11
Table 2 IS Management Practices Questionnaire Items ................................. 44
Table 3 IS Value Questionnaire Items ...................................................... 46
Table 4 IS Risk Questionnaire Items ...................................................... 47
Table 5 IS Effectiveness Questionnaire Items ........................................... 48
Table 6 Sample Characteristics .............................................................. 55
Table 7 Confirmatory Factor Analysis Results ............................................ 56
Table 8 Factor Loadings ...................................................................... 58
Table 9 Correlation Matrix, ICR, AVE .................. ' ................................... 61
Table 10 Outer Model Loadings — Structural Model .................................... 62
Table 11 Hypotheses and Results ........................................................... 67
Table 12 Auditor/IS Professional Path Comparison ...................................... 70
Table 13 Hypotheses and Results IS Auditors/IS Professionals ........................ 71

viii

LIST OF FIGURES

Figure 1 IS Portfolio Management ......................................................... 8

Figure 2 General Research Framework ..................................................... 15
Figure 3 Research Model ..................................................................... 16
Figure 4 Control Objectives for IT .......................................................... 18
Figure 5 Structural Model Results ........................................................... 63

ix

CHAPTER 1: INTRODUCTION

1.1 The problem and need for study

Organizational attempts to align business and information systems (IS) objectives
have led to the increasing use of program management offices to track projects and align
them with organizational goals. As an extension to the program office, firms have begun
to employ IS portfolio approaches to manage their entire universe of software application
projects. More than twenty years ago McFarlan (1981) first espoused the concept of
managing a firm’s IS projects and applications as one large portfolio designed to align
software application projects with business strategies. Research since McFarlan’s initial
conceptualization has been sorely lacking. A review of the literature only finds a study
by Jeffery and Leliveld (2004) who study CIO best practices. Other emerging research
includes initial studies by Burke and Shaw (2007) who examine portfolio management
from a real options perspective and Karhade (2007) who examines portfolio management
from an enterprise risk perspective. Furthermore, practitioners have only recently
embraced the notion of portfolio management (Waxer 2005). Reasons for adopting
portfolio management include high levels of IS spending in organizations as well as the
need to align IS with organizational goals (Maizlish et a1. 2005). With the increasing
complexity of projects and business needs, CIO’s must monitor the entire IS enterprise
from a number of different perspectives. First, IS project staffing levels and IS staff
skills-sets are increasingly becoming critical issues for managing the firm’s
infrastructure. The use of IS portfolio management should facilitate improved human
resource management to ensure the appropriate human resources are working on the right

projects and facilitate tracking of project financials and alignment with business goals.

Second, budgetary concerns are critical due to pressures of managing limited financial
resources for hardware, software and human resources. Finally, alignment represents
another major area that CIO’s must incorporate into their planning and execution of IS
strategy.

Unlike previous research at the project level, research on IS portfolio management
is important because of its organizational level implications. At the organizational level
managing budgets, resources and time become much more critical than at the project
level. This is due to the added complexity of managing multiple projects and spreading
resources among multiple projects. Also, goal alignment between projects and
organizational strategy becomes more important due to the need to examine the level of
fit between each and every project and application across the entire portfolio (Benko et
a1. 2003). Furthermore, IS risks become more driven by market and organizational
demands (Bonham 2005). Therefore, organizations that utilize a portfolio management
approach should incorporate what Benko and McFarlan (2003) refer to as the
characteristics of the information frontier. The characteristics of the IS portfolio reflect
organizational transparency, velocity, reduced transactional friction and role blurring.
Organizational transparency refers to the fact that there is a large amount of information
that is available regarding today’s public companies both internally and extemally. The
IS portfolio must provide transparency with regard to information storage and retrieval.
Information must be accessed easily for reporting purposes in support of the business.
Velocity refers to the speed at which information travels. The IS portfolio must provide
information quickly for decision making purposes. Reduced transactional friction refers

to decreases in costs, as well as greater communication and coordination revolving

IQ

around transactions. Transactional friction within the IS portfolio should be reduced due
to greater applications synergy, process integration and the availability of more timely
information. Finally, role blurring refers to changes in traditional roles that organizations
previously had of one another. For instance a supplier might become more of a partner
when enabled with a view into their customer’s supply chain. The IS portfolio possesses
the capability to easily accommodate joint ventures or alliances through information
access and integration across organizations. Each of these characteristics represents a
key attribute necessary for a properly functioning IS portfolio that has the ability of
supporting an organization in its attempt to achieve its organizational goals. Nevertheless
achieving a properly functioning portfolio is subject to a variety of risks. While these
risks also exist at the project level they are magnified at the portfolio level, due to
interdependencies among projects and resource allocation among projects. Examples of
such risks can include the following. Human resources in terms of role clarity or the
adequacy of skills sets as well as strategic planning that aligns business needs and IS
capabilities. Other IS risks include the ability to monitor and evaluate projects. Thus this
research will extend the project management literature to focus on an organizational level
phenomenon, (IS portfolio management), that has not been extensively researched.
1.2 Research Questions

Portfolio management is an understudied phenomenon with considerable
opportunities for examination. Organizations are using portfolio and program
management practices to supervise the implementation and ongoing use of IS across their
entire enterprises. Reasons for this trend include the increasing complexity of IS in terms

of the number of systems, the need for integration between systems, allocation of human

resources across projects, dollars invested in information systems and quickly changing
business processes (Benko et al. 2003), GVIaizlish et a1. 2005). As a result, practitioners
are attempting to increase the effectiveness of their information systems and mitigate IS
risks. This situation causes a number of questions to arise.

What management practices do organizations implement in order to supervise the
portfolio management process in terms of risk, value and eflectiveness?

What management practice used in organizations to manage portfolios and programs are
critical precursors to the ongoing operation of an eflective IS function?

How do IS management practices affect IS risk, IS value and IS effectiveness?

This dissertation will investigate three important areas to address the
aforementioned questions. This analysis will provide a more complete understanding of
IS portfolio management, with respect to IS portfolio management practices, the IS risk
return (i.e. value) tradeoff as well as IS effectiveness. The following will briefly
introduce the areas of inquiry. Information systems management practices in use within
organizations are assumed to be precursors to managing IS effectively, where
management practices are controls put in place to properly manage IS. Value is the
positive contribution of IS to the organization, where IS value is the degree of
responsiveness and flexibility that the IS portfolio possesses in its effort to meet the needs
of the business. IS risk can be viewed as the remaining uncertainty that exists after
applying project management processes and practices to interdependent IS projects
(McFarlan 1981). Finally IS effectiveness can provide a strategic contribution to
organizational goals (Chan et a1. 1997). As a result Chan et al narrowly define IS
effectiveness as a combination of IS contributions to organizational effectiveness,

product development and market linkages. From a portfolio perspective, therefore,

effectiveness, value and risk are integrally linked together because the mitigation of risk
and attention to value can help achieve effective use of IS across the organization. Thus,
this dissertation poses the following research question to address the implementation and
use of portfolio management practices:

How does IS risk and value mediate the relationship between organizational IS
management practices and IS effectiveness in organizations?

1.3 Importance of the Research and Contributions

Previous research has failed to adequately investigate the phenomenon of IS portfolio
management in a detailed manner. This research will inform both practitioners and
scholars about the use of portfolio management practices and examine macro level risks
in terms of the overall IS portfolio. From an acaderrric point of View, theory will be
applied to the IS project portfolio phenomenon to examine the antecedents of an
effectively managed portfolio. Management practices, IS risk and value as well as IS
effectiveness inherent to the organization will be studied. Furthermore, this research will
validate a portfolio level of risk measure, an IS value measure at the portfolio level and
empirically test their impact upon perceptions of IS effectiveness.

The research will provide practical information in a number of ways. First, this
research will identify the most important portfolio management practices that
organizations should use to support the proper operation of their IS fimction. Second the
research will determine which practices significantly mitigate an IS function’s level of
risk as well as facilitate IS value. Third, the research will determine if IS value and risk
significantly impact IS effectiveness. As with any research, there is an inherent risk that
this study will/not yield the expected results for a variety of reasons. For example, there is

currently a lack of significant reported findings about the impact of IS portfolio

management practices upon IS value, IS risks and subsequently IS effectiveness.
Although prior research in the project risk area points to the role of controlling risks in
order to achieve desired outcomes, the resulting research might not elucidate any
significant findings for the impact of risk and value on IS effectiveness. Lack of
significant findings will however provide valuable insights. The research might raise
concerns regarding the impact of the IS management practices used in this study.
Nevertheless any concerns raised will create an opportunity to conduct further research
into the specific management practices used in this study as well as examine other
management practices.
1.4 Scope and Outline of the Study

This dissertation research will examine organizational level perceptions of IS
portfolio management practices. Furthermore, this research will gather information
related to perceptions of the effectiveness of the IS portfolio, IS value and IS risk. Data
will be collected from cross sectional surveys conducted across a wide variety of
organizations, including Fortune 1000 firms and large not-for-profit agencies. Large
firms are the primary targets for this study since they utilize a complex enterprise of
applications hardware and software that lends itself toward the use of portfolio
management. The scope of the dissertation will be limited to a cross-sectional study that
will gather data regarding perceptions of organizational level phenomena for IS portfolio
management.
1.5 Organization of the Dissertation

Chapter two of this dissertation will present the theoretical underpinning and

literature review. Chapter three will introduce the research model and hypotheses that

will examine the IS portfolio management phenomenon. Chapter four will provide a
detailed discussion of the methods employed. Chapter five will present the results of this
analysis and chapter six will contain a discussion of the results, limitations of the study

and conclusions that give information for practitioners and future directions for research.

CHAPTER 2: THEORETICAL UNDERPINNINGS
Information systems portfolio management is a combination of proj ects, processes
and people that combine together to aid in an organization’s attempt at achieving its goals
and objectives as depicted in figure 1 below. Projects refer to the IS systems that are in
place or planned that support an organizations’ goals and objectives through digitization
of business processes. People refer to those that manage the IS systems in place while
processes refer to the business functions that support the proper management and control

of the IS function.

IS Portfolio Management

\

 

 

Meets Beside mg
Portfolio Theory Absorptive Capacity Contingency Theory

 

 

 

 

 

*-
This study will investigate IS portfolio management through a multi-theoretical lens.
Given that program management and portfolio management are firm-level activities, this
dissertation utilizes contingency theory to frame the use of portfolio management. This
research also applies modern portfolio theory in order to explain how and why certain
projects are included in the portfolio. Finally, absorptive capacity is used to support the

integration and use of portfolio management practices among various managers to

 

effectively use IS. The following will give an overview of each theoretical perspective
and how it supports IS portfolio management.
2.1 Contingent Perspective

The basic premise of the contingent perspective revolves around the following
assumptions: 1) there is no universal best way to manage an organization; 2) the design
of an organization and its subsystems must fit with the environment; 3) effective
organizations must have a proper fit with the environment and also between its
subsystems; and 4) the needs of an organization are better satisfied when the organization
is properly designed and the management style is appropriate for the tasks undertaken
(Wade et al. 2006). This perspective is appropriate for IS portfolio management for a
variety of reasons. The implementation and use of IS across organizations varies
according to context. Variability in context can mean that platforms and software
configurations may differ. However outcomes within and between industries can still
yield positive results. Furthermore, management styles and practices can differ by
industry or by organization, yet still have the potential to contribute to IS and
organizational success.

IS projects vary based upon a number of factors within an organization, as well as
across organizations. Factors affecting IS project outcomes in business environments
include the economy, budgetary and time constraints, technology configurations and
human resources. As a result, a variety of configurations that incorporate key factors can
contribute to IS project and portfolio success. Contingency theory is an excellent
framework for IS portfolio management since there is no single way to organize and a

variety of inputs can lead to a positive outcome. Specific to IS portfolios, McFarlan

(1981) supports a contingency approach to IS portfolio management. The notion of
multiple ways to control projects is dependent upon the projects’ characteristics. An
example of such is user involvement through stakeholders and ad hoc teams that vary by
project (McFarlan 1981).

In sum, contingency theory has been used in IS research as a framework to help
explain variance in complex IS phenomena. Thus, multiple and often conflicting
contingencies are normal occurrences in today’s complex business environments.
Organizations are routinely faced with a number of potentially conflicting contingencies.
Given there is not one right way to organize, implementing and maintaining portfolios of
information systems are subject to the nuances of each individual organization. Thus,
differing organizational configurations and management practices can result in effective
IS and positive organizational performance. Therefore, IS research in organizations
requires the use of the systems approach in contingency theory since it views
organizations as entities that maintain many potentially conflicting contingencies that
interact with one another to affect performance. Specific to the IS portfolio, contingency
theory will be used as an overarching‘meta-theory to elucidate how multiple IS related
contingencies affect perceptions of IS effectiveness, IS value and IS risk.

2.2 Financial Portfolio Perspective

Portfolio management for information systems is akin to that of modern portfolio
theory (Markowitz 1959). Modern portfolio theory dictates that a mix of financial assets
(stocks) be held to maximize return while minimizing risk. Therefore, the overall
investment strategy seeks to create an optimal portfolio by considering the relationship

between risk and return. The risk of a particular stock should not be examined on a

10

standalone basis, but rather in relation to how that particular stock's price varies in

relation to the overall market portfolio. Therefore, each investment has the potential to

significantly impact the value to the organization (Milgrom et al. 1992). Benko and

McFarlan (2003) adapt the financial portfolio approach by examining each IS project

within the IS portfolio in terms of its risk profile and its contribution to organizational

goals. Table 1 compares both the financial and project portfolios.

Table 1

Financial & Project Portfolio Comparison

 

Financial Portfolio

Project Portfolios

 

 

 

 

 

 

Assets Financial instrument w/distinct Projects w/distinct characteristics
characteristics
Diversification Multiple financial instruments Many project variables-scope,
can reduce risk approach, vendor, project mgr
can reduce risk
Goals Income & capital gains Profitability & growth
Cost reduction
Asset Allocation Invest according to investment Invest according to org intentions
goals
Connection Correlation Interdependency

 

 

 

The ultimate goal of using a portfolio approach to manage software applications is

to give a firm the ability to more effectively derive value by allowing information

technology to support business objectives. Just as there are a number of variations in

approaches to financial portfolios, practitioners hold varying views about constructing IS

project portfolios, as well as how the portfolio should be managed. A review of the IS

literature reveals a paucity of information related to IS portfolio management with a few

notable exceptions. McFarlan (1981) was the first to introduce the idea of IS portfolio

management. This concept was extended by Weill and Vitale (1999), who presented a

framework for assessing the health of an IS applications portfolio. Jeffrey and Leliveld

ll

 

(2004) provide CIO insights regarding the state of portfolio management in
organizations. Emerging research examines real options in portfolio management
through the lens of complexity theory (Burke 2007). While other emerging research from
Karhade (2007) examines risk within the portfolio in regards to the risk appetite
organizations have when engaging in project selection. Cameron (2005) provides an
overview of the IS portfolio phenomenon for researchers; however, Cameron’s research
was primarily based upon practitioner insights. Additional practitioners, such as Datz
(2003), have also provided additional commentary about why portfolio management is an
essential tool for enhancing the impacts of IS.

Practitioners such as Bonham (2005) and Maizlish and Handler (2005) have
provided “how to” guides on implementing and maintaining an IS project portfolio.
Furthermore, Benko and McFarlan (2003) also provided a framework to organize IS
projects as a single portfolio. The practitioners who have put forth “how to” guides for
portfolio management all rely heavily on the notion that an IS portfolio is akin to a
financial portfolio, such as a mutual fund. The authors rely on modern portfolio theory
and portfolio selection (Markowitz 1952; Markowitz 1959) to support their hypothesis
that an IS portfolio can be similar to a financial portfolio. In the same vein that a properly
selected mix of financial assets can mitigate risk and increase overall financial returns,
the proper selection of IS projects aligned with an organization’s strategic objectives can
increase organizational returns (i.e. increases in IS value) and reduce IS risks.

2.3 Absorptive Capacity
Successful implementation and use of IS is linked to effective management of a

number of processes associated with the planning, acquisition and implementation of an

12

organization’s portfolio of IS (Cash et al. 1988). These processes or practices are
employed by both business and IS management for the purpose of effective system use.
Proper IS use is therefore dependent upon the development of IS-related knowledge and
processes that exist among an organizations’ IS managers and business managers
(Boynton et al. 1994). Shared knowledge between IS and business managers is an
“understanding and appreciation for the technologies and processes that affect their
mutual performance” (Nelson et al. 1996). Knowledge sharing activities include such IS
tasks as requirements gathering, project planning and project implementation. Within
each of these activities, discourse occurs between IS and business managers at both the
individual and group levels. These knowledge sharing activities are critical situations
where high levels of knowledge sharing can facilitate performance. When knowledge is
not adequately shared between the two groups, the opportunity for the implementation of
less than optimal systems is highly probable. Thus knowledge sharing between business
and IS groups creates the opportunity to decrease inherent IS risks, establish and maintain
sound management practices surrounding an organizations’ portfolio of systems, as well
as facilitate higher levels of effectiveness in the IS portfolio.

Absorptive capacity fits well with IS portfolio management because they both
rely on sharing knowledge between individuals or groups in order to achieve a
competitive advantage. In general the use of absorptive capacity can facilitate knowledge
sharing between individuals and or groups. Furthermore from an IS perspective,
knowledge sharing between the business and the IS function within organizations must be
seamless. This is necessary to ensure properly functioning management practices for the

ability of the organization to successfully complete IS development projects, as well as to

13

ensure proper maintenance and ongoing monitoring of the IS portfolio. Therefore,
absorptive capacity in the context of shared knowledge, implicitly underpins IS specific -
management expertise. Thus, due to this implicit assumption, absorptive capacity serves
as a theoretical lens for this research but will not be explicitly measured as a construct in
the research model that follows in the next section.

In conclusion the use of multiple theoretical perspectives to explain the IS
portfolio management phenomenon is useful due to its organizational complexity and
portfolio management’s necessity in large multifaceted organizations. Portfolio theory
supports the notion that the composition of the portfolio with regard to risk, is subject to
managerial choices on the inclusion of specific projects. While contingency theory
recognizes that organizations can have a unique portfolio relative to other organizations,
each organization still has the ability to organize a portfolio through a variety of different
processes that can contribute positively to the organization’s strategic goals. Finally
absorptive capacity recognizes the need for knowledge sharing in so much that many IS
tasks need both the input of IS and business personnel to achieve optimal solutions for
business problems. The following chapter will introduce the research model and

hypotheses to be tested.

14

Chapter 3: RESEARCH MODEL AND HYPOTHESES

3.1 Overview of Conceptual Model

The purpose of properly managing an IS project portfolio is to ensure that the
organization’s portfolio contributes to IS effectiveness through creating value while
mitigating risk. Nevertheless, IS projects and IS portfolio outcomes are both subject to
the interactions of IS portfolio management practices, IS value and IS risk. Value and
risk similarly represent the risk/retum tradeoff in a financial portfolio. Depicted in Figure
2 is the general research framework that illustrates the impact of an organization’s IS
portfolio management practices on its risk/value profile (i.e., IS value & IS risk) and,
subsequently, on the organizations’ IS portfolio's effectiveness. Risk and value are
assumed to be mediators or precursors to management influence. Thus, the mediator
relationship is hypothesized since it is believed that risk mitigation as well as value
creation in an IS context stems from proper management practices in place. Thus the
risk/value proposition can either negatively or positively impact IS effectiveness. Given
the gap in the literature, the results of this research will attempt to answer the following
research question originally stated in Chapter One.

How does the risk/value tradeofl mediate the relationship between organizational

IS portfolio management practices and IS portfolio effectiveness in
organizations?

 

 

 

l
I

13 Portfolio
Effectiveness

 

 

 

 

 

 

 

 

15

Figure 3 below depicts the research model that will be used to empirically test the
hypotheses. Consistent with the general research fi'amework, IS portfolio management
practices are hypothesized to impact risk and value, and risk and value are hypothesized

to impact the IS portfolio's effectiveness.

: IS Portfolio Value

- Responsiveness
- Flexibility i
v 1
E

&

1 IS effectiveness
(Second Order Factor)

 

l IS Portfolio Mgmt Practices '
(Second Order Factor)
Monitor 8. Evaluate Perf
Define Strategic IS Plan
Manage IT Investment

l Assess & Manage IS Risk
Manage IS HR

- operational efficiency
l - mgmt effectiveness

1
(

l
I—

l
l
l

 

 

 

IS Portfolio Rick
- Portfolio Risk
- S ner

‘1 Y gy

I . ”control Variables
l- l - Size
5 - Experience

Figure 3 Research Model

 

 

3.2 Areas of Inquiry
The following section will introduce each area of inquiry specified in the high
level research model depicted in Figure 2. After these areas of inquiry have been

introduced, each hypothesis will be presented.

16

3.2.1 IS Portfolio Management Practices

Managerial controls in the form of rules and procedures serve as the basis for
properly ftmctioning organizations. The controls enacted by managers pave the way for
properly operating organizational processes. IS portfolio management practices are
formal codified rules and standard operating procedures for the supervision of IS. These
practices are specific and tactical in nature, and are carried out by management. The
practices attempt to influence outcomes so that results are favorable. Thus, IS portfolio
management practices can serve as a form of administrative control over the management
of IS portfolios. Administrative controls are those that involve monitoring, measuring
and taking corrective action (Langfield-Smith 1997). These practices can also be thought
of as ex ante forms of control. Ex ante controls provide necessary information to direct
or guide individual or group actions (F larnholtz et al. 1985).

The IS literature discusses IS management practices in a variety of different ways.
Reich and Benbasat (1996) examine management practices such as planning efforts
between IS and the rest of the business. Others researchers, including Powell and Dent-
Micallef (1997), examine IS management practices in the form of activities such as CEO
commitment and business process redesign. Zmud (1982) examines IS management
practices as ways to expand or constrain behavior by formalizing or centralizing
organizational structure. Thus, the IS literature presents management practices as high
level tools to control behavior within the context of information systems.

The literature, however, does not address how specific managerial level practices
and controls guide behavior toward properly managed projects, programs or portfolios.

Specific management tactics for the implementation and management of IS have been

17

codified within the Control Objectives for IT (COBIT) fi'om the IT Governance Institute.
This set of standards is recognized globally as an IS management tool that has the
capability to derive value from IS. The COBIT framework consists of four domains, plan
and organize, acquire and implement, deliver and support as well as monitor and
evaluate. That all work together to support an organizations’ framework for supporting

the IS function.

A

i Information Systems Controls )

 

 

 

 

 

7 A a l

Acquire 8- Implement Plan & Organize Monitor 3. Evaluate Deliver 8. Support i

 

 

 

 

I
I
I
o
I

 

 

 

 

 

 

figure 4 Control Objectives for IT
The Acquire and Implement domain serves as a framework for providing application
solutions. The Plan and Organize domain provides direction for application solution and
service delivery. The Deliver and Support domain provides direction for ensuring
applications meet user needs. Within each domain there is a series of specific practices in
each domain that each play an important role in shaping an effective IS function. This
study will only examine practices specified in the Plan & Organize domain and the

Monitor & Evaluate domain. These two domains are chosen because the practices

18

specified in these domains speak specifically to the day to day high level management of
an IS portfolio.

Given the gap in the literature regarding IS management practices, the initial
phase of this study will attempt to determine which management practices are most
important to the management of IS portfolios. This report includes a review of the
practitioner literature and interviews with senior IS managers at a few large organizations
regarding the most important IS management practices they use. Consistent with the
COBIT framework, the management practices considered to be most important are
consistent with the results of the practitioner interviews. Thus the five practices to be
most important are listed as follows: 1) monitor and evaluate IS performance, 2) define a
strategic IS plan, 3) manage the IT investment, 4) manage IS human resources, and 5)
assess and manage IS risks. These practices fall primarily within the Plan & Organize
domain while the monitor and evaluate performance practice falls in the Monitor &
Evaluate domain. The following section describes each IS management practice within
the context of COBIT, and provides supporting arguments from the IS literature.
3.2.1.1 Monitor and Evaluate IS Performance

The COBIT framework specifies that there should be a general framework for
monitoring IS' contributions to the management of the IS portfolio. Methods are in place
to collect relevant monitoring data based upon clearly defined performance objectives.
Furthermore, a performance assessment process is in place to periodically review IS
performance against targets. As a result, IS in organizations should employ something
akin to an application profile (Batiste 1986). As such, the application profile is a

management tool to objectively highlight the benefits of systems that are in place. Others

19

such as Belcher and Watson (1993) view portfolio performance as weighing the
difference between benefits and costs. Thus, mechanisms must be in place that enable
the organization to adequately measure the IS contribution to organizational goals and
objectives. Thus, the monitor and evaluate IS performance construct can be defined as a
control process that examines IS impact upon organizational goals and objectives.
3.2.1.2 Define a Strategic [8 Plan

According to the COBIT framework, strategic planning should be a top down
approach that incorporates both business and IS objectives. Therefore, business and IS
strategies should be integrated and linked to organizational goals (Reich et al. 1996).
Furthermore, both the business and IS plans must be consistent (Henderson et al. 1988).
Derived fi'om this top down approach, high level business/IS strategy should be translated
into a strategic IS plan that integrates both costs and risks. Strategic IS plans‘ success is
determined by their degree of alignment, analysis, cooperation and improvement in
capabilities (Segars et a1. 1998). Thus, the strategic IS plan is the “process of deciding
the objectives for organizational computing” (Lederer et al. 198 8). Lederer and Sethi
(1996) go on to update their understanding of strategic IS planning by extending their
definition to articulate that strategic IS planning is “a process of identifying a portfolio of
computer based applications that will assist an organization in executing its business
plans and realizing its business goals.” Thus, the define a strategic IS plan construct, can
be defined as a control process that utilizes strategic IS portfolio planning to enable an

organization the opportunity to achieve its goals and objectives.

2O

3.2.1.3 Manage the IT Investment

In some cases, IS costs can constitute over half an organization’s operating
budget. As the cost of IS becomes larger, so does the need for sound financial
management of the IS flmction (Benko et al. 2003). The COBIT framework indicates
that there must be a financial management framework in place for the IT function that
properly evaluates business cases for new IS investments. New IS investments should be
properly prioritized and financial metrics should be in place to track the IS contribution to
business results. Examples of financial methods in the IS literature include the use of real
options to value and prioritize IS investments (Bardhan et al. 2004). Furthermore ratios
specific to IS are used such as the following, IS budget as a percentage of revenue,
percentage of IS budget spent on staff and percentage of IS budget spent on training the
IS staff, to serve as a means to examine IS impact of IS on the organization (Mahmood et
a1. 1993). Thus, the manage the IS investment construct, can be defined as a control
process that utilizes widely accepted financial techniques to enable an organization to
track its IS portfolio assets.
3.2.1.4 Manage IS Human Resources

Human resources are critical for the operation of any organization. Thus, IS
human capital can considerably impact the role as a strategic enabler that IS plays in
organizations (Roepke et al. 2000). Within the COBIT framework, IS personnel should
have properly defined roles, and should acquire and maintain the proper competencies to
adequately perform their duties. Baroudi (1985) and Igbaria and Siegel (1992) find that
role ambiguity increases turnover intentions in IS personnel. Additionally, Powell and

Dent-Micallef (1997) find that training for IS personnel complements IS as a competitive

21

advantage. Thus, managing IS human resources, can be defined as a control process that
aims to ensure that an organizations’ human capital is properly equipped to add value to
the IS portfolio to enable an organization the opportunity to achieve its goals and
objectives.
3.2.1.5 Assess and Manage IS Risks

IS risk is inherent in all projects and may adversely affect the IS portfolio, as well
as the organization as a whole. According to the COBIT framework, IS risk assessment
must therefore take place along with plans to prioritize and implement responses to IS
risks. Furthermore, ongoing IS risk management should be integrated with the
organization’s enterprise-wide risk management fi'amework. A large body of IS literature
supports the notion that IS risk is an important part of implementing and maintaining
systems. These IS risks can be generally thought of as affecting IS at an operational level
as opposed to a strategic or portfolio level. Examples of operational risks that impact
specific projects or maintenance efforts can be requirements, user, group or complexity
related (Wallace et al. 2004). However risks should be thought of in a strategic sense due
to their impact upon the organization. Thus, the assess and manage IS risks construct,
can be defined as a control process that aims to continually monitor IS risk as well as join
together both IS risk and organizational level risk in an effort to properly mitigate
organizational risk.
3.2.2 IS Portfolio Value Overview

The value of an IS portfolio indicates to what degree IS can readily support the
business across the enterprise. This value represents both the proactive and reactive

position of the IS portfolio. Proactive IS portfolio’s can be described in terms of their

22

flexibility while reactive IS portfolio’s can be described in terms of their responsiveness.
The following will describe each position. A proactive portfolio is based on its ability to
act in anticipation of future problems, needs, or changes. The ability of the IS function to
anticipate business moves can be measured in terms of the degree of flexibility that the IS
portfolio maintains. Flexibility, in general, has been described as enabling organizations
to control environments (De Leeuw et al. 1996). Specific to IS, the IS portfolio can be
viewed as its own entity that can control or in this case anticipate the future problems,
needs or changes of the business (i.e. the environment inside the organization) that the
portfolio supports. From another perspective on flexibility, previous IS literature has
viewed flexibility in terms of an organization's IS infrastructure (Broadbent et al. 1997;
Byrd et al. 2000). Broadbent and Weill (1997) also distinguish IS infrastructure based
upon whether it is technical or human resources related. Regarding IS portfolio
management, flexibility can be regarded as an anticipatory, or forward looking, effort that
can readily integrate management initiatives. These initiatives help the organization to
gain an advantage in streamlining processes relative to the industry in which it competes,
or enable the organization to seek out strategic alliances, partnerships, mergers or
acquisitions. IS Value is thus derived from IS flexibility due to its anticipatory nature in
regards to business needs. The IS portfolio can be looked upon as a partner or driver in
meeting organizational goals and objectives as opposed to a hindrance or laggard. Thus
IS flexibility can be defined as an anticipatory, or forward-looking, stance of the IS
function that can readily integrate management initiatives.

On the other hand, there is the responsiveness construct that also helps to

determine value. Responsiveness in an organizational context has been referred to as

23

how quickly a firm responds to environmental signals (Zaheer 1997). Specific to the IS
portfolio, a reactive position refers to the degree of responsiveness to changes in business
needs. Changing business needs may require that processes be modified due to forces
that might be driven externally by competitive industry pressures or new government
regulations. Other examples of the need for IS to be responsive include an organization's
need to absorb additional data due to a hostile takeover or merger. IS value is thus
derived. from the responsiveness of the IS portfolio’ ability to aid in the changing needs
of the business. Thus, IS responsiveness can be defined as the velocity in which an
organization’s IS portfolio can react to externally driven change. Therefore, IS value
across the enterprise requires a high degree of responsiveness and flexibility across an
organization's applications to provide organizational benefits.
3.2.3 IS Portfolio Risk Overview

The primary focus of any portfolio, whether financial or information systems—
related, is the proper identification and management of risk (Benko et al. 2003). From an
IS perspective, McFarlan (1981) views IS risk as “what” remains after the application of
project management methods, where “what” refers to uncertainty that cannot be
controlled with the use of project management. Because a number of potential risks are
associated with any IS portfolio, an organization's objective should be to mitigate those
risks to reasonable levels or, in other words, to reduce uncertainty by applying sound
project management techniques.

The following briefly discusses a small portion of the previous literature devoted

to project level IS risk,’ and introduces the notion of risk at the portfolio level. Wallace,

 

l A large body of IS literature supports the notion that IS risk is an important part of implementing and maintaining systems. Refer to
the review by Ropponen (1999) for more detailed information regarding 18 risk literature.

24

Keil and Rai (2004) find that higher project risks lead to negative outcomes, such as
lower application process performance. Jiang, Klein and Discenza (2001) find that
behavioral-related risks, such as lack of user experience and lack of role definition,
reduce satisfaction with system usage and quality, and lead to negative IS impacts on the
organization. Other project level risks may be ascribed to team composition, business
environment, planning and control (Wallace et a1. 2004), or to new technologies or skill
levels (Keil et a1. 1998). From a portfolio perspective, behavioral-related risks such as
role definition need to be clearly defined, especially when staff are expected to work on
more than one project at a time. Other issues include reporting relationships for staff that
are assigned to multiple projects. Planning and control also are magnified at the portfolio
level as organizations attempt to synchronize projects within programs and to coordinate
multiple programs across business units. Furthermore, because portfolio level risk is
enterprise-wide, its effects are undoubtedly felt at the organizational level. Other
possible effects of risks are magnified for IS satisfaction, performance and impact upon
the organization. Overall the risk at the portfolio level is much more complex than
project level risk. Dependencies among projects, personnel and processes at the portfolio
level become more interrelated thereby increasing uncertainty. The increase in
uncertainty at the portfolio level leads to more organizational implications than do
individual projects. The following will give an overview of the two primary areas of risk
to be examined in this study.

This study evaluates two primary areas of risk. These are general IS portfolio
risk, and lack of synergy among applications, people and process. Where IS portfolio

risk is an organization's aggregate IS project risk (McFarlan 1981). IS portfolio risk is

25

the possibility of a loss that surrounds large enterprise wide implementations in terms of
both staff and dollars invested. The possibility of a loss can be spread across all systems
implementations and large re-engineering efforts. Budgetary concerns carry risks when
either costs or time limits are exceeded across all IS project within the organization.
Other areas within the IS portfolio that carry higher possibility of losses include the use
of legacy systems that use outdated technology or require very specialized skill-sets to
perform maintenance. Wide spread use of older systems can lead to higher overall costs
for the portfolio as compared to those organizations that use newer technology.

With regard to lack of synergy within the IS portfolio, lack of synergy refers to a
lack of compatibility among applications, users and processes. Compatibility in general
refers to a technical artifact's ability to share any type of information with any other
artifact (Duncan 1995). More specifically, with regard to the IS portfolio, synergy can be
viewed as among applications (i.e., data transfer/conversion) as Duncan suggests.
Furthermore the notion of compatibility can be extended to apply to personnel and the
applications they use, the applications and the business processes they host as well as
compatibility among the technical infrastructure and applications. As a result of lack of
synergy (i.e. compatibility), risk increases in the applications portfolio. Low synergies
increase the possibility of a loss due to incompatibility. An example of this includes data
that fall out between applications. Another example is when an organization attempts to
run an application in an architectural environment that the application was not designed
for. Finally, from a human resources perspective, when staff members are not properly
trained to use a particular application, then it is possible that the application will not be

used to its full capability, and that it will not be used as intended. Thus risk at the IS

26

portfolio level is a more general and broad conceptualization than risk arising at the
project level. This portfolio risk encompasses enterprise-wide uncertainty with regard to
implementation concerns, budgets and synergies as opposed to examining only project
level concerns such as requirements gathering or team composition.

3.2.4 IS Portfolio Effectiveness Overview

The effectiveness of an organization’s portfolio of systems is determined by their
contribution to business effectiveness. Effectiveness of an IS portfolio can be looked
upon as its readiness to provide service to the business, providing the business with
desired results or maintaining the capabilities to adequately support the organization in
meeting is goals and objectives. Previous research implies that if IS is utilized correctly it
can be a key differentiator in organizational success. Successful implementations are
thought of in terms of addressing budget and time concerns. While properly utilized IS
goes beyond the implementation phase to have a positive impact on the business by
reducing or mitigating IS risks that lead to sub-par performance. Furthermore properly
utilized IS can contribute to reaching organizational goals. Thus, properly utilized or
agile IS can improve productivity and promote quality gains that eventually translate into
enhanced financial performance.

Previous research views IS effectiveness as IS success (Delone et al. 1992).
Following on from DeLone and McLean, Chan et a1 (1997) shows that IS effectiveness is
a combination of user information satisfaction and organizational impact. This study will
incorporate Chan et al’s IS effectiveness construct into its proposed model; however,
only the organizational impact items will be used. The IS effectiveness construct will be

conceptualized as a second order factor comprised of two constructs that, include 18’

27

contributions to operational efficiency and management effectiveness. The constructs are
defined as follows. Information systems contribution to operational efficiency assists
managers with day-to-day organizational tasks such as contributions to the establishment
of market linkages regarding the organization's supply chain and customer relationships.
Information systems contribution to management effectiveness assists managers in their
decision making capabilities such as the creation and enhancement of products and
services that aid the organization in new product development and extensions of existing
products and services. Thus, this study will adapt the IS effectiveness construct to reflect
perceptions of IS portfolio effectiveness. The sections that follow are an overview of
each hypothesis.
3.3 Hypothesis 1 - Management Practices and IS Portfolio Value

Properly fimctioning managerial controls can serve as the foundation for creating
IS value. Management practices that ensure that IS is properly aligned with the business,
gives IS the ability to respond efficiently and effectively to management. An IS portfolio
that can quickly implement management's actions gives an organization the opportunity
to effectively compete in its marketplace. Thus, constructs for the value of an IS
portfolio are introduced into the research model. Value, in terms of responsive and
flexible IS portfolios should be outcomes of efficient and effective management
practices. Therefore management practices should provide IS with a basis from which to
conduct efficient operations, applications management, IS strategic planning and cost
containment. In other words, the result of a highly valuable IS portfolio should be
efficient operation. Without a high value IS portfolio, organizations can appear to be

laggards in their industry due to their failure to match competitors' changing processes, or

28

their inability to integrate new partnerships or alliances. From the perspective of each
management practice (monitoring IS, strategic planning, managing the IT investment,
managing human resources & risk management), the following discusses how 18' value
can be increased. The first practice, monitoring, should accurately observe IS
performance and give management an additional tool for decision making. The more IS
related processes an organization can monitor, the more likely that both best practices and
deficiencies in IS management can be identified. Thus, IS value can be derived from
properly managed systems and processes revolving around the information systems
portfolio. Organizations that have properly functioning evaluation systems in place
should be able to create more valuable IS across their portfolios than organizations that
don't.

Furthermore, strategic planning in the IS function is a critical organizational
activity that must bring together both the business and IS perspectives to create high
performing IS that serve the business. IS value is thus created by the synergy between IS
strategic planning and business strategic planning. High value IS will, in part, result from
strategic planning that accommodates the input of both the business and IS functions.

Previous research has shown that many organizations' IS investments have
reached 50% or more of total capital expenditures (Mahmood et al. 1993), and recent
research shows that total IS spending levels can range from 2-15% of an organization's
gross revenue, depending on the industry (Gomolski et al. 2007). As a result, the
investment in IS has become a critical factor that determines whether organizations

achieve profitability. Improperly managed IS expenses can significantly and negatively

29

impact the value of IS in many organizations. Thus, IS value is, in part, realized by
maintaining cost controls across projects.

Conducting periodic risk assessments enhances IS value by reducing uncertainty,
which ensures business continuity for IS, as well as the protection of valuable data. The
periodic nature of the risk assessment allows IS value to be enhanced by iterative
planning that identifies potential risks and creates contingency plans to eliminate
uncertainty. Ultimately IS value is positively impacted by mitigating IS specific risk.

From a resource based view of the firm, human resources represent key
components in organizations that excel (Powell et al. 1997). IS value thus can be created
when organizations invest heavily in IS related human resources. When organizations
support their staff in a manner that recognizes them as valued resources, it is logical that
the IS portfolio is fully supported; thus IS value can be derived by properly managing IS
human resources.

Management practices that pr0perly institutionalize processes revolving around
the IS portfolio plan for changing business needs. These practices provide the
organization the opportunity to incorporate continuous improvement into daily operations
that mirror the changes necessary to compete in hyper competitive markets.

Specific to the responsiveness construct, management practices enable the
capability for IS to react to business changes. For example, change brought about by
external forces is provided for in organizational as well as IS risk assessment. The
management practice pertaining to IS risk assessment should incorporate the possibility
of external change. Those organizations that embrace the iterative nature of the risk

assessment practice should be more responsive than those organizations that do not.

30

Another example is from a human resources context, IS human resources should not
overload personnel with current projects since there is always the possibility of
unforeseen circumstances that might need addressing due to external pressure. Due to the
possibility of external change, greater instanstiation of IS management practices should
help an IS portfolio better respond toward changing business needs that are thrust open
the organization from external sources. Therefore the following hypothesis is offered:

H 1A — Greater use of IS management practices increases responsiveness within the IS
portfolio.

From a flexibility standpoint, value is derived when planned organizational
change can be easily incorporated into the IS portfolio. When organizations plan for
change, (i.e. in terms of new reporting relationships) the IS portfolio has the ability to
work with management to address changes in a more controlled or forward looking
manner. IS portfolio’s that have the ability, to absorb internally driven change quickly,
derive more value than IS portfolio’s that cannot absorb change very quickly. Processes
that most embody the concept of flexibility are strategic planning in the IS function,
managing the IT investment as well as managing IS human resources. Anticipating the
needs of the organization that the IS portfolio supports allows for structural change more
quickly. Shifts in allocations of dollars invested and personnel in a timely manner allow
for organizations to have an easier time reaching their goals and objectives with regard to
generating revenue and reducing costs. IS management practices that help the IS
portfolio readily incorporate internally driven change create the opportunity for
flexibility. Instantiated processes and procedures gives the IS portfolio the ability to
anticipate the organizations’ future problems, needs or process changes. Therefore the

following hypothesis is offered:

31

HIB - Greater use of IS management practices increases flexibility within the IS
portfolio.

3.4 Hypothesis 2 - Management Practices and 18 Portfolio Risk

When practices specifically intended for IS management are codified in
organizations and enacted by managers, those practices should aid in proper IS portfolio
management. Therefore, those controls should mitigate enterprise wide IS portfolio risk.
IS specific practices should also properly control the systems development process and
the maintenance of enterprise-wide systems. The reasons include the high costs of
systems development and maintenance, the high costs of IS infrastructure and the IS risks
that can harm the organization. IS portfolio level risk results from IS projects that have
enterprise wide impact. These risks include age of applications (Swanson et al. 2000),
budgetary oversight (Barki 2001) and user satisfaction (McKeen et a1. 1994).

. The following is an explanation of IS portfolio risk showing how each of the five
practices previously mentioned serve as a basis for controlling IS management and
mitigating IS portfolio risk.

The ability to identify and evaluate potential risks is critical for reducing
uncertainty within the IS portfolio. Continuous monitoring must be in place in order to
identify and act upon negative performance deviations. Periodic evaluations ensure that
the IS function is always a positive contributor to the organization. Examples include
monitoring legacy applications' viability to support organizational business processes,
and associated maintenance support contracts. Other monitoring includes close attention

to budgetary concerns. Thus, risk should be mitigated with this evaluation.

32

IS strategic planning that is properly aligned with business needs serves as a key
contributor to organizational efficiency. Planning that incorporates organizational goals
can also mitigate risk across the portfolio by synchronizing projects and resources.
Possible cost savings can be identified in advance, along with proper forecasting of
human resources.

Because IT budgets are such a large portion of many organizations' administrative
budgets (Maizlish et al. 2005), appropriate budgeting that properly supports
organizational goals and eliminates impr0per expenses should, in fact, decrease risk in
the IS portfolio. Cost savings can be identified by long range budgeting. Examples
include proper management of software licenses, desktop replacement or maintenance
contracts. Vendor management is another key area in terms of the use of outside human
resources or the purchase of new equipment. Furthermore, determinations regarding
make or buy decisions (in terms of application development) are also critical. Thus
proper cost control should mitigate risk across the portfolio.

Properly managing staff is one of the most critical areas of any organization,
regardless of organizational type or business function. Technical staff members may be
considered more critical than other organizational staff members due to the specialized
knowledge they possess. The following factors can mitigate risk to the IS portfolio.
Staff roles should be clearly defined to reduce any ambiguity about the duties that are to
be performed. Furthermore, structural assurance in terms of the general workplace must
be adequate, meaning that IS technical staff members should feel safe and confident in
their position in the organization they work for. Any role ambiguity or lack of structural

assurance causes many employees to perform below their optimum, or possibly to leave

33

the organization, thereby increasing expenses to replace them and train their
replacements.

IS risk assessment and management are ongoing activities that identify possible
business uncertainties due to IS and synchronize those uncertainties with the
organization. IS risk is inherent in all projects and may adversely affect the IS portfolio
and the organization as a whole. IS risk assessment must therefore be done along with
plans to prioritize IS risk and implement responses to those IS risks. Ongoing IS risk
management should be integrated with the organization’s enterprise wide risk
management framework. A direct examination of the IS portfolio's risk should lead
organizations to incorporate possible risks into their strategic and tactical IS plans. Thus,
a thorough IS risk assessment should properly identify and address the majority of risk,
which then should be mitigated. Therefore the following hypothesis is offered:

H2A — Greater use of IS management practices reduces portfolio risk within the IS
portfolio.

Lack of compatibility or synergy can increase many risks associated with systems
maintenance. These risks range from platform incompatibility to data conversion or
adjustments that need to be made on a periodic basis between applications. Synergy
should thus be created among applications and personnel due to risk integration and
organization wide strategic planning.

Enacted management practices allow for processes that eliminate low levels of
synergy among applications, personnel and business processes. Applications that pass
data among them more easily in organizations reduce risk as opposed to those
organizations where data routinely falls out or has to be transferred by intermediate

processes. Increased synergy among applications protects data integrity and reduces

34

rework. Personnel in organizations who’s skills are prOperly matched with the systems
they maintain and manage have higher levels of synergy with the IS portfolio than in
organizations that do not properly match skill-sets to applications. Finally, when
business processes in organizations are well matched with the available technology and
personnel skill-sets, the synergy within the IS portfolio should be higher than in
organizations where skill-sets and applications are not properly matched with the
available technology. Manual work-arounds and rework are the norm in organizations
that do not properly match the technology and skill-sets with the business processes in
place. When the aforementioned areas are properly controlled through appropriate
management there should be increased synergy within the IS portfolio that in turn help

mitigate risk. Therefore the following hypothesis is offered:

H2B — Greater use of IS management practices increases synergy within the IS portfolio.

3.5 Hypothesis 3 — IS Portfolio Value and IS Effectiveness

The IS portfolio's responsiveness (i.e., reactionary position) can be of great value
to organizations that continually change business processes in response to either internal
management mandates or exogenous shocks fiom the business environment. When
responsiveness is high, organizations have the capacity to make decisions faster, facilitate
a higher velocity of change within the organization and respond to the external
environment more rapidly than when responsiveness is low. Internal operations can run
more smoothly within and beyond the IS function. Higher velocity in decision making
with more accurate and timely data should increase management's effectiveness.

Therefore the following hypothesis is put forth:

35

H3A — Highly responsive IS portfolios increase IS eflectiveness.

Flexibility represents the IS portfolio's proactive position. When flexibility is
high, organizations have the capacity to use the IS portfolio as a tool for competitive
advantage. IS serves as the key to spearheading new efforts to attract and maintain
relationships with customers. New markets become available with the use of an IS
portfolio that can seamlessly integrate new marketing opportunities. Other opportunities
also become available, such as the abilities to enter partnerships, alliances and absorb
potential takeovers. Thus operational efficiency and managerial effectiveness can be
greatly enhanced. Therefore the following hypothesis is put forth:

H33 — Highly flexible IS portfolios increase IS effectiveness.

3.6 Hypothesis 4 - IS Portfolio Risk and IS Effectiveness

Because risk might cause IS to negatively impact the organization, the objective
of any organization is to mitigate such risk surrounding the IS portfolio whenever it is
cost effective to do so, given the probabilities and effects of negative outcomes. From a
portfolio perspective, managing legacy applications in terms of maintenance and cost
control represents an area of possible risk mitigation. Organizations that properly control
the aforementioned risks within their IS portfolio will be more likely than organizations
that don't to effectively manage their information systems portfolio. Thus, the following
hypothesis is put forth:
H4A — Lower levels of IS portfolio risk increase IS effectiveness.

Synergy surrounding the IS portfolio plays a critical role in the organization's

ability to manage effectively and run an efficient operation, where synergy refers to the

36

degree of compatibility among applications, processes and people. Therefore a lack of
synergy is an indication of increased risk throughout the portfolio. Applications that
work seamlessly with one another eliminate the need for conversion programs or
maintenance and eliminate the risk of data fallout. Furthermore, when the applications in
use properly match the organization's technical infiastructure, there is less of a need for
technical support and specialized expertise to keep the enterprise up and running. The
compatibility of staff member’s technical skill-sets with the applications in use at the
organization also serves as a key source of operational efficiency. Other areas where lack
of synergy hinders IS effectiveness can be between applications, business processes and
users. When business processes can be run with little human interaction at each step in
the process, high levels of synergy exist between the applications and the business
process. Furthermore, when users feel comfortable using the organization's applications,
they should feel at ease relying on the systems for the purpose of managerial decision
making. Therefore the following hypothesis is put forth:

H4B - Lower levels of synergy reduce IS eflectiveness.

This completes the introduction of the hypotheses. The next chapter will introduce the
research method.

37

CHAPTER 4: RESEARCH METHODS

Chapter four describes the research design and methods that will be employed to
test the hypotheses. The chapter begins with a statement of the research purpose and
objectives. Next, the chapter outlines the research methodology, including the unit of
analysis, sampling procedure, data collection, instrument development and data analysis
procedures. Finally, the chapter concludes by directing attention toward the expected
results and implications of the study.
4.1 Research Methodology

This section describes the specific research methodology utilized in this study and
discusses the relevant units of analysis. The sampling procedure is identified and the
proposed data collection method is described. Furthermore, this section provides a
description of the instrumentation to be used in this report. Finally, the results and
analysis of interviews are presented.
4.2 Unit of Analysis

The organization represents the unit of analysis for this research, since each
organization has its own management practices in place as well as its own portfolio of
software applications. Hence, each organization is uniquely positioned to compete in its
given economic environment. In order to be included in this study, organizations must
already have some sort of centralized way of managing IS. Thus, the organizations will
either have a program office or tools in place that allow an enterprise level view of all
projects and applications within the organization.

For purposes of testing the hypotheses and research model, the target level of

analysis is the organization, whereby each organization will be asked to provide at least

38

one manager as an informant. The informant will be a representative of the IS function;
be or she will be knowledgeable about the practices of the IS function and how the IS
portfolio enables the organization. The types of targeted IS informants will be CIO’s,
senior IS managers, project managers, business analysts, auditors and portfolio managers.
Since the entire survey will be completed by one informant (i.e. all of the variables will
be collected from a single source), the survey results are susceptible to common methods
variance. Therefore, Harmon’s one factor test will be performed to attempt to detect the
presence of common methods variance.

4.3 Sampling Procedure

The target population for this study includes portfolio and program managers, as
well as other IS managers that are familiar with IS management practices. Furthermore,
IS auditors will be targeted due to their organizational as well as their IS knowledge.
Therefore, the target population will be derived from organizations that currently employ
IS portfolio and program management to supervise the implementation and use of IS, or
that have a centralized manner in which to manage IS. These organizations will most
likely be Fortune 1000 companies and large government agencies, given the complexity
and scale that is required in an organization for the use of portfolio and program
management.

The next step in identifying the sample is the sampling frame. The flame is a
listing of elements from which the actual sample will be drawn and should be consistent
with and representative of the phenomenon. The targeted sampling frame for this
research is the IT Service Management Forum (an industry group dedicated to the

advancement of the Information Technology Infrastructure Library (ITIL) practices) and

39

the Information Systems Assurance and Control Association (ISACA). The IT Service
Management Forum provides an excellent source of respondents, since many of their
members are IT professionals that possess business skills and usually sit at the
intersection of IT and the business in their respective organizations. The ITSMF
association was identified due to its appreciation for IS portfolio and program
management techniques as well as their use of ITIL practices, which are similar to the
COBIT framework.

Next, a determination is made for the optimal sample size for this study.
Therefore, the desired power for the sample is important, especially since power
represents the probability that a significance test will reject the null hypothesis when a
particular alternative value of the parameter is true. With regard to this study, the
expectation is that the hypothesized model will explain a moderate amount of variance in
the dependent variable IS effectiveness. Therefore a target R2 of .3 is sufficient for a
medium effect size. In addition, a reasonable level of power for the sample is .80. Given
these two criteria, R2 and power, the optimal sample size is 84 at a .05 level of
significance (Cohen et al. 2003). Next, there will be a discussion of the tool used to
evaluate the research model and the appropriate sample size given the use of the
evaluation tool.

This study will utilize Partial Least Squares (PLS) to analyze the data. Particular
attributes of PLS are as follows: PLS allows for a small number of cases to be analyzed,
including those with modest sample sizes less than 150. PLS is preferred over regression
due to the ability of PLS to obtain data regarding discriminant validity between the

constructs. The independent variable (management practices) and the dependent variable

40

(IS effectiveness) are second order factors, and PLS has the ability to properly evaluate
the effects of second order factors. Finally, PLS can evaluate data samples that have
distributions that depart from normality.

4.4 Data Collection

Data will be collected by surveying IS managers and IS audit managers regarding
their perceptions about IS management practices, IS risk, IS value and the effectiveness
of IS in their respective organizations. The choice of survey versus experiment is driven
by a number of factors. First and foremost is the fact that IS portfolio/program
management is an intricate undertaking that exists in large organizations for the purpose
of maintaining complex enterprises. Therefore, the replication of such an environment in
a laboratory setting is not possible. Furthermore, the use of surveys can provide the
researcher with a high level of external validity. Laboratory experiments, on the other
hand, provide the researcher with a high level of control in data collection and therefore
allow any external influences to be removed. Additionally, the survey method allows
respondents to remain anonymous to increase their willingness to participate.

This study combines an initial phase of qualitative inquiry with the
aforementioned survey methods. Three organizations were identified that use some form
of IS portfolio management techniques. The organizations that participated in the
exploratory interviews were a large publicly traded provider of confectionary products, a
Detroit 3 auto maker and a not for profit hospital chain. A single informant at each
organization (i.e. a senior level IS executive) was asked open-ended questions regarding
his or her IS management practices involving project management, project monitoring

and project planning. Other areas of inquiry included staff competencies and the

41

approach to complying with regulatory standards (see Appendix A for the questionnaire
and Appendix B for answers to the questionnaire).
4.5 Data Collection Method

Survey data will be collected via web survey. The web survey and data collected
will be hosted at www.zipsurvey.com. Although response rates are sometimes lower
with web surveys, research shows there is no sampling bias or differences in the content
of responses (Matz 1999).
4.6 Instrumentation

The survey instrument was designed to allow respondents to provide their
perceptions regarding the areas of inquiry (management practices, value, risk and IS
effectiveness) that represent the high level constructs in the proposed research model. In
addition to demographic questions, the survey items consisted of seven point Likert scale
questions. Demographic information served as a control mechanism in the model.
Variables

The research model used for the study incorporated both previously tested items
and new items that directly address the organizational impacts of IS portfolio
management. The dependent variable, IS effectiveness, incorporates the previously
tested constructs operational efficiency (Chan et al. 1997; Jiang et a1. 2001) and
management effectiveness (Chan et al. 1997). Mediator variables for IS Value
(responsiveness & complexity) as well as portfolio risk variables (risk & synergy) were
created based upon discussions with practitioners and academics. Finally, the exogenous
independent variables, management practices, were sourced from the COBIT framework

and are previously untested in the literature.

42

The following provides details of the survey items for the management practices
as well as the other constructs. The results of the interviews helped substantiate the IS
management practices in use at the sample of organizations. These results were mapped
directly to the COBIT structure and transformed into survey question items. Listed
below in Table 2 are all the management practices and the corresponding questionnaire
items. Each item is a COBIT control item listed in the control framework, and has been
adapted into the form of a question. After the listing of IS management practices
questionnaire items, Table 3 lists the items for IS value, both responsiveness and
flexibility. These items were created by generating a list of processes and procedures that
are likely to be very important to most organizations based upon previous academic and
practitioner literature. The list was then reviewed by other researchers and a final list
was decided upon among the researchers. Table 4 lists the questionnaire items for IS
risks, portfolio and synergy while Table 5 lists the items for IS effectiveness. Given that
the COBIT items as well as the IS value and IS risk items have not been previously
tested, a confirmatory factor analysis was performed to determine the validity of the
construct items. Furthermore, the other items (i.e. the IS effectiveness dependent
variable), albeit previously used by other researchers, were included in the factor
analysis. The results of pilot testing determined that each of the management practices
are indeed separate factors and the five management practices form a reflexive second
order factor. The pilot testing results also indicate that the IS effectiveness constructs are

two separate factors that form a reflexive second order factor.

43

TABLE 2

IS Management Practices Questionnaire Items

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Construct Source Questionnaire Item
Monitor and MEI .1 A general framework exists for monitoring the IS
Evaluate IS portfolio management processes specific to the delivery
Performance of IS services.
0")
ME1.2 Performance objectives are clearly defined.
MEI .2 Methods are in place to collect relevant monitoring data.
ME1.3 A monitoring method is in place (e.g. a balanced score
card) that provides an all around view of IS performance
ME1.4 A performance assessment process is in place to
periodically review IS performance against targets.
ME1.5 Performance reporting on IS across the enterprise is
available for senior Management/Executive board
Define a Strategic P01 .1 Evaluation of business cases ensures that the IS portfolio
IS Plan is a source of value for the organization
(IV)
POI.2 Business and IS strategies are integrated clearly linking
enterprise goals with IS goals
POI.3 There is a performance assessment in place to evaluate
existing strategic IS plans
POI .4 A strategic IS plan is in place integrating costs with risks
POl.5 Tactical IS plans are derived from strategic plans
Manage the IT P051 A financial management framework is in place for the IS
Investment (IV) function that properly evaluates business cases for new IS
investments
PO52 A prioritization plan is in place for new IS investments
POS.3 There is an IS budgeting plan in place that incorporates
the entire IS portfolio
POS.4 There is a process in place to compare actual IS costs to
budgeted IS costs
POS.5 A process is in place to track IS contribution to business

 

 

results

 

 

 

44

 

Table 2 Continued

 

 

 

 

 

 

Construct Source Questionnaire Item
Manage IS HR (IV) PO7.1 IS personnel recruitment processes are in line with the
organizations
PO7.2 IS personnel maintain the proper competencies to fulfill
their roles
PO7.3 IS staff roles are properly defined
PO7.4 IS staff are provided with adequate amounts of training
to maintain their knowledge, skills and abilities
PO7.5 Critical dependency on key individuals is minimized

(e.g. in terms of knowledge sharing, documentation,
succession planning etc)

 

 

 

 

 

 

 

Assess and Manage PO9.I IS risk management is integrated within the
IS Risks (IV) organizations’ risk management framework
PO9.2 An IS risk framework is in place that incorporates
criteria to evaluate risks
PO9.3 Possible risk events are identified with potential impacrs
PO9.4 An IS risk assessment process is in place across the
entire IS portfolio '
PO9.6 There is a plan in place to prioritize IS risk
PO96 There is a plan in place to implement responses to IS risk

 

 

 

 

 

45

 

Table 3
IS Value Questionnaire Items

 

 

 

 

 

 

 

 

Construct Questionnaire Item
IS Portfolio Value To what extent is the IS portfolio responsive to meeting current application needs
— Responsiveness '
(Mediator)
To what extent is the IS portfolio responsive to integrating emerging application
needs
To what extent is the IS portfolio responsive to addressing user driven business
process redesigns
To what extent is the IS portfolio responsive to integrating emerging business
opportunities
To what extent is the IS portfolio responsive to changes in the organizational
regulatory environment
IS Portfolio Value To what extent is the IS portfolio readily able to support the following within your
- Flexibility organization
(Mediator)

 

Adding new customers

 

Adding new vendors

 

Creating strategic alliances with other organizations

 

Integrating new technology
Entering new markets

 

Changes in organizational strategy

 

Changes in organizational structure

 

Emerging business processes

 

 

 

implementation of new business models

 

46

 

Table 4
IS Risk Questionnaire Items

 

 

 

 

 

 

 

 

 

 

Construct Questionnaire Item

IS Portfolio Risk Across the IS portfolio what % of your core applications are l — 5 years old, 6 — 10

(Mediator) years, older than 10 years) — split into 3 questions
Across the IS portfolio what % of projects are delayed in their implementation
Across the IS portfolio what % of projects run over budget
Across the IS portfolio what % of the applications meet user needs in terms of cost?
Across the IS portfolio what % of the applications meet user needs in terms of time?
Across the IS portfolio what % of the applications meet user needs in terms of
satisfaction?

IS Portfolio Risk - To what degree is there synergy among the applications in the IS portfolio

Synergy (Mediator)

 

To what degree is there synergy between the applications in the IS portfolio and the
skill sets of the IT personnel?

 

To what degree is there synergy between the IS portfolio and the training that is
provided to the IT personnel?

 

To what degree is there synergy between the applications in the IS portfolio and the
current technical infrastructure

 

To what degree is there synergy between the applications in the IS portfolio and
your organization’s business processes

 

 

 

To what degree is there synergy between the applications in the IS portfolio and the
staff that use the system

 

47

 

Table 5

IS Effectiveness Questionnaire Items

 

 

 

 

 

 

 

 

 

Construct Source Questionnaire Item

Operational Chan et a1 1997 How adequate is the portfolio of information systems in

Efficiency meeting the needs of your area of responsibility

(DV)
How adequate is the portfolio of information systerrrs in
meeting the needs of the different variety of users it serves
To what degree does the IS portfolio process contribute to the
creation of new products and services
To what degree does the IS portfolio process contribute to the
enhancement of existing products and services

Jiang et a1 2001 How adequate is the portfolio of information systenrs in

contribuu'ng to revenue generation and cost reduction across
the organization

Management Chan et a1 1997 To what degree does the IS portfolio process contribute to the

Effectiveness improvement of managerial decision making

(DV)

 

To what degree does the IS portfolio process contribute to the
improvement of budgeting and planning processes

 

To what degree does the IS portfolio process contribute to the
management of your span of confiol (over your subordinates)

 

To what degree does the IS portfolio process contribute to
customer relationship management

 

 

 

 

To what degree does the IS portfolio process contribute to
supply chain management

 

48

 

4.7 Survey

The survey was administered using the online survey tool (www.zipsurvey.com).
It was hosted entirely online and accessed by respondents via a secure URL. The survey
was administered to two different organizations: the IT Service Management Forum
(itSMF) and the Information Systems Audit and Control Association (ISACA). The
itSMF survey was conducted in May 2007 using the sampling frame of the Great Lakes
Chapter of the itSMF, in which its professional membership of 260 is located throughout
the state of Michigan. The survey link was forwarded by the chapter president to the
organizations’ membership. This was done because the organization does not make their
members’ e-mail addresses available for any reason. The majority of the respondents
completed the survey within 48 hours of the e-mail announcement (approximately 80%
or 39 respondents). Two weeks later, a reminder notice was sent out in order to obtain
additional surveys (the remaining 20% or 10 respondents were collected). There were 49
total responses received out of a total membership of 260, for a total response rate of
18.8%.

With the ISACA organization, the survey was conducted in September 2007 using
the sampling frame of the Houston Chapter, in which there are approximately 950
professional members. Similar to the itSMF, the survey link was forwarded to the
chapter president for dissemination to the entire group due to privacy concerns. Again,
the majority of the respondents completed the survey within 48 hours of the e-mail
announcement (again approximately 80% or 40 respondents). Two weeks later, a
reminder notice was sent out in order to obtain additional surveys (the remaining 12

responses were collected).

49

Sample Size

The resulting total sample size was 101 respondents. F orty-nine respondents were
from itSMF. The total population of potential respondents was 260. The response rate
was therefore 18.8% from itSMF. Also note there were 104 hits to the website from the
itSMF members. The remainder of the responses, 52, came from the Houston ISACA
Chapter. The total population of potential respondents was 950. The response rate was
therefore 5.3% from ISACA. Overall, the response rate was 8.3%. Also note there were
131 hits to the website for the ISACA members. The overall small sample size might be
due to a few factors. Since the distribution of the survey was through each organization’s
membership contact (due to privacy concerns) it was not known if all of the sample
population was contacted. This might be due to inaccurate e-mail addresses, firewalls or
the possibility that the email generated by the organizations’ membership might have
been classified as spam to some e-mail servers. Furthermore according to Matz (I999)
response rates to web surveys have been found to be lower than paper surveys however

the low response rates were not biased.

50

CHAPTER 5: ANALYSIS OF DATA

5.1 Introduction

Chapter Five details the research results. First, the qualitative results are
discussed. Second, the sample size obtained, as well as sample characteristics. Third, a
factor analysis is conducted. Fourth, the research model is evaluated, including a
discussion of the results of each hypothesis. F ifih, a test for mediation is completed.
Finally, an analysis is conducted for the different types of respondents. The Chapter
concludes with a summary of the research findings.
5.2 Qualitative Findings

Overall results indicate major differences among the organizations interviewed.
The organizational configurations for how IS supports the business differ substantially
among organizations. Configurations for IS functions range from totally centralized to
totally decentralized. For instance, at the hospital chain, the IS staff is centralized
utilizing a program office approach as well as using liaisons to interact with the various
hospital departments. The program office maintains a single database of all projects that
require over two weeks of effort to complete. Within the database, financial and non-
fmancial metrics such as man-hours and projected tirnelines are maintained. The entire
portfolio of applications is continually monitored for return on investment and possible
risks. On the other hand, the Detroit 3 auto maker’s IS function supports business
activities within each business unit. A program office or portfolio management approach
is not in use, as each business unit maintains local control for its own IT. Budgeting and

project tracking are all maintained within the business unit, with the exception of a few

51

large organization wide projects. These projects are tracked by a small project office at
the organizational level.

In terms of proj ect management support, tools for project tracking and
prioritization also differ significantly. Tools are used at some firms, but not used at
others. In the case of the hospital chain, a portfolio management software tool is used to
track a variety of financial and non-financial metrics on all IS efforts that are large
enough to be considered projects (i.e., a project that requires over two weeks’ worth of
effort). This tracking represents both monitoring and evaluating performance, as well as
managing the IS investment management practices. The confectionary and the Detroit 3
auto maker currently use a mix of Microsoft tools such as Excel and Project to monitor
their portfolios. Although all the organizations have processes in place for project
definition, justification and strategic planning, each of these elements differs significantly
between organizations. In terms of IS human resources, the confectionary organization
uses a staffing database that tracks availability and skill-sets to match the right people to
the right projects world-wide (i.e. relates to the manage IS human resources management
practice). The confectionary organization and the hospital chain both conduct IS risk
assessment and strategic planning in conjunction with the businesses they support (i.e. an
example of the define a strategic IS plan management practice and assess and manage IS
risks management practices). Overall, each organization uses a significantly different
method to organize its IS function to support the business. Thus, the results of the
interviews support the use of contingency theory to frame this research, in so much as

there is no single way to organize and each organization develops to create a fit with its

environment.

52

5.3 Preliminary Analysis -— survey, sample size, sample characteristics

Across both organizations that were surveyed the sample characteristics are as
follows. A majority of the respondents, 57% held the title of manager or above. A large
percentage of the respondents (35%) chose other as their title. Many of these in the other
category were business analysts or IT auditors. Most of the respondents listed their
reporting responsibilities at the corporate level (74%) as opposed to reporting to a sub-
unit or corporate division (26%). Other demographic characteristics include respondent
experience levels with respect to time spent in the industry, at an average of 15.4 years,
time spent in their respective positions, at almost 7 years, and time spent with their
current employer, at just over 4 years. Revenue reporting indicates that just over half of
the organizations had over $1 billion in sales, while approximately 20% had sales of less
than $100 million. A majority of the respondents, 55%, report that their operations are
primarily global. The scope of the organizations’ activities includes a significant
presence in the financial, technological and service related sectors. Also, 85% of the
respondents indicated that they belonged to for-profit enterprises. Finally, with respect to
employees, 60% of the respondents worked at firms with fewer than 5,000 employees,
while the other 40% worked organizations with over 5,000 employees. The initial study
design was meant to target large organizations; however, a significant portion of the
respondents came from smaller organizations with respect to number of employees and
revenue, which was unexpected. These results do not represent a risk of undermining the
results of the study since these firms indicated that they do use a centralized manner in
which to monitor there IS functions. The inclusion of these smaller firms do however

provide an opportunity to introduce a large amount of variability into the results. Finally,

53

the results from all the other demographic questions were as expected. Table 6 below

details the demographic characteristics of the sample collected.

54

Table 6
Sample Characteristics

r..e.
analysts, IT auditors)

 

55

5.4 Factor Analysis
Prior to testing the hypothesized model, an examination of the measurement
model is conducted through confirmatory factor analysis utilizing structural equation

modeling. Table 7 that follows presents the results of the confirmatory factor analysis.

 

 

 

 

 

 

Table 7
Confirmatory Factor Analysis Results
Fit Indices
CH .932
TLI (NNFI) .911
RMSEA .057
90% CI (.046 - .067)
x2 = 828.54 with 626 degrees of freedom

 

 

Using an ad hoc measure of fit by dividing the estimated chi-square by the degrees of
freedom (x2/dt), one can get an estimation of goodness of fit (Bollen 1989). Good model
fit can be demonstrated by obtaining a ratio of approximately 2 or less. In the current
model the resulting ad-hoc statistic (828/626) or 1.32, provides one indication good
model fit. Also RMSEA (.057) does provide a fair to level of fit of the data to the model.
Browne and Cudeck (1993) consider RMSEA results between .05 and .08 to be an
indicator of a fair level of fit of the data to the model. Further indications of a fair level
of fit is that the results for the comparative fit index (CFI = .932) and the Tucker Lewis
Index, or non-normed fit index (NNF I = .911) suggest that the data’s fit to the model is
fair in that it does not reach the recommended cutoff to indicate good model fit by Hu &

Bentler (1999), as well as Kaplan (2000).

56

 

These results suggest that there is a moderate fit of the data to the model however these
results cannot be considered an excellent level of fit. The following table 8 lists the final
questionnaire items and their loadings as a result of the confirmatory factor analysis.

Note that an LM test was performed to remove items that resulted in high cross loadings.

57

Factor Loadings

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Scale Item Factor Scale Item Factor
Loading Loading

Monitor & Evaluate Operational
Performance (MEP) Efiiciency
MEPI .644 0E2 .61 7
MEPZ .805 0133 .474
MEP3 .779 0134 .769
MEP6 .680

Management

Eflectiveness
Define a Strategic Plan M131 _901
$184 .860 ME2 .844
$185 .817 ME3 .667
Manage the IT Investment Portfolio

Risk(proj

overruns)
M113 .872 PR4 .764
M114 .616 PR5 .633
Manage IS Human Resources Portfolio

Risk(user

needs)
MHRl .644 PR6 .602
MHR2 .767 PR7 .816
MHR3 .847 PR8 .529
MHR4 .627
Assess & Manage IS Risks Synerg)
ASR3 .890 SS .881
ASR4 .932 S6 .741
ASRS .801
Flexibility
F4 .615
F5 .659
F9 .852
Responsiveness
R1 .742
R2 .844

 

 

 

 

 

 

 

 

 

Note that, given the results of the factor analysis, the age of portfolio risk (age) construct
was dropped from the model. The age (risk) items might not have loaded to due the fact
the items asked the respondents to give interval estimations on age. Also the respondents
may not have been certain on the age of applications. Items from the management
constructs that were eliminated may have been due to the similar nature of the items and
their interpretations by the respondents. The similarity of management practices
constructs is supported by the high correlations among them, see table 9 for results. For
the responsiveness construct only the items that dealt with current and future applications
held together. This might be due to the respondents thinking specifically about
applications only and not the broader business. On the other hand the flexibility
construct only had items that held together that included integrating new technology,
entering new markets and implementing new business models which speak to the broader
business as opposed to the specific application or task. The effectiveness constructs also
had items culled in that the Operational efficiency construct only had items that held
together that spoke to meeting the needs of the business. The management effectiveness

construct had items that held together that spoke directly to managerial issues.

59

5.5 Model Evaluation
For the structural and measurement model the resulting items from the CFA are tested
utilizing the partial least squares structural equation modeling technique. This method is
preferred over continuing to use the SEM based technique for the CFA due to the small
sample size and the ad hoc tests performed on a subset of the data.
5.5.1 Measurement Model

The measurement model was evaluated by using partial least squares to assess the
constructs’ psychometric properties. PLS is often used to evaluate research models with
moderate sample sizes (i.e. less than 150) (Gefen et al. 2000). PLS assesses reliability
and validity by calculating the internal composite reliability (ICR) and the average
variance extracted (AVE). The ICR is interpreted in much the same way as Cronbach’s
Alpha; according to Fomell and Larker (1981), an ICR of .7 is sufficient. The ICRs
reported in Table 9 suggest sufficient reliability for all constructs; the lowest reported is
.91. The AVE measures variance explained relative to measurement error. A valid
construct that consistently measures what is intended has an AVE greater than .5 (Chin
1998). Table 9 shows that all the constructs have an AVE of at least .57, indicating

sufficient convergent reliability.

60

Table 9
Correlation Matrix, ICR, AVE

 

Construct Mean SD. [ICR 1 2 3 4 5 6 7 8 9 10 ll 12

 

1.Responsiveness 4.56 1.25 .95 .89

 

2.Flexibility 4.75 1.09 .90 .81 .81

 

3.PortfolioRiskl 4.79 2.26 .91 .31 .44 .92

 

 

 

 

 

 

 

 

 

 

 

4.Synergy 4.83 .98 .91 .72 .79 .37 .92

5.0perational 4.53 .99 .91 .73 .71 .33 .67 .75

Efficiency

6Managerial 4.15 1.29 .95 .54 .62 .35 .63 .74 .89

lEffectiveness

7.MEP 4.25 1.71 .96 .53 .41 .23 .44 .41 .28 .89

8.818 4.22 1.52 .95 .56 .49 .38 .55 .45 .42 .80 .89

9.MHR 4.43 1.34 .94 .57 .63 .32 .69 .50 .49 .45 .57 .86

10.MII 4.39 1.45 .93 .53 .48 .27 .60 .49 .46 .70 .86 .52 .86

11.ASR 4.13 1.56 .97 .67 .60 .27 .74 .58 .49 .68 .75 .59 .69 .92
12.Portfolio 6.02 1.70 .94 .62 .72 .53 .65 .58 .45 .24 .33 .51 .37 .39 .89
Risk2

AVE .79 .65 .84 .68 .57 .79 .80 .79 .75 .74 .85 .80

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Discriminant reliability requires that constructs be distinct from one another. The
square roots of the AV Es for two latent variables must each be greater than the
correlations between those two variables (F omell et al. 1981). An examination of the
correlations among constructs in Table 9 above indicates that all of these relationships

pass the discriminant reliability test. Where table 10 below lists the outer model loadings.

61

Table 10

Outer Model Loadings — Structural Model

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Scale Item Outer Scale Item Outer
Model Model
Loading Loading

onitor & Evaluate Flexibility
Performance (MEP)
MEPI .852 F4 .73 8
MEP2 .916 F5 .886
MEP3 .903 F9 .95 5
MEP6 .889

Responsiveness
Define a Strategic Plan R1 .868
$184 .860 R2 .899
$185 .81 7

Synergy
Manage the IT Investment SS .952
MII3 .944 S6 .949
M114 .91 7

Operational

Efi‘iciency
Manage IS Human Resources 0E2 .863
MHRI .867 0133 .804
MHR2 .9 1 0 0E4 .9 l 4
MHR3 .929
NIHR4 ,8 34 Management

Effectiveness

MEl .95 1
Assess & Manage IS Risks ME2 .931
ASR3 .957 ME3 .900
ASR4 .971
ASR5 .940 Portfolio Risk(proj

overruns)

PR4 .908

PR5 .93 3

Portfolio Risk( user

needs)

PR6 .873

PR7 .898

PR8 .845

 

 

 

 

 

62

 

5.5.2 Structural Model

Partial Least Squares structural models are interpreted in the same manner as
regression models. The path coefficients represent standard betas, while the R2 amount
represents the variance explained. Given the hypothesized research model, figure 5

shows the results of the hypothesis tests. A discussion of each hypothesis follows.

' IS Portfolio Value
4 Responsiveness

R2=.42

,,,,___ *i

F _ fl 1
. H3A: .27.
“IA? 13-65 ”“ ‘ IS Portfolio Value B

Flexibility ,
R2=.24 j -

|__ __ __

H18: [3 .49 It". H38. B .47 ”a

 

IS Portfolio j ' H2A:[3—.3| m. Portfolio Risk I H4A:l3—.05 NS ,3 Effectiveness
Management " A ~ »~ ‘fi (budgetary) .

Practices ' ‘ R =_1o

mean-12.07 ' a _ J H4B:B.04 NS,

F’_““Tl
, Portfolio Risk j
(Users) 1. .
. ‘l R2=.18 '
H2CI B .48 nu

l H4C:B.08 NS

_ _ __ __ Sign—licence
l— ] «Level '

I . I
j Synergy ‘ 1.: “'10 l

- p<.05 .
"' - p <.01
" 45-201

 

Figure 5 Structural Model Results

63

5.5.2.1 Hypothesis 1A

Results for hypothesis 1A indicate that portfolio level management practices
explain variance in the responsiveness of the IS portfolio. The variance explained in
responsiveness is R2 = .42; the path coefficient is B = .65, and is significant at the p<.001
level. This result suggests that the controls that IS management considers most vital to
operating the IS function are indeed significant antecedents to IS portfolio value. From a
responsiveness perspective, the IS function has the ability to readily address change that
is due to complex business conditions.
5.5.2.2 Hypothesis 1B

Results for hypothesis 1B indicate that portfolio level management practices
explain variance in the flexibility of the IS portfolio. The variance explained in the
flexibility construct is at R2 = .24; the path coefficient is B = .49, and is significant at the
p<.001 level. From a flexibility perspective, the portfolio also has the ability to readily
address change that is created internally to match new business processes or
organizational strategy.
5.5.2.3 Hypothesis 2A

Significant results for hypothesis 2A suggest that risk in terms of budgetary
concerns (both in terms of time and cost) is mitigated by the use of portfolio level
management practices. The variance explained in budgetary risk is at R2 = .10; the path
coefficient is B = -.31, and is significant at the p<.001 level. Managerial controls in place

for planning and risk assessment can be the driving force behind risk mitigation.

64

5.5.2.4 Hypothesis 2B

Significant results for hypothesis 2B suggest that risk in terms of user satisfaction
is rrritigated by the use of portfolio level management practices. The variance explained
in risk in temrs of unmet user needs is R2 = .18; the path coefficient is [3 = .42, and is
significant at the p<.001 level. These results suggest that the managerial controls in place
lead to increased satisfaction on the part of users, since their needs are met in terms of 18
that support their operation.
5.5.2.5 Hypothesis 2C

Significant results for hypothesis 2C suggest that lack of synergy within the IS
portfolio is mitigated by the use of IS portfolio management practices. Results indicate
variance explained at R2 = .23; the path coefficient is B = .48, and is significant at the
p<.001 level. This suggests that synergy across the portfolio can be derived from properly
implemented portfolio management practices. Business processes, personnel skill-sets
and technical infrastructure within the portfolio work together more harmoniously when
supported by managerial controls that properly guide the IS portfolio.
5.5.2.6 Hypothesis 3A

Significant results for hypothesis 4A suggest that IS portfolio value in terms of
responsiveness is a significant antecedent to IS effectiveness. The path coefficient 0 =
.27 is significant at the p<.10 level. This finding suggests that an IS function that
maintains a moderate degree of responsiveness towards changing business needs
contributes to positive perceptions of IS effectiveness. External forces that exert pressure
on the performance of the IS portfolio can thus be absorbed to maintain positive

perceptions of IS effectiveness.

65

5.5.2.7 Hypothesis 3B

Significant results for hypothesis 4B suggest that IS portfolio value in terms of
flexibility is a significant antecedent to IS effectiveness. The path coefficient [3 = .47 is
significant at the p<.01 level. This finding suggests that an 18 function that is proactive,
and has the capability to act in anticipation of future problems, needs, or changes,
contributes to positive perceptions of IS effectiveness. Overall hypothesis 3 has a
significant impact on the IS effectiveness in which the R2 =.61.
5.5.2.8 Hypothesis 4

None of the hypothesized risks were found to have a significant impact upon IS
effectiveness. None of the path coefficients for hypothesis 5A, 5B or 5C reached the .05
level of significance. This might be due to respondents not believing that risk in itself
would generally be associated with negative effects upon 18 effectiveness. This will be
discussed fiirther in the next section. Table 11 below summarizes the results of the
hypotheses.
5.5.2.9 Control Variables

Control variables were tested in the model for size of the organization and the
experience of the respondents. Size of the organization was measured in terms of
revenue and number of employees both nationally and internationally. Experience was
measured in terms of the respondents’ time at their current employer as well as their time
in the industry. Results indicate that neither size nor experience were significant

predictors of the dependent variable IS effectiveness.

66

Table 11
Hypotheses and Results

Hypothesis

H1 0
H2 ' 0
H3 ectrveness

H4

 

5.6 Evaluation of the Use of a Second Order Structure

The second order structures are evaluated by examining the correlations of the
latent variables as well as evaluating the validity. In both the second order factors
utilized (management practices and IS effectiveness), the correlations are statistically
significant at the p < .01 level. As expected, the correlations among the management
practices are moderately high. This is more than likely due to the integrated nature of the
management controls performed. The evaluation of validity is conducted by examining
the magnitude of the path coefficients from the second order factor to the first order
factor (Segars et al. 1998). Results indicate that each first order factor has a significant (p
<.001) relationship with its respective second order factor (for both management
practices and IS effectiveness). Furthennore, the second order factors correlate with each
other at r=.56, indicating that they are discriminant from each other.
5.7 Test for mediation

The significant mediator constructs for value in terms of both flexibility and
responsiveness were tested to determine if they truly mediated the relationship between
management practices and IS effectivenessz. According to Baron and Kenny (1986), the

test for mediation includes 3 steps. First, the mediator must be regressed on the

 

Note that the risk variables were not tested for mediation srnce they were not Significant predictors of 18 effectiveness.

67

independent variable. The results indicate that both flexibility (B = .60 p <.001) and
responsiveness ([3 = .64 p <.001) are significantly predicted by the management practices
second order construct. The second step involves regressing the dependent variable on
the independent variable. The results indicate that the management practices construct is
a significant predictor of IS effectiveness ([3 = .56 p <.001). The final step involves
regressing the dependent variable on the independent variable and the mediators. Results
indicate that the flexibility construct is significant ([3 = .40 p <.01). However, the
responsiveness construct was not found to be significant (([3 = .28 p <. 12). Thus, the
conditions for true mediation according to Baron and Kenny have been satisfied for the
flexibility construct but not for the responsiveness construct.
5.8 Analysis between Auditors & IS professionals

Since audit professionals are attuned to examining risk and have a heightened
awareness of many risks in their organizations, it is plausible that audit professionals
might have different perceptions from IS professionals. Differences between the type of
professional might relate to risk, management controls or quite possibly the effectiveness
of the 18 function. A post hoc analysis is performed by examining each type of
respondent separately by analyzing the means of the constructs between the respondents
as well as analyzing the respondent types in the research model. Type is categorized by
which group the respondent came from. Those individuals responding from itSMF are
considered to have an IS professional orientation whereas those individuals responding
fiom ISACA are considered to have an audit professional orientation. Although it is
possible for an individual to have similar job titles in each group (i.e. for instance

business analyst). It is presupposed that an individual belonging to either group,

68

regardless of job type or title maintains a similar psychological orientation that is
comparable to their group only. Thus the many job types are not a concern. Each sample
by respondent type results in two samples of approximately 50 respondents. A t-test of
means for each construct, results in no significant difference between respondent types.
The following will discuss the results of examining each respondent type separately in
the research model.

Although the sample size of 50 is considered to be low for employing structural
equation modeling techniques it is not a concern for the reliability of the results in PLS.
Low sample size is not a problem since the minimum sample size in a PLS run model
should be at least 10 times the number of items in the most complex construct (Gefen et
a1. 2000). In research model used here, the most complex construct has 5 items.
Therefore a minimum sample size required is 5 times 10 or 50 respondents.

In order to determine whether there are any differences between respondent types
based upon the research model, a pooled calculation is performed (see Kiel et al. (2000)
for details). Results are given in Table 12 as follows. In Table 12, column 1 for the path
coefficients relates to the model run with auditor respondents and column 2 refers to the

path coefficients for the IS professional respondents.

69

Table 12

Auditor/IS Professional Path Comparison

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Model Relationships Path Coefficients Path coefficient

(hypotheses) differences
(1) vs. (2)
[t-values]3

(1) (2)

HlA - Management .70 .61 4.26***

Practices 9 Responsiveness

HlB - Management .66 .58 3.87***

Practices 9 Flexibility

H2A — Management .24 .39 -.4.97***

Practices 9 Budgetary Risk

H2B - Management .48 .33 5.42***

Practices 9 User Risk

H2C - Management .66 .48 8.84***

Practices 9 Synergy

H3A - Responsiveness 9 .52 .12 8.90***

18 Effectiveness

H3B - Flexibility 9 IS .03 .63 -11.94***

Effectiveness

H4A Budgetary Risk9 IS .04 .07 -1.10

Effectiveness

H4B User Risk 9 IS -.22 .04 -8.59***

Effectiveness

H4C Synergy 9 IS .39 -.004 -11.55***

Effectiveness

*** p<.001

Results based upon the t statistics indicate that all the paths in the model are

significantly different from one another based upon respondent type, with the exception
of hypothesis 4A which examines the impact of portfolio risk in terms of budgetary
concerns on the dependent variable information systems effectiveness. Furthermore, the
differences in the models indicate that auditors were more concerned with risk in terms of

lack of synergy impacting the IS effectiveness construct (i.e. significant impact), while in

the IS professionals model, synergy does not have a significant impact on IS

I effectiveness. Other differences include that the auditors have more concern with the

70

 

reactionary position of the IS portfolio. The responsiveness construct had a significant
impact upon IS effectiveness in the auditor model whereas, in the IS professionals model,
the responsiveness construct did not have a significant impact upon IS effectiveness. The
flexibility construct, on the other hand, was significant in its impact upon IS effectiveness
in the IS professionals model but was not significant in the auditors model. Finally, in
the auditor model, the management practices construct did not have a significant impact
upon the portfolio risk construct (budgetary concerns). In the IS professionals model,
however, the management practices construct had a significant impact on the portfolio
risk construct (in terms of budgetary concerns). All other hypotheses were found to be
equally significant in both models at the p <.001 level and are the same as the overall
model. Thus, one can conclude that each type of respondent is indeed different. Table 13
below summarizes the results of the hypotheses for the separate auditor and IS

professional models.

Table 13
Hypotheses and Results
IS Auditors/IS Professionals

 

 

 

 

 

 

 

 

 

 

 

Hypothesis Hypothesized Linkage Auditors IS
Support Professional
Support
H] A Management Practices 9 IS Portfolio Value Yes No
(responsiveness)
H1B Management Practices 9 IS Portfolio Value No Yes
(flexibility)
H2 A Management Practices 9 IS Portfolio Risk No Yes
(budgetary concerns)
HZB Management Practices 9 IS Portfolio Risk Yes Yes '
(synergy) l
H3 IS Portfolio Value 9 IS Effectiveness Yes Yes i
H4 IS Portfolio Risk 9 IS Effectiveness Yes No j

 

 

71

CHAPTER 6:.DISCUSSION AND CONCLUSION

6.1 Introduction

Chapter 6 presents the theoretical and managerial implications of this research
along with recommendations for managers. The recommendations are followed by
limitations of this research and the study is concluded by presenting directions for future
research.
6.2 Research Implications

The purpose of this study was to extend the academic literature on IS portfolio
management originally espoused by McFarlan (1981) and more recently Jeffery and
Leliveld (2004). Recent research has been sorely lacking in the IS portfolio management
area and there is a great need to expand researchers’ as well as managers’ understanding
of why organizations are turning to the use of IS portfolio management techniques to
oversee their IS functions. The overall goal of this study was to gain an understanding of
how organizations viewed their IS portfolios in terms of how they were managed. Other
goals of this study were to measure the perceived value of organizational IS portfolios as
well as perceptions of portfolio level IS risk. Finally perceptions of IS effectiveness from
a managerial perspective were measured.

Constructs

The management practices examined in the study were sourced from the COBIT
framework, for two primary reasons. The first is the use by auditors of the COBIT
framework as a tool to properly control the IS function. These controls are meant to
ensure the smooth operation of the IS firnction as well as to reduce risks within the IS

function and the organization. The next reason for using the COBIT framework is that it

72

mirrors much of the IS literature in its attempt to provide empirical evidence for proper
management techniques for the IS function. The use of COBIT thus provides a
comprehensive framework that incorporates the vast majority of managerial controls
necessary for the proper functioning of the IS portfolio. With regard to the use of the
COBIT fiamework in the research model, the results will allow researchers to understand
how accounting control measures for an IS function work to manage an IS portfolio.
Furthermore the COBIT managerial controls, validated as questionnaire items through
confirmatory factor analysis, should be considered for use in future portfolio level 18
studies.

The other constructs introduced in this study includes portfolio level concepts of
measuring risk and value. Risk and value represent perceptions of the effects of the IS
portfolio at portfolio level. Again by validating these constructs, researchers can now
consider using these in future research to further understand IS portfolios.

Results

Finally the results of the model determined whether management practices (i.e.
COBIT framework), mediated through value and risk, did indeed explain variance in the
dependent variable IS effectiveness. Results suggest that: 1) the portion of the COBIT
framework examined in this study provides significant predictors of IS value and IS risk;
2) IS value significantly impacts the dependent variable IS effectiveness; and 3) the risk
variables did not have a significant impact upon the dependent variable IS effectiveness.

The following will give an overview of the contributions to theory.

73

Contingency Theory

The IS portfolio is an organizational level element that must reflect the objectives
of the organization. Proper alignment of the IS function to the business strategy is
essential to the organization in its attempt to achieving its goals. Nevertheless, IS
portfolios within organizations need not be exactly the same across organizations to
achieve organizational objectives. Consistent with contingency theory are the
assumptions that 1) there is no universal best way to manage; 2) effective organizations
must have a proper fit with the environment and also between its subsystems; and 3) the
needs of an organization are better satisfied when the organization is properly designed
and the management style is appropriate for the tasks undertaken. The qualitative
research results indicate that among the three organizations surveyed, each organization
had a vastly different configuration for its IS portfolio, in their attempt to align the
portfolio with the business. For the three organizations surveyed, two organizations use a
centralized configuration for its portfolio. One organization uses a portfolio office while
the other organization uses a program office. The third organization is totally
decentralized in its effort to provide IS services at the business unit level while at the
corporate level large projects are tracked. Next the final contingency theory assumption
that an organization and its subsystems must fit with their environment, is demonstrated
by the following quote from a healthcare provider’s effort to adhere to the external
regulatory environment created by the I-IIPAA law:
“The HIPAA Law is quite clear on protecting a patient ’s privacy as it relates to health
information. Our IT systems go to great lengths to require user authentication to access

patient data; to track what user is viewing patient data, when they are viewing patient

74

data, and where they are viewing patient data; and to keep date-stamped logs of access
to patient data. We review these logs to ensure that only authorized access has
occurred. ”
IS portfolios —- regardless of how they are configured (i.e. centralized/decentralized) to
serve the business -- can have a positive affect on the business they support. Furthermore
organizations continually examine how their IS functions fit with the external
environment (i.e. in its attempt to incorporate HIPAA). Thus the qualitative results show
how contingency theory would apply to IS portfolio management. The following will
give an overview of how portfolio theory fits the study.
Portfolio Theory

Consistent with Benko and McFarlan (2003), the qualitative results indicate
organizations attempt to view projects as different assets within the project portfolio with
distinct characteristics, goals and asset allocations (i.e. investments according to
organizational intentions). All three organizations require a business case to justify its
inclusion as a new project. Evaluation criteria include costs, human resources in terms of
man-hours, as well as the impact upon current and future operations. The following
quote gives an example of what the organizations are attempting to achieve when
examining a new project.
“A business case is required for all projects. Projects are evaluated on a Profit Index
and Strategic Alignment to company goals. "
The aforementioned quote uses a profit index which is an organizational rubric to
determine if the cost of a project meets an acceptable internal rate of return which is

simply determined if a project does not cost too much to implement. Furthermore an

75

analysis of strategic alignment is conducted for any proposed projects to determine if the
projects meet the goals and objectives of the organization. Thus, implicitly,
organizations engage in managing their 18 portfolios in the same manner as a financial
portfolio manager would make decisions for a financial portfolio or mutual fund. The
following will give an overview of the theoretical implications of absorptive capacity in
the study.
Absorptive Capacity

Absorptive capacity among IS personnel and the business personnel they support
is demonstrated through the Detroit Three automakers’ use of business cases to justify
new projects, as well as quality gates throughout the duration of the project in which both
business and IS participate. Another example of absorptive capacity in the organization
is the use of the Project Conception Brief (PCB) by the confectionary company created
by the project stakeholder and reviewed by the IS portfolio office for feasibility. These
aforementioned examples demonstrate how those participating in the IS function work
with the business to share knowledge-regarding potential and ongoing projects. The
following will give the theoretical implications based upon the results of the research
model.
Research Model

From a research standpoint, the model employed in this study reveals the
following: First, the portion of the COBIT framework utilised has been validated through
the use of confirmatory factor analysis. The questionnaire items for the COBIT
constructs can be used for ongoing research at the organizational level. Although this

represents the first study to attempt to incorporate a portion of the COBIT framework as

76

constructs, researchers should attempt to replicate these initial findings to ensure they are
accurate. Next, the use of portfolio level constructs for value and risk have also been
validated. The use of the value constructs, responsiveness and flexibility has been
validated through confirmatory factor analysis. Furthermore, risk items in terms of users
and budgetary concerns have also been validated. Again, since this is the first study to
incorporate the use of such constructs, researchers should attempt to replicate the findings
of this study to ensure these constructs are indeed representative of the phenomena they
intend to model. Also, the significant result for hypothesis 3 (the impact of value on IS
effectiveness) shows how portfolio level constructs can explain variance in the dependent
variable IS effectiveness. Furthermore, the test for mediation shows the flexibility
construct to fully mediate the link between the independent second order factor,
management practices, and the dependent variable, IS effectiveness. Researchers should
also note that perceptions of risk do not have an impact on or explain variance in the
dependent variable IS effectiveness. This might be due to managers’ acknowledging
risk; however, risk in itself does not have an impact on effectiveness or the ability of
organizations to effectively integrate systems for the purpose of supporting an
organization in its strategic objectives and goals. However, in examining the split
sample, the findings indicate that auditors (i.e. professionals specifically trained to
examine risk) do believe that the presence of risk can negatively impact effectiveness,
whereas IS professionals (i.e. professionals who are not specifically trained to
incorporate risk into their jobs) do not see risk as a significant concern in the IS portfolio.
This is significant, since risk in general in organizations is an increasingly important issue

that executives are committing resources to and examining. However, the findings

77

suggest that IS professionals do not see risk as an issue. This might be problematic since
monitoring and control within organizations must increasingly be automated due to the
velocity of transactions and the complexity of processes. If IS professionals are not
attuned to risk, adequate monitoring might not take place. This is because 18
professionals are in control of the day-to-day operations of the IS portfolio, while
auditors only examine operations on periodic bases. Since auditors might not examine
processes for months or possibly years it is critical that IS professionals can identify and
monitor risk.
Use of Second Order Factors

Researchers often attempt to measure the impact of constructs individually to gain
a clear understanding of particular phenomena. Measuring the impact of constructs in
isolation in most cases is preferred; however this does not necessarily reflect how
interconnected many business operations are. Recently more researchers have begun
incorporating second order factors into their models to more accurately reflect the
intricate nature of the phenomena they intend to study and measure. This study has
attempted to model the IS portfolio phenomena, realizing that many different actors are
involved in these processes. Furthermore the nature of the managerial practices utilised
implies that they all must be used together to facilitate the operations of the IS portfolio.
Thus the use of second order factors for both the exogeneous independent variable
management practices, and the dependent variable, IS effectiveness is appropriate and

necessary.

78

6.3 Managerial Implications

Since IS portfolio management is a new concept to many IS practitioners, this
study serves as a basis to understand how managers think about their functions
holistically. The following will give a brief summary of each major area of inquiry
regarding what the results of this study might mean to IS practitioners.
Managerial Practices/Controls

The results of this research give IS managers a better understanding of which
management practices serve as significant controls for the day-to-day operation of an
organization’s IS function. Although many other controls exist for the management of the
IS function, managers should not ignore the findings of this study. The use of the COBIT
framework has been found to be a useful tool for IS managers. With regard to each
practice investigated, findings suggest the following: beginning with the Plan and
Organize portion of the framework, strategic planning for the IS portfolio in terms of how
the business is supported is critical. Given the results of the interviews as well as
surveys, the findings suggest that it is important for both IS personnel and business
personnel to be involved in the process. The findings also suggest that planning must be
a continuous, ongoing process. From an IS investment standpoint, a financial framework
should be in place for managing IS assets as well as project budgets. With regard to
human resources, the results suggest that organizations would be well served by paying
close attention to IS specific human resources in terms of their roles and how their
services are employed most efficiently to support the IS portfolio. The survey results for
the final Plan & Organize management control, assessing and managing IS risks, suggests

that organizations must engage in a risk assessment process that is iterative and includes

79

input from personnel working to support the IS portfolio as well as the business personnel
they support. Finally, findings from the monitor and evaluate performance management
practice (nOte this practice is from the Monitor & Evaluate domain of the COBIT
framework) suggests that the objectives for the IS portfolio must be clearly defined and
measured against specific performance criteria, and assessment of performance is
ongoing and periodic. Overall findings for the managerial practices suggest
organizations should attempt to employ these specific practices in their IS portfolios to
support an efficiently running IS function. The following will examine perceptions of the
risk/return tradeoff surrounding the IS portfolio.

Risk/ Value T radeofl

The value proposition for portfolio level information systems is that portfolio level IS can
have a positive impact upon business operations. Value in terms of how IS can both
anticipate and react to change brought forth by the business can be viewed as an asset to
the organization. Change within any organization is inevitable. The ability of the IS
function to absorb change, whether generated internally or externally can, in part
determine how well an organization can meet its goals and objectives. Therefore a highly
valuable IS portfolio can be viewed as an asset to the organization. Valuable IS portfolios
will have the ability to offset portfolio level risks in terms of how users make use of the
system as well as offset any possible negative outcomes in terms of IS portfolio budgets.
Thus when organizational IS portfolios absorb change they can help mitigate IS portfolio

level risk and positively impact the organization.

80

E flectiveness

Results indicate that managers should be concerned with the successful Operation
of the IS portfolio from both an operational and managerial perspective. Enacted
managerial processes pave the way for organizations to possibly mitigate risk as well as
add value to the portfolio. The risk/value tradeoff allows for IS to positively contribute to
organizational goals and objectives. Thus managers should strive to ensure the
precursors Of a successful IS portfolio are in place so as to have the possibility of
enhancing the organization. Today’s organizations can only excel in their industries with
a heavy reliance on automated processes. Therefore IS effectiveness should be a goal for
any organization that is attempting to excel in their organizational missions.
Implications for IS Auditors

Information systems auditors are attuned to identifying and mitigating 18 specific
risk within organizations. Auditors serve as an obj ective resource tO management in
helping them understand how their managerial decisions, as they relate tO the IS
portfolio, might positively or negatively impact the organization. Results suggest that IS
auditors indeed understand the impact Of risk upon the effectiveness of the IS portfolio.
Therefore IS managers should maintain and possibly increase IS auditor participation at
all phases of systems development, maintenance and ongoing Operations. This
involvement should positively contribute tO the IS portfolio such that management
practices increase controls surrounding the IS portfolio. An increase in controls should

lead to better contributions of the IS portfolio to an organizations’ goals and Obj ectives.

81

Implications for IS professionals

The results of this study suggest information systems managers understand the
use Of managerial controls surrounding the IS portfolio and how the effective use of these
managerial practices positively impact the IS function. These managers also have a clear
understanding Of how an effective IS portfolio can positively contribute to an
organizations’ success. Thus IS managers and IS auditors are nO different in their views
of how IS contributes tO the organization. Nevertheless, IS managers and IS auditors
differ in their attitudes on risk. IS managers are not specifically trained tO identify risk
and to have an understanding Of how those risks might negatively impact the
organization. The results suggest that IS managers do believe risk assessment and on-
going risk management are both important. However they do not believe that the
presence Of risk plays any part in the effectiveness of an IS portfolio. This View Of risk
maintained by IS managers can negatively affect ongoing Operations of an organization.
Risk assessment and planning for risk mitigation should be an increasingly prominent
portion Of the IS function. Furthermore IS managers should be trained on how important
risk assessment and planning are tO mitigating those risks.
Implications for Stakeholders

Executive officers as well as board members Of organizations can use these results
to understand how their management views the importance Of the IS portfolio. They can
also use these results to identify training opportunities in their organizations surrounding
risk as well as shaping the internal control structure of their organizations. PrOper

implementation and use Of management practices surrounding the IS portfolio should

82

create a basis for the success Of the strategic agenda that is set forth by an organizations’
stakeholders.
6.4 Recommendations

Managers should use this research as a basis to ensure their IS portfolios are
supported by the right mix of management practices that serve as managerial controls.
These controls should be devised and implemented by both IS managers as well as
business managers with the help Of the organization’s audit staff. The involvement of all
essential parties to the process should ensure that value is derived from IS and risk is
mitigated. Special emphasis should be given to increasing IS managers’ awareness as to
how risk plays an integral part in whether an organization can achieve its goals and
Obj ectives. The result should be that the IS portfolio positively contributes to the success
Of the organization in achieving its goals and Objectives.
6.5 Limitations

This research is limited in a number of ways. The survey is cross sectional in
nature, thus introducing the possibility Of ambiguity in causal direction for the constructs.
Additionally, only one informant per firm is surveyed. Therefore, common methods
variance is a possibility. TO test this, Harmon’s one factor test was performed to
determine if common methods variance was a problem. Harmon’s one factor test is
completed by comparing measurement model results with a one factor model tO
determine the presence of any methods bias (Sanchez 1995). According to Sanchez et a1,
if the one factor model demonstrates a worse model fit than the measurement model, then
common methods variance is not a problem. Results Of the measurement model indicate

a x2 Of 984.89 with 707 degrees Of freedom and the one factor model results in a x 2 Of

83

2518.17 with 859 degrees Of freedom. Since the one factor model exhibits worse model
fit than the original measurement model, then the use of a single instrument with a single
respondent suggests that common methods bias does not exist in this data. Nevertheless
responses from only one individual might limit the variety Of opinions that might be
gained if more organizational actors had been surveyed. A single respondent might not
be truly representative Of the perceptions at the firm. Furthermore the limited sample size
of only 101 firms may also not be applicable to a wide variety of industries or differing
organizational structures that undoubtedly exist across organizations. Also, using two
types of respondents (i.e. IS auditors and IS professionals) may have provided different
results from a uniform sample Of respondents that were of similar professional
orientation. Finally, the limited number Of independent variables used (i.e. 5 in the
second order factor for management practices) may not be truly representative of the
myriad Of controls and management practices that are employed in managing an
organization’s entire IS portfolio. Nevertheless, the results of this study provide
researchers with a basis for portfolio management research going forward. Furthermore
this study validates the use of the COBIT framework for future use and introduces
portfolio level constructs for value and risk, as well as explaining a high level Of variance

in the dependent variable IS effectiveness.

84

6.6 Future Research

Future research should examine other areas Of the COBIT framework not
investigated by this study. The other areas Of the Plan and Organize portion of the
COBIT framework that should be investigated are as follows: 1) define the information
architecture; 2) determine the technical direction; 3) define IT processes, organization
and relationships; 4) communicate management aims and direction; 5) manage quality;
and manage projects. Each Of these high level control Obj ectives can provide researchers
with a basis to begin a research stream that has the possibility of bridging the gap
between accountants and IS professionals. Outside Of the Plan and Organize portion of
the COBIT framework, other areas such as Acquire and Implement, Deliver and Support
and Monitor and Evaluate should be investigated. Examining these other areas of the
framework in organizations may determine whether these controls if enacted, do indeed
control the IS function in organizations. The controls should in turn support the IS
function in its attempt to aid the organization in achieving its goals and Obj ectives.

Outside Of the COBIT area, other firture research should examine how
information systems at the portfolio level impact organizational performance outcomes
such as financial and/or Operational measures. For example, hypothesized relationships
between 18 management practices and financial measures such as net income, return on
investment or return on assets should be examined. Furthermore, portfolio level
constructs should be modeled with Operational measures Of success that are industry
specific to determine whether the constructs explain variance in Operational measures of

success .

85

Other research should investigate psychological phenomena at the organizational
level such as climates (Schneider 1975) for information systems (Boynton et a1. 1994).
Furthermore, organizational orientations for or against the IS department should also be
investigated to determine the successfulness of the IS function. Success can be measured
by effectiveness or impact upon financial outcomes. A positive orientation toward IS by
business managers and other users should positively impact the effectiveness of the IS
function; however, this expectation should be confirmed through research. Another area
of inquiry is the inherent value (Rokeach 1979) that managers place in the IS function,
both in terms of line managers and their counterparts, the IS managers. Those
organizations that have managers who positively value IS should excel in the
effectiveness of their IS functions. Differences or the lack thereof between IS
management and business management should be reflected in the climates for IS and
organizational values. These psychological behaviors should have an impact either
positively or negatively upon how IS portfolios are managed and contribute to IS
effectiveness.

In conclusion, the purpose of this research was to re-engage academic literature
with the portfolio management phenomenon first espoused by McFarlin. The COBIT
framework has been tested to give researchers a foundation for future investigation into
management controls for IS. Furthermore, organizational/portfolio level constructs have
been created and confirmed through factor analysis to give researchers additional tools to
measure perceptions of the risk/return tradeoff at the portfolio level of investigation.
Finally, the research model introduced gives valuable insight into variance explained in

the IS effectiveness construct at the portfolio level.

86

Appendix A
Open Ended Interview Questions

Organizational Overview .
1. Please describe the overall organizational structure and how the IT dept supports the
organization

2. Describe any major differences in the implementation and management of systems before and
after the implementation of portfolio management tools.

3. How is the program ofi'rce structured in terms of reporting relationships, project support etc
New Projects
1. Is there a business case required for all projects? Is there visibility for small projects,

threshold for what constitutes a large project.

2. For new projects, is there a baseline assessment completed prior to systems implementation to
determine what is success?

Planning
1. To what degree is IT risk assessment tied to organizational risk assessment?

2. To what degree is IT strategic planning and organizational strategic planning linked?
Project Monitoring
1. What metrics are in place to evaluate IT — overall portfolio of systems & by project (both
quantitative & qualitative metrics if any)?

2. Is continuous monitoring in place for current applications in terms of business alignment,
value, risk & cost/benefit? If so what metrics are measured?

3. How are projects prioritized?

4. To what degree are projects tracked centrally (i.e. in the program office/portfolio management
software)? Is there a dollar or resource threshold or are all projects tracked?

Staff Competencies
1. To what degree does IT staff/management have working knowledge of financial concepts (i.e. to
apply metrics/ analyze data) that may be used in IT program/portfolio management?

2. To what degree are resources leveraged across multiple projects/program office, (how often do
roles change for staff -— how do you minimize the effect of IT employee turnover, how does the
use of contractors/consultants play into resource allocation/utilization)?

3. What type of training is in place for project managers (both new and experienced)?

Regulatory/Standards
1. What practices within IT have been put in place to comply with HIPPA and other regulatory
mandates?

2. Are any widely accepted frameworks in use in the IS dept i.e. control objectives for IT (COBIT)
or information technology infrastructure library (ITIL)?

87

Appendix B
Answers to Open Ended Interview Questions

Detroit Three Automaker Interview

1. How is the program office structured in terms of reporting relationships, project
support etc

DaimlerChrysler Corporation (Chrysler Group) does not currently have a program office.
The Leadership Team is discussing this concept and potential responsibilities.

2. Is there a business case required for all projects? Is there visibility for small projects,
threshold for what constitutes a large project

A business case is required for all projects. Projects are evaluated on Profit Index and
Strategic Alignment to company goals. Projects over $200K are considered “large”
projects.

3. What metrics are in place to evaluate IT — overall portfolio of systems & by project
(both quantitative & qualitative metrics if any)?

Profit Index, Strategic Alignment are Cost are metrics by which the company evaluates
its portfolio on an annual basis.

4. Is continuous monitoring in place for current applications in terms of business
alignment, value, risk & cost/benefit? If so what metrics are measured?

Once a project is approved, there are four quality gates the project passes through prior to
completion. The Strategic Alignment, Profit Index, etc. are evaluated/re-evaluated on an
annual basis.

5. For new projects, is there a baseline assessment completed prior to systems
implementation to determine what is success?

The four quality gates the project must pass measure implementation success. The
project is also subject to a Strategic Assessment Survey where the customer evaluates
system performance against expectations.

V 6. To what degree are resources leveraged across multiple proj ects/program office, (how
often do roles change for staff — how do you minimize the effect of IT employee
ttu'nover, how does the use of contractors/consultants play into resource
allocation/utilization)?

Resources may have responsibility for more than one project at a time but the shared
resource resides within a Functional group. The resource for example would not be

88

working on a Manufacturing project at the same time as a Product Development project.
Same holds true for our contract/ supplemental resources.

7. How are projects prioritized?
Projects are prioritized/reprioritized on an annual basis to review Profit Index and

Strategic Alignment. This exercise is conducted in a company wide Cross-Functional
senior level forum.

8. What type of training is in place for project managers (both new and experienced)?

Several levels of training are available for our Project Managers. . .starts with basic
Project Management tools and progresses to advanced Project Management
methodologies. Certification is supported with outside organizations.

9. Do you use a common set of project management tools/software as well as program or
IS portfolio management software?

Yes. Standard/common tools are used across all of IT for Portfolio and Project
Management. -

10. To what degree does IT staff/management have working knowledge of financial
concepts (i.e. to apply metrics/analyze data) that may be used in IT program/portfolio
management?

Project Managers have the lead to gather the F inancal data for evaluation by our IT
Finance organization. The data reflects lifetime costs of the program/system.

11. To what degree is IT risk assessment tied to organizational risk assessment?

Projects are assigned a “risk” factor. This is not tied to organization risk other than to
business goal/performance and Profit Index.

12. To what degree is IT strategic planning and organizational strategic planning linked?
IT Strategy & Planning has two facets — IT strategy and Corporate Strategy/Business
Strategy. Strategic planning for the organization considers not only the IT strategy but

how the organization supports the business.

13. To what degree are projects tracked centrally (i.e. in the program office)? Is there a
dollar or resource threshold or are all projects tracked?

A standard tool is used to track projects however, the Functional IT groups are

responsible for reporting. The Emotional Groups are provided with visibility to the
entire project portfolio on a monthly basis.

89

15. What practices within IS have been put in place to comply with Sarbanes Oxley?

All means have been taken to assure compliance. Automated and manual processes are
used. Compliance has added significant workload and cost to projects affected.

90

Confectionary Interview

1. How is the program office structured in terms of reporting relationships, project
support etc

Portfolio mgmt office not embedded in the global program office. The Portfolio mgmt
office assess all projects in terms of risk, integration and finance. Enterprise architecture
is in a separate function staffed by 2 resources. There is currently 1 researcher
responsible for finding best practices and examining industry norms. A data architect will
be added soon.

2.15 there a business case required for all projects? Is there visibility for small projects,
threshold for what constitutes a large project

Called a Project Initiation Brief (PIB) - asks user to provide information on SCOpe,
business requirements (business solutions group — BSG analysts — functional area
experts), provides a threshold determined by effort required (combination of hours &
dollars) to complete the project

3. What metrics are in place to evaluate IT — overall portfolio of systems & by project
(both quantitative & qualitative metrics if any)?

Currently project level metrics only, metrics are in place for pipeline projects and project
delivery, going to SAP portfolio mgmt during 2008

4. Is continuous monitoring in place for current applications in terms of business
alignment, value, risk & cost/benefit? If so what metrics are measured?

Value, risks, costs and benefits monitored and assigned score for each project, - these are
incorporated into PB

5. For new projects, is there a baseline assessment completed prior to systems
implementation to determine what is success?

See Q#2 PIE

6. To what degree are resources leveraged across multiple proj ects/program office, (how
often do roles change for staff — how do you minimize the effect of IT employee
turnover, how does the use of contractors/consultants play into resource
allocation/utilization)?

IT aligned with business functions, near future initiative to cross train resources on

different applications/programming languages, turnover is very low, employees are
encouraged to come up with their own development goals, contractors do backfillin g only

91

or to fill a skill gap, (contractors must bid on-line for Wrigley work), resources are
managed globally throughout Wrigley and assigned by global resource managers

7. How are projects prioritized?

Projects are assigned a % in each category (transform the business, grow the business,
run the business (i.e. ongoing/maintenance) strategy and capital requirements are taken
into consideration

The strategy takes into consideration the following corporate priorities such as reduce
operating expenses, drive revenue growth, CRM, improve efficiency, reduce complexity
and the removal of old technology

Enhancements are scored the same way as new projects
8. What type of training is in place for project managers (both new and experienced)?
Initial project manager academy to take place 1St Qtr ‘07

9. Do you use a common set of project management tools/software as well as program
or IS portfolio management software?

Yes, inherent in portfolio mgmt office, no software yet (i.e. SAP in ’08) the pipeline is
managed using HP ticket mgmt system (“Service Manager”)

10. To what degree does IT staff/management have working knowledge of financial
concepts (i.e. to apply metrics/analyze data) that may be used in IT program/portfolio
management?

IS staff all have familiarity with utilizing the business case (PIB). It involves pricing
issues including budgeting. There is also a finance function with the IT function

1 1. To what degree is IT risk assessment tied to organizational risk assessment?

At the implementation stage the risk assessment is tracked monthly, but not sure if its tied
to organizational risk assessment

12. To what degree is IT strategic planning and organizational strategic planning linked?

Multiple business strategies, encapsulated in Project Conception Brief (PCB), it serves as
a strategy and planning document. Requirements gathering information is taken from the
PCB and translated into a gap analysis, the completed gap analysis is sent back to the

business unit that issued the PCB to obtain business agreement on alignment issues

13. To what degree are projects tracked centrally (i.e. in the program office)? Is there a
dollar or resource threshold or are all projects tracked?

92

All tracked in the PMO office (tracked in HP service manager), threshold is 160 hrs for
small projects

14. What practices within IS have been put in place to comply with Sarbanes Oxley?

SOX audit compliance results feed into IS program/proj ects

93

Healthcare Provider

1) Is there a program office in place or a centralized project office structured in
terms of reporting relationships, project support etc

Yes — However, within the IT Department, we have designated “sections” (e. g.,
Clinical Systems, Business Systems, Communications, Support, etc.). Within these
sections, reporting relationships have been established, and the relationships among
sections have been established.

Depending on the project, several sections will be involved (e.g., A new clinical
system requires infrastructure — i.e. Communications, Support, Servers, etc. —
and the project “owner” then coordinates efforts across sections.)

2) Is there a business case required for all projects?

Yes - All IT projects are prioritized based on need and available resources. We
have a committee that has membership from IT Management and Corporate
Management that sets the priority of each project.

3) Is there visibility for small projects, threshold for what constitutes a large project?

Unfortunately, there are more requests for IT resources than IT has; therefore,
“smaller” projects sometimes are not as ‘Visible” as “larger” projects. There isn’t a
solid metric for determining what is a large vs. a small project: a project is loosely
defined as large or small based on scope, need, length, resources required, etc.

After the committee prioritizes our projects, the ones that can be accomplished or
begun within a given year are our focus. They are often a mix of large and small
projects, but once they are integrated with the carryover projects from the prior year,
we determine how many of the new projects can be started/completed. When our IT
resources are comrrritted or depleted (e. g., project # 52), any additional projects (i.e.
project #53, #54, etc.) are bumped to the next year for reprioritizing with new
projects.

4) What metrics are in place to evaluate IT — overall portfolio of systems & by
project (both quantitative & qualitative metrics if any)?

We participate in a national benchmarking service to evaluate our operations, and we
have done very well in nearly every category. I am not at liberty to elaborate on this
any further.

5) Is continuous monitoring in place for current applications in terms of business
alignment, value, risk & cost/benefit? If so what metrics are measured?

94

Our Corporate Management Team continuously evaluates our entire operations (i.e.,
IT, clinical, financial, etc.) in these terms and other terms, too. However, I am not at
liberty to elaborate further.

6) For new projects, is there a baseline assessment completed prior to systems
implementation to determine what is success?

I am not aware of any formal baseline assessment that is done to establish
“success measures.”

7) To what degree are resources leveraged across multiple proj ects/program office,
(how often do roles change for staff — how do you minimize the effect of IT
employee turnover, how does the use of contractors/consultants play into resource
allocation/utilization)?

Generally, we do not use contract /consultants for IT projects, however, we
occasionally contract some (limited) PC and server support when needed. When
a project involves a software/hardware vendor, we often require their help
implementing the project. This is a “side-door” use of contract help on projects,
but is necessary if the project is a new software/hardware platform. We need the
vendor’s help to decrease the time-to-implementation cycle.

8) How are projects prioritized?

All IT projects are prioritized based on need and available resources. We have a
committee that has membership from IT Management and Corporate
Management that sets the priority of each project.

9) What type of training is in place for project managers (both new and
experienced)?

We have formal and informal training for our project managers. The formal training
is classroom based, on-line/CD based, and/or mentor based. The informal training is
experience based.

10) Do you use a common set of proj ect management tools/ software as well as
program or IS portfolio management software?

No — However, we are just beginning to put together a recommendation for IT
Management to consider, but it so preliminary at this point that I can’t comment on it
(the truth is: I don’t know what that “group” is going to recommend).

11) To what degree does IT staff/management have working knowledge of financial

concepts (i.e. to apply metrics/analyze data) that may be used in IT
program/portfolio management?

95

Several years ago, IT Management recognized the need for a specialized group of
individuals -— Management Engineering types (we call them BOAs: Business
Operation Analysts) that have working knowledge in multiple disciplines:
Management Engineering, Financial, or Clinical and IT Operations. These specialists
assist our project managers, wherever their combination of skills are needed.

12) To what degree is IT risk assessment tied to organizational risk assessment?

Our business operations — clinical and financial — are heavily, if not entirely, IT-
based. Therefore, IT has taken the lead in assessing the operational risks for the entire
organization: we call it Business Continuity. Not only are we concerned about
Business Continuity from the perspective of the viability of our “business,” but
because we are a healthcare provider, external organizations — governmental,
reimbursement, etc. — are concerned about our Business Continuity programs, too.

13) To what degree is IT strategic planning and organizational strategic planning
linked?

Like a “hand-and-glove.” See #11.

14) To what degree are projects tracked centrally (i.e. in the program office)?

We do not have a “program office,” however, all IT projects are coordinated
through the Chief Information Officer’s office a his leadership committee.
15) Is there a dollar or resource threshold or are all projects tracked?

All projects are tracked, regardless of their “size,” because they ultimately impact the
budgeted dollars for the IT Department.

16) What practices within IS have been put in place to comply with HIPAA etc?

Time and space do not permit me to list everything we are doing to comply with
HIPAA, nor can I reveal all the security measures we have in place, so I’ll just give
you a rather general statement:

WWW.I-IIPAA.ORG and other web sites help organizations to understand HIPAA
and our obligations/requirements as it relates to the letter and spirit of the Law. Our
organization is committed to as compliant as we know how to be. I have phrased it
this way because some of the HIPAA regulations are vague and are subject to
interpretation. While these areas are being clarified, we do our best to comply. When
a definitive clarification is available, we do our best to comply.

96

For example: The HIPAA Law is quite clear on protecting a patient ’s privacy as it
relates to health information. Our IT systems go to great lengths to require user
authentication to access patient data; to track what user is viewing patient data, when
they are viewing patient data, and where they are viewing patient data; and to keep date-
stamped logs of access to patient data. We review these logs to ensure that only

authorized access has occurred.

97

BIBLIOGRAPHY

Bardhan, I., Bagchi, S., and Sougstad, R. "Prioritizing a portfolio of information
technology investment proj ects," Journal of Management Information Systems
(21:2) 2004, pp 33 - 60.

Barki, H., Rivard, S., Talbot, J. "An integrative contingency model of software project
risk management," Journal of Management Information Systems (17:4) 2001, pp
37 - 69.

Baron, R.M., Kenny, D.A. "The moderator-mediator variable distinction in social
psychological research: Conceptual, strategic and statistical considerations,"
Journal of Personality and Social Psychology (51 :6) 1986, p 9.

Baroudi, J .J . "The impact of role variables on IS personnel work attitudes and intentions,‘
MIS Quarterly (9:4) 1985, pp 341 - 356.

Batiste, J .L. "The application profile," MIS Quarterly (10:3) 1986, pp 207 - 213.

Belcher, W.L., and Watson, H.J. "Assessing the value of Conoco‘s EIS," MIS Quarterly
(17:3) 1993, pp 239 - 253.

Benko, C., and McFarlan, F .W. Connecting the Dots: Aligning Projects with Objectives
in Unpredictable Times Harvard Business School Press, Boston, Massachusetts,
2003.

Bollen, K.A. Structural Equations with Latent Variables John Wiley & Sons Inc., New
York, 1989.

Bonham, S.S. IT Project Portfolio Management Artech House, Norwood, MA, 2005.

Boynton, A., Zmud, R., and Jacobs, G. "The influence of IT management practice on IT
use in large organizations," MIS Quarterly (18:3) 1994, pp 299 - 318.

Broadbent, M., and Weill, P. "Management by maximzHow business and IT managers
can create IT infrastructures," Sloan Management Review (38:3) 1997, pp 77-92.

Browne, M.W., Cudeck, R. "Alternative ways of assessing model fit," in: Testing
Structural Equation Models, K.A. Bollen, Long, J .8. (ed.), Sage, Newbury Park,
CA., 1993, pp. 136 - 162.

Burke, J .C., Shaw, M.J. "Implications of complexity theory on real option analysis in IT

portfolio management," in: Americas Conference on Information Systems,
Keystone, Colorado, 2007.

98

Byrd, T.A., and Turner, D.E. "Measuring the flexibility of information technology
infrastructure: Exploratory analysis of a construct," Journal of Management
Information Systems (17:1) 2000, pp 167 - 208.

Cameron, B.H. "IT portfolio management: Implications for IT strategic alignment,"
Americas Conference on Information Systems, Omaha, NE, 2005, pp. 398 - 405.

Cash, J .I., McFarlan, F., and McKenney, J.L. Corporate Information Systems
Management Irwin, Homewood, IL, 1988.

Chan, Y.E., Huff, S.L., Barclay, D.W., and Copeland, D.G. "Business strategic
orientation, information systems strategic orientation and strategic alignment,"
Information Systems Research (8:2) 1997, pp 125 - 150.

Chin, W.W. "The partial least squares approach to structural equation modeling," in:
Modern Methods for Business Research, G.A. Marcoulides (ed.), Lawrence
Erlbaum, Mahway, NJ, 1998, pp. pp. 295 - 336.

Chin, W.W., Marcolin, B, Newsted, P.R. "A partial least square latent variable modeling
approach for measuring interaction effects: Results from a monte carlo simulation
study and an electonic-mail emotion/adoption study," Information Systems
Research (14:2) 2003, pp 189 - 217.

Cohen, J ., Cohen, R, West, S.G., and Aiken, L.S. Applied Multiple
Regression/Correlation Analysis for the Behavioral Sciences Lawrence Erlbaum
Associates, Mahwah, NJ 2003.

Datz, T. "Portofolio Management: How to Do it Right," in: CIO Magazine, 2003.

De Leeuw, A.C.J., and Volberda, H.W. "On the concept of flexibility:A dual control
perspective," Omega (24:2) 1996, pp 121 - 139.

Delone, W., and McLean, E. "Information systems success: The quest for the dependent
variable," Information Systems Research (3:1) 1992, pp 60 - 95.

Duncan, N.B. "Capturing flexibility of information technology infrastructure: A study of
resource characteristics and their measure," Journal of Management Information
Systems (12:2) 1995, pp 37 - 57.

Flamholtz, E.G., Das, T.K., and Tsui, A.S. "Toward an integrative framework of
organizational control," Accounting, Organizations and Society (10:1) 1985, pp
35 - 50.

Fomell, C., and Larcker, D. "Evaluating structural equation models with unobservable

variables and measurement error," Journal of Marketing Research (18:3) 1981, pp
pp. 39 - 50.

99

Gefen, D., Straub, D., and Boudreau, M. "Structural equation modeling and regression:
Guidelines for research practice," Communications of the Association for
Information Systems (4) 2000, pp 1 - 77.

Gomolski, B., and Smith, M. "Gartner 2006-2007 IT spending and staffing report: North
America," Gartner, p. 77.

Henderson, J .C., and Sifonis, J.G. "The value of strategic IS planning: Understanding
consistency, validity and IS markets," MIS Quarterly (12:2) 1988, pp 187 - 200.

Hu, L., Bentler, P.M. "Cutoff Criteria for fit indexes in covariance structure analysis:
Conventional criteria versus new alternatives," Structural Equation Modeling
(6:1) 1999, pp 1 - 55.

Igbaria, M., and Siegel, S.R. "The reasons for turnover of information systems
personnel," Information & Management (23) 1992, pp 321 - 330.

Jeffrey, M., and Leliveld, 1. "Best Practices in IT Portfolio Management," MIT Sloan
Management Review (45:3) 2004, pp 41 - 49.

Jiang, J.I., Klein, G., and Discenza, R. "Information Systems Success as impacted by
risks and development strategies," IEEE Transactions on engineering
management (48:1) 2001, pp 46 - 55.

Kaplan, D. Structural Equation Modeling Foundations and Extensions Sage Publications,
Thousand Oaks, CA, 2000.

Karhade, P. "IT portfolio management: An enterprise risk management based
perspective," in: Americas Conference on Information Systems, Keystone,

Colorado, 2007.

Keil, M., Cule, P.E., Lyytinen, K., and Schmidt, R.C. "A framework for identifying
software project risks," Communications of the ACM (41:11) 1998, pp 76 - 83.

Keil, M., Tan, B.C.Y.,Wei,K.,Saarinen,T. "A cross-cultural study on escalation of
commitment behavior in software projects," MIS Quarterly (24:2) 2000, p 27.

Kenny, D.A., Judd, C.M. "Estimating the nonlinear and interactive effects of latent
variables," Psychological Bulletin (96) 1984, pp 201 - 210.

Langfield-Smith, K. "Management control systems and strategy: A critical review,"
Accounting, Organizations and Society (22:2) 1997, pp 207 - 232.

Lederer, AL, and Sethi, V. "The implementation of strategic information systems
planning methodologies," MIS Quarterly (12:3) 1988, pp 445 - 461.

100

Lederer, AL, and Sethi, V. "Key perscriptions for strategic information systems
planning," Journal of Management Information Systems (13: 1) 1996, pp 35 - 62.

Mahmood, M.A., and Mann, G.J. "Measuring the organizational impact of information
technology investment: An exploratory study," Journal of Management
Information Systems ( 10:1) 1993, pp 97 - 123.

Maizlish, B., and Handler, R. IT Portfolio Management Step-By—Step: Unlocking the
Business Value of Technology John Wiley & Sons, Hoboken, New Jersey, 2005.

Markowitz, H. "Portfolio selection," The Journal of Finance (7:1) 1952, pp 77 - 91.

Markowitz, HM. Portfolio Selection: Efficient Diversification of Investments John Wiley
& Sons, New York, New York, 1959.

Matz, M.C. "Administration of web versus paper surveys: Mode effects and response
rates," University of North Carolina, Chapel Hill, 1999, p. 87.

McFarlan, W.F. "Portfolio approach to information systems," Harvard Business Review
(59:5) 1981, pp 142 -150.

McKeen, J .D., Guimaraes, T., and Wetherbe, J .C. "The relationship between user
participation and user satisfaction: An investigation of four contingency factors,"
MIS Quarterly (18:4) 1994, pp 427 - 452.

'Milgrom, P., and Roberts, J. "Chapter 14 The classical theory of investments and
finance," in: Economics, Organization and Management, Prentice Hall, 1992, pp.
448 - 479.

Nelson, K., and Cooprider, J. "The contribution of shared knowledge to IS group
performance," MIS Quarterly (20:4) 1996, pp 409 - 429.

Powell, TC, and Dent-Micallef, A. "Information technology as a competitive advantage:
The role of human, business and technology resources," Strategic Management
Journal (18:5) 1997, pp 375 - 405.

Reich, B.H., and Benbasat, I. "Measuring the linkage between business and information
technology objectives," MIS Quarterly (20:1) 1996, pp 55 - 81.

Roepke, R., Agarwal, R., and Ferratt, T.W. "Aligning the IT human resource with
business vision: The leadership initiative at 3M," MIS Quarterly (24:2) 2000, pp
327 - 353.

Rokeach, M. "From individual to institutional values: With special reference to the values

of science," in: Understanding Human Values, M. Rokeach (ed.), Free Press, New
York, 1979, pp. 47 - 70.

101

Sanchez, J.I., Korbin, W.P.,Viscarra, D.M. "Corporate support in the aftermath of a
natural disaster: Effects on employee strains," Academy of Management Journal
(38:2) 1995.

Schneider, B. "Organizational Climates: An Essay," Peronnel Psychology (28) 1975, pp
447 - 479.

Segars, A.H., and Grover, V. "Strategic information systems planning success: An
investigation of the construct and its measurement," MIS Quarterly (22:2) 1998,
pp 139 - 163.

Swanson, EB, and Dans, E. "System life expectancy and the maintenance effort:
Exploring their equilibration," MIS Quarterly (24:2) 2000, pp 277 - 297.

Wade, M., and Tomasevic "Theories used in IS research - contingency theory," 2006.

Wallace, L., Keil, M., and Rai, A. "Understanding software project risk: A cluster
analysis," Information & Management (42) 2004, pp 115 - 125.

Waxer, C. "Portfolio management how Lowe's grows," in: CIO Magazine, 2005.

Weill, P., and Vitale, M. "Assessing the health of an information systems applications
portfolio: An example from process manufacturing," MIS Quarterly (23:4) 1999,
pp 601 - 624.

Zaheer, A., Zaheer,S. "Catching the Wave: Alertness, responsiveness and market
influence in global electronic networks," Management Science (43:11) 1997 , p 16.

Zmud, R.W. "Diffusion of modern software practices: Influence of centralization and
formalization," Management Science (28:12) 1982, pp 1421 - 1431.

102

103

 

   

ililiiiljlliijjlliMilli;I

‘ _-_______Ai _