LIFE CYCLE COST-BASED RISK MODEL FOR ENERGY PERFORMANCE
CONTRACTING RETROFITS
By
George H. Berghorn

A DISSERTATION
Submitted to
Michigan State University
in partial fulfillment of the requirements
for the degree of
Planning, Design and Construction – Doctor of Philosophy
2014

ABSTRACT
LIFE CYCLE COST-BASED RISK MODEL FOR ENERGY PERFORMANCE
CONTRACTING RETROFITS
By
George H. Berghorn
Buildings account for 41% of the primary energy consumption in the United States, nearly half
of which is accounted for by commercial buildings. Among the greatest energy users are those in
the municipalities, universities, schools, and hospitals (MUSH) market. Correctional facilities are
in the upper half of all commercial building types for energy intensity. Public agencies have
experienced reduced capital budgets to fund retrofits; this has led to the increased use of energy
performance contracts (EPC), which are implemented by energy services companies (ESCOs).
These companies guarantee a minimum amount of energy savings resulting from the retrofit
activities, which in essence transfers performance risk from the owner to the contractor.

Building retrofits in the MUSH market, especially correctional facilities, are well-suited to EPC,
yet despite this potential and their high energy intensities, efficiency improvements lag behind
that of other public building types. Complexities in project execution, lack of support for data
requests and sub-metering, and conflicting project objectives have been cited as reasons for this
lag effect. As a result, project-level risks must be understood in order to support wider adoption
of retrofits in the public market, in particular the correctional facility sub-market.

The goal of this research is to understand risks related to the execution of energy efficiency
retrofits delivered via EPC in the MUSH market. To achieve this goal, in-depth analysis and
improved understanding was sought with regard to ESCO risks that are unique to EPC in this

market. The proposed work contributes to this understanding by developing a life cycle costbased risk model to improve project decision making with regard to risk control and reduction.
The specific objectives of the research are: (1) to perform an exploratory analysis of the EPC
retrofit process and identify key areas of performance risk requiring in-depth analysis; (2) to
construct a framework describing the sources of and mitigation strategies employed for assessing
key risks in EPC retrofits; (3) to develop a strategy for analyzing and evaluating risks for EPC
retrofits focused on managing expected costs throughout the project life cycle, and use data
collected through this strategy to develop and parameterize a risk model; and (4) to demonstrate
the applicability of the proposed life cost-based risk model through a pilot application to a case
study site.

Five major contributions to the body of knowledge resulting from the research include: (1) a
consensus-based assessment of ESCO risk management; (2) characterization of EPC retrofit
risks borne by ESCOs; (3) an empirical evaluation of scenario failure mode and effects analysis
and its application to this domain; (4) development and pilot application of a life cycle costbased risk model; and (5) future expansion of the research approach to other domains.

The researcher envisions that full implementation of the research will further encourage the
growth of the energy services industry, and support focused retrofits in complex building types
that typically can benefit the most from such work. Ultimately, this will reduce the energy
consumption of public sector buildings to levels that are more fitting with the global principles of
sustainability and responsible management of constrained resources.

Copyright by
GEORGE H. BERGHORN
2014

ACKNOWLEDGMENTS

This dissertation is the result of significant effort and teamwork over the past several years. As
this chapter of my formal education comes to a close I am reminded that this is but the midpoint
of a life spent learning. I am forever indebted to the many teachers, friends, and family who have
helped bring me to this place.

I am grateful beyond words to Dr. Matt Syal for his wisdom, patience, guidance, support, and
encouragement throughout this journey. His expertise, enthusiasm, and wisdom have been
invaluable and I hope to uphold his high standards as I move on from this endeavor.

I am very appreciative for the members of my advisory committee: Dr. Scott Witter, Dr. Satish
Joshi, and Dr. Mike Mazor. Their insights, valuable input, and support throughout this journey
were exceptional, and for this I say thank you.

I am thankful to Dr. Bob von Bernuth for introducing me to the Construction Management
program. Thanks are also due to Dr. Tariq Abdelhamid for his lessons about much more than
critical path scheduling. I am appreciative of the support and friendship of the faculty, staff, and
research assistants at the Center for Community and Economic Development.

It has been said that to be rich in friends is to be poor in nothing. First, I thank my friends in
research – Sagata Bhawani, Joe Diffendal, Angelo Garcia, Jawanda Jackson, Samarth Jain, and
Anthony Sparkling for their friendship, support, and encouragement. I especially want to thank

v

Daniel Duah and Aman Singh for their enduring friendship that has become more like family. I
would not be where I am presently without the support of wonderful work colleagues and friends
– Jack Bergeron, Margie Clark (and Dan), Brian Craig, Matt Dunham, Tim Fox, Sean Huberty,
Jim Lynch, Dr. Evan Montague, Dr. Michael Nealon, Dr. Stephanie Shanblatt, Art Theusch, and
Daniella Torcolacci. I am indebted to the six best brothers an only child could have – Andy
Chen, Darin Ninness, Frank Spano, Shawn Stanford and the two that started my graduate school
journey with me, John Burrows and Carlos Gonzalez.

I am most appreciative to the professionals that enabled the success of this research. Tommy
Norris and Paul Sheldon have become valued friends and I look forward to our continued
personal and professional interactions. I thank Steve Kmenta for his insights into SFMEA.
Special thanks are due the participants in the data collection section of this research. This work
could not have been completed without their willingness to participate, expertise, and insights.

Finally, none of this would have been possible without the unending love and support of my
awesome family. My grandfather, Primo Galli, got me interested in construction in his workshop
when I was just a child. My parents, George F. and Mary Berghorn instilled in me a love for
learning at a young age and have always supported me, regardless of the pursuit. Thanks, too, for
the endless lunches as I completed this work. My in-laws, Harvey and Pat Riley have been
supportive throughout the journey and I enjoyed “talking shop” with Harvey, a retired
construction estimator. Finally, to my wonderful wife Emily and amazing daughter Caylie – you
have endured more than can be expected and kept a smile on your face, and mine, throughout. I
could not have done this without you – let’s take that vacation I owe you.

vi

TABLE OF CONTENTS

LIST OF TABLES ....................................................................................................................... xiii
LIST OF FIGURES ..................................................................................................................... xix
KEY TO ABBREVIATIONS .................................................................................................... xxiii
CHAPTER 1
INTRODUCTION ...........................................................................................................................1
1.1 INTRODUCTION ...................................................................................................................1
1.1.1 Energy Performance Contracting (EPC) ....................................................................3
1.1.1.1 Correctional Facilities and Energy Performance Contracting .................................5
1.1.1.2 Contractual Forms of Energy Performance Contracting ..........................................6
1.1.2 Energy Characteristics of Relevant MUSH Sector Building Types...........................9
1.1.3 Energy Consumption and Efficiency in Correctional Facilities ...............................12
1.1.4 Life Cycle Cost Analysis and Energy Performance Contracting .............................14
1.2 NEED STATEMENT ............................................................................................................17
1.2.1 Premise #1 – Construction and Energy Efficient Retrofits Carry Risks for
Contractors ...............................................................................................................19
1.2.2 Premise #2 – There is a Need for Energy Efficient Retrofits in the MUSH
Market.......................................................................................................................21
1.2.3 Premise #3 – An Energy Efficiency Gap Exists in Correctional Facilities ..............24
1.2.3.1 Additional Pressure on Limited Capital Project Funding .......................................27
1.2.3.2 Complexities in Project Execution ...........................................................................28
1.2.3.3 Lack of Support for Data Requests and Sub-Metering ............................................29
1.2.3.4 Technical Scope Considerations and Conflicting Objectives ..................................29
1.2.4 Premise #4 – Uncertainty and Risk Must be Understood and Managed ..................30
1.3 RESEARCH GOAL AND OBJECTIVES ............................................................................35
1.3.1 Objective 1: Energy Performance Contract Retrofit Process ................................................35
1.3.2 Objective 2: Risk Framework for MUSH Market Energy Performance Contract
Retrofits .................................................................................................................................36
1.3.3 Objective 3: Life Cycle Cost-Based Risk Analysis and Evaluation......................................36
1.3.4 Objective 4: Pilot Application if the Life Cycle Cost-Based Risk Model ............................37
1.4 RESEARCH METHODS ......................................................................................................37
1.4.1 Objective 1: Energy Performance Contract Retrofit Process ...................................37
1.4.2 Objective 2: Risk Framework for MUSH Market Energy Performance
Contract Retrofits .....................................................................................................39
1.4.3 Objective 3: Life Cycle Cost-Based Risk Analysis and Evaluation ........................41
1.4.4 Objective 4: Pilot Life Cycle Cost-Based Risk Model ............................................42
1.5 RESEARCH SCOPE AND LIMITATIONS ........................................................................43
1.5.1 Research Scope ........................................................................................................43
1.5.2 Research Limitations ................................................................................................44
1.6 PROJECT OUTPUTS/RESEARCH CONTRIBUTIONS ....................................................45

vii

1.7 DISSERTATION ORGANIZATION ...................................................................................45
1.8 CHAPTER SUMMARY .......................................................................................................46
CHAPTER 2
LITERATURE REVIEW AND PRELIMINARY RISK FRAMEWORK ...................................47
2.1 INTRODUCTION .................................................................................................................47
2.2 ENERGY PERFORMANCE CONTRACTING RETROFITS ............................................48
2.2.1 History and Current Status .......................................................................................48
2.2.2 Energy Performance Contract Structure and Elements ............................................51
2.2.2.1 Energy Performance Contracting Elements ............................................................52
2.2.2.2 Energy Performance Contract Life Cycle ................................................................55
2.2.3 Savings Models ........................................................................................................62
2.2.3.1 Shared Savings .........................................................................................................62
2.2.3.2 Guaranteed Savings .................................................................................................63
2.2.4 Information Categories for EPC Retrofits ................................................................64
2.2.4.1 Sources of Quantitative Information ........................................................................67
2.2.4.2 Sources of Expert Knowledge ..................................................................................71
2.3 RISK IN CONSTRUCTION .................................................................................................72
2.3.1 Key Concepts ...........................................................................................................72
2.3.2 Decision Making Under Risk and Uncertainty ........................................................74
2.3.2.1 Bounded Rationality .................................................................................................74
2.3.2.2 Cognitive Biases and Energy Efficiency ..................................................................77
2.3.3 Risk in Construction and Energy Performance Contracting ....................................79
2.3.3.1 Construction Risk .....................................................................................................79
2.3.3.2 Energy Performance Contracting Risk ....................................................................81
2.3.3.3 Preliminary Risk Framework ...................................................................................82
2.3.4 Risk Management Framework and Methods ...........................................................90
2.3.4.1 Risk Identification and Classification ......................................................................90
2.3.4.2 Risk Analysis and Evaluation ...................................................................................91
2.3.4.3 Mitigation/Allocation ...............................................................................................92
2.3.5 Risk Identification, Analysis, and Evaluation Methods ...........................................93
2.3.5.1 Failure Mode and Effects Analysis ..........................................................................96
2.3.5.2 Scenario-Based FMEA (SFMEA)...........................................................................100
2.3.6 Expertise Elicitation and Delphi Technique ...........................................................104
2.3.6.1 Elicitation in Qualitative Risk Assessment .............................................................104
2.3.6.2 Delphi Technique ...................................................................................................106
2.4 LIFE CYCLE COST ANALYSIS .......................................................................................114
2.4.1 LCCA Overview ....................................................................................................115
2.4.2 Applications of LCCA to the Built Environment ...................................................117
2.4.2.1 Indirect Benefits of Sustainable Building Retrofits ................................................118
2.4.2.2 Example LCCA Applications ..................................................................................119
2.4.3 Addressing Risk and Uncertainty in LCCA ...........................................................121
2.5 CHAPTER SUMMARY .....................................................................................................122

viii

CHAPTER 3
RESEARCH METHODS AND DATA COLLECTION ............................................................123
3.1 CHAPTER OVERVIEW.....................................................................................................123
3.2 OBJECTIVE 1 – ENERGY PERFORMANCE CONTRACT RETROFIT
PROCESS ............................................................................................................................124
3.3 OBJECTIVE 2 – RISK FRAMEWORK FOR MUSH MARKET EPC PROJECTS .........126
3.3.1 Objective 2a – Energy Performance Contract Expertise Elicitation ......................126
3.3.1.1 Application of the Delphi Technique to Construction Engineering and
Management ...........................................................................................................127
3.3.1.2 Elicitation Strategy .................................................................................................134
3.3.2 Objective 2b – Data Collection and Analysis - Refined Risk Framework .............137
3.3.2.1 Data Collection ......................................................................................................138
3.3.2.2 Data Analysis .........................................................................................................139
3.3.2.3 Refined Risk Framework Construction ..................................................................140
3.3.2.4 Energy Performance Contract Retrofit Process Model .........................................141
3.4 OBJECTIVE 3 – RISK ANALYSIS AND EVALUATION ..............................................141
3.4.1 Risk Analysis Strategy Development .....................................................................144
3.4.1.1 Modified FMEA Matrix ..........................................................................................147
3.4.1.2 Modified Risk Criticality Measure .........................................................................147
3.4.2 Analyze Risk Criticality .........................................................................................148
3.4.3 Evaluate Critical Risks ...........................................................................................153
3.4.4 Parameterize Risk Framework and Construct Risk Model ....................................153
3.5 OBJECTIVE 4 – PILOT LIFE CYCLE COST-BASED RISK MODEL ...........................154
3.6 CHAPTER SUMMARY .....................................................................................................156
CHAPTER 4
ENERGY PERFORMANCE CONTRACT RISK FRAMEWORK AND RETROFIT
PROCESS MODEL .....................................................................................................................157
4.1 INTRODUCTION ...............................................................................................................157
4.1.1 Glossary ..................................................................................................................159
4.1.2 Chapter Objectives .................................................................................................160
4.2 EXPERTISE ELICITATION ..............................................................................................160
4.2.1 Delphi Panelist Recruitment and Selection ............................................................161
4.2.2 Conduct of the Delphi Panel and Achievement of Consensus ...............................162
4.2.3 Determination of Risk Tolerance and Expertise ....................................................164
4.3 DATA COLLECTION – DISCUSSION AND INFERENCES .........................................165
4.3.1 MUSH Market-Specific Risks ...............................................................................166
4.3.2 Risk Identification Methods, Timing, and Project Responsibility .........................168
4.3.3 Risk Evaluation Methods .......................................................................................170
4.3.4 Risk Identification ..................................................................................................172
4.3.4.1 Identification of Most Important Risk Categories ..................................................175
4.3.4.2 Panelist Characteristics Influencing Determination of Most Important
Risk Categories .......................................................................................................177
4.3.4.3 Role of Contractual Language and Occupant Behavior in Risk Importance ........180
4.3.4.4 Selection of Risk Categories for Further Analysis .................................................183

ix

4.3.5

4.4

4.5

4.6

4.7

Define Energy Performance Contract Retrofit Project Phases and Key
Decisions ................................................................................................................187
4.3.5.1 Project Development ..............................................................................................187
4.3.5.2 Energy Audit ...........................................................................................................189
4.3.5.3 Retrofit Design .......................................................................................................190
4.3.5.4 Project Execution ...................................................................................................191
4.3.5.5 Energy Performance ..............................................................................................192
4.3.6 Risk Categories by Project Phase ...........................................................................193
REFINED ENERGY PERFORMANCE CONTRACT RISK FRAMEWORK .................196
4.4.1 Changes from Preliminary Framework ..................................................................196
4.4.1.1 Risk Category and Risk Modifications ...................................................................202
4.4.1.2 Potential Risk Causes and Risk Controls/Mitigation Strategies ............................203
4.4.1.3 Risk Category Identification Frequency and Relative Risk Importance ................203
4.4.2 Relationship Among Delphi Data, Risk Framework, and Retrofit Process
Model ......................................................................................................................204
EPC RETROFIT PROCESS MODEL DEVELOPMENT..................................................204
4.5.1 Information is a Barrier to Energy Efficient Retrofit Adoption .............................206
4.5.2 Need for Expert Information ..................................................................................207
4.5.3 Review of Existing Models ....................................................................................208
4.5.4 Information Needs ..................................................................................................211
EPC RETROFIT PROCESS MODEL ELEMENTS ..........................................................211
4.6.1 Conceptual Grounding ...........................................................................................212
4.6.2 Key EPC Retrofit Decisions and Information by Project Phase ............................212
4.6.3 Project Development ..............................................................................................213
4.6.3.1 Decisions Made During the Project Development Phase ......................................213
4.6.3.2 Quantitative Information ........................................................................................215
4.6.3.3 Expert Knowledge ..................................................................................................216
4.6.4 Energy Audit ..........................................................................................................216
4.6.4.1 Decisions Made During the Energy Audit Phase ...................................................217
4.6.4.2 Quantitative Information ........................................................................................217
4.6.4.3 Expert Knowledge ..................................................................................................218
4.6.5 Retrofit Design .......................................................................................................219
4.6.5.1 Decisions Made During the Retrofit Design Phase ...............................................220
4.6.5.2 Quantitative Information ........................................................................................220
4.6.5.3 Expert Information .................................................................................................221
4.6.6 Project Execution and Energy Performance ...........................................................222
4.6.6.1 Decisions Made During the Project Execution and Energy Performance Phases 222
4.6.6.2 Quantitative Information ........................................................................................223
CHAPTER SUMMARY .....................................................................................................224

CHAPTER 5
RISK-BASED LIFE CYCLE COST MODEL ............................................................................225
5.1 INTRODUCTION ...............................................................................................................225
5.1.1 Glossary ..................................................................................................................225
5.1.2 Chapter Objectives .................................................................................................227

x

5.2 RISK ANALYSIS AND EVALUATION FRAMEWORK DEVELOPMENT
PROCESS............................................................................................................................228
5.2.1 Review Selected Risk Categories ...........................................................................230
5.2.2 Construct Failure Scenario Maps ...........................................................................230
5.2.3 Evaluate Risk Criticality ........................................................................................232
5.2.4 Recommend and Evaluate Mitigation Strategies ...................................................237
5.2.5 Calculate the Expected Life Cycle Value ..............................................................237
5.3 SCENARIO-BASED FMEA DATA COLLECTION AND ANALYSIS ..........................238
5.3.1 SFMEA Panel Composition ...................................................................................239
5.3.2 Data Analysis and Results ......................................................................................241
5.3.2.1 Risk Scenarios and Scenario Maps ........................................................................241
5.3.2.2 Expected Cost and Risk Priority Number ..............................................................253
5.3.2.3 Most Critical Risks .................................................................................................254
5.3.2.4 Relationship Between EC and RPN .......................................................................255
5.3.2.5 Differences in Panelist Criticality Analysis ...........................................................261
5.4 FRAMEWORK DEVELOPMENT.....................................................................................263
5.4.1 Life Cycle Cost-Based Model ................................................................................265
5.4.1.1 Risk Scenario Probabilities and Costs ...................................................................265
5.4.1.2 Mitigation Strategies Cost-Benefits .......................................................................269
5.4.1.3 Expected Life Cycle Value......................................................................................270
5.4.1.4 Decision Solutions ..................................................................................................271
5.4.2 Framework Database ..............................................................................................271
5.4.3 User Interface .........................................................................................................273
5.5 CHAPTER SUMMARY .....................................................................................................273
CHAPTER 6
PILOT APPLICATION OF LIFE CYCLE COST-BASED RISK MODEL ..............................275
6.1 INTRODUCTION ...............................................................................................................275
6.1.1 Glossary ..................................................................................................................276
6.1.2 Chapter Objectives .................................................................................................277
6.2 DEMONSTRATION OF LIFE CYCLE COST-BASED RISK MODEL ..........................277
6.2.1 Development of User Interface ..............................................................................278
6.2.1.1 User Interface Layout .............................................................................................279
6.2.1.2 Start Screen ............................................................................................................279
6.2.1.3 User Preferences Screen ........................................................................................284
6.2.1.4 Review Scenarios Screen ........................................................................................285
6.2.1.5 Risk Data Entry Screen ..........................................................................................285
6.2.1.6 Mitigation Screen ...................................................................................................286
6.2.1.7 ELCV Results & Decisions Screen .........................................................................286
6.2.2 Supporting Structures for Variable Development ..................................................291
6.3 TEST CASE CHARACTERISTICS ...................................................................................294
6.3.1 Case Study Selection ..............................................................................................295
6.3.1.1 Case Study Location ...............................................................................................297
6.3.1.2 Probabilistic Scenarios for Test Case ....................................................................300
6.3.2 Risk Analysis and Evaluation Results ....................................................................300
6.3.3 Sensitivity Analysis ................................................................................................306

xi

6.4 LIFE CYCLE COST-BASED MODEL FUNCTION AND REFINEMENT ....................310
6.5 CHAPTER SUMMARY .....................................................................................................311
CHAPTER 7
SUMMARY AND FUTURE RESEARCH .................................................................................314
7.1 INTRODUCTION ...............................................................................................................314
7.2 SUMMARY OF OUTPUTS BY OBJECTIVES ................................................................316
7.3 RESEARCH CONTRIBUTIONS .......................................................................................322
7.3.1 Consensus-Based Assessment of ESCO Risk Management ..................................322
7.3.2 Characterization of EPC Retrofit Risks Borne by ESCOs .....................................323
7.3.3 Empirical Evaluation of Improved FMEA Method ...............................................324
7.3.4 Development of Life Cycle Cost-Based Risk Model .............................................325
7.3.5 Expansion of Research Approach to Other Domains .............................................325
7.4 RECOMMENDED AREAS OF FUTURE RESEARCH ...................................................326
7.4.1 Integration of Modeled Energy Performance with Project Risks ..........................326
7.4.2 Development as an Intelligent Decision Support System ......................................327
7.4.3 Critical Success Factors for EPC Retrofits in Complex Facilities .........................329
7.5 CHAPTER SUMMARY .....................................................................................................330
APPENDICES .............................................................................................................................332
APPENDIX A.1: Preliminary Questionnaire for Risk Tolerance, Risk Identification,
and Risk Management in Energy Performance Contracting........................................................333
APPENDIX A.2: Delphi Round 1 Questionnaire for Risk Tolerance, Risk Identification,
and Risk Management in Energy Performance Contracting........................................................336
APPENDIX A.3: Delphi Round 2 Questionnaire for Risk Identification and Risk
Management in Energy Performance Contracting .......................................................................347
APPENDIX B: Delphi Panel Description and Expert Determination .........................................350
APPENDIX C.1: Detailed Analysis of Data Collected for Developing Life Cycle
Cost-Based Risk Model in Delphi Round 1 .................................................................................370
APPENDIX C.2: Detailed Analysis of Data Collected for Developing Life Cycle
Cost-Based Risk Model in Delphi Round 2 .................................................................................388
APPENDIX D: SFMEA Panel Results ........................................................................................393
APPENDIX E: Risk Model Pilot Test - Case Study Details .......................................................467
REFERENCES ............................................................................................................................473

xii

LIST OF TABLES

Table 1-1. CBECS Building Types, Definitions, and Sub-Categories ......................................... 10
Table 1-2. Instances of ESCO Risk During the Performance Contracting Process...................... 32
Table 2-1. IPMVP Option Descriptions........................................................................................ 60
Table 2-2. Comparison of EPC Contract Types ........................................................................... 62
Table 2-3. Potential Cognitive Biases During Energy Efficient Project Lifecycle ...................... 77
Table 2-4. Sources of Risk by Phase in BOT and PPP Construction Projects ............................. 81
Table 2-5. Preliminary Risk Framework for ESCOs Undertaking MUSH Market
EPC Retrofit Projects .................................................................................................................... 84
Table 2-6. Comparison Between Traditional FMEA and Scenario FMEA ................................ 100
Table 3-1. Guidelines for Expert Selection................................................................................. 131
Table 3-2. Scoring Rubric for Expert Determination ................................................................. 132
Table 3-3. Summary of Question Focus Areas and Need for Consensus ................................... 140
Table 3-4. Occurrence Measure Ranking Scale .......................................................................... 151
Table 3-5. Severity Measure Ranking Scale ............................................................................... 151
Table 3-6. Detection Measure Ranking Scale............................................................................. 152
Table 3-7. Selected Correctional Facility Case Study Characteristics ....................................... 156
Table 4-1. Summary of Part III Questions and Consensus Results ............................................ 163
Table 4-2. Research Participant Risk Tolerance Profiles ........................................................... 165
Table 4-3. Summary Expertise Attributes of Delphi Members .................................................. 165
Table 4-4. Elicited Knowledge: MUSH Market-Specific Risks ................................................. 167
Table 4-5. Constructed Knowledge: Risk Identification Process ............................................... 169
Table 4-6. Constructed Knowledge: Risk Evaluation Methods.................................................. 170

xiii

Table 4-7. Identification of Risk Categories in the Preliminary Risk Framework .................... 173
Table 4-8. Additional Risks Identified by Panelists ................................................................... 174
Table 4-9. Risk Importance Ranking from Delphi Panel Data ................................................... 177
Table 4-10. Panelist Characteristic Indicator Variables ............................................................. 179
Table 4-11. Spearman’s Rho and Significance Values – Panelist Characteristics ..................... 179
Table 4-12. Risk Causes and Control Measures for Top Risk Categories.................................. 185
Table 4-13. Risk Category Importance by Project Phase ........................................................... 195
Table 4-14. Refined Risk Framework for ESCOs Undertaking MUSH Market
EPC Retrofit Projects .................................................................................................................. 197
Table 4-15. Quantitative Information Needed During Energy Audit Phase ............................... 218
Table 5-1. FMEA Panel Participant Risk Tolerance and Expertise Determination ................... 240
Table 5-2. Potential Root Causes ................................................................................................ 242
Table 5-3. SFMEA Risk Scenarios ............................................................................................. 245
Table 5-4. Impact of Facility Factors on Prior Probabilities ...................................................... 253
Table 5-5. Consensus Assessment of Most Critical Risks.......................................................... 254
Table 5-6. Analysis of the Curve Describing the Severity and Cost Relationship ..................... 257
Table 5-7. Instances of Identical EC and RPN Priorities............................................................ 259
Table 5-8. Identical RPN Values with Large EC Variance ........................................................ 259
Table 5-9. Identical EC Values with Large RPN Variance ........................................................ 260
Table 5-10. RPN and EC Yield Mixed Risk Management Priorities ......................................... 260
Table 5-11. Panelist Variance in Key Risk Analysis Measures.................................................. 263
Table 5-12. Project-, Facility-, and External Environmental-Based Values for
Probability of Cause .................................................................................................................... 266
Table 6-1. Variables Supported by the Risk Data Screen ........................................................... 292

xiv

Table 6-2. Case Study Location – Selected Characteristics ....................................................... 297
Table 6-3. Data Obtained for Case Study Location .................................................................... 298
Table 6-4. Probabilistic Variable Values .................................................................................... 301
Table 6-5. Model Output – Ten Most Critical Risks .................................................................. 302
Table 6-6. Ten Risks to Which the Model is Most Sensitive ..................................................... 307
Table A-1. List of Previously-Identified Risk Categories for ESCOs Engaged
in Performance Contracting ........................................................................................................ 342
Table A-2. Features of Top Three Identified Risk Categories .................................................... 346
Table B-1. Expertise Assessment for Participant ER-1-E .......................................................... 351
Table B-2. Expertise Assessment for Participant ER-2-E .......................................................... 352
Table B-3. Expertise Assessment for Participant ER-3-U .......................................................... 353
Table B-4. Expertise Assessment for Participant ER-4-M ......................................................... 354
Table B-5. Expertise Assessment for Participant ER-5-M ......................................................... 355
Table B-6. Expertise Assessment for Participant ER-6-E .......................................................... 356
Table B-7. Expertise Assessment for Participant ER-7-M ......................................................... 357
Table B-8. Expertise Assessment for Participant ER-8-M ......................................................... 358
Table B-9. Expertise Assessment for Participant ER-9-M ......................................................... 359
Table B-10. Expertise Assessment for Participant ER-10-E ...................................................... 360
Table B-11. Expertise Assessment for Participant ER-11-E ...................................................... 361
Table B-12. Expertise Assessment for Participant ER-12-M ..................................................... 362
Table B-13. Expertise Assessment for Participant ER-14-E ...................................................... 363
Table B-14. Expertise Assessment for Participant ER-15-M ..................................................... 364
Table B-15. Expertise Assessment for Participant ER-16-M ..................................................... 365
Table B-16. Expertise Assessment for Participant ER-17-U ...................................................... 366

xv

Table B-17. Expertise Assessment for Participant ER-18-E ...................................................... 367
Table B-18. Expertise Assessment for Participant ER-19-E ...................................................... 368
Table B-19. Expertise Assessment for Participant ER-20-E ...................................................... 369
Table C-1. Panelists’ Risk Tolerance.......................................................................................... 370
Table C-2. Modified Panelists’ Risk Tolerance Categories........................................................ 370
Table C-3. Number of Years in the EPC Retrofit Industry......................................................... 371
Table C-4a. Number EPC Retrofits ............................................................................................ 372
Table C-4b. Number EPC Retrofits Per Year ............................................................................. 372
Table C-5. Percentage of EPC Retrofits in the MUSH Market .................................................. 372
Table C-6a. Experience with MUSH Sub-Markets .................................................................... 373
Table C-6b. Experience with Corrections Facility EPC Retrofits .............................................. 373
Table C-7. Experience with Project Financing and Financial Incentives ................................... 374
Table C-8. Retrofit Strategies Frequency of Occurrence in MUSH Market Projects ................ 374
Table C-9a. Number of Retrofit Measures Deployed on Projects .............................................. 375
Table C-9b. Retrofit Strategies Deployed by Panelists – Response Rate for
Each Measure .............................................................................................................................. 375
Table C-10. Energy-Related Credentials Held by Panelists ....................................................... 376
Table C-11. Highest Relevant Degree Held by Panelists ........................................................... 376
Table C-12. Experience as a College or University Faculty Member ........................................ 377
Table C-13. Publications Related to EPC ................................................................................... 377
Table C-14. Invited Presentations on a Related Subject ............................................................. 378
Table C-15. Consideration of Individual Risk Categories .......................................................... 379
Table C-16. Additional Risks Identified by Panelists ................................................................. 380

xvi

Table C-17a. Do MUSH Market EPC Projects Have Different Risks from nonMUSH Projects? ......................................................................................................................... 381
Table C-17b. Riskier Market Sectors Identified by Panelists Perceiving a Different
Risks in MUSH Market Projects................................................................................................. 381
Table C-17c. Segments with Greater Risk Identified by Panelists Identifying MUSH
Market as the Riskier Market ...................................................................................................... 381
Table C-18. Phase I Delphi Panel Risk Importance Rankings ................................................... 383
Table C-19. Risk Causes and Control Measures for Top Risk Categories ................................. 385
Table CC-1. Project Phases Most Frequently Supported............................................................ 388
Table CC-2. Additional Risks Identified by Panelists ................................................................ 389
Table CC-3. Risk Importance Ranking – Round 1 and Round 2 Delphi Panel Data ................. 390
Table CC-4. Risk Identification Methods Used by Delphi Panelists.......................................... 391
Table CC-5. Risk Evaluation Methods Used by Delphi Panelists .............................................. 392
Table D-1. Failure Mode Occurrence Rank Table...................................................................... 393
Table D-2. Failure Mode Severity Rank Table ........................................................................... 393
Table D-3. Failure Mode Likelihood Rank Table ...................................................................... 394
Table D-4. Panelist Consensus SFMEA Worksheet ................................................................... 395
Table D-5. Panelist RP-01-E SFMEA Worksheet ...................................................................... 407
Table D-6. Panelist RP-02-E SFMEA Worksheet ...................................................................... 419
Table D-7. Panelist RP-03-O SFMEA Worksheet ..................................................................... 431
Table D-8. Panelist RP-04-M SFMEA Worksheet ..................................................................... 443
Table E-1. Baseline Energy and Water Consumption ................................................................ 468
Table E-2. Building Subject to EPC Retrofit Activities ............................................................. 469
Table E-3. ECM Descriptions ..................................................................................................... 470
Table E-4. Proposed ECMs not Selected for Project .................................................................. 470

xvii

Table E-5. Proposed Costs and Savings for Recommended ECMs............................................ 471
Table E-6. Project Financial Summary ....................................................................................... 472
Table E-7. Base Year Condition and Expected Energy and Water Savings ............................... 472

xviii

LIST OF FIGURES

Figure 1-1a. United States ESCO – 1990 Projected Through 2014 ............................................... 4
Figure 1-1b. ESCO Revenues by Market – 2006 and 2008............................................................ 5
Figure 1-2a. Comparison of Shared Savings and Guaranteed Savings EPC Models ..................... 7
Figure 1-2b. EPC Contractual Models in Private and Public Sector Projects ................................ 8
Figure 1-3. Relative End-Use Energy Intensities Among MUSH Sector Building Types ........... 11
Figure 1-4. Energy Performance Contract Flowchart with Instances of Contractor’s Risks........ 31
Figure 1-5. Research Objectives, Methods, and Outputs.............................................................. 36
Figure 2-1 Literature Review Categories ...................................................................................... 47
Figure 2-2 Energy Performance Contracting Cash Flow and Savings Model .............................. 51
Figure 2-3. Percentage of Projects Installing Various Retrofit Measures .................................... 54
Figure 2-4. U.S. Department of Energy Commercial Buildings Portal Screenshot ..................... 69
Figure 2-5. ASHRAE Service Life and Maintenance Cost Database Web Interface
Screenshot ..................................................................................................................................... 71
Figure 2-6. Performance of ESCO Savings Guarantees ............................................................... 82
Figure 2-7. Evaluation of Selected Risk Management Strategies by Project Life
Cycle Phase, Risk Management Phase, and Organizational Maturity Toward Risk .................... 96
Figure 2-8. Comparison of Failure Modes and Failure Scenarios .............................................. 102
Figure 2-9. Example Failure Scenarios Over a Product Life Cycle ........................................... 103
Figure 2-10. Expected Cost Components in Failure Scenarios .................................................. 103
Figure 2-11. Example Life Cycle Cost Categorization ............................................................... 116
Figure 3-1. Objective 1 Methods and Outputs ............................................................................ 125
Figure 3-2. EPC Risk Expertise Elicitation Process Model and Objective 2 Methods
and Outputs ................................................................................................................................. 138

xix

Figure 3-3. Objective 3 Methods and Outputs ............................................................................ 144
Figure 4-1. Risk Management Process and Methods Mapped to Dissertation Chapters ............ 158
Figure 4-2. ESCO Industry Shares by Business Type ................................................................ 162
Figure 4-3. Risk Identification Methods Reported in Second Delphi Panel............................... 169
Figure 4-4. Risk Evaluation Methods Reported in Second Delphi Panel ................................... 171
Figure 4-5. Range and Distribution of Risk Consideration Frequency ...................................... 173
Figure 4-6. Pareto Histogram of Relative Importance Scores for Risk Categories .................... 176
Figure 4-7. Energy Performance Contract Retrofit Phases ......................................................... 188
Figure 4-8. Project Decisions and Costs Over the Project Life Cycle ........................................ 194
Figure 4-9. Most Important Risk Category Average Score by Project Phase ............................ 195
Figure 4-10. Relationship Between EPC Process Model and ESCO Risk Framework .............. 204
Figure 4-11. Residential Energy Retrofit Decision Process Model ............................................ 209
Figure 4-12. Sustainable Building Retrofit Process Model ........................................................ 210
Figure 4-13. Energy Performance Contract Retrofit Process Model .......................................... 214
Figure 5-1. Risk Analysis and Evaluation Process ..................................................................... 228
Figure 5-2. Risk Analysis and Evaluation Framework Development Steps ............................... 230
Figure 5-3. Example Failure Scenario ........................................................................................ 231
Figure 5-4. Example Failure Scenario Map ................................................................................ 232
Figure 5-5. Calculation of Expected Cost ................................................................................... 233
Figure 5-6. Use of Risk Scenarios and Expected Cost Over Project Life Cycle ........................ 234
Figure 5-7. Example Risk Scenario Maps Developed by SFMEA Panel ................................... 244
Figure 5-8. EC vs RPN for all Panelist Responses and Consensus Measure (n=380) ............... 258
Figure 5-9. EC vs RPN for all Panelist Responses and Consensus Measure (n=303) ............... 261

xx

Figure 5-10. Panelist Expected Cost Variation Trend Line Plot ................................................ 262
Figure 5-11. Risk Analysis and Evaluation Framework for EPC Retrofits ................................ 264
Figure 6-1. User Interface Screen Selection ............................................................................... 279
Figure 6-2. Job Data Input Screen .............................................................................................. 280
Figure 6-3. Facility and Project-Specific Data Input Screen ...................................................... 281
Figure 6-4. Energy Conservation Measures Input Screen .......................................................... 283
Figure 6-5. Data Inputs on the ELCV Results & Decisions Screen ........................................... 287
Figure 6-6. Project Life Cycle Value at Risk Plot ...................................................................... 290
Figure 6-7. Most Critical Risks Value at Risk Plot .................................................................... 291
Figure 6-8. Pilot Model Run Results Screen .............................................................................. 303
Figure 6-9. Pilot Model Run Results Screen Plots ..................................................................... 303
Figure 6-10. Model Risks by Project Phase ................................................................................ 305
Figure 6-11. Monte Carlo Simulation Results – Distribution of EC Values .............................. 307
Figure 6-12. Model Sensitivity to Deterministic and Probabilistic Parameters.......................... 308
Figure 6-13. Probabilistic Input Parameter Distributions ........................................................... 309
Figure 6-14. Model Sensitivity to Probabilistic Equipment Service Life Parameter
Values ......................................................................................................................................... 310
Figure A-1. TIAA-CREF Questionnaire Comparison Portfolios ............................................... 338
Figure D-1. Risk Scenario Map 1A ............................................................................................ 455
Figure D-2. Risk Scenario Map 1B............................................................................................. 456
Figure D-3. Risk Scenario Map 1C............................................................................................. 457
Figure D-4. Risk Scenario Map 1D ............................................................................................ 458
Figure D-5. Risk Scenario Map 1E ............................................................................................. 459
Figure D-6. Risk Scenario Map 1F ............................................................................................. 460

xxi

Figure D-7. Risk Scenario Map 2A ............................................................................................ 461
Figure D-8. Risk Scenario Map 2B............................................................................................. 462
Figure D-9. Risk Scenario Map 2CD .......................................................................................... 463
Figure D-10. Risk Scenario Map 2E ........................................................................................... 464
Figure D-11. Risk Scenario Map 2F ........................................................................................... 465
Figure D-12. Risk Scenario Map 2G .......................................................................................... 466
Figure E-1. Parnall Correctional Facility .................................................................................... 467
Figure E-2. Parnall Correctional Facility Cell Block 7............................................................... 469

xxii

KEY TO ABBREVIATIONS

BBN

Bayesian Belief Network

BTU

British Thermal Unit

C&I

Commercial and Industrial

CBECS

Commercial Buildings Energy Consumption Survey

CCF

Hundred Cubic Feet

CDD

Cooling Degree Days

D

Detection

EC

Expected Cost

ECPV

Expected Cost Based on Present Value

ECM

Energy Conservation Measures

ELCV

Expected Life Cycle Value

ELCPV

Expected Life Cycle Value Based on Present Value

EPC

Energy Performance Contract

ESCO

Energy Service Company

ESPC

Energy Savings Performance Contract

FEMP

Federal Energy Management Program of the U.S. Department of Energy

FMEA

Failure Mode and Effects Analysis

HDD

Heating Degree Days

IGA

Investment Grade Audit

IPMVP

International Performance Measurement and Verification Protocol

KBTU

Thousand BTUs

xxiii

kVA

Kilo Volt Ampere

kW

Kilowatt

kWh

Kilowatt Hours

LCCA

Life Cycle Cost Analysis

M&V

Measurement and Verification

MMBTU

Million BTUs

MUSH

Municipalities, Universities, Schools, and Hospitals

NEB

Non-Energy Benefit

O

Occurrence

O&M

Operations and Maintenance

OPR

Owner’s Project Requirements

Pcause

Probability of the Cause of a Risk Scenario

Peffect

Probability of the End Effect of a Risk Scenario

RM

Risk Management

RPN

Risk Priority Number

S

Severity

SFMEA

Scenario-Based Failure Mode and Effects Analysis

SPV

Single Present Value

xxiv

CHAPTER 1
INTRODUCTION

1.1 INTRODUCTION
High performance sustainable buildings (HPSB) are a part of the broader green building market
and defined as those buildings that utilize integrated design, are highly efficient in their use of
energy and water, improve indoor environmental quality for building occupants, and minimize
the negative environmental impacts of material consumption and use (U.S. Department of
Energy 2008). These buildings result in reduced life cycle costs for building operation and
maintenance and reduced energy and water consumption, leading to increased demand for such
buildings. With 40% of total energy use in the United States coming from the buildings sector
(U.S. Department of Energy 2011a; Morganstern et al. 2008), significant economic advantages
exist for buildings that exhibit high standards of energy efficiency. Energy efficient building
practices have been identified as one of the most cost effective measures for controlling
operating costs in buildings (Drumheller and Wiehagen 2004; Dong et al. 2005; Harvey 2009).
The U.S. non-residential green building market is projected to triple from approximately $43-$54
million in 2010 to $145 billion by 2015; major retrofit and renovation projects are expected to
experience a three to fivefold increase to $18 billion during that same time. Energy efficiency is
expected to be a key element of up to two-thirds of these retrofit projects between 2010 and 2015
(Russo 2011).

Commercial buildings account for nearly half of the primary energy consumption in the United
States (U.S. Department of Energy 2011a). Among buildings with the greatest energy intensity

1

(measured by British Thermal Units [BTUs] consumed per square foot of building area) are
hospitals, nursing homes, food service and sales facilities, educational, and public safety
facilities. While the U.S. Department of Energy (DOE) does not further disaggregate its
Commercial Building Energy Consumption Survey (CBECS) data beyond the building types
mentioned above, an analysis of the data reveals that prison and jail facilities are in the upper
half of all commercial building types for energy intensity (U.S. Department of Energy 2011a).
Average energy intensities between 120,000 and 221,100 BTUs per square foot were recorded
for buildings of the “public order and safety” type in the CBECS survey (U.S. Department of
Energy 2011b). Correctional facility data, obtained from California and Wisconsin, shows these
facilities had average energy intensities between 163,000 BTU/sf (California Department of
General Services 2011) and 193,400 BTU/sf (Wisconsin Department of Administration 2011)
respectively, thereby confirming the CBECS data. Energy efficient retrofit is a growing and
important part of both the construction and energy economies – utilities spent $6.7 billion on
demand-side management programs in 2011 (Forster et al. 2013), which is projected to increase
to between $6.5 and $15.6 billion by 2025 (Barbose et al. 2013). These utility expenses, along
with owner-financed improvements, had a net impact of $574 billion to the U.S. economy in
2010, which was three times the investment made in conventional energy generation capacity
(Laitner 2013).

Despite high energy intensities, many public agencies have reduced capital budgets to fund
retrofit programs, even among projects which enhance efficiencies and thus save funds
(Bharvirkar et al. 2008). The continued operation of inefficient mechanical, electrical, and
plumbing equipment can lead to increased utility costs, thereby exacerbating this situation

2

(Bhattacharjee et al. 2010). These factors have contributed to the increased use of energy
performance contracts (EPCs), particularly in what has been termed the MUSH (municipalities,
universities, schools, and hospitals) market (Hopper et al. 2005).

1.1.1 Energy Performance Contracting (EPC)
Energy performance contracting is a project delivery mechanism that finances retrofits using
projected utility savings gained through improved energy efficiency. As a result, capital is not
required at project startup; rather, the work can be financed over a period of years, termed the
contract performance period. EPCs are generally executed by an energy service company
(ESCO), which performs the work and receives payment as a result of accumulated utility
savings which exceed baseline consumption. ESCOs are defined as businesses that provide a
full-range of energy efficiency services and include performance contracting as a core
component of the energy efficiency business (Goldman et al. 2005; Hopper et al. 2007).

Businesses cannot be classified as ESCOs in the absence of engaging in performance
contracting. This is an important concept, because it implies that ESCOs, by definition,
guarantee some level of performance in their energy efficiency services, which in turn creates
inherent risks to the ESCO throughout the EPC delivery process. The level to which these firms
can manage and mitigate these risks defines a successful ESCO. As a result, the concepts of risk
analysis, risk management, decision-making under uncertainty, and EPCs are interrelated and
must be treated as such.

3

Energy service company revenues have grown significantly in the United States over the past 20
years, with annual growth rates of approximately 9% between 2009 and 2011, (Figure 1-1a),
with a projected doubling to between $10.6 and $15.3 billion by 2020. This growth outpaced the
U.S. Gross Domestic Product, and its continued growth despite the economic recession indicates
that the EPC model may in fact be somewhat insensitive to the economy due to its ability to
deliver capital improvements with little-to-no up-front investment (Stuart et al. 2013).
Approximately 85% of ESCO revenue in 2011 was the result of energy efficiency projects,
thereby providing an analogous measure to the size of the EPC industry at that point in time.

Figure 1-1a. United States ESCO – 1990 Projected Through 2014
(Source: Stuart et al. 2013)

4

The MUSH market is the dominant sector for EPC retrofits, representing a 63% share of total
energy ESCO revenue in 2011, a 9% growth since 2006 (Figure 1-1b) (Satchwell et al. 2010).
While this market sector has been characterized as being mature, Stuart et al. (2013) reported
remaining market potential of between $52 and $94 billion as of 2012, which represents potential
annual energy savings of 211-311 trillion BTUs.

Figure 1-1b. ESCO Revenues by Market – 2006 and 2008
(Source: Larsen et al. 2012a; Stuart et al. 2013)
1.1.1.1 Correctional Facilities and Energy Performance Contracting
Correctional facility retrofit projects are well-suited to EPCs, as indicated by a framework
developed by the Australian Energy Performance Contracting Association (AEPCA 2000).
These projects are characterized by centralized buildings in a discrete location, typically
consisting of 50 or fewer buildings (often less), large gross square footages (>250,000 SF), high
energy bills ($500,000-$1,000,000/year are not uncommon), facilities that are generally more
than 20 years old, and due to capital constraints, more than 10-15 years have passed since their
last upgrade.

5

Projects may be procured using a single-phase or multi-phase approach. In the former, the
request for proposals (RFPs) leads to selection of an ESCO to conduct all phases of work; in the
latter, two RFPs are issued – one for selecting an ESCO at the design phase and preparation of
the technical audit and a second one for the investment grade audit (IGA), energy savings
guarantee, construction, and measurement and verification (M&V). Some RFPs include all
phases of work in a single contract while others divide the scope into separate work packages for
the audit, construction, and M&V, regardless of whether the work packages are bid separately.
The energy savings guarantee and M&V portion of the contract (or separate contract, if used)
generally includes provisions for the annual calculation of savings and may include a designated
time period (two to three years) of intense M&V activity after completion of construction, upon
which further annual savings are based. Some contracts may stipulate different calculation
methods during the construction phase, due to incomplete installation of energy conservation
measures (ECMs) and intense utility consumption due to construction activity. Additionally,
savings may be stipulated, where both parties agree to performance levels of individual ECMs
during project development or they may be measured and verified following options in the
International Performance Measurement and Verification Protocol (IPMVP) (EVO 2012). The
IPMVP is discussed in greater detail in Chapter 2.

1.1.1.2 Contractual Forms of Energy Performance Contracting
There are two predominant contractual forms of EPCs: (1) shared savings and (2) guaranteed
savings (Figure 1-2a). In the shared savings model, the ESCO shares the value of energy savings
with the project owner under a pre-determined arrangement that is dependent on project-specific
factors. Under this model, the ESCO typically secures financing from a third-party entity,

6

meaning that both parties bear some degree of credit and performance risk. Since the ESCO’s
share of the savings is the only form of payment they receive, their share of the savings is
typically higher in the early stages of the project when design and construction costs are

Figure 1-2a. Comparison of Shared Savings and Guaranteed Savings EPC Models
(Source: Hopper et al. 2005)
incurred, and decrease over the project life cycle (Hopper et al. 2005; Bertoldi et al. 2006; CCI
2009a; Larsen et al. 2012a).

Under the guaranteed savings approach, the ESCO guarantees the customer a certain quantity of
energy savings via the guaranteed savings contract, ensuring that the associated value of those
savings is enough to repay lease costs to the financier and to pay for the cost of the retrofit work
itself (Deng 2011). When using this model, the ESCO generally grants all excess savings to the
project owner, and agrees to be liable for any shortfalls in actual energy savings. In the
guaranteed savings model, the owner typically secures funding directly from a third-party lender
7

(although the ESCO may assist with securing financing), which eliminates credit risk for the
ESCO; however, in this model the ESCO retains project performance risks via the guaranteed
savings contract (Larsen et al. 2012a).

The dominant contractual form in the United States is the guaranteed savings model. According
to Goldman et al. (2005), 67% of public sector performance contract projects reviewed over a 15
year period utilized guaranteed savings (Figure 1-2b). According to the authors, public project
owners reported that they preferred the stronger savings guarantee, and ESCOs preferred the
lower financing costs for public projects and the ability to place more focus on matters directly
related to project performance. This is reversed in the developing world and in countries with
nascent ESCO markets, where the stronger preference is to execute shared-savings contracts.
This frees project owners from bearing any financial risk, and frees banks from retaining credit
risk, often in countries that have poorly-established banking infrastructures (Okay and Akman
2010).

Figure 1-2b. EPC Contractual Models in Private and Public Sector Projects
(Source: Larsen et al. 2012a)

8

As stated above, the guaranteed savings model allocates a larger share of performance-related
risk to the ESCO. These risks occur throughout the project life cycle and result primarily from
three sources: (1) development phase concerns related to customer and project characteristics as
well as potentially unrecoverable costs during this time; (2) the financial guarantee proffered by
the ESCO which results from actual levels of energy performance versus guaranteed levels of
performance; and (3) the lack of ESCO-initiated change orders in such projects, which make
additional costs difficult or impossible to recover (Kansas Corporation Commission, Energy
Division 2014; Oregon Department of Energy 2014). These risks are discussed in greater detail
in sections 1.2.1 and 1.2.4, and serve as a key underpinning of the need for this research.

1.1.2 Energy Characteristics of Relevant MUSH Sector Building Types
As defined above, the MUSH market includes municipalities, universities, schools, and hospitals.
These four sub-markets comprise a wide variety of building types and include correctional
facilities. These buildings are neither well-described nor well-represented in existing energy
consumption data sets, such as CBECS (U.S. Department of Energy 2013a). The CBECS data
can be queried by principal building activity and government ownership to identify buildings that
are representative of the MUSH market, to include other public order and safety, nursing home,
hospital, outpatient health, office, education, public assembly, and non-refrigerated warehouse
facilities. The “other public order and safety” buildings were queried based on 24-hour operation
to generate a data set that is analogous to correctional facilities and jails. Building type
descriptions are provided in Table 1-1, and illuminate some limitations of this highly-aggregated
data set; however, CBECS data provides a framework for understanding energy intensities and
enegy end-uses of a variety of building types despite its limitations.

9

Table 1-1. CBECS Building Types, Definitions, and Sub-Categories
Building Type
Summary Definition
Example Sub Categories
Education
Buildings used for academic or technical  Preschool or daycare
classroom instruction, such as
 Elementary/middle/high
elementary, middle, or high schools, and
school
classroom buildings on college or
 College or university
university campuses.
 Career or vocational training
Healthcare
Buildings used as diagnostic and
 Hospital
(Inpatient)
treatment facilities for inpatient care.
 Inpatient rehabilitation
Healthcare
Buildings used as diagnostic and
 Medical office
(Outpatient)
treatment facilities for outpatient care.
 Clinic or other outpatient
health care
Non-Refrigerated Buildings used to store goods,
 Non-refrigerated warehouse
Warehouse
manufactured products, merchandise,
raw materials, or personal belongings
(such as self-storage).
Nursing home
Buildings used to offer multiple
 Nursing home
accommodations for long-term residents  Assisted living or other
involving skilled nursing.
residential care
Office
Buildings used for general office space,
 Government office
professional office, or administrative
 City hall or city center
offices. Medical offices are included here  Administrative office
if they do not use any type of diagnostic
 Mixed-use office
medical equipment (if they do, they are
 Bank or financial institution
categorized as an outpatient health care
 Contractor's office
building).
 Non-profit or social services
Other public order Buildings used for the preservation of
 Jail or penitentiary
a
and safety – 24
law and order or public safety.
hour operation
Public assembly
Buildings in which people gather for
 Social or meeting space
social or recreational activities, whether
 Recreation
in private or non-private meeting halls.
 Library
 Armory
Service
Buildings in which some type of service  Vehicle storage/maintenance
is provided, other than food service or
 Post office or postal center
retail sales of goods.
 Copy center or printing shop
 Kennel
Notes:
a\ “Other public safety” is a sub-set of the “public safety” building type that excludes police and
fire stations. Facilities operating 24-hours per day were queried to provide an analog for
correctional facilities, since that building type cannot be disaggregated from CBECS data.
(Data source: U.S. Department of Energy 2003 Commercial Buildings Energy Consumption
Survey [CBECS])

10

Energy intensities varied by building type, from 67,112 BTU/sf (non-refrigerated warehouse) to
258,819 BTU/sf (healthcare-inpatient), with an average of 122,282 BTU/sf (U.S. Department of
Energy 2011b). As might be expected, end-use energy intensities among these building types are
dominated by heating, ventilation, and air conditioning (HVAC) (average 72,000 BTU/sf; 56%
of building energy intensity), lighting (average 19,369 BTU/sf; 17% of building energy
intensity), and water heating (average 14,274 BTU/sf; 13% of building energy intensity). Enduse energy intensities for various MUSH market building types are depicted in Figure 1-3.

Figure 1-3. Relative End-Use Energy Intensities Among MUSH Sector Building Types
(Source: U.S. Department of Energy 2003 Commercial Buildings Energy Consumption Survey)
Given the limited available data on energy consumption and a very small amount of literature
focused on energy efficiency in correctional facilities, the CBECS data were reviewed to find
other building types with similar energy end-use intensities of the “other public order and safety”
type. Nursing home and hospital building types were found to have the most similar end-use
intensities among dominant energy uses. Additionally, healthcare buildings are an important part
of the MUSH sector, due to their ubiquity in the built environment and the high energy
11

intensities of these buildings. A relatively rich literature is also available for hospital energy
efficiency that helps establish the context for the development of the proposed risk framework in
the context of correctional facilities. Key elements of this literature that are relevant to the
subject work include:


Energy efficiency frameworks, such as the American Society of Heating, Refrigeration,
and Air Conditioning Engineers (ASHRAE) Advanced Energy Design Guide (AEDG)
for Large Hospitals (ASHRAE 2012), the Advanced Energy Retrofit Guide for
Healthcare Facilities (U.S. Department of Energy 2013b), the 2007 Sustainable
Healthcare Energy Challenge (Burpee et al. 2009), and the Green Guide for Healthcare
(Burpee et al. 2009).



Suggested energy targets for various building systems (ASHRAE 2012)



Research-based approaches to energy efficiency retrofits in hospitals that can be directly
transferred to correctional facility retrofit frameworks, to include life cycle cost-based
technical approaches (Kolokotsa et al. 2012).

While not the focus of this research, healthcare facility energy efficiency research can be used to
inform the development of this research framework, given the similarity in energy consumption
and operational profiles of these buildings and correctional facilities.

1.1.3 Energy Consumption and Efficiency in Correctional Facilities
Less attention has been focused specifically on guidelines for reducing energy consumption in
correctional facilities; however, drivers for such work have been given as: (1) reducing
recidivism through inmate job training related to sustainability activities and (2) reducing

12

operating costs (Morgan 2010; Webster 2010; Sheldon and Atherton 2011; Sheldon et al. 2011).
These reduced costs have been equated to saving $1,000 to $1,250 per inmate per year (CEC and
CBC 2001; Sheldon et al. 2011), which can be equated to $44 to $55 million in annual
operational savings in Michigan alone. A small body of research has focused on the effects of
design and engineering on inmate behavior (e.g., violence level, health effects) due to indoor
environments; energy efficient and sustainable retrofits may provide such facility improvements
which can consequentially lead to improved inmate behavior (Moore 1980; Al-Hosany and
Elkadi 2002; Nalbone 2004; McMillan 2010; Wener 2012). Such practices were codified in
2010, when the American Correctional Association (ACA) amended its correctional facility
accreditation standard through the inclusion of the “ACA Standard on Sustainability” and the
attendant “Public Correctional Policy on Environmentally Responsible and SustainabilityOriented Practices” (American Correctional Association 2012).

Despite this body of research and recent attention provided by a new accreditation standard,
limited literature exists related to design and retrofit concepts related to energy consumption (and
efficiency) and guidelines to promote efficient construction and retrofit practices in correctional
facilities. The National Institute of Justice released the first national comprehensive guidance on
implementing sustainable technologies in correctional facilities, titled the “Greening Corrections
Technology Guidebook” in 2011 (Sheldon and Atherton 2011). This guidance document focuses
on sustainability practices in seven key systems, four of which directly impact energy
consumption: (1) lighting; (2) HVAC; (3) plug-in appliances, to include pumps and motors; and
(4) energy, to include renewable energy, passive cooling, and transportation considerations, as
well as one system, water efficiency, which has mediating effects on energy consumption. This

13

document does not specifically address technical solutions to achieve these sustainability goals;
however, management, implementation, and funding issues are addressed, including a discussion
about the role of EPC in procuring projects which have inadequate capital funding.

Two example guidelines for assessing the environmental performance of correctional facilities
are the “Energy Efficiency Design Guide for California Detention Facilities,” first published in
1990 (CEC and CBC 2001), and the Building Research Establishment’s Environmental
Assessment Method (BREEAM) Prisons scheme document, first published in 2008 (BRE Global
2012). The California document provides guidance for three primary building systems (building
envelope, lighting systems, and mechanical systems) with a fourth category devoted to ongoing
considerations, to include transformers, commissioning, and maintenance. Security concerns are
included with regard to minimum interior and exterior lighting levels, ingress and egress control
with skylights, and the location within the facility of mechanical plant components.

BREAAM Prisons addresses similar systems, although it places more emphasis on carbon
dioxide emissions, low- or zero-carbon emission strategies, sub-metering, acoustics, and
occupant comfort. It provides less technical implementation detail than the California guidance;
however, it places significant emphasis on security considerations, integrating these concerns
into nearly every category and issue.

1.1.4 Life Cycle Cost Analysis and Energy Performance Contracting
Life cycle cost analysis (LCCA), and a related cash flow analysis, are critical elements of the

14

IGA performed in conjunction with the planning phase of an EPC (Hansen 2006). The U.S. DOE
has created specific guidance for the use of LCCA in federal projects in accordance with
Executive Order (EO) 13123, calling for the comparison of existing equipment costs against
ECMs specified through an EPC project (Fuller 2005). The DOE guidance utilizes the EO 13123
definition of life cycle costs, “…the sum of present values of investment costs, capital costs,
installation costs, energy costs, operating costs, maintenance costs, and disposal costs over the
life-time of the project, product, or measure.” This is a relatively traditional definition of built
environment LCCA and does not include the large array of non-energy benefits (NEBs) in which
may be included in EPC retrofits. Non-energy benefits have the potential to positively affect the
financial performance of EPC projects; however, their use and calculation has been the subject of
considerable debate.

A small number of studies over the past 8-10 years have focused on NEBs. In a survey of ESCOs
and building managers conducted by Birr and Singer (2008), 93% of respondents felt that
building maintenance and operation costs savings were either significant or extremely significant
in motivating their organization to consider entering into an EPC. A survey conducted by
Jennings and Skumatz (2006) focused on additional benefits from the commissioning process
found that the primary benefits related to similar life cycle cost activities – finding and correcting
operational deficiencies, increasing operations and maintenance staff knowledge, and addressing
equipment maintenance considerations.

The ability to incorporate NEBs into LCCA varies across public projects. The federal energy
savings performance contract (ESPC) program is not clear about permitting eligible savings from

15

operations and maintenance and avoided capital costs (Larsen et al. 2012b). Larsen et al. (2012b)
examined legislation from six states (Florida, Hawaii, New Jersey, New Mexico, Pennsylvania,
and Virginia) to ascertain whether they permit the inclusion of NEBs in energy efficiency
projects. All six states permitted the inclusion of operations and/or maintenance benefits and five
permitted the use of avoided capital costs; however, only two of the six states (New Jersey and
Pennsylvania) allowed the inclusion of other NEBs, and two states limited the amount and type
of NEBs that may be included in a project.

Michigan’s Public Act 625 of 2012 allows for the inclusion of a variety of NEBs in energy
performance contracts issued for state facilities, to include measures that reduce capital
avoidance costs, capital improvement costs, maintenance costs, and operating costs (CostEffective Governmental Energy Use Act 2012). Examples provided in the law include:


Water consumption and sewage reduction devices and practices to include waterconserving equipment, low-impact landscaping features, equipment that permits
recycling of water, to “treated municipal effluent,” condensate and grey water recapture
devices, and water system sub-metering.



Operation and maintenance practices.



Indoor air quality improvements.



Life safety measures that lead to long-term operating cost reductions, including those
measures related to compliance with the Americans with Disabilities Act.



Any other building infrastructure improvement that leads to utility or operational cost
savings.

16

Hughes and Muessel (2000) identified an important life cycle cost consideration in executing
federal ESPCs under the Federal Energy Management Program (FEMP) – the need to calculate
life cycle costs and benefits of the entire package of ECMs, not just on an individual ECM-byECM basis, as included in early versions of technical guidance for the FEMP (Fuller and
Petersen 1995). This not only ensures that ECMs provide maximum benefits to the building
system, but also increases the potential to find synergies among them (e.g., envelope and lighting
changes can result in decreased cooling loads, thereby requiring a smaller HVAC system).

Since LCCA and the long-term energy savings guarantee feature of EPC projects are so closelyrelated, it is reasonable to conclude that a complete understanding of life cycle costs of the
portfolio of ECMs installed in such projects is critical to overall project success. Despite the
important role that LCCA plays in making retrofit decisions utilizing EPC, historically there
have been challenges with the method. These challenges center on the lack of data for
maintenance and replacement costs as well as the presence of uncertainty and risk in many
LCCA elements, and their attendant lack of inclusion in analysis (Kishk et al. 2003; Brink 2012).
LCCA methods must give appropriate consideration to uncertain or incomplete data as well as
risk factors inherent with system inputs in order to provide a complete analysis. This is
particularly important in the case of EPC, where the performance guarantee is based on energy
use reductions over the performance period of the contract.

1.2 NEED STATEMENT
The use of EPCs is growing in popularity among MUSH market owners, with overall industry
growth rates increasing rapidly in the past several years. This represents a marked increase from

17

the slowing growth trend observed during the late 1990s and early 2000s (Mills et al. 2006;
Satchwell et al. 2010). This actual and projected growth over the past five years exists despite the
downturn observed in the buildings sector and restricted lending as credit rules were temporarily
tightened. Much of this optimism resulted from the American Recovery and Reinvestment Act,
which had significant focus on public sector building retrofit funding (Satchwell et al. 2010).

Despite growing use of EPCs in the MUSH market, a relatively limited body of related literature
restricts the ability for research to inform practice in this important sector of the built
environment. Since performance contracting is essentially a mechanism to transfer energy
efficiency performance risks from the owner to the ESCO, project-level risks specific to ESCOs
must be better-understood. This need has been noted by Hansen (2006), and confirmed by
Hansen (2013), in providing a list of broad risks that ESCOs should be particularly attune to;
however, this work does not provide an evaluative framework for the assessment and evaluation
of risk in EPCs. A small body of literature identifies other risks in performing EPCs, but like the
previous work, the majority of the effort is directed to risk identification. A comprehensive risk
assessment framework for ESCOs is required to fully understand and implement EPC,
particularly in emerging markets and sub-markets, such as correctional facilities.

The confluence of the increased use of EPCs in the MUSH market with the need to better study
the management of ESCO risk leads to the following four premises which guide the need for this
research.

18

1.2.1 Premise #1 – Construction and Energy Efficient Retrofits Carry Risks for
Contractors
During the life cycle of an EPC retrofit, traditional construction project-related risks combine
with the unique risks borne by the ESCO in achieving the performance guarantee to create a
unique situation that merits further study. As a result of this risk convergence, the construction,
technical, and performance risks are transferred from the owner to the ESCO in an EPC,
therefore making this arrangement inherently risky for the ESCO.

Construction is an inherently risky enterprise due to the unique nature of projects as well as their
size and complexity, and numerous contractual and management interactions among diverse
stakeholders, which all occur in an environment with distinct political, economic, and social
factors (Zavadskas et al. 2010; Banaitiene and Banaitis 2012). As a result, project risks have
been well-studied in the literature, at the level of overall project risks (Tah and Carr 2000; Tah
and Carr 2001; Zavadskas et al. 2010; Banaitiene and Banaitis 2012; Goh and Abdul-Rahman
2013) and at the level of individual project phases, elements, and critical success factors such as
the project planning phase (Diab 2012), time and cost (Doloi 2012), safety (Hallowell and
Gambatese 2010), and delay (Assaf and Al-Hejji 2006). Willis (2008) suggested that a specific
risk assessment method is needed for the specification of ECMs, resulting from the identification
of twelve emerging technologies that did not perform as expected, and in many cases actually
consumed more energy and caused damage to other equipment in linked systems. This assertion
is of direct concern to ESCOs, as ECM failure potentially means that the performance guarantee
will not be obtained. As a result, an understanding of contractor risk in the EPC process is
warranted.

19

Hansen (2006) proposed a framework for identifying and mitigating the unique risks in EPC
projects. While not explicitly defined elsewhere in the literature, several studies support and
expand upon various elements of this proposed framework, which includes ten risk categories for
ESCOs to consider when undertaking EPCs (eight from Hansen 2006; two additional from the
literature that were not identified by Hansen). Since the primary risk category for ESCOs
undertaking EPCs is failing to achieve guaranteed performance, this framework can be
considered a first step toward identifying critical risk categories for ESCOs in this regard. These
risks are discussed in greater detail in Chapter 2, but are identified briefly, as follows:


Customer Pre-Qualification – includes concerns related to the building owner, buildingspecific features, and possible future building reuse and/or repurposing (AEPCA 2000;
Walker and Dominick 2000; Bertoldi and Rezessy 2005; Shonder and Hughes 2005;
Hansen 2006; Mills et al 2006);



Project Development (Hansen 2006);



Energy Audit Quality (Mozzo 2001; Hansen 2006; Mills et al 2006; Sankey 2007);



Equipment Selection and Installation – includes differential risks among passive (e.g.,
building envelope) and active (e.g., mechanical system, lighting) retrofit strategies.
(Shonder and Hughes 2005; Hansen 2006; Mills et al 2006; Shang et al. 2008; Wang and
Chen 2008; Willis 2008; Jinrong and Enyi 2011);



Commissioning (Stum 2000; Hansen 2006; Sankey 2007);



Operations and Maintenance Practices (AEPCA 2000; Hansen 2006; Mills et al 2006);



Measurement and Verification of Savings – includes the behavioral and systems
implications of the “rebound effect” (Hertwich 2005; Herring and Roy 2007; Strand
2011) and risks related to external conditions, such as variations in weather during the

20

performance period (Schweitzer et al. 2000; Hansen 2006; Mills et al 2006; Mozzo 2006;
Shang et al. 2008; Larsen et al 2012a);


Project Management Over the Project Life Cycle (Hansen 2006);



Construction-Specific Concerns (Smith and Ferber 1996; AEPCA 2000; Sankey 2007;
Silberman 2010); and



Volatility of Energy Prices (Bertoldi and Rezessy 2005; Shonder and Hughes 2005; Mills
et al 2006; Shang et al. 2008; Wang and Chen 2008; Berghorn 2012).

1.2.2 Premise #2 – There is a Need for Energy Efficient Retrofits in the MUSH Market
Sixty three percent of ESCO revenue is derived from MUSH market projects, and another 22%
of ESCO revenue has been attributable to federal government projects (Stuart et al. 2013).
Several factors make this market particularly attractive to EPC-driven work, including high rates
of owner occupancy, which eliminates the split incentive problem; the relatively low financial
risk of MUSH market clients; legislation-driven energy efficiency mandates that increase
motivation to perform such work; and constrained capital budgets that often demand alternative
financing strategies to fund retrofits (McCabe 2011). While ESCOs have been active in the
MUSH market for over two decades, there is significant remaining market potential for energy
efficiency retrofit. An analysis conducted by the Lawrence Berkeley National Laboratory in
2010 estimated that unmet energy efficiency retrofit opportunities in larger MUSH market
facilities could yield annual energy savings of 160 million MMBTU and lifetime savings of 2.4
billion MMBTU, which would require approximately $35 billion in additional ESCO investment
(Satchwell et al. 2010).

21

The primary motivation for the use of EPC retrofits in the MUSH market is the lack of capital
financing, which often manifests as a challenge to address a broader range of capital investment
needs beyond energy efficiency (Hopper et al. 2004). This is seen by the typical inclusion of
more lighting retrofits and non-energy improvements (e.g., building envelope, environmental
remediation) in MUSH market projects than what is typically found in federal markets (Hopper
et al. 2004). The authors further identified important barriers to increased EPC implementation in
the MUSH sector. The primary barrier identified was the lack of enabling legislation in certain
states, particularly that which is well-designed or strongly supported. Legislatively-limited
contract lengths were noted as a barrier to fully realizing the opportunities available through
EPCs by limiting technical work packages to only those with relatively faster paybacks. Other
barriers included poor project performance history, which has led some states to avoid exploring
or expanding EPC work, the need to educate customers, and a sense that “the low-hanging fruit
is already picked” – meaning that the least-risky and easiest to implement retrofit strategies have
already been implemented (Hopper et al. 2004).

Energy consumption and efficiency potential in the MUSH market is difficult to understand due
to limitations in available datasets. For example, CBECS data is categorized by building type,
ownership, and occupant type, potentially leaving a gap in the knowledge of whether an office
building would be considered part of the MUSH market or not. This is an important distinction
because MUSH market buildings are often the most energy-intensive among similar structures
because they are typically older and they are generally significant energy users owing to the
unique and mission-specific function of many of these facilities. Water utilities and wastewater
treatment facilities account for up to 33 percent of municipal energy use, and the healthcare

22

sector operates continuously (e.g., around the clock) and abides by strict health and safety
requirements which can require specialized air filtration and increased air flow and exchange
rates (Irwin et al. 2011).

Irwin et al. (2011) identified over 137,000 MUSH organizations in the United States. The
CBECS 2003 survey identified 386,000 education buildings, 129,000 healthcare buildings, and
71,000 public order and safety buildings (U.S. DOE 2011b). The survey also identified 1.77
million buildings in the office, public assembly, warehouse, and other types. Applying the U.S.
Department of Energy estimate that 24% of commercial floor space is under government
ownership or management (Irwin et al. 2011), there are an estimated 426,480 office, public
assembly, warehouse, and other building types under government control. When added to the
number of education, healthcare, and public order and safety buildings, the MUSH market can be
assumed to comprise 1.01 million buildings. Subjecting square footages from the CBECS survey
to a similar analysis yields an estimated 20.8 billion square feet of floor area exists in the
national MUSH market (32.2% of the national commercial building stock floor area). This
square footage has been estimated to use between 2.08 quadrillion (analysis of CBECS data) and
3.87 quadrillion (Irwin et al. 2011) BTU per year, which costs between $21.9 and $40.7 billion
annually.

Savings from energy efficiency retrofits executed under performance contracts in the MUSH
sector have historically been approximately 20 percent of the utility bill baseline (Hopper et al.
2005). The MUSH market could realize annual energy savings of between $4.4 and $8.1 billion
if the entire building inventory was retrofitted using performance contracting to achieve 20

23

percent savings, thereby highlighting the significant potential in this market that might be
realized through improved understanding of EPCs.

1.2.3 Premise #3 – An Energy Efficiency Gap Exists in Correctional Facilities
The United States has more correctional facilities today than at any other time in history. The
number of prisons nearly doubled between 1974 and the turn of the 21st Century (Lawrence and
Travis 2004), and there are currently 1,292 state, federal, and private prisons in the United States
(Stephan 2008), with the vast majority built before 1961 (Baker and Forbes 1997). This aging
infrastructure requires a management decision to either retrofit existing facilities or undertake
new construction. After significant growth throughout the mid-1900s, new construction of
prisons began to slow down in the late 1990s, with limited “hot-spots” of new construction in
just a few states; however, the number of retrofit projects increased 35% between 1995 and 1997
(Dallao 1997).

The nationwide incarceration rate increased during the same time period (1980s to present),
leading to overcrowded facilities and added pressure on facility infrastructure (GAO 2012).
Negative impacts on infrastructure such as increased water and electricity consumption,
increased wear on plumbing and food service equipment, and increased needs for general facility
maintenance have been reported. The U.S. Bureau of Prisons (BOP) reported a 15% increase in
maintenance costs between 2006 and 2011 and a 37% increase in electricity costs, across all ages
of facilities, to include even those most recently constructed (GAO 2012). This has had a
negative impact on total operating budgets for state and federal correctional agencies, which
typically direct a significant portion of their budgets to supervision activities. As of February

24

2012, the BOP reported 150 unfunded major repair projects at a total cost of approximately $346
million – some of which have implications for energy efficiency, such as roof repairs (GAO
2012).

The last several years have seen reductions in public corrections agency budgets which have
significantly outpaced relatively small reductions in incarceration rates (Scott-Hayward 2009;
Glaze and Parks 2012). Some states have experimented with early release initiatives as a costsavings measure; however, these programs are often fraught with social, economic, and political
challenges. This environment has created challenges for funding new capital projects, including
those projects designed to improve aging infrastructure and save operational costs. Nationally,
state correctional expenses for capital projects decreased from a 28 year high of 13.6% of the
aggregate overall corrections budget in 1986 to a low of 2.7% in 2010 (the last year of the study
period) (Kyckelhahn 2012).

The above-referenced conditions have created a “perfect storm” for the retrofit of correctional
facilities. Aging facilities, higher occupancy rates, deteriorating infrastructure, reduced capital
budgets, and increased utility consumption along with attendant utility rate increases mean that
retrofit projects can provide significant benefits by reducing operational costs, reducing end-use
utility intensities, and maximizing the overall facility benefit of such projects conducted under
constrained capital budgets. Limited research has been conducted related to correctional facility
retrofits and construction in general – as a result, a new approach is required to study this work
and develop a model for understanding and expanding work in this MUSH sub-market.

25

EPC has been used to fund retrofit projects in correctional facilities despite the presence of
budgetary restrictions by leveraging operational budgets to pay for construction and equipment
costs through annualized utility savings. The use of this financing mechanism has been growing
among correctional facility retrofit projects, and they have demonstrated the ability to return or
exceed the guaranteed savings amount among recent projects (Berghorn and Vallad 2013). At the
same time, EPC implementation has been growing across the U.S. construction market
(Satchwell et al. 2010). Despite these favorable conditions, energy intensity reduction efforts
among corrections departments often lag behind those of other public agencies.

The State of Connecticut undertook a benchmarking program for 110 of its state facilities
between 2005 and 2008, utilizing the Energy Star Portfolio Manager (U.S. Environmental
Protection Agency 2014). Twenty one percent of the benchmarked buildings had a benchmark
score of 75 or above, thereby making them eligible for Energy Star recognition – 15% of these
buildings scored in the next lowest quartile (scores of 50-74). One hundred percent of the 15
Department of Corrections buildings included in this program fell at or below scores of 49, with
nearly 87% of their facilities scoring 24 or less (Commission on Enhancing Agency Outcomes
2010). This trend in correctional facilities has been seen in other states as well. Correctional
facilities in California reduced their energy intensities between 2003 and 2010; however, their
net reduction of 2.0% lagged behind the statewide facilities average of 6.3% (California
Department of General Services 2011). A similar trend has been observed in Wisconsin.
Between 2005 and 2010, state facilities experienced a 9.8% decrease in energy intensity, while
the Department of Corrections reduced energy intensity by 3%, when adjusted for weather
(Wisconsin Department of Administration 2011).

26

The Connecticut data illustrates that correctional facilities may inherently consume more energy
per square foot than many other state-owned building types, which have been addressed with
some level of success by energy efficiency retrofits. As the data show, while correctional
facilities have undertaken retrofit projects, they have not led to the same level of savings as those
realized elsewhere in the public sector. This gives rise to the notion of an “energy efficiency
gap” within such facilities. Several reasons have been given for this gap, to include additional
pressure on limited capital project funding, complexities in project execution, lack of support for
data requests and sub-metering, and technical scope considerations and conflicting objectives.

Four possible reasons for this observed energy efficiency gap are: (1) additional pressure on
limited capital project funding, (2) complexities in project execution, (3) lack of support for data
requests and sub-metering, and (4) technical scope considerations and conflicting objectives.
These reasons are discussed below, and explain unique aspects of correctional facilities that may
contribute to the perceived energy efficiency gap in such facilities.

1.2.3.1 Additional Pressure on Limited Capital Project Funding
Reductions in public budgets have had the effect of limiting the ability to finance capital
projects. While this is likely a constant condition across multiple agencies, corrections agencies
face additional pressure from rising costs due to an increased number of incarcerated persons and
the attendant operational and utility costs of such increases in occupancy (GAO 2012).

Additionally, higher facility occupancy reduces the serviceable life of infrastructure and leads to
increased pressure on maintenance budgets. This combination of factors has led public entities to

27

seek alternative project financing and delivery models, such as EPCs (Commission on Enhancing
Agency Outcomes 2010; Wisconsin Department of Administration 2011).

1.2.3.2 Complexities in Project Execution
The use of effective construction management practices has been cited as one way that ESCOs
can ensure successful EPC performance (McQuade and Piotrowski 1993; Hansen 2006). Among
the critical roles highlighted for the construction manager is the ability to keep the project on
schedule by minimizing delays and reducing the costs attributable to such delays. Construction
and retrofit projects taking place at operational secure facilities must place a primary emphasis
on the safety and security of the facility and general public (Ginoza et al. 2003); however,
required security activities can have a negative impact on the project schedule if not managed
properly.

An example of required security measures can be found in the Oklahoma Department of
Corrections Facility Construction Security Standards (Oklahoma Department of Corrections
2012). This standard requires daily check-in and check-out of all contractors at the facility’s
central control station; weekly or more frequent compliance inspections of equipment storage
areas, vehicles, and the construction site; and interviews of construction personnel for knowledge
of information contained in the standard. Additionally, daily tool inventories during mobilization
and demobilization are required, the worksite may require physical isolation from the rest of the
facility, and employees may be subject to regular searches. All of these factors can add
significant time to the project by reducing the number of work hours available in a scheduled
work day.

28

1.2.3.3 Lack of Support for Data Requests and Sub-Metering
Vanneste (2010) reported that funding to support data needed to plan and implement
sustainability-related capital projects was largely unavailable. This specifically limited the ability
of the Washington Department of Corrections to report energy use for each facility in the Energy
Star Portfolio Manager, which was mandated by new legislation in 2009-2010. The author also
identified that current funding levels effectively prevented the installation of submeters at facilities, thereby limiting data collection to the campus-level. This highly aggregated
data hampers efforts to analyze energy performance at the building level, and would likely need
to be considered during the procurement of an EPC project, in order to facilitate proper
measurement and verification activities. This can be a state-by-state issue; recently-passed
legislation in Michigan allows for sub-meter installation as well as data collection and reporting
activities to be included in EPCs (Cost-Effective Governmental Energy Use Act 2012).

1.2.3.4 Technical Scope Considerations and Conflicting Objectives
In addition to factors common to all construction projects (e.g., safety, quality, schedule
management), correctional facility construction projects must consider four specific issues: (1)
technology, (2) security, (3) accessibility, and (4) sustainability (Phillips and Griebel 2003).
While there are numerous potential design, engineering, and construction challenges with each of
these areas, a key concern is the interaction among these elements. Technology and security, if
considered alone, may dominate other important design elements (Phillips and Griebel 2003).
Conflicting objectives, specifically related to ECMs that pose concerns for security, can
potentially add costs and limit overall energy performance.

29

EPC projects must also consider the relationships between first costs and life cycle costs. While
this is not an attribute specific to corrections projects, it is particularly important when evaluating
design alternatives in this sector given the exceptionally high operating costs of prisons as
compared to their first costs (GAO 1991).

1.2.4 Premise #4 – Uncertainty and Risk Must be Understood and Managed
The limited literature focused on ESCO risks in EPC retrofits was briefly summarized in section
1.2.1. Figure 1-4 graphically depicts potential risk instances for ESCOs undertaking an EPC
retrofit. It is based on a prototype EPC which was procured using a single-phase approach with
separate contracts for two individual work packages: (1) technical audit and (2) energy services
performance contract (construction and M&V), with annual reviews of savings. The EPC project
flowchart is developed primarily from Hughes et al. (2003) and Energy Services Coalition
(2011). The numbered instances of ESCO risk are based on the risk categories described in
section 1.2.1. These are further expanded upon in Table 1-2, which is a companion to Figure 1-4.

Despite the knowledge represented in Figure 1-4 and Table 1-2, the interrelationships among
these risks are still relatively unknown, a comprehensive risk management framework for ESCO
risks in EPC retrofits does not exist, and specific applications of a risk framework to correctional
facility retrofits is lacking. This requires an approach that connects the sources of risk with the
decisions made under conditions of risk and uncertainty, in light of the information used by
ESCO professionals when making critical project decisions.

30

Figure 1-4. Energy Performance Contract Flowchart with Instances of Contractor’s Risks
(Based on Hughes et al. 2003 and Energy Services Coalition 2011)
31

Table 1-2. Instances of ESCO Risk During the Performance Contracting Process
# Risk Category
Risk
1 Client Selection Factors
Mandated short performance periods may limit technical
approach; ESCO may not get all necessary information from
owner; Current building uses/occupancies may not reflect
future uses/occupancies; Unknown latent conditions may exist.
2 Project Development
Overhead rates used for bidding may not be feasible once
technical approach is developed; Up-front costs may be
difficult to recover once technical approach is developed.
3 Energy Audit Quality
The IGA must include a thorough risk assessment; Improperlyestablished or disputed baseline can give rise to conflict and
result in improper M&V.
4 Project Development
Negotiated terms may impact results of technical analysis and
require re-work to accurately reflect new conditions.
5 ECM Selection and
Individual ECM performance must be considered as a part of
Installation
overall portfolio performance; Allowable non-energy benefits
may be difficult to quantify; Lack of risk-based LCCA may
limit robustness of findings.
6 ECM Selection and
Additional costs incurred to renogetiate ECM portfolio;
Installation
Contract may be terminated after the IGA if it reveals that
desired performance levels cannot be obtained.
7 ECM Selection and
Uncertainty in economic parameters (energy cost escalation,
Installation
discount rate, inflation rate) may limit robustness of findings ;
ECMs may not perform as expected; ECM portfolio does not
reflect future building uses/occupanices.
8 Measurement &
Poorly-developed M&V plans increase technical and financial
Verifiation
risk to ESCOs; Poorly-designed M&V sampling protocols may
not properly represent performance of installed ECMSs; M&V
protocols that do not include non-energy benefits may
understate system performance; Failure to include O&M may
lead to missed savings opportunities.
9 Client Selection Factors
Client is unable to secure financing.
10 Project Management Over Hand-offs between project phases lead to information loss and
the Project Life Cycle
loss of continuity by ESCO team.
11 Construction-Specific
Schedule and cost growth resulting from delays – likely to
Concerns
impact financial analysis that the technical audit and EPC
scope of work is based on; Delays may result from latent site
conditions, site access restrictions, and re-work.
12 Commissioning
Failure to commission systems may result in unknown suboptimal system performance conditions; Failure to
commission may miss opportunities to verify performance;
Commissioning may lead to re-work if systems are not
installed properly – this may lead to schedule and cost growth.

32

Table 1-2 (cont’d)
# Risk Category
13 Construction-Specific
Concerns
14 Operations and
Maintenance Practices
15 Measurement &
Verifiation

16 Measurement &
Verifiation

Risk
Failure to deliver the project as stated in the scope of work
may lead to re-work, ultimately resulting in schedule and cost
growth.
Poorly understood responsibilities by each party if the O&M
plan is unclear; The institution may not perform O&M work to
specification; Institution self-performance or contracted O&M
may reduce potential revenue for ESCO.
Poorly-developed M&V plans and sampling protocols can
create additional risk for the ESCO (see item # 9); Failure to
include O&M in the M&V plan may lead to missed savings;
Disputes in establishing the initial baseline and procedures and
conditions for recalculating the baseline may add risk to the
ESCO; Behavioral factors lead to a “rebound effect” in the
performance of installed ECMs
Annual savings less than costs require the ESCO to cover the
difference and thus lose revenue.

Understanding decisions under uncertainty and risk requires a hybrid approach of quantitative
and qualitative approaches. Quantitative information typically comes from energy models,
system design tools (e.g., Trane Trace), and building system energy consumption data, which are
key elements of the IGA. Qualitative information is represented by the expertise of decision
makers, realized through severity assessments of risks and their associated mitigation strategies,
as well as understanding owner’s project objectives and requirements. This hybrid qualitativequantitative approach provides a more complete assessment of risks. Automated systems such as
energy models cannot by themselves replace the need for expertise in making decisions, such as
the designer’s role in optimizing energy performance, owner’s requirements, constructability,
and facility operating parameters (Abaza 2008). This information must be provided by technical
experts and decision-makers in order to make quality decisions about sustainability-related
challenges (Cash et al. 2003). These factors make knowledge-based approaches appropriate to
building energy design problems, because numerical methods do not provide designers with a

33

quick means to assess options, whereas analytical methods do not deal well with the multitude of
variables inherent in these problems (Kalogirou 2009). Therefore a method that combines both
types of information must be considered in developing an ESCO risk model.

Decisions made under conditions of uncertainty and risk can lead to poorly-performing project
outcomes over lengthy life cycles, as exist in EPC retrofits. A body of research has developed
around incorporating uncertainty analysis into LCCA (Reigle and Zaniewski 2002; Kapp and
Girmscheid 2005; Pan et al. 2012) at the model parameter level (Reigle and Zaniewski 2002;
Dell’Isola and Kirk 2003) and project level (Pan et al. 2012).

Pan et al. (2012) identified key project-level risks by life cycle phase, which underscores the
need to develop a framework to understand and ultimately analyze risks in relation to project life
cycle phases:


Design and Purchase – price fluctuations, technical innovation, quality of data regarding
the project, site investigation quality and results.



Installation – latent conditions, schedule and cost growth factors.



Facility Operation – fluctuations and changes in human costs and energy prices,
operational risks.



Maintenance – uncertainties in O&M parameters, variations between planned and
executed O&M activities.



Disposal – risks related to the approval of disposal options and the costs of inactive
facilities.

34

Based on the foregoing discussion, there is therefore a need to understand risk at the project
level. EPC projects in general have been shown to contain certain inherent risks, and MUSH
market (specifically correctional facilities) retrofit work has the potential to add to the project
risk profile. Reducing risk in EPCs is particularly important given the tremendous potential for
energy efficient retrofit represented by the MUSH market. Corrections, as a sub-market of
MUSH, may have some of the greatest potential for retrofit due to continuous facility operation,
above average energy intensities among MUSH market buildings, unique project parameters, and
financial and systems constraints on additional capital investment. This potential must be
balanced with the development of a framework to understand the factors contributing to
contractors risk when undertaking EPC projects at such facilities.

1.3 RESEARCH GOAL AND OBJECTIVES
The main goal of this research is to understand risks related to the execution of energy efficiency
retrofits delivered via EPC in the MUSH market through the development of an integrated risk
model with focus on correctional facilities. Research objectives have been developed to support
the achievement of the research goal, and they are described below and in Figure 1-5.

1.3.1 Objective 1: Energy Performance Contract Retrofit Process
To perform an exploratory analysis of the energy performance contract retrofit process and
identify key areas of performance risk requiring in-depth analysis.

35

Figure 1-5. Research Objectives, Methods, and Outputs
1.3.2 Objective 2: Risk Framework for MUSH Market Energy Performance Contract
Retrofits
To construct a framework describing the sources of and mitigation strategies employed for
assessing key risks in energy performance contract retrofits.

1.3.3 Objective 3: Life Cycle Cost-Based Risk Analysis and Evaluation
To develop a strategy for analyzing and evaluating risks for energy performance contract
retrofits focused on managing expected costs throughout the project life cycle, and use data
collected through this strategy to develop and parameterize a risk model.
36

1.3.4 Objective 4: Pilot Application if the Life Cycle Cost-Based Risk Model
To demonstrate the applicability of the proposed life cost-based risk model through a pilot
application to a case study site.

1.4 RESEARCH METHODS
A complete discussion of the structured methods used to decompose the research goal and
objectives into logical and related research steps is provided in Chapter 3. An overview of
methods is provided in this section, with the conceptual research model depicted in Figure 1-5.

1.4.1 Objective 1: Energy Performance Contract Retrofit Process
To perform an exploratory analysis of the energy performance contract retrofit process and
identify key areas of performance risk requiring in-depth analysis.

Step 1 - Describe the Energy Performance Contract retrofit process and identify potential
risk factors for contractors: The primary focus of this objective is to compile information
about the EPC process as it is currently used in the United States with greater attention given to
the MUSH market, specifically correctional facility retrofits. Information was compiled about
the history and use of EPCs in the United States, the dominant contractual forms of EPC and
related savings models, and financial considerations related to performance periods and the
disposition of excess savings. This information was obtained from published literature (refereed
journals and EPC-related books), government reports on state and federal EPC projects,
government (primarily from the Lawrence Berkeley Laboratory) and industry reports, one
identified Master’s thesis and one identified PhD dissertation, and a review of contracts, audits,

37

and final reports prepared by ESCOs, evaluated using the content analysis procedure described
by Molenaar et al. (2009).

Additionally, a preliminary interview was conducted with an internationally-experienced EPC
professional, in order to refine the scope of the research, to obtain confirmation of some of the
identified risks, and to identify additional risks not apparent from the literature and document
review. This work served as the basis for developing the conceptual risk framework which was
used when developing questionnaires for later research objectives. That preliminary risk
framework is the deliverable resulting from this work step and is presented in Chapter 2.

Step 2 - Study risk management and investigate its application in EPC retrofit projects:
The intent of this step was to understand formal and informal risk management processes used in
construction engineering and management (CEM). In conjunction with understanding the EPC
retrofit process in Step 1, this information was used to better define risk management methods
that could be applied to this research. A comprehensive literature review of risk management
techniques deployed in CEM research related to EPC, build-operate-transfer (BOT), and publicprivate partnership (PPP) projects was conducted to identify appropriate methods for further
study. Due to the limited amount of literature directly related to EPC retrofits, research
conducted on BOT and PPP projects was examined as these project types share common features
with EPC, to include unique financing mechanisms, a partnership between the owner and
contractor, often lengthy performance periods, and contractually-guaranteed levels of
performance of the completed project. The conceptual risk framework developed in Step 1, along

38

with the literature review of risk management methods conducted in this step, resulted in the
selection of a risk management strategy for this research, as discussed in Chapters 2 and 3.

1.4.2 Objective 2: Risk Framework for MUSH Market Energy Performance Contract
Retrofits
To construct a framework describing the sources of and mitigation strategies employed for
assessing key risks in energy performance contract retrofits.

Step 3 - Develop expertise elicitation strategy for expert knowledge in EPC retrofits and
risk: Expert knowledge was a significant source of data for this research. As a result, a strategy
was needed to first identify domain experts and then to elicit such expertise from these selected
professionals. Methods for expertise elicitation were obtained through a literature review, in
particular a review of the techniques developed by Duah (2014), which was a complementary
research project conducted in the same laboratory at Michigan State University.

Step 4 - Elicit EPC retrofit expert knowledge: Using the strategy developed in Step 3, relevant
expertise was elicited from qualified EPC retrofit experts in order to yield data related to four
specific areas of focus, which divided the survey into four distinct parts: (1) risk behavior and
tolerance, (2) professional experience, (3) risk management process, and (4) life cycle cost
analysis.

Expertise was elicited through the use of a questionnaire, which collected data in the four
categories listed above. In part three of the questionnaire (risk management process) elicitation

39

was directed toward: (1) identification of risk categories and risks related to not meeting the
performance guarantee in EPC retrofit projects; (2) parameters, uncertainties, and risks inherent
in conducting LCCA as part of the EPC retrofit process; (3) a rapid assessment of the relative
importance of identified risk categories, and (4) risk mitigation strategies. Methods employed in
steps 3 and 4 are discussed in greater detail in Chapter 3.

Step 5 - Analyze expert knowledge obtained in Step 4 and develop an EPC risk framework
for ESCOs: Elicited expertise was used to refine the preliminary ESCO risk framework
developed in Step 1. Additionally, participants were asked to provide a rapid assessment of the
risk categories that are the most important in terms of their potential negative impact when
developing and executing EPC retrofits. There was an a priori assumption that many of the risks
contributing to not attaining the performance guarantee were related to the selection, installation,
and operation of ECMs. The nature of EPC retrofits bases the performance guarantee on realized
energy savings resulting from the installed ECM package. It was therefore reasonable to expect
that risk management processes would be heavily-focused on ECM selection and installation;
however, other important risk categories were also identified. This refined risk framework
(described in Chapter 4) included the following improvements to the conceptual risk framework:


Confirmation or elimination of risk categories and risks based on elicited expertise;



Addition of risk categories and risks;



Two measures of importance for each risk category in EPC retrofits; and



Identification of risk causes and mitigation strategies.

40

Step 6 - Analyze key decisions made during the EPC retrofit process: The expert knowledge
that was elicited was used to map the decisions made during the EPC retrofit process based on
the risk framework developed in Step 5. As a result, an EPC retrofit process (EPC-RP) model,
presented in Chapter 4, was developed to identify information requirements, key decisions, and
related risks throughout the life of often lengthy EPC projects.

1.4.3 Objective 3: Life Cycle Cost-Based Risk Analysis and Evaluation
To develop a strategy for analyzing and evaluating risks for energy performance contract
retrofits focused on managing expected costs throughout the project life cycle, and use data
collected through this strategy to develop and parameterize a risk model.

Step 7 - Develop a risk analysis strategy that incorporates project life cycle costs and the
probability of risk scenario occurrence: Based on the risk management method selected in
Step 2, complementary methods were sought that would permit analysis of risk-related costs and
understanding of risks throughout a project’s life cycle. A risk analysis method was sought that
could be conducted using a workshop or panel format, continuing the ability to elicit expertise
from ESCO professionals through this phase of the research.

Step 8 - Analyze risk criticality using traditional and cost-based methods: The risk analysis
strategy developed during Step 7 was deployed in this work step and permitted the researcher to
gain an in-depth understanding of the heuristics used by experts when making project decisions
under conditions of known or suspected risk. This aspect of the risk analysis approach was
important to the development of the risk model, represented by Step 10.

41

Risk analysis was focused on finding relative measures of risk criticality. A risk analysis panel
was convened for this work step, which also provided focus for the risk evaluation process in
Step 9, and risk model development and parameterization in Step 10. Additionally, traditional
and cost-based risk criticality measures were developed by the panel to allow the researcher to
determine the relationships among different risk analysis approaches.

Step 9 - Evaluate critical risks through the analysis of risk control strategies: The same risk
analysis panel from Step 8 was used to provide further evaluation of the most critical risks by
identifying relevant mitigation strategies. These strategies were evaluated for their cost and
likelihood of success in order to construct cost-benefit relationships among the analyzed risks
and the control measures identified in this work step. This information was used as an input to
Step 10. Results of the risk analysis (Step 8) and risk evaluation are provided in Chapter 5.

Step 10 - Parameterize risk framework and construct risk model: Risks and their mitigation
strategies, along with their attendant costs and likelihoods of occurrence and successful control
were mapped, along with important facility and project factors along with heuristics and
expertise used by risk panelists in order to develop a framework for the creation of a life costbased risk model. This model framework and parameterization is provided in Chapter 5.

1.4.4 Objective 4: Pilot Life Cycle Cost-Based Risk Model
To demonstrate the applicability of the developed life cost-based risk model through a pilot
application to a case study site.

42

Step 11 - Pilot-test proposed life cost-based risk model: The risk model framework developed
in Step 10 was fully developed in this step and implemented through application to a case study.
The risk tool was further refined based on results of this pilot application. The case was a
representative example of a recently-conducted correctional facility EPC retrofit and included
analysis of facility-specific characteristics, results of the IGA, the energy guarantee, and installed
ECMs and retrofit measures. The purpose of the pilot was to evaluate the applicability of the
research in evaluating significant risks faced by ESCOs when conducting EPC retrofits, through
a method that reflects the decision-making process utilized by ESCO professionals, utilizing the
case study method.

1.5 RESEARCH SCOPE AND LIMITATIONS
1.5.1 Research Scope
The scope of this research is the creation of a life cycle cost-based risk model for understanding
long-term risks associated with project-level factors. The contextual focus of this work is MUSH
market buildings, with an emphasis on correctional facilities. While the first two objectives of
the research are focused on understanding the EPC retrofit process and risk identification for the
entire project life cycle, this project does not undertake a complete risk analysis and evaluation
of all elements of ESCO risk in such projects. Instead, project phases and risk categories that
were shown to have the greatest potential impact on project performance were the focus of the
latter two objectives. Additionally, this research is premised on the notion that risks can best be
controlled during the earliest project phases, when the costs of change are lowest and the
potential impact of those changes is the greatest (Kmenta and Ishii 2001; Horsley et al. 2003;

43

Kishk et al. 2003). As a result, risks related to project execution are not considered, except in the
context of their upstream effects on risks that occur in earlier project phases.

1.5.2 Research Limitations
Limitations of this research include:


The scope of the risk identification model is limited to project failure based on failure to
achieve guaranteed performance;



This research examines risk from the perspective of the ESCO (contractor’s risk); clientonly risks are not considered; and



The risk management process used in this research was not intended to be used as a
design tool to select among multiple retrofit measures or to optimize risk and energy
performance. Instead, the focus of this research was to obtain a comprehensive
understanding of project-level risks and conduct a thorough evaluation of the most
critical risk categories by understanding cost-related impacts of risks and their associated
mitigation strategies. This information is used to inform the risk management approach
taken by ESCOs for a given design or set of design alternatives before their contract is
finalized, when changes to the work scope and delivered costs are still possible.



While acceptable for the selected method, the size of the risk analysis and evaluation
panel did not permit statistical analysis or modeling of results, thereby limiting cross-case
comparisons among individual panelists and the ability to predictively model panelist
results.

44

1.6 PROJECT OUTPUTS/RESEARCH CONTRIBUTIONS
The primary output of the research is the development of a life cycle cost-based risk model for
evaluating the most significant ESCO risks in the development and execution of EPC retrofit
projects in MUSH market buildings, particularly correctional facilities. This primary output is
supported through the development of the following intermediate outcomes:


Preliminary ESCO risk framework for MUSH building EPC retrofits;



Refined ESCO risk framework for MUSH building EPC retrofits, focused on correctional
facilities;



EPC retrofit process model with key risks;



Life cycle cost-based risk criticality and mitigation framework; and



Life cost-based risk model and pilot test results.

1.7 DISSERTATION ORGANIZATION
The dissertation is organized into seven chapters. This chapter provided an introduction to the
research, to include background on the problem, key premises underlying the research, and goals
and objectives of the work. Chapter 2 provides a review of the relevant literature in three key
areas – (1) energy performance contracting and the selection of energy conservation measures,
(2) risk in construction and energy performance contracting, and (3) project life cycle risk
analysis. The research methodology, developed in part through the literature review, is presented
in Chapter 3. The contractor’s EPC risk framework and EPC retrofit process model are presented
in Chapter 4 and the life cycle cost-based risk model is presented in Chapter 5. Implementation
of the model through a pilot application to a case is described in Chapter 6, and the research
summary, conclusions, and directions for future research are provided in Chapter 7.

45

1.8 CHAPTER SUMMARY
This chapter provided the project overview and a discussion of the need to undertake this
research. The project goal and objectives were described, along with research methods
supporting the completion of each goal. The scope of the research along with its limitations were
presented, as was a discussion of the key outputs and contributions to the body of construction
management and energy retrofit knowledge made by this project.

46

CHAPTER 2
LITERATURE REVIEW AND PRELIMINARY RISK FRAMEWORK

2.1 INTRODUCTION
This chapter presents a review of literature providing the context for this research with regard to
decision making in energy performance contract (EPC) retrofits and related concepts of risk
management in construction and engineering and the life cycle cost implications of retrofit
projects with lengthy contracts. The literature review has been organized around three broad
categories: (1) energy performance contracting, (2) risk in construction, and (3) life cycle cost
analysis (Figure 2-1). Understanding these three related categories of literature is essential to
understanding the EPC retrofit decision making process, the sources and nature of risks to energy
service companies (ESCOs), and the tools available to help identify and manage these risks.

Figure 2-1 Literature Review Categories

47

The literature from the first two categories is presented later in this chapter as a preliminary risk
framework for ESCOs undertaking EPC retrofits. The purpose of this a priori framework is to
organize the relatively small and disperse literature related to EPC risks to assist with the
development of the primary data collection instrument in Chapter 4. This also represents the first
deliverable from this research.

2.2 ENERGY PERFORMANCE CONTRACTING RETROFITS
This section reviews the history and current state of EPC retrofit projects, with a focus on the
municipalities, universities, schools, and hospitals (MUSH) market. The guaranteed savings
model is described, as it is the predominant model used in the United States and among MUSH
market projects. Finally, information needs for making retrofit decisions under the EPC
contracting method are discussed with particular attention focused on the notion of ESCO risk
and the types of information used to make risk-based decisions.

2.2.1 History and Current Status
Energy performance contracting dates back over 100 years; however, in its modern form the
concept gained significant traction globally in the mid-1980s following the OPEC oil crisis
(Hansen 2006; ICF and NAESCO 2007; Larsen et al. 2012a; Deng et al. 2014). The earliest form
originated from government mandates that required utility companies to provide energy services
to their customers – ESCOs contracted these services to the utilities. This continued through the
late 1980s when energy prices fell dramatically and the construction cost for new energy
generating technologies became more expensive (Hansen 2006; ICF and NAESCO 2007).

48

As a result of reduced energy prices, project values decreased and ESCOs were finding it more
difficult to meet their long-term project guarantees. As a result, they began to default on their
payments and their shared savings guarantees and the industry shifted away from shared savings
to guaranteed savings as a means to reduce uncertainty and risk for financiers, equipment
suppliers, and owners by decoupling performance from energy prices (Hansen 2006). Between
1982 and 1996, 27% of the EPC projects in the National Association of Energy Service
Companies (NAESCO) database utilized shared savings; that number dropped to 3% between
1996 and 2000 (Goldman et al. 2002).

These changes in contractual savings have come as the ESCO industry has experienced
significant growth. Industry revenues have grown significantly over the past 20 years, from less
than $500 million in 1990 to an estimate of over $15 billion in 2020 (Stuart et al. 2013).

The retrofit strategies employed on institutional sector EPC projects changed between 1990 and
2008. The greatest increase in retrofit measures over that time was the use of non-energy related
strategies, growing from 3% of all projects in 1990 to 24% in 2008 (Larsen et al. 2012a). The use
of distributed generation also increased over that time, from 3% to 8% of projects. Lighting only
retrofits decreased from 25% in 1990 to 3% in 2008, indicating that much of the “low hanging
fruit” was already “picked” and more comprehensive retrofits are becoming the norm.

The increasing complexity of EPC retrofits is supported by the fact that the number of retrofit
measures used on institutional EPC retrofits approximately doubled between 1990 and 2008, the
mean investment per square foot more than doubled over that time, and 60% of survey

49

respondents reported increased project installation costs (Larsen et al. 2012a). Additionally, over
50% of projects used total facility consumption data to develop project baselines; less than 12%
of projects used end-use targeted measures.

The value of annual savings per square foot increased approximately 25% between 1990 and
2008 and the simple payback period length increased approximately 50% (Larsen et al. 2012a).
However, during this same time period, the number of institutional projects with excess savings
decreased significantly. This may be the result of limited incentives for ESCOs to exceed the
savings guarantee (Larsen et al. 2012a). In an earlier study, Hopper et al. (2005) reported that
19% of 517 EPC projects examined experienced savings shortfalls; Larsen et al. (2012a)
observed that approximately 16% of the 436 EPC projects they examined experienced a savings
shortfall.

While the ESCO market and the demand for EPC retrofits have grown significantly over the past
20 years, there are still issues to be addressed as this market matures. The MUSH market
accounts for approximately 69% of EPC revenues; however, penetration of EPC projects into
this market has been limited. As described in Chapter 1, an additional $4.4 to $8.1 billion of
energy savings (at a 20% target energy efficiency rate) is possible on an annual basis from the
MUSH market alone. Additionally, further work is required to investigate the monetization and
inclusion of non-energy benefits in EPC retrofits to create additional opportunities for owners
and ESCOs alike (Jennings and Skumatz 2006; Birr and Singer 2008; Larsen et al. 2012b).

50

2.2.2 Energy Performance Contract Structure and Elements
Energy performance contracting is a project delivery and financing method that provides turnkey
service to deliver a set of energy efficiency-related upgrades to an owner, typically via a
performance guarantee issued by the ESCO, which is financed through the annual energy savings
that result from the retrofit work (ICF and NAESCO 2007; Appleman et al. 2010; Seeley 2012).
The retrofit work is essentially completed using a design-build approach (ICF and NAESCO
2007). Because the work is financed through the accrual of operational cost savings during the
length of the project performance period (Figure 2-2), owners are typically not required to
provide significant up- front capital to finance the work and instead can access operations and
maintenance funds (e.g., budgets for utility payments, equipment replacement) for project
financing.

Figure 2-2 Energy Performance Contracting Cash Flow and Savings Model
(Source: Shonder et al. 2010)

51

In the idealized EPC cash flow and savings model presented in Figure 2-2, operational costs,
such as utility bills, equipment maintenance, and equipment replacement burden 100% of
available capital before the EPC retrofit begins. During the project performance period, annual
savings from efficiency upgrades are first used to offset debt service (e.g., loan payments for
financing the project); any remaining savings are then available to the owner. Debt service
payments are completed by the end of the performance period; at that point, the owner retains all
savings obtained through the EPC retrofit work, and experiences an overall reduction in
operational costs.

2.2.2.1 Energy Performance Contracting Elements
EPC retrofits are typically conducted by ESCOs, which are defined as businesses that provide a
full-range of energy efficiency services and have performance contracting as a key element of
their energy efficiency offerings (Goldman et al. 2002; Hopper et al. 2005; ICF and NAESCO
2007; Larsen et al. 2012a; Stuart et al. 2013). The inclusion of performance contracting as a key
business element is important because the guarantee offered through EPC means that ESCOs
inherently retain risks throughout the performance contracting retrofit delivery process.

The provision of the performance guarantee is one of four key elements of EPC retrofits
recognized by the ESCO industry (ICF and NAESCO 20070). These elements include: (1)
turnkey service, (2) comprehensive retrofit measures, (3) project financing, and (4) performance
guarantee. These elements essentially define the ESCOs scope of work during the EPC retrofit
process, and provide a working definition for the comprehensive nature of the work conducted as
part of an EPC.

52

Through EPC, an ESCO typically provides a complete suite of services (i.e., turnkey service) to
include an energy audit (preliminary audit and investment grade energy audit), engineering and
architectural design, construction management, and commissioning, as well as assistance with
securing financing. Operation and maintenance (O&M) services may also be provided during the
performance period; however, this is variable depending on the project goals and complexity,
owner ability to self-perform O&M, and contractual provisions. Similarly, while measurement
and verification (M&V) is an important part of an EPC project, it may or may not be included as
part of the turnkey services provided by the ESCO. Projects that are measured and verified
through stipulation do not require comprehensive M&V. Additionally, less-comprehensive
retrofits such as lighting-only retrofits typically only develop the baseline using equipmenttargeted metrics (Larsen et al. 2012a).

Typical EPC projects include a comprehensive suite of retrofit measures, also called energy
conservation measures (ECMs) that address the owner’s project requirements (OPR). Larsen et
al. (2012a) identified 26 different categories of retrofit measures installed in private, K-12
school, and other public EPC retrofits (Figure 2-3). The most commonly installed measures
include lighting, controls, distribution and ventilation, boilers, water conservation, chillers, and
improvements to the building envelope.

ESCOs often provided funding early in the history of EPC retrofits in the United States,
primarily due to the unwillingness of financial institutions to provide capital arising from their
lack of familiarity with EPC (ICF and NAESCO 2007; Bhattacharjee et al. 2010). This evolved
through time to the current condition, whereby ESCOs no longer directly provide financing, due

53

Figure 2-3. Percentage of Projects Installing Various Retrofit Measures
(Source: Larsen et al. 2012a)
to a mature financial market that is willing to work with such projects, although ESCOs may still
provide assistance to the owner in recommending the appropriate financing vehicle (ICF and
NAESCO 2007). Public projects are typically funded through debt financing (e.g., loans) or
lease-based financing (Hansen 2006; ICF and NAESCO 2007), although bonds are also
frequently used to fund MUSH market EPC projects (Hansen 2006; CCI 2009b). Power purchase
agreements may also be used when EPC projects involve the deployment of distributed
generation or combined heat and power technologies (ICF and NAESCO 2007). Financing costs
54

increased during the economic downturn in the mid-2000s, although this did not appear to
materially deter the ESCO industry from providing EPC services during this time (Satchwell et
al. 2010).

Some form of a performance guarantee is included in many EPC projects (ICF and NAESCO
2007). When examining the NAESCO database for EPC projects between 1990 and 2008,
Larsen et al. (2012a) found that 73% of public and institutional projects utilized a performance
guarantee, whereas only 40-45% of private sector projects included such a guarantee. Among
projects with guaranteed performance, the guaranteed savings model was vastly preferred across
sectors – 92% of public and institutional projects and 71-80% of private sector projects used this
contractual form. A more complete discussion of EPC savings models is included in section
2.2.3.

2.2.2.2 Energy Performance Contract Life Cycle
The life cycle of an EPC retrofit project can be divided into five phases: (1) Project
Development, (2) Energy Audit, (3) Retrofit Design, (4) Project Execution, and (5) Energy
Savings, which are described below.

Project Development, Energy Audit, and Retrofit Design: The first three phases of the EPC
process encompass the work conducted by ESCOs with regard to the decision to bid, the
response to a request for proposals which often includes a preliminary technical audit,
completion of the investment grade audit (IGA), an analysis of potential energy savings and cash
flows, and the final development of the retrofit design (AEPCA 2000; Waste Reduction Partners

55

2008; Petersen 2009; Tetreault and Regenthal 2011). A critical element of this phase is the
completion of the IGA as it includes several pre-requisites that define the remainder of the EPC
scope of work to include development of the baseline, against which performance guarantees are
developed and expected levels of performance from the specified energy conservation measures
(ECMs) (Sankey 2007). Critical areas addressed by the IGA include (AEPCA 2000; Ganji and
Gilleland 2002; ASHRAE 2011; Baechler et al. 2011):


Identify opportunities for energy savings through detailed energy calculations, derived
from a site assessment, analysis of utility bill data, and review of current energyconsuming equipment performance data;



Fully define the scope of work to be undertaken during the project execution and project
performance phases to include M&V procedures; and



A comprehensive financial analysis that identifies retrofit implementation costs,
guaranteed levels of performance, and cash flows during the performance period of the
project.

The scope of work to be delivered and M&V plans may be impacted by system upgrades
required under the current version of the building code that is in force during the term of the
retrofit (Hansen 2006). This may require added work scope items merely to make systems codecompliant, whether or not those systems were an integral component of one or more ECMs. As
information is gathered relative to the development of the IGA, ESCOs must also analyze the
potential impacts of humans on the retrofit, to include behavioral impacts known as the “rebound
effect” (Hertwich 2005; Herring and Roy 2007; Strand 2011), the ability for maintenance
personnel to operate newly-installed systems based on the ESCO’s assumptions (Hansen and

56

Brown 2004), and potential changes in the operational profile of the building (e.g., hours of
operation, temperature and humidity set points, building use) (Baechler et al. 2011). Indoor
environmental quality issues may also be addressed during the IGA (Hansen and Brown 2004)
and a complete assessment of risks related to each ECM should also be completed (Hansen
2006).

Project Execution: Once the retrofit design is complete and ECMs have been specified, work
commences on the actual retrofit activities. While this phase of the work is similar to other
construction projects, there are unique considerations for work conducted through an EPC. The
overarching concern is that delays, procurement issues, productivity losses, equipment
substitutions, substandard performance of ECMs, and delayed commissioning can all impact the
project’s objectives and the attainment of energy performance goals (Sankey 2007; Silberman
2010). Delays on projects with statutorily-mandated contract terms will shorten the period of
time during which energy savings may be accrued; such delays almost always lead to monetary
claims against the ESCO (Silberman 2010).

System commissioning in EPC projects may need to address issues differently than in traditional
construction, to include interaction between commissioning, M&V, and required documentation
to support any included non-energy benefits (NEBs) (Stum 2000). Jennings and Skumatz (2006)
found that negative impacts to the project schedule were cited by survey participants as the
primary negative outcome of undertaking commissioning. Silberman (2010) suggested that
ECMs should be commissioned as they are installed in order to avoid potential delays in project
execution that could lead to claims against the ESCO. Stum (2000) also found that the intensity

57

of commissioning and the rigor of the M&V plan may be inversely related; however, there is a
question as to the optimal level of intensity in each component that would yield an acceptable
result.

Jennings and Skumatz (2006) found that commissioning can assist in the valuation of non-energy
benefits (NEBs) accrued through the retrofit process. The application of such benefits in EPC
projects is inconsistent due to legislative variation among jurisdictions undertaking the work
(Larsen et al. 2012b) and the lack of a standardized protocol for quantifying these benefits (Birr
and Singer 2008). Including these benefits is potentially important to project economics, as Tso
et al. (2003) found the payback period for commissioning costs could be reduced by
approximately 19% when including NEBs.

Energy Savings: Once ECMs are installed and commissioned, the project moves into the energy
savings phase. As stated above, many jurisdictions statutorily limit the overall contract length for
EPCs, so the owner benefits if the project can quickly move to the energy savings phase, where
efficiency-related operational savings are accrued. Two interrelated elements of the energy
savings phase established during the IGA are the energy consumption baseline and the M&V
plan (AEPCA 2000; Mozzo 2001; Waste Reduction Partners 2008; Petersen 2009; Tetreault and
Regenthal 2011). The establishment of the baseline is critical, as it is the basis upon which all
energy savings are based and it may be adjusted during the energy performance phase if changes
are detected in system operational parameters (FEMP 2007). The baseline may be established
through measurement and analysis of existing conditions or by stipulation; the latter option
potentially reduces risk exposure for all parties to an EPC (Mozzo 2001). Adjustments to the

58

baseline are considered to be either routine or non-routine (EVO 2012). Routine adjustments
include “energy-governing factors” that are likely to change during the energy performance
period, such as weather. Non-routine adjustments are also “energy-governing factors,” but those
considered less likely to change, such as facility size, occupant type, volume of conditioned
space, or facility operating profile (EVO 2012).

Measurement and Verification Options: The International Performance Measurement and
Verification Protocol (IPMVP) is an internationally-recognized, consensus-based M&V standard
that was first developed in the mid-1990s through consultation with hundreds of professionals
representing 12 countries (IPMVP Committee 2002). The IPMVP recognizes four options for
M&V, referred to as options A through D (EVO 2012), which are described in Table 2-1.
Options A and B establish the M&V boundary as the system components directly related to the
equipment impacted by the EPC retrofit. Option C utilizes the entire facility as the measurement
boundary in order to manage total facility energy performance. Option D is used when baseline
or reporting period data are not available and requires calibrated whole-building simulation,
using a measurement boundary at either the whole building or installed system scale.

Stipulation in Option A: Stipulation is an option that is available when the ESCO and owner
agree to hold a performance term constant, regardless of actual performance during the energy
performance period (FEMP 2002). Stipulation may be allowed in Option A if three conditions
are met: (1) measures can be reasonably estimated, (2) they are documented, and (3) they
contribute a relatively small amount of uncertainty toward the overall savings guarantee (FEMP
2002). Data used to estimate and stipulated savings may include manufacturers’ or national

59

Table 2-1. IPMVP Option Descriptions
Savings Calculation
IPMVP Option
Methods
Engineering calculation of
A. Retrofit Isolation: Key
baseline and reporting
Parameter Measurement
period energy from:
Savings are determined by field
 Short-term or
measurement of the key
continuous
performance parameter(s) which
measurements of key
define the energy use of the ECM’s operating
affected system(s) and/or the
parameter(s); and
success of the project.
 Estimated values.
Measurement frequency ranges
from short-term to continuous,
depending on the expected
variations in the measured
parameter, and the length of the
reporting period. Parameters not
selected for field measurement are
estimated.

Typical Applications
A lighting retrofit where
power draw is the key
performance parameter that is
measured periodically.
Estimate operating hours of
the lights based on facility
schedules and occupant
behavior.

Routine and non-routine
adjustments as required.

Estimates can based on historical
data, manufacturer’s specifications,
or engineering judgment.
Documentation of the source or
justification of the estimated
parameter is required. The plausible
savings error arising from
estimation rather than measurement
is evaluated.
B. Retrofit Isolation: All
Parameter
Measurement
Savings are determined by field
measurement of the energy use of
the ECM-affected system.
Measurement frequency ranges
from short-term to continuous,
depending on the expected
variations in the savings and the
length of the reporting period.

Short-term or continuous
measurements of baseline
and reporting period
energy, and/or
engineering computations
using measurements of
proxies of energy use.
Routine and non-routine
adjustments as required.

60

Application of a variable
speed drive and controls to a
motor to adjust pump flow.
Measure electric power with a
kW meter installed on the
electrical supply to the motor,
which reads the power every
minute. In the baseline period
this meter is in place for a
week to verify constant
loading. The meter is in place
throughout the reporting
period to track variations in
power use.

Table 2-1 (cont’d)
IPMVP Option
C. Whole Facility
Savings are determined by
measuring energy use at the whole
facility or sub-facility level.

Savings Calculation
Methods
Analysis of whole facility
baseline and reporting
period (utility) meter data.

Routine adjustments as
required, using techniques
such as simple
Continuous measurements of the
entire facility’s energy use are taken comparison or regression
analysis.
throughout the reporting period.

D. Calibrated Simulation
Savings are determined through
simulation of the energy use of the
whole facility, or of a sub-facility.

Non-routine adjustments
as required.
Energy use simulation,
calibrated with hourly or
monthly utility billing
data. (Energy end use
metering may be used to
help refine input data.)

Typical Applications
Multifaceted energy
management program
affecting many systems in a
facility. Measure energy use
with the gas and electric utility
meters for a twelve month
baseline period and throughout
the reporting period.

Multifaceted energy
management program
affecting many systems in a
facility but where no meter
existed in the baseline period.

Simulation routines are
demonstrated to adequately model
actual energy performance
measured in the facility.

Energy use measurements,
after installation of gas and
electric meters, are used to
calibrate a simulation.

This Option usually requires
considerable skill in calibrated
simulation.

Baseline energy use,
determined using the
calibrated simulation, is
compared to a simulation of
reporting period energy use.

(Source: EVO 2012)
association-developed performance curves, manufacturer’s specifications, standard lighting
tables, and government weather data. The use of stipulation has grown in EPCs as there is a
significant benefit to all parties (ICF and NAESCO 2007). Owners can direct more of the project
value toward capital improvements by reducing the complexity, and therefore cost, of M&V and
the ESCO can reduce their long-term performance risk (ICF and NAESCO 2007). Due to the
nature of stipulation, excess savings are not possible and therefore cannot be accrued.

61

2.2.3 Savings Models
As discussed in section 2.2.1, the use of shared savings has waned in favor of guaranteed
savings. Customers prefer guaranteed savings due to its lower financing costs (most MUSH
market entities are tax-exempt), greater certainty in savings (the ESCO is contractually-obligated
to achieve guaranteed performance), and lower transaction costs, since the ESCO is focused
solely on performance, to the exclusion of financial risks (Hopper et al. 2005). Despite the
current focus on the guaranteed savings model, this section will briefly discuss and compare both
shared savings and guaranteed savings. A summary of the key aspects of each contract type is
presented in Table 2-2.
Table 2-2. Comparison of EPC Contract Types
Finance
Performance
Balance
Project-Specific
Contract Type
Risk
Risk
Sheet
Financing
Shared Savings
ESCO
ESCO
ESCO
Yes
a
Guaranteed Savings Customer
ESCO
Customer
Yes
Notes:
a\ ESCO bears finance risk if guaranteed savings is ESCO-financed
(Sources: Hopper et al. 2005; IFC 2011)
2.2.3.1 Shared Savings
The shared savings model of EPC requires that the ESCO carry the credit risk of the customer in
addition to the technical performance risk of the project (Hansen 2006; ICF and NAESCO 2007;
IFC 2011; Larsen et al. 2012a). The ESCO also bears the risk of energy rate increases that
exceed the agreed-upon escalation factor in the contract (ICF and NAESCO 2007). If there are
no energy savings, the facility owner pays the utility bill as usual; while the ESCO does not
receive any payments, they also do not owe anything to the owner (ICF and NAESCO 2007). In
the event that the project does not result in energy savings, the ESCO is required to pay for the
up-front project costs (equipment is owned by the ESCO until the project is turned over at the
62

end of the performance period); however, if the project does achieve net savings, they are
divided with the owner based on agreed-upon terms. While this contractual form has declined in
use in the U.S., Da-li (2009) reports that it is still preferred in the building sector in China, and
Okay and Akman (2010) report its preference in the developing world, since owners limit their
financial risk and can keep these projects off their balance sheets. This latter situation is
particularly important as it enables an owner to minimize exposure of their credit capacity, which
can be a benefit in developing economies which typically have less-robust financial institutions
(IFC 2011).

2.2.3.2 Guaranteed Savings
As stated earlier in this section, the U.S. ESCO industry has moved nearly exclusively to
guaranteed savings for EPC retrofits. This contractual form has a decidedly different risk profile
compared to shared savings, and releases the ESCO from financial risk (Table 2-2); however, the
performance guarantee must ensure that a wider range of costs are recoverable, to include debt
service, M&V fees to the ESCO, and any maintenance obligations or other incremental costs
stipulated by the contract (IFC 2011). The main advantages of this method, compared to shared
savings for owners, are that public or non-profit customers can secure financing at lower rates by
carrying the debt on their own balance sheet, more comprehensive projects are possible because
the ESCO is less highly-leveraged through not financing the work, and the amount of energy
saving is guaranteed. If there is a shortfall in the guaranteed amount of energy savings, the ESCO
will pay the shortfall amount to the owner using previously-agreed upon utility rates and
escalation factors; in this way, the ESCO does not guarantee cost savings, rather a quantity of
saved energy (CCI 2009a).

63

The ESCO benefits by reducing their risk profile through elimination of carrying owner credit
risk and by being able to assemble larger, more complex projects which potentially carry more
value (European Commission 2014). Additionally, the ESCO is generally paid up-front for the
construction costs of the project via the owner’s financing, and ongoing payments are directed
toward ongoing costs (e.g., O&M and M&V costs, if included in the contract) (Hopper et al.
2005).

Establishing the appropriate level of guarantee is a subject of much debate, but little research has
been conducted on this subject. Goldman et al. (2002) examined the difference between ESCOguaranteed savings to customers and the ESCO’s own predicted savings estimates for 15
companies. Seven companies reported that they guarantee 100% of predicted savings, while six
reported guarantees between 50% and 100% of predicted savings. Two companies guaranteed
less than 50% of predicted savings. Establishing the appropriate guarantee can be a risky
proposition, especially since Goldman et al. (2002) found that 30% of the 369 projects examined
had actual savings that were less than predicted savings. Hopper et al. (2005) reviewed 534 EPC
projects and found that 34% had shortfalls between predicted savings and actual savings; 57% of
these projects had shortfalls in excess of 10%. Only 12% of projects used stipulation in the
Hopper et al. (2005) study; 14% of projects used stipulation in the Goldman et al. (2002) study.

2.2.4 Information Categories for EPC Retrofits
The EPC process relies on both quantitative and qualitative sources of data when developing the
project technical requirements. The IGA is the manifestation of the information needed for a
successful EPC project, and can be divided into two primary categories – energy accounting

64

information and O&M related information (Hansen 2002). The quality of information gathered
and reported throughout the life cycle of an EPC project is a major determinant of that project’s
success (Allen et al. 2006). The specific types of information categorized thusly include:


Monthly utility bill data:
o Electricity – usage (kWh), demand (kW and kVA), power factor penalties (if
assessed), and the total bill (dollars);
o Other fuels – billing period, consumption units (e.g., CCF, gallons), cost per unit,
and the total bill (dollars); and
o Energy use normed to BTUs or therms to permit cross-fuel source comparisons;



Energy consumption per unit of product (for industrial facilities);



Facility information:
o Conditioned gross square footage of building(s);
o Operational profile, end-use(s), occupant types, building ages, age(s) of previous
retrofits; and
o Known locations of hazardous materials (e.g., asbestos) and code non-compliant
infrastructure;



Current and estimated future energy prices;



Technical requirements of and eligibility for utility rebate, incentive, and grant programs;



Emissions reductions resulting from energy efficiency;



Benchmark data based on EPC retrofit goals;



Calculated costs based on existing data collection:
o Cost of delay (CoD), where

65

CoD= -(energy cost savings for period + O&M savings for period) + initial investment
prorated; and
o Cost avoidance (Ca), normalized to changed conditions (e.g., increased
conditioned gross floor area), where
Ca = (baseline cost X current energy cost) – current costs;


O&M costs (e.g., training, staff time on O&M-related activities);



O&M service log (e.g., frequency and severity of required O&M activities);



Inventory of all energy-consuming equipment (location, use, operational profile,
descriptive information, nameplate information);



Equipment replacement and upgrade costs; and



New equipment costs (purchase, O&M, and replacement), energy consumption, and
operating parameters.

The IGA also requires information related to the owner’s project requirements (OPR), relevant
statutory requirements or limitations, special operating parameters for the facility (e.g., the
presence of medically-fragile patients), specific site access and security requirements, climate,
and building conditions.

The various types of information needed for the retrofit process (as listed above) can be
described as being quantitative or qualitative (Samuel 2010; Duah 2014). Quantitative
information generally includes published standards and literature that is agreed upon by domain
experts (Duah 2014). This may include published information about retrofit technologies, cost
information from published databases (Duah 2014), equipment operating cost databases (such as

66

the ASHRAE Service Life and Maintenance Cost Database) (Abramson et al. 2005), energy
simulation analyses from simulation software (e.g., DOE2, eQuest, TRNSYS), system design
tools (e.g., Trane TRACE, Carrier HAP), building information modeling results, design guides
(e.g., ASHRAE 2012), information available from corporate databases on previous projects, and
company-wide planning and risk mitigation checklists (such as implemented by Honeywell,
Johnson Controls, and McKinstry, among others).

Qualitative information is generally considered to be expert knowledge, and consists of
experience-based judgment possessed by domain professionals (Duah 2014). In the case of EPC
retrofits, qualitative information may include heuristics employed when making project-level
decisions (e.g., the decision to bid, risk-based decision making with regard to retrofit measures)
and the incorporation of domain-level expertise in the development and execution of EPC
retrofits. The first information communicated to the ESCO is typically qualitative and is directed
by the OPR and the experience of the ESCO. This is used to provide an initial response to those
requirements, often without the ability to compile large amounts of information or perform a
preliminary audit.

2.2.4.1 Sources of Quantitative Information
The following section appraises various sources of quantitative information for non-residential
energy efficiency retrofits. A variety of quantitative information sources and models developed
by the U.S. DOE and third-party organizations are available for the non-residential market.

67

State energy efficiency databases provide technical and financial information about installed
technologies. Examples of state energy efficiency databases include the California Database for
Energy Efficient Resources (DEER) (California Public Utilities Commission 2014) and the
Michigan Energy Measures Database (MEMD) (Michigan Public Service Commission 2014),
which includes data for residential and non-residential applications. Both databases include
information on various energy savings technologies and measures to include energy savings
potential estimates, typical measures installed in the marketplace, and cost and benefit data of
more energy-efficient measures.

The U.S. DOE Commercial Buildings Portal (Figure 2-4) provides access to a large number of
U.S. DOE research products and databases (some of which are described below), including:


Advanced Energy Design Guides and Advanced Energy Retrofit Guides



Buildings Performance Database



Commercial Buildings Resource Database



Energy Modeling Software

Both ASHRAE and the U.S. DOE have created energy efficient design guides for a number of
building types, to include office buildings, schools, and hospitals (ASHRAE 2012; U.S.
Department of Energy 2013b). The ASHRAE guides provide information and tools to support
50% energy savings over ASHRAE Standard 90.1-2004, and 30% savings over ASHRAE
Standard 90.1-1999 through their Energy Efficient Design Guides (AEDG). The U.S. DOE
Advanced Energy Retrofit Guides (AERG) do not use a benchmarked standard; instead, the
guide offers flexibility to meet individual building retrofit needs and uses a pre-1980 reference

68

Figure 2-4. U.S. Department of Energy Commercial Buildings Portal Screenshot
(Source: U.S. Department of Energy 2014a)
building that was previously developed by the U.S. DOE. Both guides provide information
related to retrofit measures, ECMs, and building strategies to achieve stated energy performance
goals; however, the U.S. DOE guides are formatted in line with the phases of an EPC retrofit,
and offer technical guidance regarding commissioning, ECMs, M&V, and O&M, in addition to
cost considerations for recommended retrofit packages (U.S. Department of Energy 2013b).

The U.S. DOE publishes two buildings databases of interest for this research – the Buildings
Performance Database (BPD) and the High Performance Buildings Database. The purpose of
both products is to disseminate information about high performance buildings, and in the case of
the BPD, to permit users to perform statistical analyses on an anonymous dataset comprising tens
of thousands of actual buildings in the U.S. (U.S. Department of Energy 2014b).

69

The U.S. DOE provides a portal containing information about commonly-used and widely
available energy models, including EnergyPlus, OpenStudio, RADIANCE, and DOE-2, in
addition to a comprehensive database of over 300 other models for whole building analysis;
codes and standards; materials, components, equipment, and systems; energy economics; and
indoor environmental quality, among other subjects (U.S. Department of Energy 2014c).

The purpose of the ASHRAE Service Life and Maintenance Cost Database is to help building
managers make better-informed decisions about life cycle costs related to equipment O&M and
replacement, and to provide engineers and facility managers with the ability to “distinguish
between low price and best value in the selection of HVAC [heating, ventilation, and air
conditioning] systems” (Abramson et al. 2005). The database currently features an internet-based
interface (ASHRAE 2014), depicted in Figure 2-5, which provides information related to
evaluating equipment service life. Mechanical equipment service life data is currently available
for 40,000 pieces of equipment. There are 155 different types of HVAC equipment and systems
included in the database, including:


Air Distribution



Cooling



Heat Rejection



Cooling Pump



Heating



Heating Pump



Control



Miscellaneous
70

Figure 2-5. ASHRAE Service Life and Maintenance Cost Database Web Interface
Screenshot
(Source: ASHRAE 2014)
2.2.4.2 Sources of Expert Knowledge
Expert knowledge used in this domain is largely derived from expertise accumulated over a
period of time performing related work. Duah (2014) provided a complete treatise on the
theoretical framework for expertise development, and he noted that the U.S. DOE and Building
Performance Institute collaborated on developing guidelines and certification standards for
professionals involved in various aspects of residential energy efficiency. Such standards do not
currently exist in the non-residential market; however, the U.S. DOE is currently developing a
series of workforce development guidelines for jobs in the commercial and industrial buildings
sector (U.S. Department of Energy 2013c). Without such industry-accepted guidelines, there is
little consistency in education for EPC retrofit professionals nor is there an accepted body of
knowledge that would result in a framework for describing domain-level expertise. A review of
project documents revealed a heavy degree of reliance on licensed Professional Engineers and
71

Certified Energy Managers; however, those bodies of knowledge are relatively broad and these
are not industry-required credentials for EPC experts. Further challenging this issue is the critical
role of risk management in EPC projects. Retrofit professionals must rely on their expertise
when making risk-based decisions, even when completing corporate risk guidelines, but the lack
of a codified body of knowledge may hamper this decision making.

2.3 RISK IN CONSTRUCTION
All human enterprises contain elements of risk including the construction industry (Dey and
Ogunlana 2004). Construction projects are often beset with risks which in turn lead to poor
performance throughout the project life cycle (Tah and Carr 2000; Zavadskas et al. 2010;
Banaitiene and Banaitis 2012). Despite this, the literature has only placed the concept of
construction risk management within the context of decision-making and management science
theories since the 1960s (Edwards and Bowen 1998).

This section of the literature review provides an overview of how risk in construction is
addressed. The section begins with a presentation of key definitions of uncertainty, risk, and risk
management, then addresses foundational theories of decision-making under risk and the
elicitation of judgment from experts. Sources of risk in construction projects and energy
performance contracting are discussed, which leads to a discussion of risk management process
applied to construction projects.

2.3.1 Key Concepts
Three key concepts must be defined before undertaking a review of the literature focused on risk

72

in the built environment and construction projects:


Uncertainty may be defined as a lack of knowledge about current conditions or some future
state. In examining uncertainty, one must distinguish between random uncertainty and
knowledge-based uncertainty (Ang and De Leon 2005). Aleatory uncertainty (that due to
randomness) cannot be systematically addressed and reduced; however, epistemic
uncertainty (that arising from gaps in knowledge) can be reduced through increased
knowledge and/or models that can address this gap. Tserng et al. (2009) demonstrated the
importance of extracting process knowledge in the effective management of project risks in
order to address this knowledge gap.



Risk is defined in a number of ways, depending on its context. Webster’s dictionary defines
risk as ‘‘possibility of loss or injury.’’ Construction researchers have re-stated that definition
as a quantifiable standard, “a measure of the probability, severity, and exposure of all the
hazards of an activity” (Jannadi and Almishari 2003). This definition is adopted for this
work. The probability of a hazard occurring alludes to uncertainty, as defined above. Hertz
and Thomas (1983) defined the relationship between risk and uncertainty such that highly
uncertain outcomes cannot be determined to be risky unless they also carry a potential for
damage or loss. Kmenta and Ishii (2004) restated damage as the potential consequence of
uncertainty. This can therefore be represented conceptually as:
Risk = Uncertainty x Consequence



Risk management (RM) is the systematic process of applying management science to address
risks in projects (Edwards and Bowen 1998; Arashpour and Arashpour 2012). The RM
process consists of risk identification (and classification), risk analysis and evaluation, risk
mitigation/allocation, and decision influencing, and risk monitoring (Edwards and Bowen

73

1998; Ahmed et al. 2007; Liu et al. 2010; Arashpour and Arashpour 2012; Banaitiene and
Banaitis 2012) framed within an appropriate project context to achieve management goals
throughout the project lifecycle (Ward and Chapman 1995).

2.3.2 Decision Making Under Risk and Uncertainty
2.3.2.1 Bounded Rationality
Economic and psychological models of decision-making traditionally held that individuals
behaved rationally, until the late 1950s when it is believed that Herbert Simon introduced the
concept of “bounded rationality” (Russell and Thaler 1985), which holds that individuals’ ability
to act rationally is limited by information availability, cognitive abilities, and limited time to
make decisions. Research between the 1970s and 2000s further explored this concept (Russell
and Thaler 1985; Kachelmeier and Shehata 1992; Fehr and Tyran 2001; Tse and Love 2001) and
studied the conditions under which such behavior takes place. Tversky and Kahneman’s (1981)
seminal empirical study demonstrated that people make different decisions about the same
decision problem when it is presented differently, thereby lending support to the notion of
bounded rationality. This “framing problem” arises when the problem’s form has a greater
influence on the decision maker than does the substance of the problem under analysis. This has
been shown to be a systematic problem, in that specific decisions can be predicted by knowing
how the problem has been framed, which has been termed as quasi-rational behavior (Russell
and Thaler 1985).

Tversky and Kahneman (1986) demonstrated that this framing problem can lead to violations of
the two widely-supported axioms of expected utility theory that define rational behavior –

74

dominance and invariance. The dominance axiom states that a decision option that is preferred in
one context and at least as good in other contexts is the preferred option. The invariance axiom
represents the inverse of the framing problem and states that the preferred decision option should
be the same regardless of the way the problem is represented. They noted that McNeil et al.
(1982) observed an increased preference for significant medical procedures when immediate
risks were reduced despite greater long-term risks of mortality. Furthermore, they noted that this
framing effect risk preference was the same for experienced physicians, business students with
significant statistical knowledge, or clinic patients.

These examples are illustrative of Kahneman and Tversky’s (1979) Prospect Theory (PT), which
holds that people make decisions based on potential losses and gains as compared to a reference
point, rather than based on an end-state condition. The theory is based on individual decisionmaker behaviors that are loss averse – in other words, losses hurt more than gains feel good. In
PT, the individual employs heuristics as their primary decision-making tool. While PT relies on
choosing among outcomes with known probabilities, Cumulative Prospect Theory (CPT) offers
the ability to provide decision weights, which can vary for gains and losses (Tversky and
Kahneman 1992). CPT demonstrates a “fourfold pattern of risk attitudes: risk aversion for gains
and risk seeking for losses of high probability; risk seeking for gains and risk aversion for losses
of low probability” (Tversky and Kahneman 1992).

As a result of previous work on rationality, quasi-rationality, PT/CPT and psychological
considerations (e.g., framing) of decision-making under risk, it is important to understand the
context of a decision problem in order to examine individual behavior under risk. Berny and

75

Townsend (1993) recognized that effective risk analysis in technical projects relies on expert
knowledge of the systems under risk. These experts tend to be risk-averse (Berny and Townsend
1993; Raftery et al. 2001), which may lead to unintentionally ignoring specific risks in a
comprehensive analysis since they tend to rely on previous experience rather than to analyze
risks with unknown likelihoods of occurrence (Wirba et al. 1996). Furthermore, construction
professionals have been shown to maintain identical risk attitudes before and after a “turning
point” event on a project and displayed more loss aversion when smaller amounts of money are
at risk (Raftery et al. 2001) than what would be predicted by PT/CPT(Tversky and Kahneman
1992).

When analyzing risks, McKim (1992) found that contractors tended to assign discrete
percentages to the likelihood of a project risk occurring as opposed to developing probability
functions, they tended to believe that risks could not happen to them, and that they largely
ignored contractual risks. As a result, McKim (1992) recommended that risk analysis for
construction projects should favor qualitative methods over statistical techniques. Despite these
individual aspects of decision-making under risk by construction professionals, Birnie (1993)
found that estimators subject to his study held the same judgmental biases and used the same
heuristics in their decision-making process as the population at large. Raftery (1994) suggested
that construction professionals could benefit from techniques that make them more aware of their
decision-making process since they do not possess significantly better decision-making abilities
or different risk attitudes than the general public.

76

2.3.2.2 Cognitive Biases and Energy Efficiency
Klotz (2011) investigated the role of cognitive biases during the early stages of energy efficiency
project delivery, when decisions made can have the greatest impact on project performance at the
least cost. His assertion is that understanding potential cognitive biases is important to
understanding the full scope of why buildings use more energy per square foot today than ever
before, despite technical advancements and proven cost-effective technologies. The literature
review conducted as part of the research identified six potential cognitive biases and placed them
in the context of the project phase where they may occur with an example of how they could
impact energy efficiency performance (Table 2-3).

Table 2-3. Potential Cognitive Biases During Energy Efficient Project Lifecycle
Project Phase Cognitive Bias
Energy Efficiency Example
Impact
Planning
Anchoring
Set an energy performance
Anchoring on minimum
goal.
standards (e.g., building
codes, LEED points) may
inhibit higher performance.
Status Quo
Adopt an integrated design
Obstacle to
approach.
implementation of new
approaches.
Design
Groupthink
Conduct a charrette.
Stifle novel ideas.
Professional Bias Use natural features to reduce
Engineers may ignore
cooling load.
natural/ passive options.
Framing
Use day lighting to reduce
Framing costs in appealing
electrical requirements.
terms is critical.
Construction
Professional Bias Specify design team
Need to overcome
participation during
professional bias for
construction.
parties to work together.
Mental
Seek incentives for meeting
Consider mental
Accounting
the energy performance goal.
accounting when seeking
incentives.
(Adapted from Klotz 2011)
This, in part, points back to the earlier discussion of PT and CPT as these theories have been
applied to the understanding of three biases examined by Klotz (2011) - framing, status quo, and
77

mental accounting. The framing problem was discussed in some detail earlier in this section. It
may be relevant to energy efficient retrofit problems when discussing the costs of various retrofit
measures; rather than discuss savings, Klotz recommends discussing them in terms of cost
avoidance, or costs if not adopted. While the EPC process is built around determination of a
savings guarantee (e.g., energy saved per unit time), avoided costs are allowed by many
jurisdictions’ EPC legislation and may be valuable in describing the full impact of not
undertaking a specific portfolio of ECMs.

The status quo bias, first described by Samuelson and Zeckhauser (1988) has been described as
continuing to choose the current situation (e.g., follow company policy, employ ECMs that
worked on previous projects) even when given new alternatives (Kahneman et al. 1991; Kwak et
al. 2010). Klotz (2011) reported that this might lead to refusal to adopt new technologies. Toole
(1998) found that architecture and engineering professionals may exhibit this bias when
considering new technologies with high levels of uncertainty related to market acceptance. That
bias was not noted when considering innovations with a high degree of technical uncertainty.
While Toole (1998) observed the status quo bias in residential energy efficiency projects, it has
also been shown to impact decision making in public-private partnership (PPP) projects (van
Buiten and Hartmann 2013).

According to mental accounting (Thaler 1985), decisions are impacted by the way possible
outcomes are coded and categorized, and explains violations of basic economic principles (Klotz
2011). Wilson and Dowlatabadi (2007) noted that this cognitive bias may lead decision makers
to partition benefits from energy efficiency retrofits such as monetary/non-monetary or energy/

78

non-energy into different mental accounts and assess the overall project benefit across multiple
accounts. Klotz (2011) suggests that this bias may manifest itself if energy costs are treated
separately from building costs in an organizational budget.

While the aim of this research does not include accounting for effects of these cognitive biases
on risk-based decisions made during the EPC retrofit process, the preceding discussion assists in
framing the nature of expertise sought from study participants in chapters 4 and 5. It also
provides context for the discussion of risk in construction and engineering in the following
section.

2.3.3 Risk in Construction and Energy Performance Contracting
2.3.3.1 Construction Risk
Construction projects are inherently risky, owing to the unique nature of each project, the
interactions among a large number of diverse stakeholders, the size and complexity of most
projects, and the political/economic/social landscape in which the project takes place (Zavadskas
et al. 2010; Banaitiene and Banaitis 2012). The cost of bearing certain risks has been proposed as
a technique to understand construction risk management; however, literature reviews have
concluded that this approach does not seem to be widely used (Yang and Lowe 2011) and upfront pricing of risks may not be incorporated in final bids, in order to improve bid
competitiveness (Laryea and Hughes 2011).

There is a clear distinction in the literature among research focused on overall project risk (Tah
and Carr 2000; Tah and Carr 2001; Zavadskas et al. 2010; Banaitiene and Banaitis 2012; Goh

79

and Abdul-Rahman 2012) versus research focused on individual project phases, elements, and
critical success factors such as the project planning phase (Diab 2012), time and cost (Doloi
2012), safety, and delay (Assaf and Al-Hejji 2006). The technical aspects of construction risk
management have been explored related to design and retrofit of structures to withstand seismic
events (Pampanin 2009) and selection of energy conservation measures (Willis 2008). A second
and significant body of the literature addresses business and decision-related aspects of risk
(Ward and Chapman 1995; Assaf and Al-Hejji 2006; Subramanyan et al. 2012; Xiang et al.
2012).

This highlights the need to find methods that adequately address the aspects of risk under review
in a given problem, to include the need to develop a comprehensive framework for addressing all
of the elements of risk that impact a specific project objective (Yang and Lowe 2011). Such a
framework must include the ability to capture expert knowledge and judgments related to the
treatment of risks in decision-making and provide a basis for incorporating that information into
a complete risk management scheme.

Broad categories of risk in construction projects have been given as arising from financial,
operational, technological, and legal aspects of the project environment (Kindinger and Darby
2000; Dey and Ogunlana 2004). These categories may include risks such as poor communication
among project team members, cost overrun, poor indoor environmental quality, design
discrepancies, delays in appointing subcontractors, low productivity, unacceptable performance
of installed technologies, and legal issues among project stakeholders (Dey and Ogunlana 2004;
Assaf and Al-Hejji 2006; Goh and Abdul-Rahman 2012; Jefferies and Chen 2012).

80

The literature has also identified risks by project phase, as presented in Table 2-4. Given the lack
of a rich literature focused on EPC risks, build-operate-transfer (BOT) and PPP projects were
examined when completing this table, as they share some characteristics with EPC, including
turnkey service, the integrated partnership approach, and the use of performance guarantees.
Table 2-4. Sources of Risk by Phase in BOT and PPP Construction Projects
Project Phase
Risk
Planning/Design Construction Operation
Technical risk (frequent design change,
X
design discrepancies, unknown site
conditions)
Credit risk
X
Bid risk
X
Completion risk
X
Cost overrun risk
X
X
Performance risk
X
X
Political risk
X
Liability risk
X
(Sources: Dey and Ogunlana 2004; Kokkaew and Chiara 2010; Goh and Abdul-Rahman
2012; Jefferies and Chen 2012)
2.3.3.2 Energy Performance Contracting Risk
As discussed earlier, EPC projects finance the work performed by the ESCO through savings
accrued from reduced operational costs. These projects are typically financed over a lengthy
performance period (12-15 years is not uncommon in the MUSH market) and benefits begin to
accrue during the energy savings phase, although some savings may occur during execution. For
ESCOs conducting EPC retrofits, the overarching issue is risk, because performance-related risks
are transferred from the owner to the ESCO (Deng et al. 2014). The level to which ESCOs can
manage and mitigate their performance risk defines a successful ESCO. In essence this means
that the concepts of risk analysis, risk management, decision-making under uncertainty, and
EPCs are interrelated and must be treated as such. The development of the retrofit design is an
81

iterative processes based on the data analysis conducted as part of the IGA and the expert
judgment of domain experts engaged in the work. Despite the large quantity of information
collected and analyzed as well as the use of expert knowledge, EPC retrofits still run the risk of
not meeting their savings guarantee. A 2005 study conducted by the Berkeley National
Laboratory analyzed 517 performance contract projects and found that 72% had excess savings
beyond the guarantee, while another 9% were fully stipulated (Hopper et al. 2005), as depicted in
Figure 2-6. While these results point to only 19% of projects failing to meet the savings
guarantee, approximately 5% of the included projects failed to meet their guarantee by 25% or
more, indicating significant potential economic risk to the ESCO in these cases.

Figure 2-6. Performance of ESCO Savings Guarantees
(Source: Hopper et al. 2005)
2.3.3.3 Preliminary Risk Framework
The research literature lacks a comprehensive risk framework for EPC retrofits. Hansen (2006)
provided an outline of key risks that ESCOs should be aware of when undertaking EPC retrofits.
Many of these risks were confirmed in an interview with the author (Hansen 2013). The existing
82

work by Hansen (2006) was built from over three decades of experience; however, it was not
verified through a literature review. Using this experience as a base, a literature review was
conducted to further examine risks faced by ESCOs when undertaking EPC retrofits. The net
result of that review was a preliminary risk framework that includes ten risk categories for
ESCOs to consider when undertaking EPCs (eight from Hansen 2006; two additional risks were
identified in the literature that were not included in Hansen 2006). Since the primary technical
risk retained by ESCOs is failure to achieve guaranteed performance, this framework can be
considered a first step toward identifying critical risk factors for ESCOs undertaking EPC
retrofits. These categories and the preliminary risk framework are depicted in Table 2-5.

Customer Pre-Qualification: The pre-qualification of customers is important to ensure that
ESCOs minimize financial performance and project management-related risks. Hansen (2006)
identified three categories of qualification factors, including: (1) financial/economic factors, (2)
facility/technical factors, and (3) people factors (e.g., commitment of management, manpower,
capacity, etc.). These issues may manifest themselves through customer-preferred short
performance periods that limit the technical approach (Bertoldi and Rezessy 2005), changes in
future occupancy and building use (Walker and Dominick 2000; Shonder and Hughes 2005),
interference with customer operations (AEPCA 2000), unknown latent conditions that can cause
delay (AEPCA 2000), improper operations and maintenance undertaken by the customer
(AEPCA 2000; Mills et al. 2006), human activity that is inconsistent with the M&V plan (Mills
et al. 2006), and the risk that the customer may go out of business prior to full payment of the
contract (Bertoldi and Rezessy 2005).

83

Table 2-5. Preliminary Risk Framework for ESCOs Undertaking MUSH Market EPC Retrofit Projects
Risk Category
Risks
References
Client Pre-Qualification Financial factors
Bertoldi and Rezessy 2005; Hansen 2006
 Client may go out of business before full contract payment.

Project Development
Energy Audit Quality

Equipment Selection
and Installation

Facility/technical factors
 Client-preferred short performance periods limit technical
approach.
 Changes in future occupancy and use.
 Unknown latent conditions.

AEPCA 2000; Walker and Dominick
2000; Shonder and Hughes 2005; Hansen
2006; Bertoldi and Rezessy 2005

People factors
 Human activity inconsistent with M&V plans .
 Improper O&M undertaken by client.
 Interference with client operations.
Costs incurred from project start-up; long development phases
can lead to difficult to recover costs.
The investment grade audit must include a risk assessment for
each proposed ECM.
Improperly-established or disputed baseline can impact
calculations of energy savings and also give rise to disputes.
Selected ECMs not aligned with findings of the IGA.
ECM package feasibility, failure to perform as designed,
uncertainty in factors used to predict performance.

AEPCA 2000; Hansen 2006;Mills et al.
2006

Reduced involvement of ESCO in ECM selection/installation,
procurement through bidding, risk of improper installation if
the ESCO is not involved during the construction phase.

84

Hansen 2006
Hansen 2006
Mozzo 2001; Mills et al. 2006; Sankey
2007
Hansen 2006
Shonder and Hughes 2005; Mills et al.
2006; Shang et al. 2008; Wang and Chen
2008; Jinrong and Enyi 2011
Hansen 2006

Table 2-5 (cont’d)
Risk Category
Commissioning

Operations and
Maintenance Practices

Measurement and
Verification of Savings

Project Management
Over the Project LifeCycle
Construction-Specific
Concerns
Volatility of Energy
Prices

Risks
Failure to commission may lead to missed opportunities to
verify ECM performance and better overall project
performance; Failure to commission may miss opportunities to
verify installed equipment performance, to ensure that
calibration, operation, and maintenance procedures are wellunderstood, and that all system documentation is turned over.
Poorly understood responsibilities by each party if the O&M
plan is unclear.
The customer may not perform O&M work to specification.
Customer self-performance or contracted O&M may reduce
revenue for ESCO.
Poorly-developed M&V plans can create additional risk for the
ESCO.
Poorly-designed M&V sampling protocols may not accurately
reflect the overall performance of the ECM package.
M&V protocols that do not capture non-energy benefits of
EPCs may understate overall system performance.
Failure to include O&M in the M&V plan may lead to missed
savings.
Failure to adopt life-cycle based management approaches may
result in failure to meet project objectives.

References
Stum 2000; Sankey 2007

Schedule growth may cause unrecoverable costs for the ESCO.
Cost growth may impact the financial analysis that the savings
guarantee is premised on - cost overruns may result from
schedule delays, latent site conditions, and field changes.
Changing prices can reduce project value.

Silberman 2010
Smith and Ferber 1996; AEPCA 2000;
Sankey 2007; Silberman 2010

85

AEPCA 2000
Mills et al. 2006
Hansen 2006
Hansen 2006
Mills et al. 2006; Mozzo 2001; Shang et
al. 2008
Larsen et al. 2012a
Schweitzer et al. 2000
Hansen 2006

Bertoldi and Rezessy 2005; Shonder and
Hughes 2005; Mills et al. 2006; Shang et
al. 2008; Wang and Chen 2008

Project Development: Energy service companies incur costs from the start of their work on the
development of an EPC project (Hansen 2006). The longer the development phase takes to
complete, the greater the risk to the ESCO that incurred costs will not be recovered, especially if
no agreement can be reached during this phase, effectively halting the project.

Energy Audit Quality: Energy audits must include ESCO verification of facility operating
parameters in establishing the baseline for energy consumption. That information, along with
equipment nameplate data, utility bill analysis, and a physical observation of building conditions
comprise an energy audit. The next stage of sophistication, known as an investment-grade audit
(IGA) incorporates a risk assessment for each proposed ECM (Hansen 2006). An improperlyestablished or disputed baseline can impact calculations of energy savings and also give rise to
disputes (Mozzo 2001; Mills et al. 2006; Sankey 2007).

Equipment Selection and Installation: The development of a portfolio of ECMs should be
aligned with the findings of the IGA (Hansen 2006). However, there is still the question of the
feasibility of the specified ECM package, and concerns about whether they will perform as
designed and what factors were included in considering predicted performance (Shonder and
Hughes 2005; Mills et al. 2006; Shang et al.2008; Wang and Chen 2008; Jinrong and Enyi
2011). Energy service companies may encounter risks during selection if the customer assumes a
greater role in this process or if bidding is required to procure ECM-related equipment and
technology. There is also a risk of improper installation if the ESCO does not have a role during
the construction phase, functioning as either a general contractor or a construction manager
(Hansen 2006).

86

Commissioning: Commissioning is an important element of all energy efficient retrofit and high
performance building projects; however, it is particularly important in an EPC, where system
performance (or lack thereof) is a critical component to achieving the performance guarantee and
release of the ESCO from financial liability. The lack of commissioning can result in missed
opportunities to verify ECM performance that would otherwise lead to better overall project
performance (Stum 2000; Sankey 2007). Commissioning can verify that installed equipment
performs as specified, that calibration and maintenance procedures are well-understood, that
proper operation procedures are understood, and that all required documentation is conveyed to
the party responsible for operation and maintenance (O&M) (Hansen 2006). Completing this
process with representatives of the owner’s staff and ESCO project management staff is
important in order to obtain agreement among all parties about future operating requirements and
savings potential (Hansen 2006).

Operations and Maintenance Practices: Operations and maintenance may be undertaken by
the ESCO, by the owner, or by a contractor to either party (Hansen 2006). Each arrangement
carries different risks to each party – to alleviate this; the O&M plan must clearly define each
party’s responsibilities (AEPCA 2000). The most obvious risk to an ESCO related to O&M is
that the customer may not perform this work to specification (Mills et al. 2006). Additional risks
may be present if self-performing O&M activities is a part of an ESCO's financial consideration
when evaluating whether to bid on work – if the customer decides to self-perform this work, or
hire their own contractor, the ESCO misses this revenue opportunity (Hansen 2006). While this
can be compensated for in other areas of the project, this may make the project less- or nonviable for the ESCO.

87

Measurement and Verification of Savings: Measurement and verification is important to
ensure that systems perform as designed and yield savings in accordance with the performance
guarantee (Hansen 2006). It is important that the ESCO and owner agree upon the M&V plan, as
this project phase is typically the longest of all phases and the information collected during this
stage establishes whether or not guaranteed performance levels are attained. A proper M&V plan
that adequately measures system performance is necessary; however, a plan that collects
unnecessary or unreliable data, that collects information that is not consistent with the ECM
being evaluated, or is too costly to implement, can create opportunities for additional risk to the
ESCO (Hansen 2006). A poorly-designed M&V sampling protocol may not accurately reflect the
overall performance of the ECM package

(Mills et al. 2006; Mozzo 2006; Shang et al. 2008).

Missing information from M&V plans can also create risks for ESCOs. Measurement and
verification protocols that do not capture non-energy benefits of EPCs (e.g., deferred
maintenance improvements) may understate overall system performance (Larsen et al. 2012a).
Similarly, O&M savings have been shown to constitute a large portion of projected savings;
however, O&M is often excluded from the M&V plan (Schweitzer et al. 2000).

Project Management over the Project Life-Cycle: The long-term nature of EPCs requires a
life-cycle management approach to ensure that project objectives are met (Hansen 2006). While
ESCOs should ensure that they take at least a portion of their fee from the guarantee to cover
initial costs, taking some of their fee from a share of the excess savings may signal to a customer
the ESCOs commitment to work with them throughout the entire performance period to ensure
that systems continue to perform well and accrue excess savings (Hansen 2006).

88

Construction-Specific Concerns: Several construction issues have been identified in the
literature that potentially add risk to an ESCO. While many of these concerns exist across a
variety of construction projects, the structure of EPCs makes ESCOs particularly vulnerable to
some of them. Schedule delays may shorten the project’s performance period, particularly in the
case of public projects with legislated maximum project durations. As a result, any delay in the
project schedule may create the potential for unrecoverable costs by the ESCO (Silberman 2010).
Cost overruns are a problem for all construction managers, but in an EPC, any additional cost
borne by the ESCO will impact the financial analysis that their savings guarantee is predicated
on (Smith and Ferber 1996; AEPCA 2000; Silberman 2010). Cost overruns may result from a
number of factors – those identified in the literature regarding EPCs include schedule delays
related to potential interference with customer operations (AEPCA 2000), unknown latent site
conditions leading to delay (AEPCA 2000), and field changes that result in delays or poor system
performance (Sankey 2007).

Volatility of Energy Prices: Volatility of energy prices is typically more of a customer risk
issue because changing prices can potentially reduce the value of the project (Bertoldi and
Rezessy 2005; Shonder and Hughes 2005; Mills et al. 2006; Shang et al. 2008; Wang and Chen
2008). Energy service companies should be aware of this issue, though, as customer sensitivity
to this concern may lead to an objective to diversify fuel mixes used in a project to hedge against
such long-term fluctuations (Berghorn 2012).

89

2.3.4 Risk Management Framework and Methods
The construction risk literature is replete with discussions of the risk management process. While
individual terms and concepts vary slightly, particularly in terms of the number of steps involved
with the process, there is broad agreement that risk management consists of three steps: (1) risk
identification and classification, (2) risk analysis and evaluation, and (3) risk control and
allocation/mitigation.

2.3.4.1 Risk Identification and Classification
In this step, all possible risk conditions are recognized and organized in such a way to permit
further analysis during later steps. Risks may be identified using a variety of methods, including
checklists, influence diagrams, interviews, use of expertise, cause and effect diagrams, failure
mode and effect analysis (FMEA), hazard and operability study, fault trees, and event trees.

An important step of risk identification is termed risk classification. In the classification step,
risks are structured in a logical way such that their individual and cumulative impacts on a
project may be understood and represented. Several classification methods have been suggested
in the literature including Perry and Hayes (1985), Cooper and Chapman (1987), Tah et al.
(1993), Dey et al. (1994), Wirba et al. (1996), Tah and Carr (2000).

Perry and Hayes (1985) developed lengthy lists of project risk categories which were then
attributed to contractors, consultants, and clients. Tah et al. (1993) developed a risk breakdown
structure (RBS) to identify the sources of risk based on their project location and their origin in
the larger project environment. Tah and Carr (2000) refined the RBS concept to that of

90

“hierarchical risk breakdown structure” (HRBS). This method uses a multi-tiered approach to
classify risks based on their overall type (internal or external), scope (global – affect entire
project; local – affect individual work packages), risk center (open-ended list of project areas),
risks (specific risks each belonging only to one risk center), and risk factors (specific factors
affecting the likelihood and severity of a risk). Dey et al. (1994) classified risk by first creating a
project work breakdown structure (WBS) and then analyzing risk separately for each “work
package” identified in the WBS. Cooper and Chapman (1987) identified risks based on their
source and magnitude, and then classified them as primary or secondary risks. This approach
then allowed the examination of dependencies among risks, which may lead to better
understanding of the potential sources of project failure (Wirba et al. 1996).

Wirba et al. (1996) developed a hybrid approach of Tah et al. (1993) and Cooper and Chapman
(1987). The RBS approach of Tah et al. (1993) was used to provide a comprehensive
classification of all project risks and the risks are then classified as primary and secondary based
on Cooper and Chapman (1987). This classification then permits the identification of
dependencies between risks and an estimate of their level of dependence.

2.3.4.2 Risk Analysis and Evaluation
Once risks have been identified, they must be analyzed based on their potential to influence the
entire system under review, which may be the project as a whole, a particular project phase, or a
specific project element (Ahmed et al. 2007; Arashpour and Arashpour 2012). Pai et al. (2003)
suggested a framework for analyzing risk factors based on a three step process of vulnerability
assessment, consequence analysis, and implementation. Risk evaluation then seeks to prioritize

91

risk events in order to develop mitigation and allocation plans. These plans may be based on past
experiences, accepted best practices, institutional knowledge, or standard practices (Ahmed et al.
2007).

Risk analysis techniques can include the construction of probability and impact grids, estimation
of system reliability, fault tree analysis, event tree analysis, collaborative processes, FMEA, and
sensitivity analysis. Risk evaluation techniques can include decision tree analysis, portfolio
management of projects, FMEA, or multiple-criteria decision-making (Ahmed et al. 2007;
Arashpour and Arashpour 2012).

2.3.4.3 Mitigation/Allocation
During risk mitigation and allocation, strategies are developed to address project risks, by either
retaining the risk (mitigation) or transferring to another party (allocation) (Perera et al. 2009).
Attempts to respond to risk events once they occur generally relies on utilizing contingency plans
(Ahmed et al. 2007). Proactive risk mitigation seeks to implement actions based on the
probability that a risk will occur, before it is actually realized. Risk mitigation and allocation
have often been considered to be the weakest stage of the RM process (Banaitiene and Banaitis
2012). Risk mitigation/allocation has been classified into four categories: (1) risk retention, (2)
risk reduction, (3) risk transfer, and (4) risk avoidance (Perera et al. 2009), and projects have
been shown to utilize a combination of techniques based on project-specific factors.

92

2.3.5 Risk Identification, Analysis, and Evaluation Methods
Qualitative and quantitative analysis are distinguished in the risk management literature. During
the identification stage, risk factors may be identified through quantitative methods or through
qualitative processes. Qualitative methods typically assess the impact and likelihood of identified
risks and can prioritize them for examination during the risk analysis and mitigation/allocation
stage of risk management (Banaitiene and Banaitis 2012). Tah and Carr’s (2001) HRBS and
fuzzy set analysis are examples of qualitative risk assessment. Quantitative analysis is concerned
with techniques to estimate the frequency and magnitude of risks, and makes use of tools such as
decision tree analysis and Monte Carlo simulation. Studies such as Kindinger and Darby (2000)
focused on development of a qualitative risk assessment method (risk factor analysis), but also
demonstrated its integration with a quantitative analytical method.

A thorough inventory of analytical methods supporting risk management is provided by Grimaldi
et al. (2012). These methods are presented in terms of their applicability to phases of a project
life cycle as well as a corporate maturity toward risk, as follows:


A novice level of maturity is focused solely on risk identification;



A normalized maturity level adds qualitative risk analysis, sometimes risk response
and monitoring and control; and



A natural level of maturity undertakes the complete risk management process, to
include quantitative analysis, and integrates the risk management into the project
management process.

93

Forbes et al. (2008) also developed and tested a matrix of risk management techniques to be
applied to various risk management phases and to evaluate specific risk categories (e.g.,
technical, social, legal) as well as the characteristics of the data about that risk category
(fuzziness, incompleteness, randomness). They then applied the case-based reasoning approach
to identify where specific techniques were used in similar risk management problems in an effort
to help guide the selection of appropriate techniques. Forbes et al. (2008) also outlined a method
to determine the applicability between two risk management methods in a specific problem
context – that of a sustainability assessment of housing. Grimaldi et al. (2012) and Forbes et al.
(2008) agree that it is important to select the appropriate methods based on the nature of the
decision problem under analysis and that understanding of the individual methods is critical to
the overall success of the analysis.

Grimaldi et al. (2012) classified 31 risk management techniques based on the following criteria:


Risk management phase;



Project life cycle phase (conceptualization [development], planning [design], and
execution); and



Level of corporate maturity with regard to risk management (novice, normalized, and
natural).

The goal of the review in this section is to identify potential risk management techniques
considering the characteristics of the problem that is the subject of this research – the risk of
failing to meet the performance guarantee in EPC retrofits. Key criteria for selection of a risk
management strategy for this research include:

94



Qualitative techniques, based on the recommendation of McKim (1992);



Applicability across the development and design phases (treated as two phases by
Grimaldi et al. 2012), as this is where the majority of risk-based decisions are made in
EPC projects, and when changes can have the greatest impact at the lowest incremental
cost (Kmenta and Ishii 2001; Horsley et al. 2003; Kishk et al. 2003);



Applicability to organizations in the first two stages of organizational maturity (novice
and normalized) per Grimaldi et al. 2012;



Techniques favoring the use of expert knowledge;



Techniques enabling the incorporation of life cycle costs as part of risk evaluation
(Research Premise #4 in Chapter 1); and



Iterative, consensus-based techniques to address Raftery’s (1994) observation that
construction professionals could benefit from being made aware of their decision-making
process.

Based on a review of Grimaldi et al. (2012), four risk management techniques that met one or
more of the selection criteria presented above were subjected to further review (Figure 2-7). It is
worth noting that the risk monitoring and control phase is absent from Figure 2-7; since the
research focuses on activities that take place in early phases of EPC retrofits, this phase has less
weight in the selection of appropriate risk management techniques.

As a result of Figure 2-7, expertise elicitation using the Delphi technique will be discussed
relative to risk identification during project development and design, as well as to contribute to
risk analysis/evaluation and risk response. Failure mode and effects analysis (FMEA)/failure

95

Figure 2-7. Evaluation of Selected Risk Management Strategies by Project Life Cycle
Phase, Risk Management Phase, and Organizational Maturity Toward Risk
(Adapted from Grimaldi et al. 2012)
mode and effects criticality analysis (FMECA) will be reviewed as the primary tools for risk
analysis and risk response because they are well-suited to all phases of the project life cycle
(Onodera 1997). There is also a body of literature supporting the incorporation of life cycle costs
into FMEA/ FMECA analysis as a measure of risk severity during the evaluation phase (Kmenta
and Ishii 2000; Rhee and Ishii 2002; Marenjak et al. 2003; Rhee and Ishii 2003; Rhee and
Spencer 2009; Chen and Zhang 2012; Liu et al. 2013). The next section includes a discussion of
FMEA/FMECA; expertise elicitation and the Delphi technique are discussed in section 2.3.6.

2.3.5.1 Failure Mode and Effects Analysis
The goal of FMEA is to help engineers focus on high-risk components, typically during design;
however, systems and processes can also be analyzed with this technique (Teng and Ho 1996;

96

Stamatis 2003; AIAG 2008; McDermott et al. 2009; Curkovic et al. 2013). The method employs
an inductive logic process, whereby a team of analysts examine potential failures in a design or
process in a bottom-up fashion, beginning with known or assumed failure modes at one level
(e.g., individual component), and then investigate its effects on higher levels (e.g., system or
operational subsystems). These methods assess the risk of failure of a system, process, design, or
service proactively – that is before they occur (Stamatis 2003). This has the net effect of
allowing these tools to inform the design process and permit improvement by identifying and
correcting potential failure modes.

FMEA was developed in the 1960s for the aerospace industry and has since been used by the
defense, nuclear, space exploration, automotive, software systems, healthcare, and construction
industries (Bednarz and Marriott 1988; Dhillon 1992; Latino 2004; Liu et al. 2013; Rahimi et al.
2013). Built environment applications of FMEA/FMECA have included an assessment of
barriers to innovation in construction (Murphy et al. 2011), broadly understanding the risks of
moisture problems in buildings (Nielsen 2002), an analysis of the effects of interior insulation on
moisture-related damage to masonry walls with wooden floor beam ends (Morelli and Svendsen
2012), risk of failure in building envelope systems (Layzell and Ledbetter 1998), and risk of life
cycle environmental impacts of a given design (Lindahl 1999). Despite the goal of risk reduction,
in its current form FMEA is unable to evaluate the cost of risks and the cost-benefit relationship
of proposed mitigation measures.

Methodological Concerns – RPN: Despite the wide application of this method, FMEA has its
own limitations, to include its complexity (Grimaldi et al. 2012), as well as criticism leveled at

97

the primary measure of failure mode criticality, the risk priority number (RPN) (Bowles 2004;
Gargama and Chaturvedi 2011; Liu et al. 2013). The RPN is constructed by assessing severity
(S), occurrence (O), and detection (D) ranks for each failure mode, utilizing a scale of 1-10; the
three measures are then multiplied to obtain the RPN, a score between 0 and 1,000 (Gargama and
Chaturvedi 2011; Liu et al. 2013).

Primary criticisms of the RPN include the use of ordinal rankings (Imbeah and Guikema 2009),
duplicate values of the RPN having very different characteristics (Kmenta and Ishii 2004), and a
lack of linguistic terms regarding priority for managing critical failures identified through RPN
values (Bowles 2004; Abdelgawad and Fayek 2010). Specific challenges also arise due to the
somewhat arbitrary nature of S, O, and D scales and the way in which they are applied in the
analysis.

Severity scores are only assigned to the end effects of a failure mode; the cause of the failure (the
failure mode) is not scored, thereby making the end effects independent of the cause and any
intermediate effects. Also at issue when conducting FMEA to asses built environment risks, each
cause or groups of causes may need to be evaluated contextually, related to severity. Mecca and
Masera (1999) illustrated that notion by creating different severity measures based on the risk
factor to which they were connected.

Multiple definitions for detection have been proposed within the context of manufacturing,
typically falling into one of three categories: (1) the ability to detect a potential cause before a
component or system is released for production (AIAG 2008); (2) the likelihood that process

98

controls will detect a root cause before the part leaves the manufacturing area (Stamatis 2003);
and (3) the chance that the customer (end-user) will detect the problem before a catastrophic
failure (Palady 1995). Kmenta and Ishii (2004) noted that this is problematic, as it is difficult to
discern which of these definitions measures the contribution towards realized risk. Several
authors have suggested eliminating the detection score from FMEA (Bowles 1998; Society of
Automotive Engineers 2001; Bowles2004) because the ranking itself is very subjective, leading
to a high degree of variation among detection scores. The traditional use of detection indicates
the likelihood that a failure mode will be detected through subsequent testing, not at the point of
occurrence; and detections occurring late in the product or process life cycle typically are
typically not cost effective, yet the detection score may not change based on when detection
happens. A method is needed that can accurately analyze and evaluate risks based on the project
or process life cycle phase where they are realized.

Several methods have been proposed to address these shortcomings (Liu et al. 2013), to include
tree analysis (Abdelgawad and Fayek 2012), risk probability and impact assessment, advanced
programmatic risk analysis and management model (Imbeah and Guikema 2009), multi-criteria
decision models (Franceschini and Galetto 2001), artificial intelligence/fuzzy analysis
(Abdelgawad and Fayek 2010; Abdelgawad and Fayek 2012), Pareto analysis (Carbone and
Tippett 2004), and cost-based models (Kmenta and Ishii 2000; Rhee and Ishii 2002; Rhee and
Ishii 2003; von Ahsen 2008).

99

2.3.5.2 Scenario-Based FMEA (SFMEA)
Gilchrist (1993) was among the first to suggest an alternate means for calculating risk criticality,
replacing the traditional RPN with expected cost, which was measured as cost multiplied by
probability. Kmenta and Ishii (2004) identified an additional shortcoming of FMEA methods,
including many of those that attempted to correct shortcomings of the RPN. A goal of FMEA is
to include a large set of empirically-derived failures, to encourage deeper review of risks that
might be ignored; however, many applications rely on mathematical functions or one-to-one
relationships between a failure mode and its end effect (e.g., cause and effect). In order to
represent multi-level cause-effect relationships, FMEA requires the use of multiple levels of
analysis (e.g., effects at the system level may become causes at the design level). Rhee and Ishii
(2002 and 2003) improved upon previous methods by incorporating life cycle cost into FMEA,
but there was limited discussion of the link between failure representation and risk evaluation
(Kmenta and Ishii 2004). Kmenta and Ishii (2000) introduced failure scenarios with expected
cost and compared the results of FMEA and follow-on activities based on expected costs and
RPN. Traditional FMEA and SFMEA are compared in Table 2-6.
Table 2-6. Comparison Between Traditional FMEA and Scenario FMEA
Failure
Cost of
Product
Total
Method
Strategy
Probability
Failure
Cost
Cost
Traditional
Increase reliability
Reduce
No change Same or
Uncertain
FMEA
increase
Scenario-Based Reduce total cost (failure Cost-based
Cost-base Cost-base Same or
FMEA
cost and product cost)
decision
decision
decision
lower
(Source: Kmenta and Ishii 2004)
Since life cycle costs are an integral element of EPC retrofits, a FMEA method that can
adequately address both issues is preferred for this research. Scenario FMEA (Kmenta and Ishii

100

2000; Kmenta and Ishii 2004) is therefore used as the primary risk analysis and evaluation tool
for this research.

Failure Scenarios: Conventional FMEA is constructed around failure modes, which describe the
cause of a failure and have associated effects. Failure modes may occur across various system
levels, and the standard practice is to represent that through the use of “nested” FMEAs, where
the effects at a lower level become causes at the next higher level, and so on. These separate
analyses lack consistency in managing information across these various levels, thereby making a
life cycle-based analysis very difficult (Kmenta and Ishii 2004). There is typically a one-to-one
relationship between the causes and effects in a given failure mode, so a cause and effect chain is
only represented by the cause and the end effect – intermediate effects are not evaluated
explicitly.

Scenario-based FMEA allows the creation of many possible “failure scenarios,” which are cause
and effect chains that can be lengthened, as needed, if new effects and new causes are identified
during analysis (Kmenta and Ishii 2000; Kmenta and Ishii 2004). Failure scenarios are found in
the risk literature (Kaplan and Garrick 1981) and have recently been applied in construction and
engineering research (Esmaeili and Hallowell 2013; Pallin 2013; Hamilton et al. 2014). This also
permits description of the entire failure chain, to include intermediate effects (Figure 2-8).

Risk Evaluation Using Expected Cost: Risk was defined earlier in this section as the product of
uncertainty (probability) and consequences (damages). Gilchrist (1983) and Kaplan and Garrick
(1981) identified cost as an acceptable measure of consequences, so for each failure scenario,

101

Figure 2-8. Comparison of Failure Modes and Failure Scenarios
(Source: Kmenta and Ishii 2004)
Kmenta and Ishii (2004) posited that risk can be calculated as expected cost (Rasmussen 1981;
Kmenta and Ishii 2000).

The expected cost can be shown mathematically as:
Expected Cost (EC) = pi x ci where pi is the probability of risk i and ci is the cost of risk i.

A hypothetical example of failure scenarios throughout a product’s life cycle is shown in Figure
2-9. There are eight possible cause-effect scenarios represented by combinations of causes being
introduced into the system and downstream discoveries of those causes through an observed
effect. As depicted in Figure 5-5, a failure cause may be discovered at various life cycle phases.
The conditional probability of this failure scenario can be calculated using Bayes’ theorem
(Bolstad 2013) which states the probability of a given scenario can be found by multiplying the
probability of the cause by the conditional probability of the end effect (i.e., the probability of
discovering the failure at point d, point b, etc.). Figure 2-10 shows the expected cost equation for
scenario a-d from Figure 2-9.

102

This also demonstrates the robustness of scenario FMEA throughout the project life cycle;
traditional FMEA separates process, design, and system cause-effect relationships into separate
FMEA sessions, and might miss lower level failure modes when aggregating up to the next level

Figure 2-9. Example Failure Scenarios Over a Product Life Cycle
(Source: Kmenta and Ishii 2004)

Figure 2-10. Expected Cost Components in Failure Scenarios
(Source: Kmenta and Ishii 2004)

103

FMEA. Scenario-based FMEA allows the same panel to examine multiple levels of causes and
effects, using the same evaluative criteria and language to describe risks.

2.3.6 Expertise Elicitation and Delphi Technique
The following section provides a brief discussion about the need for expertise elicitation in
qualitative risk assessment and describes several elicitation techniques found in the literature.
The Delphi technique is explored in greater detail, in connection with the screening of risk
management techniques in the previous section.

2.3.6.1 Elicitation in Qualitative Risk Assessment
Risk assessment consists of qualitative and quantitative methods that may be used separately or
collectively as part of the analysis (Liu et al. 2010). Quantitative methods may not be able to
effectively reduce uncertainty in all decision parameters due to limitations in knowledge. As a
result, expert knowledge, experience, and judgment may be sought to represent a problem
qualitatively, which ultimately requires the elicitation of this information from experts to be used
in the analysis. The current literature on expert elicitation has focused on three broad topics: (1)
methods of elicitation, (2) the role of heuristics and human cognitive processes on elicitation, and
(3) methods of aggregating elicited information from multiple experts (Mosleh et al. 1988; Moon
and Kang 1999; Engel and Dalton 2012).

Qualitative elicitation methods include focus groups that yield risk rankings and identification of
decision alternatives, whereas quantitative methods can lead to the development of probability
distributions of risk occurrence based on expert information. Numerous elicitation techniques

104

have been researched for the impact of the framing of the elicitation technique on the type of
expert behavior elicited, particularly as related to decisions made under risk (Ranasinghe and
Russell 1993; Crosetto and Filippin 2013). A hybrid qualitative-quantitative method has been
proposed by Engel and Dalton (2012).

Bonano et al. (1990) suggest that there are four fundamental cognitive processes that experts
engage in when making judgments: (1) identification of decision alternatives, (2) screening of
alternatives, (3) decomposition of the judgment task into smaller elements, and (4) quantification
of comparative judgments about decision alternatives. These processes must be represented in an
elicitation strategy in order to effectively capture the information desired from the experts. These
cognitive processes intersect with psychological research related to heuristics and biases.
Tversky and Kahneman (1974) operationalized a definition of heuristics as principles that help a
decision-maker reduce complex decision variables into simpler judgments. They identified three
widely-used heuristics: (1) representativeness, (2) availability, and (3) adjustment and anchoring,
as well as judgmental biases corresponding with each.

Methods of expert judgment aggregation can be divided into mathematical and behavioral
approaches (Engel and Dalton 2012). Mathematical approaches are typically undertaken when
interaction between experts is not possible or not desired, and utilizes methods such as
averaging(Keeney and von Winterfeldt 1991), Bayesian aggregation (Adams 2008), aggregation
based on the analytic hierarchy process (AHP), and applications of fuzzy set theory (Moon and
Kang 1999), which can incorporate variable weights assigned to each expert based on prior
performance in training and calibration sessions (Engel and Dalton 2012). Behavioral approaches

105

seek to build consensus among experts and can utilize the Delphi technique (Ranasinghe and
Russell 1993; Schieg 2007).

The use of expert judgment has been shown to be effective in risk-based decisions related to
engineered component failure analysis and seismic risk analysis (Mosleh et al. 1988). However,
concerns have arisen over reliance on “rules of thumb” and common sense of experts, the lack of
standardized frameworks for expert information aggregation, and a limited use of normative
expertise in structuring elicitation exercises (Mosleh et al. 1988). Knowledge-based (De Zoysa
and Russell 2003a; De Zoysa and Russell 2003b; Tserng et al. 2009), linguistic (Liu et al. 2010),
and Delphi technique (Ranasinghe and Russell 1993; Schieg 2007) approaches have been
suggested to combat these concerns.

2.3.6.2 Delphi Technique
Construction research is driven by the need to account for the actions of people engaged in the
process of design, building, and operations, and as such, requires the careful application of social
science research methods (Toole 2006). A suitable research design is needed to ensure that
research questions can be answered appropriately (Keeney et al. 2011). Research designs used in
construction engineering and management research may be categorized as being experimental or
qualitative, theoretical or empirical (Toole 2006). Experimental-based research has been called
the “gold standard,” mainly due to its strength in assuring internal validity of results (Vogt 2005;
Trochim 2006); however, it is likely not practical nor is it appropriate for the study of humanrelated factors in construction management research (Toole 2006). Two methods that have

106

shown success for application to human-related factors in construction industry have been case
study analysis and the use of surveys (Duah 2014).

Case study is a qualitative research technique whereby a researcher intensely studies an
individual or a setting over time. This approach lacks a single methodological framework and
instead frequently relies on multiple sources of data, such as semi-structured interviews, direct
observation, or participant observation (Trochim 2006). Analysis includes a description of the
cases and emergent themes derived through observation and data collection (Miles et al. 2014,
Yin 2014). Survey research systematically elicits information from participants about the
attribute(s) of interest to the researcher and may consist of a questionnaire or interview.
Interviews offer the researcher greater flexibility than a traditional survey questionnaire and
permit open-ended responses that enable the researcher to be more probative when needed
(Trochim 2006).

The Delphi technique is a method of survey research that was developed by the RAND
Corporation during the early days of the Cold War to study technological applications related to
defense (Vogt 2005; Keeney et al. 2011). Delphi has received a great deal of recognition and use
in a number of disciplines that focus on research involving human interactions and behaviors
(Keeney et al. 2011) and has more recently been shown to offer a robust method for construction
researchers (Rajendran and Gambatese 2009; Hallowell and Gambatese 2010). Delphi utilizes
the judgments of qualified experts in a well-structured technique that permits the researcher to
maintain control over participant biases (Hallowell and Gambatese 2010). Keeney et al. (2011)
stated that the underlying principle of Delphi is the assumption that the group opinion is more

107

valuable than individual opinions. Chapter 3 includes a discussion of notions of group opinion
and its representativeness of consensus or merely “majority rule;” however, for the sake of this
discussion, this premise has been accepted at its face value. Delphi utilizes an iterative, multiround survey process to combine individual expert opinion into group consensus, and is
particularly useful where there may be wide divergence among expert opinion and where it is not
possible to convene such experts in a single meeting (Yousuf 2007).

Several variations of Delphi have been presented in the literature (Turoff 1970; Linstone and
Turoff 1975; Mitroff and Turoff 1975; Rauch 1979; Keeney et al. 2011), to include the classical,
modified, policy, and decision Delphi. Classical Delphi utilizes a relatively unstructured first
round to facilitate idea generation and often uses three or more survey rounds. Modified Delphi
usually replaces the first round with face-to-face interviews or focus groups (Keeney et al. 2011).
In both cases, achieving consensus among the participants is the overarching goal.
While achieving consensus among participants is associated with traditional or modified Delphi,
this reflects the so-called “Lockean Perspective,” whereas variations exist that do not seek such
agreement and instead seek to address reality through alternate means (Mitroff and Turoff 1975).
The policy Delphi (Turoff 1970) and the decision Delphi (Rauch 1979) are two such variations
where the interpretation of reality and the creation of reality are the goal, respectively; however,
the decision Delphi enables panelists to give structure to potentially broad lines of inquiry and
aid the decision-making process through this newly constructed reality (Rauch 1979). The
decision Delphi may be well-suited to fully understanding and evaluating risks, particularly in
the context of decisions made over the life of complex processes.

108

Regardless of the method used, key elements of Delphi include contributions of expert
knowledge by panel members; assessment of the group consensus and outlying opinions; an
opportunity for individuals to revise their responses, as needed; and some degree of anonymity
among individual responses (Linstone and Turoff 1975; Hsu and Sandford 2007). The iterative
and anonymous inclusion of a large number of panelists is important in order to avoid dominance
and Von Restorff biases (Hsu and Sandford 2007; Hallowell and Gambatese 2010; Keeney et al.
2011). Dominance bias is whereby one member of the group exercises control of group direction
and feedback. Bias attributed to the Von Restorff effect is described as the condition whereby
respondents recognize and remember more extreme events more accurately and more often than
less extreme events (Hallowell and Gambatese 2010).

Delphi rounds could theoretically be iterated infinitely until consensus is reached; however, the
research is clear that the return on investment in terms of robustness of findings versus the effort
expended diminishes after the third round, therefore the decision of the appropriate number of
rounds to use is a practical one, particularly in light of diminishing response rates when more
rounds are used (Hsu and Sandford 2007; Hallowell and Gambatese 2010; Keeney et al. 2011;
Duah 2014). Skulmoski et al. (2007) reported that fewer than three rounds may be adequate to
attain research goals when there is a high degree of homogeneity among panelists. The goal of
the first round is typically to provide scope to the research topic (Powell 2003), which then leads
to construction of a survey instrument through qualitative analysis of the results for use in
subsequent rounds (Powell 2003; Hsu and Sandford 2007). Alternative approaches to the first
round include the use of a semi-structured questionnaire coupled with an extensive literature
review, and pilot testing of questionnaires to help diagnose and treat problems with the

109

questionnaire construction and its feasibility of administration (Powell 2003; Hsu and Sandford
2007).

The second and subsequent rounds seek to quantify earlier findings, often through rating or
ranking techniques (Powell 2003) or seek to re-frame questions that have not achieved consensus
(Hallowell and Gambatese 2010). In the latter case, it is suggested that panelists are provided
with median response values along with the range of round one responses and their first round
response, and then be given the chance to modify their response or respond to the re-framed
question. The median is often used to de-emphasize biased responses and minimize contrast bias
(Hallowell and Gambatese 2010). Hallowell and Gambatese (2010) suggested a method whereby
medians are reported along with reasons for outlying responses in the third round, in a final
attempt to achieve consensus] (Hallowell and Gambatese 2010).

Two typical ways to measure the attainment of consensus are the statistical approach and
through the use of percentages. Statistics used are typically measures of central tendency (mean,
median, and mode) and level of dispersion (standard deviation and inter-quartile range) (Hasson
et al. 2000). Median and mode are favored through the use of a Likert-type scale, and as
described above, can also help minimize contrast bias and further diminish previously-biased
responses (Keeney et al. 2011). Response frequency criteria are assessed as having achieved
consensus when a certain percentage of expert responses fall inside a prescribed range adopted
prior to the commencement of data collection (Keeney et al. 2011). The lack of clearly-defined
measures of consensus and adequate explanation of the threshold values used to attain consensus
is a methodological weakness observed in a large number of Delphi studies (Evans 1997).

110

Keeney et al. (2011) found disagreement on a standard threshold value for percentage consensus,
and cited varying thresholds identified in the research of 51%, 66%, 70%, 75%, and 80%. Thus,
the researcher must exercise appropriate judgment to develop an appropriate threshold that is
applicable to the domain and topic being studied.

A critical success factor of any Delphi study is the combined expertise of the panelists, which is
defined by the size of the panel and the qualifications of each expert panelist (Powell 2003).
There is a lack of guidance on the appropriate panel size to be used in Delphi (Keeney et al.
2006). While the reliability of the consensus achieved by the panel will increase with a larger
number of participants, Murphy et al. (1998) reported there is little empirical data to support a
connection between panel size and overall reliability and validity of the research, although a
diverse panel may ensure better performance through the consideration of different perspectives.
Skulmoski et al. (2007) stated that a panel of 10-15 experts may be sufficient when the group is
homogenous. Hallowell and Gambatese (2010) found that panels of 8-12 highly qualified experts
are adequate for effective results in construction and engineering management research.

The success of a Delphi study depends on the expertise of the participants comprising the panel
(Powell 2003); however, the characteristics to determine expertise are ambiguous and defined on
a project-specific basis (Hallowell and Gambatese 2010). Thus, there is a correlation between the
quality of the research findings and the quality of the expert panel. Delphi panels are not
statistical samples of a larger population (Hasson et al. 2000; Keeney et al. 2006; Duah 2014),
and instead are constructed using purposive sampling techniques (Hasson et al. 2000; Oliver

111

2006; Tongco 2007) such as expert sampling, snowball sampling, and critical case sampling
(Teddlie and Yu 2007).

The following panelist characteristics have been suggested to promote the successful use of
Delphi (Duah 2014):


Willing and able to make a valid contribution (Powell 2003);



Reflect current domain knowledge, perceptions, and relatively impartial to findings
(Jairath and Weinstein 1994);



Highly trained and competent within their domain related to the issue being studied (Hsu
and Sandford 2007);



Panelists are nominated from among the targeted groups of experts (Ludwig 1994);



Panelists are identified through literature searches and/or recommendations from other
recognized domain experts (Gordon 1992); and



Primary stakeholders with various interests related to the issue being studied (Hsu and
Sandford 2007).

Reliability and Validity of Delphi: Reliability refers to the quality of the measurement
employed in the research, and addresses the consistency and stability of research findings – in
essence, whether the work is repeatable and yields similar results (Vogt 2005; Trochim 2006).
Delphi enhances reliability through the avoidance of group biases and groupthink (Keeney et al.
2011). Unlike other research methods where bias can be introduced through the research,
particularly by the interactions of subjects, Delphi depends on the unbiased judgment of experts
(Hallowell and Gambatese 2010). Such biases are reduced, and reliability is enhanced, due to the

112

anonymity of participants, which limits the ability for one or more respondents to dominate the
process, and through iteration which enables respondents to reformulate answers from previous
rounds.

Validity is built on a foundation of reliability, and refers to a method that accurately measures
what it is supposed to measure and research designs that help collect data appropriate for the
construct being studied (Vogt 2005). External validity is concerned with the generalizability of
findings, and internal validity speaks to whether the research yielded a confident cause and effect
relationship (e.g., is there another cause that can explain the observed effects?).

Validity can be measured as being content- or criterion-related (Keeney et al. 2011). Content
validity is concerned with the ability of the research tool to sample the complete range of the
attribute under study (DeVon et al. 2007). Evidence of content validity in Delphi is based on the
following methodological characteristics (Keeney et al. 2011):


Results are based on group opinion, which provides greater validity than individual
opinions. The entire process is based on confirmative judgments from expert panelists.



A less-structured, qualitative round allows experts to scale items and iteration allows for
an opportunity to review and judge the suitability of that scale.

Criterion-related validity is established when a test can accurately predict criterion or indicators
of a construct (Keeney et al. 2011). A number of studies have shown that Delphi helps achieve
such validity through its iterative survey rounds, achieving consensus, and observed accuracy in

113

long-range forecasting (Keeney et al. 2011). The authors also reported on a number of threats to
external and internal validity.

Threats to external validity may arise due to the inability to generalize findings, since Delphi
panels are ephemeral, and consist of experts who may not be representative of the larger
population from which they are drawn (Keeney et al. 2011). Such threats may include:


Selection -The makeup of the Delphi panel may lead to results that are not repeatable
with other panels, and the inclusion of non-experts on the panel may yield different
results from the experts;



History - Outside events may influence expert’s responses between Delphi rounds;



Situation – The characteristics of the Delphi design, to include timing, number of rounds,
feedback, and lack of agreement on thresholds for consensus can limit generalizability;



Reactivity – There is little accountability for views expressed by panelists and since true
anonymity cannot be guaranteed, some experts may be influenced by others;



Natural loss - Participants may drop out or lose interest between successive Delphi
rounds; and



Researcher bias – The open qualitative round is designed to capture a broad range of
ideas from experts, which is then reduced by the researcher based on content analysis.

2.4 LIFE CYCLE COST ANALYSIS
This category presents an overview of life cycle cost analysis (LCCA) and its applicability to the
current research. Also discussed here are LCCA publications that incorporate risk and
uncertainty into the analysis.

114

2.4.1 LCCA Overview
Life cycle cost analysis is a method for assessing alternatives by considering significant costs
over the economic life of each alternative (Dell’Isola and Kirk 2003), and is particularly wellsuited for fixed assets and other products (Woodward 1997; Asiedu and Gu 1998). Since its
inception, it has enjoyed significant use in the construction industry. Korpi and Ala-Risku (2008)
found that 62% of the 55 examined LCCA case studies were in the construction industry; another
18% came from the energy industry. Life cycle cost analysis enables an investor to make
decisions based on the total costs and benefits accrued over the entire economic life of an
investment. In the building industry this economic life includes:


Initial costs for design and construction of buildings;



Costs incurred during the operational life of the building; and



End of life or demolition costs.

Harvey (1976 – cited in Woodward 1997) proposed a four step general life cycle cost model:


The cost elements of interest are the cash flows that occur during the asset’s life, which
includes all costs from purchase through end of life disposal.



Defining the cost structure involves grouping costs in order to identify potential tradeoffs.
Identification of tradeoffs is an important concept toward developing an optimal life
cycle cost. The groupings are defined in a manner that is appropriate to the system under
review in order to facilitate this tradeoff analysis (Harvey 1976 - cited in Woodward
1997). Figure 2-11 depicts a three category cost grouping that might be relevant to the
design of building retrofit measures.

115

Figure 2-11. Example Life Cycle Cost Categorization
(Based on Woodward 1997)


A cost estimating relationship mathematically describes the cost of the asset under review
(e.g., product or activity) as a function of one or more independent variables, such as
energy costs, running hours, and labor costs (Emblemsvag 2001). This information may
come from historical costs databases or expertise (Korpi and Ala-Risku 2008), and
utilizes defined relationships (e.g., linear, parabolic, hyperbolic) between the dependent
and different independent variables.



Establishing the method of life cycle cost formulation entails selecting an appropriate
methodology to evaluate the life cycle cost of the asset under review. Examples of LCCA
methods include those developed by Kaufman (1970 – cited by Woodward 1997), the
cost-benefit analysis developed by Hanley and Spash (1994), and the requirements of the
Federal Energy Management Program (FEMP) (NIST 1995).

116

The frameworks presented by Kaufman (1970 – cited in Woodward 1997), Hanley and Spash
(1993), and the FEMP guidance (NIST 1995), while slightly different, require discounting of
costs to the base period and calculation of the life cycle cost (Woodward 1997; Kishk et al.
2003). Present worth and annualized LCCA are most often used in design economic analysis,
which seeks to find the most cost-effective design solution which meets project requirements
(Dell’Isola and Kirk 2003). Investment economic analyses typically use payback period, return
on investment, or savings to investment ratio to decide whether an investment is justified, and if
so, to determine the most cost-effective course of action (Dell’Isola and Kirk 2003).

Cost-effectiveness analysis, also termed benefit-cost analysis (BCA), can be used whenever
dollar value calculations of the benefits provided by the alternatives under review are either
unnecessary or impractical (Dell’Isola and Kirk 2003). Similarly, BCA can be used to examine
alternatives with identical costs but different benefits. This is possible because the BCA
calculation allows separate calculation of life cycle costs and benefits while incorporating the
discount rate, and inflation rate. This method also includes discounting future costs to their
present value, and may be considered one of the methods for calculating net present value.

2.4.2 Applications of LCCA to the Built Environment
Recent literature focused on built environment sustainability and energy costs/savings is rich
with cost-benefit evaluations; however, EPC-specific literature is again lacking. Larsen et al.
(2012a) addressed this concept in the aggregate sense by reporting simple payback times for 15
years-worth of EPC retrofit projects. The simple payback time was defined as project installation
costs (financing costs were excluded) divided by the dollar value of annual energy and O&M

117

savings. This method was selected for ease of reporting to a diverse audience and for consistency
with previous research conducted by the same group at the Berkeley National Laboratory (e.g.,
Goldman et al. 2005). Additionally, other metrics of investment performance are only valid when
there is no interim cash flow generated (e.g., internal rate of return), which is often not the case
in EPC projects. When evaluating data from a sample of 2,484 projects, it was observed that K12 schools had the longest median payback times for all retrofit strategies studied (7–15 years),
while median payback times for public/institutional sector projects (excludes K-12 schools) was
7–10 years for onsite generation, non-energy, and major HVAC retrofits. The median payback
time for lighting only retrofits was 3 years in ESCO projects targeting public sector customers.
Additionally, public-sector projects had median BCA values of 1.1 (K-12 schools) up to 2.6
(health and hospitals). The authors posited that lower observed BCA ratios are likely the result of
larger amounts of deferred maintenance work being financed through the EPC vehicle, thereby
limiting the potential for energy savings to a smaller subset of work scope items. Within this
context that EPC retrofits have demonstrated positive BCA values, several example LCCA
papers in the built environment are described briefly below.

2.4.2.1 Indirect Benefits of Sustainable Building Retrofits
A variety of research has also been conducted in an attempt to quantify indirect benefits of
sustainability-based retrofits and new green construction. Singh (2009) found BCA values of 3-8
for LEED-certified office buildings in Michigan, where the benefits were derived from increased
worker health and productivity as a result of improved health and well-being. A 2003 study
conducted for the Seattle Office of Sustainability and Environment (SBW 2003) quantified BCA
ratio values for incorporating LEED in two state office buildings. Results showed BCA ratios of

118

0.78-1.1, 1.49-2.16, and 1.19-1.72 when considering only direct financial benefits, including
indirect benefits, and considering community-wide benefits, respectively. The Rocky Mountain
Institute was able to quantify worker productivity gains from eight different work-environment
case studies with lighting upgrades (Romm and Browning 1994).

Larsen et al. (2012b) reported that approximately 40% of public sector projects reported the
inclusion of non-energy benefits (NEBs) in their EPC projects, mostly generated by O&M
savings. Birr and Singer (2008) analyzed the barriers to the use of NEBs in EPC retrofit projects.
They suggested that blending the value of quantifiable NEBs into the project mix of longer and
shorter payback measures would add value and be attractive to public customers. Survey
participants identified data availability and management; however, O&M savings and avoided
utility system costs were cited as regularly-included NEBs. Other areas for monetizing and
including NEBs included the use of emissions reduction markets, avoided environmental
compliance savings, and health and productivity benefits. While Birr and Singer (2008) reported
that the latter category of NEBs might be the most difficult to quantify, Singh (2009) empirically
demonstrated the relationship between green construction principles and these indirect benefits,
and Berghorn and Vallad (2013) reported anecdotal evidence to support such claims after an
EPC retrofit was performed in a correctional facility. Jennings and Skumatz (2006) found
positive BCA ratio values as high as 3.1when considering NEBs from commissioning projects.

2.4.2.2 Example LCCA Applications
Chew et al. (2004) utilized a combination of quality parameters and LCCA to develop a
framework for making building maintainability decisions. This framework was applied to

119

decisions made during the design phase in an attempt to develop buildings that are more
maintainable and cost effective. Frangopol and Liu (2007) demonstrated an approach whereby
maintenance and management decisions for deteriorating infrastructure were made using an
optimization method that considered structure condition, safety, and life cycle cost. Furuta et al.
(2004) took a similar approach, examining service life as one of the decision factors for
optimization.

Lee et al. (2013) took a different approach, using LCCA as an integral part of a simulation to set
target building performance levels and allowable financing for construction costs. The simulation
was suggested for use by borrowers and lenders during early phase decision making, and it
reduced uncertainties related to project cost-related risk (level of investment needed for retrofit)
and project performance risks (the level of performance achieved after retrofit to attain positive
cash flow).

Consideration of Indirect Benefits: Lucuik and Meil (2004) examined the complete life cycle
costs of various designs of commercial wall systems. They found that the total life cycle costs
changed significantly when incorporating indirect costs, defined in this case as broader societal
environmental impacts typically found in a life cycle analysis. Carter and Keeler (2008) were
able to monetize and include tradable emissions values for nitrogen oxides (NOx) as part of an
LCCA of green roof systems. Gu et al. (2007) developed an integrated environmental-economic
LCCA method called the life cycle green costs assessment method that monetized environmental
loads through the assumption of an emissions tax that could be incorporated into the initial and
operating costs of the building.

120

2.4.3 Addressing Risk and Uncertainty in LCCA
Nearly half of the papers reviewed by Korpi and Ala-Risku (2008) were deterministic, meaning
that they arrived at a life cycle cost value without considering uncertainty in the analysis. Kapp
and Girmscheid (2005) articulated important sources of uncertainty in building projects owing to
their relatively long life cycles (30-100 years), to include challenges in predicting exact future
values of inflation and price increases and the useful lifetime of individual assets/investments.

Emblemsvag (2001) proposed an activity-based LCCA framework. This method emphasized the
modeling of uncertainty using Monte Carlo simulation, which facilitated the identification of
critical success factors, and in this case contributed to long term profitability. Kirkham et al.
(2002) developed probability density functions from a data set of over 450 hospitals across the
United Kingdom in order to accurately model the relationship between physical characteristics of
these buildings and their facility management costs.

Zhu et al. (2012) used a probabilistic LCCA method to compare building mechanical system
options. They examined the feasibility of installing a ground source heat pump versus a single
zone split system that used heat pumps. The probabilistic LCCA was facilitated through the use
of Monte Carlo simulation, and utilized directly collected data as well as data from the literature
and published performance data. By comparing deterministic and probabilistic methods of
conducting the LCCA, the authors found that the latter method yielded more reliable conclusions
and provided more critical information to the decision maker. Garber et al. (2013) also employed
Monte Carlo simulation to provide a risk-based assessment of a ground source heat pump
compared to four other potential HVAC systems. They employed an energy simulation platform

121

called TRNSYS to parameterize key input values for components of the system under review.
Brink (2012) analyzed upgrade options for a municipal water system and divided input data into
three types: (1) cost inputs (dollars), (2) schedule inputs (years), and (3) cost drivers (value).
Minimum, most likely, and maximum values for each of the 20 inputs were used to construct
Monte Carlo distributions. The analysis revealed different results using a deterministic LCCA
model, thereby supporting the need to more broadly implement probabilistic models to obtain
more accurate results.

2.5 CHAPTER SUMMARY
This chapter identified and reviewed literature related to three broad categories of energy
performance contracting, risk in construction engineering and management, and life cycle cost
analysis. The first category gave an overview of the EPC method, its history and current use, and
information categories used when developing EPC retrofit projects. The second category
examined the nature of risk and concepts of decision making under conditions of uncertainty and
risk. That was followed by an overview of risk in construction and EPC, a discussion of the risk
management process and various risk evaluation methods, and finally an overview of expertise
elicitation and use of the Delphi technique. The third section reviewed basic concepts of LCCA
and examined its application in the built environment and how concepts of risk and uncertainty
are addressed methodologically.

122

CHAPTER 3
RESEARCH METHODS AND DATA COLLECTION

3.1 CHAPTER OVERVIEW
This chapter presents a discussion of the research methods used in this study. This research
includes four objectives, as described in section 1.3. The conduct of the research and connection
to project objectives is presented in this chapter and was previously depicted in Figure 1-5.

Objective 1 steps include a preliminary ESCO expert interview as well as a literature and
document review which summarizes the EPC process; the use of EPC in the MUSH market,
specifically correctional facility retrofit projects; risk identification, classification, analysis, and
mitigation in EPC retrofits; LCCA methods and sources of uncertainty and risk in LCCA; and
analytical methods for LCCA and risk management. Collectively, this information was used to
construct a preliminary ESCO risk framework and identify a proposed risk management method
to be used in this research; this literature review and preliminary risk framework are presented in
Chapter 2.

Work steps in Objective 2 add to what is learned from the preliminary interview and the
literature and document review by eliciting expertise from ESCO professionals about sources of
project-level risk to contractors related to non-attainment of the performance guarantee in EPC
retrofit projects, including an assessment of the most critical risk categories. This data was used
to construct an EPC retrofit process model and refine the preliminary risk framework from
Objective 1.

123

Objective 3 work steps are primarily concerned with thoroughly evaluating and analyzing the
most important risk categories, as identified in Objective 2. A cost-based risk analysis and
evaluation approach will be used to assess risk criticality based on its economic impact
throughout the project life cycle. This in turn leads to the development of a framework for a life
cycle cost-based risk model, which is parameterized in the final work step of Objective 3.

Pilot application of the life cycle cost-based risk model and its refinement is the focus of
Objective 4. Primary activities directed toward this final research objective include the
evaluation of a correctional facility EPC case study using the risk model.

3.2 OBJECTIVE 1 – ENERGY PERFORMANCE CONTRACT RETROFIT PROCESS
The purpose of the work conducted in Objective 1 was to construct greater understanding of the
EPC retrofit process and to gain insight into the types of risks faced by ESCOs when performing
such projects. The work in this objective was initiated through a comprehensive review of the
literature which provided the necessary theoretical and current state-of-knowledge underpinnings
for the dissertation. Three categories of literature were reviewed to provide appropriate context
for the research: (1) energy performance contracting, (2) construction and EPC risk, and (3) life
cycle cost analysis. The work related to Objective 1 and its relation to other objectives is
depicted in Figure 3-1.

The EPC literature was reviewed primarily within the context of MUSH market projects, to
understand its various contractual forms, its history and current use in the MUSH market, and the
types of information needed during planning and design phases. The risk literature was reviewed

124

Figure 3-1. Objective 1 Methods and Outputs
to understand general concepts of decision making under uncertainty and risk, frameworks for
the assessment and management of project- and technical-based risks, applications of these
frameworks to construction and building projects methods of risk evaluation tailored to the
research undertaken in this thesis, and a discussion of methods used for the elicitation of
expertise in a risk management process. The LCCA literature was reviewed to understand how
concepts related to life cycle costing are applied in the built environment and how uncertainty
and risk are addressed in this analysis. The first two focus areas from the literature review were
used to develop the expertise elicitation strategy in Objective 2 and the risk analysis and
evaluation strategy in Objective 3. The latter area was used to inform the parameterization of the
risk framework in Objective 3 and model application in Objective 4.
125

The primary deliverable of the work conducted in Objective 1 was the development of a
preliminary ESCO risk framework for MUSH building EPC retrofits, presented in Chapter 2.
This framework was developed through the review of the two literature categories identified
above, along with an interview with an EPC professional with over 30 years of experience. The
preliminary framework was focused on identifying broad categories of relevant risks, termed risk
categories, with example risks and causes belonging to each category. This deliverable served as
an input to the work undertaken in Objective 2, which refined the framework through elicited
expertise from a panel of ESCO professionals with regard to comprehensive risk identification,
risk importance, risk causes, and mitigation activities.

3.3 OBJECTIVE 2 – RISK FRAMEWORK FOR MUSH MARKET EPC PROJECTS
Objective 2 consists of two sub-objectives: (1) Objective 2a is focused on development of an
expertise elicitation strategy and (2) Objective 2b addresses data collection and analysis of
elicited expertise, as it relates to refinement of the risk framework first developed in Objective 1
and development of an EPC retrofit process model. In the context of the risk management
process described in Chapter 2, data collected in Objective 2 is used to complete the risk
identification step.

3.3.1 Objective 2a – Energy Performance Contract Expertise Elicitation
The central data collection activity of this research is the elicitation of expertise from ESCO
practitioners who have had significant roles in developing, designing, and executing EPC
retrofits in the MUSH market. This section discusses the development of a research approach
that maximizes the ability to collect and analyze qualitative data in the form of expert knowledge

126

and judgments, which are framed by individual attitudes and approaches to risk. A strategy for
eliciting this expertise is presented in this section, along with a discussion of considerations
made when constructing the sample population for this study.

3.3.1.1 Application of the Delphi Technique to Construction Engineering and Management
This research is premised largely on qualitative information. As discussed in chapters 1 and 2,
EPC retrofit projects rely significantly on expert judgment and implicit knowledge to make
decisions under conditions of uncertainty and risk. Furthermore, as discussed in section 2.3.2,
such expertise-based decision making is subject to cognitive biases. As a result, a research
method is needed that can capture the judgment of experts and resolve any apparent differences
among expert opinions. The Delphi technique has been proposed as a useful research method for
construction engineering and management research that addresses limitations inherent in existing
experimental techniques due to the complexity of the domain and the presence of biases among
research subjects (Hallowell and Gambatese 2010). This method has been found to be
particularly applicable when source agreement is desired through a refereed process to collect,
aggregate, and organize expertise from potentially unique or divergent information sources (Pill
1971; Powell 2003; Yousuf 2007). Furthermore, Yousuf (2007) stated that an advantage of the
method is its ability to collect such data when time, distance, and other logistical factors make it
difficult for such an expert panel to be convened in a single in-person event.

As described in Chapter 2, The Delphi technique is a group process that is facilitated by the
researcher in order to elicit knowledge from a qualified panel of experts to reliably structure the
group’s communication (Linstone and Turoff 1975). Experts participate in multiple rounds of

127

questionnaires until a pre-determined measure of consensus is reached. While achieving
consensus among participants has typically been associated as an overarching goal of Delphi,
methodological variations exist that instead seek to create alternatives or address “wickedly
unstructured problems” through the presence of conflict among panelists (Mitroff and Turoff
1975). The policy Delphi seeks to generate strongly opposing views regarding potential
resolutions to a policy issue (Turoff 1970; Turoff 1975). The decision Delphi seeks to create the
future reality, rather than predict that future condition, which is typically the focus of the reliance
on building consensus in traditional Delphi (Rauch 1979). The decision Delphi then aids the
decision-making process through analysis of the newly constructed reality that was created by
panelists, comprised of decision-makers from the domain being studied (Rauch 1979).

Since the goal of this study is to fully explore and construct an understanding of risks present in
the EPC retrofit process and to evaluate those risks as they relate to retrofit decisions made
throughout the project development and execution process, a hybrid Delphi technique is used for
this study, combining the traditional and decision Delphi methods. As a result, panelists must be
recruited from among domain experts who occupy positions of decision-making authority within
their organizations. Regardless of the method used, the use of Delphi contributes to a robust
qualitative approach through the following methodological aspects:


Accuracy – obtained through knowledge elicited from a properly constructed panel of
experts (Hallowell and Gambatese 2010);



Precision – obtained through consensus (traditional Delphi), iterative rounds of feedback
(traditional Delphi and decision Delphi), and knowledge construction (decision Delphi)
(Rauch 1979; Hallowell and Gambatese 2007);

128



Minimize bias – assured through anonymity (Hallowell and Gambatese 2010), although
quasi-anonymity (participants were known but individual responses were anonymous)
was shown to improve response rates due to panelists wishing to provide input to
minimize the chance that their competitors would have undue influence (Rauch 1979).
This study only identified participants by company name (when such a release was
consented to) or by description (e.g., N nationwide independent ESCOs have
participated, N nationwide equipment manufacturer ESCOs have participated); and



Account for judgment – Delphi enables researchers to study questions which do not lend
themselves to traditional statistical analysis due to the nature of the question, lack of
objective data, or lack of agreement among professionals (Hallowell and Gambatese
2010).

Participant selection is critical, as the success of a Delphi study depends largely on the
expertise of the participants comprising the panel (Powell 2003). Hasson et al. (2000) suggested
that selection of participants requires a balance between finding those with interest in the subject
(so that they remain engaged throughout the Delphi study), which may lead to bias, and those
with current domain knowledge who are impartial to the outcome so as to provide unbiased
information. Diversity among the panel members may bring different perspectives to the study
and enable a wider range of alternatives (Murphy et al. 1998); however, this effect may be
minimized in large panels that are overly heterogeneous. Keeney et al. (2006) reported a lack of
guidance on the appropriate panel size, with reviewed studies ranging from hundreds to
thousands of participants. Murphy et al. (1998) advocated for larger panels despite their own
assertion that there is little empirical data to support a connection between panel size and

129

reliability or validity of the research. Hasson et al. (2000) identified panel sizes in the literature
of between 15 and 60 participants, and cautioned that larger panels lead to the generation of
additional data, which can cause problems during analysis. Hallowell and Gambatese (2010)
provided guidance on the selection of panel members relative to Delphi studies in the
construction engineering and management domain. Their research recommended panels
consisting of 8-12 participants so long as the panel is diverse and members are highly-qualified
within their domain.

Regardless of panel size, the literature is clear that Delphi panels are not statistical samples of a
larger population and thus should not be treated as such (Hasson et al. 2000; Keeney et al. 2006;
Duah 2014). The Delphi panel may be constructed using criteria sampling or purposive sampling
techniques (Hasson et al. 2000). Purposive sampling is defined as a non-probability sampling
technique where the researcher exerts influence on the selection of the sample based on
participants’ expert knowledge of the domain being researched or their willingness to participate
(Oliver 2006; Tongco 2007); this definition is particularly applicable to Delphi research.

The identification and selection of panelists for inclusion in this research utilized three purposive
sampling methods identified by Teddlie and Yu (2007): (1) expert sampling, (2) snowball
sampling, and (3) critical case sampling. Identification and selection of panelists was based on
the guidelines presented by Hallowell and Gambatese (2010) for construction engineering and
management and Duah (2014) for energy efficient retrofit. In considering these existing
guidelines, several selection criteria were difficult to align with the overall scheme of EPC
retrofits, due to the lack of literature focused on this domain as compared to construction

130

engineering and management in general. Additionally, Duah’s (2014) expertise framework
focused specifically on residential energy retrofits, a related but separate domain from EPC
retrofits in commercial buildings. Given these variations, this research defines analogous
relationships among selection criteria and EPC retrofits, as needed. For example, Duah (2014)
requires 10 residential retrofits per year to define expertise; one MUSH market retrofit project
can have a longer project life cycle than 10 residential retrofits. Based on the foregoing, expert
selection guidelines used in this research are provided in Table 3-1.
Table 3-1. Guidelines for Expert Selection
Identification and Recruitment (based on Hallowell and Gambatese 2010)
 Member of the National Association of Energy Service Companies (NAESCO)
 Employed by a NAESCO-accredited company
 Member of American Correctional Association (ACA) Clean and Green Committee
 Member of the Construction and Maintenance Institute for Criminal Justice Agencies (CMI)
 Past presenter at ACA or CMI conferences or the Green Prisons Annual National
Symposium on Sustainability in Corrections – topics focused on EPC retrofits
 Previous participant in similar expert-based studies
Expert Qualifications (at least five out of nine qualifications must be met)
 8 years EPC experience with no break in industry work during that time (Duah 2014)
 Work on at least 8 previous EPC projects in a professional (design, engineering, or
construction) capacity (based on Duah 2014)
 Average project impacted at least three building systems (e.g., lighting, central plant
decoupling, envelope) (based on Duah 2014)
 Active role in seeking financing and financial incentives (based on Duah 2014)
 Professional registration - CEM, CxA, PE, or AIA (based on Hallowell and Gambatese 2010)
 Advanced degree in engineering or a related field to building construction, operations,
design, or energy (at least BS) (based on Hallowell and Gambatese 2010)
 Faculty member in an energy engineering or management program at an accredited college
or university (based on Hallowell and Gambatese 2010)
 Two peer-reviewed publications or one book/book chapter related to EPC (based on
Hallowell and Gambatese 2010)
 Invited conference presenter (based on Hallowell and Gambatese 2010)
Based on Duah’s (2014) attribute weighting and Hallowell and Gambatese’s (2010) flexible
scoring system, the expertise scoring rubric in Table 3-2 was developed in order to provide a
relative measure of expertise among the panelists. The rubric is connected to the expert selection

131

Table 3-2. Scoring Rubric for Expert Determination
Experience
At least 8 years professional EPC experience
At least 8 previous EPC projects
At least 75% of EPC project experience is in the MUSH
market
At least three building systems impacted per project, on
average
Role in financing and financial incentives per project
Professional registration
Advanced degree in a related field (BS or higher)
Faculty member
Peer-reviewed publications
Invited conference presenter

Points
1
1
1.5
0.5
0.5
2
2
1
1
0.5

guidelines in Table 3-1, and emphasizes professional experience in the EPC process, particularly
in MUSH market projects, involvement in building systems and financing/incentive aspects of
projects, professional registration, and advanced education in a related field. While a threshold
value of points was not used, as was the case with expert qualification categories, a goal was
established that panelists would have a score of at least 4.5 points out of the 8.5 points available
for professional experience, registration, and education.

Reaching consensus among panelists is an important part of traditional Delphi, and is important
for several of the questions posed to participants in this study. The literature includes several
measures of consensus and many threshold values to establish the point at which consensus has
been reached. This study utilized a two-round Delphi questionnaire to investigate consensus- and
non-consensus- (e.g., knowledge construction) driven aspects of ESCO risks in the EPC retrofit
process. The first Delphi round was preceded by the development of the preliminary risk
framework, which was the result of a literature review, an open-ended interview with a domain
expert, and a semi-structured interview with a test panelist possessing over 30 years of

132

experience in correctional facility energy retrofits. The test panelist interview was used to
finalize development of the questionnaire for the Delphi panel; the final version of the instrument
was re-administered to the test panelist concurrently with administration with the rest of the
panelists. A second round of the Delphi questionnaire was employed for questions where
consensus was not reached in round one.

A variety of statistical parameters are commonly used in Delphi studies to indicate when
consensus has been reached, and include percent agreement, mean, median, standard deviation,
and/or mode (Hasson et al. 2000; Hsu and Sanford 2007; Hallowell and Gambatese 2010). This
study uses percentage of agreement among panelists which is a commonly reported parameter. A
predetermined percentage agreement of 70% or more was used as an indicator of consensus
(Duah 2014).

The questionnaire also included the ability to detect the difference between participants
achieving consensus and achieving agreement. Evans (1997) questioned the working definition
of consensus based on the review of 30 consensus-based expert panels in pharmacoeconomics
research. The research revealed three uses of the term: (1) views that are “acceptable” to all
panelists, (2) the same view that is held by all panelists, and (3) the majority view. Keeney et al.
(2006) explained that most studies opt to measure the extent to which participants reach
agreement with one another, fundamentally ignoring whether the “correct” answer has been
found or whether true consensus has been reached. To address this concern, several items in the
questionnaire elicited expertise with regard to elements of the preliminary risk framework. This
enabled the researcher to capture panelists’ reactions to information obtained through the

133

literature review and the preliminary expert interview and provided them the ability to add other
risk-related information, as necessary. This approach deemphasized the need for panelists to
merely agree with one another, and instead sought triangulation among multiple data sources
represented in the preliminary risk framework and provided participants with the opportunity to
contribute their expertise to the construction of reality represented by the decision Delphi
questions.

3.3.1.2 Elicitation Strategy
Duah (2014) provided a complete review of knowledge elicitation techniques to address
problems related to: (1) eliciting tacit or intuitive knowledge from experts, (2) eliciting
knowledge from experts that may be unavailable due to their difficulty in articulating this
knowledge, and (3) having a knowledge elicitor with limited domain knowledge. This research
generally follows the framework for elicitation described by Duah (2014). The elicitation
strategy for this research consisted of two elements: (1) knowledge elicitor training and (2) the
selection and use of appropriate elicitation techniques.

Knowledge elicitor training was used to help achieve a principal goal in the elicitation process,
to help experts articulate their knowledge which is often unstructured and tacit, in order to
document expert knowledge in a usable format (Duah 2014). The training of the knowledge
elicitor included a review of existing research and industry-specific literature as well as
interactions with members of the ESCO industry beyond just the administration of the Delphi
questionnaire. The first part of this training, the literature review, was presented in Chapter 2 and
included information about the energy performance contracting industry, sources of contractor’s

134

risk in construction engineering and management, and cost-based approaches to assessing risk.
Industry interactions included appointment of the researcher to a national sustainability in
corrections research committee, work with the Clean and Green Committee of the American
Correctional Association, presentation of an EPC industry workshop at the 2nd National
Symposium on Sustainability in Corrections, and participation in an EPC industry panel
presentation during the American Correctional Association 2014 Winter meeting. All
interactions have focused on the use of EPC retrofits and have involved work with facility
management professionals and ESCO staff in business development, engineering, construction,
and project management functions.

Since the selection of knowledge elicitation techniques depends on characteristics of the
domain and the experts under study, this research followed the framework for selection of
elicitation techniques provided by Duah (2014) given the similarity between domains – energy
efficiency retrofits – while making appropriate changes to reflect the differences between
residential retrofits and MUSH market EPC retrofits. Based on Duah’s (2014) framework for
selection of elicitation techniques, semi-structured interviews, job shadowing, and the Delphi
technique have been selected for this research.

Semi-structured interviews were used to collect data from the Delphi panel, which was
preceded by the preliminary EPC expert interview and the test panelist interview. These were
conducted as open-ended interviews in order to better explore key concepts related to the
research objectives. The test panelist interview was also used to clarify information gaps
remaining after the literature review and to test the structure of the questionnaire which would be

135

administered to the complete Delphi panel. The final semi-structured questionnaire was
developed based on refinements to the previous open-ended documents, and focused less on
capturing basic domain knowledge (e.g., questions such as, “please describe the phases of an
EPC retrofit project” were rejected) and instead focused on tacit knowledge, non-linear
information, and probative questioning, such as questioning about controls used to manage risks
identified as most important under various project conditions. The preliminary and finalized
questionnaires are provided in Appendix A.1 and Appendix A.2, respectively.

Job shadowing of experienced ESCO and correctional industry facility energy management
professionals was undertaken in order to gain fundamental knowledge about the early phases of
the EPC retrofit process in MUSH market buildings,. The objective was to observe their early
interactions when the investment grade audit (IGA) was being conceptualized, developed, and
finalized, in order to gain insights about how the retrofit decision process occurs and the types of
information used, as well as to gain experience with the risk-based decisions made by ESCOs
during the earliest phases of EPC projects. This participation also enabled the researcher to
become familiar with the language used by ESCOs in order to facilitate better communication
with panelists during the Delphi rounds. This was also critical during data compilation and
analysis in order to establish the basis for identifying emerging themes. The researcher was
invited to observe these otherwise closed meetings. So as not to interrupt the meeting and ESCOclient interactions with questions, requests for further information or clarification were made
after the conclusion of each meeting and through follow-up emails and meetings with the ESCO
team members.

136

The Delphi technique is especially useful for unstructured and semi-structured problems which
rely less on inferential statistics and more on the knowledge of domain experts (Pill 1971; Powell
2003). The ability to build consensus among experts using a refereed communication process for
areas of disagreement or to construct knowledge in complex decision scenarios are hallmarks of
the Delphi technique that make it particularly useful for this research, as is the knowledge
construction aspect of the decision Delphi (Linstone and Turoff 1975; Rauch 1979). Since this
research is focused largely on development, audit, and design phase activities (e.g., procurement,
IGA completion, and retrofit design) based on input gathered from experts with moderate and
high levels of experience, the technique is well-suited to this project.

As shown in this section, a combination of elicitation techniques was employed for this research,
which placed an emphasis on knowledge elicitor training and used a hybrid of the traditional and
decision Delphi techniques as the dominant elicitation method (Figure 3-2). This figure also
depicts the activities conducted in Objective 2.

3.3.2 Objective 2b – Data Collection and Analysis - Refined Risk Framework
Data was collected following the EPC risk expertise elicitation model depicted in Figure 3-2.
Based on Hallowell and Gambatese’s (2010) review of Delphi studies in construction
engineering and management, this project sought to impanel between 12 and 20 domain experts
over two rounds. A third round was considered, if needed, to deeply explore the reasons for
outlying responses if any exist after round 2; however, the research points to diminishing returns
in achieving consensus after the third round (Hallowell and Gambatese 2010).

137

Figure 3-2. EPC Risk Expertise Elicitation Process Model and Objective 2 Methods and
Outputs
3.3.2.1 Data Collection
Participants whose expertise about EPC retrofit project risks was to be elicited were selected
based on the qualification rubric described in section 3.3.1. The characteristics of the panel and
determinants of expertise for each participant are provided in Appendix B. In total, 19
participants were included in the Delphi panel – ten represented independent ESCOs, seven
represented equipment manufacturer-based ESCOs, and two represented utilities. This
distribution approximates the nationwide share of ESCO ownership, and was an important part
of the purposive sampling strategy employed by this study.

138

The expert questionnaire was divided into four parts: Part I - Risk Tolerance; Part II –
Professional Experience and Determination of Expertise; Part III – Risk Management Process,
and Part IV – Life Cycle Cost Analysis. Part I questions used the TIAA-CREF Asset Allocation
Evaluator (AAE) to assess each panelist’s attitudes toward risk by using the analog of allocating
assets in a financial portfolio (TIAA-CREF 2013). Out of 31 total questions, six were for Part I,
11 were for Part II, ten were for Part III, and four were for Part IV. Data collected in Part IV of
the questionnaire was not included in this research; rather, the intent of these questions was to
explore possible future research related to concepts of life cycle costing in considering nonenergy benefits (NEBs) and potentially non-recoverable project costs. As a result, data collected
from Part IV questions are not discussed further in this dissertation.

Consensus was not sought for questions in parts I and II, as those parts were focused on
identifying and classifying participants’ levels of expertise and their risk tolerance. Consensus
was sought for four of the ten Part III questions. Consensus was not sought for questions that
sought to elicit information about specific decision processes used in the identification and
evaluation of risks, as well as the causes of the most significant risks and the methods used to
control them on projects. Those questions were intended for “knowledge construction” as
previously described in the discussion of the decision Delphi (Rauch 1979). This information is
summarized in Table 3-3.

3.3.2.2 Data Analysis
Data was analyzed following the elicitation process model in Figure 3-2. Key aspects of this
process included elements of knowledge elicitor training, observance of emerging patterns in the

139

Table 3-3. Summary of Question Focus Areas and Need for Consensus
Question No.
Question Focus
Consensus Sought
Part I – Risk Tolerance
TIAA-CREF 1-6
Risk Tolerance
Not Needed
Part II - Professional Experience and Determination of Expertise
1a-1k.
Expert Criteria
Not Needed
Part III - Risk Management Process
2a.
Risk Identification
Yes - Knowledge Elicitation
2b.
Yes - Knowledge Elicitation
2c.
MUSH Market-Specific Risks
Yes - Knowledge Elicitation
3a.
Risk Identification Methods
No - Knowledge Construction
3b.
Risk Identification Timing
No - Knowledge Construction
3c.
Risk Identification Responsibility No - Knowledge Construction
3d.
Risk Evaluation Methods
No - Knowledge Construction
3e.
Identification of Most Important Yes - Knowledge Elicitation
Risk Categories
4a.
No - Knowledge Construction
4b.
No - Knowledge Construction
data, and inferences that were drawn from both. The application of knowledge elicitor training to
data analysis centered on the use of the preliminary risk framework to evaluate responses.
Additionally, EPC professionals encountered through industry interactions provided guidance
and feedback throughout data analysis and construction of intermediate outputs.

Findings and inferences from elicited expertise are provided in Chapter 4, Appendix C.1, and
Appendix C.2. The data analysis resulted in inputs to the refined risk framework and to the EPC
retrofit process model. These outputs are briefly described below and are fully discussed in
Chapter 4. These are the primary deliverables from Objective 2.

3.3.2.3 Refined Risk Framework Construction
The elicited expertise from the Delphi panel was analyzed in the context of the preliminary risk
framework developed in Chapter 2. Refinements were made to the 12 original risk categories and

140

the risks assigned to each. Further information about specific ways in which these risks are
realized on EPC retrofit projects and mitigation strategies used for their control were included,
and a risk importance scoring was used to identify those risks that panelists identified as being
the most important toward meeting the performance guarantee. The refined risk framework is the
primary output of this objective and is described in Chapter 4.

3.3.2.4 Energy Performance Contract Retrofit Process Model
During development of the refined risk framework, it became apparent that the quality of the
IGA has significant influence on the overall level of project risk. Many of the causes and
mitigation strategies elicited from panelists were focused on issues related to information quality
and availability, which could be expected given that a significant activity during the IGA is the
collection and analysis of large quantities of information about the facility under review. Much
of the elicited expertise addressed the types of information used during each phase of the EPC
retrofit process to assist in the decision making process. As a result, the development of an EPC
retrofit process model provides the ability to analyze key decisions made throughout the project
life cycle. This model was also used to establish the system boundary for the risk analysis and
evaluation process in Objective 3.

3.4 OBJECTIVE 3 – RISK ANALYSIS AND EVALUATION
The focus of work in Objective 3 is to develop an appropriate risk analysis and evaluation
strategy for EPC retrofits that has the ability to incorporate life cycle costing as a measure of
risk. Since these projects have lengthy performance periods (i.e., 12-15 years is common),
unmitigated risks that become apparent after the contract is signed can lead to significant

141

financial implications for ESCOs. Delphi panelists provided a risk importance score for each
risk category in the preliminary framework. Importance scores were analyzed and resulted in the
inclusion of energy audit quality- and ECM selection and installation-related risks in Objective 3
risk evaluation and analysis activities.

As discussed in Chapter 2, a single method for conducting risk evaluation and analysis was
sought that incorporated qualitative techniques (McKim 1992), was applicable to early design
phases (Grimaldi et al. 2012), that favored the use of expert knowledge, that enabled the
incorporation of life cycle costing as part of the risk evaluation, and was consensus-based
(Raftery 1994). Based on a review of Grimaldi et al. (2012), failure mode and effects analysis
(FMEA) was selected as the risk evaluation tool because it is widely understood, it provides a
comprehensive framework for data collection and understanding of complex system dynamics
(Grimaldi et al. 2012), and places of a focus on understanding the primary modes of system or
product failure resulting from technical factors (Stamatis 2003). Since the focus of this research
is a system failure (i.e., failing to meet the performance guarantee as a result of project-level
risks), FMEA is particularly well-suited. Additionally, FMEA addresses some of the
shortcomings of other methods identified by Grimaldi et al. (2012), including single event focus
(event trees), overlooking dependencies among systems elements (event trees), missing the
nature of risks (interviews), and a lack of appropriate ranking tools (risk index).

Two types of FMEA are widely used – the design FMEA and the process FMEA. Both are
conducted at system, subsystem, and component levels (Stamatis 2003; McDermott et al. 2009),
indicating the ability to analyze risks at various life cycle phases. Conducting the FMEA in this

142

way requires multiple analyses, which must be conducted sequentially, in order to examine the
complete life cycle of a design or process (Kmenta and Ishii 2005). This has been identified as a
methodological shortcoming due to the additional analysis time, the need to impanel a larger
number of experts, and the linear nature of the “nested FMEAs” over a design or process life
cycle, which prevents an iterative analysis of complex risk relationships (Kmenta and Ishii
2004).

Another FMEA method variant (FMECA) includes a measure of risk criticality; it is worth
noting that this has increasingly become incorporated in standard FMEA methods (AIAG 2008).
Risk criticality is typically determined through the calculation of the risk priority number (RPN);
however, there are shortcomings of this method, as described in Chapter 2. Liu et al. (2013)
conducted a comprehensive review of literature related to the risk evaluation methods used in
FMEA, directed at addressing the shortcomings of the RPN method. Cost-based methods were
identified as one of the 33 distinct evaluation methods identified by the authors, and six papers
were reported as using this approach.

A life cycle cost-based approach to risk evaluation is desirable for this research since EPC
retrofits are premised on financial guarantees made by the ESCO, which must remain robust over
lengthy project performance periods. Much of the analysis in the IGA is presented in terms of
ESCO costs, cash flow, guaranteed savings, and project value, highlighting the importance of
developing a measure of risk that utilizes similar terms.

143

Figure 3-3 depicts the relationships among the works steps in Objective 3, as described in the
sections below. Data collection and analysis relative to this objective is fully described in
Chapter 5.

Figure 3-3. Objective 3 Methods and Outputs
3.4.1 Risk Analysis Strategy Development
Based on the desire to analyze risks in an iterative fashion throughout the project life cycle and
to utilize cost-based terms to express the measure of risk criticality, scenario-based FMEA
(SFMEA) was selected for further review. SFMEA and its methodological enhancements over

144

traditional FMEA methods were detailed in Chapter 2; a summary of relevant characteristics for
the development of the Objective 3 research methods are described below.

SFMEA was first proposed by Kmenta and Ishii (2000) and refined by Kmenta and Ishii (2004).
The goal of this method is to overcome barriers in conventional FMEA related to the handling of
interrelated risks and more accurately define and evaluate risk events. This method also uses cost
as a measure of risk, replacing the RPN which has been challenged by numerous authors. Costbased models were first proposed over 20 years ago; an early adopter of this method was
Gilchrist (1993). Additional cost-based models have been used by Rhee and Ishii (2002), Rhee
and Ishii (2003a), Rhee and Ishii (2003b), Spencer and Rhee (2003), von Ahsen (2008),
Carmignani (2009), Rhee and Spencer (2009), and Chen and Zhang (2012). These models
achieved the goal of evaluating risks by their cost impacts by beginning with a conventional
FMEA framework, and modifying elements necessary to incorporate cost information when
substituting for the traditional RPN method.

A key characteristic of SFMEA is the construction of failure scenarios, which replace the
analysis of single point failure modes found in traditional FMEA. Conventional FMEA is built
around failure modes, which describe the cause of a failure and have associated effects. Failure
modes may occur across various system levels, and the standard practice is to represent that
through the use of “nested” FMEAs, where the effects at a lower level become causes at the next
higher level, and so on. As described earlier in this section, these separate analyses lack
consistency in managing information across levels, thereby making a life cycle-based analysis
very difficult (Kmenta and Ishii 2004). There is typically a one-to-one relationship between the

145

causes and effects in a given failure mode, so a cause and effect chain is only represented by the
cause and the end effect – intermediate effects are not evaluated explicitly.

Scenario-based FMEA allows the creation of many possible “failure scenarios,” which are cause
and effect chains that can be lengthened, as needed, if new effects and new causes are identified
during analysis (Kmenta and Ishii 2000; Kmenta and Ishii 2004). Failure scenarios are found in
the risk literature (Kaplan and Garrick 1981) and have recently been applied in construction and
engineering research (Esmaeili and Hallowell 2013; Pallin 2013; Hamilton et al. 2014). Failure
scenarios were meant to be constructed by panelists participating in the FMEA. As a result, the
scenario development process and the scenarios themselves are akin to the construction of
Bayesian Belief Networks (BBNs), which were more thoroughly described in Chapter 2.

The construction of a BBN is a participatory decision-making strategy whereby expertise is
elicited from participants in a structured manner so that complex systems with a great deal of
uncertainty can be reduced through graphical representation (Hart and Pollino 2008). A key
strength of this approach is that complex causal relationships that are not easily expressed in
mathematical terms can be analyzed to gain insight into the relationships among variables under
study. This is especially relevant in this case, where data collected in Objective 2 was through
elicited expertise and was qualitative in nature. The construction of BBNs in the risk analysis
process permits FMEA participants to structure and understand the system under review.

BBNs use three probabilities to represent participants’ beliefs: (1) prior probability, (2)
conditional probability, and (3) posterior probability. These are incorporated in the SFMEA

146

method as the probability of the cause and the conditional probability of the end effect given the
intermediate effects present in the scenario. These probabilities are then analyzed using Baye’s
Theorem, which enables the belief network to be updated if new evidence causes a change to the
state of the network (Grinstead and Snell 1997). The use of point estimates of probabilities was
preferred by construction professionals studied by McKim (1992), who noted that these
individuals tend to assign discrete percentages to a likelihood of a risk as opposed to developing
probability functions. Thaheem and DeMarco (2013) also found that probability functions were
used by less than 30% of construction professionals responding to a risk management methods
survey. The use of SFMEA requires modifications to the traditional FMEA method. These are
identified and discussed briefly in the following sections.

3.4.1.1 Modified FMEA Matrix
A matrix was developed specifically for this research based on standard documents as provided
by AIAG (2008) with modifications per Kmenta and Ishii (20004). The matrices used for this
research are provided in Appendix D. Modifications to the matrix included the collection causes
and effects for each risk scenario, any specific facility factors that give rise to individual causes,
percent likelihoods (i.e., probabilities) for the risk cause and end effect given any intermediate
effects, and relative cost data for each risk scenario.

3.4.1.2 Modified Risk Criticality Measure
The purpose of the criticality analysis is to identify the risk scenarios having the greatest impact
on the ESCO’s financial performance in EPC retrofit projects. This research utilizes expected
cost as included in SFMEA as the primary measure of risk criticality (Kmenta and Ishii 2000;

147

Rhee and Ishii 2002; Kmenta and Ishii 2004). The authors modeled the relationships between
occurrence, severity, and detection and probability and cost. This effort gave rise to the notion
that a 1:1 relationship does not exist between the values of the RPN and expected cost. Criticality
measures were evaluated based on their relational differences, and three scenarios were detected:
(1) identical RPN values can have large expected cost variances, (2) identical expected cost
values can have different RPN values, and (3) the RPN and expected cost values can yield mixed
priorities for risk control (Kmenta and Ishii 2004). Additionally, the authors found that in 10,000
simulated FMEAs, the RPN priority and expected cost priority never matched. As a result, the
risk analysis strategy developed in this objective will also collect measures of occurrence,
severity, and detection from panelists to construct RPNs for each risk scenario. This data will be
used to empirically test the modeled relationships between RPN and expected cost observed by
Kmenta and Ishii (2004).

3.4.2 Analyze Risk Criticality
The risk analysis strategy utilizing SFMEA described above was conducted through a FMEA
panel of domain experts representing various facets of EPC retrofits. The recommended panel for
conducting a FMEA includes individuals who have professional responsibility and authority for
the system or process being evaluated (Stamatis 2003). Panels must reflect the multiple
disciplines represented in the process and system under review and multiple technical disciplines
should be represented, to the extent possible (Teng and Ho 1996; Stamatis 2003; McDermott et
al. 2009). Recommended panel sizes range from four to nine and should include participation by
end-users or clients (Stamatis 2003; McDermott et al. 2009). McDermott et al. (2009) also
recommend balancing the panel composition among participants with more experience and those

148

with less. The rationale for this is that less experienced panelists may perceive issues differently
than those panelists for whom these projects have become familiar, thereby potentially revealing
additional issues that could be missed by those who routinely perform this work.

A panel comprised of four professionals was assembled for this task. Panelist expertise and risk
tolerance measures are described in Chapter 5. All members were convened for a half-day
SFMEA meeting. An expert facilitator was retained to co-lead the process with the researcher.
This was critical to ensure that the SFMEA achieved its stated goals and analyzed risk scenarios
in a comprehensive manner. The researcher thoroughly explained the goals of the SFMEA and
shared the modified matrix with the facilitator prior to convening the panel. The EPC retrofit
process model was also explained, as was the refined risk framework, so panelists could begin to
understand the connections among the process model, the SFMEA goals, the development of
failure scenarios, and the cost-related information required of participants. A fellow researcher
was included as a secondary recorder and analyst to ensure that SFMEA details were transcribed
accurately and to assist with performing life cost-based calculations during the SFMEA panel.
Panelists engaged in an iterative process consisting of the following steps:
1. Construct risk scenarios – Panelists were provided with an example scenario, previously
constructed by the researcher, in order to visualize the related concepts of risk scenarios,
cause and effect probabilities, cost, and expected cost. Participants were also given a list
of the 27 risks belonging to the IGA quality and ECM selection and installation risk
categories. Members determined the priority order for these risks before beginning to
construct scenarios, aided by the facilitator and the researcher.

149

2. Identify critical facility factors – As risk scenarios were being constructed, panelists were
asked to provide a measure of specific facility-related factors that could give rise to
specific risk causes. For example, the age of the facility and the age of the last major
retrofit is related to the probability of the facility containing asbestos-containing
materials, lead-based paint, thus requiring necessary code upgrades.
3. Assign probabilities and relative costs – Panelists individually provided probabilities and
relative costs for each risk scenario. Relative costs were used to represent the
consequence of an individual risk scenario based on the percentage of the ESCO’s
margin, defined as the difference between calculated savings and guaranteed savings.
This was advantageous for two reasons: (1) panelists were readily able to assign a
percentage value of the margin that could be at risk in each scenario and (2) these values
captured the essence of the financial risk in each scenario without requiring significant
participant time to derive construction cost estimates for each individual scenario. Once
probabilities and costs were recorded for each panelist, consensus measures were
gathered using the median of the panelist values. The median was selected for its ability
to minimize the impact of outliers on this small data set, and provide a more robust
measure of central tendency among the four responses. Probability and cost data is
provided for each panelist and the median values in Appendix D.
4. Assign occurrence, severity, and detection scores – Panelists were then prompted to score
each scenario using traditional measures of criticality using custom tables developed for
this research (tables 3-4, 3-5, and 3-6). RPN values were calculated for each scenario and
also for the median case, and are presented in Appendix D.

150

Table 3-4. Occurrence Measure Ranking Scale
Likelihood of Failure Criteria: Occurrence of Causes
Very High
90+ in 100 projects
80 in 100 projects
High
70 in 100 projects
60 in 100 projects
50 in100 projects
Moderate
40 in 100 projects
30 in 100 projects
20 in 100 projects
Low
10 in 100 projects
Very Low
<1 in 100 projects

Rank
10
9
8
7
6
5
4
3
2
1

Table 3-5. Severity Measure Ranking Scale
Effect
Criteria: Severity of Effect on Project
Failure to Meet Safety or
May endanger lives of personnel.
Regulatory Requirements
May singularly lead to failure to achieve performance
guarantee or endanger safety of personnel.
Requires changes to the scope of work based on
Major Project Impact
client requirements and project goals; may endanger
future working relationship.
Requires recalculation of energy guarantee.
Requires recalculation of key IGA elements/findings
or similar.
Inconvenience to ESCO staff and client staff time or
Moderate Project Impact
after execution phase.
Inconvenience to ESCO staff time in or after
execution phase.
Slight inconvenience to ESCO staff and client staff
time in earliest project phases.
Minor Project Impact
Slight inconvenience to ESCO staff time in earliest
project phases.
No effect
No effect.

Rank
10
9
8
7
6
5
4
3
2
1

5. Conduct comparative analysis of expected cost and RPN – The calculated values of
expected cost and RPN were compared to detect the presence of a relationship among the
measures and to fully describe any discrepancies observed between them. A detailed
discussion of this analysis is provided in Chapter 5.

151

Table 3-6. Detection Measure Ranking Scale
Detection Likelihood and
Early Project Phase Timing
Criteria: Likelihood and Timing of Detection
Almost impossible to detect
Cannot detect until after energy guarantee and price
until execution phase
guarantee have been fixed by ESCO.
commences
Very remote chance of detection Slight chance of detection before energy guarantee
before execution phase
and price guarantee have been fixed by ESCO, but
only through significant additional scope completed
by ESCO.
Remote/May be after execution Failure mode may be detected before entering
execution through past project experience.
Very low/May be after
Failure mode may be detected before entering
execution
execution through modeled results.
Low/May be after execution
Failure mode may be detected before entering
execution through a combination of visual inspection,
laboratory testing, or direct measurement/observation.
Moderate/Before execution
Failure mode is detected before entering execution
through past project experience.
Moderately high/Before
Failure mode is detected before entering execution
execution
through modeled results.
High/Before execution
Failure mode is detected before entering execution
through a combination of visual inspection, laboratory
testing, or direct measurement/observation.
Very high/Before execution
Design solutions have been highly correlated with
early detection ability in previous projects of a similar
type.
Almost certain/Before
Failure mode cannot occur because it is addressed
execution
fully through design solutions.

Rank
10
9

8
7
6
5
4
3
2
1

6. Identify risk mitigation strategies – Risk mitigation strategies were developed from three
sources: (1) review of the literature, (2) elicited expertise from Objective 2, and (3)
interview with an ESCO expert. The refined risk framework contains a number of risk
mitigation strategies identified by the Delphi panelists; however, that list was not
complete for all identified risks. The literature review and interview with the ESCO
expert were used to provide missing information and confirm data obtained from the
panel. This final step of the SFMEA is discussed below and in Chapter 5.

152

3.4.3 Evaluate Critical Risks
Mitigation strategies were evaluated for each scenario, first using a Boolean operator; risks could
either be retained or allocated. In each case, data was collected regarding the efficacy of the
strategy in eliminating risk and the cost to undertake the activity was identified. This led to the
creation of multiple mitigation scenarios for each selected risk scenario, based on combinations
of the selected approach (i.e., retain or allocate), the effectiveness of the measure, and its cost.
These were used to construct cost-benefit relationships for each risk scenario and its attendant
mitigations strategies, thereby enabling decisions to be made about effective strategies to reduce
the risk of the most critical scenarios that are present in a given EPC retrofit.

3.4.4 Parameterize Risk Framework and Construct Risk Model
The principal output from Objective 3 is the life cycle cost-based risk criticality and mitigation
framework. By parameterizing that framework, a risk model is constructed to encapsulate the
qualitative and quantitative data collected from experts in objectives 2 and 3. Parameterization
consists of assigning functions and values for the following elements of the framework:


Connect identified facility factors with probabilities of cause;



Define relationships among the following four parameters and individual risk scenarios:
(1) equipment service life and replacement cost; (2) equipment maintenance cost; (3) fuel
cost escalation; and (4) weather-related variation;



Develop probability density functions for the four parameters listed above using external
data; and



Determine criteria weights for the four parameters listed above.

153

Criteria weights for each of the four parameters obtained from each SFMEA panelist will be
applied to the values of expected cost for each relevant scenario. Additionally, the SFMEA data
in Appendix D will be analyzed to detect whether a relationship exists between assigned
probabilities and costs and the risk tolerance and experience level of panelists. If such a
relationship is detected, this will be included as part of the framework parameterization to allow
end-users the ability to model risk scenarios based on either the panel-developed probabilities
and costs or user-specified probability and cost values. This is discussed in greater detail in
Chapter 5. Model construction is also described in Chapter 5 and consists of representing the
parameterized framework in a data collection and analysis tool that utilizes end-user inputs about
facility and EPC project characteristics to determine cost-benefit relationships for risks and
mitigation strategies, using automatic or end-user inputs for values of probability and cost.

3.5 OBJECTIVE 4 – PILOT LIFE CYCLE COST-BASED RISK MODEL
The life cycle cost-based risk model developed in Objective 3 will be pilot tested through
application to one case study site, selected in accordance with parameters described below, and
described in Chapter 6. Facility and EPC project characteristics from this case study will be used
to provide the inputs to run the life cycle cost-based risk model. Model implementation is
conducted primarily to test model usability and usefulness of outcomes. The model will be tested
for its applicability, that is, that it can be applied by users with available data in typical retrofit
projects during the energy audit phase. This is important given the potentially large number of
input parameters to the risk model.

154

Sensitivity analysis will be conducted to determine the response of model outputs to changes in
model parameter values. Since the framework was parameterized using responses from a panel
of four experts, the resultant data set is somewhat limited. Monte Carlo simulation can be used to
examine model performance when varying individual parameter values in an effort to address
model uncertainties resulting from the small data set. This process affords the ability to
determine the relative importance of individual parameters in influencing the modeled result and
provides insight for model refinement.

Implementation will also consider whether the model yields results that are feasible, and balance
risk mitigation, life cycle costs, and technical approach in assisting decision makers to better
evaluate project level risks. As a result of this final step and the sensitivity analysis described
above, the model will be refined, as needed, to address the issues of model applicability,
feasibility, and parameter development.

To pursue this objective, a correctional facility case study has been identified where an EPC
retrofit has completed within the past two years, and the facility is currently in the measurement
and verification phase. Key selection criteria included the presence of identified project risk
categories and life cycle cost implications as described by the Delphi panel. Additionally, a site
was sought that was located in either climate zone 5 or 6, in order to minimize variation in the
technical approach in response to climatic conditions. A facility was also sought that included a
representative sample of ECM strategies, to include short (e.g., <5 years) and longer (e.g., >5
years) payback periods and strategies that utilized stipulated performance as well as performance
monitoring and verification. A summary of case study characteristics is provided in Table 3-7.

155

Table 3-7. Selected Correctional Facility Case Study Characteristics
Facility Name
[Years Built]
Project Cost
Total Building
Project
(Total Savings)
Area (SF)
Construction Types and
Components [Total Annual Payback
[EPC Project SF]
Security Levels
Savings]
Period
 Primary structures are
 Lighting
Parnall
brick, pre-cast concrete
Retrofit
Correctional
block, steel, and glass
Facility
$12,891,626
 Water
 Additional housing units
[1925-2002]
($15,646,197)
10 yrs
Efficiency
are weatherized pole barns
[$1,881,274]

Retrofit
with sealed concrete floors
905,220
HVAC
and plaster-board walls
[~250,000]
 Security level I
Despite being public facilities, information about correctional facilities and physical access to
these structures can be difficult to obtain because of significant security concerns. The researcher
was able to obtain access to EPC project teams and decision makers, gain site access in order to
visually inspect the scope of installed ECMs, and access all EPC-related work products
(solicitations and contracts, investment grade audit results, and construction documents).

3.6 CHAPTER SUMMARY
This chapter described the research approach for the identification, analysis, and evaluation of
project-level risks faced by ESCOs in the development and execution of EPC retrofits in
correctional facilities and other MUSH market buildings. The four objectives include: (1)
attainment of knowledge about the EPC retrofit process; (2) elicitation of expertise from ESCO
professionals through the Delphi technique to construct a risk framework and a retrofit process
model; (3) analysis and evaluation of identified risks using SFMEA and expected cost, and
parameterization of this framework as a life cycle cost-based risk model; and (4) pilot application
of the developed risk model using a correctional facility case study.

156

CHAPTER 4
ENERGY PERFORMANCE CONTRACT RISK FRAMEWORK AND
RETROFIT PROCESS MODEL

4.1 INTRODUCTION
The need for increased energy efficient retrofit in the municipalities, universities, school, and
hospitals (MUSH) market and the role of energy performance contracting (EPC) in promoting
the wider adoption of energy efficient retrofit practices was established in Chapter 1. A number
of risk sources for energy service companies (ESCOs) undertaking MUSH market EPC retrofits
were discussed in the first two chapters, three of which have particular relevance to this research:
(1) ESCOs bear the primary project performance risk, which must be properly understood and
mitigated during early project phases; (2) EPC project decision making can be shaped by
cognitive biases, based on available information, during the development and design of EPC
projects; and (3) the availability and quality of information has a significant impact on activities
and decisions made during the energy audit phase and during retrofit design. These issues
informed the development of the research design presented in Chapter 3.

This and the following two chapters are grounded by the premise that ESCOs must fully
understand and control the risks that they bear throughout an EPC retrofit project at the earliest
project life cycle phases. This chapter is the first of three that focus on the risk management
process (Figure 4-1). This process involves understanding the context within which EPC retrofit
decisions are made, the types of information resources available for decision making, and the
identification of risks inherent in EPC retrofit projects. The result of this understanding was the

157

Figure 4-1. Risk Management Process and Methods
Mapped to Dissertation Chapters
concurrent development of a refinement to the preliminary risk framework presented in Chapter
2 and an EPC retrofit process model. Both outputs from this chapter help to frame the Chapter 5
work. The refined risk framework identifies the two risk categories assigned the greatest
importance by research participants, and the risks belonging to the selected risk categories
provide the basis for scenario development by identifying risk causes and potential mitigation

158

strategies. The EPC retrofit process model serves as the process map for conducting the scenariobased failure mode and effects analysis (SFMEA), establishing the context for the panel’s
analysis.

Chapter 5 is primarily concerned with the development and implementation of an analytical and
evaluative framework using SFMEA for the risks identified in this chapter. The complete
SFMEA serves as the framework to develop a life cycle cost-based risk model. That model is
implemented in Chapter 6 through a pilot application to an actual case study location – a
correctional facility that recently underwent EPC retrofit activity. Application of the model
results in completing the risk management process.

4.1.1 Glossary
Frequently used terms which may be used interchangeably in this chapter are defined as below:


ESCO: energy service company, contractor;



Owner: building owner, client;



Qualitative Information: expert knowledge, expertise, experience-based judgment, casebased reasoning, ESCO advice;



Quantitative Information: published information, databases, simulation results,
documents/reports, checklists, procedures, figures, pictures; and



Retrofit Measures: energy conservation measures (ECMs).

159

4.1.2 Chapter Objectives
The primary goal of this chapter is to identify project-level risks to ESCOs undertaking EPC
retrofits in MUSH market facilities. The work is premised on the preliminary risk framework
that was described in Chapter 2. Primary data to achieve this goal consists of expertise that was
elicited from ESCO professionals using the Delphi technique. In order to achieve these goals,
this chapter has the following objectives:


Refine the preliminary (a priori) risk framework by eliciting knowledge from a panel of
ESCO experts about risk identification and prioritization for further analysis;



Determine the presence of an effect between each panelist’s amount of ESCO experience,
risk tolerance, position type, and project focus and the risk identification decisions made
by each panelist;



To identify the key decisions made under conditions of uncertainty and risk during the
retrofit process; and



To connect information sources used throughout the retrofit process with the decisions
that are made in early project phases.

4.2 EXPERTISE ELICITATION
Primary data collection consisted of eliciting domain expertise from 19 ESCO professionals
following the strategy described in Chapter 3. Panelists were selected based on their expertise,
also as described in Chapter 3, and responded to a two round Delphi questionnaire consisting of
three categories of questions: (1) participant risk tolerance, (2) professional experience and
determinants of expertise, and (3) risk management process. This section includes a discussion of

160

the recruitment and selection of panelists, a summary of their levels of expertise, and key
findings and inferences drawn from the data collection effort.

4.2.1 Delphi Panelist Recruitment and Selection
Participants were selected for this study using purposive sampling techniques (Barbour 2001;
Oliver 2006; Tongco 2007), in order to focus on experts who are engaged in EPC work with a
variety of different ESCOs. The overarching goal of sampling was to ensure that only contractors
possessing a high level of knowledge about MUSH market EPC retrofit projects would be
included, and that those individuals had roles involved with decision-making during the retrofit
process. Expert sampling was therefore a cornerstone of the combination approach used;
however, identifying and gaining access to such experts was difficult in many cases. To help
recruit such individuals, a sequential sampling technique known as snowball sampling was used.
This method has been shown to assist in the recruitment of participants who are difficult to
access for a variety of reasons (Sadler et al. 2010). Upon completion of the questionnaire,
panelists were asked to provide referrals to other domain experts who met pre-determined expert
selection criteria and who would be likely to participate in this study. Experts were also
identified in connection with critical cases, which consisted of correctional facility EPC retrofits
from the past three to five years.

Even though constructing a representative industry sample was not the goal of participant
recruitment and selection activities, the panel was generally representative of the overall
ownership share of the ESCO market held by three main business types: (1) independent ESCOs
and other energy companies, (2) building equipment manufacturer ESCOs, and (3) utility

161

company ESCOs. Nationwide ESCO industry shares by business type for 2008 (the most recent
year for which data were available) are shown in Figure 4-2. The data shown for study
participants is based on the proportional representation of each type of business among the

Note: Due to rounding errors, ownership shares do not add to 100%
Figure 4-2. ESCO Industry Shares by Business Type
(Source: Larsen et al. 2012a – nationwide data)
panelists. Some observed variation between the study sample and ESCO population is present,
likely attributable to the use of snowball sampling and expert sampling in constructing the panel
(Sadler et al. 2010).

4.2.2 Conduct of the Delphi Panel and Achievement of Consensus
Consensus was not sought for Part I and Part II questions, as their focus was on determinants of
risk tolerance and expertise for individual panelists. The ten Part III questions sought to elicit
expertise about seven knowledge categories relative to the risk management process: (1) MUSH
market-specific risks, (2) risk identification methods, (3) risk identification timing, (4) risk
identification responsibility, (5) risk evaluation methods , (6) risk identification, and (7)
identification of the most important risk categories. Consensus was a goal of four out of ten Part

162

III questions; the other six questions were intended to assist with knowledge construction, as
described in Chapter 3, and reflective of the decision Delphi technique (Rauch 1979).
After the first Delphi round, consensus was achieved for three of the four consensus-based
questions (75%), as shown in Table 4-1. Consensus was not achieved for question 2b, which was
focused on new risks identified by panelists during the first round interviews. Additionally, while
consensus was achieved on questions 2c (MUSH market-specific risk categories) and 3e (risk
importance ranking), further detail was sought from panelists; further detail was also sought with
regard to questions 3a (risk identification methods) and 3d (risk evaluation methods). A second
Delphi round was conducted using an online survey platform (Appendix A.3) to facilitate
participant responses. As part of the second round, panelists were provided with their first round
responses, as well as the consensus measures attained for each question through the first Delphi
round. The second round resulted in achieving consensus for question 2b and attaining the
additional information needed for questions 2c, 2d, 3a, and 3e; at that point, the Delphi process
was terminated. A detailed analysis of all data obtained from both rounds of the Delphi panel is
included in Appendix C.1 and Appendix C.2.
Table 4-1. Summary of Part III Questions and Consensus Results
Knowledge
Delphi Consensus Status
Question
Goal
1st Round
2nd Round
Number
Question Focus
2a.
Risk Identification
Elicitation
Achieved
2b.
Elicitation
Not Achieved Achieved
2c.
MUSH Market-Specific Risks
Elicitation
Achieveda
Achieveda
a
3a.
Risk Identification Methods
Construction Not Needed
3b.
Risk Identification Timing
Construction Not Needed
3c.
Risk Identification Responsibility Construction Not Needed
3d.
Risk Evaluation Methods
Construction Not Neededa
3e.
Identification of Most Important
Elicitation
Achieveda
Achieveda
Risk Categories
4a.
Construction Not Needed
4b.
Construction Not Needed
Notes:
a\ Additional information was sought in the second Delphi round.

163

The results of the Delphi process provided the data for this portion of the research and consisted
of recorded conversations with panelists, which were subsequently transcribed by the researcher,
as well as field notes collected during the interviews. Each recording was played back then
transcribed; transcripts were reviewed at least twice during data analysis. Respondents were
contacted when information was unclear or difficult to understand, either due to the quality of the
recording or due to the context within which the information was provided. Frequent notes were
included with the transcripts in order to identify important statements and identify emerging
patterns based on the content of the data.

4.2.3 Determination of Risk Tolerance and Expertise
The TIAA-CREF Asset Allocation Evaluator (AAE) was used to assess each panelist’s attitudes
toward risk by using the analog of allocating assets in a financial portfolio (TIAA-CREF 2013).
This was considered to be an important element of data collection and analysis because
individual responses to the questionnaire may be dependent on participants’ attitudes. Panelists
were provided the six questions from the AAE as Part I of the Delphi questionnaire. A complete
discussion of the AAE, its use in this study, and panelists’ responses are provided in Appendix
C; however, a summary of panelists’ risk tolerance profiles, as evaluated by the AAE, is
provided in Table 4-2. It should be noted that only 18 panelists completed the AAE as one
refused to answer these questions.

Eleven questions making up Part II of the Delphi questionnaire sought information about
panelists’ relevant experience. The respective level of expertise possessed by each panelist was
determined using the rubric based on Hallowell and Gambatese (2010) and Duah (2014) that was

164

Table 4-2. Research Participant Risk Tolerance Profiles
Risk Tolerance
% of Respondents Firm Typesa
Conservative
17% M, U
Moderately Conservative
11% M, U
Moderate
28% E, M
Moderately Aggressive
33% E, M
Aggressive
11% E
Notes:
a\ E = Independent ESCO; M = Manufacturer-Based ESCO; U =
Utility-Based ESCO
described in Chapter 3. The complete list of determinants of expertise for each panelist is
presented in Appendix B; a summary of selected expertise criteria is provided in Table 4-3.
Table 4-3. Summary Expertise Attributes of Delphi Members
Minimum
Attribute
Value
Mean/Mode
a
Total Points
4.5 (31%)
Mean = 8 (55%)
Total Categoriesa
5 (50%)
Mean = 7.2 (72%)
Years involved with EPC projects
5
Mean = 16
Total number of projects
10
Mean = 81
Proportion MUSH market projects
33%
Mean = 90%
Number of building systems impacted/project 3-4
Mode = 5-7
Notes:
a\ Points and categories refer to Tables 3-1 and 3-2

Maximum
Value
12 (83%)
10 (100%)
33
300
100%
10+

4.3 DATA COLLECTION – DISCUSSION AND INFERENCES
This section discusses the analysis of consensus and non-consensus data obtained through the
comprehensive expertise elicitation strategy, focused on questions related to the risk
management process. The data obtained through the Delphi rounds was analyzed based on
consensus of elicited knowledge, consensus in modifications to the risk categories and risks in
the preliminary risk framework, elements of knowledge elicitor training (literature review and
industry interactions), quantitative information, and inferences and emerging patterns drawn
from the data.

165

Based on the comprehensive elicitation strategy described in Chapter 3 and the analysis of the
resultant data, consensus was achieved among all expert panelists regarding the level of ESCO
risk in MUSH market projects and in select sub-markets, risk identification, and the relative
importance of identified risk categories. The elicited knowledge was mutually acceptable to
participants and was deemed to be applicable to ESCO industry practices for EPC retrofit
planning and project management. Additionally, knowledge was constructed from six nonconsensus questions, which was focused on understanding industry practices related to risk
management. This section discusses the results of the analysis and reports findings related to the
knowledge categories related to the Delphi questionnaire.

4.3.1 MUSH Market-Specific Risks
Table 4-4 reports on consensus achieved and knowledge constructed by experts relative to the
knowledge category of MUSH market-specific risks. Consensus was sought and obtained for this
question in order to reject or confirm the assertion that MUSH market projects, specifically
correctional facility retrofits, carry additional risks and as such require the construction of a
specific risk. Eighty-two percent of the Delphi panelists indicated that there are differences in the
risk profiles between MUSH and non-MUSH segments; non-MUSH segments were defined as
commercial and industrial (C&I) retrofits. Of the panelists reporting a difference in risks among
projects in the two markets, 78% indicated that at least one or more MUSH sub-markets is riskier
than C&I, (64% - all MUSH sub-markets; 14% - correctional facilities only, 7% hospitals only).
Two other respondents indicated their belief that C&I projects are riskier in all cases.

166

Table 4-4. Elicited Knowledge: MUSH Market-Specific Risks
Knowledge Subcategory
Elicited Knowledge
Sources of Risk: C&I
 Greater likelihood of building use changes.
Projects
 Potential revenue losses during retrofit activities.
 Greater credit risk.
 Preferred short contract periods (e.g., 3-5) years limit the
scope of the project to short payback period items.
Sources of Risk:
 Costs related to security protocols, specifically due to reduced
Correctional Facility EPC
daily productivity and the need for additional project staff.
a
Retrofits
 Health and safety protocols that override the ability to cycle
HVAC equipment off during low utilization times.
 Project conflicts arising from the “militaristic structure” of
correctional agencies and external pressures they face, such as
reduced budgets, overcrowding, and security.
 Difficulties of scheduling and conducting work in a
continuously-operated facility.
 Security concerns with installed ECMs and vandalism of
equipment after installation.
 Risks are dependent on the security level of the correctional
institution where work is taking place, to include:
o Schedule delays due to security protocols; these are
typically more stringent in higher security facilities.
o Schedule delays arising from the need to relocate
inmates from work areas.
o Lost time due to correctional officers assigned to project
oversight being assigned elsewhere in the facility based
on emergent needs.
Sources of Risk: Hospital
 Difficulties of scheduling and conducting work in a
EPC Retrofitsa
continuously-operated facility.
 Critical areas, such as surgical suites, must remain operational
throughout the retrofit project. A highly competent
construction manager is needed to manage the work given
these concerns.
Notes:
a\ Elicited knowledge relative to risks across all MUSH market facility types are presented in the
refined risk framework in this chapter. Only the riskiest MUSH building types are included here.
As a result of the open-ended responses obtained from several participants during the first round,
in the second Delphi round panelists were asked to identify which MUSH market facility types
had the greatest project risk profiles. Facility types included correctional facilities, hospitals, K12 schools, mission-critical facilities, and continuously operated facilities; the latter facility type

167

was inclusive of correctional facilities and hospitals which operate continuously by definition.
The percentage of responses for each facility type was 67%, 47%, 13%, 53%, and 47%,
respectively. When accounting for respondents who did not identify correctional facilities, but
did identify the inclusive category of continuously operated facilities, the response frequency
increased from 67% to 87%. Additionally, just 7% of respondents believed that MUSH market
facility EPC retrofits are no riskier than private sector EPC projects. As a result, it is reasonable
to conclude that MUSH market EPC retrofits in general, and correctional facility projects
specifically, have a higher project risk profile than other markets and building and types.

4.3.2 Risk Identification Methods, Timing, and Project Responsibility
The first Delphi round encouraged panelists to provide open-ended responses to these questions
in order to fully understand the range of techniques and processes used relative to these risk
management functions. Additional detail was sought in the second round regarding risk
identification methods. Panelists were asked to identify the risk identification methods they
typically use; index values for this question came from the open-ended responses in the first
round and the work conducted by Grimaldi et al. (2012) and Thaheem and DeMarco (2013).
Table 4-5 and Figure 4-3 report on this constructed knowledge.

As can be seen in Figure 4-3, panelists most frequently rely on expertise to identify project risks
(80%), followed by brainstorming (73%), and the use of checklists (73%). Techniques that are
highly structured and require more staff and analysis time to complete were less favored by
panelists. SWOT analysis and FMEA are used much less frequently during risk identification,
with frequencies of 27% and 13%, respectively. The results of the second round responses

168

Table 4-5. Constructed Knowledge: Risk Identification Process
Knowledge Category
Constructed Knowledge
Risk Identification
See Figure 4-3.
Methods
Risk Identification
 100% of respondents indicated that the risk identification process
Timing
begins early in the project.
 58% of respondents indicated that the risk identification process
is iterative, and takes place at designated milestones during the
project, particularly during phase hand-offs among project staff.
Risk Identification
 100% of panelists indicated that their organizations have
Responsibility
designated individuals to manage the risk identification process;
however, their specific job functions varied.
 84% of respondents indicated use of a team approach which
typically involved project managers managed the risk
identification process specific to their phase of the work (e.g.,
IGA development, construction, M&V).
 Risk review teams were typically multi-disciplinary and included
the project developer, energy engineers, the design engineer, the
construction manager, assurance engineers (for M&V concerns),
and in some cases finance and legal staff members.
 Executive management (vice president or above) review of risks
was indicated by 26% of the panelists.
validated the risk identification approach taken by this research, combining interviewing with
expertise/expert judgment and a de-facto checklist (the a priori risk framework).

Figure 4-3. Risk Identification Methods Reported in Second Delphi Panel
169

4.3.3 Risk Evaluation Methods
Table 4-6 reports on panelists’ responses regarding risk evaluation methods. The overarching
finding was that rigorous analytical methods are employed less than 50% of the time as part of
the risk management process, and more robust methods were needed to improve the approach to
managing identified risks. One panelist reported a desire to use probabilistic methods, given the
variation inherent in some of the data used to model retrofit performance (e.g., climate,
equipment service life, energy price escalation, schedule variation); however, difficulty
incorporating such methods directly with energy modeling activities was cited as a barrier for
wider adoption.
Table 4-6. Constructed Knowledge: Risk Evaluation Methods
Knowledge Subcategory Constructed Knowledge
Risk Evaluation
 In general, ESCOs do not engage in formal methods of risk
Techniques
evaluation.
 The majority of panelists reported using informal or experiencebased methods.
 One panelist reported some use of probabilistic methods for risk
evaluation to include Monte Carlo simulation (via Crystal Ball)
and the use of PERT inputs for schedule-related risks.
 One panelist reported use of an internally-developed tool that
evaluates risks as having either “low” or “high” impacts relative
to associated costs. The “high risk” cost value is added to the
project contingency if mitigation is not feasible.
As with risk identification, the second round Delphi questionnaire sought further detail about the
specific methods used for risk evaluation. Panelists responded to techniques identified during the
first round as well as methods identified by Grimaldi et al. (2012) and Thaheem and DeMarco
(2013). Those results are depicted in Figure 4-4.

As can be seen in Figure 4-4, the use of expertise and simpler methods to implement was
reported most frequently. Brainstorming (60%), expertise/expert judgement (60%), checklist

170

Figure 4-4. Risk Evaluation Methods Reported in Second Delphi Panel
(53%) and risk mapping (53%) were the most commonly-reported risk evaluation methods.
Interestingly, 40% of respondents reported using expected monetary value, which relies on a
large amount of historical data to implement (Grimaldi et al. 2012). The above average use of
this technique may indicate that ESCO professionals seek out and prefer methods that express
risk as a conditional cost given scenarios that may or may not occur during a project.

While the data is reported in this chapter as part of data collection and analysis activities, the
insights gained are most pertinent to the risk analysis and evaluation strategy development in
Chapter 5. Based on the knowledge constructed from the Delphi panel, the risk evaluation
strategy used for this research must include the following characteristics and features:


Incorporate expertise of ESCO professionals, especially key decision makers;



Utilize scenario-based expected cost measures; and



Utilize probabilistic functions for evaluation parameters, where feasible.
171

4.3.4 Risk Identification
Consensus was sought, and obtained in the first Delphi round with regard to the twelve risk
categories included in the preliminary risk framework (Table 2-5). Panelists were asked the
frequency with which they consider each risk category in their MUSH market EPC retrofit
projects, using a seven-point Likert-type scale (Vagias 2006), which utilized the following
response categories: (1) 100% - Every Time, (2) ~90% - Usually, (3) ~70% - Frequently, (4)
~50% - Sometimes, (5) ~30% - Occasionally, (6) <10% - Rarely, and (7) 0% - Never.
Positive risk identification was denoted when a panelist indicated that they considered the risk in
categories one through six. Consideration was defined as addressing a risk category through
contractual means (e.g., inclusion of an energy rate escalation term), via technical means (e.g.,
conduct a peer review of energy model results), through project management (e.g., specific team
member assigned to coordinate activities among planning, engineering, and execution groups), or
by financial means (e.g., include a “hedge” factor to account for potential uncertainty).

Consensus was reached for all 12 risk categories (1A-1C and 2-10) as depicted in Table 4-7.
Sixty-six risks were identified as belonging to these 12 risk categories. Achieving consensus also
confirmed the selection and identification of the 12 risk categories in the preliminary framework.

The distribution of risk identification responses was also examined using a boxplot (Figure 4-5).
Review of the boxplot indicates that identification frequency for all 12 risk categories was
negatively skewed and the data set contained a number of outliers. These are denoted by
“○”(mild outlier) and “*” (extreme outlier) which are defined as Q1-1.5xIQR|Q3+1.5xIQR and
Q1-3xIQR|Q3+3xIQR, respectively. The boxplot further reveals that the least amount of

172

Table 4-7. Identification of Risk Categories in the Preliminary Risk Framework
Risk Category
Frequencya
1. Client Selection Factors
1A. Financial Factors
84% (84%)
1B. Facility/Technical Factors
84% (100%)
1C. People Factors
95% (95%)
2. Project Development
95% (100%)
3. Energy Audit Quality
95% (100%)
4. ECM Selection and Installation
100% (100%)
5. Commissioning
100% (100%)
6. Operations and Maintenance Practices
95% (95%)
7. Measurement and Verification of Savings
100% (100%)
8. Project Management Over the Project Life-Cycle
74% (74%)
9. Construction-Specific Concerns
95% (100%)
10. Volatility of Energy Prices
89% (89%)
Notes:
a\ Value in parentheses is after all outliers were removed. Outliers were defined as
Q1-1.5*IQR and Q3+1.5*IQR. Shaded rows include risk categories with increased
identification frequency after removal of outliers.

Figure 4-5. Range and Distribution of Risk Consideration Frequency
173

variation in panelist responses to frequency of risk consideration were with regard to energy
audit quality, ECM selection and installation, and M&V.

Panelists were afforded the opportunity to add additional risk categories that did not appear in
the a priori framework; 11 additional risk categories were identified. Analysis of transcripts and
a summative review of each of the newly-identified risk categories resulted in 10 being classified
as individual risks belonging to existing risk categories, and one was retained as a new risk
category (Table 4-8). This resulted in an additional 12 risks being identified as belonging to the
relevant risk categories, bringing the number of risks identified by Delphi panelists to 78.
Table 4-8. Additional Risks Identified by Panelists
Risk

Disposition

Political Risks

Moved to individual risk
under "Client Selection
Risks - People Factors"
Moved to individual risk
under "Construction Phase
Risks"
Moved to individual risk
under "Development Phase
Risks"

Productivity Losses in
Corrections Projects
Safety
Changing Financial
Incentives
Cost of Doing Nothing or
Self-Implementing
Public Procedural Risks
Time-Based Risk
Timing
Design Development
Staff Turnover During
Project Lifecycle
Perception of the
Performance Contracting
Industry

Delphi Consensus Status
1st
2nd Round
Round
N %
%
Consensus
4 21% 100% Achieved
1

5%

87%

Achieved

1
1

5%
5%

60%
80%

Not Achieved
Achieved

1

5%

80%

Achieved

1
2
1
1

5%
11%
5%
5%

73%
73%
73%
67%

Achieved
Achieved
Achieved
Not Achieved

5%

80%

Achieved

11%

100%

Achieved

Moved to individual risk
under "Energy Audit
Quality"
Moved into individual risk
1
under "Project Management
Over the Project Life Cycle"
Create New Risk Category
2

174

Consensus was not achieved in the first Delphi round for the newly identified risk category, or
the other ten risks in Table 4-8. As a result, during the second Delphi round panelists were asked
the rate their consideration frequency of the new risk category, “Perception of the Performance
Contracting Industry,” and they were asked for their concurrence with the other ten risks.
Consensus was achieved for the inclusion of the new risk category, with 100% agreement during
the second round. Consensus was also achieved for eight of the ten risks identified by panelists.
Design development (67%) and safety (60%) did not meet the threshold value for participant
agreement (70%), and were thus excluded from further analysis.

4.3.4.1 Identification of Most Important Risk Categories
The questionnaire included three subcategories for this knowledge category: (1) risk importance
scoring of top three risk categories, (2) identification of risk causes belonging to each risk
category, and (3) identification of risk control strategies. Delphi panelists were asked to identify
the most important risk categories among those presented to them during the interview. The
purpose of this review was to identify the risk categories that would be analyzed and evaluated in
Chapter 5. Consensus was sought on this question, and panelists were asked to assign a de-facto
risk importance score rank by identifying the three risk categories that they believed to have the
greatest contribution to failing to meet guaranteed performance. There were 57 such available
votes (59 were actually recorded because one panelist identified five important risk categories).
A Pareto histogram of the relative importance scores for each risk category is provided in Figure
4-6. From the Pareto histogram, six risk categories were identified and ranked in the upper two
quartiles for risk importance scores: (1) energy audit quality, (2) project development, (3) ECM

175

Figure 4-6. Pareto Histogram of Relative Importance Scores for Risk Categories
selection and installation, (4) construction-specific concerns, (5) commissioning, and (6)
measurement and verification (M&V).

Risk consideration frequencies (Table 4-7) were reviewed for these six risk categories, and they
were ranked again. The results of ranking using quartile analysis/Pareto histogram of risk
importance scoring and ranking due to risk consideration frequencies are shown in Table 4-9.

The use of consideration frequencies eliminated the variation in risk category importance ranks.
As a result, during the second Delphi round, panelists were asked to rank each of the six risk
categories from 1 (most important) to 6 (least important). Results are shown in the rightmost
column of Table 4-9. Consensus was achieved if two of the three ranking measures had rank
176

order agreement for a given risk category. By that measure, four risk categories achieved
consensus (energy audit quality, ECM selection and installation, commissioning, and M&V),
while two did not (construction specific concerns and project development).
Table 4-9. Risk Importance Ranking from Delphi Panel Data
Consideration
a
a,b
a
Pareto/Quartile
Frequency
Delphi Round 2
Risk Category
Energy audit quality
1 (10)
4/1 (95%/100%)
1 (2.4)
ECM selection and installation
2 (8)
1/1 (100%/100%)
2 (3.0)
Construction-specific concerns
2 (8)
4/1 (95%/100%)
6 (3.9)
Commissioning
4 (7)
1/1 (100%/100%)
4 (3.6)
Project development
5 (5)
4/1 (95%/100%)
3 (3.1)
M&V
5 (5)
1/1 (100%/100%)
5 (3.7)
Notes:
a\ Numbers in parentheses are the raw values (e.g., non-ranked) for each variable.
b\ Values are ranks with outliers retained (left of slash) and with outliers removed (right of
slash).
4.3.4.2 Panelist Characteristics Influencing Determination of Most Important Risk Categories
As was seen in Table 4-9, the disagreement across the three risk category ranking measures
could be the result of the framing problem described by Kahneman (1981), Russell and Thaler
(1985), Tversky and Kahneman (1986), and Klotz (2011). If this is indeed the case, it would be
expected to find no relationship among the three variables: (1) risk importance score
(Pareto/quartile analysis), (2) risk consideration frequency, and (3) round 2 Delphi scores. First
correlation was used to determine the relationship between the risk importance score and risk
consideration frequency, since both of these variables included all 12 risk categories. Since
neither variable is normally distributed, Spearman’s rank order correlation was chosen for this
analysis instead of Pearson’s correlation (Onwuegbuzie et al. 2007). This test has the additional
benefit of being relatively insensitive to outliers, which is cited as a methodological concern with
Pearson’s correlation (Chok 2010). Results of the analysis indicated a strong positive monotonic
relationship between risk importance score and risk consideration frequency (ρ=0.701, n=12,
177

p<0.05). Before analyzing the relationship among the three variables, a Spearman correlation
coefficient was again calculated for the same variables as before, but now including only the six
levels of the variables included in Table 4-9. No evidence of a significant correlation was
detected in this test (ρ=0.441, n=6, p=0.381). This is likely a result of the extremely small
dataset; as such, analysis of the relationship among the three variables was not conducted. Based
on the initial finding of a strong and significant relationship between risk importance score and
risk consideration frequency, it is reasonable to reject the presence of a framing effect for
responses to these variables.

Other characteristics which may affect selection of the most important risk categories include
panelist’s length and nature of professional ESCO experience and their risk tolerance level.
These reflect some of the underlying conditions of the status quo and professional biases as
described by Klotz (2011) with respect to energy efficiency programs. Like the analysis of the
presence of a framing problem, the purpose of this research is not a comprehensive
characterization of the impacts of cognitive biases; however, it is important to recognize when
the potential for such bias exists in data sets such as this one, collected using primarily
qualitative means.

Given the non-normal distribution of the dependent variable (DV), risk consideration frequency,
Spearman’s correlation was again used to detect a relationship among the selected DV and four
indicator variables (IV), one of which was measured two ways: (1) panelist’s length of
professional ESCO experience, (2) risk tolerance level (raw risk score from the TIAA-CREF
AAE), (3) risk tolerance level (transformed raw risk score into the ordinal scale described in

178

Table 4-2), (4) job level, and (5) project phases worked on. The variables are described in Table
4-10 and the results of the correlation analysis are detailed in Table 4-11.
Table 4-10. Panelist Characteristic Indicator Variables
Variable
Type
Values
Length of Professional Scale
Years of experience
ESCO Experience
Risk Tolerance (Scale) Scale
TIAA-CREF AAE scores - 0-96
Risk Tolerance
Ordinal TIAA-CREF AAE scores – 1 (conservative) through 5
(Ordinal)
(aggressive)
Job Level
Nominal 1 – technical staff; 2 – management; 3 – executive
Project Phases
Nominal 1 – pre-execution; 2 – execution or later; 3 – both; 4 - other
Table 4-11. Spearman’s Rho and Significance Values – Panelist Characteristics
Indicator Variables
Length of
Professional Risk
Risk
ESCO
Tolerance Tolerance
Dependent Variable
Experience
(Scale)
(Ordinal) Job Level
Risk Consideration
ρ = 0.150
ρ = 0.017
ρ = 0.050 ρ = -0.121
Frequency - All Risk
p<0.05
Not Sig
Not Sig
Not Sig
Categories

Project
Phases
ρ = -0.144
p<0.05

Based on the results of the correlation analysis, when examining the full data set (i.e., all 12 risk
categories), risk consideration frequency and length of professional ESCO experience have a
slightly positive, significant relationship (ρ = 0.150, N=216, p<0.05); the relationship with
project phases that the panelist regularly works in is slightly negative and significant (ρ = -0.144,
N=216, p<0.05). The individual relationships between risk consideration frequency and job level
and risk consideration frequency and risk tolerance were insignificant. Collectively, only 4.3% of
the variation in the DV is explained by both significant IVs. If bias is present, as indicated by the
two significant IV relationships, their net effect is very small; therefore this is not deemed a
concern with this data set.

179

It was initially believed that the most likely reason for the lack of an observed correlation
between risk tolerance and risk consideration frequency was due to differential understanding of
the context of the decision problem on the part of the panelists. The effects of this “framing
problem” have been well-documented in the literature (Tversky and Kahneman 1981; McNeil et
al. 1982; Russell and Thaler 1985; Tversky and Kahneman 1986). The potential presence of a
framing problem was tested, and as reported earlier in this section did not seem to be present,
thereby eliminating contextual issues as a possible reason for the observed lack of correlation.
Kahneman and Tversky’s (1979) Prospect Theory stated that individual decision maker
behaviors are often shaped by the fact that losses hurt more than gains feel good, thereby
implying an inherent bias toward risk aversion. Berny and Townsend (1993) and Raftery et al.
(2001) found that construction and technical project managers tended to be risk-averse. While
the risk tolerance questionnaire administered to the Delphi panelists suggested that ESCO
professionals have wide-ranging risk tolerances, the data on risk identification frequency was
positively skewed for each risk category, indicating that even the most risk averse panelists
frequently give consideration to the identified risks on their projects. The researcher, therefore
surmises that the lack of correlation between panelist risk tolerance and risk consideration
behavior is the result of the average tendency of participants to give a high degree of
consideration to risks, regardless of their individual attitude or tolerance toward risk.

4.3.4.3 Role of Contractual Language and Occupant Behavior in Risk Importance
Delphi panelists provided insight into which risk categories could be mitigated through
contractual means. This typically took one of two forms, either through directly allocating a
greater share of the risk to a third-party via contract language (e.g., the client, a subcontractor) or

180

by including a safety factor into the contract to address such risks. Of the thirteen risk categories
(12 from the preliminary framework and one from the refined framework), seven contained
aspects that could be controlled contractually: (1) facility/technical factors, (2) people factors, (3)
energy audit quality, (4) construction-specific concerns, (5) O&M practices, (6) M&V, and (7)
energy price volatility. Specific aspects of these risk categories that panelists identified as being
controlled through contractual means are discussed below:


Possible future changes in the facility’s use profile and attendant changes in energy
consumption that relate to facility/technical factors are frequently addressed in project
contract documents.
o One panelist identified a situation where a public healthcare facility changed
from being used for medically fragile individuals to persons with
developmental disabilities.
o The latter population was more active than the original population, requiring
longer schedules for lighting and space conditioning in the new facility use
scheme.
o Such change was managed contractually via language limiting the ESCOs’
responsibility for increased energy consumption resulting from the changed
facility use.



Human behaviors and activities (risks due to people factors) that are inconsistent with
the retrofit design were also mentioned as being mitigated contractually.
o ESCOs may employ continuous commissioning to quickly identify and
remedy such issues.

181

o The use of continuous commissioning and the responsibility of the client to
manage occupant behavior are frequently specified in the contract.


Energy audit quality-rated risks are often addressed contractually when the audit
work is outsourced to a third party firm.



Construction-related risks are controlled in a similar manner to energy audit quality
risks when a third party firm is involved; many ESCOs contract the actual retrofit
construction and equipment installation activities to a third party. The use of an
approach similar to integrated project delivery was noted by two panelists as a means
to control construction-related risks through contractual means.



As described by panelists, O&M and M&V responsibilities are frequently outlined in
the contract and include the requirement to adhere to designated equipment
maintenance schedules, the requirement to maintain operation of any control systems,
and the responsibilities, types, and duration of M&V activities.



Over half of the panelists stated that energy price volatility is addressed contractually
through the use of mutually agreed-upon energy escalation rates and the specification
of the savings guarantee in terms of energy units (e.g., therms, kWh, CCF) saved, not
dollars saved.

As stated previously in this section, risks related to the behaviors and activities of occupants, as
well as general changes regarding building operation are frequently addressed contractually.
Despite the use of this control strategy, several panelists alluded to the fact that it can be difficult
to detect such behavioral changes. One participant indicated the use of continuous
commissioning in order to monitor and provide timely feedback regarding unexpected energy

182

consumption changes, which often result from behaviors attributed to the rebound effect, as
described by Hertwich (2005), Herring and Roy (2007), and Strand (2011). Several panelists
indicated that they do not realize these issues on their projects, and they frequently use the
project development phase and their expertise to help control these risks. Many ESCOs indicated
that they offer free training regarding appropriate O&M practices and the use of the specific
technologies they install as part of the retrofit, as well as training related to the new features and
enhancements made to the facility once the project is complete. This is done to supplement
contractual requirements and help build a positive relationship with the customer in order to
avoid potential disputes once the project is in the energy savings phase.

4.3.4.4 Selection of Risk Categories for Further Analysis
Based on the previous analysis, four risk categories were identified for potential further analysis:
(1) energy audit quality, (2) ECM selection and installation, (3) commissioning, and (4) M&V.
This enabled the remainder of the research to focus on empirical analysis and evaluation of a
targeted group of risks that represent those that have the greatest potential impact on the outcome
of EPC retrofit projects.

Commissioning-related risks primarily consisted of net effects resulting from the failure to
commission and the extra costs of commissioning associated with implementing and managing a
controls program. Risks related to M&V included measurement concerns (improper
measurement, lack of sub meters, and use of an improper baseline) and issues related to the
M&V plan (failure to include O&M, understatement of system performance by neglecting the
effects of NEBS, and poorly-designed protocols that inaccurately reflect system performance). In

183

both cases these risks are realized during and after execution, at which point the contract has
been signed and energy savings have been guaranteed. As a result, there is limited opportunity
for the ESCO to pursue change orders during these project phases. Furthermore, some of the
early phase risks identified by Delphi panelists (i.e., before the contract is signed) incorporated
commissioning- and M&V-related issues, thereby enabling their treatment during the next steps
of this research, risk analysis and evaluation. As a result, no further evaluation of these risk
categories or their related risks is planned as part of this research.

As described earlier in this chapter, the energy audit phase of EPC retrofits consists of
conducting the facility audit, conducting an analysis of previous utility bills, and development of
the energy model based on facility information. This leads to two critical decisions arising from
this phase: (1) establishment of the facility’s utility baseline and (2) identification of areas of
improvement which serve as inputs to the retrofit design phase. Proper establishment of the
baseline is critical as it directly impacts energy savings calculations and an improper baseline can
lead to disputes (Mozzo 2001; Mills et al. 2006; Sankey 2007). As depicted in Figure 4-13, the
baseline also impacts M&V-related risks and the execution phase. The retrofit design determines
the selection and installation of ECMs as well as the technical scope of the project, and
influences the complexity and cost of the M&V plan. As a result, both the energy audit quality
and ECM selection and installation risk categories were selected for further analysis and
evaluation.

Based on the foregoing, risks related to the energy audit quality and ECM selection and
installation are the focus of risk analysis and evaluation activities in chapters 5 and 6.

184

Table 4-12 includes expertise elicited from panelists with regard to causes of and control
measures used for energy audit quality-related risks and equipment selection and installationrelated risks. Achieving consensus among panelists was not a goal for the two knowledge
categories addressing individual risks (e.g., risk causes) and their associated control measures
(e.g., mitigation strategies) because the intent was to fully-describe the risk-based decision
making process with regard to the top two identified risk categories. Since risk causes and
mitigation strategies would be subject to review by SFMEA panelists as part of the risk scenario
Table 4-12. Risk Causes and Control Measures for Top Risk Categories
Risk Category
Energy Audit
Quality

Risk Causes
Existing Conditions
 Facility age – code update
 Misunderstanding existing conditions,
such as possible presence of asbestos
Facility Stakeholder Concerns
 Differing stakeholder needs
Inexperience
 Failure to understand facility
operations and EPC goal – lack of
EPC experience
Information and Analysis
 Calculation errors
 Conducting the IGA too quickly
 Energy model calibration error
 Inaccurate/incorrect or disputed
baseline
 Information availability and accuracy
 Missing information
 Missing risk assessment for each ECM
Project Complexity and Guaranteed
Savings
 Establishment of the guarantee amount
- difficult balance between providing a
large-enough project to generate client
excitement and ESCO value and
hedging on savings
 Overstatement of issues leading to
mismatch with technical needs;
understatement of issues leading to
reduced project value

185

Risk Control Measures
Existing Conditions
 Complete set of facility drawings
 Conduct a code review
Information and Analysis
 Complete set of facility drawings
 Contingency factors
 Identify risks for each ECM
being evaluated and the entire
ECM portfolio
 Include potential but unverified
concerns in the IGA
 Interview all facility staff and
back-brief them on the findings
to “ground truth” the audit results
and uncover any missing
information
 Third-party internal review based
on historical projects
Project Complexity and Guaranteed
Savings
 Contingency factors
 Third-party external reviews
 Use stipulation and IPMVP
Option A wherever appropriate

Table 4-12 (cont’d)
Risk Category
ECM Selection
and Installation

Risk Causes
Contractual Concerns
 Owners and/or project designers and
engineers are used to design-bid-build
in the public sector and are not used to
designing based on fixed budgets and
do not understand EPC cost structures
 Disconnect between the design and
audit intent – improper efficiency
target; different operating schedule
implemented than what was planned;
different control system installed
 Subcontractor quality and reliability
 Unqualified and unsafe contractors
create additional risks
ECM-Specific Issues
 Cheapest equipment sometimes
selected – leads to sub-optimal O&M
savings
 ECM package constructability and
feasibility
 ECMs are not aligned with the IGA
findings
 Failure to perform as designed
 Uncertainty in factors used to predict
performance
Facility Factors
 Installation location
Occupant Concerns
 Comfort complaints can add extra
work after construction is complete
 Lighting is difficult to demonstrate
before installation; owner and
occupants can be unhappy with light
quality after a retrofit
Project Complexity and Guaranteed
Savings
 “Low-hanging fruit has been picked”
– increasing complexity of ECMs
 Lowest-cost solutions may create
value concerns during O&M
Security Concerns (Correctional Facilities)
 Accessibility to inmates/physical
security

186

Risk Control Measures
 Information – equipment-specific
Contractual Concerns
 Assign a team member to align
outsourced design team with
internal energy auditing team
 Coordinate phase handoffs
between project developer and
energy engineer
 Designate a team member to
coordinate contractor and ESCO
team members.
 Pre-qualify subcontractors
 Robust CM practices
 Use ESCO’s own forces for
complex aspects of system
design, whenever possible
 Use EMR ratings to assess
subcontractor safety
ECM-Specific Issues
 Identify risks for each ECM
being evaluated and the entire
ECM portfolio
 In-house design teams for
specialty ECMs (e.g., lighting)
that can have complex design and
performance issues
Project Complexity and Guaranteed
Savings
 In-house design teams for
specialty ECMs that can have
complex design and performance
issues

construction process, assessing the validity of the knowledge constructed by Delphi panelists
was possible.

4.3.5 Define Energy Performance Contract Retrofit Project Phases and Key Decisions
Energy performance contract retrofit projects can generally be organized into five phases - (1)
project development, (2) energy audit, (3) retrofit design, (4) project execution, and (5) energy
performance, although some variation occurs depending on the procurement process used.
Tetreault and Regenthal (2011) identified a preliminary and final audit phase, separated by the
design phase for the federal Energy Savings Performance Contracting (ESPC) model and the
European Association of Energy Services Companies identified a four phase model that moves
directly from a preliminary audit to detailed engineering analysis and design (Petersen 2009).
The final phase model was developed through data collected from the Delphi panel and literature
reviewed as part of knowledge elicitor training. The generalized five phase model is depicted in
Figure 4-7. Each project phase contains a brief description of key actions taken and decisions (in
italics) made by the ESCO during that phase, which informs the development of the retrofit
process model. This information was the result of analysis of Delphi data using the elicitation
strategy developed for this project, as mentioned above. A brief description of each phase is
provided after Figure 4-7.

4.3.5.1 Project Development
The project development phase includes tasks related to the assessment of a request for proposal
(RFP) or other procurement documentation. This phase culminates with the ESCO’s decision

187

Project
Development
•Assess scope of work
•Assess facility factors
•Assess technical
factors
•Assess people
factors (occupants,
client, etc.)
•Decision to bid

Energy Audit
•Perform facility audit
•Conduct utility bill
analysis
•Develop energy
model
•Establish baseline
•Identify areas of
improvement

Retrofit Design

Project
Execution

•Refine energy model •Install ECMs
•Estimate energy
•Perform other
savings
required
construction
•Conduct cash flow
analysis
•Commission systems
•Conduct life cycle
cost analysis
•Prioritize ECMs and
finalize design
•Determine the % of
savings to guarantee

Energy
Performance
•Implement M&V plan
•Perform O&M in
accordance with plan
•Annual performance
review
•Revise baseline as
needed

Figure 4-7. Energy Performance Contract Retrofit Phases
whether or not to bid on the work. This “go/no-go” decision is typically based on an analysis of
factors related to the facility and technical aspects of the project, the finances of the client, and
the people involved in the project (e.g., building owners, occupants, client staff), and includes an
analysis of the project scope of work.

Facility and technical factors include characteristics of the building or buildings to be retrofitted.
These may include potential changes in future occupancy and use of the facility and unknown
latent conditions, such as the presence of hazardous materials, infrastructure that needs to be
abandoned, or mechanical and electrical equipment that is not code compliant. These issues may
be evaluated through a preliminary or walk-through energy audit.

People factors can include the existence of an existing business relationship between the ESCO
and owner (be it previously successful or unsuccessful), human behavior that overrides the
energy savings benefits of the retrofit (i.e., the rebound effect) and that is, therefore, inconsistent
with M&V plans, improper O&M activities undertaken by customer as a result of limited
188

capacity to understand and/or perform the work, and project interference with building
operations due to the unique nature of building occupants or operating schedules. Examples of
the latter concerns include preservation of clean, conditioned air during retrofit activities for
medically-fragile occupants, and movement restrictions due to differing security levels inside a
correctional facility.

Financial factors include concerns about the client’s ability to secure financing and make lease
payments throughout the project life cycle. The project development phase culminates in an
ESCO’s decision whether or not to bid on a project.

4.3.5.2 Energy Audit
As described in Chapter 2, energy audits do not necessarily conform to a third-party standard;
however, ASHRAE has developed a standard for energy audits, categorized as Level 1, Level 2,
and Level 3 (ASHRAE 2011; Baechler, Strecker, and Shafer 2011). A Level 1 audit may be
considered a preliminary audit, consisting of a rapid assessment of building performance,
installed technologies, and an assessment of potentially-applicable incentives and grant
programs. Such audits may be performed during the project development phase to assist the
ESCO in deciding whether to bid or to assist them in developing their RFP response. Both the
Level 2 and Level 3 audits require a more detailed analysis of building systems, consisting of an
assessment of energy sources and end uses, discrepancies in system operations, a thorough
analysis of utility bill data, and identification of potential ECMs for each system, to include a
range of possibilities, associated costs, and prioritization of ECMs based on limiting factors such
as cost, infrastructure capacity, building end-use, and system interactions. The key difference

189

with the Level 3 audit, also termed the investment grade audit (IGA), is the collection of
operating data across longer time periods, the use of whole-building energy simulation modeling,
and modeling of each ECM for performance and life cycle cost characteristics (ASHRAE 2011).

The energy audit also serves to identify building conditions that may increase the project cost if
detected during project execution. This can include the presence of outdated infrastructure which
requires an upgrade to be compliant with current standards, the presence of hazardous materials
requiring remediation during retrofit activities, a decision whether supporting infrastructure (e.g.,
steam pipes, ductwork) for systems to be retrofitted should be removed or abandoned in-place, a
review of missing details on as-built drawings, or an assessment of unknown site conditions.
A key aspect of the energy audit phase is the development of, and multi-party agreement to, the
facility’s energy baseline. This process involves an assessment of utility bill data, a review of
facility operating parameters and the condition of building equipment, a review of the building’s
operating profile (e.g., hours of operation, ventilation and conditioned air requirements, the
presence of occupants requiring controlled indoor environments), and fitting the energy model to
observed operating parameters and utility consumption. Ultimately, the goal of the energy audit
phase is to identify areas of improvement for the building and assess a variety of strategies to
provide those improvements. Determination of the utility baseline and identification of areas for
improvement are the two key decisions made during this project phase.

4.3.5.3 Retrofit Design
The primary goal of the retrofit design phase is to prioritize ECMs and develop a final
engineering design based on energy models and economic analyses. This ensures that the design

190

meets the owner’s project requirements (OPR) and that it provides sufficient savings during the
performance period to fully pay for the retrofit and associated work. By refining the energy
model to develop an estimate of energy savings and completing a cash flow analysis for the life
of the project and a life cycle cost analysis of selected retrofit measures, ECMs can be prioritized
based on their ability to meet energy-related, non-energy-related, and financial OPR. Completion
of the retrofit design phase results in two key decisions: (1) a prioritized list of potential ECMs
and the resulting final retrofit design and (2) a determination of the amount of savings to be
offered as the guarantee (often referred to as deration).

4.3.5.4 Project Execution
Selected ECMs are installed along with any other required construction during the project
execution phase. While this may include necessary upgrades in order to be in compliance with
building codes, removal of hazardous materials, or upgrading infrastructure to support the newlyinstalled retrofit measures, this can also include measures not directly-related to ECMs such as
capital improvements to the building envelope. To illustrate the concept of how the consequence
of a risk cause can change during the project’s life cycle, asbestos-containing material that is
discovered during the project execution phase results in an unrecoverable project cost for the
ESCO, whereas the costs associated with that same issue when discovered during project
development or the energy audit phase can potentially be recovered.

As mentioned previously, the facility type and its operating profile may limit construction to
specific times or create additional restrictions on construction activities. Hospitals frequently
require additional protection of indoor air quality during construction; schools may limit

191

construction activities to non-occupied times; and correctional facilities often have detailed and
lengthy procedures for daily check-in and check-out, tool security, and relocation of inmates
during construction activities.

Besides installation of retrofit measures, the other primary activity during the project execution
phase is the commissioning of newly-installed systems and existing systems that were impacted
by the new ECMs. This is required to ensure that the building and its systems operate optimally
and in accordance with the OPR. Additional savings in EPCs have also been demonstrated
through a robust commissioning program (Stum 2000; Jennings and Skumatz 2006).

4.3.5.5 Energy Performance
Once the ECMs have been installed and commissioned, the project moves into the energy
performance phase. During this phase, the M&V plan is implemented. This plan should be
developed at the same time the engineering design is finalized, and needs to be agreed to by the
owner and the ESCO. Chapter 2 included a description of the four options for M&V included in
the International Performance Measurement and Verification Protocol (IPMVP) including a
discussion about the role of stipulation. Briefly, options A and B focus on the M&V of individual
systems (Option A utilizes partially-measured isolated systems, and may include stipulation,
where Option B utilizes fully-measured isolated systems), Option C requires whole-building
M&V (includes existing and retrofitted components), and Option D requires calibrated wholebuilding simulation and is used when baseline data is not available.

192

An important corollary to executing the M&V plan is ensuring that system O&M is performed to
required levels. Improperly-maintained systems can lead to missed opportunities for energy
savings due to sub-optimal performance. O&M may be performed by the owner, through their
own forces or through contract, or the ESCO may provide a service contract separately from the
EPC contract.

Annual assessments of energy performance are used to determine whether the guarantee is being
met by the ESCO; insufficient energy performance results in the ESCO remunerating the owner
in the amount of the shortfall. With mutual agreement among the parties, the energy baseline
may be revised during this phase; typically this occurs if the operating profile of the building has
changed (e.g., end-use, hours of operation, specific occupant requirements), if required O&M is
not being performed by the owner, or if the owner makes any additional changes to the building
that are outside the scope of the EPC contract.

4.3.6 Risk Categories by Project Phase
Over the course of a typical EPC retrofit project with a 12-15 year performance period, decisions
regarding the project design are made during the earliest parts of the project life cycle. Since
EPC retrofits generally contain little to no opportunity for ESCO-driven change orders, there is
limited opportunity for cost recovery once the performance contract is signed. As a result, it is
posited that performance risks are generally greater during the latter phases of the project, after
the energy savings guarantee is signed. The most important risk management actions would
likely take place during the earliest project phases, when decisions are made that have long-term
impacts on project performance and can effectively mitigate risk by including them in the

193

project’s pro forma before the energy savings guarantee is signed. Figure 4-8 summarizes the
relationship between cost and the ability to influence design, risk, and uncertainty. Polygon X on
Figure 4-8 describes a space termed the “window of opportunity” (Horsley et al. 2003), a limited
period of time during which the project design can be improved so that performance is enhanced,
project costs are reduced, and project-related risks are most effectively managed (Kmenta and
Ishii 2001; Kishk et al. 2003).

X

Figure 4-8. Project Decisions and Costs Over the Project Life Cycle
(Adapted from Kmenta and Ishii 2001; Horsley et al. 2003; Kishk et al. 2003)
Based on elicited expertise related to the knowledge categories pertaining to risk identification
and risk importance scoring, each risk category was assigned to the project phases where they
occur and a mean risk importance score was recorded. Project life cycle phase assignments and
risk importance scores are provided in Table 4-13. This relationship is depicted graphically in
Figure 4-9, which provides a conceptual understanding of how risk is realized over the life cycle
of a typical MUSH market EPC retrofit project. As expected, the risk importance score is highest
in the earlier phases, since decisions made here have lasting impact over the length of the
performance contract; this corresponds to the time when the cost of changes is the lowest and the
ability to influence project performance is the greatest. An assessment of the overall project risk
194

profile, with regard to the level of performance risk, is provided in Chapter 6 as part of the pilot
application of the risk model.
Table 4-13. Risk Category Importance by Project Phase
Mean Risk
Importance
Project Phase Scorea
Project
7
Development

Risk Category
Financial Factors b
Facility/Technical Factors b
People Factors b
Project Development
Energy Audit Quality
Energy Audit 10
ECM Selection and Installation
Retrofit
5
Design
Volatility of Energy Prices
Commissioning
Project
7.5
Execution
Construction-Specific Concerns
O&M Practices
Energy
4
Performance
M&V
c
Project Management Over the Project Life Cycle Not Assigned 2
Notes:
a\ The mean score is the average frequency of all risk categories in a given
project phase.
b\ While listed separately, these risk categories were treated as one for the
purposes of calculating the mean score, since they were sub-factors of “client
selection risks.”
c\ A single phase score was not assigned because this risk category occurs in each
project phase.

Figure 4-9. Most Important Risk Category Average Score by Project Phase
195

4.4 REFINED ENERGY PERFORMANCE CONTRACT RISK FRAMEWORK
The a priori risk framework presented in Chapter 2 was refined as a result of data collection and
analysis as described in section 4.3. This a posteriori framework informed the development of
two other deliverables of this research. The EPC retrofit process model, discussed later in this
chapter, is premised largely on the expertise elicited from the Delphi panel and the information
encapsulated in the refined risk framework. The ESCO risk framework also provided the inputs
to the first steps in the SFMEA process, by providing potential risk causes from which the panel
developed risk scenarios and by providing risk controls and mitigation strategies for evaluation.

4.4.1 Changes from Preliminary Framework
Based on data collected from the Delphi panel, the preliminary risk framework was refined in
three ways: (1) risk categories and risks were modified, (2) risk causes and mitigation strategies
were added, and (3) risk category identification frequencies and relative risk importance scores
were added. Table 4-14 depicts the refined (a posteriori) risk framework, and its components and
modifications from the preliminary risk framework are presented below.

196

Table 4-14. Refined Risk Framework for ESCOs Undertaking MUSH Market EPC Retrofit Projects
Risk Category
Client Selection
Factors

Project
Development

Delphi %a
84%
(84%)

Quartb,c
1(3)

84%
(100%)

1(3)

95%
(95%)

1(3)

95%
(100%)

3

Risks and Causes
Financial Factors
 Customer may not be able to get financing.
 Unacceptable customer debt ratio.
Facility/Technical Factors
 Building staff overrides equipment schedules or set points.
 Changes in future occupancy and use.
 Clients either prefer or have mandated shorter contract lengths
which limit technical scope of work.
 Facility age – code update requirements.
 Interference with building operations due to unique schedules or
facility needs (e.g., schools, hospitals, and prisons).
 Unknown latent site and facility conditions.
People Factors
 Improperly-performed O&M by client’s forces.
 Occupants that require special management during project
execution.
 “Rebound effect” - human activity inconsistent with M&V.
 Client self-implementation of specific work packages.
 Cost to do nothing and operate inefficiently.
 Costs incurred from project start-up; long development phases
can lead to difficult to recover costs.
 Lack of experience in a given market and/or with utility rebate
programs can add significant workload.
 Long procurement times for MUSH products can outdate pricing
by a year or longer.
 Political issues – owner’s project decision makers in MUSH
market may be elected, term-limited, etc. and cause a project to
lose continuity.
 Potential utility rebates may expire before the project is awarded.
 Public entities frequently change contract documents, terms, and
conditions.
 Some procurement methods require completion of an audit before
a contract is awarded.
 Timing of public projects can cause delays resulting in price
increases.

197

Controls/Mitigation
 Utilize standardized qualification
method.




Address concerns contractually.
Conduct code review.
Conduct a feasibility study.



Address concerns with potential
behavior of occupants contractually.
Offer training to operators and
building occupants.






Document conversations among
parties.
Track approvals given to the project.
Use escalation factor for price growth
if procurement takes too long.

Table 4-14 (cont’d)
Risk Category
Energy Audit
Quality

Delphi %a
95%
(100%)

Quartb,c
3

Risks and Causes
Existing Conditions
 Facility age – code update.
 Misunderstanding existing conditions, such as possible presence
of asbestos.
Facility Stakeholder Concerns
 Differing stakeholder needs.
Inexperience
 Failure to understand facility operations and EPC goal – lack of
EPC experience .
Information and Analysis
 Calculation errors.
 Conducting the IGA too quickly.
 Energy model calibration error.
 Inaccurate/incorrect or disputed baseline.
 Information availability and accuracy.
 Missing information.
 Missing risk assessment for each ECM.
Project Complexity and Guaranteed Savings
 Establishment of the guarantee amount - difficult balance
between providing a large-enough project to generate client
excitement and ESCO value and hedging on savings.
 Overstatement of issues leading to mismatch with technical
needs; understatement of issues leading to reduced project value.

198

Controls/Mitigation
Existing Conditions
 Complete set of facility drawings.
 Conduct a code review.
Information and Analysis
 Complete set of facility drawings.
 Contingency factors.
 Identify risks for each ECM being
evaluated and the entire ECM
portfolio.
 Include potential but unverified
concerns in the IGA.
 Interview all facility staff and backbrief them on the findings to “ground
truth” the audit results and uncover
any missing information.
 Third-party internal review based on
historical projects.
Project Complexity and Guaranteed
Savings
 Contingency factors .
 Third-party external reviews.
 Use stipulation and IPMVP Option
A wherever appropriate.

Table 4-14 (cont’d)
Risk Category
ECM Selection
and Installation

Delphi %a
100%
(100%)

Quartb,c
3

Risks and Causes
Contractual Concerns
 Owners and/or project designers and engineers are used to
design-bid-build in the public sector and are not used to
designing based on fixed budgets and do not understand EPC
cost structures.
 Disconnect between the design and audit intent – improper
efficiency target; different operating schedule implemented than
what was planned; different control system installed.
 Subcontractor quality and reliability.
 Unqualified and unsafe contractors create additional risks.
ECM-Specific Issues
 Cheapest equipment sometimes selected – leads to sub-optimal
O&M savings.
 ECM package constructability and feasibility.
 ECMs are not aligned with the IGA findings.
 Failure to perform as designed.
 Uncertainty in factors used to predict performance.
Facility Factors
 Installation location.
Occupant Concerns
 Comfort complaints can add extra work after construction is
complete.
 Lighting is difficult to demonstrate before installation; owner and
occupants can be unhappy with light quality after a retrofit.
Project Complexity and Guaranteed Savings
 “Low-hanging fruit has been picked” – increasing complexity of
ECMs.
 Lowest-cost solutions may create value concerns during O&M.
Security Concerns (Correctional Facilities)
 Accessibility to inmates/physical security.

199

Controls/Mitigation
 Information – equipment-specific.
Contractual Concerns
 Assign a team member to align
outsourced design team with internal
energy auditing team.
 Coordinate phase handoffs between
project developer and energy
engineer.
 Designate a team member to
coordinate contractor and ESCO
team members.
 Pre-qualify subcontractors.
 Robust CM practices.
 Use ESCO’s own forces for complex
aspects of system design, whenever
possible.
 Use EMR ratings to assess
subcontractor safety.
ECM-Specific Issues
 Identify risks for each ECM being
evaluated and the entire ECM
portfolio.
 In-house design teams for specialty
ECMs (e.g., lighting) that can have
complex design and performance
issues.
Project Complexity and Guaranteed
Savings
 In-house design teams for specialty
ECMs that can have complex design
and performance issues.

Table 4-14 (cont’d)
Risk Category
Commissioning

Delphi %a
100%
(100%)

Quartb,c
2

Risks and Causes
 Commissioning is often taken out of the project if cost overruns
are projected.
 Failure to commission can effect system performance.
 Implementing and managing a controls program can add extra
cost and delay to commissioning and closeout.

Operations and
Maintenance
Practices

95%
(95%)

1





Measurement
and Verification
of Savings

100%
(100%)

2







Project
Management
Over the Project
Life-Cycle

74%
(74%)

1





Cheapest equipment sometimes selected – leads to sub-optimal
O&M savings.
Missed opportunities to verify ECM performance if fail to
commission.
Missed opportunities to ensure that calibration, operation, and
maintenance procedures are well-understood and documented.
Failure to include O&M in the M&V plan may lead to missed
savings.
Improper measurement.
Lack of sub meters.
M&V protocols that do not capture non-energy benefits of EPCs
may understate overall system performance.
Poorly-designed M&V sampling protocols may not accurately
reflect the overall performance of the ECM package.
Use of an inaccurate baseline.
ESCO personnel turnover.
Handoffs to different managers at each phase; can often be a long
time between handoffs.

200

Controls/Mitigation
 Balance amount of commissioning
with project specifics to save costs.
 Commission every point of a building
automation system.
 Commission with the customer.
 ESCO self-performs commissioning.
 Improve communications and
collaboration with controls vendors.
 Delineate ESCO and owner
responsibilities for O&M in the
contract.
 Stipulate factors that the ESCO does
not have control of.






Internal M&V review process.
Review M&V plan at the same time
as energy savings are being
calculated.
Stipulate factors that the ESCO does
not have control of.

Document everything properly.

Table 4-14 (cont’d)
Risk Category
ConstructionSpecific
Concerns

Delphi %a
95%
(100%)

Quartb,c
3

Volatility of
Energy Prices

89%
(89%)

1

Risks and Causes
 Change orders are generally not allowed.
 Cost growth.
 Facility operating profile may limit times of the year when work
can be done – long waits until then and short construction seasons
may be common.
 Long lead times for equipment.
 Poor project handoffs among engineers and CMs.
 Schedule growth.
 Unknown site and facility conditions.
 Unqualified or unprofessional sub-contractors.
 Work productivity losses due to correctional facility security
procedures.
 Changing prices can reduce project value.
 Difficult to predict energy rate increases more than 2-3 years
from the present time.

Controls/Mitigation
 Build safety factors into proposals.
 Early involvement of CM in project
team.
 Pre-qualify subcontractors.
 Strong construction management.
 Utilize a mandatory handoff process.






Guarantee a quantity of energy saved,
not energy costs.
Frequently monitor energy rates.
Use an acceptable energy rate
escalation factor in the contract.
Use a floor rate to protect against
large rate decreased.

Perception of
100%
N/Ad
 Lack of knowledge regarding EPC enabling statutes by the ESCO
the Performance (100%)
limits some customers’ understanding of how they can use EPC.
Contracting
 Unethical behavior negatively impacts the industry as a whole.
Industry
Notes: a\ Percent values indicate the frequency of risk category identification by Delphi panelists. The value in parentheses is the frequency with outliers
removed.
b\ Quartile rankings come from Delphi panelists voting for the top three risk categories they believe can most negatively impact performance.
c\ Client selection factors were presented to Delphi panelists as three separate risks; however, many indicated that these should be treated as a single risk.
The value in parentheses indicates the quartile ranking when each individual client selection factor was considered collectively as a single consolidated
risk category.
d\ None of the panelists identified this risk category as being among those they believe most negatively impact project performance, therefore no quartile
score could be calculated.

201

4.4.1.1 Risk Category and Risk Modifications
Risk categories and their associated risks were modified, deleted, moved to different categories,
or added to the framework, based on the elicited expertise of the Delphi panel. Examples of such
modifications include:


The risk that the “client may go out of business before full contract payment” under the
client pre-qualification/financial factors risk category was removed. Response from
panelists was overwhelming that ESCOs typically have little concern about a client’s
long-term ability to make payments since at that point, they are paying the financier and
not the ESCO. Additionally, most panelists replied that in the MUSH market the greater
concern is the ability for a public entity to secure financing, due to diminished bond
ratings or a debt ratio that is too high. As a result, two new risks which better reflect
concerns related to public client financing were added to this risk category: (1) customer
may not be able to get financing and (2) unacceptable customer debt ratio.



Several panelists provided additional details specific to MUSH market retrofits with
regard to the “project development” risk category. These included risks such as lack of
experience in a given market (specifically mentioned for corrections retrofits) and issues
specific to the complex and often political nature of these projects. Those risks included
long procurement times for MUSH projects, the political nature of public agency decision
makers (e.g., elected, term-limited) that can cause a loss of continuity for the project, and
the fact that some public procurement methods require the completion of a technical audit
before the contract is awarded, thereby placing additional financial risk on the ESCO.



The “project management over the project life cycle” risk category was better developed
through the Delphi panel. This was included in the preliminary risk framework based on

202

Hansen (2006) and the preliminary ESCO expert interview; however, details were not
well-developed at that time. Panelists provided that detail and several also identified that
risk management activities are reviewed coincident to project phase transitions.


Two panelists identified a risk which could not be placed with a corresponding risk
category, thus a new one was added to the framework. “Perception of the Performance
Contracting Industry” was added as a risk category that addressed concerns arising from
ESCOs lack of knowledge surrounding legislative authorization for EPC programs,
thereby limiting the scope of projects they can provide to their clients. Additionally,
unethical practices were raised as a concern that can manifest itself on individual projects
through client mistrust and miscommunication.

4.4.1.2 Potential Risk Causes and Risk Controls/Mitigation Strategies
During the process of providing risk importance scores, panelists were asked to provide causes
and potential control strategies for their top three ranked risk categories. This was important not
only in comprehensively defining each risk category, but also in providing inputs to the risk
scenario development process described in Chapter 5.

4.4.1.3 Risk Category Identification Frequency and Relative Risk Importance
Risk identification frequency and relative risk importance scores, as described above, were added
to the risk framework to provide a measure of the Delphi panel’s specific risks of concern. This
is particularly useful for future research when identifying and evaluating project risks with
panels comprised of different experts, and can help to facilitate cross-case comparisons between
this data set and others.

203

4.4.2 Relationship Among Delphi Data, Risk Framework, and Retrofit Process Model
The expertise elicited from Delphi panelists was used as the primary data source for developing
the EPC retrofit process model, as well as the refined risk framework. During analysis, patterns
emerged relative to the relationships between the incidence and detection of project risks across
project phases and the types of information used in making key decisions. Panelists provided
information that allowed risk categories to be matched to the quantitative information and
expertise used when making decisions throughout the retrofit process, which established a link
between the two outputs of this chapter, depicted in Figure 4-10.

Figure 4-10. Relationship Between EPC Process Model and ESCO Risk Framework
4.5 EPC RETROFIT PROCESS MODEL DEVELOPMENT
ESCOs use a variety of information sources when developing and executing EPC retrofits. These
sources can be categorized as being either quantitative information (e.g., energy model outputs,
databases, design guidelines) or expert knowledge, as defined by Duah (2014). In order to fully
204

understand the context in which risks are borne by ESCOs when undertaking EPC retrofits, it is
critical to examine how different types of information are used during the retrofit process.

Information quality and availability is critical to the proper analysis and evaluation of risks
throughout the project life cycle (Zavadskas et al. 2010). Knowledge acquisition has been given
as a critical function during risk identification (Chapman 1998; Thaheem and De Marco 2013)
and as the first step during qualitative risk analysis (Chapman 2001). An EPC retrofit process
map that specifies information sources and project risks over the project life cycle assists in
connecting the work undertaken during a retrofit with its related risks.

Ultimately, there are three overarching needs for the development of an EPC retrofit process
model:


Panelists ranked energy audit quality as the most important risk category across all three
measures used: (1) Pareto/quartile, (2) consideration frequency, and (3) Delphi round 2.
The IGA is essentially a comprehensive information collection and analysis task during
which time facility information and financial information are used to design retrofit
measures, specify M&V elements, and calculate the performance guarantee. As a result,
the connection between information sources, project phases, and key decisions must be
well-understood;



As part of developing a comprehensive understanding of the context in which EPC
retrofit decisions are made, the ways in which information is used when making
decisions needs to be understood; and

205



Information availability and quality has also been identified as a potential barrier to the
wider adoption of energy efficient retrofits.

4.5.1 Information is a Barrier to Energy Efficient Retrofit Adoption
Cagno et al. (2013) reviewed several frameworks that have been proposed to explain the energy
efficiency gap – the difference between potentially profitable energy efficiency retrofits and
those that are actually realized. Issues related to information were present in all of the
frameworks reviewed by the authors, either as a primary element or as an underlying issue that
influences primary framework elements. Sorrell et al. (2000) (cited by Cagno et al. 2013),
provided a major contribution by developing a framework focused on four economic (market and
non-market failures) and non-economic (behavioral and organizational) theories. Information
and its use in decision making was identified as a key barrier in three out of the four frameworks.

Cagno et al. (2013) further refined Sorrell et al.’s (2000) original taxonomy by conducting a
comprehensive literature review focused on missing elements, overlaps, and implicit
interactions. The authors proposed further empirical investigation of information barriers, to
include lack of information on costs and benefits, unclear information provided by technology
suppliers, mistrust of information, and information issues related to energy contracts arising from
poor communication and unclear information provided by energy suppliers.

While this research is not concerned with identifying or classifying such information barriers,
their presence needs to be accounted for in order to understand the EPC retrofit decision process.

206

As a result, the final model must include connections among sources and types of information,
EPC retrofit project phases, and their attendant decisions made under conditions of risk.

4.5.2 Need for Expert Information
The taxonomy proposed by Sorrell et al. (2000) included two information-related barriers as part
of the behavioral theoretical framework. This classification involved the ways in which people
make decisions based on imperfect information. Very often decisions are made using “rules of
thumb” or heuristics, as a result of imperfect information and the ways in which information is
presented. This was described by Klotz (2011) as part of a broader discussion of cognitive biases
in the use of such information. Ma et al. (2012) classified the selection of appropriate ECMs as a
multi-objective optimization problem, and they stated that the problem can be developed and
solved either using a model-based approach or through the use of an expert system. The latter
option uses information elicited from disciplinary experts, rather than relying on modeled
outcomes.

This use and reliance on expert-based information provides the entry-point to the system for
cognitive biases due to information quality, availability, and the use of heuristics.
Duah (2014) and Syal et al. (2014) identified the critical role that expert information plays in the
energy efficient retrofit process. Expertise is frequently used as a part of the EPC retrofit
decision process, particularly during risk identification and evaluation, which was noted by the
Delphi panel and in the literature (Banaitiene and Banaitis 2012). The reliance on expert
knowledge, whether in a formal or informal context, is therefore an important element of the
EPC retrofit process model, and its sources and use should be included.

207

4.5.3 Review of Existing Models
A variety of energy efficient construction process models can be found in the literature (Horsley,
France, and Quatermass 2003; Juan et al. 2009; Kolokotsa et al. 2009; Jones and Bogus 2010;
Samuel 2011; Ferreira et al. 2013; Syal et al. 2014). These are broadly applicable across a range
of building types and energy efficiency goals, including new construction of university
dormitories delivered using a private finance initiative, single family residential retrofits, new
construction and retrofit projects that seek to improve indoor environmental quality, and
commercial buildings. While none of these models address EPC retrofits or explicitly address
MUSH market buildings, they do provide a basis for developing the EPC retrofit process model.
The decision frameworks developed by Syal et al. (2014) and Ma et al. (2012) were reviewed to
assist with the development of the model.

Syal et al. (2014) initially developed an energy retrofit decision process (ERDP) model to
understand the types of information and their interrelationships as part of developing an
intelligent decision support system (IDSS) to support enhanced residential energy efficient
retrofits. The ERDP model focuses on key decisions related to the identification, prioritization,
and installation of retrofit measures (Figure 4-11). Since the focus of the IDSS developed by
Syal et al. (2014) is the provision of expert information to homeowners as part of the overall
energy efficient retrofit process, the model focuses on aspects related to selecting and installing
retrofit measures. As a result, the project development process, to include the energy audit and
contractual decisions, are not included in the model. The ERDP model does, however, provide an
excellent example of the integration of quantitative and qualitative information as well as the use
of expert knowledge during its development process.

208

Figure 4-11. Residential Energy Retrofit Decision Process Model
(Source: Syal et al. 2014)
Ma et al. (2012) developed a process model for the identification and implementation of retrofit
measures across a variety of building types (Figure 4-12), which provides significant focus to
pre- retrofit activities and the types of information and resources needed to support retrofit
activities. The model also recognizes the critical role played by M&V once retrofit measures
have been installed. While this process model expands on the ERDP developed by Syal et al.
(2014) through consideration of pre-retrofit and post-retrofit activities within the context of
information needed for decision making, risk assessment is presented as a single step in the
decision process. Analysis of data using the knowledge elicitation strategy reveals that risk
assessment is an iterative process that occurs during all EPC retrofit project phases, with primary
emphasis during project development, energy audit, and retrofit design phases. This evidences
the need to fully incorporate risk assessment throughout the EPC retrofit process model.
209

Figure 4-12. Sustainable Building Retrofit Process Model
(Source: Ma et al. 2012)
210

4.5.4 Information Needs
As described above, the EPC retrofit process is complex and lengthy. Decisions must be made
during each project phase with regard to the technical, financial, and political aspects of these
projects, and robust risk assessment is required to ensure that these decisions are made
appropriately. Furthermore, the success of these projects relies on a significant amount of
quantitative information and expert knowledge applied across a broad range of subjects. Ma et al.
(2012) described six key elements that can impact the success of sustainable building retrofits:
(1) policies and regulations, (2) client resources and expectations, (3) building-specific
information, (4) other uncertainty factors , (5) human factors, and (6) retrofit technologies.

Samuel (2011) developed a framework for information needed to address barriers to residential
energy efficient retrofit based on construction phases where these barriers and information needs
are present. Information was classified as addressing barriers related to project performance,
cost, and construction management and installation.

Both papers point to the need for an information framework in creating the EPC retrofit process
model. While these models do not inherently provide comprehensive treatment of the
information needs for EPC retrofits, they provide an example of the connection between needed
information and the decisions they support. This highlights the potential to use elements of the
preliminary risk framework to serve as the information model for the EPC retrofit process model.

4.6 EPC RETROFIT PROCESS MODEL ELEMENTS
Based on a review of elicited and constructed knowledge from the Delphi panel, a large number

211

of ESCO risks in EPC retrofits are based on information and its use (Table 4-14). Decisions are
made throughout the project life cycle that rely on appropriate and timely information, and many
of those decisions can have significant implications on project risks. The development of an EPC
retrofit process model is a preliminary effort to categorize the information needed throughout the
EPC retrofit process, aligned with key decisions and their relationship to project-level risks.

4.6.1 Conceptual Grounding
A key underpinning of the EPC retrofit process model is the relationship among the ability to
influence design, risk, and uncertainty; the cost to do so; and potential performance impacts,
measured as cost savings on an EPC retrofit, at different stages in the project life cycle. This
relationship, as depicted in Figure 4-8, evidences the notion that cost and performance risks can
be efficiently controlled by effective decision making early in the project.

As described in Table 4-14, many of the causes for the two top-ranked risk categories
(particularly IGA quality-related risks) given by the Delphi panelists resulted from inexperience,
lack of information, improper information, or misinterpretation by ESCO professionals. Key to
depicting the role of information in the retrofit decision process is connecting decisions made
during critical project phases with information upon which those decisions are made.

4.6.2 Key EPC Retrofit Decisions and Information by Project Phase
The previous sections of this chapter were focused on the use of elicited expertise and
constructed knowledge to construct a refined risk framework for ESCOs undertaking MUSH
market EPC retrofits. This knowledge also contributed to understanding the connection among

212

key project decisions and risk categories, the risk profile of each phase, and the types of
information utilized in such projects. That information is brought together through the
development of the EPC retrofit process model.

The EPC retrofit process model is depicted in Figure 4-13. Key elements include EPC retrofit
project phases, quantitative information, expert knowledge, key decisions, and risks. Each of
these model elements are described in the sections below, organized by project phase.

4.6.3 Project Development
The preliminary risk framework identified three sub-factors related to the “client selection” risk
category. These were financial factors, facility/technical factors, and people factors. Project
development risks also occurred during this phase, which consisted of contractual- and clientrelated issues as well as issues with external parties.

4.6.3.1 Decisions Made During the Project Development Phase
The principal decision made by the ESCO during the project development phase is the decision
to bid based on project- and customer-specific characteristics. The risk importance score for the
project development phase risk category (score=7) ranked third, behind energy audit risks
(score=10), and project execution (score=7.5).

213

Figure 4-13. Energy Performance Contract Retrofit Process Model

214

4.6.3.2 Quantitative Information
Key information needs and sources as identified by the panelists included:


The preliminary project feasibility study, often conducted as a conceptual, or Phase I
energy audit, which relies on the following information sources:
o Utility bills;
o Past retrofits;
o Facility information (e.g., conditioned gross facility square footage, building ages,
locations of hazardous materials [e.g., asbestos], and code non-compliant
infrastructure); and
o Rapid assessment of retrofit potential. This rapid assessment is based on
information about utility rebates and incentives that the project may potentially
qualify for, and benchmarking of the facility against similar facilities in similar
climate zones, as found in the U.S. Department of Energy Commercial Buildings
Portal and available buildings databases.



The customer credit rating is used primarily to determine the client’s ability to access
capital. Most panelists indicated use of public data for this purpose, to include the Data
Universal Numbering System (DUNS) (Dunn & Bradstreet 2014). Panelists who
conducted EPC retrofits for the U.S. Department of Housing and Urban Development
reported mandatory use of the Public Housing Assessment System (PHAS), a tool that
measures the performance of public housing agencies across the United States (U.S.
Department of Housing and Urban Development 2014).

215

4.6.3.3 Expert Knowledge
Panelists reported reliance on expert knowledge when assessing risk categories in the project
development phase and in support of the decision to bid. Expert knowledge was used as follows:


Customer capacity to undertake O&M - This was specifically reported as a concern in K12 school retrofits, where schools typically have a small maintenance staff that often has
limited training in operating complex systems. This was also listed as a concern in some
correctional facility retrofits, owing to small physical plant and maintenance staff
complements at many facilities. Panelists reported that there are no rules to determine
whether a client is adequately prepared to undertake O&M activities; rather, the
determination is made on a case-by-case basis and a review of retrofit technologies and
the qualification of the client’s building operations staff.



Assess the likelihood of the building being closed or significantly changing its use profile
(e.g., days/hours of operation, occupant population) during the performance period - This
determination is often based on past experience with similar facility types, the same
owner (e.g., state agency, school board), and a general knowledge of environmental and
fiscal conditions affecting the facility’s future use.

4.6.4 Energy Audit
The principal activity during the energy audit phase is the conduct of the IGA and reporting of
results. As a result, this phase is primarily concerned with information collection and analysis.
The Delphi panel was clear that this phase requires significant time and attention and should be
approached deliberately and comprehensively. One panelist identified a “tension” between the

216

business development function and the need to spend time to conduct a thorough IGA, and
attributed the cause of many potential risks to rushing through the IGA.

4.6.4.1 Decisions Made During the Energy Audit Phase
While decisions made in this phase are primarily related to the conduct of the IGA, the audit
results have a significant impact on “downstream” project phases (e.g., those that occur after the
IGA is finalized). As a result, phases that are beyond the “window of opportunity” described by
Horsley et al. (2003) are shaped by activities and decisions arising from the energy audit phase.
The two primary decisions resulting from the energy audit phase are development of the energy
baseline and identifying areas for improvement for the retrofit design. An incorrect or disputed
baseline can give rise to miscalculations during retrofit design, and errors related to M&V during
project execution. Problems with identifying areas for improvement can have similar negative
effects, and can result in missed opportunities for other retrofit measures that are not included.

4.6.4.2 Quantitative Information
This phase benefits from a diversity of information types and sources, as shown in Table 4-15.
The sheer scope and volume of the information required by the IGA can lead to missing
equipment during surveys, lighting miscounts, and inaccurate or missing information necessary
to effectively assess the current state of facility energy use. Much of this information is provided
by the client and it can often be difficult for the ESCO to recognize when information is missing
or out of date. This information includes facility as-built drawings, equipment operating
schedules, and utility bills, as well as that obtained through direct observation (e.g., facility

217

Table 4-15. Quantitative Information Needed During Energy Audit Phase
Information Description
Information Description
 Monthly utility bill data
 Emissions reductions resulting from
o Electricity – usage (kWh), demand
energy efficiency.
(kW and kVa), power factor penalties
 Benchmark data based on EPC retrofit
(if assessed), and the total bill ($);
goals.
o Other fuels – billing period,
o Cost of delay (CoD), where
consumption units (e.g., CCF,
CoD= -(energy cost savings for period +
gallons), cost per unit, and the total
O&M savings for period) + initial investment
bill ($); and
prorated
o Energy use normed to BTUs to permit
 Cost avoidance (Ca), normalized to
cross-fuel source comparisons.
changed conditions (e.g., increased
 Energy consumption per unit of product
conditioned gross floor area), where
(for industrial facilities).
Ca = (baseline cost X current energy cost) –
current costs
o
O&M costs (e.g., training, staff time on
 Facility information:
O&M-related activities).
o Conditioned gross square footage of
o O&M service log (e.g., frequency and
building(s);
severity of required O&M activities).
o Operational profile, end-use(s),
occupant types, building ages, age(s)
o Inventory of all energy-consuming
of previous retrofits; and
equipment (location, use, operational
o Known locations of hazardous
profile, descriptive information,
materials (e.g., asbestos) and code
nameplate information).
non-compliant infrastructure.
 Technical requirements of and eligibility
for utility rebate, incentive, and grant
programs.
 Current and estimated future energy
 Emissions reductions resulting from
prices.
energy efficiency.
 Technical requirements of and eligibility
 Benchmark data based on EPC retrofit
for utility rebate, incentive, and grant
goals.
programs.
condition, light fixture counts) and informal information collected from building occupants and
managers (e.g., desired indoor environmental conditions, confirmation of IGA findings).

4.6.4.3 Expert Knowledge
Expert knowledge is a significant factor for the success of the energy audit phase. Two primary
types of expert knowledge are utilized:

218



Experience with similar project and facility types – Panelists, particularly those involved
with correctional facility EPC retrofits, identified past project experience as a critical
factor to the success of the project. These projects have a great deal of unique factors,
including reduced productivity due to daily check-in and check-out procedures and
population control issues once inside the facility, security needs for all retrofit activities
and technologies, and the hierarchical nature of corrections departments that can create a
“command and control” approach to these projects. These factors, among others, need to
be proactively recognized and understood during the energy audit phase to minimize risks
in later project phases.



Contract length restrictions - While the length of the project performance period is used
as quantitative information when completing energy models and financial analyses,
expertise is required during early project phases to understand customer preferences and
legislatively-mandated contract length terms, and their impact on the scope of work.

4.6.5 Retrofit Design
Activities during the retrofit design phase are directed toward the final retrofit design. This is
primarily an analytical phase, where data obtained during the energy audit phase are subjected to
further analysis focused on ECM and facility technical, energy, and project financial parameters.
Risks related to ECM selection and installation are realized here, as are risks related to volatility
of energy prices and the way in which these costs are escalated.

219

4.6.5.1 Decisions Made During the Retrofit Design Phase
The key decisions occurring during the retrofit design phase are the final design of the selected
ECM package based on findings from the IGA and the percentage of accrued savings to be
guaranteed. One panelist commented on the latter issue, stating that he discusses the difference
between modeled savings and guaranteed savings (often termed “deration”) with clients as a risk
mitigation strategy. His belief was that this approach allows him to share risks with the client by
negotiating the deration against client-driven scope that does not have an associated energy
savings component. This can result in agreeing to a higher energy escalation factor, or stipulating
savings that would normally be measured and verified, as a way to deliver the client’s desired
scope while minimizing the ESCOs risk exposure.

These decisions directly impact the project’s performance risk; that is, the ability for the project
to return the guaranteed savings without requiring the ESCO to incur additional costs during the
project execution and energy performance phases.

4.6.5.2 Quantitative Information
Information used to develop the package of retrofit measures includes:


Financial analysis (cash flow, LCCA), based on the following information:
o Current and estimated future energy prices;
o Technical requirements of and eligibility for utility rebate, incentive, and grant
programs;
o Equipment replacement and O&M costs ;
o Cost of delay;

220

o Cost avoidance; and


Outputs from computational energy models such as DOE2, eQuest, and TRNSYS, which
are used to predict retrofit performance based on facility-specific factors. Panelists
indicated that limited probabilistic ability of such tools prevents the development of
predictive, scenario-driven model runs to address key uncertainties when designing
ECMs. Heo et al. (2012) proposed a Bayesian method to address such concerns; however,
most panelists were unaware of such options.



Results of system design tools such as Trane TRACE that are directed toward specific
types of equipment or particular manufacturers.



Advanced Energy Retrofit Guides, such as those published by ASHRAE (ASHRAE
2012) and the U.S. Department of Energy (U.S. Department of Energy 2013b).

4.6.5.3 Expert Information
Two primary forms of expert information used during the retrofit design phase are:


Owner and facility needs. Energy service company professionals must balance project
needs with the intent and requirements of the facility owner, and of the facility itself. For
example many public buildings, particularly schools and correctional facilities, have had
a decades-long history of deferred capital expenditures. As a result, significant roof and
building envelope problems persist in these buildings and clients often wish to include
major upgrades as part of the EPC scope of work, despite the absence of a direct payback
from these measures. ESCO professionals report negotiating such scope with clients,
either through discussion of deration, as described above, or through the use of project

221

contingency fees in the later years of the performance period, once guaranteed savings
have been assured.


Non-energy benefits (NEBs). While some NEBs can be, and are treated like energy
savings (i.e., water savings), the inclusion of most in an EPC retrofit relies on previous
experience of the ESCO with these measures, and willingness by the project team to
attempt creative approaches to finding additional savings opportunities. NEBs are most
often addressed through stipulation, and panelists reported an unwillingness to attempt
M&V protocols for less-clearly defined benefits.

4.6.6 Project Execution and Energy Performance
4.6.6.1 Decisions Made During the Project Execution and Energy Performance Phases
Decisions made during these phases take place outside of the “window of opportunity” described
by Horsley et al. (2003) when decisions can have maximum positive impact on design, level of
risk, and overall project performance at minimal cost. These phases are, therefore, not the
primary focus of the EPC retrofit process model, since decisions made once construction begins
have limited ability to positively influence the project and are likely to incur higher costs.
Despite that fact, decisions are made during earlier project phases that are realized during project
execution and energy performance, such as design elements that are inconsistent with IGA
results, thereby requiring equipment substitution or field changes during construction or updates
and revisions to the energy baseline.

Project performance during these latter two phases can directly influence M&V risks by creating
inconsistences with the planning documents that served as an input to performance guarantee

222

calculations and by creating conditions that necessitate changes to the baseline. Performance
during project execution and performance can also directly affect performance risk. According to
multiple panelists, since the driver for EPC projects is being able to meet the performance
guarantee, cost controls are of paramount importance and tend to be the deciding factor when the
ESCO detects a need to make changes during execution and performance.
The totality of the project occurring up until execution and performance phases influences the
risks that are inherent in these later stages. As a result, specific interactions among earlier risk
categories in the earlier project phases and later project risk categories are not depicted in the
retrofit process model. Panelists identified the following risk categories as occurring during these
phases:


Project execution, includes commissioning- and construction-related risks.



Energy performance, includes risks related to O&M practices and M&V.

4.6.6.2 Quantitative Information
While the Delphi panel identified risks related to failure to commission or improper
commissioning, they also regarded the commissioning process as an information gathering and
verification step. In many cases it is the only time that equipment can be verified against the
design intent and performance specifications, creating an opportunity to minimize risks through a
process that itself bears its own risks. It is treated here as a source of quantitative information;
however, its placement during later project phases indicates a higher cost and lower potential
impact of changes made as a result of commissioning-related information.

223

4.7 CHAPTER SUMMARY
This chapter presented data from the Delphi panel related to risk identification as well as causes
and mitigation factors used. The refined risk framework and EPC retrofit process model were
developed from this information. These outputs are important in the development of a risk
analysis and evaluation method in Chapter 5. The retrofit process model will be used to guide
participants in the SFMEA panel as they construct and evaluate risk scenarios, paying attention
to the time-bound nature of decisions and realized risks throughout the project life cycle. The
refined risk framework provides additional input to the SFMEA panel relevant to risk
identification and mitigation measures. Additionally, the Delphi panelists provided an
assessment of the most important risk categories for ESCOs undertaking EPC retrofits. Two risk
categories, energy audit quality, and ECM selection and installation were selected for further
analysis and evaluation via the SFMEA process in Chapter 5.

224

CHAPTER 5
RISK-BASED LIFE CYCLE COST MODEL

5.1 INTRODUCTION
Chapter 4 described the collection and analysis of data via a Delphi panel consisting of 19 energy
service company (ESCO) industry experts. The panel provided information regarding risk
identification, unique risk aspects of municipality, university, school, and hospital (MUSH)
market retrofits, and a prioritization of the top risk factors that can negatively affect the success
of energy performance contracting (EPC) projects. Consensus was reached on these issues by
panel members. The data was used to develop two related outputs, which served as inputs to the
work conducted in this chapter. First, the preliminary framework for ESCO risks in MUSH
market EPC retrofits, presented in Chapter 2, was refined in Chapter 4 through modifications
made as a result of elicited and constructed knowledge from the Delphi panel. Second, the EPC
retrofit process was modeled in Chapter 4, which incorporated sources of information used in
making key decisions during each project phase. These outputs were developed as inputs to the
failure mode and effects analysis (FMEA) method used to analyze and evaluate risks related to
investment grade audit (IGA) quality and energy conservation measures (ECM) selection and
installation, described in this chapter. While data collection efforts in Chapter 4 were focused
broadly on MUSH market retrofits, efforts undertaken in this chapter have a primary focus on
correctional facility EPC projects.

5.1.1 Glossary
Frequently used terms in this chapter are defined as below:

225



EC: expected cost; a measure of risk that assesses the probability of the occurrence and
missed detection of a fault, in terms of cost over the life cycle of the fault condition (Gilchrist



1993); calculated as EC= Pcause x Peffect x Cost;



ECPV: Present value of the expected cost;



ELCV: expected life cycle value; the amount of risk (expressed as expected cost) effectively
mitigated at a given cost, over the project life cycle; calculated as ELCV = EC Costmitigation - Riskremaining;



FMEA: failure mode and effects analysis;



Life Cycle Cost: The systematic consideration of all costs and revenues associated with the
acquisition, use and maintenance and disposal of an asset (Kishk et al. 2003);



Life Cycle Cost-Based FMEA: FMEA analysis that uses cost as a measure of criticality,
includes scenario-based FMEA;



Posterior Probability: The conditional probability of an end effect occurring after considering
all intermediate effects preceding it; Peffect;



Prior Probability: The probability of risk cause occurring before taking into account further
evidence in the risk scenario; cause probability; Pcause;



Risk Scenario: A cause-effect chain describing a root cause, end effect, and potentially
intermediate effects, which lead to the calculation of an EC value;



Risk Scenario Map: Multiple related scenarios organized together in a map to describe a
particular system or function of interest;

226



RPN: risk priority number; traditional measure of risk criticality utilized by FMEA that
multiplies ordinal rankings (1-10 scale) of risk occurrence, risk severity, and risk detection to
prioritize risks for further treatment;



Scenario-Based FMEA: FMEA method that analyzes risk by constructing cause-effect chains
to describe failures rather than defining a single cause-single failure mode relationship; and



Single Present Value: SPV; calculates the present value of a future cash amount, for one-time
amounts; SPV factor is calculated as SPV =

1
(1 d)t

where d is the discount rate and t is the

year at which SPV is calculated.

5.1.2 Chapter Objectives
The primary goal of this chapter is the analysis and evaluation of ESCO risks in MUSH market
EPC retrofits. The research activities described in this chapter focused on a sub-market of
MUSH by analyzing these risks in correctional facility projects. Based on the selection criteria
presented in Chapter 2, scenario-based failure mode and effects analysis (SFMEA) was utilized
to meet the objectives of this chapter. These criteria included the ability of the method to address
life cycle costs through risk evaluation, to favor the use of expert knowledge, to provide insight
during project design and development stages. Results of the SFMEA were used to develop a life
cycle cost-based risk criticality and mitigation framework, which is the main output of this
chapter. The framework was then parameterized and developed into a life cycle cost-based risk
model which is pilot tested in Chapter 6. The following objectives guide the work in this chapter:


To implement risk analysis and evaluation through the use of SFMEA, utilizing
previously elicited and constructed knowledge to establish a framework for risk analysis;

227



To analyze risk criticality and understand the methodological relationships between risk
priority number (RPN) and expected cost (EC) through the use of elicited expertise;



To evaluate risks through developed risk control strategies; and



To parameterize the risk analysis and evaluation framework developed in this chapter and
construct a risk model to assess the cost-based risk of project-level factors related to the
investment grade audit (IGA) and the selection and installation of energy conservation
measures (ECMs).

5.2 RISK ANALYSIS AND EVALUATION FRAMEWORK DEVELOPMENT PROCESS
The building blocks for constructing the risk analysis framework include a priori knowledge,
represented by the preliminary risk framework (Chapter 2) and a posteriori knowledge which
was obtained through elicited expertise; the latter was the subject of the outputs from Chapter 4.
This information was needed to identify risks and prioritize them for further analysis.
Additionally, qualitative data regarding risk causes and controls/mitigation strategies was
collected and analyzed during this step. This chapter presents efforts to quantify this data through
a risk analysis and evaluation process, which is depicted in Figure 5-1.

Figure 5-1. Risk Analysis and Evaluation Process
As discussed in chapters 2 and 3, the quantitative risk analysis technique selected for this
research was SFMEA, described by Kmenta and Ishii (2000), Kmenta (2002), Kmenta and Ishii

228

(2004), and Rhee and Spencer (2009). Reasons for selection included the ability of the technique
to analyze risks based on expert knowledge, its systematic and comprehensive approach to
analyzing potential failures, and SFMEA’s theoretical grounding in Bayesian belief networks
(BBN), enabling the development of probabilistic measures of risk which are applied to causeeffect scenarios. Strengths of applying this technique to the research specifically relate to the
ability for experts to develop risk scenarios based on their collective expertise and evaluate them
using likelihood estimates of occurrence and costs. Using cost as a consequence measure is
particularly suitable for construction engineering and management professionals, as this is a
regularly-calculated and monitored metric. In a survey of 271 construction professionals from 56
countries, Thaheem and DeMarco (2013) found that expected monetary value was favored as a
risk analysis technique by approximately 30% of respondents; furthermore, 65% of respondents
reported seeking risk software outputs related to cost. Cost has also historically been a significant
focus of quantitative risk analysis techniques, generally applied to the cost estimate (Edwards
and Bowen 1998).

The development of the framework for the SFMEA consists of five steps: (1) review selected
risk categories in the refined risk framework; (2) construct failure scenario maps; (3) evaluate
risk criticality for each scenario; (4) recommend and evaluate mitigation strategies; and (5)
calculate the expected life cycle value (Figure 5-2). A panel of four ESCO and retrofit
professionals was assembled to conduct the analysis represented by each step of the framework.
Panelist recruitment and selection, as well as final panel composition, is discussed in section 5.3.
Each step of the process is described in detail in the sections below.

229

Figure 5-2. Risk Analysis and Evaluation Framework Development Steps
5.2.1 Review Selected Risk Categories
Risk causes related to the IGA quality and ECM selection and installation risk factors from the
refined framework were reviewed. Panelists reviewed the 27 potential risk causes for both
categories and made a consensus-based decision whether to evaluate them further. Decision
criteria included the group’s agreement that the listed items accurately reflected root causes of
project-level risks, that the items were assigned to the correct risk category, and that the items
were not duplicated within the same risk category. The selection of potential root causes was
subject to validation during the next step, construct failure scenario maps.

5.2.2 Construct Failure Scenario Maps
Failure scenarios are the central element of SFMEA. As opposed to traditional failure mode and
effects analysis (FMEA), where the unit of analysis is the relationship between a single failure
cause and single failure mode, scenarios enable the creation of a cause-effect chain (Kmenta and
Ishii 2004). This chain consists of a root cause, an end effect, and possibly one or more
intermediate effects (Figure 5-3). The root cause defines the first order event that gives rise to the
elements of the failure scenario. The end effect is the verbal description of the risk consequence
of the given scenario. Intermediate effects may further define the cause-effect relationship, and
230

Figure 5-3. Example Failure Scenario
are placed between the root cause and the end effect. These describe the sequence of events that
connect the root cause to the end effect in higher order scenarios (e.g., where the root cause does
not lead directly to the end effect). The probability of the cause and that of the end effect given
any intermediate effects, termed the conditional probability of the end effect, constitutes the risk
likelihood.

As a scenario is constructed and evaluated, panelists may lengthen the chain in either direction
either as a result of detecting and including a new root cause, new intermediate effects, or a new
end effect. Multiple related scenarios may be organized into a risk map which describes the
interrelationships among causes, intermediate effects, and end effects in a single system. This
permits the creation of a one-to-many relationship among causes and effects, as one root cause
can lead to several end effects and one end effect can have many root causes. Figure 5-4 depicts
a scenario map which includes the scenario from Figure 5-3, and demonstrates how the causeeffect chain may be lengthened, how additional intermediate effects may be evaluated, and the
one-to-many relationship potential among causes and end effects in failure scenario maps.

Panelists were provided with example scenario maps and undertook an iterative process that
began with the identification of a single root cause followed by intermediate and end effects
231

Figure 5-4. Example Failure Scenario Map
connected to that root cause. The scenario maps were revised repeatedly throughout the
facilitated process as long as consensus was achieved among panelists.

5.2.3 Evaluate Risk Criticality
The assessment of risk criticality in FMEA entails identifying the risks that have the greatest
overall impact on the system under review. As described in Chapter 2, criticality is assessed in
traditional FMEA through the calculation of the RPN, which is the product of three independent
measures of the failure mode: (1) occurrence (O), (2) severity (S), and (3) detection (D)
(Stamatis 2003; Gargama and Chaturvedi 2011; Liu et al. 2013). Each measure is ranked on an
ordinal scale with values ranging between 1 and 10, therefore, the RPN score ranges from 1 to
1,000. Several methodological concerns have been raised with the calculation of the RPN and its
use to inform decisions under conditions of risk. Three such concerns include the use and
improper arithmetic operation of ordinal values (Imbeah and Guikema 2009), duplicate values of

232

the RPN that have very different characteristics (Gilchrist 1993; Kmenta and Ishii 2004), and a
lack of linguistic terms regarding priority for managing critical failures identified through RPN
values (Bowles 2004; Abdelgawad and Fayek 2010). Alternate criticality measures proposed to
address these shortcomings include cost-based techniques (Gilchrist 1993; Kmenta and Ishii
2000; Rhee and Ishii 2002; Rhee and Ishii 2003; Kmenta and Ishii 2004; von Ahsen 2008).

Cost-based risk criticality measures have focused on the use of EC, which is defined as the
product of the probability that a cost will be incurred and the cost (Gilchrist 1993; Kmenta and
Ishii 2004). The determination of EC relies on an assessment the probability of the risk cause
(prior probability) and the conditional probability of the end effect (posterior probability). With
cost as the measure of consequence, Bayesian statistics are applied to calculate the EC value for
each scenario (Figure 5-5).

Figure 5-5. Calculation of Expected Cost
(Source: Kmenta and Ishii 2004)
Through its incorporation with risk scenarios, EC can also be used to assess costs over the
project life cycle, as depicted in Figure 5-6, which shows a risk scenario map that was created to
evaluate a root cause originating during the energy audit phase. As a result of intermediate

233

Figure 5-6. Use of Risk Scenarios and Expected Cost Over Project Life Cycle
effects, this map enables the assessment of EC for four scenarios which leads to the analysis of
risks in four of the five EPC retrofit project phases.

Given the calculus of the conditional probability of the end effect, each scenario is evaluated
based on risk events occurring throughout the project life cycle. For example, the EC for
scenario 4 is calculated as the products of the probability of the ESCO having limited experience
with a given facility type (identified in this case as correctional facilities), the conditional
probability of a savings shortfall/erosion of the ESCO margin given the project schedule being
too short, the effect of improper construction timing and phasing, a failure to understand facility
operations and client goals resulting from the root cause, and the cost impact of the end effect.
This method effectively analyzed an energy audit phase risk (ESCO’s lack of experience) as it is

234

realized during the project execution phase, when a short project schedule is detected during
construction activities.

Kmenta and Ishii (2004) gave the latter point as another methodological improvement of
SFMEA over the RPN method. Several definitions of detection are offered in the literature and in
practice (Palady 1995; Stamatis 2003; AIAG 2008), which creates confusion as to which
definition actually measures the contribution towards risk. Because of this confusion, several
authors have suggested eliminating the detection score from FMEA (Bowles 1998; Society of
Automotive Engineers 2001; Bowles2004) due to a high degree of subjectivity of the ranking
and large variation among detection scores. The risk scenario map presented in Figure 5-6
addresses detection through the conditional probability of the end effect, given its placement in
project space and time, and utilizes the same concept of probability as was used to assess the
scenario’s root cause. Again, the Bayesian calculus of SFMEA then multiplies the conditional
probability by the scenario cost, thereby ensuring that the issue of risk detection is quantified
appropriately.

In order to adequately address the time value of money inherent in this analysis, and to truly
enable this method to evaluate life cycle costs, the resultant EC for each risk scenario must be
discounted based on the time in the project life cycle where the risk is realized. The single
present value (SPV) factor is used to calculate the present value of a one-time future amount
given a discount rate. The equation to determine the present value of a given EC using SPV is
(Fuller and Petersen 1995):
1

ECPV =

(1

235

)

where ECPV = the present value of the EC, ECt = the future expected cost at time t, d = the
discount rate, and t = the time in years when the risk is realized.

For the purposes of the remainder of this chapter, the non-discounted value of EC will be used
for comparing the cost-based and RPN-based measures of risk criticality. The ECPV is useful
when evaluating the life cycle cost implications associated with the risk scenarios, both
individually and in the aggregate. It is particularly necessary when providing information to aid
decisions about selecting mitigation measures, which incur costs in current dollars and
evaluating those costs and benefits in light of expected costs that have been discounted to the
period where the risk is realized. The ECPV will be used in Chapter 6 during pilot application of
the model.

The risk analysis and evaluation framework evaluated occurrence, severity, and detection ranks
and incorporated the RPN, despite criticisms of the method. This was done to enable empirical
analysis of the relationship between the RPN and EC and to evaluate the appropriateness of the
selected risk criticality measure. Kmenta and Ishii (2004) used 10,000 simulated FMEAs to
evaluate the two methods against one another. This was done to provide robustness to the
arguments in favor of EC first posed by Gilchrist (1993); however, the analysis was premised on
three assumptions: (1) there is a relationship between occurrence and probability that is
consistent, (2) cost can be expressed as a function of severity and this relationship can be
consistently mapped, and (3) a relationship exists between detection and the probability of nondetection. These three relationships were used to derive complementary EC values for each of

236

the 1,000 possible RPN combinations. Through the elicitation of expertise related to contributory
measures of risk (O, S, and D for RPN and prior and posterior probabilities and cost for EC), this
research facilitates the evaluation of the relationship of the two risk criticality measures using
panel-derived data and test the conclusions of the original study (Kmenta and Ishii 2004).

5.2.4 Recommend and Evaluate Mitigation Strategies
The refined risk framework included control measures and mitigation strategies for some of the
potential risk causes. These were provided by Delphi panelists during knowledge elicitation and
construction, as described in Chapter 4. These measures were not intended to be an inclusive list,
rather, panelists were provided the opportunity to respond to open-ended questions about the
approaches they take to controlling risks once they have been identified. As a result, some risks
had multiple mitigation strategies listed in the framework while others had none. Once the data
from the SFMEA panel was analyzed, a separate ESCO expert (one of the original Delphi
panelists) with significant risk management experience was asked to provide mitigation
strategies for each risk scenario. These consisted of selections among strategies that were
contained in the refined risk framework, the ability to add others to the list, and the ability to add
new measures, particularly in the cases of potential causes that had no associated controls or
mitigation strategies. Each mitigation strategy was assigned an efficacy score, which is a
measure of the amount of the initial risk that is controlled by investing in mitigation. This is an
important feature of calculating the ELCV, below.

5.2.5 Calculate the Expected Life Cycle Value
The expected life cycle value (ELCV) is defined as the amount of risk that has been effectively

237

mitigated, when subtracting the cost of mitigation and the remaining risk in the project due to
mitigation strategies with less than 100% efficacy. It can be calculated based on the EC in
current dollars or the ECPV, accounting for the year where the risk is realized and discounted to
the present year. Formulaically, it appears as follows:
ELCV = EC - Costmitigation - Riskremaining or ELCVPV = ECPV - Costmitigation - Riskremaining
where Riskremaining = (1-Efficacymitigation) x EC or Riskremaining = (1-Efficacymitigation) x
ECPV

The ELCV can be calculated for each risk scenario, for each scenario map, or for the total project
risk (e.g., the sum of all scenario maps). This measure provides decision support to project teams
when evaluating the cost-effectiveness of selected mitigation strategies, in light of the
incremental change in costs when risk scenarios are controlled or mitigated.

5.3 SCENARIO-BASED FMEA DATA COLLECTION AND ANALYSIS
This section describes expertise elicitation efforts to complete the risk management process for
ESCOs undertaking correctional facility EPC retrofits, begun with risk identification in Chapter
4. This data was also used to parameterize the risk analysis and evaluation framework described
in section 5.2, such that a life cycle cost-based risk model could be constructed from an analysis
of the data. Primary data collection efforts were conducted using a panel of four domain experts,
thus, expert knowledge was again sought as the central element of this research. This section
includes a discussion of panelist recruitment and selection, as well as the collection and analysis
of data leading to framework parameterization.

238

5.3.1 SFMEA Panel Composition
The recommended panel for conducting a FMEA includes individuals with ownership of the
system and process being evaluated (Stamatis 2003). Panelists must reflect the multiple
disciplines represented in the process and system under review (Teng and Ho 1996; Stamatis
2003; McDermott et al. 2009). As a result, while recommended panel sizes range from four to
nine (Stamatis 2003; McDermott et al. 2009), the most critical aspect of its composition is that
key project functions and technical disciplines are represented, to include end users of the
process being analyzed (i.e., customers). An experienced team leader is important to help guide
the team’s deliberative processes by functioning as a facilitator, allowing panel participants to
make final decisions during the FMEA.

Following this guidance, a panel was assembled that reflects domain-level expertise in the key
components of the system under review, namely the conduct of the IGA and the selection and
installation of ECMs. As a result, the following experts were recruited to the panel:


Corrections business development manager, representing a national independent ESCO;



Construction manager, representing a regional independent ESCO;



Correctional agency physical plant division manager, representing a state agency client;
and



Commissioning manager, representing a manufacturer based ESCO and MUSH market
client.

As part of the qualification process, panelists completed parts I and II of the Delphi questionnaire
(risk tolerance and professional experience/determination of expertise), with additional questions

239

focused on the level of their positions and the project phases they typically support; those results
are depicted in Table 5-1.
Table 5-1. FMEA Panel Participant Risk Tolerance and Expertise Determination
Attribute
RP-01-E
RP-02-E
RP-03-O RP-04-M
Job Level
Management
a
Project Phases Supported
All
All
All
All except PD
Risk Tolerance

Aggressive

Moderate

Moderate

Total Points

5

3.5

4

Total Categories
Years involved with EPC projects

6
25+

4

5
20

c

3
15
75%
5

Moderately
Conservative
b
2
2
<1

Total number of projects
53+
6
2
Proportion MUSH market projects 100%
100%
100%
Number of building systems
>10
5
3
impacted/project
Notes:
a\ PD = project development
b\ This panelist had <1 year of experience with EPC projects; however, this participant had over
9 years of experience as a commissioning manager, having worked on 49 projects during that
time. During that time this panelist also developed standards and procurement guidance for
initiating performance contracting projects as an owner’s representative. This individual has an
additional 35+ years of experience in mechanical construction.
c\ This panelist had 3 years of experience with EPC projects; however, this individual had 4
years of experience in commissioning and over 15 years of mechanical construction experience.
An expert facilitator was retained to co-lead the SFMEA panel, along with the researcher. This
was critical to ensure that the panel achieved its stated goals and thoroughly analyzed risk
scenarios in a comprehensive and timely manner. The researcher explained the goals of the
SFMEA and shared the specifically-developed forms with the facilitator prior to convening the
panel. A fellow researcher was included as a secondary recorder and analyst to ensure that
SFMEA details were transcribed accurately and to assist with performing life cycle cost-based
calculations during the FMEA panel.

240

Data collection activities of the expert panel utilized a custom FMEA worksheet that was
developed specifically for the SFMEA process used in this research. The data collection strategy
was depicted in Figure 3-3 and consisted of facilitated, open-ended responses from panelists with
consensus as the goal for elicited risk measures (i.e., O, S, D, probabilities, and cost). If
consensus could not be reached, median values of the range of panelists’ responses were used.
The median was selected as it has the tendency to minimize the effects of outliers (Agresti and
Findlay 2009). With a small data set (N=4), outliers could unduly influence results, therefore, a
more robust measure of central tendency was needed.

5.3.2 Data Analysis and Results
This section discusses significant findings from the collection and analysis of risk analysis and
evaluation data. The complete data set compiled from SFMEA panel members is located in
Appendix D.

5.3.2.1 Risk Scenarios and Scenario Maps
Panelists agreed upon seven root causes identified in the refined risk framework for the IGA risk
category; three potential causes were reconfigured as intermediate effects, and three were not
considered by the panel (Table 5-2). Upon review of the ECM selection and installation risk
category section of the refined risk framework, panelists agreed upon nine root causes; four
potential causes were reconfigured as intermediate effects, and two were not considered (of
which one was deemed to be a duplicate entry) (Table 5-2).

241

Table 5-2. Potential Root Causes
Potential Risk Root Cause
Energy Audit Risk Category
Facility age – code update
Misunderstanding existing conditions
Differing stakeholder needs
Failure to understand facility operations and EPC goal – lack of
EPC experience
Calculation errors
Conducting the IGA too quickly
Energy model calibration error
Inaccurate/incorrect or disputed baseline
Information availability and accuracy
Missing information
Missing risk assessment for each ECM
Establishment of the guarantee amount - difficult balance
between providing a large-enough project to generate client
excitement and ESCO value and hedging on savings
Overstatement of issues leading to mismatch with technical
needs; understatement of issues leading to reduced project value
ECM Selection and Installation Risk Category
Owners are used to design-bid-build in the public sector and are
not used to designing based on fixed budgets and do not
understand EPC cost structures
Disconnect between the design and audit intent – improper
efficiency target; different operating schedule implemented than
what was planned; different control system installed
Subcontractor quality and reliability
Unqualified and unsafe contractors create additional risks
Cheapest equipment sometimes selected – leads to sub-optimal
O&M savings
ECM package constructability and feasibility
ECMs are not aligned with the IGA findings
Failure to perform as designed
Uncertainty in factors used to predict performance
Installation location
Comfort complaints can add extra work after construction is
complete
Lighting is difficult to demonstrate before installation; owner
and occupants can be unhappy with light quality after a retrofit
“Low-hanging fruit has been picked”
Lowest-cost solutions may create value concerns during O&M
Accessibility to inmates/physical security

242

Panel Disposition
Retain
Retain
Retain
Retain as Two Root Causes
Remove
Retain
Retain as Intermediate Effect
Retain as Intermediate Effect
Retain
Retain as Intermediate Effect
Remove
Remove
Retain
Retain
Retain as Intermediate Effect
Retain
Retain
Retain First Half - Second
Half as Intermediate Effect
Retain
Remove – Duplicate with
ECM package feasibility
Retain
Retain as Intermediate Effect
Remove
Retain as Intermediate Effect
Retain
Retain
Retain as Intermediate Effect
Retain

Once root causes were agreed upon, panelists constructed 12 risk scenario maps, consisting of 77
individual risk scenarios. The risk scenario maps are provided in Appendix D; two example
scenario maps are provided in Figure 5-7, and a list of scenarios, their causes, intermediate
effects, and end effects is provided in Table 5-3. The 77 scenarios included effects from each of
the five EPC project phases identified in Chapter 4, thereby making an assessment of project life
cycle costs feasible through the implementation of this risk analysis and evaluation framework.

243

Figure 5-7. Example Risk Scenario Maps Developed by SFMEA Panel
244

Table 5-3. SFMEA Risk Scenarios
Risk Cause
Intermediate Effect 1
Presence of AsbestosMisunderstanding
Containing Materials
Existing Conditions
(ACM)
1A1
Presence of AsbestosMisunderstanding
Containing Materials
Existing Conditions
(ACM)
1A2
Misunderstanding
Presence of Lead-Based
Existing Conditions
Paint
1A3
Misunderstanding
Presence of Lead-Based
Existing Conditions
Paint
1A4
Misunderstanding
Presence of FuelExisting Conditions
Related Contaminants
1A5
Misunderstanding
Presence of FuelExisting Conditions
Related Contaminants
1A6
Misunderstanding
Presence of Buried
Existing Conditions
Infrastructure
1A7
Misunderstanding
Presence of Buried
Existing Conditions
Infrastructure
1A8
Facility Age and Current Fire Suppression and
Code Requirements
Alarm System Upgrades
1B1
Facility Age and Current Electrical System
Code Requirements
Upgrades
1B2
Facility Age and Current Plumbing System
Code Requirements
Upgrades
1B3
Facility Age and Current Mechanical System
Code Requirements
Upgrades
1B4
Facility Age and Current Universal Access
Code Requirements
(ADA) Upgrades
1B5

Intermediate Effect 2

Intermediate Effect 3

End Effect
Remove/Modify
Manage in Place
Remove/Modify
Manage in Place
Remove/Modify
Manage in Place
Remove/Modify
Manage in Place
Modify Affected
System(s)
Modify Affected
System(s)
Modify Affected
System(s)
Modify Affected
System(s)
Modify Affected
System(s)

245

Table 5-3 (cont’d)
Risk Cause
Facility Age and Current
Code Requirements
1B6
Facility Age and Current
Code Requirements
1B7

Intermediate Effect 1
OSHA-Related
Upgrades
Specialty-Standard
Upgrades

1C1

Overstatement of Issues

1C2

Overstatement of Issues

Improper Equipment
Sizing
Improper Equipment
Repurposing

1C3

Overstatement of Issues

Improper Equipment
Repurposing

1C4

Overstatement of Issues

Customer-Specified
Issues

1C5

Understatement of
Issues

Improper Equipment
Sizing

1C6

Understatement of
Issues

Improper Equipment
Repurposing

1C7

Understatement of
Issues

Customer-Specified
Issues

1C8

Understatement of
Issues

Missed Opportunities

Intermediate Effect 2

Intermediate Effect 3

Inability to Meet
Performance
Expectations
Inaccurate Prediction
of Service Life/O&M
Schedule
Changes to the Size of
the Project Scope
Inability to Meet
Performance
Expectations
Inaccurate Prediction
of Service Life/O&M
Schedule
Changes to the Size of
the Project Scope
Inability to Meet
Performance
Expectations

246

Inability to Meet
Performance
Expectations
Inability to Meet
Performance
Expectations

Inability to Meet
Performance
Expectations
Inability to Meet
Performance
Expectations

End Effect
Modify Affected
System(s)
Modify Affected
System(s)
Savings Shortfall/
Erosion of
ESCO's Margin
Difficulty Getting
Financing
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin

Table 5-3 (cont’d)
Risk Cause
ESCO Lack of
Experience with Facility
Type
1D1
ESCO Lack of
Experience with Facility
Type
1D2
ESCO Lack of
Experience with Facility
Type
1D3

Intermediate Effect 1
Failure to Understand
Facility Operations and
Client Goals
Failure to Understand
Facility Operations and
Client Goals
Failure to Understand
Facility Operations and
Client Goals

ESCO Lack of
Experience with Facility
Type

Failure to Understand
Facility Operations and
Client Goals

ESCO Lack of
Experience with Facility
Type
ESCO Lack of
Experience with Facility
Type
ESCO Lack of
Experience with Facility
Type
ESCO Lack of
Experience with Facility
Type
ESCO Lack of
Experience with Facility
Type

Failure to Understand
Facility Operations and
Client Goals
Failure to Understand
Facility Operations and
Client Goals
Failure to Understand
Facility Operations and
Client Goals
Failure to Understand
Facility Operations and
Client Goals
Failure to Understand
Facility Operations and
Client Goals

1D4

1D5
1D6
1D7
1D8
1D9

Intermediate Effect 2
Overestimate Client
Capacity to Perform
O&M
Overestimate Client
Capacity to Perform
O&M
Overestimate Client
Capacity to Perform
O&M
Lack of Foresight
About Future Building
Use and Occupant
Changes
Lack of Foresight
About Future Building
Use and Occupant
Changes

Intermediate Effect 3

Improper Construction
Timing and Phasing

Project Schedule too
Short

Overestimate
Productivity Rates

Project Schedule too
Short

Overestimate
Productivity Rates
Improper
Consideration of
Security Concerns

Underestimate Project
Costs

247

End Effect
Missed
Opportunity for
O&M Contract

Reduced Savings Due
to Poorly-Performed
O&M
Reduced Savings Due
to Poorly-Performed
O&M

Reduced Project
Value for ESCO
Savings Shortfall/
Erosion of
ESCO's Margin

Improper Baseline

Savings Shortfall/
Erosion of
ESCO's Margin

Improper Retrofit
Design

Project Schedule too
Short

Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin

Table 5-3 (cont’d)
Risk Cause
ESCO Lack of
Experience with Facility
1D10 Type
ESCO Lack of
Experience with Facility
1D11 Type

1E1

Differing Stakeholder
Needs

Intermediate Effect 1
Failure to Understand
Facility Operations and
Client Goals
Failure to Understand
Facility Operations and
Client Goals
Inconsistent Information
About Facility
Operating Parameters
(e.g., too hot, too cold,
too much airflow, not
enough airflow)

1E2

Differing Stakeholder
Needs

Codes and Standards
Upgrades

1E3

Differing Stakeholder
Needs

Codes and Standards
Upgrades

1E4

Differing Stakeholder
Needs

Increased Project
Complexity

1E5

Differing Stakeholder
Needs

1F1

IGA Conducted Too
Quickly

Increased Project
Complexity
Miss Critical Functions
and Savings
Opportunities

1F2

IGA Conducted Too
Quickly

Miscount Existing
Equipment (e.g., Lights)

Intermediate Effect 2
Improper
Consideration of
Security Concerns
Improper
Consideration of
Security Concerns

Intermediate Effect 3

Retrofit Design Does
Not Satisfy All
Stakeholders
Retrofit Design Does
Not Satisfy All
Stakeholders

User Modification of
ECMs/Override
Operating Parameters
Comfort-Related
Callbacks after
Construction Complete
Controls Scope of
Work Lacks Adequate
Detail and Specificity
Comfort-Related
Callbacks after
Construction Complete

Increased Project
Complexity
Retrofit Design Does
Not Satisfy All
Stakeholders
Controls Scope of
Work Lacks Adequate
Detail and Specificity

248

Underestimate Project
Costs
Occupant Use of
Equipment for Other
Purposes

End Effect
Savings Shortfall/
Erosion of
ESCO's Margin
Legal Impacts

Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin

Table 5-3 (cont’d)
Risk Cause

Intermediate Effect 1

Intermediate Effect 2

1F3

IGA Conducted Too
Quickly

Miscount Existing
Equipment (e.g., Lights)

Energy Model
Calibration Errors

1F4

IGA Conducted Too
Quickly

Inaccurate Baseline

Energy Model
Calibration Errors

1F5

IGA Conducted Too
Quickly

Missing Facility
Information

Energy Model
Calibration Errors

IGA Conducted Too
Quickly
Facility Information
Unavailable/
Inaccurate
Facility Information
Unavailable/
Inaccurate

Missing Facility
Information

1F6
1F7

1F8
2A1

Security Concerns

2A2

Security Concerns

2A3

Security Concerns

2A4

Security Concerns

Inaccurate Baseline
Missing Facility
Information
Technology Breakage/
Damage Possibility by
Occupants
Technology Breakage/
Damage Possibility by
Occupants
Technology Breakage/
Damage Possibility by
Occupants
Technology Breakage/
Damage Possibility by
Occupants

Intermediate Effect 3

Energy Model
Calibration Errors

Change in Design
Intent for Impacted
Equipment
Change in Design
Intent for Impacted
Equipment

Rework

Equipment Impacts

Total Costs of
Operation

Project Schedule
Impacted

Delay in Accruing
Savings

249

End Effect
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin

Table 5-3 (cont’d)
Risk Cause

2A5

Security Concerns

Intermediate Effect 1
Occupant Use of
Equipment for Other
Purposes

2B1

Reduced Availability of
“Low Hanging Fruit”

Increase in Project
Complexity

2B2

Reduced Availability of
“Low Hanging Fruit”

IGA Scope Increase
Increase in Project Costs (No Additional Fee)
Less Opportunity for
Non-Energy Benefits
Project Schedule
(NEBs)
Impacted

Legal Impacts
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin

Failure to Meet Owner
Project Requirements

Rework

Savings Shortfall/
Erosion of
ESCO's Margin

Comfort Complaints

Savings Shortfall/
Erosion of
ESCO's Margin

Reduce Project Scope

Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin

Safety Incident

Legal Impacts

Reduced Availability of
“Low Hanging Fruit”
2B3
Some ECMs (e.g.,
Lighting) are Difficult to
Demonstrate before
2CD1 Installation
Some ECMs (e.g.,
Lighting) are Difficult to
Demonstrate before
2CD2 Installation
Some ECMs (e.g.,
Lighting) are Difficult to
Demonstrate before
2CD3 Installation

2CD4 Operational Concerns
2CD5 Safety Concerns

Failure to Meet Owner
Project Requirements

Intermediate Effect 2

Limited/Less
Attractive Scope

Failure to Meet Owner
Project Requirements
Failure to Meet Owner
Project Requirements
Failure to Meet Owner
Project Requirements

250

Intermediate Effect 3

Increased Opportunity
for Diminished Savings

End Effect

Table 5-3 (cont’d)
Risk Cause

Intermediate Effect 1

Intermediate Effect 2

Intermediate Effect 3

Unqualified
Subcontractors

Rework

Project Schedule
Impacted

Delay in Accruing
Savings

Unsafe Working
Conditions/Injuries
Unsafe Working
Conditions/Injuries

Safety Incident

Project Schedule
Impacted

2E3

Unqualified
Subcontractors
Unqualified
Subcontractors

2E4

Unreliable
Subcontractors

Rework

2E5

Unreliable
Subcontractors

Unsafe Working
Conditions/Injuries

Unreliable
Subcontractors
Prepare Technical Scope
of Work Based on
Standard Public DBB
Procurement
Prepare Technical Scope
of Work Based on
Standard Public DBB
Procurement
Prepare Technical Scope
of Work Based on
Standard Public DBB
Procurement

Project Schedule
Impacted

Legal Impacts
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin

Require ESCO to use
Fixed Overhead & Profit Overstate Project Soft
Rates
Costs

Savings Shortfall/
Erosion of
ESCO's Margin

Disconnect Between
Divide Procurement into IGA and Retrofit
Separate Phases
Design

Savings Shortfall/
Erosion of
ESCO's Margin

Disconnect Between
IGA and Retrofit
Design

Savings Shortfall/
Erosion of
ESCO's Margin

2E1
2E2

2E6

2F1

2F2

2F3

Incorrect Vendor
Selection

Safety Incident
Project Schedule
Impacted

Delay in Accruing
Savings

Safety Incident

Project Schedule
Impacted

251

End Effect
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin

Table 5-3 (cont’d)
Risk Cause
Select Cheapest
Available Equipment to
Meet Specifications
2G1
Select Cheapest
Available Equipment to
Meet Specifications
2G2
Actual Climate
Conditions Outside
Range of Planned
Conditions
2G3
Actual Climate
Conditions Outside
Range of Planned
Conditions
2G4

Intermediate Effect 1

Intermediate Effect 2

Reduced Equipment
Service Life

End Effect
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin

Uncertainty in Factors
Used to Predict
Performance

Savings Shortfall/
Erosion of
ESCO's Margin

Increased O&M Cost

2G5

Reduced Energy
Performance
Uncertainty in Factors
ECM Constructability & Used to Predict
Feasibility Issues
Performance

2G6

ECM Constructability &
Feasibility Issues
Rework

2G7

ECM Failure to Perform
as Designed

Reduced Energy
Performance

2G8

ECM Failure to Perform
as Designed

Rework

Reduced Equipment
Service Life

Reduced Energy
Performance

252

Intermediate Effect 3

Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin
Savings Shortfall/
Erosion of
ESCO's Margin

5.3.2.2 Expected Cost and Risk Priority Number
Panelists assigned measures of risk criticality to the 77 developed risk scenarios described in
Table 5-4. Measures were first assigned individually by each panelist; group consensus was then
developed from their individual responses. During the process of assigning prior probabilities,
panelists identified several facility-related factors that impact the way these probabilities were
assigned. Table 5-4 summarizes the facility factors that were identified and their impacts on the
calculation of prior probabilities.
Table 5-4. Impact of Facility Factors on Prior Probabilities
Risk
Scenario Facility Factor
Criteria
1A1/2
Presence of asbestos
Built/renovated before 1985
Built/renovated 1985-2000
Built/renovated after 2000
1A3/4
Presence of lead-based paint
Built before 1980
Built after 1980
1A5/6
Presence of fuel contamination Fuel tanks in vicinity of ECM work
Fuel tanks not in vicinity of ECM
work
1A7/8
Presence of buried
Assume for all correctional
infrastructure
facilities

Prior
Probability
1.00
0.75
0.25
1.00
0.00
0.85
0.00
0.80

Cost was calculated as the proportion of the ESCO margin that would be at risk if the given risk
scenario was left unmitigated. The end effect “savings shortfall/erosion of ESCO’s margin”
appeared in most scenarios. Remaining end effects included “difficulty getting financing,”
“reduced project value for ESCO,” “legal impacts,” “remove/modify,” “manage in place,” and
“modify affected system(s).” Panelists agreed in all cases that these consequences could be
adequately addressed in terms of the proportion of the ESCO’s project margin that would
potentially be at risk. In order to convert relative costs to actual costs, which was required to
calculate EC, a $2,000,000 project with a 20% margin was assumed.

253

After EC-related metrics were scored, panelists repeated the process of individually and
collaboratively recording RPN-related scores. Customized scales for O, S, and D were developed
for this research, following guidance in Stamatis (2003) and McDermott et al. (2009). The scales
are included in Appendix D.

5.3.2.3 Most Critical Risks
The consensus measure (median) was applied to panelists’ individual assessments of EC and
RPN. Values of EC and RPN were ranked from 1 (highest criticality value) to 77 (lowest
criticality value); the ten most critical risks are identified in Table 5-5.
Table 5-5. Consensus Assessment of Most Critical Risks
Risk
RPN
Risk
EC
Scenario Rank RPN Scenario Rank EC
2A2
1
410
2B1
1
$153,600.00
2A3
1
410
2F3
2
$104,625.00
2CD3
3
371
2B2
3
$102,400.00
1F7
4
352
1F8
4
$91,350.00
2CD2
5
340
1F7
5
$86,275.00
1F8
6
330
2B3
6
$76,800.00
2F3
6
330
2F1
7
$67,500.00
1A7
8
322
2F2
8
$64,125.00
1D11
9
319
1A1
9
$64,000.00
1D3
10
315
1D8
10
$63,000.00
Notes:
Shaded rows identify instances of the same risk scenario
appearing among the ten most critical risks using both EC and
RPN
The two-way analysis of RPN and EC yielded 17 discrete risk scenarios. In a traditional FMEA,
the most critical risks would be submitted for further analysis with regard to risk mitigation and
control techniques. The most important result of this pairwise analysis is that the data makes it
clear that there is not a one-to-one relationship between RPN and EC. Only three of the ten most
critical risks appeared in both of the ten most critical risks list when using RPN and EC, and in
254

no case was the rank identical for a given risk scenario among the top ten using both assessment
scales. Scenarios are approximately evenly divided between those addressing energy audit phase
root causes (risk scenarios preceded with a 1) and those related to ECM selection and installation
(risk scenarios preceded with a 2). Criticality scores obtained for the two most critical risks
(scenarios 2A2 and 2B1) represented approximately the same proportion of available points
using each rating technique; values were equal to 41% of the highest possible RPN and 38.4% of
the highest possible EC. This relationship degenerates by the time the tenth highest risk is scored
– the proportional scores are now 31.5% for RPN and 15.8% for EC.

5.3.2.4 Relationship Between EC and RPN
Further analysis is warranted based on the observed lack of a relationship between EC and RPN
after a review of the ten most critical risks derived using each criticality assessment method.
Analysis consists of the following two aspects: (1) assessment of two of the relational
assumptions given by Kmenta and Ishii (2004) and (2) assessment of the linear relationship
between the calculated criticality measures.

Two assumptions made by Kmenta and Ishii (2004) regarding their assessment of the
relationship between EC and RPN, using modeled results, are tested in this section: (1) there is a
relationship between occurrence and probability that is consistent and (2) cost can be expressed
as a function of severity and this relationship can be consistently mapped.

The relationship between occurrence and prior probability (e.g., probability of cause) was
analyzed using Spearman’s correlation, given that neither variable was distributed normally. A

255

strong, positive, monotonic relationship was detected between occurrence and prior probability
(ρ=0.986, N=77, p<0.01). This was expected given that both scales were developed based on the
likelihood or frequency of a risk cause occurring on projects, and scoring levels were consistent
between the two scales. For example, a rank of 3 on the occurrence scale indicated that a cause
occurred in 20 out of 100 projects; this corresponded to a value for Pcause of 0.20.

Kmenta and Ishii(2004) stated that a consistent relationship exists between severity and cost;
however, the scale of the cost variable may change by industry. For example, failure costs in the
aerospace industry are likely to be greater than those in consumer electronics manufacturing.
Since there is no available comparison for this data set, the first step is again to determine the
degree of relationship among the two variables. Using Spearman’s correlation, a strong, positive,
monotonic relationship was detected between severity and cost (ρ=0.706, N=77, p<0.01). To
determine the ESCO industry-specific scale of the cost variable, the equation of the line
describing the relationship between these variables was sought using the curve estimator function
of SPSS version 22 (IBM 2014) to determine the best fit curve for the data. Based on a visual
review of the scatterplot of severity and cost, three models were selected for inclusion in the
curve estimation analysis: (1) cubic, (2) power, and (3) S. Treating cost as the dependent variable
and severity as the independent variable yielded regression analysis results as shown in Table 56. The power model provided the strongest correlation between the variables (R=0.762) and also
2

explained the greatest amount of variation in cost as a function of severity (R =0.580) with the
smallest error term of the three models (σest=0.548). The equation for the curve describing this
relationship is given as:

256

ln(cost) = ln(1162.38) + (2.2 X ln(S)).
where 1162.38 is a constant and S = the severity score.
Table 5-6. Analysis of the Curve Describing the Severity and Cost Relationship
Parameter Value
Cubic Model
Power Model S Model
R
0.596
0.762
0.501
2
0.356
0.580
0.494
R
Significance
Standard Error of the Estimate

p<0.01
56,592.9

p<0.01
0.548

p<0.01
0.597

Based on the above analysis, both modeled assumptions made by Kmenta and Ishii (2004) were
determined to exist in the data set generated by SFMEA panelists.

The relationship between EC and RPN was further examined in addition to the assumptions
reviewed previously. Due to non-normality in the residuals of both variables, Spearman’s
correlation was again used to detect the presence of a relationship between the two. A moderate,
positive, monotonic relationship was detected between EC and RPN (ρ=0.562, N=77, p<0.01).
This indicates that 31.6% of the variation in one variable is explained by the value of the other.
This may be an artifact of panelists scoring EC and RPN for each risk scenario within the same
brief window of time; however, out of 308 scoring opportunities (77 scenarios each scored by
four panelists), there were fewer than 20 instances of a panelist verbalizing the use of one value
as a reference for the other. Additionally, panelists did not see calculated EC and RPN values in
real time; they only saw their probabilities, relative cost, and O, S, and D values in an attempt to
minimize bias. Based on the test statistic, 68.4% of the observed variation remains unexplained
via this relationship between variables, thereby providing evidence of a relationship that is other
than one-to-one between EC and RPN. Visual inspection of the scatterplot describing the
relationship between the two criticality measures leads to a similar conclusion (Figure 5-8).
257

Figure 5-8. EC vs RPN for all Panelist Responses and Consensus Measure (n=380)
Through the 380 instances (5 were eliminated due to panelists not providing individual scores) of
calculated values of EC and RPN shown in Figure 5-8 (data from each of the four panelists and
the consensus measure, the median value), the failure priorities for the two measures equaled
each other in just 11 instances (3.8%), which are described in Table 5-7. As review of the data
reveals, only one risk scenario received the same failure priority in more than one set of scores.
Risk scenario 1F1 received identical EC and RPN priorities in the consensus case and by panelist
RP-04-M; however, their respective scores differed greatly. Panelists RP-04-M assigned the
highest possible priority to this scenario whereas the consensus case assigned this a lower-mid
range priority rank of 43 out of 77. This also provides an example of the mixed prioritization of
results that can occur when utilizing the RPN method of determining risk criticality.

258

Table 5-7. Instances of Identical EC and RPN Priorities
Risk Scenario Panelist(s)
Priority Score
1C7
RP-04-M
47
1D2
RP-02-E
34
1E2
RP-04-M
68
1FI
Consensus (Median) Case 43
RP-04-M
1
2A4
RP-04-M
5
2E3
RP-03-O
71
2F2
RP-02-E
34
2G3
RP-04-M
61
2G7
RP-02-E
34
2G8
RP-01-E
73
Rank order disagreement was reviewed further, with minimum, maximum, and mean values of
disagreement (|RankRPN-RankEC|) calculated for each panelist and the consensus case. The
mean value across these cases was: (1) |RankRPN-RankEC|min=0, since each case had at least one
instance of rank order agreement; (2) |RankRPN-RankEC|max=61, where the maximum possible
value of rank order disagreement is 76; and (3) |RankRPN-RankEC|mean=17. Within the 96.2% of
instances with observed risk priority disagreement, several observations were noted that further
indicated the challenges of RPN use.

Instances were observed where identical RPN values had a wide range of EC values, as shown in
Table 5-8. In the given example, the same RPN yielded a variance of $72,000 in the EC.
Table 5-8. Identical RPN Values with Large EC Variance
Scenario Pcause
1C8
0.8
2F2
0.8

Peffect Cost
EC
O
0.1
$20,000.00
$1,600.00 9
0.8
$200,000.00
$128,000.00 9

S
9
9

D
2
2

RPN
162
162

Conversely, identical values of EC also yielded a large variance in RPN scores (Table 5-9).
259

Table 5-9. Identical EC Values with Large RPN Variance
Scenario Pcause
2F3
0.50
2G6
0.25

Peffect Cost
EC
O
0.2
$100,000.00
$10,000.00 6
0.50
$80,000.00
$10,000.00 3

S
8
8

D
9
6

RPN
432
144

Several instances were also detected where RPN and EC values yielded mixed priorities for risk
mitigation, as shown in Table 5-10. The risk scenario with the higher RPN value has a
significantly lower EC than the scenario with the higher RPN. This represents information that is
difficult to use by managers when making decisions about which risks to prioritize.
Table 5-10. RPN and EC Yield Mixed Risk Management Priorities
Scenario Pcause
1B4
0.80
2F3
0.50

Peffect Cost
EC
O
0.5
$20,000.00
$144,000.00 9
0.2
$100,000.00
$10,000.00 6

S
6
8

D
3
9

RPN
162
432

Finally, given the differences in possible combinations that lead to RPN and EC values, there is
the potential for significant variation of EC around a given RPN value. Multiplying O, S, and D
yielded 1,000 possible numerical combinations which result in 120 possible unique RPN values
(Kmenta and Ishii 2004). The data generated across all five cases in the SFMEA panel resulted
in 84 unique values of the RPN; 226 unique values were generated for EC. This creates the
expectation that a great deal of EC variation should be seen around each RPN value in the
dataset. This is demonstrated in Figure 5-9. The average EC variation around the RPN is 68% of
the range of EC values in this study. Variation ranged from 5% to 200%, with the greatest
amount of variation generally occurring around the largest RPN values.

While these examples further serve to demonstrate the irregular relationship between EC and
RPN, the most important observation for decision making is that EC quantifies risks in monetary
terms instead of dimensionless, ordinal scales. As a result, much can be said about the EC value
260

Figure 5-9. EC vs RPN for all Panelist Responses and Consensus Measure (n=303)
when making decisions. For example, an EC of $150,000 is twice as risky as one of $75,000 and
the representation in monetary terms means that appropriate mitigation techniques can be
selected that yield an optimal cost-benefit relationship.

5.3.2.5 Differences in Panelist Criticality Analysis
The consensus measures used for values of probabilities, cost, and O, S, and D showed
significant variation from individual panelist scores for each measure. As discussed in Chapter 4,
it was hypothesized that differences in each participant’s background contribute to their
individual analysis of project-level risks. These background attributes include their risk
tolerance, their professional tenure with ESCOs or with conducting EPC retrofits, the level of
their job, and the project phases during which they most frequently work. Since the unit of
analysis is the individual decision-maker, the SFMEA panel provides only four data points for
261

each measure of interest (e.g., probabilities, costs), where the measure of interest is examined for
each of the 77 scenarios. The small size of this data set makes traditional statistical analysis
difficult to use in detecting the strength and significance of relationships among these indicator
variables. The use of a sample size calculator indicated that a sample size of four would yield a
confidence interval of no better than 50%. While this limits the ability to analyze panelist data in
such a manner, trend lines of EC values for each scenario were prepared (Figure 5-10).

Figure 5-10. Panelist Expected Cost Variation Trend Line Plot
The panelist with the most conservative risk tolerance score (RP-04-M – moderately
conservative) generally assigned the greatest EC value to each scenario, except among those with
the lowest overall EC values. Panelist RP-03-O had a moderate risk tolerance, and the EC trend
was generally where one might expect to see it, in the middle of panelist scores. Deviation was
observed from what was expected due to risk tolerance with panelist RP-01-E (aggressive) and
panelist RP-02-E (moderate). Risk seekers would be expected to have the lowest trend in EC
values; in this case, the aggressive panelist evaluated EC higher than the moderate panelist RP262

02-E. This may be a factor of the moderate panelist scoring lower than expected, rather than the
aggressive panelist scoring higher.

The length of professional tenure related to EPC retrofits was evaluated next for each panelist.
Panelists RP-01-E and RP-03-O each have 20+ years of ESCO and/or EPC experience, whereas
panelists RP-02-E and RP-04-M had less domain experience, despite lengthy backgrounds in
mechanical construction. The most experienced panelists had EC trends that were approximately
in the middle of observed trend lines; the least experienced panelists had EC trends at the high
and low ends of the observed trend lines. The inclusion of length of professional tenure as an
interaction variable may explain some of the observed variation when considering just risk
tolerance behaviors, as level of experience has been shown to be a critical aspect of the risk
management process (Akintoye and MacLeod 1997; Akintoye et al. 1998); however, this cannot
be ascertained without further analysis. Table 5-11 provides measures of panelist variance across
key risk analysis measures, and indicates that the relationships noted above do not appear to
remain true for RPN scores.
Table 5-11. Panelist Variance in Key Risk Analysis Measures
Response Range
Key Measure Median Value
RP-01-E
RP-02-E
0.20-1.0
0.02-1.0
0.15-1.0
P
cause

Peffect
EC
RPN

RP-03-O
0.20-1.0

RP-04-M
0.20-1.0

0.00-1.0

0.00-1.0

0.00-1.0

0.00-1.0

0.00-1.0

$0-153,600
12-576

$0-$259,200
4-1,000

$0-$320,000
18-432

$0-$204,800
8-448

$0-$240,000
12-648

5.4 FRAMEWORK DEVELOPMENT
Figure 5-11 represents the design of the risk analysis and evaluation framework used to evaluate
ESCO risks when delivering EPC retrofits, with a focus on correctional facility projects. The

263

Figure 5-11. Risk Analysis and Evaluation Framework for EPC Retrofits
three primary components of the framework include the user interface, the database, and the life
cycle cost-based risk model.

The risk model is the central component of the framework and incorporates elicited and
constructed knowledge from ESCO experts, risk-specific information from the SFMEA panel
described earlier in this chapter, and quantitative information from the framework database. The
framework development steps were described in earlier sections and included knowledge from
the a priori risk framework, the a posteriori risk framework, and the EPC retrofit process model.

264

5.4.1 Life Cycle Cost-Based Model
The life cycle cost-based model uses the information obtained from the SFMEA to represent
decision maker expertise, risk preferences, and external information in examining the financial
impacts of risk scenarios and their associated mitigation strategies. There are four elements of
the model: (1) risk scenario probabilities and costs, (2) mitigation strategies cost-benefits, (3)
ELCV, and (4) decision solutions.

5.4.1.1 Risk Scenario Probabilities and Costs
Calculation of Risk Probabilities: Based on the expertise elicited from SFMEA panelists, 77
risk scenarios are available for criticality analysis. Prior and posterior probabilities for each
scenario are derived from one of the following sources:


SFMEA panelist-assigned values;



User inputs; or



Project, facility, and external environment characteristic-driven values.

The latter option consists of specific project, facility, and external environment characteristics
that were identified by SFMEA expert panelists as affecting the calculation of prior probabilities.
The classifications used for these instances are shown in Table 5-12.

Project Factors: Prior probability values were generally obtained from SFMEA panelists;
however, variables related to ECM service life-related costs were generated through a random
number obtained from a probability density function (PDF). The PDF was generated based on

265

ECM-specific service life values in the ASHRAE Service Life and Maintenance Cost Database
(ASHRAE 2014).
Table 5-12. Project-, Facility-, and External Environmental-Based Values for Probability
of Cause
Factor
Prior corrections EPC experience

Frequency of client use of designbid-build (DBB) procurement
Difficulty in demonstrating ECM
results prior to installationa

Related Risk
Scenarios
Variable Levels
Project Factors
1D1-1D11
 No experience - 0 prior projects
 Minimal experience - 1-2 prior
projects
 Moderate experience - 3-5 prior
projects
 Significant experience - 6+ prior
projects
2F1-2F3
Continuous values 0%-100%
2CD1-2CD3

Discrete values:
 ECM is (Yes) difficult to
demonstrate
 ECM is not (No) difficult to
demonstrate
Difference between ECM service
2G2, 2G7,
Continuous values - the probability of
life and probabilistically-derived
and 2G8
service life that is lower than the
service life
design parameter based on a
probabilistically-derived random
number
Facility Factors
Facility age
1A1 and 1A2  Built or renovated before 1985
 Built or renovated 1985-2000
 Built or renovated after 2000
Facility ageb
1A3 and 1A4  Built or renovated before 1980
 Built or renovated after 1980
Retrofit disturbance of fuel tanks
1A4 and 1A5 Discrete values:
and related piping
 Yes
 No
External Environment Factors
Difference between annual normal 2G3 and 2G4 Continuous values - the probability of
heating degree days (HDD) and
annual HDD differing from the design
probabilistically-derived HDD
parameter based on a probabilisticallyderived random number
Difference between annual normal 2G3 and 2G4 Continuous values - the probability of
cooling degree days (CDD) and
annual CDD differing from the design
probabilistically-derived CDD
parameter based on a probabilisticallyderived random number

266

Pcause



0.9
0.8



0.6



0.4

0.0-1.0


1.0



0.0

0.0-1.0







1.00
0.75
0.25
1.0
0.0




0.85
0.00

0.0-1.0

0.0-1.0

Facility Factors: Cause probabilities were developed through expertise elicited from SFMEA
panelists. Renovation was defined as a major renovation activity that would have likely
contacted the hazardous material of concern (e.g., previous steam line repairs resulted in removal
of asbestos pipe wrap, previous door frame repair involved either lead-based paint removal or
encapsulation). Panelists also indicated that they use a fixed prior probability of 0.85 for the
likelihood of encountering fuel-related contamination during correctional facility retrofits;
however, their primary decision making regarding this point is based on whether construction
activities that are directly or indirectly involved with installing retrofit measures will disturb fuel
tanks or associated piping.

External Environment Factors: Prior probability values were generated from a random number
that was obtained from a PDF. The PDF was generated based on historical annual HDD and
CDD values, which could then be compared to the design value of these parameters. In a
nationwide study of energy consumption at U.S. Air Force bases, Griffin (2008) determined that
HDD overall had 4.5 times the impact on energy consumption per square foot of building area as
CDD. The risk model provides the ability to identify the climate zone of the project location. For
climate zones 4 and above, the model utilizes HDD; CDD are utilized for climate zones 1
through 3.

Example Rules: The above relationships were developed in the model through conditional
statements using IF- THEN-ELSE logic. This enabled the model to represent the knowledge
elicited from the SFMEA panelists to calculate EC values for each affected scenario. As an

267

example, the probability of encountering lead-based paint that requires remediation resulting
from the facility’s age would be represented as:
IF (# square feet) built or renovated before 1980 >0 THEN Pcause = 1.0 x SF ELSE
IF (# square feet) built or renovated after 1980 THEN Pcause = 0.0 x SF

Calculation of Risk Scenario Costs: As with the calculation of probabilities, costs for each
scenario are derived from one of the following sources:


SFMEA panelist-assigned values;



User inputs; or



Externally-derived data.

Scenario-based FMEA panelists were asked to assign relative costs as a proportion of the
ESCO’s margin for a given retrofit project. This reflected the fact that the most frequent end
effect identified by SFMEA panelists was “savings shortfall/erosion of ESCO’s margin” which
was expected, given the focus of EPC retrofit projects on maintaining the margin in light of
realized energy savings performance. Other end effects appearing in scenarios included
“difficulty getting financing,” “reduced project value for ESCO,” and “legal impacts.” Panelists
agreed in all cases that these consequences could be adequately addressed in terms of the
proportion of the ESCO’s project margin that would potentially be at risk if these risk scenarios
were left unmitigated.

The first two risk scenario maps had end effects of “Remove/Modify” or “Manage in Place” and
“Modify Affected System(s),” respectively. In these cases, construction estimates were used

268

whenever possible, since risk consequences could be directly assessed utilizing industry standard
cost databases, such as those developed by RS Means (RS Means 2014). Such cost calculation
methods were applied to risk scenarios 1A1 (asbestos abatement), 1A3 and 1A4 (lead-based
paint abatement and management in-place, respectively), and 1A5 and 1A6 (fuel-related
contaminants). Data for asbestos abatement costs per building square foot were derived from
Azen et al. (1992) and Whitestone Research (2009), with costs converted to 2014 dollars through
the use of consumer product indices for the base years when research was conducted and the
current year. Data from RS Means (RS Means 2014) supported costs estimates for all variables
in this section.

5.4.1.2 Mitigation Strategies Cost-Benefits
The second element of the model is focused on determining the costs and benefits of mitigation
strategies for each risk scenario. Potential mitigation strategies were identified through two
sources: (1) the refined risk framework presented in Chapter 4 and (2) risk mitigation review
conducted by an ESCO industry professional. Mitigation can typically follow one of two
management schemes; risks can be retained and managed or allocated to another party via
contractual or technical means (Ahmed et al. 2007; Perera et al. 2009; Banaitiene and Banaitis
2012). For the purposes of this model, only retained risks are subject to mitigation activities, and
mitigation is defined as an activity that enables potential risks to be detected and managed before
the energy performance guarantee contract is signed, after which time the ESCO has limited
opportunities for cost recovery due to unknown or unmitigated risks.

269

Mitigation strategies were organized by risk scenario map, and the model provided the
opportunity to select appropriate strategies for each of the 12 groups of risk scenarios. Two
measures are required of each mitigation strategy in order to effectively model them: (1) the
efficacy of the strategy, and (2) the cost of the strategy. Since consensus was not achieved for
these measures from an expert panel, the model required manual entry of these values by the
analyst, except for those for which cost-estimating relationships were derived from industry
standard databases (e.g., asbestos sampling costs).

Efficacy is the assessment of how effective the measure is at mitigating the original source of
risk. For example, conducting an asbestos survey in a building may not adequately characterize
100% of the asbestos-containing materials, thereby leaving some residual amount of unmitigated
risk; efficacy is, therefore, measured as the percent effectiveness of a given measure. There may
be a relationship between efficacy and cost, where increased intensity of the mitigation strategy
may result in greater efficacy at a higher cost. This cost-benefit relationship is the central
component of interest for this part of the model. An important consideration is the ability to
make decisions about risk control and mitigation strategies at given levels of cost.

5.4.1.3 Expected Life Cycle Value
The third element of the model is the calculation of the net effects of project risks and mitigation
strategies using the ELCV metric. Values of unmitigated value at risk, mitigated value at risk,
and ELCV are calculated and presented in terms of EC and percentage of the ESCO’s margin.
To accomplish this, values of expected cost for each scenario are calculated from the earlier
model element focused on risk probabilities and costs, mitigation costs and efficacies are carried

270

forward from the previously-discussed model element. The remaining risk is then calculated for
each scenario, as the portion of the EC remaining after mitigation, owing to the efficacy of each
strategy. This results in the calculation of a cost-benefit relationship for each scenario, where
values below one represent mitigation strategies that cost more to implement than the value they
mitigate. This element of the model also examines the value of ELCV-related cost parameters
throughout the project life cycle, by assigning to each scenario the project phase where end
effects are realized.

5.4.1.4 Decision Solutions
The final element of the model is the use of ELCV-related data in providing advice about the risk
management process. The model highlights instances of negative cost-benefit ratios and negative
values of ELCV, which indicate that elected mitigation strategies cost more to implement than
the value of the mitigated risk. The decision solutions element, therefore, supports the ability of
the decision-maker to find optimal combinations of risk mitigation strategies to maximize the
value of the ELCV. To support this process, the model also provides a graphical representation
of the unmitigated value at risk, the mitigated value at risk, the ELCV, and the average costbenefit value across each project phase.

5.4.2 Framework Database
The database providing facility-, probability-, and cost-related data utilizes data sources as
described in the EPC retrofit process model. This is delivered through four primary subdatabases: (1) IGA data, (2) cost data, (3) climate data, and (4) equipment replacement and

271

maintenance life/cost. Relevant IGA data is provided directly by the model user through the
interface, described below. Relevant information from the IGA includes the following:


Facility and project-specific information
o Size and age of buildings undergoing retrofit activities;
o The presence of fuel tanks and piping;
o Design values for HDD and CDD; and
o Utility rates and escalation factors.



Energy guarantee-related information
o Contract length;
o Total and annual guaranteed energy savings; and
o Total predicted savings.



An ECM-by-ECM analysis of the following
o Whether the ECM is difficult to demonstrate;
o Design values for ECM service life;
o Replacement cost; and
o Design energy savings.

As described earlier in this section, cost data was derived from industry standard construction
cost estimating databases, such as RS Means (RS Means 2014), and research that has been
directed to defining asbestos remediation costs per square foot of building area (Azen et al. 1992;
Whitestone Research 2009). Climate data consisted of annual normal HDD and CDD values as
well as 62 years of CDD and HDD data. The latter dataset was used to generate the PDF by first
identifying the data’s distribution and using a random number generator seeded with the

272

parameters of that distribution. Equipment service life data was obtained from the ASHRAE
Service Life and Maintenance Cost Database. Probabilistic values for ECM service life costs
were generated from that dataset, using cost data from the IGA, following the method as
described for probabilistic HDD/CDD values. These values were then used to calculate
probabilistic energy savings, using the results of the multivariate regression model developed by
Griffin (2008), which found that for every one unit increase in HDD energy consumption per
square foot increases by 0.032 KBTUs (or 0.00032 therms). A 1:0.007 relationship was observed
for CDD.

5.4.3 User Interface
The user interface has the primary purpose of enabling the user, defined here as the decision
maker seeking advice on risk management actions, to interact with the model. The interface
enables two-way communication, whereby the user can provide information that is translated
into the model, and where the model can provide information back to the user in a format that is
understandable, in order to facilitate support of decision making.

5.5 CHAPTER SUMMARY
This chapter discussed development of the risk analysis and evaluation framework for ESCO
risks when undertaking EPC retrofits, focused on correctional facilities. Using potential risk root
cause and mitigation strategy information elicited from the Delphi panel, the framework used
SFMEA to analyze and assess risks. A four member expert panel was convened to conduct a
SFMEA analysis of 28 causes, which resulted in the analysis of 23 causes via the development of
77 different risk scenarios organized into 12 scenario maps.

273

The SFMEA panel analyzed risks using two measures of criticality – RPN and EC. This
provided the opportunity for the research to empirically test the differences between the two
methods and evaluate the findings of Kmenta and Ishii (2004). While relationships were detected
between individual contributory parameters of both RPN and EC, the measures of criticality
generally did not exhibit a one-to-one relationship with one another. The use of EC also provides
a cardinal measure of risk criticality, enabling decision making on the basis of the difference in
values. This was found to be a challenge with the use of the RPN score, where higher numbers
do not necessarily indicate greater amounts of risk. Several scenarios were shown where EC
varied greatly at the same value of RPN and vice versa, as well as the situation where RPN and
EC scores yielded mixed priorities for decision making. At given median values of the RPN, the
EC varied by an average of 68%, and in some instances was as high as 200%. As a result of this
analysis, EC was demonstrated to be the more effective measure to support decision making.

The parameter values obtained via the SFMEA panel along with external sources of information,
project-, facility-, and external environment-related factors, and decision maker heuristics were
captured and represented via a life cycle cost-based risk model framework. This framework
consists of a user interface, a database of external quantitative information, and the model which
utilizes expert panel data, quantitative information, and expert knowledge to represent risk
scenarios and associated mitigation costs over the project life cycle. A pilot-scale application of
this model is conducted in the next chapter.

274

CHAPTER 6
PILOT APPLICATION OF LIFE CYCLE COST-BASED RISK MODEL

6.1 INTRODUCTION
Chapter 5 described the risk analysis and evaluation method used to complete the risk
management process for energy service company (ESCO) risks when undertaking energy
performance contracting (EPC) retrofit projects, with a focus on correctional facilities. Scenariobased failure mode and effects analysis (SFMEA) was selected as the analytical method for its
ability to incorporate expert knowledge and determine risk criticality through a consensus-based
process. The SFMEA panel identified and prioritized potential risk root causes from the refined
risk framework (Chapter 4) and developed 77 risk scenarios that were analyzed using risk
priority number (RPN) and expected cost (EC). The relationship between RPN and EC was
analyzed following research conducted by Kmenta and Ishii (2004), and determined that EC is a
preferred measure of risk criticality given its use of measures that support improved decision
making.

The risk analysis and evaluation framework was then developed into a model through the
inclusion of quantitative information from a database and the use of a user interface. The model
enabled users to evaluate risk mitigation decisions based on facility and project characteristics,
qualitative and quantitative data as described in the EPC retrofit process model (Chapter 4) and
heuristics used by the SFMEA panelists and the user in approaching project level risks.

275

This chapter focuses on the pilot-scale implementation of the life cycle cost-based risk model for
EPC retrofits in correctional facilities, to determine its applicability and usability to the types of
risks faced in such projects and the suitability of such a model for effective decision making.

6.1.1 Glossary
Frequently used terms in this chapter are defined as below:


EC: expected cost; a measure of risk that assesses the probability of the occurrence and
missed detection of a fault, in terms of cost over the life cycle of the fault condition (Gilchrist
1993); calculated as EC= Pcause X Peffect X Cost;



ECPV: expected cost based on present value;



ELCV: expected life cycle value; the amount of risk (expressed as expected cost) effectively
mitigated at a given cost, over the project life cycle;



ELCVPV: expected life cycle value based on present value; the amount of future risk
discounted to the present that is effectively mitigated at a given cost, over the project life
cycle;



Risk Scenario: A cause-effect chain describing a root cause, end effect, and potentially
intermediate effects which lead to the calculation of an EC value;



Risk Scenario Map: Multiple related scenarios organized together in a map to describe a
particular system or function of interest;



Scenario-Based FMEA: SFMEA; FMEA method that analyzes risk by constructing causeeffect chains to describe failures rather than defining a single cause-single failure mode
relationship; and

276



Single Present Value: SPV; calculates the present value of a future cash amount, for one-time
amounts; SPV factor is calculated as SPV =

1
(1 d)t

where d is the discount rate and t is the

year at which SPV is calculated.

6.1.2 Chapter Objectives
The model framework developed in Chapter 5 was concerned with the simultaneous use of
several information sources to support decision making regarding risk management strategies
over the project life cycle. Information sources included knowledge elicited from the SFMEA
panel, direct user inputs, and quantitative information from external sources related to costs,
climate, equipment replacement and maintenance, and investment grade audit- (IGA) related
data. The goal of this chapter is to demonstrate the applicability and use of the risk model by
ESCO professionals. The following objectives were established in order to accomplish this goal:


Demonstrate the functioning of the life cycle cost-based risk model and test on a
correctional facility; and



Verify the function of the model based on the application to the case study and finalize
the system, as needed.

6.2 DEMONSTRATION OF LIFE CYCLE COST-BASED RISK MODEL
As shown in Chapter 5, quantitative information and expert knowledge could be represented in a
model framework to reflect the outcomes of decisions made regarding the management of EPC
retrofit project risks. The core function of the model utilized risk relationships and data identified
and developed from the following sources:

277



Elicited expertise and constructed knowledge from the Delphi panel of 19 ESCO experts
discussed in Chapter 4:
o Risk identification, potential root cause identification, and a partial list of controls
and mitigation strategies; and
o Information sources and their use to support decision making under conditions of
risk at each project phase.



Elicited expertise from the SFMEA panel, described in Chapter 5:
o Seventy-seven risk scenarios organized into 12 risk scenario maps;
o Probabilities of cause, conditional end effect probabilities, and relative costs for
each scenario; and
o Facility-, project-, and external environmental-related factors that influence risks.



Quantitative data related to construction costs estimates, probabilities of cause given
factors listed above, climate data, equipment service life, and IGA-related information.

The next sections describe how the user interface was developed as well as supporting structures
for variable development in the model system.

6.2.1 Development of User Interface
The purpose of the user interface is to allow the decision maker seeking risk management advice
the ability to interact with the model in an organized manner that enhances system
understanding. The user interface also provides communication to the user via risk management
advice derived from the results of the model. This section explains each element of the user

278

interface, its critical functions, how decision making is represented, and how connections to
quantitative information are made.

6.2.1.1 User Interface Layout
The interface consists of five input screens and one combination input/output screen. The model
was built in Microsoft Excel; each screen consists of a separate workbook. The screens consist
of: (1) starting screen, (2) user preferences screen, (3) review scenarios screen, (4) risk data entry
screen, (5) mitigation screen, and (6) an expected life cycle value (ELCV) results & decisions
screen. These screens are selected using the Excel workbook tabs following a sequential
numbering format for ease of use (Figure 6-1).

Figure 6-1. User Interface Screen Selection
6.2.1.2 Start Screen
The start screen allows the user to provide information about the retrofit project in four
categories: (1) job data, (2) facility and project data, (3) energy guarantee information, and (4)
energy conservation measures.

The job data box is used to capture information about the project team conducting the retrofit
and is generally not used in model calculations. One exception is the data entry option for the
firm’s level of previous corrections EPC experience. This was a significant issue raised
numerous times by Delphi panelists and SFMEA participants; as a result, this information is
captured here and values are assigned based on their experiences. Those values become the
probability of cause for scenarios 1D1-1D11. The job data input screen is depicted in Figure 6-2.
279

Figure 6-2. Job Data Input Box
The facility and project data box allows the user to provide information about facility, project,
and external factors which were initially developed by SFMEA panelists. As discussed in the
previous chapter, these factors potentially have a significant impact on the calculation of a
number of probabilities of risk cause and costs. A screen shot of the interface is provided in
Figure 6-3.

Notable among these inputs is the ability to distinguish between manual data input requirements
(solid green cells) and automatically-calculated values (diagonally-hatched cells). Instructions
are highlighted in yellow, and non-applicable information based on previous inputs is shaded in
black. These help ensure that users are entering data in the appropriate locations on the
worksheet and also provides interactive guidance if error conditions are detected.

280

Figure 6-3. Facility and Project-Specific Data Input Box
The lower portion of the screen provides mean probabilistic values for annual heating degree
days (HDD) and cooling degree days (CDD). These values are called from a supporting
worksheet which utilizes a random number generated from a probability function derived from
over 60 years of HDD data. One probabilistic HDD value is recorded for each year of the
contract, as provided in the job data input screen. Probabilistic facility-wide energy consumption
estimates are then calculated based on changes between the normal annual or modeled HDD and
the probabilistic annual HDD value for each contract year. The calculation of this impact relies
on the linear relationship observed by Griffin (2008) between HDD and energy consumption per
square foot of building area. Consumption was converted from thousand British Thermal Units
281

(KBTUs) to therms (natural gas consumption) and kilowatt hours (kWh) for electricity
consumption. Probabilistic consumption values are then multiplied by the total facility square
footage subject to retrofit activities, obtained previously on this screen, and the respective utility
unit rates which were obtained directly from the IGA.

Utility rate escalation factors provided by the FEMP (Rushing et al. 2013) are applied to each
year’s resultant total energy cost increase or decrease to account for future conditions in energy
markets. These factors are derived by fuel type (i.e., natural gas and electricity), by end-use (e.g.,
industrial vs. residential consumers), and by U.S. Census Region. Finally, each year’s
probabilistic HDD value is compared to the probability density function (PDF) for HDD; only
values equal to or exceeding the 95% confidence level are passed to the cost field of the
appropriate risk scenario (2G4). This was done to reflect SFMEA panelist experience with
reduced energy performance that led to contractual disputes during severe climate conditions that
deviated significantly from normal annual conditions.

The energy guarantee box captures data about baseline energy consumption, the contract
length, total savings and annual guaranteed savings, as well as total energy savings and the
ESCO’s margin, also called the contractor’s cash flow. This value automatically calculates and is
important as it is used to parameterize relative costs in scenarios that do not use fixed or
probabilistic costs when calculating the EC.

The energy conservation measures (ECMs) box of the start screen provides summary
information about ECMs selected for the retrofit, as shown in Figure 6-4. The ECM description

282

Figure 6-4. Energy Conservation Measures Input Box
mirrors data from the IGA, and the primary equipment field utilizes a drop-down menu from
which the main type of equipment used for each ECM is selected. Each ECM may have multiple
entries, as is seen with the ECM number 3, if that ECM utilizes more than one type of
mechanical or other equipment. In that case, the original ECM numbers are supplemented by a
lower case letter as an additional identifier, such as 3a. With ECM 3, the retrofit measures were
separated into gas-fired condensing boilers and gas-fired rooftop units with direct expansion
(DX) cooling.

An assessment is made by the user as to whether the ECM is difficult to demonstrate, which
helps construct the probability of cause for risk scenarios 2CD1 through 2CD3. The design
service life is then entered by the user, and a probabilistic service life is calculated by the model,
utilizing the user input and ECM-by-ECM service life data provided by ASHRAE (2014). A
random number is generated from the constructed probability distribution function, which results
in the value entered in the “Prob” cell for service life. If that value is less than the design value,
further assessment is made as to whether the service life is less than the contract length, as
entered in the energy guarantee information data screen and as shown by ECM 1 in the figure. A
283

separate random number is also generated to assess how many pieces of equipment may be
affected by this shorter service life. For example, there are five domestic hot water heaters
installed in ECM 1; the random number generator assigned three of these units as having a
reduced service life of eight years. Since eight years is less than the length of the contract term
(10 years in this case), the manually-entered replacement cost (derived from the IGA) in current
dollars is discounted using the SPV, with the time variable entered as the early failure year; in
the example case, the SPV was based on a time of eight years. The sum of the discounted
replacement values for all mechanical ECMs is used to generate cost parameters for risk
scenarios 2G7 and 2G8.

This screen includes entries for non-mechanical system ECMs; however, due to data limitations
for the ASHRAE database (ASHRAE 2014), not all mechanical equipment can be listed and
non-mechanical equipment (e.g., lighting, water conservation devices) is not evaluated.

6.2.1.3 User Preferences Screen
The user preferences screen permits the user to select their desired method for risk scoring.
Options are to accept the expert panel data, derived from the SFMEA panel, or to manually enter
values for each scenario. A link is provided to take the user directly to the risk data entry screen
in order to facilitate a review of the panel data before deciding which method they wish to use.
At the time of model development, a third option was explored – model-derived probability and
relative cost values based on the user’s background. This was to be based on the characteristics
of the SFMEA panelists with regard to risk tolerance, professional tenure in ESCO projects, job
level within the organization, and the project phases that the individual supports. Due to the

284

small sample size of the SFMEA panel, establishing such quantitative relationships was
impossible; further work that can be conducted in this regard is discussed in Chapter 7.

6.2.1.4 Review Scenarios Screen
The “Review Scenario” screen is required in order to review each risk scenario, both to ensure
understanding of the relationship among various causes, intermediate effects, and end effects, as
well as to decide which of the 77 scenarios will be included in the model. A comments field is
provided to explain user decisions for selection or non-selection of scenarios. To facilitate ease
of use and understanding, selection fields are color-coded to alert users to their selection;
selected risks appear in green, non-selected risks are red, and risks with no data entered with
regard to selection are in yellow.

6.2.1.5 Risk Data Entry Screen
As stated earlier, the risk data entry screen is used to review probability and cost values for risk
scenarios during the user preferences selection process. This screen also reflects the data that has
been either brought forward from the expert SFMEA panel, or empty green cells appear
indicating that the user has selected to manually enter data, which is also accomplished on this
screen. Users also have the ability to enter a value for the year when each risk scenario is
realized, which is used in the calculation of the ECPV. Those risks not selected for further
analysis are highlighted to allow the user to focus on just their selected risks.

285

6.2.1.6 Mitigation Screen
For the purposes of the model, risk mitigation is defined as activity that enables risk to be
controlled before the energy savings guarantee is signed. It is generally accepted that the ESCO
has limited ability to initiate change orders if they detect unmitigated risk conditions after that
point. The mitigation screen provides a list of mitigation strategies associated with each scenario
map. Users may accept these strategies for the project, or overwrite the “other” field with their
own preferred strategy. An efficacy value must also be provided for each strategy. Efficacy is
given as the amount of risk, expressed as a percentage that is effectively mitigated through each
given strategy. Users must then enter a cost value for each mitigation strategy. The cost may be
entered as a lump sum, or fields are available to enter labor cost, labor hours, material cost, and
equipment cost associated with each mitigation measure. With the exception of strategies
calculated using quantitative information provided elsewhere in the model, values must be
manually entered for each strategy.

6.2.1.7 ELCV Results & Decisions Screen
This screen provides a hybrid of user inputs and outputs to assist the user in making effective
decisions (Figure 6-5). Sections 6A through 6C are based on the ten most critical risks, as ranked
by the value of the present value of the EC (ECPV). Section 6A identifies those risk scenarios by

286

Figure 6-5. Data Inputs on the ELCV Results & Decisions Screen

287

number along with their attendant ECPV values. Section 6B provides the user with dropdown
lists containing the risk mitigation strategies developed in the previous screen. In addition to the
efficacy and mitigation costs carried forward from the mitigation screen, the model calculates the
remaining risk and the cost-benefit relationship for each scenario. Remaining risk is calculated
as:
(1-Efficiacymitigation) X ECPV
Cost-benefit is calculated as:
(EC-Riskremaining)/Costmitigation

The cost-benefit relationship provides the user with a rapid screening tool for assessing the
financial performance of each selected mitigation strategy. Cost-benefit values below 1 indicate
a mitigation measure that costs more to implement than it saves. The final value calculated by
the model is the ELCVPV, which was described in Chapter 5 as the ELCV discounted to the
present. The ELCVPV is effectively the value of the mitigated risk, once mitigation costs and
remaining risks have been subtracted. Like the cost- benefit ratio, optimal solutions of the model
maximize the value of ELCVPV across the range of risk scenarios examined. A negative
ELCVPV value indicates the same thing as a cost-benefit value below one. The model provides
color coding to help the user quickly focus on the risk factors with the best and worst outcomes.
Negative ELCVPVs are highlighted in red; green highlighting is used for ELCVPVs that
represent effective mitigation of 75% or more of the EC for a given risk.

288

The rest of the ELCVPVs are shaded yellow, indicating that further mitigation may be required
for those risks.

Similar shading is used for the three summary values that connect the input portions of the screen
with the output portions. The unmitigated value at risk is the sum of the ECPV values for the ten
most critical risks and reflects the percentage of the ESCO margin that is potentially at risk if
those risk scenarios remain unmitigated. The mitigated value at risk is the portion of the original
value that remains at risk after mitigation has been conducted. This includes the cost of
mitigation which is borne by the ESCO, as well as the remaining risk. The ELCVPV can then be
expressed as the difference between the unmitigated and mitigated values at risk, and reflects the
financial value of successful risk mitigation activities. Unmitigated Value at Risk calculations
that are greater than 40% of the ESCO’s margin are flagged in red; those that are less than 20%
are shaded green, and yellow cells denote cases that fall in between those values. The same
shading is used for unmitigated value at risk calculations. The ELCVPV utilizes the reverse of
those color scales – values greater than 40% of the margin are shaded green and lower values are
in yellow and red boxes. This makes sense as the goal of the model is to maximize values of
ELCVPV while minimizing mitigation costs.

Section 6D presents the value at risk, ELCV, and cost-benefit data in a more efficient format to
aid decision making. First, each risk scenario’s project life cycle assignment, as described in
Chapter 5 and as noted on the risk data entry screen, is used to aggregate these values at each

289

point in the project life cycle where they are realized (Figure 6-6). This provides the ability to
determine the future value of risks avoided at the point in the project where the model is used.

Figure 6-6. Project Life Cycle Value at Risk Plot
Project cash flow can also be overlaid on this time series plot to determine the appropriate
project timing for making risk management investments, particularly those at lower cost-benefit
values.

The model also facilitates the examination of value at risk for each of the ten most critical risks
(Figure 6-7). Mitigation strategies that contribute the most (2B1) and least (1F6) to the overall
project ELCVPV can quickly be detected. These relationships enable the user to determine which
scenarios could be improved further through the pursuit of different mitigation strategies,
additional effort to improve the efficacy of a given strategy at minimal additional cost, or
290

Figure 6-7. Most Critical Risks Value at Risk Plot
through the addition of multiple strategies to a single risk scenario if the additive effect would
further enhance ELCV.

6.2.2 Supporting Structures for Variable Development
In addition to the user interface screens, the model contains several screens that support the
development of variables. While the various user interface screens support the development of
model variables through direct input and calculated values based on user inputs (e.g., the job
data, facility and project-specific data, energy guarantee information, and energy conservation
measures screens), several call functions and instances of IF-THEN-ELSE logic are used to
further support enhanced variable development.

291

Five hidden screens (risk data, references, mitigation, probability data, and costs) contain
readable and writable model data; however, to minimize errors, these screens are not available
for end-user access. The risk data screen includes the values of Pcause, Peffect, and cost for the 77
scenarios. The default condition for these cells is storage of the SFMEA panelist consensus
scores for each variable; however, values are automatically written for risk cause probabilities in
the 1A scenario map, estimated costs for scenarios 1A1 through 1A6, and risk cause probabilities
for scenarios 2G3-2G4 and 2G7-2G8 (Table 6-1).
Table 6-1. Variables Supported by the Risk Data Screen
Scenario Variable Variable Calculation
1A1
Pcause
Average of square footage per age category * Pcause given by panel
1A1
Cost
Asbestos abatement plan cost + $5.88 SF floor area impacted by ECMs
1A2
P
Average of square footage per age category * P
given by panel
1A2
1A3
1A3
1A4
1A4
1A5
1A5
1A6

cause

Asbestos abatement plan cost + $5.88 SF floor area impacted by ECMs
Average of square footage per age category * Pcause given by panel
SF painted surface disturbed by ECMs * average lead paint abatement
cost/SF

Pcause
Cost

Average of square footage per age category * Pcause given by panel
SF painted surface disturbed by ECMs * average lead paint encapsulation
cost/SF

Pcause
Cost

IF tanks/piping will be disturbed THEN Pcause = 0.85 ELSE Pcause = 0.0
IF tanks/piping will be disturbed THEN number of tanks of each size *
cleanup cost/tank, IF tanks are suspected leaking THEN number
suspected leaking of each size * leaking cleanup cost/tank ELSE 0
IF tanks/piping will be disturbed THEN Pcause = 0.85 ELSE Pcause = 0.0
IF tanks/piping will be disturbed THEN number of tanks of each size *
cleanup cost/tank, IF tanks are suspected leaking THEN number
suspected leaking of each size * leaking cleanup cost/tank ELSE 0
IF ECM data reveals an average number that are “difficult to
demonstrate” >0 THEN Pcause =average number of such ECMs ELSE 0

1A6

Pcause
Cost

2CD1-3

Pcause

2G3

Pcause

2G4
2G4

cause

Cost
Pcause
Cost

Pcause
Cost

IF HDDprob>0 THEN 1 ELSE 0
∑ IF HDDprob>=HDD0.95 THEN Total Energy Costprob X SPV factor
ELSE use SFMEA panel value of relative cost

292

Table 6-1 (cont’d)
Scenario Variable
2G7
Pcause
2G8
P

Variable Calculation
IF Service Lifeprob < contract length THEN ∑ECMs where Service

2G7
2G8

IF Service Lifeprob < contract length THEN Cost = ∑Costreplacement X
SPV factor ELSE Cost = 0

cause

Cost
Cost

Lifeprob/number of ECMs ELSE use SFMEA panel value of Pcause

The references screen organizes the variable values that are presented to the user via dropdown
boxes. These, in turn, serve as inputs elsewhere in the model. For example, the dropdown list for
the experience with corrections retrofits variable is stored here, and connects descriptions of
variable levels (e.g., “moderate experience – 3-5 prior projects”) to their associated input values
for Pcause (in this case, Pcause = 0.60).

The mitigation references screen serves a similar function as described for the references screen,
in this case strictly supporting mitigation strategy-related variables.

The probability data screen contains the functions for all variables that are based wholly or
partially on selecting a value from a probability density function. These include the base case and
probabilistic case for calculation of scenarios involving HDD, CDD, and mechanical equipment
service life. Distributions were selected that best fit the original data, and values from the
resultant probability functions were selected from among these distributions.

The costs screen supports the development of scenario costs derived through construction
estimation. This was applied to scenarios 1A1 and 1A2 (asbestos-related risks), 1A3 and 1A4
(lead-based paint-related risks), and 1A5 and 1A6 (fuel contaminant-related risks). In general
these data were derived from RS Means construction cost estimating databases (RS Means 2014)
293

and were used to calculate risk scenario costs directly through the use of facility factors
described by SFMEA panelists. For example, as shown in Table 6-1, the calculation of leadbased paint costs utilizes the following relationship:
Average of square footage per age category x Pcause, as given by the panel
This utilized average square-foot costs for lead-based paint abatement in RS Means (RS Means
2014) to calculate the scenario cost.

Asbestos abatement costs were calculated somewhat differently. RS Means (RS Means 2014)
contains significant amounts of data about asbestos abatement costs; however, there is a great
deal of variability among unit costs due to several factors, to include the physical condition of
the asbestos-containing material, the type of asbestos-containing material (e.g., pipe wrap, sprayon fireproofing, ceiling tiles), and the intensity level of the abatement activity. Research was
sought that could provide a relationship between building characteristics, presence of asbestos,
and unitary abatement costs. Research conducted by Azen et al. (1992) and Whitestone Research
(2009) found similar square foot costs for asbestos abatement in public buildings, when adjusting
for inflation between the research dates. This unit cost could then be applied to the estimate of an
equivalent floor area square footage impacted by retrofit activities, provided via the user
interface, to estimate the quantity of asbestos present and calculate a cost for relevant scenarios.

6.3 TEST CASE CHARACTERISTICS
The final part of demonstrating the function of the model was to test its use on a case study
location. This section discusses the criteria for selection of the case study location, the data

294

collected from the case study, as well as the model results when applied to risk management of
the EPC retrofit process.

6.3.1 Case Study Selection
Purposive sampling was used to select the case study location. As opposed to probabilistic
sampling, where the goal is to sample individuals from a population such that findings can be
generalized to the population as a whole, purposive sampling seeks to focus on specific
characteristics of interest within that population (Oliver 2006; Teddlie and Yu 2007; Tongco
2007). As was described in chapters 3 and 4, the Delphi panel was constructed via the same
strategy, where the focus was on finding knowledgeable experts with specialized knowledge of
the research subject, willingness to participate in the research, and the reliability of experts
(Oliver 2006; Tongco 2007; Duah 2014).

As a result, the following characteristics were used when selecting the case study location:


Capacity and willingness of ESCO and facility agency staff to participate in the study.
This included the ability and willingness to share the IGA report, final retrofit design,
performance guarantee documents, and contracts;



Highly experienced key informants (ESCO and facility owner staff) with regard to
correctional facility EPC retrofits; and



Locations were sought that recently (i.e., within the past five years) entered the energy
savings phase of an EPC project, such that data would be readily available and key
informants would have recent knowledge of project characteristics.

295

Based on the case study location selection criteria, the steps for testing the life cycle cost-based
risk model included:
1. Identify case study location;
2. Review the IGA report, final retrofit design, financial performance, and contract for case
study;
3. Visit case study location and perform a visual inspection of building conditions and
installed ECMs;
4. Researcher inputs data into risk model using project-, facility-, and external environmentrelated information from the case study:
a. Variables and parameter values obtained from SFMEA panelists; and
b. Probabilistically-generated values through the model;
5. Run risk model to obtain value at risk and ELCV;
6. Provide risk results and recommendations to ESCO and client representatives from the
case study location; and
7. Asses system function and utility; finalize the system based on user inputs and
recommended modification.

The researcher visited the case study location and had extensive interactions with both the ESCO
project representative and the agency representative over an 18 month period. The researcher and
the agency representative published a case study about three recent EPC retrofits in correctional
facilities, and spent several months discussing the case study location and its suitability for this
research. Both individuals were also members of the SFMEA panel, therefore their domain

296

expertise related to risk management was elicited in a general sense, and then again with respect
to the selected case study through telephone calls, face-to-face meetings, and site visits.

6.3.1.1 Case Study Location
The selected case study is a correctional facility located in the lower peninsula of Michigan that
completed the pre-execution and execution phases of an EPC retrofit in 2011. It was selected
because it was representative of a number of similar correctional facility EPC retrofit projects
completed during the same time period. Complete case study location details are provided in
Appendix E. An overview of the selected location is provided in Table 6-2.
Table 6-2. Case Study Location – Selected Characteristics
Project Characteristics
ESCO Experience with Significant experience – 6+ projects
Correctional Facility EPCs:
Facility Characteristics
Building Vintages: 1925-2002
Building Area 905,220
(SF):
Construction Types:  Brick, pre-cast concrete
Energy  Lighting retrofit
Conservation  Water efficiency
block, steel, and glass
Measures:  Decouple steam
 Weatherized pole barns
with sealed concrete
plant and condition
floors and plaster-board
independently
walls
 Controls
Security Level: I – Minimum Sec.
Climate Zone: 5
Energy Performance Contract and Savings Guarantee Information
Baseline Electricity 9,442,290
Project Value: $12,890,098
(kWh):
Baseline Natural Gas 560,016
Guaranteed $1,195,334
(therms):
Savings/Year:
Baseline Steam 1,059,181
Payback Period: 10 yrs
(therms):
Data was collected by the researcher after reviewing the IGA, energy savings guarantee, and
contractual documents, followed by a field visit to the case study location and interactions with
the agency representative. This individual had significant experience implementing EPC projects
297

in a correctional setting (20 years) and was thus considered a knowledgeable informant for this
research. This individual also participated in the SFMEA panel as member RP-03-E – relative
expertise scores are available in Chapter 5.

Data was obtained from agency and ESCO reports, as well as the knowledgeable informant and a
member of the ESCO staff who developed the EPC retrofit at the case study location (Table 6-3).
Table 6-3. Data Obtained for Case Study Location
Information Required by Model
Information Obtained
Project Information – Job Data
Job Name: Parnall Correctional Facility
Firm's Prior Corrections EPC Experience: Significant Experience - 6+ Prior
Projects
Project Information – Facility and Project-Specific Data
a,b
Building Area Built/Renovated Before 1980: 602,600 SF
a

Building Area Built/Renovated 1980-1985: 0 SF
a,b
Building Area Built/Renovated 1985-2000: 14,260 SF
Building Area Built/Renovated After 2000: 23,939 SFa,b
% SF Impacted by ECMs Built/Renovated Before 1980: ~35%
% SF Impacted by ECMs Built/Renovated 1980-1985: ~10%
% SF Impacted by ECMs Built/Renovated 1985-2000: ~100%
% SF Impacted by ECMs Built/Renovated After 2000: ~100%
Will retrofit activities disturb fuel tanks and piping? No
Area of Painted Surfaces to be Disturbed: ~250 SF
Percent Client Uses DBB Procurement: 85%
Climate Zone: 5
Nearest City in Model DB: Lansing, MI
Normal Annual HDD 6909
Natural Gas Unit Rate (CCF): $0.7782
Electricity Unit Rate (kWh): $.07850
Project Information – Energy Guarantee Information
Baseline Annual Electricity Consumption (kWh): 9,442,290
Baseline Annual Natural Gas Consumption (therms): 560,016
Contract Length (Years): 10
Total Guaranteed Savings ($): $12,890,098.00
Annual Guaranteed Savings ($): $1,195,334.00
Total Predicted Savings ($): $15,646,197.00
Financing Costs: $2,161,000.00
Hedged Savings ($)/Cash Flow: $545,931.00
298

Table 6-3 (cont’d)
Information Required by Model
Information Obtained
Energy Conservation Measures
ECM #1: Renegotiate Incinerator/Steam Contract
ECM #2: New HVAC for CB9, CB10 - Hydronic Radiant Panel Heating System and RoofMounted Air Rotation Units
ECM #3: Decentralized Building Utilities - Gas Fired Unit Heater; Gas Fired Infrared Heater;
Gas Fired Condensing Boiler; Domestic Hot Water; Gas Fired Rooftop Unit with DX
Cooling
ECM #4: Campus Lighting Retrofit - Interiors
ECM #5: Renegotiate Incinerator/Steam Contract
ECM #6: Campus Water Conservation
ECM #7: Decommission Former Textiles Building
ECM #8: Minimize Heat in Bldg 31 and Minimize Heat in Cell Block 8 - Hydronic Unit Heater
ECM #9: Replace Dishwasher
Notes:
a\ Risk scenarios addressing the presence of asbestos containing materials were not used in the
pilot test; it is the State of Michigan’s policy to separately address and pay for the abatement of
any asbestos discovered during the construction process and not otherwise burden the contractor.
b\ Approximately 250,000 square feet of building area was subjected to retrofit activities.
Additional details about installed ECMs were obtained by reviewing the project schedule of
values. Based on data availability and the model limitations of the model, several assumptions
were made when compiling this data:


For the purposes of the life cycle cost-based risk model, only the primary system
components were of interest (e.g., rooftop units, boilers), as opposed to associated piping,
electrical, and minor controls elements. Cost data provided in the IGA and schedule of
values was rather coarse with regard to this level of detail. As a result, estimates of
replacement cost may be higher than actual costs, as they can be expected to include
system accessories. Estimated unitary costs were used, whenever possible, by isolating
project overhead and profit from specific item costs.



In some cases (e.g., ECM #8), the schedule of values did not isolate the cost of individual
mechanical systems. As a result, costs were apportioned to individual systems in each
299

ECM based on estimates of the type of equipment being installed, and through an
analysis of unitary costs provided in the IGA alongside the schedule of values.


Non-mechanical ECMs (e.g., lighting, dishwasher replacement, water conservation
measures) were excluded from the analysis. Probabilistic data regarding installed service
life was only available for mechanical systems, as collected by ASHRAE (ASHRAE
2014). Even with a large, nationwide data set, some equipment types relevant to this
research were missing from the ASHRAE database.



Only ECMs used to achieve the energy savings guarantee were included; therefore, ECM
#1 (incinerator steam cost savings) was not included in the analysis.

6.3.1.2 Probabilistic Scenarios for Test Case
Once the data described above was input through the user interface, probabilistic variable values
were developed, as described in Table 6-4. These values were related to the random selection of
annualized HDD from a probability distribution of over 60 years of such data from the project
location and equipment service life data obtained from ASHRAE (2014).

6.3.2 Risk Analysis and Evaluation Results
The pilot model was run with 75 of the 77 identified risk scenarios. Scenarios 1A1 and 1A2
(asbestos abatement and management in-place, respectively) were not included due to the State
of Michigan separately funding asbestos-related mitigation on such projects.

Based on the deterministic and probabilistic model inputs, including selection of the expert panel
inputs for risk probabilities and costs (except where overridden formulaically, as discussed

300

previously in this chapter), the following ten risk scenarios were assessed as being most critical
(Table 6-5). A $138,800 difference was observed between the top-most ranked risk and the
bottom-most risk.
Table 6-4. Probabilistic Variable Values
Probabilistic
Prob. Reference
Variable
Value
Value
Result on Model Parameters
Did not increase probabilistic facility-wide
annual energy costs because no single year
Mean
HDDprob exceeded the 95% level of the PDF.
HDD
HDDprob

of 7730

6909

As a result the ECPV2G4=$0 since Pcause=0.

Service Lifeprob1

N/A

20

Service Lifeprob2a

N/A

30

No change – prob service life >= 10 years
No change – prob service life not calculated –
no similar equipment type found in ASHRAE
2014

Service Lifeprob2b

10

30

Service Lifeprob3a-3b

N/A

30

No change – prob service life >= 10 years
No change – prob service life not calculated –
no similar equipment type found in ASHRAE
2014

Service Lifeprob3c

10

30

No change – prob service life >= 10 years

Service Lifeprob3d

18

30

No change – prob service life >= 10 years

Service Lifeprob3e

26

30

No change – prob service life >= 10 years

Service Lifeprob4

N/A

15

No change – prob service life not calculated

Service Lifeprob5

N/A

15

No change – prob service life not calculated

Service Lifeprob6

N/A

30

No change – prob service life not calculated

Service Lifeprob7

N/A

30

No change – prob service life not calculated

Service Lifeprob8a

N/A

30

No change – prob service life not calculated

Service Lifeprob8b

18

30

No change – prob service life > 10 years

Service Lifeprob9

N/A

20

No change – prob service life not calculated
No change – since no values of the
probabilistic service life were shorter than the
contract term, these costs are not calculated

Replacement Costprob1-9 N/A

Varies

301

Table 6-5. Model Output – Ten Most Critical Risks
Rank Risk Scenario Expected Cost
Rank Risk Scenario Expected Cost
1
2B1
$221,890.99
6
2B3
$112,545.12
2
2F3
$171,293.78
7
2F1
$110,512.12
3
2B2
$150,060.16
8
2F2
$104,986.51
4
1F8
$133,867.15
9
1F6
$83,089.95
5
1F7
$126,430.08
10
1B2
$83,089.95
Notes:
a\ Shaded cells indicate risk scenarios that originated from the same risk scenario map
Figures 6-8 and 6-9 depict the results screen obtained through the pilot model run. Some
observations of interest for decision makers immediately become clear from these results. First,
the ten most critical risks were responsible for a potential erosion of 218.1% of the ESCO margin
if left unmitigated. Seventy percent of the initial unmitigated risk was recovered over the project
life cycle through the implementation of risk mitigation strategies, resulting in an ELCVPV of
$912,383, or 153.3% of the ESCO margin. It is important to note that while 64.8% of the ESCO
margin remains at risk, the improvement described above was after mitigating just the top 10 out
of 75 possible risk scenarios in the model. Further mitigation efforts will improve the values of
Mitigated Value at Risk and ELCVPV.

In making recommendations, the first risks to examine are those with cost-benefit ratios of less
than one, indicating that these risks cost more to mitigate than the value that was potentially
recovered and those approaching a value of one, indicating an approximately break-even
relationship. There is no such case present in the modeled results, as the lowest cost-benefit
value among mitigation strategies is that for risk scenario 2F1 with a value of 2.05. This risk
scenario is based on the root cause of a client preparing the EPC scope of work based on terms

302

Figure 6-8. Pilot Model Run Results Screen

Figure 6-9. Pilot Model Run Results Screen Plots
303

and conditions found commonly in typical design-bid-build public procurement which requires
overstated soft costs resulting from the use of standard, fixed overhead and profit rates. The
ELCVPV results and decisions screen also provides insight into these relationships graphically,
as shown in Figure 6-9.

The value at risk over the project life cycle plot (left side of Figure 6-9) shows that the ten most
critical project risks are concentrated in the energy audit and retrofit design phases, so early
decisions regarding risk management are critical. The value at risk – 10 most critical risks plot
(right side of Figure 6-9) indicates that to further reduce the project’s mitigated value at risk,
relatively poorer performing mitigation strategies such as 1F8 and 2F1 can be targeted for more
intensive activity, with the intent to increase efficacy or lower cost solutions to risk management
and control.

As discussed in Chapter 4, the most important risks occurred during the earliest project phases;
lower risk importance scores were assigned to risk categories occurring later in the project.
Reasons for this included the inability to address risks contractually once the energy savings
guarantee was signed, thereby requiring risks to be addressed early in the project life cycle.
Because activities related to the IGA and ECM selection and installation have such a large
impact on later project phases, these risk categories were scored as having among the highest risk
importance scores. A review of all 75 risk scenarios included in the model yields expected results
for the level of risk realized at each phase of the project life cycle (Figure 6-10). The upper graph
depicts the project risk profile using the ECPV while the lower graph used the EC. In general,

304

Figure 6-10. Model Risks by Project Phase
project risks increased toward the latter project phases, as expected based on the project
decisions and costs model presented in Chapter 4 (Figure 4-8), underscoring the need to gain
adequate control of project risks during the earliest phases. In the model output using ECPV, a
more pronounced reduction in risk is noted during the energy savings phase. This is somewhat
expected given that this is the longest phase, occupying nearly 8 years in the case study. As a
result of the present value calculation, risks realized later in the project have lower present
values, which are reflected in the Figure 6-10.
305

6.3.3 Sensitivity Analysis
Sensitivity analysis was conducted to determine the impact of inputs on the results of the life
cycle cost-based risk model. Key areas of interest included the sensitivity of the model to
changes in deterministic and probabilistic parameters. The use of probabilistic inputs in the
model made it difficult to assess risk criticality in a single model run because the results changed
with each iteration, so Monte Carlo analysis was used for this step.

ModelRisk by Vose Software (Vose Software 2014) was used to conduct the Monte Carlo
analysis. In order to determine the distribution of possible model outcomes (ELCV), probabilistic
parameters were included as described earlier in this chapter; deterministic parameters were
calculated using the risk modeling function of ModelRisk. For each risk scenario, the minimum,
maximum, and mean panelist values of Pcause, Peffect, and cost were recorded and used as inputs
to a triangular distribution. Since each risk scenario only had four data points, the selection of a
data distribution can be challenging. The main reason for selecting the triangular distribution is
its use of either the mean or mode in its construction. With smaller sample sizes, distributions
that emphasize central tendency can be more reliable than those that emphasize extreme values
(U.S. Environmental Protection Agency 2001). Parameters that relied on probabilistic functions
were populated with the appropriate distribution in lieu of the triangle distribution. The model
was run 10,000 times during the Monte Carlo simulation, which resulted in a normal distribution
of ECPV values over those trials (Figure 6-11). Expected cost distribution values were:


Minimum – $2,330,929



Maximum – $3,976,759



Mean - $3,075,946
306

Figure 6-11. Monte Carlo Simulation Results – Distribution of EC Values
The Monte Carlo results were analyzed to determine which scenarios the model result is most
sensitive to. This was done to compare the pilot application results, which resulted from a single
model run, to the simulated range of possible values. The ten risks for which the model is most
sensitive were determined for the Monte Carlo simulation by examining the contribution of each
risk scenario to uncertainty of the conditional mean of EC at the 99th percentile of the output
distribution. These risks are listed in Table 6-6. The delta value represents the difference between
the lowest and highest values for each scenario that contribute to the output value.
Table 6-6. Ten Risks to Which the Model is Most Sensitive
Risk Scenario Delta
Risk Scenario Delta
2B3
$191,096.03
2B2
$157,455.18
1A7
$189,360.33
1F1
$136,499.67
2B1
$166,794.80
1E4
$131,124.63
2G4
$158,121.56
1F8
$127,283.17
1B7
$158,104.74
2CD1
$119,533.99
307

Sensitivity of the model to the probabilistic inputs was determined first. This was accomplished
by turning off the distribution-based cause, effect, and cost parameters for the deterministic
variables, as described above, and only using panelist-derived values. Probabilistic input values
were allowed to vary within the bounds established by the model. Results of this sensitivity
analysis are provided in Figure 6-12. As can be seen, the deterministic inputs have the greatest
overall contribution to uncertainty of the model output, which may be the result of having just
four probabilistically-derived risk scenarios (2G3, 2G4, 2G7, and 2G8) in the pilot model,
compared to 71 deterministic scenarios..

Figure 6-12. Model Sensitivity to Deterministic and Probabilistic Parameters
Sensitivity of the model to the two probabilistic parameters (HDD and equipment service life)
was also assessed. The distribution of the two input parameters is depicted in Figure 6-13. The
distribution of probabilistic equipment service life-driven values of ECPV is positively-skewed;
equipment service life-related values of ECPV appear to be uniformly distributed across a much
smaller range of values than the HDD-driven values.
308

Figure 6-13. Probabilistic Input Parameter Distributions
Sensitivity of the model outputs to the probabilistic HDD- and equipment service life-related
inputs is shown in Figure 6-14. The model showed more sensitivity to the distribution of HDD-

309

Figure 6-14. Model Sensitivity to Probabilistic Equipment Service Life Parameter Values
related input values. Whether the parameter values for equipment service life are from the high
or low range of the distribution, there is little effect on the model outcome.

6.4 LIFE CYCLE COST-BASED MODEL FUNCTION AND REFINEMENT
The demonstration of the life cycle cost-based risk model received generally positive feedback
from ESCO and agency client representatives. The model was particularly useful in providing a
basis to illustrate the presence of latent risks that do no surface until later stages of projects. This
was found to be helpful toward discovering and treating such risks early in the project process.
Additionally, the ability to utilize panel data or override those parameters with values that reflect
individual preferences toward risk was recognized as a strength of the system.

The agency client representative recognized the value of the probabilistic weather data and its
use in creating increased energy consumption scenarios. Such a scenario was observed over the
winter of 2013-2014, when temperatures were much colder than long term averages. As a result,
310

one of their projects experienced some performance degradation as a result of sub-optimal ECM
performance in extreme climatic conditions. The ability to continually “game” the model to
derive high and low deviations in HDD values was regarded as a strength of the approach.

Both informants found the ability to generate scenarios by selecting different mitigation
strategies and altering the efficacy and costs of such strategies, to be effective decision aids in
evaluating different management approaches to long-term risk control and management. In both
cases, participants indicated a desire to be able to select more than one mitigation strategy per
risk scenario to better represent management approaches that employ multiple controls.
Additionally, they believed that the results screen would be easier to navigate if there was a
visual reference to remind the user of the construction of the risk scenarios under review, as the
alphanumeric coding was difficult to recall.

One participant also indicated that the model would be strengthened through the ability to
optimize retrofit design parameters along with their attendant project-level risks. Further
refinement of this model could include additional research and development activities to enable
this model to exchange information with an energy modeling platform, such as EnergyPlus or
DOE2. Work in that regard would be focused on finding optimal solutions that balance energy
performance, cash flow, and life cycle project risks.

6.5 CHAPTER SUMMARY
The risk analysis and evaluation framework developed in Chapter 5 sought to connect
quantitative data with expertise to effectively create and manage risk scenarios. The framework

311

was designed to help ESCO professionals make better-informed decisions about the long-term
effects of project-level risks over the course of the project life cycle. The framework was
developed into a model through the connection of the framework with a quantitative database
and the connection to expert knowledge through a user interface.

This chapter focused on verification of the model function and its applicability to the types of
decision problems under review. It provided the opportunity to demonstrate the suitability of a
life cycle cost-focused risk management system in assessing risks on projects with long
contractual terms. The applicability of the model was demonstrated by selecting a correctional
facility case study that had recently undergone an EPC retrofit, and is currently in the energy
savings phase. Personnel involved with this project, from both the ESCO and in the facilities
management function of the correctional agency, were knowledgeable, engaged, and willing to
participate in long-term research of EPC retrofit risks.

The applicability of the life cycle cost-based risk model was demonstrated using data from the
case study. The advice provided to visualize and to manage life cycle risks during early project
phases received positive feedback. Areas of concern among the professionals involved in
framework review were the ability to include multiple mitigation strategies for each risk scenario
and the ability to automatically generate optimal solutions that maximize ELCV without
manually searching for and assigning mitigation strategies to each risk. Finally, a suggestion was
made to enhance the tool’s applicability by connecting it to standard energy modeling software,
such that the risk tool can also be used as a design advisor. In that way, life cycle cost impacts of

312

project risks can be incorporated into the project design prior to contract execution. This is
recommended as an area for future research.

313

CHAPTER 7
SUMMARY AND FUTURE RESEARCH

7.1 INTRODUCTION
In this chapter, a summary of the research is presented, followed by a review of the research goal
and the output of each objective in support of attaining that goal. The key contributions and
conclusions of this research are presented next, and finally recommended areas for future
research are proposed.

The main goal of this research is to understand risks related to the adoption of energy efficiency
retrofit practices in the municipalities, universities, schools, and hospitals (MUSH) market. To
achieve this goal, in-depth analysis and improved understanding was sought with regard to
contractors’ risks that are unique to energy performance contracts (EPC) performed in this
market. This was supported through the development of a model that enabled the management of
risks across the project life cycle. To help develop this model, a preliminary framework was
developed, based on literature and a preliminary expert interview, to identify, describe, organize,
and classify risks faced by energy service companies (ESCOs) when undertaking EPC retrofit
projects.

The preliminary risk framework was refined through the extensive and comprehensive study of
ESCO risks in MUSH market projects through a knowledge elicitation strategy. The preliminary
framework revealed that construction and engineering professionals rely heavily on their
expertise to identify and analyze risks. Concurrent with the development of the refined risk

314

framework was the assessment of EPC project phases, the key decisions made during each phase,
and the quantitative and expert sources of knowledge used to support the decision-making
process in an EPC retrofit process model. A risk analysis and evaluation framework was
proposed, using the refined risk framework and retrofit process model as inputs to identify key
risks and assess their criticality based on values of expected cost. The expected cost data,
combined with the information sources identified in the retrofit process model, was used to
develop a life cycle cost-based risk model. The application of the risk model was then
demonstrated through a pilot scale implementation on a correctional facility case study.

Chapter 1 provided the background, established the need and associated premises, outlined the
goal and related objectives, outlined the methodology, and identified the scope, limitations, and
projected outcomes of this research. The literature related to energy performance contracting,
risk in construction, and project life cycle cost considerations was examined in Chapter 2.
Chapter 3 detailed the research methods and data collection efforts to support the research goal
and objectives. The first stage of the energy management process, risk identification, was
pursued in Chapter 4 through a comprehensive expertise elicitation strategy, which was used to
elicit and compile relevant domain knowledge related to project-level risks in EPC retrofits.
Chapter 5 detailed the development of a risk analysis and evaluation framework which was
parameterized as a life cycle cost-based risk model. The applicability and suitability of the model
was demonstrated through a pilot scale implementation using a correctional facility retrofit case
study in Chapter 6.

315

7.2 SUMMARY OF OUTPUTS BY OBJECTIVES
The main goal of this research, supporting increased adoption of MUSH market EPC retrofits
through enhanced understanding of ESCO risks inherent in the process, was supported through
the development of a life cycle cost-based risk model. The model enabled the incorporation of
quantitative information and expertise and included probabilistic elements to capture some of the
variation inherent in EPC retrofits. The purpose was to represent key decisions made and
information used to support those decisions during the risk management process undertaken by
industry professionals. This section evaluates the research output of each objective and their role
in achieving the research goal.

Objective 1 - To perform an exploratory analysis of the energy performance contract
retrofit process and identify key areas of performance risk requiring in-depth analysis:
Buildings in the United States consume a great deal of energy. Correctional facilities are among
the upper half of buildings for energy consumption. Despite their generally large size, historical
exclusion from energy efficiency retrofits, and high energy intensities, these facilities lag behind
the energy efficiency improvement of other public building types. Lack of capital to finance such
projects has been cited as a reason for this efficiency gap. The use of EPC retrofits has been
growing quickly, particularly in the MUSH market, in response to constrained public budgets
and decades of deferred maintenance.

The use of EPC for delivering energy efficiency retrofits is particularly attractive for public
sector clients, not only because its financing mechanism facilitates the development of projects,
but because the performance guarantee effectively allocates performance risk from the client to

316

the ESCO. The transfer of risk and lengthy contract terms inherent with EPC retrofits (12-15
years is not uncommon; some jurisdictions permit 20 year terms) create the need for ESCOs to
better understand risk management processes and the financial impacts of risks over the long
project life cycle. A review of risk management methods employed by the construction and
engineering management (CEM) industry, as well as by researchers in the same domains
indicates that expertise is heavily relied on for risk identification and qualitative risk analysis. A
wide variety of more rigorous qualitative and quantitative methods have been used in CEM
research and despite the analytical methods selected, a four step risk management process is
commonly encountered throughout the discipline.

Using this process as a guide, risks borne by ESCOs during EPC retrofit projects were studied
through the creation of an a priori risk framework. This framework utilized an extensive
literature review and preliminary interview with an ESCO professional possessing 30+ years of
experience. Ten risk categories were identified through this process; one category was
subdivided into three, for a total of twelve categories. This a priori risk framework was an
important first step in characterizing ESCO risks and was used as an input to the work steps
conducted as part of Objective 2.

Objective 2 - To construct a framework describing the sources of, and mitigation strategies
employed for assessing key risks in energy performance contract retrofits: The literature
review, preliminary expert interview, and a priori risk framework highlighted the importance and
use of expertise in making risk-based decisions in the CEM industry and domain-specific
research. As a result, an expertise elicitation strategy was developed relative to the risk

317

management process outlined in Objective 1. The elicitation strategy utilized comprehensive
knowledge elicitor training, the Delphi technique, semi-structured interviews, and job
shadowing. The use of Delphi, a well-documented consensus-building technique, encouraged
participation via semi and total anonymity (as requested by individual panelists), enabled
panelists’ use of professional language and standards that were familiar to them when describing
their knowledge, and reduced bias while maintaining acceptable levels of reliability and validity.

Based on the ESCO risk expertise elicitation strategy, 19 ESCO professionals participated in a
two-round interview process, completed primarily via telephone during the first round, and
through an online survey platform in the second round. Expertise was elicited and compiled with
regard to risk identification, risk importance scoring, risk causes and control strategies, and
methods for risk identification and evaluation used in the ESCO industry. Questions sought two
types of knowledge: (1) constructed and (2) elicited. Achieving consensus was the goal of
elicited knowledge, in order to best represent the collective expertise and decision-making
criteria of the panelists, whereas the goal of constructed knowledge was to gain in-depth
understanding of practices and techniques employed by ESCOs in the management of risks. No
previous studies could be found where knowledge was elicited from ESCO professionals within
the context of risk management; therefore, this strategy represented a new approach in this body
of research.

The elicited knowledge was used to refine the a priori risk framework through the inclusion of
measures of risk consideration frequency and risk importance, modification of risk categories
and their associated risks, and the inclusion of risk causes and control strategies. Risk importance

318

scoring by the Delphi panelists led to the selection of two risk factors for further analysis in
Objective 3 – energy audit quality and ECM selection and installation. An EPC retrofit process
model was developed concomitantly with the refined risk framework. This model examined key
risk-based decisions made during each phase of the EPC retrofit project life cycle and connected
sources of quantitative information and expert knowledge to the specific decisions and retrofit
project activities that were based on them. These two outputs served as important inputs to the
work steps completed in Objective 3.

Objective 3 - To develop a strategy for analyzing and evaluating risks for energy
performance contract retrofits focused on managing expected costs throughout the project
life cycle, and use data collected through this strategy to develop and parameterize a risk
model: The risk management process was completed in Chapter 5, which focused on analyzing
risks based on their criticality and evaluating possible mitigation strategies for their overall
impact on project outcomes. In order to accomplish this, a method was sought that would capture
the decision making process and expertise of knowledgeable professionals and express measures
of risk dimensionally, in order to improve management outcomes.

Failure mode and effects analysis (FMEA) was selected as the central element of this strategy for
its focus on events that contribute to systemic failure (i.e., the failure to achieve the energy
performance guarantee due to risk events throughout the project life cycle), its ability to address
the complete risk management process in a single method, and its widespread use and acceptance
among the risk management community. While not often applied to project management in
general and the management of energy efficiency retrofits, specifically, this method has been

319

used to examine failure modes in the built environment and it has demonstrated its ability to
utilize expertise in an organized and robust analytical process. Scenario-based FMEA (SFMEA),
a methodological improvement over traditional FMEA, was selected for this research for its
ability to construct variable length cause and effect chains, its use of expected cost (EC) to
express levels of risk, and its focus on examining risk at various points in a system life cycle.

A four member expert panel was convened to complete the SFMEA process and they used the
refined framework to identify potential risk causes from which the analysis began. An empirical
evaluation of the differences between risk criticality measures used in traditional FMEA and
SFMEA demonstrated the strength and robustness of the EC method. The panel identified 77
different risk scenarios related to two risk categories applied to the context of correctional
facility EPC retrofits – energy audit quality and ECM selection and installation. Project-, facility,
and external environment-factors influencing risks were identified by the panel and their
individual decision-making was captured through independent and consensus-based scoring of
risk probabilities and costs. An additional expert examined mitigation strategies.

The net result of these efforts was the representation of the risk analysis and evaluation process
in a model framework. The goal of the model was to decompose the decision-making process
relative to risk management in EPC retrofits by incorporating expertise, quantitative information,
and uncertainty to assist with decision-making over lengthy project life cycles. Consensus risk
scoring and external factors were developed as model parameters, along with probabilistic
parameters related to variability of climate conditions and ECM service life. External data, as
described by the EPC retrofit process model, was developed as a supporting database for the

320

model and a user interface was constructed using Microsoft Excel. The interface supported the
entry of data by both the user and through automated selection of SFMEA panel expert values, as
well as communicated risk management results to the user by providing a measure of risk
mitigation called the expected life cycle value (ELCV). The ELCV measures the amount of
unmitigated risk, expressed as EC that has been effectively mitigated once efficacy and cost of
risk mitigation efforts have been accounted for.

Objective 4 - To demonstrate the applicability of the developed life cost-based risk model
through a pilot application to a case study site: The functional capacity and applicability of
the modeled framework to improve the correctional facility EPC retrofit risk management was
assessed through a pilot test with a case study site. The site was selected as it was recently the
subject of an EPC retrofit and was presently in the energy savings phase, the ESCO and agency
staff members involved in the project were engaged with the project and willing to share their
knowledge and project documents, and the site was representative of several other such projects
either recently completed or currently under contract.

Results of the pilot test verified its applicability to the domain and its suitability to study risk
management problems inherent with such projects. Modeled case study results, which led to
system refinement, indicated that there is strong potential to extend the use of this system more
broadly.

321

7.3 RESEARCH CONTRIBUTIONS
Despite the generally wide adoption of energy efficiency practices, and despite having some of
the greatest need to implement energy efficiency in existing buildings, MUSH market facilities
have experienced difficulties in securing public funding to pay for this work. This research
contributes to attaining the full market potential in MUSH market EPC retrofits, particularly in
correctional facilities. These buildings have unique challenges for retrofit, and despite their
above average need for improving energy efficiency, have exhibited slower than average
improvements in this area. The following sections highlight and briefly discuss the contributions
of this research.

7.3.1 Consensus-Based Assessment of ESCO Risk Management
Consensus was reached among a panel of ESCO industry experts with regard to risk
management and the knowledge of project risks that is used in decision-making. The Delphi
technique was used for this step, which is a consensus-based method that utilizes a questionnaire
to collect data from pre-qualified, expert participants. The fact that consensus was achieved for
the questions included in the study indicates that the elicited knowledge was acceptable to the
panelists and applicable to the problem under review.

This method enhances the reliability and validity of the data. Reliability is enhanced through the
anonymity of participants, which avoids group biases and groupthink. The use of group opinion
helps to ensure validity and is preferred to individual opinions. Additionally, panelists were
selected using purposive sampling, which helps ensure that only domain experts were included in

322

the research, thereby pre-qualifying the sources of the elicited knowledge before data collection
began.

This consensus-based approach and the data it provided is a contribution. In addition to the
dearth of research related to EPC risks in general, the consensus-based approach, in particular, is
an important contribution in that it ensures that early research directed on the subject is properly
grounded in methods that are both reliable and valid.

7.3.2 Characterization of EPC Retrofit Risks Borne by ESCOs
Despite a robust literature pertaining to CEM risk management, there is a lack of research
directed toward understanding risk management in EPC retrofits. This is particularly
troublesome, since one of the central features of EPC projects is the complete transfer of
performance risk from the client to the ESCO. As a result, while clients can view energy
efficiency projects delivered in this way as a risk management strategy, ESCOs must actively
manage their risks and ensure successful project outcomes. As a first step in managing the risks
faced by ESCOs, they must be identified and understood to facilitate future research on those
that are most critical.

The first related contribution in this section was the development of an a priori risk framework
which was guided by the literature and a preliminary expert interview. This served to create a
conceptual grounding for the remainder of the work and was used to develop the expertise
elicitation strategy for this research. A refined framework was developed based on the elicited
expertise of 19 ESCO professionals, and represents the second contribution in this section. In

323

addition to the conceptual grounding offered by the preliminary framework, this effort better
delineated and organized risk categories and assigned specific risks, assigned risk categories to
the project life cycle phases where they are realized, included measures of risk importance and
frequency of consideration of risks in MUSH projects, and the identification of potential risk
causes and mitigation measures. These efforts represent a first-of-their kind comprehensive study
of risks in EPC retrofit projects.

7.3.3 Empirical Evaluation of Improved FMEA Method
Risk analysis and evaluation was accomplished through the use of SFMEA. While this
represented several methodological improvements over the traditional FMEA approach, little
empirical evaluation of its use and comparison with traditional FMEA had been conducted.
SFMEA was selected for this research due to its focus on risk scenario development and
measurement of risk criticality using expected cost, as several Delphi panelists had underscored
the importance of financial measures when assessing EPC project risks. During implementation,
SFMEA panelists were asked to analyze risks using traditional means and the EC method
inherent to SFMEA. Based on the data obtained from the panel, both methods were analyzed and
their suitability for decision-making was evaluated. The empirical analysis of the criticality
measures is in itself a contribution to the risk literature. The application of FMEA to energy
efficient retrofit project risks, through the modified SFMEA technique, represents another
contribution to the study of construction risks via a method that comes from the manufacturing
and quality domain.

324

7.3.4 Development of Life Cycle Cost-Based Risk Model
The development of the life cycle cost-based risk model is an advancement over traditional risk
management methods in that latent project risks, those that may not occur until many years into
the project, can be examined during the early stages of the project life cycle. Furthermore, model
development utilized a combination of expertise, quantitative information, and probabilistic
functions to create and define model parameters.

The model is expected to help ESCOs make informed decisions about the approach they take
towards risk management. By clearly demonstrating the financial implications of unmitigated
risks, the cost and efficacy of selected mitigation strategies, and their net benefit on levels of risk
by phase and to the overall project, risk management-related decisions should be improved. The
ELCV, a metric to evaluate the cost-benefit relationship of risk mitigation strategies and their
impact on the project was developed, and taken with the life cycle cost-based risk model,
represents a significant contribution to energy efficiency, EPC, and risk management domains.

7.3.5 Expansion of Research Approach to Other Domains
Finally, the research approach employed in this study to identify, analyze, and evaluate risks
based on their expected costs over the project life cycle can be replicated in other domains. The
ability to incorporate expertise, quantitative information, and probabilistic inputs strengthens the
ability to represent any system under review, regardless of domain, and leads to the creation of
robust findings that are broadly applicable. While the frameworks used in this research relied
strictly on knowledge elicited from domain experts, the approach can be expanded to elicit
knowledge from end-users, stakeholders, and regulators for incorporation with the model.

325

7.4 RECOMMENDED AREAS OF FUTURE RESEARCH
This section proposes three additional areas which are recommended for future research.

7.4.1 Integration of Modeled Energy Performance with Project Risks
Decision support systems (DSS) have been widely used to assist with the design of energy
efficient buildings and the selection of ECMs in energy efficient retrofits (Horsley et al. 2003;
Juan et al. 2009; Kolokotsa et al. 2009; Chidiac et al. 2012; Ferreira et al. 2013). Typically these
DSS optimize energy performance with environmental, social, and/or economic outcomes, as
described by Ferreira et al. (2013). Far fewer efforts have focused on optimizing energy
performance and project risks. Heo et al. (2012) used a Bayesian approach to calibrate energy
models based on parameter uncertainty. Jackson (2010) used risk management tools to support
the selection of energy efficiency investment options. None of this research examined the effect
of project-level risks as part of the energy efficiency performance or ECM selection process.

Through the incorporation of energy modeling techniques and identification of critical project
factors, a DSS can be developed that seeks an optimal solution based on objective functions that
seek to minimize the life cycle costs of project risks while maximizing energy performance and
minimizing project costs. The risk model that was developed for this research was focused on
project-level factors that give rise to risks; however, many of these factors incorporated design
features and conditions that could not be addressed without the incorporation of the energy
analysis and modeling conducted as part of the retrofit design. Examples of this include:


Risk Scenario 1E1 - differing stakeholder needs result in a savings shortfall or erosion of
the ESCO’s margin due to user modification of ECMs and override of operating

326

parameters: lack of knowledge of the ECM-by-ECM modeled energy performance and
assumptions to include run times, set points, and operating schedules make these effects
difficult to assess in the project risk model.


Risk Scenario 2G3 – actual climate conditions occurring outside the range of design
parameters result in a savings shortfall or erosion of the ESCO’s margin due to
uncertainty in modeled parameters and reduced equipment service life: comprehensive
quantification of this risk scenario relies on understanding modeled energy and
equipment service life/performance parameters, which cannot be assessed solely by
project-level risks.

The expansion of the risk model from this research through integration with building energy
models and a more detailed assessment of project-, facility-, and external environment-related
factors would enable the creation of a multiple criteria design advisor. The goal of the tool would
be to guide the retrofit design phase of EPC projects by minimizing life cycle project-level risks
while maximizing energy performance.

7.4.2 Development as an Intelligent Decision Support System
Duah (2014) and Syal et al. (2014) demonstrated the applicability of an intelligent decision
support system (IDSS) framework, to improving the energy efficient retrofit process in the
residential buildings context. An IDSS is a decision support system that includes a knowledge
management system that utilizes expertise to help users make decisions, and may be particularly
applicable to unstructured and semi-structured problems. The components of an IDSS include a
data management system, a knowledgebase management system, and a user interface. An

327

inference engine is used to integrate quantitative information and expertise and infer new
decision-making options based on stored expertise (Duah 2014).

The current research explored the use of quantitative information and expert knowledge in the
EPC retrofit process, and developed a retrofit process model to represent those relationships, as
well as the presence of risks and the timing of key decisions throughout the project life cycle.
The life cycle cost-based risk model utilized a database that incorporated many of the
quantitative sources from the model and applied expertise derived from the SFMEA panel, by
enabling its use as a source for input values to existing risk scenarios. The small sample size of
the SFMEA panel, while acceptable for the given method, was too small to draw inferences from
with regard to individual decision-making and biases, thus limiting the use of expertise in the
model.

Grounded by the refined risk framework and the risk analysis and evaluation process, expertise
should be elicited from a larger sample of ESCO professionals that focuses on capturing their
decision-making process with regard to risk identification, scenario construction, and
determination of risk criticality. Determinants of potential decision-making bias should also be
refined and captured from this larger group to develop predictive models of risk behavior and
decision-making related to risk management. This predictive model can be used to develop the
elements of knowledgebase management system. When combined with the data management
system utilized in this research and a refined user interface, to include development of an
inference engine, this research can thus be expanded to take the form of a robust, knowledge-

328

based decision support system to assist users in making life cycle cost-based risk management
decisions in EPC retrofit projects.

7.4.3 Critical Success Factors for EPC Retrofits in Complex Facilities
During data collection activities, many ESCO experts provided their thoughts about what makes
some projects successful while others are less so. Many of the stated factors were not directly
applicable to the current research, as they did not impact attainment of the energy guarantee.
Rather, several identified factors related to the quality of the ESCO-client relationship, the
political landscape of the client agency, client leadership and commitment to the EPC model,
communication throughout the project, facility-specific operating parameters, and the
development of mutually-acceptable measurement and verification strategies.

These examples far exceed the scope of the risk model and indicate the presence of many
complex, interrelated factors that may govern successful outcomes of EPC retrofit projects. The
empaneled experts for the current research reached consensus on the idea that correctional
facility EPC retrofits have unique complexities and may indeed be riskier than other facility
types; however, a subset of like buildings was noted, to include hospitals, continuously-operating
facilities, and mission critical facilities. There is, therefore, an inherent hierarchy of building
types, ordered by the complexity of EPC retrofits taking place within their walls.

Based on the foregoing, a comprehensive assessment and analysis of critical success factors in
these complex building types can be undertaken. The perspective gained from both ESCOs and
their clients will be used to enhance the management of such projects and improve outcomes. As

329

a result, the use of EPC retrofits among these buildings will be increased and energy
performance of their related market sectors will be improved.

7.5 CHAPTER SUMMARY
This chapter provided a summary of the research goal and objectives, and the outputs that led to
developing a life cycle cost-based risk model for ESCOs undertaking EPC retrofit projects in
MUSH market buildings, specifically correctional facilities. The research activities and outputs
were summarized for each objective and contributions to the body of knowledge as well as
conclusions of this research were highlighted. Recommendations for future research were
proposed, focused on expansion of the work within this domain and further work in related
domains.

The goal of this research was to use enhanced risk management understanding and modeling to
better understand the risks to ESCOs that are inherent in EPC retrofits, particularly in building
sub-markets that have a significant need to increase energy efficiency. Focus was given to
MUSH market buildings and correctional facilities as a sub-market, owing to limitations on the
availability of capital financing and decades of deferred maintenance. These factors have created
a near-crisis level condition in correctional facility operating costs and reduced energy
performance across the country. The researcher envisions that full implementation of the life
cycle cost-based risk management model will further encourage the growth of the ESCO
industry, and support focused retrofits in complex building types that typically can benefit the
most from such work. Ultimately, this will reduce the energy consumption of public sector

330

buildings to levels that are more fitting with the global principles of sustainability and
responsible management of constrained resources.

331

APPENDICES

332

APPENDIX A.1

Preliminary Questionnaire for Risk Tolerance, Risk Identification, and Risk Management
in Energy Performance Contracting

Phase I Interview Questions
1. Please provide a description of your role in energy performance contract projects performed
in MUSH market buildings.
a. How many such projects have you worked on?
b. Which sub-markets have you worked in (provide standard definitions)?
c. How many years have you been working on such projects?
d. What is your educational/professional background (e.g., engineer, architect,
construction manager, etc.)
2. These questions relate to previously identified risk categories for ESCOs engaged in
performance contracts (see attached).
a. Do you consider any of these risk categories when developing energy performance
contract projects? If so, which ones?
b. Are there any risk categories that you consider that are not included in this list?
c. Do MUSH market energy performance contract projects have different risk categories
than non-MUSH market projects? If so, please describe those differences.
3. What is your risk management process for energy performance contracts?
a. How do you identify risks in these projects? Do you use formal or informal methods?
Follow up with questions about knowledge of/use of FMEA, tree analysis, Monte
Carlo, risk-based LCCA, or other? - why/why not are these methods used?
b. What method or methods do you use for evaluating the impact of a risk on the
project?
c. At what stage of the project do you identify risks?
d. What are the top 3 reasons (risk categories) you believe an energy performance
contract would not meet the performance guarantee?
4. For each of the risk categories identified in the previous question, what are the effects on the
ESCO if these risks are not prevented or corrected?
a. For each item listed, what can cause these risks?
b. For each item listed, are there controls implemented that either prevent these risks or
detect negative impacts of these risks if they occur?
c. Do combinations of these risks pose a greater threat than these risks individually?

333

5. How do you evaluate potential energy conservation measures individually and as a portfolio
for inclusion in an energy performance contract?
a. Do you use life cycle cost analysis? If so, how do you define parameters (discount
rate, inflation rate, energy costs)? If not, why not? What performance periods are
typically used?
b. Do you consider future uncertainty of life cycle costs (discount rate, inflation rate,
energy costs)? Uncertainty of project categories (cost growth, schedule growth)
c. Are non-energy benefits included in this analysis? If so, please describe. If not, why
not?
REFERENCE FOR QUESTION 1b
Definition of MUSH Market and Sub-Markets
1.
2.
3.
4.

Municipal agencies (state/local government)
Universities/colleges
K-12 schools
Hospitals
REFERENCE FOR QUESTION 3
List of Previously-Identified Risk Categories for ESCOs Engaged in Performance
Contracting

1. Customer Pre-Qualification
a. Financial factors
i. Customer may go out of business before full contract payment
b. Facility/technical factors
i. Customer-preferred short performance periods limit technical approach
ii. Changes in future occupancy and use
iii. Unknown latent conditions
c. People factors
i. Human activity inconsistent with M&V plans
ii. Improper O&M undertaken by customer
iii. Interference with customer operations
2. Project Development
a. Costs incurred from project start-up; long development phases can lead to difficult to
recover costs.
3. Energy Audit Quality
a. The investment grade audit must include a risk assessment for each proposed ECM
b. Improperly-established or disputed baseline can impact calculations of energy savings
and also give rise to disputes.

334

4. Equipment Selection and Installation
a. Selected ECMs not aligned with findings of the IGA.
b. ECM package feasibility, failure to perform as designed, uncertainty in factors used
to predict performance.
c. Reduced involvement of ESCO in ECM selection/installation, procurement through
bidding, risk of improper installation if the ESCO is not involved during the
construction phase.
5. Commissioning
a. Failure to commission may lead to missed opportunities to verify ECM performance
and better overall project performance.
b. Failure to commission may miss opportunities to verify installed equipment
performance, to ensure that calibration, operation, and maintenance procedures are
well-understood, and that all system documentation is turned over.
6. Operations and Maintenance Practices
a. Poorly understood responsibilities by each party if the O&M plan is unclear.
b. The customer may not perform O&M work to specification.
c. Customer self-performance or contracted O&M may reduce revenue for ESCO.
7. Measurement and Verification of Savings
a. Poorly-developed M&V plans can create additional risk for the ESCO. Poorlydesigned M&V sampling protocols may not accurately reflect the overall
performance of the ECM package.
b. M&V protocols that do not capture non-energy benefits of EPCs may understate
overall system performance.
c. Failure to include O&M in the M&V plan may lead to missed savings.
8. Project Management Over the Project Life-Cycle
9. Construction-Specific Concerns
a. Schedule growth may cause unrecoverable costs for the ESCO.
b. Cost growth may impact the financial analysis that the savings guarantee is premised
on - cost overruns may result from schedule delays, latent site conditions, and field
changes.
10. Volatility of Energy Prices
a. Changing prices can reduce project value.

335

APPENDIX A.2

Delphi Round 1 Questionnaire for Risk Tolerance, Risk Identification, and Risk
Management in Energy Performance Contracting

Phase II Interview Questions
QUESTIONNAIRE PART I – YOUR RISK BEHAVIOR AND TOLERANCE
Note: Please complete Part I prior to conducting Part II over the telephone or in-person.
The following questions are based on a well-known and vetted risk-based portfolio selection tool
used by TIAA-CREF (source: https://ais4.tiaa-cref.org/asstallocguidance/nsjsp/dms.do). These
questions are intended to measure your individual approach to risk, and will help with further
analysis of the data collected in Part II. Your individual raw answers to these questions will not
be released except as aggregate analysis of all respondents to the questionnaire, even if you
consent to the release of your name and company affiliation.
Please read the background information before answering each corresponding question.
QUESTION 1 BACKGROUND INFORMATION: Inflation, the rise in prices over time, can
erode your investment return. Long-term investors should be aware that, if portfolio returns are
less than the inflation rate, their ability to purchase goods and services in the future might
actually decline. However, portfolios with long-term returns that significantly exceed inflation
are associated with a higher degree of risk.
1. Which of the following portfolios is most consistent with your investment philosophy?
Please mark your answer with an X in the box to the left of your selected option.
a. Portfolio A will most likely exceed long-term inflation by a significant margin and
has a high degree of risk.
b. Portfolio B will most likely exceed long-term inflation by a moderate margin and has
a high to moderate degree of risk.
c. Portfolio C will most likely exceed long-term inflation by a small margin and has a
moderate degree of risk.
d. Portfolio D will most likely match long-term inflation and has a low degree of risk.
QUESTION 2 BACKGROUND INFORMATION: Portfolios with the highest average returns
also tend to have the highest chance of short-term losses. The table at right provides the average
dollar return of four hypothetical investments of $100,000 and the possibility of losing money
(ending value of less than $100,000) over a one-year holding period.

336

2. Please select the portfolio with which you are most comfortable.
Please mark your answer with an X in the box to the left of your selected option.
Probabilities After 1 Year
Possible Average Value
Chance of Losing Money
at the End of One Year
at the End of One Year
a. Portfolio A
$106,000
16%
b. Portfolio B
$107,000
21%
c. Portfolio C
$108,000
25%
d. Portfolio D
$109,000
28%
Data supplied by Ibbotson Associates.
QUESTION 3 BACKGROUND INFORMATION: Investing involves a trade-off between
risk and return. Historically, investors who have received high long-term average returns have
experienced greater fluctuations in the value of their portfolio and more frequent short-term
losses than investors in more conservative investments have.
3. Which statement best describes your investment goals?
Please mark your answer with an X in the box to the left of your selected option.
a. Protect the value of the account. In order to minimize the chance for loss, I am
willing to accept the lower long-term returns provided by conservative investments.
b. Keep risk to a minimum while trying to achieve slightly higher returns than the
returns provided by investments that are more conservative.
c. Balance moderate levels of risk with moderate levels of returns.
d. Maximize long-term investment returns. I am willing to accept large and sometimes
dramatic fluctuations in the value of my investments.
QUESTION 4 BACKGROUND INFORMATION: Historically, markets have experienced
downturns, both short-term and prolonged, followed by market recoveries. Suppose you owned a
well-diversified portfolio that fell by 20% (i.e. $1,000 initial investment would now be worth
$800) over a short period, consistent with the overall market.
4. Assuming you still have 10 years until you begin withdrawals, how would you react?
Please mark your answer with an X in the box to the left of your selected option.
a. I would not change my portfolio.
b. I would wait at least one year before changing to options that are more conservative.
c. I would wait at least three months before changing to options that are more
conservative.
d. I would immediately change to options that are more conservative.

337

QUESTION 5 BACKGROUND INFORMATION:

The graph at right (Figure A-1) shows the
hypothetical results of four sample portfolios over a
one-year holding period. The best potential and
worst potential gains and losses are presented. Note
that the portfolio with the best potential gain also
has the largest potential loss.

Figure A-1. TIAA-CREF Questionnaire
Comparison Portfolios
5. Which of these portfolios would you prefer to hold?
Please mark your answer with an X in the box to the left of your selected option.
a. Portfolio A
b. Portfolio B
c. Portfolio C
d. Portfolio D
6. I am comfortable with investments that may frequently experience large declines in
value if there is a potential for higher returns.
Please mark your answer with an X in the box to the left of your selected option.
a. Agree
b. Disagree
c. Strongly Disagree

338

QUESTIONNAIRE PART II – UNCERTAINTY, RISK MANAGEMENT, AND LIFE
CYCLE COSTING IN ENERGY PERFORMANCE CONTRACTING
Note: Please complete Part I prior to conducting Part II over the telephone or in-person. If
possible, please complete Question 1 of Part II prior to conducting the rest of Part II over the
telephone or in-person.
The following questions are based on your experiences in performing energy performance
contract retrofit projects. All tables will be filled out by the researcher during the telephone or
face-to-face interview. The tables and questions are provided here for your review; however, no
action is needed on your part until the interview is conducted.
1. Please provide a description of your role in energy performance contract projects performed
in MUSH (Municipal agencies [state/local government] – includes correctional facilities,
Universities/colleges, K-12 Schools, and Hospitals) market buildings.
Sub-Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing financial
incentives on these projects?
1f. On average, how many building systems were impacted on each project
you worked on (e.g., envelope, lighting, central plant decentralization,
water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in engineering or a
related field to building construction, operations, design, or energy? If yes,
what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy engineering or
management program at an accredited college or university?
1j. Have you published at least two peer-reviewed publications or one book
or book chapter related to EPC? If yes, how many of each type?
1k. Have you ever been an invited conference presenter on a related
subject? If yes, how many times?

Your Answer

2. These questions relate to previously identified risk categories for ESCOs engaged in
performance contracts (see Table A-1 for questions 2a and 2b).
a. Do you consider any of these risk categories when developing and executing
energy performance contract projects? If so, which ones?
b. Are there any risk categories that you consider that are not included in this list? If
yes, we will add these to Table 1 and complete the questions about frequency and
reasoning for consideration.

339

c. What are the top 3 reasons (Risk Categories from Table A-1) you believe an
energy performance contract would not meet the performance guarantee? Do you
base this selection on one or more of the following?
i. Probability of occurrence?
ii. The seriousness of the end effect – measured through cost or another
factor?
iii. Difficulty in detecting the risk factor?
iv. Ability to mitigate a risk factor once identified?
3. For each of the risk categories identified in the previous question, what are the effects on the
ESCO if these risks are not prevented or corrected (Please refer to Table A-2)?
a. For each of the three categories listed, what can cause these risks?
b. For each item listed, are there controls implemented that either prevent these
risks, detect negative impacts of these risks if they occur, and/or mitigate these
risks? If so, what is done? Are formal methods used (e.g., status meetings, project
risk response audits, risk response planning, earned value analysis, none, or
other)?
c. Do combinations of these risks pose a greater threat than these risks individually?
4. Do MUSH market energy performance contract projects have different risk categories than
non-MUSH market projects? If so, please describe those differences.
5. What is your risk management process for energy performance contracts?
a. How do you identify risks in these projects? Do you base this on experience,
previous projects, client requirements, case based reasoning, contractual
requirements? Do you use formal or informal methods? Follow up with questions
about knowledge of/use of documentation review, brainstorming, checklist
analysis, root cause analysis, interviewing, SWOT analysis, assumption analysis,
cause/effect diagramming, system/process flow chart, influence diagram, Delphi
technique, none, other (Thaheem and DeMarco 2013) - why/why not are these
methods used?
b. At what stage of the project do you identify risks?
c. Who on the project team identifies these risks and performs these functions (e.g.,
sales lead, PM, engineer, attorney)?
d. What method or methods do you use for evaluating the impact of a risk on the
project? Follow up with questions about knowledge of/use of FMEA, tree
analysis, Monte Carlo, risk-based LCCA, none, or other? - Why/why not are these
methods used?
6. How do you evaluate potential energy conservation measures individually and as a portfolio
for inclusion in an energy performance contract?
a. Do you use life cycle cost analysis? If so, how do you define parameters (discount
rate, inflation rate, energy costs)? If not, why not? What performance periods are
typically used?

340

b. Do you consider future uncertainty of life cycle cost analysis model parameters
(discount rate, inflation rate, energy costs)? Uncertainty of project factors (cost
growth, schedule growth, identified risk categories)?
c. Do you build sunk costs (e.g., project management during M&V, inclusion of
costs for security/escort staff, some NEB-related work) and loss leaders (e.g.,
additional no-fee services, some NEB-related work) into these analyses? On what
basis do you include values for these elements?
d. Are non-energy benefits included in this analysis (e.g., water conservation, O&M
savings, avoided capital costs, tradable emissions credits, improved health/wellbeing/productivity)? If so, please describe. If not, why not?

341

1. Client Selection
Factors
1A. Financial
factors
1B. Facility/
technical factors
1C. People factors

1. 2. Project
Development
2. 3. Energy Audit
Quality

- Customer may go out of business before
full contract payment
- Customer-preferred short performance
periods limit technical approach
- Changes in future occupancy and use
- Unknown latent conditions
- Human activity inconsistent with M&V
plans
- Improper O&M undertaken by customer
- Interference with customer operations
- Costs incurred from project start-up; long
development phases can lead to difficult
to recover costs.
- The investment grade audit must include
a risk assessment for each proposed ECM
- Improperly-established or disputed
baseline can impact calculations of
energy savings and also give rise to
disputes.

342

Never

Rarely
(<10%)

Occasionally
(~30%)

Sometimes
(~50%)

Frequently
(~70%)

Non-Comprehensive Examples

Usually
(~90%)

Risk Category

Every Time

Table A-1. List of Previously-Identified Risk Categories for ESCOs Engaged in Performance Contracting
(Accompanies Question 2)
Do you consider any of these risk categories when
developing and executing energy performance
contract projects?

Why?

Table A-1 (cont’d)

3. 4. Equipment
Selection and
Installation

4. 5. Commissioning

- Selected ECMs not aligned with findings
of the IGA.
- ECM package feasibility, failure to
perform as designed, uncertainty in
factors used to predict performance.
- Reduced involvement of ESCO in ECM
selection/installation, procurement
through bidding, risk of improper
installation if the ESCO is not involved
during the construction phase.
- Failure to commission may lead to
missed opportunities to verify ECM
performance and better overall project
performance.
- Failure to commission may miss
opportunities to verify installed
equipment performance, to ensure that
calibration, operation, and maintenance
procedures are well-understood, and that
all system documentation is turned over.

343

Never

Rarely
(<10%)

Occasionally
(~30%)

Sometimes
(~50%)

Frequently
(~70%)

Non-Comprehensive Examples

Usually
(~90%)

Risk Category

Every Time

Do you consider any of these risk categories when
developing and executing energy performance
contract projects?

Why?

Table A-1 (cont’d)

5. 6. Operations and
Maintenance
Practices

- Poorly understood responsibilities by
each party if the O&M plan is unclear.
- The customer may not perform O&M
work to specification.
- Customer self-performance or contracted
O&M may reduce revenue for ESCO.
6. 7. Measurement
- Poorly-developed M&V plans can create
and Verification of
additional risk for the ESCO.
Savings
- Poorly-designed M&V sampling
protocols may not accurately reflect the
overall performance of the ECM
package.
- M&V protocols that do not capture nonenergy benefits of EPCs may understate
overall system performance.
- Failure to include O&M in the M&V
plan may lead to missed savings.
7. 8. Project
Management Over
the Project LifeCycle

344

Never

Rarely
(<10%)

Occasionally
(~30%)

Sometimes
(~50%)

Frequently
(~70%)

Non-Comprehensive Examples

Usually
(~90%)

Risk Category

Every Time

Do you consider any of these risk categories when
developing and executing energy performance
contract projects?

Why?

Table A-1 (cont’d)

8. 9. ConstructionSpecific Concerns

10. Volatility of
Energy Prices
11. Other (Please
Specify)
12. Other (Please
Specify)
13. Other (Please
Specify)
14. Other (Please
Specify)

- Schedule growth may cause
unrecoverable costs for the ESCO.
- Cost growth may impact the financial
analysis that the savings guarantee is
premised on - cost overruns may result
from schedule delays, latent site
conditions, and field changes.
- Changing prices can reduce project value.

345

Never

Rarely
(<10%)

Occasionally
(~30%)

Sometimes
(~50%)

Frequently
(~70%)

Non-Comprehensive Examples

Usually
(~90%)

Risk Category

Every Time

Do you consider any of these risk categories when
developing and executing energy performance
contract projects?

Why?

Table A-2. Features of Top Three Identified Risk Categories
(Accompanies Question 4)
Identified Risk Category
Potential Cause(s)

346

Controls Used

APPENDIX A.3

Delphi Round 2 Questionnaire for Risk Identification and Risk Management in Energy
Performance Contracting
1. Please enter your participant ID sent to you in the email that linked to this survey:

2. Please select the EPC retrofit project phase or phases you most frequently support:
Project Development
Energy Audit
Retrofit Design
Project Execution (Construction and Commissioning)
Energy Savings (O&M and M&V)
Other (please specify)
3. In the first survey, 11 participants identified a total of 10 new risks that were added to matrix
that you responded to. The number in parentheses indicates the number of panelists
identifying each new risk. Please indicate below whether you concur with the addition of
each risk. You may refer to the risk matrix in the document that was attached to the email
you received to see all of the other risk categories and individual risks. A comment field is
available if you wish to add your thoughts.
Agree with
Addition of this
Risk
Political Risks - Client Selection Risks (4)
Productivity Losses in Corrections Projects (1)
Safety (1)
Changing Financial Incentives (1)
Cost of Doing Nothing or Self-Implementing (1)
Public Procedural Risks (1)
Time-Based Risk (1)
Timing (1)
Design Development (1)
Staff Turnover During Project (2)

347

Disagree with
Addition of this
Risk

4. Two participants identified a risk which did not fall under any existing risk category. The
new risk category was identified as "Perception of the Performance Contracting Industry"
and included two concerns:
 Lack of knowledge regarding EPC enabling statutes by the ESCO limits some customers’
understanding of how they can use EPC.
 Unethical behavior negatively impacts the industry as a whole.

Never

(<10%)
Rarely

(~30%)
Occasionally

(~50%)
Sometimes

(~70%)
Frequently

(~90%)
Usually

Every Time

Please rank the frequency with which you consider this risk on your projects. As with the first
interview, consideration means that you address these risks contractually, by project
management, through technical means, or via financial assurance.

Perception of the Performance
Contracting Industry
5. 78% of panelists believe that MUSH market EPC retrofits have a higher level of project risk
than commercial and industrial sector EPC retrofits. Panelists provided five MUSH market
facility types that they believed had higher levels of risk than other buildings in the MUSH
market. Which facility types do you believe have higher levels of project risk (select all that
apply)?
Correctional facilities
Hospitals
K-12 schools
Continuously operated facilities (e.g., 24/7 operation) of all types, including hospitals,
correctional facilities, etc.
Mission-critical facilities
EPC retrofits in MUSH market buildings do not inherently have any more risk than
commercial and industrial EPC retrofits
Other (please specify)
6. In the first survey, you were asked to identify the three most important risk categories you
consider when developing and implementing MUSH market EPC retrofit projects. Based on
your importance scoring, the top 6 risk categories are listed below. Importance scores are in
parentheses next to each risk category. Please rank these from 1 to 6, with a score of 1
indicating the most important risk category to consider (as one panelist stated, "the things
that keep me awake at night when I have a project underway") and a score of 6 indicating the
least important among these 6.
348

1

2

3

4

5

6

Energy Audit Quality (10)
ECM Selection and Installation (8)
Project Development Risks (8)
Construction-Specific Concerns (8)
Commissioning (7)
Measurement and Verification of Savings (5)
7. Please select which method or methods you use to identify risks during projects.
Brainstorming

Monte Carlo Simulation

Cause and Effect Diagram/Ishikawa Diagram

Risk Mapping, Risk Matrix, Probability
and Impact Matrix

Checklist

Root Cause Analysis

Delphi

Sensitivity Analysis

Expertise/Expert Judgement

Strengths, Weaknesses, Opportunities,
and Threats (SWOT)

Failure Mode and Effects Analysis (FMEA)

None

Interviewing

Other (please specify methods used)

8. Please select which method or methods you use to analyze risks during projects. Risk
analysis is defined as determining the level of impact that a risk has on the project.
Brainstorming

Failure Mode and Effects Analysis
(FMEA)

Cause and Effect Diagram/Ishikawa Diagram

Interviewing

Checklist

Monte Carlo Simulation

Decision Tree Analysis

Risk Mapping, Risk Matrix, Probability
and Impact Matrix

Delphi

Strengths, Weaknesses, Opportunities,
and Threats (SWOT)

Expected Monetary Value

None

Expertise/Expert Judgement

Other (please specify methods used)

349

APPENDIX B

Delphi Panel Description and Expert Determination

Use of Purposive Sampling
Participants were selected for this study using purposive sampling techniques (Barbour 2001;
Oliver 2006; Tongco 2007), in order to focus on experts engaged in energy performance
contracting (EPC) work with a variety of energy service companies (ESCOs). Rather than draw a
large sample to be representative of all such ESCOs and types of EPC retrofits, this study
necessitated participation from experts with significant experience in performing retrofits
specifically in municipalities, universities, schools, and hospitals (MUSH) market buildings.
A number of purposive sampling methods have been identified in the literature. Teddlie and Yu
2007 identified a four-category typology for purposive sampling methods:
 Sampling to achieve representativeness or comparability
 Sampling special or unique cases
 Sequential sampling
 Sampling based on combinations of purposeful techniques
This study uses the latter strategy, a combination approach of expert sampling, snowball
sampling, and critical case sampling to guide selection of individual participants, cases, and
documents. The overarching goal of the sampling was to provide expertise from contractors
possessing a high level of knowledge about MUSH market EPC retrofit projects. Expert
sampling was, therefore, a cornerstone of the combination approach used; however, identifying
and gaining access to such experts was difficult in many cases. As a result, a sequential sampling
technique known as snowball sampling (also called chain sampling) was used, as it has been
shown to assist in the recruitment of participants who are difficult to reach for a variety of
reasons (Sadler et al. 2010). Through this technique, upon completion of the interview
questionnaire, respondents were asked to provide referrals to other domain experts who would be
likely to participate in this study and who met pre-determined expert selection criteria. This was
particularly important with this group of practitioners, as unsolicited requests for participation
led to relatively low participation rates (18.5%) and most ESCOs do not readily publish contact
information for individuals who might qualify as domain experts. Experts were identified in
connection with critical cases, which were defined based on key attributes that were central to
the goal and objectives of this research, including:
 Individuals and projects were selected from the MUSH market;
 Individuals with experience in, and projects from the correctional facilities domain were
particularly important; and
 Project documents, especially risk management checklists, were sought from firms with
significant MUSH market retrofit experience and well-documented and widelydisseminated corporate risk management programs.
350

Delphi Panel – Determination of Expertise
Expertise was determined for each participant following a rubric based on Hallowell and
Gambatese (2010) and Duah (2014), as described in Tables 3-1 (expertise selection guidelines)
and Table 3-2 (expertise scoring rubric). Expertise was determined two ways:
1. A threshold number of expert selection categories were met (five out of nine
qualifications met).
2. A goal of at least 4.5 points attained out of 8.5 available points obtained by following the
scoring rubric.
Determinants of expertise are presented in Tables B-1 through B-19 for study participants.
Table B-1. Expertise Assessment for Participant ER-1-E
Company Type: Independent and Other ESCOs
Title: Corrections Market Business Development
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

351

Response
25 + yrs
53 +
All
M - corrections
Yes

Points
1
1
1.5
N/A
0.5

3-5

0.5

No

0

No

0

No

0

No

0

Yes -14

0.5

5
6

Table B-2. Expertise Assessment for Participant ER-2-E
Company Type: Independent and Other ESCOs
Title: Senior Business Development Manager
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?

1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

352

Response
22
50-60
At least 90%
All markets,
primarily M and
U
No

Points
1
1
1.5

0

HVAC,
0.5
mechanical,
envelope,
lighting,
decentralization,
co-gen, water,
controls
No
0
No

0

No

0

No

0

Yes – 10-12

0.5

4.5
5

Table B-3. Expertise Assessment for Participant ER-3-U
Company Type: Utility Company ESCOs
Title: Senior Project Manager
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?

1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

353

Response
9 years
10
8
MUSH, Yes –
corrections
work
No

Points
1
1
1.5

All listed plus
controls,
vending
machine
optimization,
renewable
CEM

0.5

MS in Info
Systems for
Technology
No

2

No

0

Yes – 2

0.5

9.5
7

0

3

0

Table B-4. Expertise Assessment for Participant ER-4-M
Company Type: Building Equipment Manufacturer ESCOs
Title: Manager, Public Housing Innovation & Best Practices
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

354

Response
19
200
200
MUSH
Yes

Points
1
1
1.5

5+

0.5

CEM

3

0.5

BS Mechanical 2
Engineering and
MBA
No
0
No

0

Yes – 3-4/yr –
about 30
10
8

0.5

Table B-5. Expertise Assessment for Participant ER-5-M
Company Type: Building Equipment Manufacturer ESCOs
Title: Project Development Manager I
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?

1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

355

Response
21 years
100s $300million in
work
100%
M,U,S,H; No
corrections
Yes on rebates

Points
1
1

Lighting,
controls, water,
mechanical
upgrades,
envelope, decentral
CEM

0.5

BS - EET

2

No

0

No

0

2/year

0.5

10
8

1.5
0.5

3

Table B-6. Expertise Assessment for Participant ER-6-E
Company Type: Independent and Other ESCOs
Title: Vice President
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

356

Response
14
50
50
M, U, S
Yes

Points
1
1
1.5

6

0.5

No

0

0.5

BS Engineering, 2
Masters in
finance and
business
No
0
No

0

1

0.5

7
7

Table B-7. Expertise Assessment for Participant ER-7-M
Company Type: Building Equipment Manufacturer ESCOs
Title: Sales Team Leader, Northeast
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

357

Response
14
20 to 30
All
K12, Univ,
State
Yes

Points
1
1
1.5

8

0.5

CEM, PE

3

BS EET MBA
- Finance

2

Yes, Part-Time

1

No

0

Yes, ~40

0.5

11
9

0.5

Table B-8. Expertise Assessment for Participant ER-8-M
Company Type: Building Equipment Manufacturer ESCOs
Title: Western Territory Team Project Developer
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?

Response
22
30
9-10
Fed (M), U, S
Yes – incentives
and grants
Lighting, temp
controls, HVAC
modernization,
few envelope,
CEM since
1992/93
BS Aerospace
engineeringmechanical
systems
No

Points
1
1
1.5

No

0

A couple of
times at Trane
and a
conference
TOTAL POINTS: 10
TOTAL CATEGORIES: 8

358

0.5
0.5

3
2

0

0.5

Table B-9. Expertise Assessment for Participant ER-9-M
Company Type: Building Equipment Manufacturer ESCOs
Title: Government Market Manager - Michigan
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

359

Response
33
300
99%
MUSH – most
M
Yes

Points
1
1
1.5

8-10

0.5

No

0

No

0

No

0

No

0

Yes – 10

0.5

5
6

0.5

Table B-10. Expertise Assessment for Participant ER-10-E
Company Type: Independent and Other ESCOs
Title: Senior Sales Executive
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?

1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?

Response
7
~25
16
MUSH –
correctional –
counties with
ESG
Yes

Points
0
1
1.5

6+

0.5

No - Pursuing
LEED AP;
looking at CEM
BS in ME and
AAS in
electronics
No

0
2
0

3 articles
1
published in
trade journals
for New Mexico
Yes – 3+ times
0.5

1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS: 7
TOTAL CATEGORIES: 7

360

0.5

Table B-11. Expertise Assessment for Participant ER-11-E
Company Type: Independent and Other ESCOs
Title: Senior Account Executive
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?

1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

361

Response
18
80-100 - $30-40
million/yr
98%
MUSH Corrections is
largest customer
segment
Yes

Points
1
1

>5

0.5

No

0

No - BSBA

0

No

0

Articles – 3 or 4

1

Yes – 5 or 6

0.5

6
7

1.5

0.5

Table B-12. Expertise Assessment for Participant ER-12-M
Company Type: Building Equipment Manufacturer ESCOs
Title: Senior Contract Manager
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

Response
14
At least 100
90%-95%
MUSH
No

Points
1
1
1.5

5+

0.5

No

0

JD – Directly
related to
function as
contract
manager
No

3

No

0

2

0.5

0

0

7.5
6

Note: There is no entry for participant ER-13 due to the inability to complete an interview after
participant identification numbers were created.

362

Table B-13. Expertise Assessment for Participant ER-14-E
Company Type: Independent and Other ESCOs
Title: Director of Construction Services
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

363

Response
5
100+
95%
All
Yes

Points
0
1
1.5

3 to 6

0.5

No

0

0.5

Yes – BS in CM 2
No

0

No

0

No

0

5.5
5

Table B-14. Expertise Assessment for Participant ER-15-M
Company Type: Building Equipment Manufacturer ESCOs
Title: Design-Build and ESPC Team Manager
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

364

Response
7
12
100%
M,S –
Corrections
through M
Yes

Points
0
1
1.5

5+

0.5

EIT

0

BS
Architectural
Engineering
No

2

No

0

Yes – 1 time

0.5

6
6

0.5

0

Table B-15. Expertise Assessment for Participant ER-16-M
Company Type: Building Equipment Manufacturer ESCOs
Title: Account Executive
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

365

Response
29
120
90%
MUSH; Yes
corrections
Yes

Points
1
1
1.5

Yes

0.5

CEM

3

BSIE; MS in
Energy Systems
Engineering
I&CE
FT faculty @
BGSU – 2 years

2

Articles – 4

1

Yes – 5+

0.5

12
10

0.5

1

Table B-16. Expertise Assessment for Participant ER-17-U
Company Type: Utility Company ESCOs
Title: Associate Product Portfolio Manager
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

366

Response
5
~40
95%
MUS
Yes –
Incentives
3-4; Lighting
and HVAC,
some envelope,
No

Points
0
1
1.5

No

0

No

0

Article – yes
ACEEE
whitepaper
Yes – four
times
5
6

1

0.5
0.5
0

0.5

Table B-17. Expertise Assessment for Participant ER-18-E
Company Type: Independent and Other ESCOs
Title: Energy Engineer
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

367

Response
10
~30
6 fed, rest
MUSH
MUSH
Yes - rebates

Points
1
1
1.5

3+

0.5

CEM and
LEED AP
MS in
Mechanical

3

No

0

No

0

0.5

2

Yes –
0.5
Numerous times
10
8

Table B-18. Expertise Assessment for Participant ER-19-E
Company Type: Independent and Other ESCOs
Title: Energy Engineering Manager
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?

Response
14
140+
90%
MUSH
Yes

Points
1
1
1.5

3+

0.5

PE and CEM

3

BS ME minor
in match

2

Associate
Faculty at
Cascadia
Community
College
No

1

1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
Yes – at least 6
related subject? If yes, how many times?
TOTAL POINTS: 11
TOTAL CATEGORIES: 9

368

0.5

0
0.5

Table B-19. Expertise Assessment for Participant ER-20-E
Company Type: Independent and Other ESCOs
Title: Engineering Manager
Question
1a. How many years have you been involved with EPC projects?
1b. How many such projects have you worked on?
1c. How many were MUSH market projects?
1d. Which MUSH sub-markets have you worked in?
1e. Did you have a role in securing financing and/or securing
financial incentives on these projects?
1f. On average, how many building systems were impacted on
each project you worked on (e.g., envelope, lighting, central plant
decentralization, water, mechanical upgrade, etc.)?
1g. Do you hold professional registration as a CEM, CxA, PE,
NCARB/AIA, or LEED AP? If yes, which one(s)?
1h. Do you hold an advanced degree (BS or higher) in
engineering or a related field to building construction, operations,
design, or energy? If yes, what is your highest relevant degree?
1i. Have you ever served as a faculty member in an energy
engineering or management program at an accredited college or
university?
1j. Have you published at least two peer-reviewed publications or
one book or book chapter related to EPC? If yes, how many of
each type?
1k. Have you ever been an invited conference presenter on a
related subject? If yes, how many times?
TOTAL POINTS:
TOTAL CATEGORIES:

369

Response
16
50+
45
MUSH
Yes

Points
1
1
1.5

5

0.5

CEM

3

MSME

2

No

0

No

0

6

0.5

10
8

0.5

APPENDIX C.1

Detailed Analysis of Data Collected for Developing Life Cycle Cost-Based Risk Model in
Delphi Round 1
PART I - RISK TOLERANCE QUESTIONS
KNOWLEDGE CATEGORY 1 – RISK TOLERANCE
The purpose of this question was to classify panelists based on their extant beliefs regarding risk.
This is potentially important in the context of this research, as the literature has shown evidence
of irrational risk behavior among contractors (McKim 1992) and a reliance on information and
heuristics to address project risks (Raftery et al. 2001; Simu 2009) which can lead to cognitive
biases in decision making. The six questions in this part of the survey utilized the TIAA-CREF
Asset Allocation Evaluator (TIAA-CREF 2013), a well-vetted tool that provides insight into
individual attitudes and tolerance toward risk (Cobb and Menas 2009). Table C-1 summarizes
the risk tolerance of the panelists, organized in ascending order of risk tolerance.
Table C-1. Panelists’ Risk Tolerance
a Percent (%)
Risk Tolerance
N
Conservative
3
16.7%
Moderately Conservative
2
11.1%
Moderate
5
27.8%
Moderately Aggressive
6
33.3%
Aggressive
2
11.1%
Notes:
a\ One panelist did not complete the risk profile tool
Risk tolerance was evaluated as a function of various characteristics of the panelists.
Characteristics of interest were identified based on a study of construction managers’ impacts on
risk management (Simu 2009), and included education in a related field, the type of firm in
which employed, years of energy performance contracting (EPC) experience, and project
experience. Comparing proportions between these panelists’ characteristics of interest and risk
tolerance could be difficult due to the small sample size of panelists (n=19), the large number of
outcomes for risk tolerance (5), and the corresponding relatively small number of outcomes for
each characteristic of interest. As a result, risk tolerance outcomes were collapsed from five to
three categories, as shown in Table C-2.
Table C-2. Modified Panelists’ Risk Tolerance Categories
Risk Tolerance
N Percent (%)
Conservative and Moderately Conservative 5 27.8%
Moderate
5 27.8%
Moderately Aggressive and Aggressive
8 44.4%
370

Restructuring the risk tolerance outcomes reveals that panelist attitudes toward risk place them
along a conservative to moderate continuum (55.6%). The single largest category of panelists,
however, was those who are considered moderately aggressive or aggressive. This gives rise to
the question of which other characteristics may be related to specific risk tolerances.
PART II – PROFESSIONAL EXPERIENCE AND DETERMINATION OF EXPERTISE
KNOWLEDGE CATEGORY 2 – NUMBER OF YEARS IN EPC INDUSTRY
1a. How many years have you been involved with EPC projects?
This question sought to elicit the number of years each panel member has been involved with
performing EPC retrofit projects. Consensus was not a goal of this question since the selected
participants had varying amounts of work experience in this field. Additionally, this question
was phrased such that a participant’s cumulative work experience with EPC retrofits was
elicited; this was particularly important in the case of experts who had varied work experience.
For example, some participants conducted EPC retrofit-related work while working for utilities,
engineering firms, or municipalities, universities, schools, and hospitals (MUSH) market
building owners/managers prior to a career with an energy service company (ESCO). This
question was phrased to capture the sum of a participant’s experience in the delivery of EPC
retrofit projects.
Responses were analyzed using equal categories beginning with 0-7 years (8 years or more EPC
experience was one of the threshold criteria for expert identification). Table C-3 contains the
number of years EPC experience held by the panel.
Table C-3. Number of Years in
the EPC Retrofit Industry
Years
N Percent (%)
0-7 Years
4
21%
8-15 Years
6
32%
16-23 Years 6
32%
24-31 Years 2
11%
32+ Years
1
5%
It can be seen that the majority of panel members (64%) have between 8 and 23 years’
experience in conducting EPC retrofit projects. A small number of participants (16%) had greater
than 24 years of experience. This is expected, since the United States ESCO industry developed
in the 1980s (ICF and NAESCO 2007); participants were thus deemed to have a maximum
possible tenure in this industry of between approximately 30 and 35 years. While 21% of
participants had less than the threshold value for years of experience, 100% of these respondents
had at least 5 years of experience.
KNOWLEDGE CATEGORY 3 – NUMBER OF EPC RETROFITS
1b. How many such projects have you worked on?
This question was intended to explore the relationship between an individual’s tenure in the EPC
retrofit industry and the number of projects they have worked on. This was included to explore
the belief that the number of projects EPC experts work on annually may be a reflection of their
role in the EPC retrofit process. Table C-4a shows the total number of projects worked on by the
371

panel and Table C-4b shows the average number of projects work on annually by panel
members.
Table C-4a. Number EPC Retrofits
# of Projects
N
Percent (%)
0-25
4
21%
26-50
5
26%
51-75
2
11%
76-100
4
21%
100+
4
21%

Table C-4b. Number EPC Retrofits Per Year
# of Projects/Year N Percent (%)
1-2
4 21%
2.1-3
3 16%
3.1-4
3 16%
4.1-5
3 16%
5.1+
6 32%

Analysis of Table C-4a reveals that the majority of panelists have completed between 51 and
over 100 projects, with a maximum value of 300 projects. The number of projects panelists
worked on annually centers on values between 2.1 and 5 (48% of panelists); however, 32% of
participants worked on 5.1 or more projects per year, which was the largest single response
category. Data in this category ranged from 7.1 to 20 projects worked annually. An analysis of
this data reveals that the experts in the latter category generally held positions such as sales
manager of an entire sector (e.g., public housing), state and regional market managers, and
individuals with corporate-level responsibility (e.g., senior contract manager, director of
construction services, and energy engineering manager).
The panelist working on the least number of EPC projects had 10 total projects (over 9 years).
That same individual had the lowest number of projects per year at 1.11. This individual was a
construction project manager, which is consistent with the relatively low number of projects per
year, since the construction phase of EPC retrofits can take 6 to 18 months to complete. The
panelist with the largest number of projects reported working on 300 EPC retrofits. The
participant with the largest annual workload reported 20 EPC projects per year. The former
panelist is a statewide market manager, and the latter panelist is a director of construction
services.
KNOWLEDGE CATEGORY 4 – MUSH MARKET EPC RETROFIT EXPERIENCE
1c. How many were MUSH market projects?
In order to determine panelists’ experience specifically with MUSH market EPC projects,
participants were asked to identify the percentage of their total portfolio of work that was
directed toward MUSH market buildings. Table C-5 shows the percentage of panelists’ overall
EPC retrofit work that took place in the MUSH market.
Table C-5. Percentage of EPC Retrofits in the
MUSH Market
Percentage of Projects
N Percent (%)
<80%
2
11%
80-90%
5
26%
91%-95%
3
16%
96-99%
2
11%
100%
7
37%
372

An analysis of Table C-5 reveals that the majority of panelists’ EPC retrofit project portfolios are
dominated by MUSH market work; 64% of participants’ portfolios are comprised of greater than
90% of such projects. This was not unexpected, given the large EPC market share given to
MUSH market projects (Satchwell et al. 2010). The largest single response category indicated
that 37% of panelists have worked exclusively in one or more MUSH sub-markets.
1d. Which MUSH sub-markets have you worked in?
In concert with the previous question, panelists were asked to identify which specific MUCH
sub-markets they have worked in. Results are displayed in Table C-6a.
Table C-6a. Experience with MUSH Sub-Markets
Submarket(s)
N Percent (%)
M Only
1
5%
MS Only
1
5%
MUS Only
4
21%
MUSH
13 68%
The vast majority of panelists (68%) have work experience in all four MUSH sub-markets
(municipalities, universities, K-12 schools, and hospitals), further establishing their qualification
to contribute their expertise to achieve the goal of this research.
Individuals were not asked specifically about their experience with correctional facility projects;
however, 7 out of the 19 panelists offered this information when responding to question 1d.
Those results are shown in Table C-6b.
Table C-6b. Experience with
Corrections Facility EPC Retrofits
Response N
Percent (%)
Yes
6
32%
No
1
5%
Unknown 12
63%
KNOWLEDGE CATEGORY 5 – FINANCIAL AID KNOWLEDGE
1e. Did you have a role in securing financing and/or securing financial incentives on these
projects?
The purpose of this question was the assess panelists’ experience with understanding the role that
utility rebates, grants, and other financial incentives play in understanding the financial aspects
of an EPC project and their contribution to the effective financing of an EPC retrofit. The
importance of financial incentives was raised in the Phase 0 interview, which confirmed earlier
case study findings that revealed the role of utility rebates in helping to finance a broader
technical scope of work in EPC retrofits, particularly in non-energy areas, for three prisons in
Michigan (Berghorn and Vallad 2013). The importance of financial incentives has also been
emphasized in the literature (Hopper et al. 2007; Satchwell et al. 2010; IFC 2011; Stuart et al.
2013). Table C-7 shows panelists’ responses to whether they have experience with financing and
financial incentives for EPC retrofits.
373

As can be seen, the majority (84%) of participants have such experience. Four respondents
indicated that their experience is only with rebates, grants, and/or incentives, and one respondent
indicated that they have experience in this area on approximately 10% of their projects.
Table C-7. Experience with Project
Financing and Financial Incentives
Response N
Percent (%)
Yes
16
84%
No
3
16%
KNOWLEDGE CATEGORY 6 – RETROFIT MEASURES KNOWLEDGE
1f. On average, how many building systems were impacted on each project you worked on
(e.g., envelope, lighting, central plant decentralization, water, mechanical upgrade, etc.)?
The purpose of this question was to elicit knowledge from panelists about their experience with
specific retrofit measures included in the design and execution of EPC projects. The average
number of measures deployed in MUSH market EPC retrofits increased to approximately 5.8 by
2008, with non K-12 school MUSH market projects averaging approximately 5 measures per
project by that same time (Larsen, Goldman, and Satchwell 2012). Retrofit measures and their
frequency of use in “public sector” projects (as defined, analogous to MUSH market) were
identified by Larsen et al. (2012), as shown in Table C-8.
Table C-8. Retrofit Strategies Frequency of Occurrence in MUSH Market Projects
Percent (%)
% Change
1990-2008
Retrofit Measure
1990-1997
1998-2004
2005-2008
Major HVAC
50%
54%
46% -8%
Lighting Only
25%
9%
3% -88%
Minor HVAC
12%
17%
13% 8%
Onsite Generation
5%
8%
11% 120%
Other
5%
4%
3% -40%
Non-Energy
3%
8%
24% 700%
In order to facilitate comparisons across multiple projects and over 150 unique retrofit measures
contained in the NAESCO database, the study used the following definitions of retrofit
measures, which were developed and discussed in previous papers (Goldman et al. 2002; Hopper
et al. 2005):
 Lighting-Only: Various lighting efficiency measures, controls, and strategies.
 Major HVAC: Major HVAC equipment replacement (e.g., boilers, chillers, cooling
towers, HVAC dist. improvements) and may include other HVAC control, highefficiency lighting, and motors measures.
 Minor HVAC: Less-capital intensive HVAC measures and controls (and exclude major
HVAC equipment replacements), and may include lighting and other measures.
 Onsite generation: Installation of onsite generation equipment and may include other
energy efficiency measures (e.g., lighting, HVAC equipment and controls, motor
efficiency measures).

374




Non-energy: Roof or ceiling replacement, asbestos abatement (i.e., measures that are not
installed primarily for their energy savings), and may include other efficiency measures
(e.g., lighting or HVAC upgrades).
Other: All other measures including domestic hot water (DHW), water conservation, and
installation of energy efficient equipment such as vending machines, laundry or office
equipment, high-efficiency refrigeration, industrial process improvements, and strategies
such as staff training or utility tariff negotiation. These individual measures may also be
included in other retrofit strategies (except lighting-only); projects categorized as
‘‘other’’ retrofit strategy only installed these types of measures.

Analysis of previous information highlights the need for domain experts to possess broad
knowledge of a variety of retrofit measures, and have experience with projects that deployed a
wide variety of retrofit measures. The number of retrofit measures deployed on panelists’
projects is given in Table C-9a.
Table C-9a. Number of Retrofit Measures
Deployed on Projects
# of Retrofit Measures N Percent (%)
3-4
5
26%
5-7
9
47%
8-10
4
21%
10+
1
5%
While specific measures were not required as part of this question, several respondents provided
information about ECMs that they typically encounter on EPC retrofits, which is summarized in
table C-9b.
Table C-9b. Retrofit Strategies Deployed by Panelists –
Response Rate for Each Measure
Retrofit Measure
N
Percent (%)
HVAC/Mechanical
5
100%
Lighting
5
100%
Envelope
5
100%
Controls
4
80%
Water
3
60%
Plant Decentralization
3
60%
Vending Machine Optimization 1
20%
Onsite Generation
1
20%
Cogeneration
1
20%
Analysis of the data in table C-9b indicates that panelists have broad expertise and experience
with a number of high- and low-utilization retrofit measures, as described in Table C-8.
Furthermore, this indicated the need to modify the expert-assessment rubric, such that at least
five building systems were impacted on an average project. The selection rubric initially required
experience with an average of three retrofit measures per project.
375

KNOWLEDGE CATEGORY 7 – RELATED CERTIFICATION AND EDUCATION
1g. Do you hold professional registration as a CEM, CxA, PE, NCARB/AIA, or LEED AP?
If yes, which one(s)?
The purpose of this question was to elicit information for the purposes of determining expertise,
following the rubric developed in Chapter 3. A review was conducted of relevant professional
certifications, licensure, and registration that are relevant to the sustainable built environment, by
examining membership rosters for various state chapters of the Energy Services Coalition, as
well as reviewing the Association of Energy Engineers for available relevant certifications.
Question options were structured from this review; however, participants were able to add
additional credentials that they possessed if they deemed them relevant. A summary of panelist
credentials is presented in Table C-10.
Table C-10. Energy-Related Credentials Held by Panelists
Credential
N Percent (%)
Engineer in Training (EIT)
1
5%
LEED Accredited Professional (LEED AP®) 1
5%
Professional Engineer (PE)
2
9%
Certified Energy Manager (CEM)
9
41%
None
9
41%
An analysis of the data in Table C-10 reveals that several panelists hold more than one credential
(N=3) and the most commonly-held professional credential is the CEM (41% of panelists). An
equal number of panelists do not hold relevant credentials. Of those, one panelist holds a
credential relevant to their primary job function (Bar Association certification for the panelist
that serves as a senior contract manager), despite the credential not being specifically energyrelated. Of the remaining eight, seven panelists are in sales and business development functions
and one is a corporate vice president.
1h. Do you hold an advanced degree (BS or higher) in engineering or a related field to
building construction, operations, design, or energy? If yes, what is your highest relevant
degree?
This question further explored the credentials held by panelists in an effort to determine their
expertise. Data related to the highest relevant degree held by panelists is shown in Table C-11.
Table C-11. Highest Relevant Degree Held by Panelists
Degree
N
Percent (%)
BS - Construction Management
1
5%
BS - Engineering Technology
2
11%
BS – Engineering
6
32%
MS – Technology
1
5%
MS – Engineering
3
16%
Juris Doctor
1
5%
No Related Degree
5
26%
The most commonly held highest degrees by panelists were a bachelor’s degree in engineering
(mechanical, aerospace, and architectural) and a master’s degree in engineering (mechanical and
376

energy systems). It should be noted that three panelists with bachelor’s degrees in engineering
also held an MBA. Of the panelists without a related degree, one entered this field through a
trades background obtained first in the U.S. Navy, and over 30 years of experience as a
mechanical tradesperson and supervisor. Of the nine panelists who do not hold a related
professional credential, four hold a related advanced degree.
KNOWLEDGE CATEGORY 8 – PROFESSIONAL SERVICE AND OUTREACH
1i. Have you ever served as a faculty member in an energy engineering or management
program at an accredited college or university?
The purpose of this question was to determine expertise through qualification and experience as
a college or university faculty member, which represents acknowledgement of domain-level
expertise by institutions of higher education that are generally independent and subject to
rigorous accreditation standards. Additionally, this question also has the potential to demonstrate
outreach through the sharing of knowledge by EPC professionals who are serving as an adjunct
faculty member. Results for this question are shown in Table C-12.
Table C-12. Experience as a College
or University Faculty Member
Response
N
Percent (%)
Yes
3
16%
No
16
84%
Of the three panelists with faculty experience, two have served as adjunct faculty in energyrelated programs at community colleges and one served as a full-time faculty member at a
doctoral degree-granting institution.
1j. Have you published at least two peer-reviewed publications or one book or book
chapter related to EPC? If yes, how many of each type?
This question provides additional insight into third-party recognition of domain-level expertise
through acceptance of publications. Table C-13 summarizes panelists’ related publications.
Table C-13. Publications Related to EPC
Response
N
Percent (%)
Yes - Article
4
21%
No
15
79%
While a relatively small number of panelists have published related works, one indicated that
they authored a white paper for the American Council for an Energy-Efficient Economy, a
national non-profit think tank that has produced a sizable body of work related to EPC via
whitepapers and conference papers.
1k. Have you ever been an invited conference presenter on a related subject? If yes, how
many times?
Similarly to the previous question, this one seeks to elicit third-party recognition of domain
expertise through invitations to present on a subject related to EPC. Table C-14 shows panelists’
experience with delivering invited conference presentations.
377

As can be seen from the data, all but one panelist has given invited presentations. That individual
has a lengthy background in construction management; however, they are a more recent entrant
to the EPC domain. The majority of panelists have been invited presenters between one and five
times (42%), with 16% of panelists having given greater than 15 invited presentations.
Table C-14. Invited Presentations on a
Related Subject
# of Presentations N
Percent (%)
0
1
5%
1-5
8
42%
6-10
5
26%
11-15
2
11%
15+
3
16%
PART III – RISK MANAGEMENT PROCESS
KNOWLEDGE CATEGORY 9 – RISK IDENTIFICATION
2a. Do you consider any of these risk categories (Table 1) when developing and executing
energy performance contract projects? If so, which ones?
This question and question 2b were intended to elicit expert knowledge in the identification of
risks that are encountered in EPC retrofit projects. This question provided a series of twelve risk
categories and non-exclusive examples of specific risks associated with each risk category.
These risk categories and example risks were identified through a preliminary interview with an
EPC expert, through a review of the literature on EPC risks, and during the Phase 0 interview.
The literature review and preliminary expert interview led to subdividing the first risk category
(Client Selection categories) into three. These are summarized in Table 2-2. This question,
therefore, gave panelists the ability to respond to pre-identified risk categories and elicited their
expertise about specific example risks and the categories that give rise to the realization of these
risks in EPC retrofits. Question 2b elicited panelists’ expertise in an open-ended format by
providing them the opportunity to identify additional risk categories that were not included in the
initial list.
Consensus was sought among panelists as to whether they consider individual risk categories.
Panelists were asked to rank their frequency of considering each risk category using a sevencategory scale; any response greater than “never” (0% consideration) was logged as an
affirmation that this risk category is identified and considered on EPC projects. That information
is shown in Table C-15.
In explaining the intent of the question to several panelists, consideration of a risk category was
defined as a risk that is identified by the project team, that is mitigated through contractual or
other means, and/or that requires a specific plan of action for its assessment and management. A
summary of the major themes that emerged through the elicitation of knowledge from the
panelists related to each risk category is presented below.

378

Table C-15. Consideration of Individual Risk Categories
Risk Category
N
Percent (%)
1. Client Selection Factors
16 84%
1A. Financial factors
16 84%
1B. Facility/technical factors
18 95%
1C. People factors
18 95%
2. Project Development
18 95%
3. Energy Audit Quality
19 100%
4. Equipment Selection and Installation
19 100%
5. Commissioning
18 95%
6. Operations and Maintenance Practices
19 100%
7. Measurement and Verification of Savings
14 74%
8. Project Management Over the Project Life-Cycle 18 95%
9. Construction-Specific Concerns
17 89%
10. Volatility of Energy Prices
16 84%
2b. Are there any risk categories that you consider that are not included in this list?
The purpose of this question was to elicit expertise in an open-ended fashion; that is, rather than
get reactions to already identified risks; panelists were given the opportunity to add additional
risk categories and individual risks. Responses to this question generally fell into two categories:
1) risks that could be classified under an existing risk category and 2) truly unique risks and risk
categories that had not been previously identified. Many risks falling into the former category
were immediately moved to the appropriate risk category; however, several required further
review before deciding how to most appropriately handle them. These risks have been discussed
with the appropriate risk category in the section above. Risks in the latter category will be
discussed in this section. Risks identified as part of this question that were not obviously part of
one of the first twelve risk categories and their ultimate disposition are provided in Table C-16.
Obtaining consensus among panelists was a goal of this question. Since these were open-ended
responses, and previous panelists were not re-interviewed after a new risk category or risk was
added, consensus was not obtained during Delphi Round 1. Consensus on the identification of
these risks and the newly-identified risk category will be sought during the second round of the
Delphi technique. Despite the inability to seek consensus in the first round of interviews, it is
worth noting that 4 panelists (21%) independently identified political risks as something to be
identified and considered in MUSH market projects.
Perception of the Performance Contracting Industry
Two panelists raised their concern that negative perception of the performance contracting
industry could harm project performance. One panelist stated their belief that about 20% of the
ESCOs operating currently are staffed with knowledgeable professionals and are sincere in
delivering a value solution, while the other 80% are prospecting for business because they think
there are large profits to be had.
Three examples were provided:
1. Companies that “subterfuge” the guarantee if performance is not on track, to ensure they
are not writing a check back to the client. This is often based on disputed changes in
379

occupancy, plug loads, etc. This harms the project directly as well as long-term client
relationships.
Table C-16. Additional Risks Identified by Panelists
Risk
Disposition
Political Risks
Moved to individual risk under "Client
Selection Risks - People Factors"
Productivity Losses in
Moved to individual risk under
Corrections Projects
"Construction Phase Risks"
Safety
Moved to individual risk under
"Construction Phase Risks"
Changing Financial Incentives Moved to individual risk under
"Development Phase Risks"
Cost of Doing Nothing or
Moved to individual risk under
Self-Implementing
"Development Phase Risks"
Public Procedural Risks
Moved to individual risk under
"Development Phase Risks"
Time-Based Risk
Moved to individual risk under
"Development Phase Risks"
Timing
Moved to individual risk under
"Development Phase Risks"
Design Development
Moved to individual risk under "Energy
Audit Quality"
Staff Turnover During Project Moved into individual risk under "Project
Lifecycle
Management Over the Project Lifecycle"
Perception of the Performance Create New Risk Category
Contracting Industry

N
4

Percent (%)
21%

1

5%

1

5%

1

5%

1

5%

1

5%

2

11%

1

5%

1

5%

1

5%

2

11%

2. Some companies move the project very quickly through development when they know
they have a “less sophisticated” client, resulting in a low-yield project, which could have
resulted in much larger benefits for the client if the ESCO invested more time up front.
For the ESCO, this limits project scope and creates projects with lower overall value.
3. Lack of understanding of enabling statutes limits the ability for some ESCOs to deliver
projects that best-meet the client’s needs. For example, a client was concerned about
being able to undertake an EPC retrofit because they were at their debt cap. Several
ESCOs told them that they could not execute a project until they cleared their debt-cap,
meanwhile, state legislation allowed publicly funded entities to finance performance
contracting without counting against the debt cap utilization. This creates the potential for
generally negative views of the industry as a whole.
KNOWLEDGE CATEGORY 10 – MUSH MARKET-SPECIFIC RISKS
2c. Do MUSH market energy performance contract projects have different risk categories
than non-MUSH market projects? If so, please describe those differences.
This question sought to elicit expertise from panelists to assess whether MUSH market EPC
retrofit projects have specific risks that are different or unique from non-MUSH projects (e.g.,
private sector). Larsen et al. (2012) defined private sector projects as consisting of commercial,
380

office, industrial, retail, hotel/hospitality, residential, and other building types; that definition was
used to clarify the question if respondents needed additional information. Table C-17a provides
panelist responses as to whether MUSH market EPC projects have different risks from other
EPC project types.
As can be seen from Table C-17a, the majority of respondents (82%) believed that MUSH
market EPC projects have different risk categories than private sector projects, and therefore,
achieved consensus on this issue. Going deeper, information was elicited from panelists as to
which market is riskier, and whether any specific sub-markets are riskier than others. Table C17b and Table C-17c show the information provided by panelists in this regard. Follow up
questions will be posed in round 2 to further clarify these responses.
Table C-17a. Do MUSH Market EPC Projects
Have Different Risks from non-MUSH Projects?
Response
N
Percent (%)
Yes
14
82%
No
3
18%
Table C-17b. Riskier Market Sectors Identified by Panelists
Perceiving a Different Risks in MUSH Market Projects
Riskier Market(s)
N Percent (%)
MUSH
9
64%
Non-MUSH
3
21%
Non-MUSH, Hospitals, and Corrections 1
7%
Non-MUSH and Corrections
1
7%
Table C-17c. Segments with Greater Risk Identified by
Panelists Identifying MUSH Market as the Riskier Market
Riskier Segment(s)
N Percent (%)
Corrections
8
89%
Hospitals
3
33%
K-12 Schools
2
22%
24/7 Facilities
1
11%
Industrial
1
11%
Data Centers/High Security/Mission Critical 1
11%
KNOWLEDGE CATEGORY 11 – RISK IDENTIFICATION - METHODS
3a. How do you identify risks in these projects? Do you base this on experience, previous
projects, client requirements, case based reasoning, contractual requirements? Do you use
formal or informal methods?
The purpose of this question was to gain insight into the methods typically employed for risk
identification. This assists in determining ESCO’s overall level of organizational maturity, per
Grimaldi et al. (2012). Panelists were asked to provide information about the risk identification
methods they employ on their projects. All panelists reported that they rely on experience to
some degree; however, the way experience is used varied significantly. Twenty percent of
respondents reported utilizing company databases that contain information from previous
381

projects that can be used to help identify risks by examining similar building types, locations,
and ECMs. Five respondents from three different ESCOs reported the use of a formal corporate
risk tool, or guidance document that is essentially a checklist constructed from the totality of
experience with performing EPC retrofits. Two respondents reported specific risk identification
objectives during a multi-stage risk management process. The first panelist indicated a risk
identification process that examines risks related to energy savings calculations and M&V,
design-related risk, and risks arising from cost and constructability concerns. The second panelist
reported a process that begins by identifying risks associated with specific project milestones
then utilizes the collective experience of the project team. Since many of these responses were
open-ended, this question will be asked again in round 2 using methods found in Grimaldi et al.
(2012) and Thaheem and DeMarco (2013).
KNOWLEDGE CATEGORY 12 – RISK IDENTIFICATION - TIMING
3b. At what stage of the project do you identify risks?
100% of respondents indicated that the risk identification process begins early in the project;
60% indicated that the process is iterative and takes place at designated points in time during the
project.
KNOWLEDGE CATEGORY 13 – RISK IDENTIFICATION - RESPONSIBILITY
3c. Who on the project team identifies these risks and performs these functions (e.g., sales
lead, PM, engineer, attorney)?
All panelists indicated that their organizations have designated individuals to manage the risk
process; however, the job functions of these individuals varied. Seventy percent of respondents
indicated that a team approach was used, which in the majority of the cases meant that the team
member tasked with overall responsibility for activities occurring during a given project phase
managed the risk identification process specific to that phase of the work. Generally speaking,
these risk review teams were multi-disciplinary and included the project developer, energy
engineers, the design engineer, the construction manager, assurance engineers (for M&V
concerns), and in some cases, finance and legal staff members. Senior project review (vice
president or above) was indicated by 30% of the panelists, and one panelist mentioned the use of
in-house project inspectors who make random site visits to review risk, among other items.
KNOWLEDGE CATEGORY 14 – RISK EVALUATION METHODS
3d. What method or methods do you use for evaluating the impact of a risk on the project?
As a companion to question 3a, this question also sought to determine the level of organizational
maturity with regard to risk for ESCOs in this study. The general sense from the panel was that
ESCOs do not engage in formal methods of risk evaluation. Sixty percent of the panelists
reported using informal methods or experience-based methods – one member reported relying on
experience as codified in a project database. One panelist reported some use of probabilistic
methods for risk evaluation to include Monte Carlo simulation (via Crystal Ball) and the use of
PERT inputs for schedule-related risks. That individual reported that the greatest barrier to more
widespread use of such methods is the inability to specify probabilistic input ranges for energy
models, which limits the use of such methods. Another panelist reported use of an internallydeveloped tool that evaluates risks as having either “low” or “high” impacts relative to associated
costs. The “high risk” cost value is then used as an input to the project’s contingency if the
project team is unable to identify a feasible mitigation strategy. That panelist went as far as to
382

say that they believed that process is “ineffective;” however, no changes to the risk evaluation
method have been planned. Since many of these responses were open-ended, this question will
be asked again in round 2 using methods found in Grimaldi et al. (2012) and Thaheem and
DeMarco (2013).
KNOWLEDGE CATEGORY 15 – IDENTIFICATION OF MOST IMPORTANT RISK
CATEGORIES
3e. What are the top 3 reasons (Risk Categories from Table 1) you believe an energy
performance contract would not meet the performance guarantee?
The purpose of this question was twofold. First, expertise was elicited from panelists about those
risks that they believe have a significant contribution to not meeting guaranteed performance.
This provided the opportunity to more deeply examine the sources of and controls used to
manage these risks (as elicited by the next two questions). Second, this question was used to
determine which risk categories were deemed by panelists as being most important in EPC
retrofits. These categories would in turn be subjected to further analysis in this research.
Two measures of risk importance were used: (1) quartile analysis of the frequency with which a
risk category was identified in this question and (2) the frequency with which risk categories
were considered by each panelist. Using those two measures yielded the results shown in Table
C-18.
Table C-18. Phase I Delphi Panel Risk Importance Rankings
Consideration
a
a,b
Risk Category
Pareto/Quartile
Frequency
Energy audit quality
1 (10)
4/1 (95%/100%)
ECM selection and installation
2 (8)
1/1 (100%/100%)
Construction-specific concerns
2 (8)
4/1 (95%/100%)
Commissioning
4 (7)
1/1 (100%/100%)
Project development
5 (5)
4/1 (95%/100%)
M&V
5 (5)
1/1 (100%/100%)
Notes:
a\ Numbers in parentheses are the raw values (e.g., non-ranked) for each
variable.
b\ Values are ranks with outliers retained (left of slash) and with outliers
removed (right of slash).
While the quartile analysis and consideration frequency generally yielded results within the same
range, outlier removal from the data set resulted in every one of the risk categories in the upper
two quartiles for risk importance score being tied for the number one ranking. Further
clarification was sought in round 2.
4a. For each of the risk categories identified in the previous question, what can cause these
risks?
The focus of this research was on the evaluation of the most important risk categories, as
determined by question 3e. As a result, risk causes and controls are published herein just for
those risk categories. The complete list for all 12 risk categories is available by contacting the
383

author at gberghorn@gmail.com. Risk causes and controls for the two risk categories selected for
further analysis (IGA quality and ECM selection and installation) are provided in Table C-18.
4b. For each item listed, are there controls implemented that either prevent these risks,
detect negative impacts of these risks if they occur, and/or mitigate these risks? If so, what
is done? Are formal methods used (e.g., status meetings, project risk response audits, risk
response planning, earned value analysis, none, or other)?
The focus of this research was on the evaluation of the most important risk categories, as
determined by question 3e. As a result, risk causes and controls are published herein just for
those risk categories. The complete list for all 12 risk categories is available by contacting the
author at gberghorn@gmail.com. See Table C-19 for risk causes and controls for the two risk
categories selected for further analysis (IGA quality and ECM selection and installation).
Table C-19. Risk Causes and Control Measures for Top Risk Categories
Risk Category
Energy Audit
Quality

Risk Causes
Existing Conditions
 Facility age – code update
 Misunderstanding existing conditions,
such as possible presence of asbestos
Facility Stakeholder Concerns
 Differing stakeholder needs
Inexperience
 Failure to understand facility operations
and EPC goal – lack of EPC experience
Information and Analysis
 Calculation errors
 Conducting the IGA too quickly
 Energy model calibration error
 Inaccurate/incorrect or disputed baseline
 Information availability and accuracy
 Missing information
 Missing risk assessment for each ECM
Project Complexity and Guaranteed Savings
 Establishment of the guarantee amount difficult balance between providing a
large-enough project to generate client
excitement and ESCO value and hedging
on savings
 Overstatement of issues leading to
mismatch with technical needs;
understatement of issues leading to
reduced project value

384

Risk Control Measures
Existing Conditions
 Complete set of facility drawings
 Conduct a code review
Information and Analysis
 Complete set of facility drawings
 Contingency factors
 Identify risks for each ECM being
evaluated and the entire ECM
portfolio
 Include potential but unverified
concerns in the IGA
 Interview all facility staff and backbrief them on the findings to “ground
truth” the audit results and uncover
any missing information
 Third-party internal review based on
historical projects
Project Complexity and Guaranteed
Savings
 Contingency factors
 Third-party external reviews
 Use stipulation and IPMVP Option
A wherever appropriate

Table C-19 (cont’d)
Risk Category
ECM Selection and
Installation

Risk Causes
Contractual Concerns
 Owners and/or project designers and
engineers are used to design-bid-build in
the public sector and are not used to
designing based on fixed budgets and do
not understand EPC cost structures
 Disconnect between the design and audit
intent – improper efficiency target;
different operating schedule implemented
than what was planned; different control
system installed
 Subcontractor quality and reliability
 Unqualified and unsafe contractors create
additional risks
ECM-Specific Issues
 Cheapest equipment sometimes selected –
leads to sub-optimal O&M savings
 ECM package constructability and
feasibility
 ECMs are not aligned with the IGA
findings
 Failure to perform as designed
 Uncertainty in factors used to predict
performance
Facility Factors
 Installation location
Occupant Concerns
 Comfort complaints can add extra work
after construction is complete
 Lighting is difficult to demonstrate before
installation; owner and occupants can be
unhappy with light quality after a retrofit
Project Complexity and Guaranteed Savings
 “Low-hanging fruit has been picked” –
increasing complexity of ECMs
 Lowest-cost solutions may create value
concerns during O&M
Security Concerns (Correctional Facilities)
 Accessibility to inmates/physical security

Risk Control Measures
 Information – equipment-specific
Contractual Concerns
 Assign a team member to align
outsourced design team with internal
energy auditing team
 Coordinate phase handoffs between
project developer and energy
engineer
 Designate a team member to
coordinate contractor and ESCO
team members.
 Pre-qualify subcontractors
 Robust CM practices
 Use ESCO’s own forces for complex
aspects of system design, whenever
possible
 Use EMR ratings to assess
subcontractor safety
ECM-Specific Issues
 Identify risks for each ECM being
evaluated and the entire ECM
portfolio
 In-house design teams for specialty
ECMs (e.g., lighting) that can have
complex design and performance
issues
Project Complexity and Guaranteed
Savings
 In-house design teams for specialty
ECMs that can have complex design
and performance issues

PART IV– LIFE CYCLE COST ANALYSIS
KNOWLEDGE CATEGORY 16 – USE OF LIFE CYCLE COST ANALYSIS
5a. How do you evaluate potential energy conservation measures individually and as a
portfolio for inclusion in an energy performance contract? Do you use life cycle cost
analysis? If so, how do you define parameters (discount rate, inflation rate, energy costs)?
If not, why not? What performance periods are typically used?
Generally all panelists reported using life cycle cost analysis (LCCA) to achieve various project
objectives during retrofit design. Reasons for including LCCA included the ESCO providing
operations and maintenance (O&M) service during the life of the contract, the fact that some
385

projects require repair and replacement costs to be included, and as a tool to educate clients
about the impact that required reinvested costs (e.g., maintenance) have on overall project
performance. One panelist reported using LCCA in only approximately 20% of projects, which
are owned by the State of Washington, which requires the use of life cycle costing for energy
projects. Other panelists reported using LCCA to analyze the impact of deferred maintenance
related to the installation of new equipment, typically involving stipulated escalation rates, which
must be mutually agreed upon by parties before the contract is signed. One panelist, an
equipment manufacturer ESCO, reported that in addition to the traditional LCCA, an internallyfocused analysis is conducted which focuses on the project’s selling price, the project markup,
the use of their own branded equipment, ongoing services to be provided during the project
performance period, whether or not continuous commissioning will be used, and the amount and
duration of measurement and verification (M&V) to be included.
Two panelists specifically reported limited use of LCCA during their projects’ design. The
panelists who reported using LCCA on 20% of projects, those owned by the State of
Washington, stated that the other 80% of clients prefer the use of simple payback period (SPP).
Since the ESCO generally does not install ECMs with less than a five year service life, the
panelist reported a degree of comfort relying on the SPP. The second panelist reported that their
firm does not use LCCA since they work for public agencies that are not making decisions based
on net present value or return on investment. This participant reported that he considers the
inflation of energy costs and matches it with the lease purchase payment in order to analyze
project cash flow, and since equipment replacement horizons are typically longer than project
performance periods, life cycle costs are less relevant and less of a concern. This was an
anomalous response, and similar information was not elicited from any other experts.
Panelists reached consensus on typical contractual lengths of between 10 and 20 years, with 1215 years being the most common. One panelist echoed information elicited from other
participants earlier in the questionnaire, stating that shorter payback periods tend to result in
projects with higher risks, because public owners typically want to maximize the scope of work
for their retrofits, which means the ESCO typically needs a longer payback period over which to
extend ECM costs.
KNOWLEDGE CATEGORY 17 – UNCERTAINTY IN LIFE CYCLE COST ANALYSIS
5b. Do you consider future uncertainty of life cycle cost analysis model parameters
(discount rate, inflation rate, energy costs)? Uncertainty of project factors (cost growth,
schedule growth, identified risk categories)?
Uncertainty was cited as an important factor to consider by several panelists. From an equipment
perspective, future degradation must be considered and ESCOs must predict this “as best as
possible.” Equipment life cycle performance tables developed by ASHRAE and manufacturers’
product data are typically used to help assess the likelihood of equipment failure and degraded
performance. This reduced performance can also lead to degraded energy savings if not detected
and mitigated in an appropriate amount of time. One panelist reported using Monte Carlo
analysis to assist in developing a “hedge” value against cumulative project risks; however, he did
not report the use of probabilistic functions in other aspects of the LCCA. Based on expertise
elicited from panelists, the selected risk evaluation strategy must:
 Include probabilistic functions; and
386



Incorporate uncertainty in other aspects of the analysis to help provide greater insight into
potential project risks.

KNOWLEDGE CATEGORY 18 – INDIRECT COSTS IN LIFE CYCLE COST
ANALYSIS
5c. Do you build sunk costs (e.g., project management during M&V, inclusion of costs for
security/escort staff, some NEB-related work) and loss leaders (e.g., additional no-fee
services, some NEB-related work) into these analyses? On what basis do you include values
for these elements?
Indirect costs identified by panelists included project management costs, overhead costs, and
unrecoverable costs related to the procurement and sales process. Panelists agreed that these
costs are generally included in their financial analysis of projects, and typically results in their
distribution over the total project cost. Typically in the MUSH market, these costs must be
factored up-front because they cannot be addressed after the contract has been signed (i.e., no
opportunity for change orders on EPC projects). Many of these costs can also be negotiated
during the development phase.
KNOWLEDGE CATEGORY 19 – NON-ENERGY BENEFITS IN LIFE CYCLE COST
ANALYSIS
5d. Are non-energy benefits included in this analysis (e.g., water conservation, O&M
savings, avoided capital costs, tradable emissions credits, improved health/wellbeing/productivity)? If so, please describe. If not, why not?
Panelists reported a great deal of interest in non-energy benefits (NEBs), both from clients and as
part of their business process. A significant driver of NEB inclusion is enabling legislation that
can determine whether they can be used, and if so, what options are authorized. The majority of
panelists include water conservation measures in their projects; however, they agreed that
because this is based on a measurable utility, these retrofit measures are generally treated as an
energy-related ECM. Other commonly-referenced NEBs included O&M savings, operational
savings, capital cost avoidance, waste management savings, daylighting controls, and building
renovations. Improved indoor air quality resulting from mechanical system upgrades in underventilated buildings was recognized as an NEB used on projects by one panelist; however, the
participant indicated that this can actually result in increased energy consumption over the
baseline, due to air change rate and airflow volume enhancements.
Panelists generally agreed that savings resulting from NEBs would only be included if they could
be stipulated, based on an agreement between the ESCO and client. This was connected to the
level of risk incurred by the ESCO, with one respondent stating they were “not going to have any
risk associated with building it [the retrofit project]."

387

APPENDIX C.2

Detailed Analysis of Data Collected for Developing Life Cycle Cost-Based Risk Model in
Delphi Round 2
PART II – PROFESSIONAL EXPERIENCE AND DETERMINATION OF EXPERTISE
KNOWLEDGE CATEGORY 2b – EPC PROJECT PHASES
2. Please select the EPC retrofit project phase or phases you most frequently support:
This question was asked in order to provide additional panelist background information. This
was used to detect any potential relationships between panelist background characteristics and
risk identification responses. Results are provided in Table CC-1. Panelist experience was
heavily concentrated in earlier project phases (e.g., pre-execution).
Table CC-1. Project Phases Most Frequently Supported
a
Project Phase
Responses
Project Development
Energy Audit
Retrofit Design
Project Execution (Construction and Commissioning)
Energy Savings (O&M and M&V)
b

13 (86.7%)
8 (53.3%)
7 (46.7%)
3 (20.0%)
5 (33.3%)
4 (26.7%)

Other (please specify)
Notes:
a\ Panelists could select multiple phases to adequately describe which
project elements they routinely worked on. As a result, the sum of
responses was greater than the number of individual respondents.
b\ Responses of “other” included:
 Sales qualification
 Contract execution – negotiation and contract development
 Utility rebate program support
 Project selling and negotiating
PART III – RISK MANAGEMENT PROCESS
KNOWLEDGE CATEGORY 9 – RISK IDENTIFICATION
3. In the first survey, 11 participants identified a total of 10 new risks that were added to
matrix that you responded to. The number in parentheses indicates the number of panelists
identifying each new risk. Please indicate below whether you concur with the addition of
each risk. You may refer to the risk matrix in the document that was attached to the email
you received to see all of the other risk categories and individual risks. A comment field is
available if you wish to add your thoughts.

388

Since consensus was not achieved for this question in the first Delphi round, panelists were
provided with a list of the newly identified risks and asked whether or not they agreed with its
inclusion – results are shown in Table CC-2.
Table CC-2. Additional Risks Identified by Panelists

Risk
Political Risks
Productivity Losses in
Corrections Projects
Safety
Changing Financial
Incentives
Cost of Doing Nothing or
Self-Implementing
Public Procedural Risks
Time-Based Risk
Timing
Design Development
Staff Turnover During
Project Lifecycle
Perception of the
Performance Contracting
Industry

Disposition
Moved to individual risk
under "Client Selection
Risks - People Factors"
Moved to individual risk
under "Construction Phase
Risks"
Moved to individual risk
under "Development Phase
Risks"

Delphi Consensus Status
1st
2nd Round
Round
N %
%
Consensus
4 21% 100% Achieved
1

5%

87%

Achieved

1
1

5%
5%

60%
80%

Not Achieved
Achieved

1

5%

80%

Achieved

1
2
1
1

5%
11%
5%
5%

73%
73%
73%
67%

Achieved
Achieved
Achieved
Not Achieved

5%

80%

Achieved

11%

100%

Achieved

Moved to individual risk
under "Energy Audit
Quality"
Moved into individual risk
1
under "Project Management
Over the Project Life Cycle"
Create New Risk Category
2

Consensus was achieved for all but two risks, thus excluding them from further analysis.
4. Two participants identified a risk which did not fall under any existing risk category.
The new risk category was identified as "Perception of the Performance Contracting
Industry" and included two concerns:
 Lack of knowledge regarding EPC enabling statutes by the ESCO limits some
customers’ understanding of how they can use EPC.
 Unethical behavior negatively impacts the industry as a whole.
Please rank the frequency with which you consider this risk on your projects. As with the
first interview, consideration means that you address these risks contractually, by project
management, through technical means, or via financial assurance.
100% of panelists agreed with the inclusion of this as a new risk category.
389

KNOWLEDGE CATEGORY 10 – MUSH MARKET-SPECIFIC RISKS
5. 78% of panelists believe that MUSH market EPC retrofits have a higher level of project
risk than commercial and industrial sector EPC retrofits. Panelists provided five MUSH
market facility types that they believed had higher levels of risk than other buildings in the
MUSH market. Which facility types do you believe have higher levels of project risk (select
all that apply)?
As a result of the open-ended responses obtained from several participants during the first round,
in the second Delphi round panelists were asked to identify which MUSH market facility types
had the greatest project risk profiles. Facility types included correctional facilities, hospitals, K12 schools, mission-critical facilities, and continuously operated facilities; the latter facility type
was inclusive of correctional facilities and hospitals, which operate continuously by definition.
The percentage of responses for each facility type was 67%, 47%, 13%, 53%, and 47%,
respectively. When accounting for respondents who did not identify correctional facilities, but
did identify the inclusive category of continuously operated facilities, the response frequency
increased from 67% to 87%. As a result, it is reasonable to conclude that MUSH market EPC
retrofits in general, and correctional facility projects specifically, have a higher project risk
profile than other markets and building and types.
KNOWLEDGE CATEGORY 15 – IDENTIFICATION OF MOST IMPORTANT RISK
CATEGORIES
6. In the first survey, you were asked to identify the three most important risk categories
you consider when developing and implementing MUSH market EPC retrofit projects.
Based on your importance scoring, the top 6 risk categories are listed below. Importance
scores are in parentheses next to each risk category. Please rank these from 1 to 6, with a
score of 1 indicating the most important risk category to consider (as one panelist stated,
"the things that keep me awake at night when I have a project underway") and a score of 6
indicating the least important among these 6.
A third data column was added to Table C-18, representing the Delphi round 2 responses to this
question, and is presented below as Table CC-3. Rank order agreement across two or more
categories was deemed as consensus that a risk category received an equivalent risk importance
score. As a result of this, four risk categories achieved consensus (energy audit quality, ECM
selection and installation, commissioning, and M&V), while two did not (construction specific
concerns and project development).
Table CC-3. Risk Importance Ranking – Round 1 and Round 2 Delphi Panel Data
Consideration
b
Risk Category
Pareto/Quartile
Delphi Round 2
Frequency
Energy audit quality
1 (10)
4/1 (95%/100%)
1 (2.4)
ECM selection and installation
2 (8)
1/1 (100%/100%)
2 (3.0)
Construction-specific concerns
2 (8)
4/1 (95%/100%)
6 (3.9)
Commissioning
4 (7)
1/1 (100%/100%)
4 (3.6)
Project development
5 (5)
4/1 (95%/100%)
3 (3.1)
M&V
5 (5)
1/1 (100%/100%)
5 (3.7)
Notes:
a\ Numbers in parentheses are the raw values (e.g., non-ranked) for each variable.
b\ Ranks with outliers retained - left of the slash; ranks with outliers removed - right of the slash.
390

KNOWLEDGE CATEGORY 11 – RISK IDENTIFICATION - METHODS
7. Please select which method or methods you use to identify risks during projects.
This question was follow-up from open-ended responses in round 1 of the Delphi survey.
Participants were provided with a list of risk identification measures described by Grimaldi et al.
(2012) and Thaheem and DeMarco (2013) in order to provide better-defined response categories.
The most frequently cited risk identification methods were expertise/expert judgement;
brainstorming; checklist; and risk mapping, risk matrix, probability and impact matrix (Table
CC-4). This placed the majority of respondent firms as being in either the “novice,”
“normalized,” or “natural” categories of corporate maturity toward risk management (Grimaldi et
al. 2012). A natural level of maturity is described as having integration between project
management and risk management.
Table CC-4. Risk Identification Methods Used by Delphi Panelists
Risk Identification Method
Responses
Brainstorming
11
Cause and Effect Diagram/Ishikawa Diagram
Checklist
Delphi
Expertise/Expert Judgement
Failure Mode and Effects Analysis (FMEA)
Interviewing
Monte Carlo Simulation
Risk Mapping, Risk Matrix, Probability and Impact
Matrix
Root Cause Analysis
Sensitivity Analysis
Strengths, Weaknesses, Opportunities, and Threats
(SWOT)
None
Internal Tool

1
11
0
12
2
6
0
7
1
4
4
2
1

KNOWLEDGE CATEGORY 14 – RISK EVALUATION - METHODS
8. Please select which method or methods you use to analyze risks during projects. Risk
analysis is defined as determining the level of impact that a risk has on the project.
This question was follow-up from open-ended responses in round 1 of the Delphi survey.
Participants were provided with a list of risk analysis and evaluation measures described by
Grimaldi et al. (2012) and Thaheem and DeMarco (2013) in order to provide better-defined
response categories. The most frequently cited risk analysis and evaluation methods were
expertise/expert judgement; brainstorming; checklist; and risk mapping, risk matrix, probability
and impact matrix, which was identical to responses for risk identification (Table CC-5). Since
firms undertaking risk analysis and evaluation are not considered to be “novice” with regard to
risk management, the majority of respondent firms were classified as either “normalized” or
natural (Grimaldi et al. 2012).

391

Table CC-5. Risk Evaluation Methods Used by Delphi Panelists
Risk Evaluation Method
Responses
Brainstorming
9
Cause and Effect Diagram/Ishikawa Diagram
1
Checklist
8
Decision Tree Analysis
3
Delphi
0
Expected Monetary Value
6
Expertise/Expert Judgement
9
Failure Mode and Effects Analysis (FMEA)
2
Interviewing
5
Monte Carlo Simulation
1
Risk Mapping, Risk Matrix, Probability and Impact Matrix
8
Strengths, Weaknesses, Opportunities, and Threats (SWOT)
None
Internal Tool

4
2
1

As a result of the expertise elicited from ESCO professionals, the risk evaluation method
selected for this research must meet the following objectives:
 Include expertise;
 Incorporate costs related to project risks as a consistent framework for evaluation; and
 Utilize probabilistic functions, where possible.

392

APPENDIX D

SFMEA Panel Results

Occurrence, Severity, and Detection Rank Tables
Table D-1. Failure Mode Occurrence Rank Table
Likelihood of Failure Criteria: Occurrence of Causes
Very High
90+ in 100 projects
80 in 100 projects
High
70 in 100 projects
60 in 100 projects
50 in100 projects
Moderate
40 in 100 projects
30 in 100 projects
20 in 100 projects
Low
10 in 100 projects
Very Low
<1 in 100 projects

Rank
10
9
8
7
6
5
4
3
2
1

Table D-2. Failure Mode Severity Rank Table
Effect
Criteria: Severity of Effect on Project
Failure to Meet Safety or
May endanger lives of personnel.
Regulatory Requirements
May singularly lead to failure to achieve performance
guarantee or endanger safety of personnel.
Requires changes to the scope of work based on
Major Project Impact
client requirements and project goals; may endanger
future working relationship.
Requires recalculation of energy guarantee.
Requires recalculation of key IGA elements/findings
or similar.
Inconvenience to ESCO staff and client staff time or
Moderate Project Impact
after execution phase.
Inconvenience to ESCO staff time in or after
execution phase.
Slight inconvenience to ESCO staff and client staff
time in earliest project phases.
Minor Project Impact
Slight inconvenience to ESCO staff time in earliest
project phases.
No effect
No effect.

393

Rank
10
9
8
7
6
5
4
3
2
1

Table D-3. Failure Mode Likelihood Rank Table
Detection Likelihood and
Early Project Phase Timing
Criteria: Likelihood and Timing of Detection
Almost impossible to detect
Cannot detect until after energy guarantee and price
until execution phase
guarantee have been fixed by ESCO.
commences
Very remote chance of detection Slight chance of detection before energy guarantee
before execution phase
and price guarantee have been fixed by ESCO, but
only through significant additional scope completed
by ESCO.
Remote
Failure mode may be detected before entering
execution through past project experience.
Very low
Failure mode may be detected before entering
execution through modeled results.
Low
Failure mode may be detected before entering
execution through a combination of visual inspection,
laboratory testing, or direct measurement/observation.
Moderate
Failure mode is detected before entering execution
through past project experience.
Moderately high
Failure mode is detected before entering execution
through modeled results.
High
Failure mode is detected before entering execution
through a combination of visual inspection, laboratory
testing, or direct measurement/observation.
Very high
Design solutions have been highly correlated with
early detection ability in previous projects of a similar
type.
Almost certain
Failure mode cannot occur because it is addressed
fully through design solutions.

394

Rank
10
9

8
7
6
5
4
3
2
1

SFMEA Worksheets

1A4
1A5
1A6
1A7
1A8

9

1.0

10

1.0

9

0.8

9

0.8

10

0.8

9

0.8

10

395

End Effect

Cost

Remove/
Modify

7

7

490

1

1

Manage in
Place

9

5

405

0.0

0.0

Remove/
Modify

7

4

280

0.3

0.3

Manage in
Place

9

5

405

0.7

0.7

Remove/
Modify

8

3

216

1

1

Manage in
Place

7

5

350

0.0

0.0

Remove/
Modify

8

3

216

0.4

0.4

Manage in
Place

7

3

210

0.6

0.6

RPN

1.0

Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions

Occurrence
10

Intermediate
Effect 3

% Likelihood
End Effect

1A3

1.0

Intermediate
Effect 2

Detection

1A2

Misunderstanding
Existing
Conditions

Intermediate
Effect 1
Presence of
AsbestosContaining
Materials (ACM)
Presence of
AsbestosContaining
Materials (ACM)
Presence of
Lead-Based
Paint
Presence of
Lead-Based
Paint
Presence of FuelRelated
Contaminants
Presence of FuelRelated
Contaminants
Presence of
Buried
Infrastructure
Presence of
Buried
Infrastructure

Severity

1A1

Cause

%Likelihood
Cause

Table D-4. Panelist Consensus SFMEA Worksheet

9

0.8

9

0.78

9

0.2

9

Universal Access
(ADA) Upgrades

1.0

8

0.95

8

1C1

Overstatement of
Issues

0.28

9

1C2

Overstatement of
Issues

0.28

8

1B2
1B3
1B4
1B5
1B6

OSHA-Related
Upgrades
SpecialtyStandard
Upgrades
Improper
Equipment
Sizing
Improper
Equipment
Repurposing

Inability to Meet
Performance
Expectations

396

End Effect
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Savings
Shortfall/
Erosion of
ESCO's
Margin
Difficulty
Getting
Financing

% Likelihood
End Effect

0.9

1B7

1B1

Intermediate
Effect 3

Cost

8

3

216

0.75

0.75

7

5

315

0.9

0.9

7

3

189

1

1

7

5

315

0.73

0.73

9

7

567

0.1

0.1

8

4

256

0.55

0.55

7

4

224

0.33

0.33

7

5

315

0.8

0.8

7

5

280

0.28

0.28

RPN

9

Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements

Intermediate
Effect 2

Detection

0.73

Intermediate
Effect 1
Fire Suppression
and Alarm
System
Upgrades
Electrical
System
Upgrades
Plumbing
System
Upgrades
Mechanical
System
Upgrades

Severity

Occurrence

Cause

%Likelihood
Cause

Table D-4 (cont’d)

1C6

Understatement of
Issues

1C7

Understatement of
Issues

1C8

Understatement of
Issues

0.53

0.53

0.53

0.53

9

CustomerSpecified Issues

Changes to the
Size of the
Project Scope

Inability to Meet
Performance
Expectations

7

Improper
Equipment
Sizing

9

Improper
Equipment
Repurposing

Inability to Meet
Performance
Expectations
Inaccurate
Prediction of
Service
Life/O&M
Schedule

Inability to Meet
Performance
Expectations

9

CustomerSpecified Issues

Changes to the
Size of the
Project Scope

Inability to Meet
Performance
Expectations

7

Missed
Opportunities

Inability to Meet
Performance
Expectations

397

Cost

7

4

252

0.23

0.23

7

4

252

0.55

0.55

8

6

336

0.5

0.5

7

6

378

0.25

0.25

7

4

252

0.55

0.55

8

4

224

0.6

0.6

RPN

9

Inability to Meet
Performance
Expectations

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

1C5

Understatement of
Issues

0.28

Intermediate
Effect 3

Improper
Equipment
Repurposing

Intermediate
Effect 2
Inaccurate
Prediction of
Service
Life/O&M
Schedule

Detection

1C4

Overstatement of
Issues

0.28

Intermediate
Effect 1

Severity

1C3

Overstatement of
Issues

Occurrence

Cause

%Likelihood
Cause

Table D-4 (cont’d)

1D4

ESCO Lack of
Experience with
Facility Type

1D5

ESCO Lack of
Experience with
Facility Type

1D6

ESCO Lack of
Experience with
Facility Type

0.6

0.6

0.6

0.6

9

Failure to
Understand
Facility
Operations and
Client Goals

7

7

Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals

Overestimate
Client Capacity
to Perform O&M

End Effect

Cost

Missed
Opportunity
for O&M
Contract

9

5

360

0.4

0.4

7

4

224

0.45

0.45

8

6

384

0.23

0.23

8

6

432

0.33

0.33

7

5

245

0.45

0.45

7

5

245

0.6

0.6

Overestimate
Client Capacity
to Perform O&M

Reduced Savings
Due to PoorlyPerformed O&M

Overestimate
Client Capacity
to Perform O&M
Lack of
Foresight About
Future Building
Use and
Occupant
Changes
Lack of
Foresight About
Future Building
Use and
Occupant
Changes

Reduced Savings
Due to PoorlyPerformed O&M

Reduced
Project Value
for ESCO
Savings
Shortfall/
Erosion of
ESCO's
Margin

Improper Baseline

Savings
Shortfall/
Erosion of
ESCO's
Margin

Improper
Construction
Timing and
Phasing

398

Improper Retrofit
Design

Project Schedule
too Short

Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

RPN

8

8

Intermediate
Effect 3

% Likelihood
End Effect

1D3

ESCO Lack of
Experience with
Facility Type

0.6

8

Intermediate
Effect 2

Detection

1D2

ESCO Lack of
Experience with
Facility Type

0.6

Intermediate
Effect 1
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals

Severity

1D1

ESCO Lack of
Experience with
Facility Type

Occurrence

Cause

%Likelihood
Cause

Table D-4 (cont’d)

1D10

ESCO Lack of
Experience with
Facility Type

1D11

ESCO Lack of
Experience with
Facility Type

0.6

0.6

0.6

4

5

6

6

Overestimate
Productivity
Rates

Project Schedule
too Short

Overestimate
Productivity
Rates

Underestimate
Project Costs

Improper
Consideration of
Security
Concerns

Project Schedule
too Short

Improper
Consideration of
Security
Concerns

Underestimate
Project Costs

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

Improper
Consideration of
Security
Concerns

Occupant Use of
Equipment for
Other Purposes

Legal
Impacts

399

Cost

9

5

225

0.6

0.6

8

7

224

0.75

0.75

8

7

280

0.48

0.48

9

8

432

0.45

0.45

8

7

336

0.16

0.16

RPN

Intermediate
Effect 3

% Likelihood
End Effect

1D9

ESCO Lack of
Experience with
Facility Type

0.6

5

Intermediate
Effect 2

Detection

1D8

ESCO Lack of
Experience with
Facility Type

0.6

Intermediate
Effect 1
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals

Severity

1D7

ESCO Lack of
Experience with
Facility Type

Occurrence

Cause

%Likelihood
Cause

Table D-4 (cont’d)

1E4

Differing
Stakeholder
Needs

1E5

Differing
Stakeholder
Needs

0.52

0.52

0.52

8

Retrofit Design
Does Not Satisfy
All Stakeholders

User Modification
of
ECMs/Override
Operating
Parameters

6

Codes and
Standards
Upgrades

Retrofit Design
Does Not Satisfy
All Stakeholders

Comfort-Related
Callbacks after
Construction
Complete

7

Codes and
Standards
Upgrades

Increased Project
Complexity

Controls Scope of
Work Lacks
Adequate Detail
and Specificity

Increased Project
Complexity

Retrofit Design
Does Not Satisfy
All Stakeholders

Comfort-Related
Callbacks after
Construction
Complete

Increased Project
Complexity

Controls Scope
of Work Lacks
Adequate Detail
and Specificity

7

7

400

End Effect

Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

Cost

7

4

224

0.65

0.65

8

6

288

0.4

0.4

7

4

196

0.4

0.4

6

3

126

0.65

0.65

7

8

392

0.6

0.6

RPN

Intermediate
Effect 3

% Likelihood
End Effect

1E3

Differing
Stakeholder
Needs

0.52

Intermediate
Effect 2

Detection

1E2

Differing
Stakeholder
Needs

0.52

Intermediate
Effect 1
Inconsistent
Information
About Facility
Operating
Parameters (e.g.,
too hot, too cold,
too much
airflow, not
enough airflow)

Severity

1E1

Differing
Stakeholder
Needs

Occurrence

Cause

%Likelihood
Cause

Table D-4 (cont’d)

0.53

9

Miss Critical
Functions and
Savings
Opportunities

8

Miscount
Existing
Equipment (e.g.,
Lights)

6

Miscount
Existing
Equipment (e.g.,
Lights)

Energy Model
Calibration
Errors

5

Inaccurate
Baseline

Energy Model
Calibration
Errors
Energy Model
Calibration
Errors

1F3

IGA Conducted
Too Quickly

1F4

IGA Conducted
Too Quickly

1F5

IGA Conducted
Too Quickly

0.53

9

Missing Facility
Information

1F6

IGA Conducted
Too Quickly

0.53

6

Missing Facility
Information

0.53

0.53

401

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

Cost

5

5

225

0.85

0.85

8

4

256

0.6

0.6

6

6

216

0.75

0.75

8

4

160

0.85

0.85

8

8

576

0.9

0.9

5

5

150

0.9

0.9

RPN

Intermediate
Effect 3

% Likelihood
End Effect

IGA Conducted
Too Quickly

0.53

Intermediate
Effect 2

Detection

1F2

IGA Conducted
Too Quickly

Intermediate
Effect 1

Severity

1F1

Occurrence

Cause

%Likelihood
Cause

Table D-4 (cont’d)

9

1F8

Facility
Information
Unavailable/
Inaccurate

0.73

6

2A1

Security Concerns

0.8

4

2A2

Security Concerns

0.8

7

2A3

Security Concerns

0.8

6

2A4

Security Concerns

0.8

7

2A5

Security Concerns

0.8

7

Energy Model
Calibration
Errors

Missing Facility
Information
Technology
Breakage/
Damage
Possibility by
Occupants
Technology
Breakage/
Damage
Possibility by
Occupants
Technology
Breakage/
Damage
Possibility by
Occupants
Technology
Breakage/
Damage
Possibility by
Occupants
Occupant Use of
Equipment for
Other Purposes

Change in
Design Intent for
Impacted
Equipment

Rework

Change in
Design Intent for
Impacted
Equipment

Equipment
Impacts

Total Costs of
Operation

Project Schedule
Impacted

Delay in Accruing
Savings

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts

402

Cost

8

8

576

0.7

0.7

5

5

150

0.7

0.7

7

6

168

0.7

0.7

5

6

210

0.7

0.7

7

6

252

0.48

0.48

7

6

294

0.15

0.15

8

6

336

0.1

0.1

RPN

Inaccurate
Baseline

Intermediate
Effect 3

% Likelihood
End Effect

0.73

Intermediate
Effect 2

Detection

1F7

Facility
Information
Unavailable/
Inaccurate

Intermediate
Effect 1

Severity

Occurrence

Cause

%Likelihood
Cause

Table D-4 (cont’d)

2CD1

2CD2

2CD3

0.8

0.85

0.85

0.85

Increase in
Project
Complexity

Limited/Less
Attractive Scope

Increased
Opportunity for
Diminished
Savings

7

Increase in
Project Costs

IGA Scope
Increase
(No Additional
Fee)

8

Less Opportunity
for Non-Energy
Benefits (NEBs)

Project Schedule
Impacted

3

Failure to Meet
Owner Project
Requirements

Rework

7

Failure to Meet
Owner Project
Requirements

Comfort
Complaints

6

Failure to Meet
Owner Project
Requirements

7

403

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

Cost

5

4

140

0.8

0.8

5

4

140

0.8

0.8

5

7

280

0.8

0.8

6

8

144

0.68

0.68

6

5

210

0.55

0.55

5

5

150

0.68

0.68

RPN

Intermediate
Effect 3

% Likelihood
End Effect

2B3

Reduced
Availability of
“Low Hanging
Fruit”
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation

0.8

Intermediate
Effect 2

Detection

2B2

Reduced
Availability of
“Low Hanging
Fruit”

0.8

Intermediate
Effect 1

Severity

2B1

Reduced
Availability of
“Low Hanging
Fruit”

Occurrence

Cause

%Likelihood
Cause

Table D-4 (cont’d)

2CD4

Operational
Concerns

0.55

7

2CD5

Safety Concerns

0.35

5

Failure to Meet
Owner Project
Requirements
Failure to Meet
Owner Project
Requirements

2E1

Unqualified
Subcontractors

0.4

4

Rework

2E2

Unqualified
Subcontractors

0.4

4

2E3

Unqualified
Subcontractors

0.4

7

Unsafe Working
Conditions/Injuri
es
Unsafe Working
Conditions/Injuri
es

2E4

Unreliable
Subcontractors

0.3

8

Rework

Project Schedule
Impacted

Delay in Accruing
Savings

2E5

Unreliable
Subcontractors

4

Unsafe Working
Conditions/Injuri
es

Safety Incident

Project Schedule
Impacted

0.25

Reduce Project
Scope
Safety Incident

Project Schedule
Impacted

Delay in Accruing
Savings

Safety Incident

Project Schedule
Impacted

Safety Incident

404

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Intermediate
Effect 3

Cost

6

7

294

0.45

0.45

5

5

125

0.15

0.15

6

4

96

0.65

0.65

7

6

168

0.25

0.25

7

5

245

0.15

0.15

6

6

288

0.4

0.4

6

4

96

0.15

0.15

RPN

Intermediate
Effect 2

Detection

Intermediate
Effect 1

Severity

Occurrence

Cause

%Likelihood
Cause

Table D-4 (cont’d)

2F3

2G1

2G2

0.75

0.75

3

Project Schedule
Impacted

3

Require ESCO to
use Fixed
Overhead &
Profit Rates

Overstate Project
Soft Costs

4

Divide
Procurement into
Separate Phases

Disconnect
Between IGA
and Retrofit
Design
Disconnect
Between IGA
and Retrofit
Design

0.75

3

Incorrect Vendor
Selection

0.5

6

Increased O&M
Cost

5

Reduced
Equipment
Service Life

0.5

405

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

Cost

6

5

90

0.5

0.5

6

4

72

0.75

0.75

4

5

80

0.48

0.48

4

5

60

0.78

0.78

7

5

210

0.5

0.5

2

8

80

0.33

0.33

RPN

Intermediate
Effect 3

% Likelihood
End Effect

2F2

0.3

Intermediate
Effect 2

Detection

2F1

Unreliable
Subcontractors
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement
Select Cheapest
Available
Equipment to
Meet
Specifications
Select Cheapest
Available
Equipment to
Meet
Specifications

Intermediate
Effect 1

Severity

2E6

Occurrence

Cause

%Likelihood
Cause

Table D-4 (cont’d)

2G4

2G5

ECM
Constructability
& Feasibility
Issues

0.45

3

Uncertainty in
Factors Used to
Predict
Performance

2G6

ECM
Constructability
& Feasibility
Issues

0.45

2

Rework

2G7

ECM Failure to
Perform as
Designed

0.24

10

Reduced Energy
Performance

2G8

ECM Failure to
Perform as
Designed

0.24

9

Rework

2G3

0.28

6

0.28

3

Reduced Energy
Performance

Reduced Energy
Performance

406

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Reduced
Equipment
Service Life

Uncertainty in
Factors Used to
Predict
Performance

Intermediate
Effect 3

Cost

6

4

144

0.18

0.18

5

3

45

0.48

0.48

3

4

36

0.45

0.45

2

3

12

0.2

0.2

5

6

300

0.48

0.48

4

4

144

0.2

0.2

RPN

Intermediate
Effect 2

Detection

Intermediate
Effect 1

Cause
Actual Climate
Conditions
Outside Range of
Planned
Conditions
Actual Climate
Conditions
Outside Range of
Planned
Conditions

Severity

Occurrence

%Likelihood
Cause

Table D-4 (cont’d)

1A5
1A6
1A7
1A8
1B1
1B2

% Likelihood
End Effect

1A4

RPN

1A3

Detection

1A2

Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements

9

4

360

1.0

0.20

10

9

900

0.0

0.10

4

10

400

0.5

0.05

Manage in
Place

10

10

1000

0.5

0.025

Remove/
Modify

10

5

450

1.0

0.10

Manage in
Place

5

5

225

0.0

0.025

Presence of Buried
Infrastructure

Remove/
Modify

5

5

250

0.5

0.05

5

250

0.5

0.25

8

1

64

1

0.1

10

Electrical System
Upgrades

Manage in
Place
Modify
Affected
System(s)
Modify
Affected
System(s)

5

8

Presence of Buried
Infrastructure
Fire Suppression
and Alarm System
Upgrades

7

2

140

1

0.15

Occurrence

1A1

Misunderstanding
Existing
Conditions

%Likelihood
Cause

Cause

Severity

Table D-5. Panelist RP-01-E SFMEA Worksheet

1

10

1

10

Intermediate
Effect 1
Presence of
AsbestosContaining
Materials (ACM)
Presence of
AsbestosContaining
Materials (ACM)

1

10

Presence of LeadBased Paint

Remove/
Modify

1

10

0.8

9

0.8

9

Presence of LeadBased Paint
Presence of FuelRelated
Contaminants
Presence of FuelRelated
Contaminants

0.9

10

0.9

10

0.7
1

Intermediate
Effect 2

Intermediate
Effect 3

End Effect
Remove/
Modify
Manage in
Place

407

Cost

Plumbing System
Upgrades

0.8

9

Mechanical System
Upgrades

0.1

2

Universal Access
(ADA) Upgrades

1

10

OSHA-Related
Upgrades

0.9

10

Specialty-Standard
Upgrades

1C1

Overstatement of
Issues

0.8

9

1C2

Overstatement of
Issues

0.8

9

1B4
1B5
1B6

1C3

Overstatement of
Issues

0.8

9

Improper
Equipment Sizing
Improper
Equipment
Repurposing
Improper
Equipment
Repurposing

Inability to Meet
Performance
Expectations

Inaccurate
Prediction of
Service
Life/O&M
Schedule

408

Inability to Meet
Performance
Expectations

End Effect
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Savings
Shortfall/
Erosion of
ESCO's
Margin
Difficulty
Getting
Financing
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

10

Intermediate
Effect 3

RPN

1

1B7

1B3

Intermediate
Effect 2

Detection

Occurrence

Intermediate
Effect 1

Cause
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements

Severity

%Likelihood
Cause

Table D-5 (cont’d)

6

6

360

1

0.2

5

4

180

1

0.05

2

8

32

0.25

0.05

6

3

180

1

0.05

2

9

180

0.5

0.1

7

4

252

0.05

0.025

2

8

144

0.01

0.01

8

5

360

0.05

0.1

Cost

1C8

Understatement of
Issues

1D1

ESCO Lack of
Experience with
Facility Type

1D2

ESCO Lack of
Experience with
Facility Type

0.8

0.8

0.7

0.7

9

CustomerSpecified Issues

Changes to the
Size of the
Project Scope

Inability to Meet
Performance
Expectations

9

Improper
Equipment Sizing

9

Improper
Equipment
Repurposing

Inability to Meet
Performance
Expectations
Inaccurate
Prediction of
Service
Life/O&M
Schedule

Inability to Meet
Performance
Expectations

9

CustomerSpecified Issues

Changes to the
Size of the
Project Scope

Inability to Meet
Performance
Expectations

9

8

8

Missed
Opportunities
Failure to
Understand Facility
Operations and
Client Goals
Failure to
Understand Facility
Operations and
Client Goals

Inability to Meet
Performance
Expectations
Overestimate
Client Capacity
to Perform O&M
Overestimate
Client Capacity
to Perform O&M

409

Reduced Savings
Due to PoorlyPerformed O&M

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Missed
Opportunity
for O&M
Contract
Reduced
Project Value
for ESCO

% Likelihood
End Effect

1C7

Understatement of
Issues

0.8

Intermediate
Effect 3

RPN

1C6

Understatement of
Issues

0.8

Intermediate
Effect 2

Detection

1C5

Understatement of
Issues

0.8

Intermediate
Effect 1

Severity

1C4

Overstatement of
Issues

Occurrence

Cause

%Likelihood
Cause

Table D-5 (cont’d)

Cost

5

6

270

0.025

0.025

8

3

216

0.25

0.1

6

3

162

0.25

0.1

8

2

144

0.3

0.25

9

2

162

0.1

0.05

7

3

168

0.5

0.05

8

6

384

0.4

0.05

1D7

ESCO Lack of
Experience with
Facility Type

1D8

ESCO Lack of
Experience with
Facility Type

0.7

0.7

0.7

Reduced Savings
Due to PoorlyPerformed O&M

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin

Improper Baseline

Savings
Shortfall/
Erosion of
ESCO's
Margin

8

Failure to
Understand Facility
Operations and
Client Goals

8

Failure to
Understand Facility
Operations and
Client Goals

8

Failure to
Understand Facility
Operations and
Client Goals

Overestimate
Client Capacity
to Perform O&M
Lack of
Foresight About
Future Building
Use and
Occupant
Changes
Lack of
Foresight About
Future Building
Use and
Occupant
Changes

8

Failure to
Understand Facility
Operations and
Client Goals

Improper
Construction
Timing and
Phasing

Project Schedule
too Short

8

Failure to
Understand Facility
Operations and
Client Goals

Overestimate
Productivity
Rates

Project Schedule
too Short

8

Failure to
Understand Facility
Operations and
Client Goals

Overestimate
Productivity
Rates

Underestimate
Project Costs

410

Improper Retrofit
Design

Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

1D6

ESCO Lack of
Experience with
Facility Type

0.7

Intermediate
Effect 3

RPN

1D5

ESCO Lack of
Experience with
Facility Type

0.7

Intermediate
Effect 2

Detection

1D4

ESCO Lack of
Experience with
Facility Type

0.7

Intermediate
Effect 1

Severity

1D3

ESCO Lack of
Experience with
Facility Type

Occurrence

Cause

%Likelihood
Cause

Table D-5 (cont’d)

8

6

384

0.1

0.075

9

6

432

0.4

0.25

7

4

224

0.15

0.10

7

3

168

0.5

0.10

8

2

128

0.5

0.33

8

3

192

0.75

0.40

Cost

1E2

Differing
Stakeholder
Needs

1E3

Differing
Stakeholder
Needs

0.7

0.7

0.7

Failure to
Understand Facility
Operations and
Client Goals

Improper
Consideration of
Security
Concerns

Project Schedule
too Short

Improper
Consideration of
Security
Concerns
Improper
Consideration of
Security
Concerns

8

Failure to
Understand Facility
Operations and
Client Goals
Failure to
Understand Facility
Operations and
Client Goals
Inconsistent
Information About
Facility Operating
Parameters (e.g.,
too hot, too cold,
too much airflow,
not enough
airflow)

Retrofit Design
Does Not Satisfy
All Stakeholders

User Modification
of
ECMs/Override
Operating
Parameters

8

Codes and
Standards
Upgrades

Retrofit Design
Does Not Satisfy
All Stakeholders

Comfort-Related
Callbacks after
Construction
Complete

8

Codes and
Standards
Upgrades

Increased Project
Complexity

Controls Scope of
Work Lacks
Adequate Detail
and Specificity

8

8

8

411

Underestimate
Project Costs

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

Occupant Use of
Equipment for
Other Purposes

Legal
Impacts

Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

1E1

Differing
Stakeholder
Needs

0.7

Intermediate
Effect 3

RPN

1D11

ESCO Lack of
Experience with
Facility Type

0.7

Intermediate
Effect 2

Detection

1D10

ESCO Lack of
Experience with
Facility Type

0.7

Intermediate
Effect 1

Severity

1D9

ESCO Lack of
Experience with
Facility Type

Occurrence

Cause

%Likelihood
Cause

Table D-5 (cont’d)

7

6

336

0.25

0.50

6

5

240

0.2

0.50

5

5

200

0.2

0.25

8

2

128

0.5

0.5

8

3

192

0.5

0.4

7

4

224

0.4

0.4

Cost

1F2

IGA Conducted
Too Quickly

1F3

IGA Conducted
Too Quickly

1F4

IGA Conducted
Too Quickly

0.8

0.8

0.8

0.8

Increased Project
Complexity

Retrofit Design
Does Not Satisfy
All Stakeholders

Comfort-Related
Callbacks after
Construction
Complete

8

Increased Project
Complexity

Controls Scope
of Work Lacks
Adequate Detail
and Specificity

9

Miss Critical
Functions and
Savings
Opportunities

9

Miscount Existing
Equipment (e.g.,
Lights)

9

Miscount Existing
Equipment (e.g.,
Lights)

Energy Model
Calibration
Errors

Inaccurate Baseline

Energy Model
Calibration
Errors

8

9

412

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

IGA Conducted
Too Quickly

Intermediate
Effect 3

RPN

1F1

0.7

Intermediate
Effect 2

Detection

1E5

Differing
Stakeholder
Needs

0.7

Intermediate
Effect 1

Severity

1E4

Differing
Stakeholder
Needs

Occurrence

Cause

%Likelihood
Cause

Table D-5 (cont’d)

8

3

192

0.6

0.3

8

3

192

0.6

0.2

9

2

162

0.8

0.2

9

2

162

0.8

0.2

9

3

243

0.8

0.3

8

3

216

0.8

0.3

Cost

1F5

0.8

9

Missing Facility
Information

1F6

IGA Conducted
Too Quickly

0.8

9

Missing Facility
Information

1F7

Facility
Information
Unavailable/
Inaccurate

0.8

9

Inaccurate Baseline

1F8

Facility
Information
Unavailable/
Inaccurate

0.8

9

Missing Facility
Information

9

Technology
Breakage/ Damage
Possibility by
Occupants

Change in
Design Intent for
Impacted
Equipment

9

Technology
Breakage/ Damage
Possibility by
Occupants

Change in
Design Intent for
Impacted
Equipment

2A1

2A2

Security Concerns

Security Concerns

0.8

0.8

Energy Model
Calibration
Errors

413

Rework

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Energy Model
Calibration
Errors

IGA Conducted
Too Quickly

Intermediate
Effect 3

RPN

Intermediate
Effect 2

Detection

Intermediate
Effect 1

Severity

Occurrence

Cause

%Likelihood
Cause

Table D-5 (cont’d)

8

3

216

0.9

0.3

8

3

216

0.9

0.3

7

2

126

0.5

0.5

7

2

126

0.5

0.5

7

4

252

0.7

0.25

6

8

432

0.7

0.25

Cost

2A4

Security Concerns

0.8

9

2A5

Security Concerns

0.8

9

2B1

Reduced
Availability of
“Low Hanging
Fruit”

2B2

Reduced
Availability of
“Low Hanging
Fruit”

2B3

2CD1

Reduced
Availability of
“Low Hanging
Fruit”
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation

0.90

0.90

0.90

0.90

Technology
Breakage/ Damage
Possibility by
Occupants

Equipment
Impacts

Total Costs of
Operation

Project Schedule
Impacted

Delay in Accruing
Savings

Technology
Breakage/ Damage
Possibility by
Occupants
Occupant Use of
Equipment for
Other Purposes

Increase in Project
Complexity

Limited/Less
Attractive Scope

10

Increase in Project
Costs

IGA Scope
Increase
(No Additional
Fee)

10

Less Opportunity
for Non-Energy
Benefits (NEBs)

Project Schedule
Impacted

10

Failure to Meet
Owner Project
Requirements

Rework

10

414

Increased
Opportunity for
Diminished
Savings

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

9

Intermediate
Effect 3

RPN

0.8

Intermediate
Effect 2

Detection

Security Concerns

Intermediate
Effect 1

Severity

2A3

Occurrence

Cause

%Likelihood
Cause

Table D-5 (cont’d)

5

3

135

0.2

0.1

3

7

189

0.1

0.1

2

4

72

0.1

0.1

9

5

450

0.8

0.8

6

5

300

0.8

0.75

4

7

280

0.8

0.9

6

6

360

0.75

0.2

Cost

Comfort
Complaints

10

Failure to Meet
Owner Project
Requirements

2CD3

2CD4

Operational
Concerns

0.70

8

2CD5

Safety Concerns

0.70

8

Failure to Meet
Owner Project
Requirements
Failure to Meet
Owner Project
Requirements

2E1

Unqualified
Subcontractors

0.8

9

Rework

0.8

9

0.8

9

2CD2

2E2
2E3

Unqualified
Subcontractors
Unqualified
Subcontractors

0.90

0.90

Unsafe Working
Conditions/Injuries
Unsafe Working
Conditions/Injuries

Reduce Project
Scope
Safety Incident

Project Schedule
Impacted

Delay in Accruing
Savings

Safety Incident

Project Schedule
Impacted

Safety Incident

415

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts

% Likelihood
End Effect

10

Failure to Meet
Owner Project
Requirements

Intermediate
Effect 3

RPN

Intermediate
Effect 2

Detection

Intermediate
Effect 1

Cause
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation

Severity

Occurrence

%Likelihood
Cause

Table D-5 (cont’d)

7

8

560

0.2

0.25

8

6

480

0.75

0.2

4

4

128

0.1

0.2

3

4

96

0.2

0.1

8

3

216

0.8

0.2

6

4

216

0.5

0.1

3

2

54

0.1

0.1

Cost

2

Rework

Project Schedule
Impacted

Delay in Accruing
Savings

2E5

Unreliable
Subcontractors

0.15

2

Unsafe Working
Conditions/Injuries

Safety Incident

Project Schedule
Impacted

0.15

2

Project Schedule
Impacted

9

Require ESCO to
use Fixed
Overhead & Profit
Rates

Overstate Project
Soft Costs

9

Divide
Procurement into
Separate Phases

Disconnect
Between IGA
and Retrofit
Design

Incorrect Vendor
Selection

Disconnect
Between IGA
and Retrofit
Design

2E6

2F1

2F2

2F3

Unreliable
Subcontractors
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement

0.8

0.8

0.8

9

Intermediate
Effect 3

416

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

0.15

Intermediate
Effect 2

RPN

Unreliable
Subcontractors

Intermediate
Effect 1

Detection

Occurrence

2E4

Cause

Severity

%Likelihood
Cause

Table D-5 (cont’d)

7

8

112

0.4

0.2

2

2

8

0.1

0.1

3

2

12

0.1

0.1

7

5

315

0.75

0.2

9

2

162

0.8

0.5

9

3

243

0.8

0.5

Cost

6

Reduced
Equipment Service
Life

0.6

7

Uncertainty in
Factors Used to
Predict
Performance

0.6

7

Reduced Energy
Performance

2G5

ECM
Constructability
& Feasibility
Issues

0.5

6

Uncertainty in
Factors Used to
Predict
Performance

2G6

ECM
Constructability
& Feasibility
Issues

0.5

6

Rework

2G2

2G3

0.5

Reduced
Equipment
Service Life

Reduced Energy
Performance

417

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Increased O&M
Cost

Intermediate
Effect 3

RPN

6

2G4

2G1

Intermediate
Effect 2

Detection

Occurrence

0.5

Intermediate
Effect 1

Cause
Select Cheapest
Available
Equipment to
Meet
Specifications
Select Cheapest
Available
Equipment to
Meet
Specifications
Actual Climate
Conditions
Outside Range of
Planned
Conditions
Actual Climate
Conditions
Outside Range of
Planned
Conditions

Severity

%Likelihood
Cause

Table D-5 (cont’d)

8

2

96

0.3

0.25

3

3

54

0.4

0.5

5

4

140

0.1

0.10

4

6

168

0.05

0.05

2

9

108

0.05

0.05

2

8

96

0.05

0.025

Cost

Reduced Energy
Performance

2G8

ECM Failure to
Perform as
Designed

0.02

1

Rework

418

Intermediate
Effect 3

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

1

Intermediate
Effect 2

RPN

0.02

Intermediate
Effect 1

Detection

Occurrence

2G7

ECM Failure to
Perform as
Designed

Cause

Severity

%Likelihood
Cause

Table D-5 (cont’d)

Cost

2

2

4

0.05

0.025

7

5

35

0.05

0.1

1A3
1A4
1A5
1A6
1A7
1A8

1B1
1B2

10

1

10

1

10

0.8

9

0.8

9

0.75

8

0.75

8

Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements

0.75

8

1

10

End Effect
Remove/
Modify

6

2

120

0.625

0.12

Manage in
Place

4

2

80

0.4

0.05

Remove/
Modify

5

6

300

0.3

0.03

Manage in
Place

5

6

300

0.8

0.03

Remove/
Modify

7

3

189

1

0.1

Manage in
Place

7

3

189

0.0

0.1

Remove/
Modify

6

6

288

0.5

0.1

Manage in
Place

5

6

240

0.5

0.06

6

6

288

0.5

0.05

7

5

350

0.8

0.08

Modify
Affected
System(s)
Modify
Affected
System(s)

419

% Likelihood
End Effect

1

Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions

Intermediate
Effect 3

RPN

10

Occurrence

1

Intermediate
Effect 2

Detection

1A2

Misunderstanding
Existing
Conditions

Intermediate
Effect 1
Presence of
AsbestosContaining
Materials (ACM)
Presence of
AsbestosContaining
Materials (ACM)
Presence of
Lead-Based
Paint
Presence of
Lead-Based
Paint
Presence of FuelRelated
Contaminants
Presence of FuelRelated
Contaminants
Presence of
Buried
Infrastructure
Presence of
Buried
Infrastructure
Fire Suppression
and Alarm
System
Upgrades
Electrical
System
Upgrades

Severity

1A1

Cause

%Likelihood
Cause

Table D-6. Panelist RP-02-E SFMEA Worksheet

Cost

1B5
1B6
1B7

9

0.2

3

Universal Access
(ADA) Upgrades

1

10

1

10

1C1

Overstatement of
Issues

0.15

2

1C2

Overstatement of
Issues

0.15

2

1C3

Overstatement of
Issues

0.15

2

OSHA-Related
Upgrades
SpecialtyStandard
Upgrades
Improper
Equipment
Sizing
Improper
Equipment
Repurposing
Improper
Equipment
Repurposing

Inability to
Meet
Performance
Expectations

Inaccurate
Prediction of
Service
Life/O&M
Schedule

420

Inability to Meet
Performance
Expectations

End Effect
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Savings
Shortfall/
Erosion of
ESCO's
Margin
Difficulty
Getting
Financing
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

0.8

Intermediate
Effect 3

6

4

216

0.2

0.05

6

3

162

0.5

0.05

6

2

36

0.0

0.04

5

2

100

0.0

0.03

6

6

360

0.15

0.03

8

4

64

0.7

0.1

9

1

18

0.5

1

6

8

96

0.2

0.06

RPN

Occurrence
9

Intermediate
Effect 2

Detection

1B4

0.8

Intermediate
Effect 1
Plumbing
System
Upgrades
Mechanical
System
Upgrades

Severity

1B3

Cause
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements

%Likelihood
Cause

Table D-6 (cont’d)

Cost

Understatement of
Issues

1C6

Understatement of
Issues

1C7

Understatement of
Issues

1C8

Understatement of
Issues

1D1

ESCO Lack of
Experience with
Facility Type

0.15

0.15

0.15

0.15

0.18

2

CustomerSpecified Issues

Changes to the
Size of the
Project Scope

Inability to Meet
Performance
Expectations

2

Improper
Equipment
Sizing

Inability to
Meet
Performance
Expectations

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

Overestimate
Client Capacity
to Perform
O&M

Missed
Opportunity
for O&M
Contract

2

Improper
Equipment
Repurposing

Inability to
Meet
Performance
Expectations
Inaccurate
Prediction of
Service
Life/O&M
Schedule

Inability to Meet
Performance
Expectations

2

CustomerSpecified Issues

Changes to the
Size of the
Project Scope

Inability to Meet
Performance
Expectations

2

3

Missed
Opportunities
Failure to
Understand
Facility
Operations and
Client Goals

421

% Likelihood
End Effect

Intermediate
Effect 3

7

3

42

0.5

0.04

7

8

112

0.1

0.03

7

8

112

0.1

0.03

7

8

112

0.5

0.05

4

7

56

0.5

0.02

8

5

120

0.2

0.02

RPN

Intermediate
Effect 2

Detection

1C5

0.15

Intermediate
Effect 1

Severity

1C4

Overstatement of
Issues

Occurrence

Cause

%Likelihood
Cause

Table D-6 (cont’d)

Cost

1D4

ESCO Lack of
Experience with
Facility Type

1D5

ESCO Lack of
Experience with
Facility Type

1D6

ESCO Lack of
Experience with
Facility Type

1D7

ESCO Lack of
Experience with
Facility Type

0.18

0.18

0.18

0.18

0.18

3

3

3

Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals

Overestimate
Client Capacity
to Perform
O&M

Reduced Savings
Due to PoorlyPerformed O&M

Overestimate
Client Capacity
to Perform
O&M
Lack of
Foresight About
Future Building
Use and
Occupant
Changes
Lack of
Foresight About
Future Building
Use and
Occupant
Changes

Reduced Savings
Due to PoorlyPerformed O&M

Reduced
Project Value
for ESCO
Savings
Shortfall/
Erosion of
ESCO's
Margin

Improper Baseline

Savings
Shortfall/
Erosion of
ESCO's
Margin

Improper Retrofit
Design

Improper
Construction
Timing and
Phasing

Project Schedule
too Short

Overestimate
Productivity
Rates

Project Schedule
too Short

422

End Effect

Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

3

Failure to
Understand
Facility
Operations and
Client Goals

Intermediate
Effect 3

7

6

126

0.5

0.05

9

6

162

0.15

0.08

9

4

108

0.1

0.08

9

4

108

0.2

0.1

5

6

90

0.1

0.06

5

6

90

0.2

0.02

RPN

3

3

Intermediate
Effect 2

Detection

1D3

ESCO Lack of
Experience with
Facility Type

0.18

Intermediate
Effect 1
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals

Severity

1D2

ESCO Lack of
Experience with
Facility Type

Occurrence

Cause

%Likelihood
Cause

Table D-6 (cont’d)

Cost

1D10

ESCO Lack of
Experience with
Facility Type

1D11

ESCO Lack of
Experience with
Facility Type

1E1

Differing
Stakeholder
Needs

0.18

0.18

0.18

0.33

3

3

3

3

4

8

120

0.2

0.05

8

6

144

0.1

0.1

8

7

168

0.2

0.2

Legal
Impacts

9

8

216

0.11

0.25

Savings
Shortfall/
Erosion of
ESCO's
Margin

9

6

216

0.8

0.05

Intermediate
Effect 3

Overestimate
Productivity
Rates

Underestimate
Project Costs

Improper
Consideration
of Security
Concerns

Project Schedule
too Short

Improper
Consideration
of Security
Concerns

Underestimate
Project Costs

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

Improper
Consideration
of Security
Concerns

Occupant Use of
Equipment for
Other Purposes

Retrofit Design
Does Not
Satisfy All
Stakeholders

User Modification
of
ECMs/Override
Operating
Parameters

423

% Likelihood
End Effect

5

Intermediate
Effect 2

RPN

Detection

1D9

ESCO Lack of
Experience with
Facility Type

0.18

Intermediate
Effect 1
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Inconsistent
Information
About Facility
Operating
Parameters (e.g.,
too hot, too cold,
too much
airflow, not
enough airflow)

Severity

1D8

ESCO Lack of
Experience with
Facility Type

Occurrence

Cause

%Likelihood
Cause

Table D-6 (cont’d)

Cost

1E4

Differing
Stakeholder
Needs

1E5

Differing
Stakeholder
Needs

1F1

1F2

IGA Conducted
Too Quickly

IGA Conducted
Too Quickly

0.33

0.33

0.33

0.25

0.25

4

Codes and
Standards
Upgrades

Retrofit Design
Does Not
Satisfy All
Stakeholders

Comfort-Related
Callbacks after
Construction
Complete

4

Codes and
Standards
Upgrades

Increased
Project
Complexity

Controls Scope of
Work Lacks
Adequate Detail
and Specificity

Increased Project
Complexity

Retrofit Design
Does Not
Satisfy All
Stakeholders

Comfort-Related
Callbacks after
Construction
Complete

4

Increased Project
Complexity

Controls Scope
of Work Lacks
Adequate Detail
and Specificity

3

Miss Critical
Functions and
Savings
Opportunities

3

Miscount
Existing
Equipment (e.g.,
Lights)

4

424

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Intermediate
Effect 3

8

8

256

0.3

0.06

9

6

216

0.4

0.05

6

6

144

1

0.01

6

3

72

0.6

0.02

6

4

72

0.33

0.02

5

4

60

0.25

0.01

RPN

Intermediate
Effect 2

Detection

1E3

Differing
Stakeholder
Needs

0.33

Intermediate
Effect 1

Severity

1E2

Differing
Stakeholder
Needs

Occurrence

Cause

%Likelihood
Cause

Table D-6 (cont’d)

Cost

Energy Model
Calibration
Errors

3

Inaccurate
Baseline

Energy Model
Calibration
Errors
Energy Model
Calibration
Errors

1F3

IGA Conducted
Too Quickly

1F4

IGA Conducted
Too Quickly

1F5

IGA Conducted
Too Quickly

0.25

3

Missing Facility
Information

1F6

IGA Conducted
Too Quickly

0.25

3

Missing Facility
Information

1F7

Facility
Information
Unavailable/
Inaccurate

0.35

4

Inaccurate
Baseline

1F8

Facility
Information
Unavailable/
Inaccurate

0.35

4

Missing Facility
Information

0.25

0.25

Energy Model
Calibration
Errors

425

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

3

Miscount
Existing
Equipment (e.g.,
Lights)

Intermediate
Effect 3

4

4

48

0.3

0.01

7

4

84

0.5

0.03

6

4

72

0.8

0.02

5

3

45

0.25

0.01

9

6

216

0.2

0.1

5

6

120

0.5

0.01

RPN

Intermediate
Effect 2

Detection

Intermediate
Effect 1

Severity

Occurrence

Cause

%Likelihood
Cause

Table D-6 (cont’d)

Cost

2

2A5

Security Concerns

0.15

2

2B1

Reduced
Availability of
“Low Hanging
Fruit”

9

Increase in
Project
Complexity

Limited/Less
Attractive
Scope

2B2

Reduced
Availability of
“Low Hanging
Fruit”

Increase in
Project Costs

IGA Scope
Increase
(No Additional
Fee)

0.15

2

2A2

Security Concerns

0.15

2

2A3

Security Concerns

0.15

2

0.8

0.8

9

Change in
Design Intent
for Impacted
Equipment

Rework

Change in
Design Intent
for Impacted
Equipment

Equipment
Impacts

Total Costs of
Operation

Project
Schedule
Impacted

Delay in Accruing
Savings

426

Increased
Opportunity for
Diminished
Savings

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

0.15

Security Concerns

Intermediate
Effect 3

5

4

40

0.25

0.01

4

4

32

0.2

0.01

7

9

126

0.25

0.02

6

4

48

0.05

0.02

7

4

56

0.05

0.05

7

3

189

0.5

0.05

9

4

324

0.25

0.2

RPN

Security Concerns

2A1

Intermediate
Effect 2

Detection

2A4

Intermediate
Effect 1
Technology
Breakage/
Damage
Possibility by
Occupants
Technology
Breakage/
Damage
Possibility by
Occupants
Technology
Breakage/
Damage
Possibility by
Occupants
Technology
Breakage/
Damage
Possibility by
Occupants
Occupant Use of
Equipment for
Other Purposes

Cause

Severity

Occurrence

%Likelihood
Cause

Table D-6 (cont’d)

Cost

Project
Schedule
Impacted

9

Failure to Meet
Owner Project
Requirements

Rework

9

Failure to Meet
Owner Project
Requirements

Comfort
Complaints

9

Failure to Meet
Owner Project
Requirements

2CD3

2CD4

Operational
Concerns

0.15

2

2CD5

Safety Concerns

0.15

2

2E1

Unqualified
Subcontractors

2B3

2CD1

2CD2

0.8

0.8

0.8

0.8

0.2

3

Failure to Meet
Owner Project
Requirements
Failure to Meet
Owner Project
Requirements

Rework

Reduce Project
Scope
Safety Incident
Project
Schedule
Impacted

427

Delay in Accruing
Savings

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

9

Less Opportunity
for Non-Energy
Benefits (NEBs)

Reduced
Availability of
“Low Hanging
Fruit”
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation

Intermediate
Effect 3

6

4

216

0.2

0.06

8

4

288

0.5

0.03

6

5

270

0.5

0.03

5

3

135

0.1

0.02

9

3

54

0.1

0.25

10

10

200

0.015

0.35

7

6

126

0.3

0.02

RPN

Intermediate
Effect 2

Detection

Intermediate
Effect 1

Severity

Occurrence

Cause

%Likelihood
Cause

Table D-6 (cont’d)

Cost

0.2

3

2E3

Unqualified
Subcontractors

0.2

3

2E4

Unreliable
Subcontractors

2E5

Unreliable
Subcontractors

2E6

2F1

2F2

Unreliable
Subcontractors
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement

Unsafe Working
Conditions/Injuri
es
Unsafe Working
Conditions/Injuri
es

Safety Incident

Project Schedule
Impacted

Safety Incident

3

Rework

Project
Schedule
Impacted

0.2

3

Unsafe Working
Conditions/Injuri
es

Safety Incident

0.2

3

Project Schedule
Impacted

6

Require ESCO to
use Fixed
Overhead &
Profit Rates

Overstate
Project Soft
Costs

6

Divide
Procurement into
Separate Phases

Disconnect
Between IGA
and Retrofit
Design

0.2

0.5

0.5

428

Delay in Accruing
Savings

Project Schedule
Impacted

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Intermediate
Effect 3

10

5

150

0.1

0.15

10

8

240

0.02

0.85

6

6

108

0.25

0.03

10

6

180

0.01

0.15

6

6

108

0.25

0.02

6

1

36

0.75

0.10

7

3

126

0.15

0.06

RPN

Intermediate
Effect 2

Detection

2E2

Unqualified
Subcontractors

Intermediate
Effect 1

Severity

Occurrence

Cause

%Likelihood
Cause

Table D-6 (cont’d)

Cost

2G2

2G3

0.5

6

Incorrect Vendor
Selection

0.25

3

Increased O&M
Cost

3

Reduced
Equipment
Service Life

0.2

3

Uncertainty in
Factors Used to
Predict
Performance

0.2

3

Reduced Energy
Performance

3

Uncertainty in
Factors Used to
Predict
Performance

0.25

0.25

Reduced
Equipment
Service Life

Reduced
Energy
Performance

429

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Disconnect
Between IGA
and Retrofit
Design

Intermediate
Effect 3

8

9

432

0.2

0.25

6

2

36

0.2

0.01

7

6

126

0.25

0.06

7

6

126

0.1

0.1

7

6

126

0.25

0.1

7

6

126

0.30

0.15

RPN

Intermediate
Effect 2

Detection

2G1

Intermediate
Effect 1

Severity

2G5

ECM
Constructability
& Feasibility
Issues

2F3

Occurrence

2G4

Cause
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement
Select Cheapest
Available
Equipment to
Meet
Specifications
Select Cheapest
Available
Equipment to
Meet
Specifications
Actual Climate
Conditions
Outside Range of
Planned
Conditions
Actual Climate
Conditions
Outside Range of
Planned
Conditions

%Likelihood
Cause

Table D-6 (cont’d)

Cost

2G7

ECM Failure to
Perform as
Designed

0.18

3

Reduced Energy
Performance

2G8

ECM Failure to
Perform as
Designed

0.18

3

Rework

430

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Rework

Intermediate
Effect 3

8

6

144

0.5

0.2

7

6

126

0.25

0.1

8

7

168

0.2

0.25

RPN

3

Intermediate
Effect 2

Detection

0.25

Intermediate
Effect 1

Severity

2G6

ECM
Constructability
& Feasibility
Issues

Occurrence

Cause

%Likelihood
Cause

Table D-6 (cont’d)

Cost

1A3
1A4
1A5
1A6
1A7
1A8

1B1
1B2

10

1

10

1

10

0.9

10

0.9

10

0.8

9

0.8

9

Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements

0.5

6

0.75

8

End Effect
Remove/
Modify

6

5

300

1

0.4

Manage in
Place

4

5

200

0.0

0.2

Remove/
Modify

6

5

300

0.3

0.3

Manage in
Place

3

5

150

0.7

0.1

Remove/
Modify

5

6

300

1

0.2

Manage in
Place

2

6

120

0.0

1

Remove/
Modify

6

7

378

0.3

0.2

Manage in
Place

2

7

126

0.7

0.1

2

3

36

0.7

0.1

6

3

144

1

0.2

Modify
Affected
System(s)
Modify
Affected
System(s)

431

% Likelihood
End Effect

1

Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions

Intermediate
Effect 3

RPN

10

Occurrence

1

Intermediate
Effect 2

Detection

1A2

Misunderstanding
Existing
Conditions

Intermediate
Effect 1
Presence of
AsbestosContaining
Materials (ACM)
Presence of
AsbestosContaining
Materials (ACM)
Presence of
Lead-Based
Paint
Presence of
Lead-Based
Paint
Presence of FuelRelated
Contaminants
Presence of FuelRelated
Contaminants
Presence of
Buried
Infrastructure
Presence of
Buried
Infrastructure
Fire Suppression
and Alarm
System
Upgrades
Electrical
System
Upgrades

Severity

1A1

Cause

%Likelihood
Cause

Table D-7. Panelist RP-03-O SFMEA Worksheet

Cost

1B5
1B6
1B7

8

0.2

3

Universal Access
(ADA) Upgrades

1

10

1

10

1C1

Overstatement of
Issues

0.3

4

1C2

Overstatement of
Issues

0.3

4

1C3

Overstatement of
Issues

0.3

4

OSHA-Related
Upgrades
SpecialtyStandard
Upgrades
Improper
Equipment
Sizing
Improper
Equipment
Repurposing
Improper
Equipment
Repurposing

Inability to
Meet
Performance
Expectations

Inaccurate
Prediction of
Service
Life/O&M
Schedule

432

Inability to Meet
Performance
Expectations

End Effect
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Savings
Shortfall/
Erosion of
ESCO's
Margin
Difficulty
Getting
Financing
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

0.75

Intermediate
Effect 3

5

3

120

1

0.2

5

3

120

0.75

0.2

3

4

36

0.2

0.1

2

4

80

0.1

0.1

2

5

100

0.1

0.1

4

5

80

0.9

0.25

2

1

8

0.05

0.3

4

5

80

0.6

0.25

RPN

Occurrence
8

Intermediate
Effect 2

Detection

1B4

0.75

Intermediate
Effect 1
Plumbing
System
Upgrades
Mechanical
System
Upgrades

Severity

1B3

Cause
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements

%Likelihood
Cause

Table D-7 (cont’d)

Cost

Understatement of
Issues

1C6

Understatement of
Issues

1C7

Understatement of
Issues

1C8

Understatement of
Issues

1D1

ESCO Lack of
Experience with
Facility Type

0.3

0.3

0.3

0.3

0.5

4

CustomerSpecified Issues

Changes to the
Size of the
Project Scope

Inability to Meet
Performance
Expectations

4

Improper
Equipment
Sizing

Inability to
Meet
Performance
Expectations

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

Overestimate
Client Capacity
to Perform
O&M

Missed
Opportunity
for O&M
Contract

4

Improper
Equipment
Repurposing

Inability to
Meet
Performance
Expectations
Inaccurate
Prediction of
Service
Life/O&M
Schedule

Inability to Meet
Performance
Expectations

4

CustomerSpecified Issues

Changes to the
Size of the
Project Scope

Inability to Meet
Performance
Expectations

4

6

Missed
Opportunities
Failure to
Understand
Facility
Operations and
Client Goals

433

% Likelihood
End Effect

Intermediate
Effect 3

4

5

80

0.6

0.25

4

5

80

0.9

0.25

4

5

80

0.6

0.25

4

5

80

0.6

0.25

4

5

80

0.7

0.25

8

3

144

0.3

0.25

RPN

Intermediate
Effect 2

Detection

1C5

0.3

Intermediate
Effect 1

Severity

1C4

Overstatement of
Issues

Occurrence

Cause

%Likelihood
Cause

Table D-7 (cont’d)

Cost

1D4

ESCO Lack of
Experience with
Facility Type

1D5

ESCO Lack of
Experience with
Facility Type

1D6

ESCO Lack of
Experience with
Facility Type

1D7

ESCO Lack of
Experience with
Facility Type

0.5

0.5

0.5

0.5

0.5

6

6

6

Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals

Overestimate
Client Capacity
to Perform
O&M

Reduced Savings
Due to PoorlyPerformed O&M

Overestimate
Client Capacity
to Perform
O&M
Lack of
Foresight About
Future Building
Use and
Occupant
Changes
Lack of
Foresight About
Future Building
Use and
Occupant
Changes

Reduced Savings
Due to PoorlyPerformed O&M

Reduced
Project Value
for ESCO
Savings
Shortfall/
Erosion of
ESCO's
Margin

Improper Baseline

Savings
Shortfall/
Erosion of
ESCO's
Margin

Improper Retrofit
Design

Improper
Construction
Timing and
Phasing

Project Schedule
too Short

Overestimate
Productivity
Rates

Project Schedule
too Short

434

End Effect

Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

6

Failure to
Understand
Facility
Operations and
Client Goals

Intermediate
Effect 3

6

2

72

0.3

0.2

7

3

126

0.3

0.2

7

3

126

0.25

0.3

6

5

180

0.7

0.3

7

3

126

0.7

0.4

7

4

168

0.7

0.4

RPN

6

6

Intermediate
Effect 2

Detection

1D3

ESCO Lack of
Experience with
Facility Type

0.5

Intermediate
Effect 1
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals

Severity

1D2

ESCO Lack of
Experience with
Facility Type

Occurrence

Cause

%Likelihood
Cause

Table D-7 (cont’d)

Cost

1D10

ESCO Lack of
Experience with
Facility Type

1D11

ESCO Lack of
Experience with
Facility Type

1E1

Differing
Stakeholder
Needs

0.5

0.5

0.5

0.8

6

6

6

6

9

3

126

0.8

0.4

7

3

126

0.7

0.2

7

4

168

0.7

0.1

Legal
Impacts

8

4

192

0.1

0.1

Savings
Shortfall/
Erosion of
ESCO's
Margin

7

6

378

0.8

0.8

Intermediate
Effect 3

Overestimate
Productivity
Rates

Underestimate
Project Costs

Improper
Consideration
of Security
Concerns

Project Schedule
too Short

Improper
Consideration
of Security
Concerns

Underestimate
Project Costs

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

Improper
Consideration
of Security
Concerns

Occupant Use of
Equipment for
Other Purposes

Retrofit Design
Does Not
Satisfy All
Stakeholders

User Modification
of
ECMs/Override
Operating
Parameters

435

% Likelihood
End Effect

7

Intermediate
Effect 2

RPN

Detection

1D9

ESCO Lack of
Experience with
Facility Type

0.5

Intermediate
Effect 1
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Inconsistent
Information
About Facility
Operating
Parameters (e.g.,
too hot, too cold,
too much
airflow, not
enough airflow)

Severity

1D8

ESCO Lack of
Experience with
Facility Type

Occurrence

Cause

%Likelihood
Cause

Table D-7 (cont’d)

Cost

1E4

Differing
Stakeholder
Needs

1E5

Differing
Stakeholder
Needs

1F1

1F2

IGA Conducted
Too Quickly

IGA Conducted
Too Quickly

0.8

0.8

0.8

0.3

0.3

9

Codes and
Standards
Upgrades

Retrofit Design
Does Not
Satisfy All
Stakeholders

Comfort-Related
Callbacks after
Construction
Complete

9

Codes and
Standards
Upgrades

Increased
Project
Complexity

Controls Scope of
Work Lacks
Adequate Detail
and Specificity

Increased Project
Complexity

Retrofit Design
Does Not
Satisfy All
Stakeholders

Comfort-Related
Callbacks after
Construction
Complete

9

Increased Project
Complexity

Controls Scope
of Work Lacks
Adequate Detail
and Specificity

4

Miss Critical
Functions and
Savings
Opportunities

4

Miscount
Existing
Equipment (e.g.,
Lights)

9

436

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Intermediate
Effect 3

8

4

288

0.7

0.8

7

3

189

0.7

0.8

8

4

288

0.7

0.8

7

4

252

0.7

0.8

8

6

192

0.9

0.3

8

4

128

0.9

0.35

RPN

Intermediate
Effect 2

Detection

1E3

Differing
Stakeholder
Needs

0.8

Intermediate
Effect 1

Severity

1E2

Differing
Stakeholder
Needs

Occurrence

Cause

%Likelihood
Cause

Table D-7 (cont’d)

Cost

Energy Model
Calibration
Errors

4

Inaccurate
Baseline

Energy Model
Calibration
Errors
Energy Model
Calibration
Errors

1F3

IGA Conducted
Too Quickly

1F4

IGA Conducted
Too Quickly

1F5

IGA Conducted
Too Quickly

0.3

4

Missing Facility
Information

1F6

IGA Conducted
Too Quickly

0.3

4

Missing Facility
Information

1F7

Facility
Information
Unavailable/
Inaccurate

0.7

8

Inaccurate
Baseline

1F8

Facility
Information
Unavailable/
Inaccurate

0.7

8

Missing Facility
Information

0.3

0.3

Energy Model
Calibration
Errors

437

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

4

Miscount
Existing
Equipment (e.g.,
Lights)

Intermediate
Effect 3

8

4

128

0.9

0.35

8

7

224

0.9

0.35

8

7

224

0.9

0.35

8

7

224

0.9

0.35

8

7

448

0.9

0.35

8

7

448

0.9

0.4

RPN

Intermediate
Effect 2

Detection

Intermediate
Effect 1

Severity

Occurrence

Cause

%Likelihood
Cause

Table D-7 (cont’d)

Cost

9

2A5

Security Concerns

0.8

9

2B1

Reduced
Availability of
“Low Hanging
Fruit”

9

Increase in
Project
Complexity

Limited/Less
Attractive
Scope

2B2

Reduced
Availability of
“Low Hanging
Fruit”

Increase in
Project Costs

IGA Scope
Increase
(No Additional
Fee)

0.8

9

2A2

Security Concerns

0.8

9

2A3

Security Concerns

0.8

9

0.8

0.8

9

Change in
Design Intent
for Impacted
Equipment

Rework

Change in
Design Intent
for Impacted
Equipment

Equipment
Impacts

Total Costs of
Operation

Project
Schedule
Impacted

Delay in Accruing
Savings

438

Increased
Opportunity for
Diminished
Savings

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

0.8

Security Concerns

Intermediate
Effect 3

7

5

315

0.8

0.2

8

5

360

0.7

0.2

8

5

360

0.8

0.15

4

6

216

0.2

0.1

8

4

288

0.1

0.15

6

2

108

0.8

0.5

6

2

108

0.8

0.5

RPN

Security Concerns

2A1

Intermediate
Effect 2

Detection

2A4

Intermediate
Effect 1
Technology
Breakage/
Damage
Possibility by
Occupants
Technology
Breakage/
Damage
Possibility by
Occupants
Technology
Breakage/
Damage
Possibility by
Occupants
Technology
Breakage/
Damage
Possibility by
Occupants
Occupant Use of
Equipment for
Other Purposes

Cause

Severity

Occurrence

%Likelihood
Cause

Table D-7 (cont’d)

Cost

Project
Schedule
Impacted

5

Failure to Meet
Owner Project
Requirements

Rework

5

Failure to Meet
Owner Project
Requirements

Comfort
Complaints

5

Failure to Meet
Owner Project
Requirements

2CD3

2CD4

Operational
Concerns

0.4

5

2CD5

Safety Concerns

0.2

3

2E1

Unqualified
Subcontractors

2B3

2CD1

2CD2

0.8

0.4

0.4

0.4

0.3

4

Failure to Meet
Owner Project
Requirements
Failure to Meet
Owner Project
Requirements

Rework

Reduce Project
Scope
Safety Incident
Project
Schedule
Impacted

439

Delay in Accruing
Savings

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

9

Less Opportunity
for Non-Energy
Benefits (NEBs)

Reduced
Availability of
“Low Hanging
Fruit”
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation

Intermediate
Effect 3

6

2

108

0.85

0.5

4

4

80

0.8

0.3

4

6

120

0.7

0.4

3

7

105

0.6

0.3

2

7

70

0.8

0.3

1

9

27

0.1

0.1

5

3

60

0.85

0.3

RPN

Intermediate
Effect 2

Detection

Intermediate
Effect 1

Severity

Occurrence

Cause

%Likelihood
Cause

Table D-7 (cont’d)

Cost

0.3

4

2E3

Unqualified
Subcontractors

0.3

4

2E4

Unreliable
Subcontractors

2E5

Unreliable
Subcontractors

2E6

2F1

2F2

Unreliable
Subcontractors
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement

Unsafe Working
Conditions/Injuri
es
Unsafe Working
Conditions/Injuri
es

Safety Incident

Project Schedule
Impacted

Safety Incident

5

Rework

Project
Schedule
Impacted

0.3

4

Unsafe Working
Conditions/Injuri
es

Safety Incident

0.4

5

Project Schedule
Impacted

8

Require ESCO to
use Fixed
Overhead &
Profit Rates

Overstate
Project Soft
Costs

8

Divide
Procurement into
Separate Phases

Disconnect
Between IGA
and Retrofit
Design

0.4

0.75

0.75

440

Delay in Accruing
Savings

Project Schedule
Impacted

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Intermediate
Effect 3

2

4

32

0.4

0.1

2

7

56

0.2

0.1

2

3

30

0.4

0.1

5

3

60

0.85

0.3

6

3

90

0.80

0.35

7

3

168

0.7

0.4

7

3

168

0.85

0.4

RPN

Intermediate
Effect 2

Detection

2E2

Unqualified
Subcontractors

Intermediate
Effect 1

Severity

Occurrence

Cause

%Likelihood
Cause

Table D-7 (cont’d)

Cost

2G2

2G3

0.75

8

Incorrect Vendor
Selection

0.5

6

Increased O&M
Cost

6

Reduced
Equipment
Service Life

0.3

4

Uncertainty in
Factors Used to
Predict
Performance

0.3

4

Reduced Energy
Performance

5

Uncertainty in
Factors Used to
Predict
Performance

0.5

0.4

Reduced
Equipment
Service Life

Reduced
Energy
Performance

441

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Disconnect
Between IGA
and Retrofit
Design

Intermediate
Effect 3

7

3

168

0.85

0.4

4

7

168

0.7

0.1

4

8

192

0.7

0.1

6

4

96

0.7

0.1

6

4

96

0.70

0.10

5

7

175

0.6

0.1

RPN

Intermediate
Effect 2

Detection

2G1

Intermediate
Effect 1

Severity

2G5

ECM
Constructability
& Feasibility
Issues

2F3

Occurrence

2G4

Cause
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement
Select Cheapest
Available
Equipment to
Meet
Specifications
Select Cheapest
Available
Equipment to
Meet
Specifications
Actual Climate
Conditions
Outside Range of
Planned
Conditions
Actual Climate
Conditions
Outside Range of
Planned
Conditions

%Likelihood
Cause

Table D-7 (cont’d)

Cost

2G7

ECM Failure to
Perform as
Designed

0.3

4

Reduced Energy
Performance

2G8

ECM Failure to
Perform as
Designed

0.3

4

Rework

442

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Rework

Intermediate
Effect 3

7

4

140

0.2

0.25

6

4

96

0.7

0.1

7

4

112

0.2

0.25

RPN

5

Intermediate
Effect 2

Detection

0.4

Intermediate
Effect 1

Severity

2G6

ECM
Constructability
& Feasibility
Issues

Occurrence

Cause

%Likelihood
Cause

Table D-7 (cont’d)

Cost

1A3
1A4
1A5
1A6
1A7
1A8

1B1
1B2

10

N/A

N/A

N/A

N/A

N/A

N/A

N/A

N/A

0.8

9

0.8

9

Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements

0.8

9

0.8

9

End Effect
Remove/
Modify

7

3

210

1

0.10

Manage in
Place

2

3

60

0

0.05

Remove/
Modify

1

1

N/A

0.00

Manage in
Place

1

1

N/A

0.00

Remove/
Modify

1

1

N/A

0.00

Manage in
Place

1

1

N/A

0.00

Remove/
Modify

3

9

243

0.1

0.70

Manage in
Place

8

9

648

0.9

0.20

6

2

108

0.8

0.05

4

6

216

0.2

0.20

Modify
Affected
System(s)
Modify
Affected
System(s)

443

% Likelihood
End Effect

1

Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions
Misunderstanding
Existing
Conditions

Intermediate
Effect 3

RPN

10

Occurrence

1

Intermediate
Effect 2

Detection

1A2

Misunderstanding
Existing
Conditions

Intermediate
Effect 1
Presence of
AsbestosContaining
Materials (ACM)
Presence of
AsbestosContaining
Materials (ACM)
Presence of
Lead-Based
Paint
Presence of
Lead-Based
Paint
Presence of FuelRelated
Contaminants
Presence of FuelRelated
Contaminants
Presence of
Buried
Infrastructure
Presence of
Buried
Infrastructure
Fire Suppression
and Alarm
System
Upgrades
Electrical
System
Upgrades

Severity

1A1

Cause

%Likelihood
Cause

Table D-8. Panelist RP-04-M SFMEA Worksheet

Cost

1B5
1B6
1B7

8

0.2

3

Universal Access
(ADA) Upgrades

1

10

0.75

8

1C1

Overstatement of
Issues

0.25

3

1C2

Overstatement of
Issues

0.25

3

1C3

Overstatement of
Issues

0.25

3

OSHA-Related
Upgrades
SpecialtyStandard
Upgrades
Improper
Equipment
Sizing
Improper
Equipment
Repurposing
Improper
Equipment
Repurposing

Inability to
Meet
Performance
Expectations

Inaccurate
Prediction of
Service
Life/O&M
Schedule

444

Inability to Meet
Performance
Expectations

End Effect
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Modify
Affected
System(s)
Savings
Shortfall/
Erosion of
ESCO's
Margin
Difficulty
Getting
Financing
Savings
Shortfall/
Erosion of
ESCO's
Margin

4

6

6

3

2

% Likelihood
End Effect

0.75

Intermediate
Effect 3

RPN

Occurrence
N/A

Intermediate
Effect 2

Detection

1B4

N/A

Intermediate
Effect 1
Plumbing
System
Upgrades
Mechanical
System
Upgrades

Severity

1B3

Cause
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements
Facility Age and
Current Code
Requirements

%Likelihood
Cause

Table D-8 (cont’d)

Cost

N/A

0.05

144

0.7

0.30

2

12

0

0.05

7

4

280

1

0.20

7

8

448

1

0.60

7

8

168

1

0.70

7

2

42

0.75

0.10

7

2

42

0.25

0.20

Understatement of
Issues

1C6

Understatement of
Issues

1C7

Understatement of
Issues

1C8

Understatement of
Issues

1D1

ESCO Lack of
Experience with
Facility Type

0.75

0.75

0.75

0.75

0.8

3

CustomerSpecified Issues

Changes to the
Size of the
Project Scope

Inability to Meet
Performance
Expectations

8

Improper
Equipment
Sizing

Inability to
Meet
Performance
Expectations

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

Overestimate
Client Capacity
to Perform
O&M

Missed
Opportunity
for O&M
Contract

8

Improper
Equipment
Repurposing

Inability to
Meet
Performance
Expectations
Inaccurate
Prediction of
Service
Life/O&M
Schedule

Inability to Meet
Performance
Expectations

8

CustomerSpecified Issues

Changes to the
Size of the
Project Scope

Inability to Meet
Performance
Expectations

8

9

Missed
Opportunities
Failure to
Understand
Facility
Operations and
Client Goals

445

% Likelihood
End Effect

Intermediate
Effect 3

7

8

168

0.75

0.15

9

8

576

0.75

0.30

9

8

576

0.25

0.20

3

9

216

0.75

0.10

7

6

336

1

0.40

4

7

252

0.8

0.70

RPN

Intermediate
Effect 2

Detection

1C5

0.25

Intermediate
Effect 1

Severity

1C4

Overstatement of
Issues

Occurrence

Cause

%Likelihood
Cause

Table D-8 (cont’d)

Cost

1D4

ESCO Lack of
Experience with
Facility Type

1D5

ESCO Lack of
Experience with
Facility Type

1D6

ESCO Lack of
Experience with
Facility Type

1D7

ESCO Lack of
Experience with
Facility Type

0.8

0.8

0.8

0.8

0.8

9

9

9

Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals

Overestimate
Client Capacity
to Perform
O&M

Reduced Savings
Due to PoorlyPerformed O&M

Overestimate
Client Capacity
to Perform
O&M
Lack of
Foresight About
Future Building
Use and
Occupant
Changes
Lack of
Foresight About
Future Building
Use and
Occupant
Changes

Reduced Savings
Due to PoorlyPerformed O&M

Reduced
Project Value
for ESCO
Savings
Shortfall/
Erosion of
ESCO's
Margin

Improper Baseline

Savings
Shortfall/
Erosion of
ESCO's
Margin

Improper Retrofit
Design

Improper
Construction
Timing and
Phasing

Project Schedule
too Short

Overestimate
Productivity
Rates

Project Schedule
too Short

446

End Effect

Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

9

Failure to
Understand
Facility
Operations and
Client Goals

Intermediate
Effect 3

7

9

567

1

0.30

7

9

567

0.6

0.30

8

8

576

0.6

0.30

8

9

648

0.9

0.70

8

8

576

0.9

0.30

9

3

243

0.8

0.70

RPN

9

9

Intermediate
Effect 2

Detection

1D3

ESCO Lack of
Experience with
Facility Type

0.8

Intermediate
Effect 1
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals

Severity

1D2

ESCO Lack of
Experience with
Facility Type

Occurrence

Cause

%Likelihood
Cause

Table D-8 (cont’d)

Cost

1D10

ESCO Lack of
Experience with
Facility Type

1D11

ESCO Lack of
Experience with
Facility Type

1E1

Differing
Stakeholder
Needs

0.8

0.8

0.8

0.2

9

9

9

9

3

5

405

0.75

0.30

9

5

405

0.9

0.70

9

6

486

0.7

0.80

Legal
Impacts

6

9

486

0.6

0.05

Savings
Shortfall/
Erosion of
ESCO's
Margin

9

8

216

0.2

0.30

Intermediate
Effect 3

Overestimate
Productivity
Rates

Underestimate
Project Costs

Improper
Consideration
of Security
Concerns

Project Schedule
too Short

Improper
Consideration
of Security
Concerns

Underestimate
Project Costs

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

Improper
Consideration
of Security
Concerns

Occupant Use of
Equipment for
Other Purposes

Retrofit Design
Does Not
Satisfy All
Stakeholders

User Modification
of
ECMs/Override
Operating
Parameters

447

% Likelihood
End Effect

9

Intermediate
Effect 2

RPN

Detection

1D9

ESCO Lack of
Experience with
Facility Type

0.8

Intermediate
Effect 1
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Failure to
Understand
Facility
Operations and
Client Goals
Inconsistent
Information
About Facility
Operating
Parameters (e.g.,
too hot, too cold,
too much
airflow, not
enough airflow)

Severity

1D8

ESCO Lack of
Experience with
Facility Type

Occurrence

Cause

%Likelihood
Cause

Table D-8 (cont’d)

Cost

1E4

Differing
Stakeholder
Needs

1E5

Differing
Stakeholder
Needs

1F1

1F2

IGA Conducted
Too Quickly

IGA Conducted
Too Quickly

0.2

0.2

0.2

0.75

0.75

3

Codes and
Standards
Upgrades

Retrofit Design
Does Not
Satisfy All
Stakeholders

Comfort-Related
Callbacks after
Construction
Complete

3

Codes and
Standards
Upgrades

Increased
Project
Complexity

Controls Scope of
Work Lacks
Adequate Detail
and Specificity

Increased Project
Complexity

Retrofit Design
Does Not
Satisfy All
Stakeholders

Comfort-Related
Callbacks after
Construction
Complete

3

Increased Project
Complexity

Controls Scope
of Work Lacks
Adequate Detail
and Specificity

8

Miss Critical
Functions and
Savings
Opportunities

8

Miscount
Existing
Equipment (e.g.,
Lights)

3

448

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Intermediate
Effect 3

6

5

90

0.1

0.30

7

5

105

0.1

0.10

7

5

105

0.2

0.20

7

5

105

0.1

0.30

9

9

648

1

0.8

9

8

576

0.4

0.2

RPN

Intermediate
Effect 2

Detection

1E3

Differing
Stakeholder
Needs

0.2

Intermediate
Effect 1

Severity

1E2

Differing
Stakeholder
Needs

Occurrence

Cause

%Likelihood
Cause

Table D-8 (cont’d)

Cost

Energy Model
Calibration
Errors

8

Inaccurate
Baseline

Energy Model
Calibration
Errors
Energy Model
Calibration
Errors

1F3

IGA Conducted
Too Quickly

1F4

IGA Conducted
Too Quickly

1F5

IGA Conducted
Too Quickly

0.75

8

Missing Facility
Information

1F6

IGA Conducted
Too Quickly

0.75

8

Missing Facility
Information

1F7

Facility
Information
Unavailable/
Inaccurate

0.75

8

Inaccurate
Baseline

1F8

Facility
Information
Unavailable/
Inaccurate

0.75

8

Missing Facility
Information

0.75

0.75

Energy Model
Calibration
Errors

449

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

8

Miscount
Existing
Equipment (e.g.,
Lights)

Intermediate
Effect 3

7

7

392

0.7

0.2

7

7

392

0.9

0.2

9

3

216

0.9

0.2

9

5

360

0.9

0.3

8

5

320

0.9

0.6

8

5

320

0.9

0.6

RPN

Intermediate
Effect 2

Detection

Intermediate
Effect 1

Severity

Occurrence

Cause

%Likelihood
Cause

Table D-8 (cont’d)

Cost

9

2A5

Security Concerns

0.8

9

2B1

Reduced
Availability of
“Low Hanging
Fruit”

9

Increase in
Project
Complexity

Limited/Less
Attractive
Scope

2B2

Reduced
Availability of
“Low Hanging
Fruit”

Increase in
Project Costs

IGA Scope
Increase
(No Additional
Fee)

0.8

9

2A2

Security Concerns

0.8

9

2A3

Security Concerns

0.8

9

0.8

0.8

9

Change in
Design Intent
for Impacted
Equipment

Rework

Change in
Design Intent
for Impacted
Equipment

Equipment
Impacts

Total Costs of
Operation

Project
Schedule
Impacted

Delay in Accruing
Savings

450

Increased
Opportunity for
Diminished
Savings

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

0.8

Security Concerns

Intermediate
Effect 3

8

8

576

0.70

0.2

8

8

576

0.80

0.1

7

8

504

0.70

0.3

8

8

576

0.90

0.7

6

8

432

0.70

0.05

7

8

504

0.8

0.70

7

8

504

0.9

0.30

RPN

Security Concerns

2A1

Intermediate
Effect 2

Detection

2A4

Intermediate
Effect 1
Technology
Breakage/
Damage
Possibility by
Occupants
Technology
Breakage/
Damage
Possibility by
Occupants
Technology
Breakage/
Damage
Possibility by
Occupants
Technology
Breakage/
Damage
Possibility by
Occupants
Occupant Use of
Equipment for
Other Purposes

Cause

Severity

Occurrence

%Likelihood
Cause

Table D-8 (cont’d)

Cost

Project
Schedule
Impacted

10

Failure to Meet
Owner Project
Requirements

Rework

10

Failure to Meet
Owner Project
Requirements

Comfort
Complaints

10

Failure to Meet
Owner Project
Requirements

2CD3

2CD4

Operational
Concerns

0.75

8

2CD5

Safety Concerns

0.5

6

2E1

Unqualified
Subcontractors

2B3

2CD1

2CD2

0.8

0.9

0.9

0.9

0.5

6

Failure to Meet
Owner Project
Requirements
Failure to Meet
Owner Project
Requirements

Rework

Reduce Project
Scope
Safety Incident
Project
Schedule
Impacted

451

Delay in Accruing
Savings

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

9

Less Opportunity
for Non-Energy
Benefits (NEBs)

Reduced
Availability of
“Low Hanging
Fruit”
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation
Some ECMs (e.g.,
Lighting) are
Difficult to
Demonstrate
before Installation

Intermediate
Effect 3

3

5

135

0.8

0.10

7

8

560

0.6

0.60

7

5

350

0.6

0.60

7

8

560

0.9

0.60

8

9

576

1

0.40

9

8

432

0.5

0.70

3

6

108

0.5

0.10

RPN

Intermediate
Effect 2

Detection

Intermediate
Effect 1

Severity

Occurrence

Cause

%Likelihood
Cause

Table D-8 (cont’d)

Cost

0.5

6

2E3

Unqualified
Subcontractors

0.5

6

2E4

Unreliable
Subcontractors

2E5

Unreliable
Subcontractors

2E6

2F1

2F2

Unreliable
Subcontractors
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement

Unsafe Working
Conditions/Injuri
es
Unsafe Working
Conditions/Injuri
es

Safety Incident

Project Schedule
Impacted

Safety Incident

6

Rework

Project
Schedule
Impacted

0.5

6

Unsafe Working
Conditions/Injuri
es

Safety Incident

0.5

6

Project Schedule
Impacted

8

Require ESCO to
use Fixed
Overhead &
Profit Rates

Overstate
Project Soft
Costs

8

Divide
Procurement into
Separate Phases

Disconnect
Between IGA
and Retrofit
Design

0.5

0.75

0.75

452

Delay in Accruing
Savings

Project Schedule
Impacted

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Legal
Impacts
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Intermediate
Effect 3

3

6

108

0.1

0.30

5

6

180

0.2

0.30

5

6

180

0.7

0.70

5

6

180

0.2

0.20

9

6

324

0.75

0.60

9

2

144

0.75

0.40

7

6

336

0.1

0.50

RPN

Intermediate
Effect 2

Detection

2E2

Unqualified
Subcontractors

Intermediate
Effect 1

Severity

Occurrence

Cause

%Likelihood
Cause

Table D-8 (cont’d)

Cost

2G2

2G3

0.75

8

Incorrect Vendor
Selection

0.8

9

Increased O&M
Cost

9

Reduced
Equipment
Service Life

0.25

3

Uncertainty in
Factors Used to
Predict
Performance

0.25

3

Reduced Energy
Performance

10

Uncertainty in
Factors Used to
Predict
Performance

0.8

0.9

Reduced
Equipment
Service Life

Reduced
Energy
Performance

453

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Disconnect
Between IGA
and Retrofit
Design

Intermediate
Effect 3

7

8

448

0.75

0.60

9

5

405

1

0.30

7

5

315

0.2

0.10

6

6

108

0.25

0.30

6

8

144

0.9

0.25

8

8

640

0.9

0.70

RPN

Intermediate
Effect 2

Detection

2G1

Intermediate
Effect 1

Severity

2G5

ECM
Constructability
& Feasibility
Issues

2F3

Occurrence

2G4

Cause
Prepare Technical
Scope of Work
Based on
Standard Public
DBB Procurement
Select Cheapest
Available
Equipment to
Meet
Specifications
Select Cheapest
Available
Equipment to
Meet
Specifications
Actual Climate
Conditions
Outside Range of
Planned
Conditions
Actual Climate
Conditions
Outside Range of
Planned
Conditions

%Likelihood
Cause

Table D-8 (cont’d)

Cost

2G7

ECM Failure to
Perform as
Designed

0.75

8

Reduced Energy
Performance

2G8

ECM Failure to
Perform as
Designed

0.75

8

Rework

454

End Effect
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin
Savings
Shortfall/
Erosion of
ESCO's
Margin

% Likelihood
End Effect

Rework

Intermediate
Effect 3

6

5

300

0.2

0.20

9

5

360

1

0.70

9

5

360

1

0.80

RPN

10

Intermediate
Effect 2

Detection

0.9

Intermediate
Effect 1

Severity

2G6

ECM
Constructability
& Feasibility
Issues

Occurrence

Cause

%Likelihood
Cause

Table D-8 (cont’d)

Cost

Risk Scenario Maps

Figure D-1. Risk Scenario Map 1A

455

Figure D-2. Risk Scenario Map 1B

456

Figure D-3. Risk Scenario Map 1C

457

Figure D-4. Risk Scenario Map 1D

458

Figure D-5. Risk Scenario Map 1E

459

Figure D-6. Risk Scenario Map 1F

460

Figure D-7. Risk Scenario Map 2A

461

Figure D-8. Risk Scenario Map 2B

462

Figure D-9. Risk Scenario Map 2CD

463

Figure D-10. Risk Scenario Map 2E

464

Figure D-11. Risk Scenario Map 2F

465

Figure D-12. Risk Scenario Map 2G

466

APPENDIX E

RISK MODEL PILOT TEST - CASE STUDY DETAILS
Case Study Location Description
The Parnall Correctional Facility is located in Blackman Township, Jackson County, Michigan.
It is currently a minimum-security prison with a capacity for 1,696 prisoners. It was originally
part of the former State Prison of Southern Michigan, until elements of the larger facility were
closed and others were divided into new facilities, as shown in Figure E-1 (Michigan Department
of Correction 2014). Parnall consists of 47 buildings, constructed between 1925 and 2002, and
includes five housing units and a land area of 45 acres.

Figure E-1. Parnall Correctional Facility
Note: Numbers refer to building locations in Table E-2
(Aerial Photograph Source: Google Earth)
467

Energy Performance Contract Overview
The Michigan Department of Technology, Management, and Budget, in conjunction with the
Michigan Department of Corrections, executed three pilot energy performance contracting (EPC)
retrofits at selected correctional facilities in 2011, which were partially funded through the
American Recovery and Reinvestment Act (Michigan Department of Licensing and Regulatory
Affairs 2011). Parnall was one of the facilities selected for this project. The Parnall Correctional
Facility is part of a four-facility campus designated as SMR, consisting of Parnall, the G. Robert
Cotton Correctional Facility, the Charles E. Egeler Reception Guidance Center, and the Cooper
Street Correctional Facility. The State of Michigan determined that Parnall was responsible for
38% of the campus’ total utility consumption, based on building area (Energy Systems Group
2010).
The utility baseline was completed for the facility and showed and annual cost for energy and
water/wastewater of $2.91 million (Table E-1). With 1,696 inmates, the utility cost per inmate
was $1,716, far exceeding national, regional, and statewide averages (Stephan 2004). Stephan’s
(2004) reported 2001 costs data was calculated using the consumer price index to 2010 costs:
 National average utility cost - $979/inmate (2010 dollars)
 Midwest Region average utility cost/inmate - $1,017/inmate (2010 dollars)
 Michigan average utility cost/inmate - $967/inmate (2010 dollars)
 G. Robert Cotton utility cost/inmate was reported as $450 (Energy Systems Group 2010)
Table E-1. Baseline Energy and Water Consumption
Utility Type
Annual Usage
Annual Cost
Electricity
9,442,290 kWh
$741,220
Natural Gas
560,016 therms
$441,405
Steam
1,083,181 therms
$1,234,500
Steam Adjustment – Textile Building (24,000) therms
($27,353)
Water and Sewer
174 Mgal – water
$520,784
157 Mgal - sewer
Total Cost:
$2,910,556
(Source: Energy Systems Group 2010)
Retrofit activities were directed toward specific technologies in targeted buildings (Table E-2).
The building numbers are aligned with the labels in Figure E-1. Most of the energy conservation
measures (ECMs) targeted older buildings, many of which were still connected to the inefficient
central steam plant or utilized original, circa 1930s vintage, hydronic heating units. Additionally,
the cell blocks utilize a five tier design (Figure E-2) and are on an uncontrolled hydronic loop
with each block receiving the same inlet temperature. As a result, the cell blocks at the front of
the loop overheat in order to maintain adequate temperature for the blocks in the back of the loop
(Energy Systems Group 2010). Overheating is controlled through increased ventilation; however,
there is routinely a wide vertical temperature variation in the cell blocks which leads to inmate
grievances (Berghorn and Vallad 2013).
Table E-2 lists selected ECMs by the building or buildings in which they were installed;
however, ECMs #1, #4, and #5 are not included in this table. Savings resulting from a negotiated
steam purchase agreement were the subject of ECM #1, thus there was no building modification
468

involved. The lighting and water conservation retrofits in ECMs #3 and #4 are campus-wide
efforts; therefore, while the buildings in Table E-2 may have included these retrofit measures,
this would not be an exhaustive list. Parnall ECMs are described in Table E-3.

Figure E-2. Parnall Correctional Facility Cell Block 7
(Photo credit: Nick Dentamaro - Jackson Citizen Patriot)
Table E-2. Building Subject to EPC Retrofit Activities
Building
Year
Number Description
Built Gross SF ECMs Included
CB 8
Correctional Security
1930
47,000
ECM #8
CB 9
Correctional Security
1932
33,800
ECM #2
ECM #3
CB 10
Correctional Security
1932
47,000
ECM #2
ECM #3
16
Correctional Security
2002
23,939
ECM #3
28
MI State Industries - Textiles
1928
145,900
ECM #6
Manufacturing
29
MI State Industries – Dye Plant
1925
13,300
ECM #3
31
Shoe and Sign Factory
1937
103,100
ECM #3
ECM #7
32
Metal Furniture Factory
1930
173,800
ECM #3
71
Main Garage
1929
16,300
ECM #3
74
Maintenance
1967
7,200
ECM #3
79
Trusty Dining Hall
1955
15,200
ECM #3
ECM #9
80
Health Care Services
1999
9,060
ECM #3
213
Pole Storage
1990
5,200
ECM #3
469

Table E-3. ECM Descriptions
ECM # Description
ECM #1 Incinerator Steam Cost Savings – purchase steam from Jackson county at a price that
is below the cost to produce steam from the central plant.
ECM #2 New HVAC System for Cell Blocks 9 and 10 – Original hydronic piping and
radiators from the 1930s have exceeded their service life; replace with radiant
panels. Install roof-mounted constant volume air rotation systems for fresh air
ventilation and air heating.
ECM #3 Decentralize Building Utility Systems – Older, less efficient buildings remain
connected to the central steam power plant; replace with natural gas systems.
ECM #4 Interior Campus Lighting Retrofit – Replacement, relamping, reballasting, and
retrofitting with T8 lamps, high output T5 lamps, and high efficiency ballasts.
ECM #5 Campus Water Conservation Retrofit – Retrofit cells with electronic plumbing
controls and install more efficient fixtures.
ECM #6 Prepare Textiles Building for Abandonment – Decommission all HVAC systems.
ECM #7 Minimize Heating 2nd Floor of Building 31 – Minimize heating in an unoccupied
space; retrofit systems based on ECM #3 to provide base level of heating to protect
fore suppression system.
ECM #8 Minimize Heating of Cell Block 8 – Minimize heating in an unoccupied space;
maintain base level of heating to protect fire suppression system and due to adjacent
occupied cell blocks; contingent on ECM #3 and utilizes elements of ECM #2.
ECM #9 Replace Cafeteria Dishwasher – Replace steam-heated model with natural gas.
In addition to the nine installed ECMs, 10 candidate retrofit measures were also considered, as
described in Table E-4. These ECMs typically had rather lengthy simple payback periods or
anticipated energy savings that were difficult to calculate. While this was not always the primary
factor in deciding not to implement, it was a significant concern.
Table E-4. Proposed ECMs not Selected for Project
ECM Description
Simple Payback Period
Solar-Assisted Domestic Hot Water Heating
377 years
Roof Replacement
355 years
Window Replacement
222 years
Integrated Building Automation System
Difficult to calculate energy savings
Exterior Lighting Retrofit
19 years
Add Air Conditioning to Lower Level Bldg 31 No savings due to new energy end-use
Exterior Security Lighting (LEP or LED)
40 years
Creamery Steam Process and Bldg 218 Boiler
16 years
Chapel Building Automation System
44 years
Electrical Power Monitoring
Difficult to calculate energy savings
(Source: Energy Systems Group 2010)
The total costs and savings associated with the nine selected ECMs are detailed in Table E-5.
The project financial summary is provided in Table E-6 and the baseline condition and annual
expected energy and water savings are detailed in Table E-7.
470

Commissioning

Table E-5. Proposed Costs and Savings for Recommended ECMs
Project Costs

Annual Project Savings

a,b

Energy
Cost
Avoidance
$724,170
$158,466
$667,177
$65,396
$205,911
$30,819
$7,560
$13,252
$8,523
$1,881,274

Total Cost
ECM
Design
Material
Labor
ECM #1
$ $ $ $ $ ECM #2
$339,818 $2,572,907 $2,281,635 $22,608
$5,216,968
ECM #3
$301,155 $2,280,170 $2,022,037 $49,655
$4,653,017
ECM #4
$50,440
$381,905
$338,672
$ $771,017
ECM #5
$121,223
$917,832
$813,926
$ $1,852,981
ECM #6
$4,757
$36,021
$31,943
$ $72,721
ECM #7
$ $ $ $ $ ECM #8
$14,661
$111,003
$98,436
$5,201
$229,301
ECM #9
$4,324
$32,736
$29,032
$ $66,092
Subtotal $836,378 $6,332,574 $5,615,681 $77,464
$12,862,097
Audit Cost
$29,528
Total
$12,891,626
Notes:
a\Costs for training and warranties were not included in this analysis.
b\ Annual maintenance costs were anticipated to be the same or less than the existing cost.
(Adapted from Energy Systems Group 2010)

471

Energy
Simple
Avoidance
Payback
0
0 yrs
139,005 therms 32.9 yrs
421,653 therms
7.0 yrs
833,067 kWh
11.8 yrs
68,797 kGal
9.0 yrs
39,100 therms
2.4 yrs
9,592 therms
0.0 yrs
16,813 therms
17.3 yrs
10,824 therms
7.8 yrs
6.8

Service
Life
20 yrs
30 yrs
30 yrs
15 yrs
15 yrs
30 yrs
30 yrs
30 yrs
20 yrs

Table E-6. Project Financial Summary
Total Project Cost: $12,891,626
Incentives/Rebates:
$0
Committed Capital Funding:
$0
Financed Investment Cost: $12,891,626
Rate of Financing:
4.50%
Term of Financing:
7 years
Total Savings over Term: $15,646,197
Annual Utility Rate Increase:
3.40%
Annual Operational Savings Increase:
3.40%
Annual Cost of Capital Increase:
0.00%
Total Net Cash Flow:
$545,931
Simple Payback:
6.80 years
(Source: Energy Systems Group 2010)
Table E-7. Base Year Condition and Expected Energy and Water Savings
Utility Type
Base Year Utility Usage Base Year Utility Cost Annual Savings
Electricity
9,442,290 kWh
$741,220
$65,396
Natural Gas
560,016 therms
$441,405
N/A
Steam
1,083,181 therms
$1,234,500
$885,777
Steam Adjustment – (24,000) thermsa
($27,353)
Note b
Textile Building
Steam Cost
N/A
N/A
$724,170
Adjustment
Water and Sewer
Water: 174 Mgal
$520,784
$205,911
Sewer: 157 Mgal
c
Totals
$2,910,556
$1,881,254
Notes:
a\ This represents the fuel load for operating the
b\ The savings attributed to the space heating fuel load (46,000 therms/year or $30,819/year) is
included in the “Steam” annual savings in the immediately preceding row.
c\ Annual savings figures are slightly mismatched between Table E-5 and this table, as a result
of rounding errors in the calculations
(Source: Energy Systems Group 2010)

472

REFERENCES

473

REFERENCES

Abaza, H. (2008). An interactive design advisor for energy efficient buildings. Journal of Green
Building 3(1): 112-125.
Abdelgawad, M. and Fayek, A.R. (2012). Comprehensive hybrid framework for risk analysis in
the construction industry using combined failure mode and effect analysis, fault trees,
even trees, and fuzzy logic. Journal of Construction Engineering and Management 138:
642-651.
Abdelgawad, M. and Fayek, A.R. (2010). Risk management in the construction industry using
combined fuzzy FMEA and fuzzy AHP. Journal of Construction Engineering and
Management 136: 1028-1036.
Abramson, Herman, and Wong (2005). Interactive Web-Based Owning and Operating Cost
Database - Final Report. ASHRAE Research Project 1237-TRP. ASHRAE: Atlanta,
Georgia, USA.
Adams, F.K. (2008). Construction Contract Risk Management: A Study of Practices in the
United Kingdom. Cost Engineering 50(1): 22-32.
AEPCA (2000). A Best Practice Guide to Energy Performance Contracts: Reducing operating
costs through guaranteed outcomes. Castle Hill, New South Wales, Australia.
Agresti, A. and Findlay, B. (2009). Statistical Methods for the Social Sciences, 4th ed. Pearson
Prentice Hall: Upper Saddle River, New Jersey, USA.
Ahmed, A., Kayis, B., and Amornsawadwatana, S. (2007). A review of techniques for risk
management in projects. Benchmarking: An International Journal 14(1): 22-36.
AIAG (2008). Potential Failure Mode and Effects Analysis, 4th Edition. Automotive Industry
Action Group. Southfield, Michigan, USA.
Akintoye, A., Taylor, C., and Fitzgerald, E. (1998). Risk analysis and management of private
finance initiative projects. Engineering, Construction and Architectural Management
5(1): 9-21.
Akintoye, A.S. and MacLeod, M.J. (1997). Risk analysis and management in construction.
International Journal of Project Management 5(1): 31-38.
Al-Hosany, N. and Elkadi, H. (2002). Sustainability approaches for incarceration architecture.
Renewable and Sustainable Energy Reviews 6: 457–470.

474

Allen, P., Green, D., and Lewis, J. (2006).Using the Web for Energy Data Acquisition &
Analysis. In Hansen, S.J. Performance Contracting: Expanding Horizons. The Fairmont
Press, Inc.: Lilburn, Georgia, USA.
American Correctional Association (2012). 2012 Standards Supplement. American Correctional
Association. Alexandria, Virginia, USA.
Ang, A.H.-S. and De Leon, D. (2005). Modeling and analysis of uncertainties for risk informed
decisions in infrastructures engineering. Structure and Infrastructure Engineering:
Maintenance, Management, Life-Cycle Design and Performance 1(1): 19-31.
Appleman, T., Owsenek, B., Clough, D., and Hanna, M. (2010). Energy savings through
performance contracting at wastewater treatment plants. In Proceedings of WEFTEC
2010. October 2-6, 2010. New Orleans, Louisiana, USA.
Arashpour, M. and Arashpour, M. (2012). A collaborative perspective in green construction risk
management. In Proceedings of the 37th Annual Conference of the Australasian
Universities Building Educators Association (AUBEA). July 4-6, 2012. Sydney,
Australia.
ASHRAE (2014). ASHRAE: Service Life and Maintenance Cost Database. Interactive database
accessed from http://xp20.ashrae.org/publicdatabase/default.asp on March 14, 2013.
ASHRAE (2012). Advanced Energy Design Guide for Large Hospitals: Achieving 50% Energy
Savings Toward a Net Zero Energy Building. ASHRAE Design Guide. Atlanta, Georgia,
USA.
ASHRAE (2011). Procedures for Commercial Building Energy Audits (2nd Ed). ASHRAE:
Atlanta, Georgia, USA.
Asiedu, Y. and Gu, P. (1998) Product life cycle cost analysis: state of the art review.
International Journal of Production Research 36(4): 883-908.
Assaf, S.A. and Al-Hejji, S. (2006). Causes of delay in large construction projects. International
Journal of Project Management 24: 349–357.
Azen, S.P., Niland, J., Hayward, S.B., Liu, K-S., Redinger, C., and Porter, P. (1992). Estimation
of the cost of asbestos abatement in public buildings in the state of California: an
application of the bootstrap. Environmetrics: 3(2): 193-209.
Baechler, M., Strecker, C., and Shafer, J. (2011). A Guide to Energy Audits. PNNL-20956,
Pacific Northwest National Laboratory. Richland, Washington, USA.
Baker, D. and Forbes, F.W. (1997). To build or not to build: aging prisons force states to
examine capital projects. Corrections Today 59(2): 84-86.
475

Banaitiene, N. and Banaitis, A. (2012). Risk Management in Construction Projects. In
N. Banaitiene (Ed.) Risk Management - Current Issues and Challenges. InTech: New
York, New York, USA.
Barbose, G.L., Goldman, C.A., Hoffman, I.M., and Billingsley, M. (2013). The Future of Utility
Customer Funded Energy Efficiency Programs in the United States: Projected Spending
and Savings to 2025. LBNL-5803E, Ernest Orlando Lawrence Berkeley National
Laboratory. Berkeley, California, USA.
Barbour, R.S. (2001). Checklists for improving rigour in qualitative research: A case of the tail
wagging the dog? British Medical Journal 322: 1115-1117.
Bednarz, S., and Marriott, D. (1988). Efficient analysis for FMEA [Space Shuttle reliability]. In
Proceedings of the Annual Reliability and Maintainability Symposium, January 26-28,
1988. Los Angeles, California, USA.
Berghorn (2012). The energy performance contracting process for retrofits at correctional
facilities. Presented at the 2nd National Symposium on Sustainability in Corrections.
October 29-November 1, 2012. Indianapolis, Indiana, USA.
Berghorn, G.H. and Vallad, E. (2013). Energy performance projects yield greening and security
benefits. Corrections Today 75:(1): 44-48.
Berny, J. and Townsend, P.R.F. (1993). Macrosimulation of project risks – a practical way
forward. International Journal of Project Management 11(4): 201-208.
Bertoldi, P., Hinnells, M., and Rezessy, S. (2006). Liberating the power of energy services and
ESCOs in a liberalised energy market. In Proceedings of the International Energy
Efficient Domestic Appliances and Lighting Conference (EEDAL06). June 21-23, 2006.
London, England, United Kingdom.
Bertoldi, P. and Rezessy, S. (2005). Energy service companies in Europe; Status report 2005.
EUR 21646 EN, European Commission, Institute for Environment and Sustainability,
Renewable Energies Unit. Ispra, Varese, Italy.
Bharvirkar, R., Goldman, C., Gilligan, D., Singer, T.E., Birr, D., Donahue,P., and Serota, S.
(2008). Performance Contracting and Energy Efficiency in the State Government Market.
LBNL-1202E, Ernest Orlando Lawrence Berkeley National Laboratory. Berkeley,
California, USA.
Bhattacharjee, S., Ghosh, S., and Young-Corbett, D. (2010). Energy service performance
contracting in construction. In Proceedings of the 46th Annual International Conference
of Associated Schools of Construction. April 7-10, 2010. Boston, Massachusetts, USA.

476

Birnie, J. (1993) A Behavioural Study Using Decision Analysis of Building Cost Prediction by
Chartered Quantity Surveyors. D.Phil. Thesis, University of Ulster - Jordanstown,
Northern Ireland, United Kingdom.
Birr, D. and Singer, T.E. (2008). NAESCO Analysis of Non-Energy Benefits of Efficiency
Retrofits for ESCOs and ESCO Customers. National Association of Energy Service
Companies (NAESCO). Washington, DC, USA.
Bolstad, W.M. (2013). Introduction to Bayesian Statistics, 2nd ed. Wiley: New York, New York,
USA.
Bonano, E.J., Hora, S.C., Keeney, R.L., and von Winterfeldt, D. (1990). Elicitation and Use of
Expert Judgment in Performance Assessment for High-Level Radioactive Waste
Repositories. NUREG/CR-5411 – SAND89-1821, Sandia National Laboratories.
Albuquerque, New Mexico, USA.
Bowles, J.B. (2004). An assessment of RPN prioritization in a failure modes effects and
criticality analysis. Journal of the Institute of Environmental Sciences and Technology
47(1): 51-56.
Bowles, J.B. (1998). The new SAE FMECA standard. In Proceedings of the Annual Reliability
and Maintainability Symposium. January 19-22, 1998. Anaheim, California, USA.
BRE Global (2012). Scheme Document SD 5054 - BREEAM Prisons 2008, Issue 4.1, 5/25/2012.
Watford, England, United Kingdom.
Brink, G. (2012). Expanding risk analysis into the world of life cycle costing. Presented at the
Palisade 2012 Risk Conference - Risk Analysis, Applications & Training. November 7-8,
2012. Las Vegas, Nevada, USA.
Bull, J.W. (1993). Life Cycle Costing for Construction. Blackie Academic & Professional:
London, England, United Kingdom.
Burpee, H., Hatten, M., Loveland, J., and Price, S. (2009). High performance hospital
partnerships: reaching the 2030 Challenge and improving the health and healing
environment. In Proceedings of the Annual American Society for Healthcare Engineering
(ASHE) Conference on Health Facility Planning, Design and Construction (PDC). March
8-11, 2009. Phoenix, Arizona, USA.
Cagno, E., Worrell, E., Trianni, A., and Pugliese, G. (2013). A novel approach for barriers to
industrial energy efficiency. Renewable and Sustainable Energy Reviews 19: 290-308.
California Department of General Services (2011). 2003-2010 State-Owned Buildings Energy
Use and Benchmarking Data (9/21/2011), http://www.documents.dgs.ca.gov/Green/
buildingsenergyuse.pdf - downloaded on November 13, 2012.
477

California Public Utilities Commission (2014). Database for Energy Efficient Resources.
Downloaded from http://www.deeresources.com/ on March 14, 2014.
Carbone, T.A. and Tippett, D.D. (2004). Project risk management using the project risk FMEA.
Engineering Management Journal 16(4): 28-35.
Carmignani, G. (2009). An integrated structural framework to cost-based FMECA: the prioritycost FMECA. Reliability Engineering and System Safety 94: 861-871.
Carter, T. and Keeler, A. (2008). Life-cycle cost–benefit analysis of extensive vegetated roof
systems. Journal of Environmental Management 87: 350–363.
Cash, D.W., Clark, W.C., Alcock, F., Dickson, N.M., Eckley, N., Guston, D.H., Jäger, J. and
Mitchell, R.B. (2003). Knowledge systems for sustainable development. Proceedings of
the National Academy of Sciences of the United States of America 100 (14): 8086-8091.
CCI (2009a). An Introduction to EPC: EPC Toolkit for Higher Education. Clinton Foundation,
Clinton Climate Initiative, and American College & University Presidents Climate
Commitment. New York, New York, USA.
CCI (2009b). Energy Performance Contract Financing Options. Clinton Foundation, Clinton
Climate Initiative, and American College & University Presidents Climate Commitment.
New York, New York, USA.
CEC and CBC (2001). Energy Efficiency Design Guide for California Detention Facilities.
P400-01-010, California Energy Commission and California Board of Corrections.
Sacramento, California, USA.
Chapman, R.J. (2001). The controlling influences on effective risk identification and assessment
for construction design management. International Journal of Project Management 19:
147-160.
Chapman, R.J. (1998). The effectiveness of working group risk identification and assessment
techniques. International Journal of Project Management 16(6): 333-343.
Chen, C-C. and Zhang, H. (2012). A FMEA-aided equipment life-cycle-cost measurement
system. In Proceedings of the 3rd International Asia Conference on Industrial
Engineering and Management Innovation. July 2012. Beijing, China.
Chew, M. Y. L., Tan, S.S., and Kang, K.H. (2004). Building maintainability - review of state of
the art. Journal of Architectural Engineering 10(3) 80–87.
Chidiac, S.E., Catania, E.J.C., Perry, H.L., Morofsky, E., and Foo, S. (2012). Computational
tools for selecting energy conservation measures for retrofitting existing office buildings.
Canadian Journal of Civil Engineering 40(5): 445-459.
478

Chok, N.S. (2010). Pearson’s Versus Spearman’s and Kendall’s Correlation Coefficients for
Continuous Data. MS Thesis, University of Pittsburgh, Pittsburgh, Pennsylvania, USA.
Cobb, B.R. and Menas, A.J. (2009). Performance of TIAA-CREF’s recommended portfolios.
Journal of Personal Finance 8 23–48.
Commission on Enhancing Agency Outcomes (2010). Final Report to the Governor, President
Pro Tempore of the Senate, and the Speaker of the House - State of Connecticut.
Connecticut General Assembly. Hartford, Connecticut, USA.
Cooper, D.F. and Chapman, C.B. (1987). Risk Analysis for Large Projects: Models, Methods
and Cases. Wiley: New York, New York, USA.
Cost-Effective Governmental Energy Use Act (2012). MCL 18.1711-18.725.
Crosetto, P. and Filippin, A. (2013). A Theoretical and Experimental Appraisal of Five Risk
Elicitation Methods. Jena Economic Research Papers #2013-009, Friedrich Schiller
University and the Max Planck Institute of Economics. Jena, Germany.
Curkovic, S., Scannell, T., and Wagner, B. (2013). Using FMEA for supply chain risk
management. Modern Management Science & Engineering 1(2): 251-265.
Da-li, G. (2009). Energy service companies to improve energy efficiency in China:
barriers and removal measures. Procedia Earth and Planetary Science 1 (2009) 1695–
1704.
Dailey, K.W. (2004). The FMEA Pocket Handbook. DW Publishing Company: Port Saint Lucie,
Florida, USA.
Dallao, M. (1997). Prison construction trends: states building fewer but larger facilities.
Corrections Today 59(2): 70-73.
De Zoysa, S. and Russell, A.D. (2003a). Knowledge-based risk identification in infrastructure
projects. Canadian Journal of Civil Engineering 30: 511-522.
De Zoysa, S. and Russell, A.D. (2003b). Structuring of risk information to assist in knowledgebased information of the life cycle risks of civil engineering projects. In Proceedings of
the 5th Construction Specialty Conference of the Canadian Society for Civil Engineering.
June 4-7, 2003. Moncton, New Brunswick, Canada.
Dell’Isola, A.J. and Kirk, S.J. (2003). Life Cycle Costing for Facilities. Reed Construction Data:
Kingston, Massachusetts, USA.
Deng, Q., Zhang, L., Cui, Q., and Jiang, X. (2014). A simulation-based decision model for
designing contract period in building energy performance contracting. Building and
Environment 71: 71-80.
479

Deng, Q. (2011). Guarantee Design on Energy Performance Contracts Under Uncertainty. MS
Thesis, University of Maryland College Park, College Park, Maryland, USA.
DeVon, H. A., Block, M. E., Moyle-Wright, P., Ernst, D. M., Hayden, S. J., Lazzara, D. J.,
Savoy, S.M., Kostas-Polston, E. (2007). A Psychometric Toolbox for Testing Validity
and Reliability. Journal of Nursing Scholarship 39(2): 155-64.
Dey, P.K. and Ogunlana, S.O. (2004). Selection and application of risk management tools and
techniques for build-operate-transfer project. Industrial Management + Data Systems
104(3/4): 334-346.
Dey, P., Tabucanon, M.T., and Ogunlana, S.O. (1994). Planning for project control through
risk analysis: a petroleum pipeline-laying project. International Journal of Project
Management 12(1): 23-33.
Dhillon, B.S. (1992). Failure modes and effects analysis – bibliography. Microelectronics
Reliability 32(5): 719-731.
Diab, M.F. (2012). Integrating risk assessment in planning for sustainable infrastructure projects.
In Proceedings of the 2012 International Conference on Sustainable Design, Engineering,
and Construction. November 7-9, 2012. Fort Worth, Texas, USA.
Doloi, H. (2012). Understanding impacts of time and cost related construction risks on
operational performance of PPP projects. International Journal of Strategic Property
Management 16(3): 316-337.
Dong, B., Kennedy, C., and Pressnail, K. (2005). Comparing life cycle implications of building
retrofit and replacement options. Canadian Journal of Civil Engineering 32: 1051-1063.
Drumheller, S. C. and Wiehagen, J. (2004). Strategies for Energy-Efficient Remodeling: SEER
2003 Case Study Report. Building America Technical Report NREL/SR-550-36474.
Duah, D.Y.A. (2014). Intelligent Decision Support System Framework for Energy Retrofits in
Existing Homes. PhD Thesis, Michigan State University, East Lansing, Michigan, USA.
Dunn & Bradstreet. (2014). About the D-U-N-S number. Downloaded from http://fedgov.
dnb.com/webform/pages/dunsnumber.jsp on April 24, 2014.
Edwards, P.J. and Bowen, P.A. (1998). Risk and risk management in construction: a review and
future directions for research. Engineering, Construction and Architectural Management
5(4): 339-349.
Emblemsvag, J. (2001). Activity-based life cycle costing. Managerial Auditing Journal 16(1):
17-27.

480

Energy Services Coalition (2011). Model Procurement and Contracting Documents – Request for
Proposals for Energy Performance Contracting Services for End-Users. Phoenix,
Arizona, USA.
Energy Systems Group (2010). Draft Energy Audit Report: Parnall Correctional Facility, August
25, 2010.
Engel, D.W. and Dalton, A.C. (2012). CCSI Risk Estimation: An Application of Expert
Elicitation. PNNL-21785, Pacific Northwest National Laboratory. Richland, Washington,
USA.
Esmaeili, B. and Hallowell, M. (2013). Integration of safety risk data with highway
construction schedules. Construction Management and Economics 31(6): 528-541.
European Commission (2014). Energy Performance Contracting. European Commission Joint
Research Centre – Institute for Energy and Transport. Downloaded from http://iet.
jrc.ec.europa.eu/energyefficiency/european-energy-service-companies/energyperformance-contracting on March 14, 2014.
Evans, C. (1997). The use of consensus methods and expert panels in pharmacoeconomic
studies: practical applications and methodological shortcomings. Pharmacoeconomics
12(2): 121 -12
EVO (2012). International Performance Measurement and Verification Protocol: Concepts and
Options for Determining Energy and Water Savings - Volume 1. EVO 10000 – 1:2012.
Efficiency Valuation Organization. Washington, DC, USA.
Fehr, E. and Tyran, J-R. (2001). Does money illusion matter? The American Economic Review
91(5): 1239-1262.
FEMP (2007). Introduction to Measurement & Verification for DOE Super ESPC Projects. U.S.
Department of Energy Federal Energy Management Program. Washington, D.C., USA.
FEMP (2002). Detailed Guidelines for FEMP M&V Option A. U.S. Department of Energy
Federal Energy Management Program. Washington, D.C., USA.
Ferreira, J., Duarte Pinheiro, M., and de Brito, J. (2013). Refurbishment decision support tools:
A review from a Portuguese user’s perspective. Construction Building Materials 49: 425447.
Forbes, D.R., Smith, S.D., and Horner, R.M.W. (2008). A comparison of techniques for
identifying risks in sustainability assessment of housing. In Proceedings of the 24th
Annual ARCOM Conference. September 1-3, 2008. Cardiff, Wales, United Kingdom.

481

Forster, H.J., Wallace, P., and Dahlberg, N. (2013). 2012 State of the Efficiency Program
Industry: Budgets, Expenditures, and Impacts - Annual Industry Report. Consortium for
Energy Efficiency. Boston, Massachusetts, USA.
Franceschini, F. and Galetto, M. (2001). A new approach for evaluation of risk priorities of
failure modes. International Journal of Production Research 39(13): 2991-3002.
Frangopol, D.M. and Liu, M. (2007). Maintenance and management of civil infrastructure based
on condition, safety, optimization, and life-cycle cost. Structure and Infrastructure
Engineering 3(1): 29 – 41.
Fuller, S.K. (2005). Guidance on Life-Cycle Cost Analysis Required by Executive Order 13123.
National Institute of Standards and Technology. Gaithersburg, Maryland, USA.
Fuller, S.K. and Petersen, S.R. (1995). Life-Cycle Costing Manual for the Federal Energy
Management Program. NIST Handbook 135, National Institute of Standards and
Technology. Washington, DC, USA.
Furuta, H., Kameda, T., Fukuda, Y., and Frangopol, D.M. (2004). Life-cycle cost analysis for
infrastructure systems: life-cycle cost vs. safety level, vs. service life. In Proceedings of
the 3rd IABMAS Workshop on Life-Cycle Cost Analysis and Design of Civil
Infrastructures Systems. March 24-26, 2003. Lausanne, Switzerland.
Ganji, A.R. and Gilleland, B. (2002). Investment grade energy audit. In Proceedings of the 25th
World Energy Engineering Congress. October 9-11, 2002. Atlanta, Georgia, USA.
GAO (2012). Bureau of Prisons: Growing Inmate Crowding Negatively Affects Inmates, Staff,
and Infrastructure. GAO-12-743, U.S. Government Accountability Office. Washington,
DC, USA.
GAO (1991). Prison Costs: Opportunities Exist to Lower the Cost of Building Federal Prisons.
GAO GGD-92-3, U.S. Government Accountability Office. Washington, DC, USA.
Garber, D., Choudhary, R., and Soga, K. (2013). Risk based lifetime costs assessment of a
ground source heat pump (GSHP) system design: methodology and case study. Building
and Environment 60: 66-80.
Gargama, H. and Chaturvedi, S.K. (2011). Criticality assessment models for failure mode effects
and criticality analysis using fuzzy logic. IEEE Transactions on Reliability 60(1): 102110.
Gilchrist, W. (1993). Modelling failure modes and effects analysis. International Journal of
Quality and Reliability Management 10(5): 16-23.

482

Ginoza, M., Ruano, R., Woodford, J., and Butler, D. (2003). Renovating historic structures:
seismic retrofits at Folsom and San Quentin state prisons. Corrections Today 65(4): 7881, 117.
Glaze, L.E. and Parks, E. (2012). Correctional Populations in the United States, 2011. NCJ
239972, Bureau of Justice Statistics. Washington, DC, USA.
Goh, C.S. and Abdul-Rahman, H. (2013). Applying risk management workshop for a public
construction project: a case study. Journal of Construction Engineering and Management
139: 572–580.
Goldman, C., Hopper, N., and Osborn, J. (2005). Review of U.S. ESCO Industry Market
Trends: An Empirical Analysis of Project Data. LBNL-52320, Ernest Orlando Lawrence
Berkeley National Laboratory. Berkeley, California, USA.
Goldman, C.A., Osborn, J.G., Hopper, N.C., and Singer, T.E. (2002). Market Trends in the U.S.
ESCO Industry: Results from the NAESCO Database Project. LBNL-49601, Ernest
Orlando Lawrence Berkeley National Laboratory. Berkeley, California, USA.
Gordon, T.J. (1992). The methods in futures research. Annals of the American Academy of
Political and Social Science, 522(1): 25-35.
Griffin, J.S. (2008). Impact of Weather Variations on Energy Consumption Efforts at U.S. Air
Force Bases. MS Thesis, Air Force Institute of Technology, Wright-Patterson Air Force
Base, Ohio, USA.
Grimaldi, S., Rafele, C., and Cagliano, A.C. (2012). A Framework to Select Techniques
Supporting Project Risk Management. In N. Banaitiene (Ed.) Risk Management - Current
Issues and Challenges. InTech: New York, New York, USA.
Grinstead, C.M. and Snell, J.L. (1997). Introduction to Probability. American Mathematical
Society: Providence, Rhode Island, USA.
Gu, L., Gu, D., Lin, B., Huang, M., Gai, J., and Zhu, Y. (2007). Life cycle green cost assessment
method for green building design. In Proceedings of Building Simulation 2007 – the
International Building Performance Simulation Association. September 3-6, 2007.
Beijing, China.
Hallowell, M.R. and Gambatese, J.A. (2010). Qualitative research: application of the Delphi
method to CEM research. Journal of Construction Engineering and Management 136: 99107.

483

Hamilton, M. C., Goldsmith, W., Harmon, R., Lewis, D., Srdjevic, B., Goodsite, M., Lambert, J.
H., and Macdonell, M. (2014). Sustainable Water Resources Management: Challenges
and Methods. In I. Linkov (Ed.) Sustainable Cities and Military Installations, NATO
Science for Peace and Security Series C: Environmental Security 2014. Springer: New
York, New York, USA.
Hanley, N. and Spash, C. (1993). Cost–Benefit Analysis and the Environment. Edward Elgar”
Cheltenham, England.
Hansen, S.J. (2013). Personal communication with Shirley J. Hansen, PhD on February 6, 2013.
Hansen, S.J. (2006). Performance Contracting: Expanding Horizons. The Fairmont Press, Inc.:
Lilburn, Georgia, USA.
Hansen, S.J. (2002). Manual for Intelligent Energy Services. The Fairmont Press, Inc.: Lilburn,
Georgia, USA.
Hansen, S. and Brown, J.W. (2004). Investment Grade Energy Audit: Making Smart Energy
Choices. The Fairmont Press, Inc.: Lilburn, Georgia, USA
Harvey, L.D.D. (2009). Reducing energy use in the buildings sector: measures, costs, and
examples. Energy Efficiency 2:139-163.
Hart, B.T. and Pollino, C.A. (2008). Increased use of Bayesian network models will improve
ecological risk assessments. Human and Ecological Risk Assessment: An International
Journal 14(5): 851-853.
Hasson, F., Keeney, S., and McKenna, H. (2000). Research guidelines for the Delphi survey
Technique. Journal of Advanced Nursing 32(4): 1008-1015 .
Heo, J., Choudhary, R., and Augenbroe, G.A. (2012). Calibration of building energy models for
retrofit analysis under uncertainty. Energy and Buildings 47: 550–560.
Herring, H. and Roy, R. (2007). Technological innovation, energy efficient design and the
rebound effect. Technovation 27: 194-203.
Hertwich, E.G. (2005). Consumption and the rebound effect: an industrial ecology perspective.
Journal of Industrial Ecology 9(1-2):85-98.
Hertz, D.B. and Thomas, H. (1983) Risk Analysis and its Application. Wiley: Chichester,
England, United Kingdom.
Hopper, N., Goldman, C., and Birr, D. (2004). The federal market for ESCO services: how does
it measure up? LBNL-54952, Ernest Orlando Lawrence Berkeley National Laboratory.
Berkeley, California, USA.
484

Hopper, N., Goldman, C., Gilligan, D., Singer, T.E., and Birr, D. (2007). A Survey of the U.S.
ESCO Industry: Market Growth and Development from 2000 to 2006. LBNL-62679,
Ernest Orlando Lawrence Berkeley National Laboratory. Berkeley, California, USA.
Hopper, N., Goldman, C., McWilliams, J., Birr, D., and Stoughton, K.M. (2005). Public and
Institutional Markets for ESCO Services: Comparing Programs, Practices and
Performance. LBNL-55002, Ernest Orlando Lawrence Berkeley National Laboratory.
Berkeley, California, USA.
Horsley, A., France, C., and Quatermass, B. (2003). Delivering energy efficient buildings: A
design procedure to demonstrate environmental and economic benefits. Construction
Management and Economics 21: 345-356.
Hsu, C-C. and Sandford, B.A. (2007). The Delphi technique: making sense of consensus.
Practical Assessment, Research & Evaluation 12(10): 1-8.
Hughes. P.J. and Muessel, T.S. (2000). Energy savings performance contracting - experience of
the U.S. Department of Energy Federal Energy Management Program. In Proceedings of
the 2000 American Council for an Energy-Efficient Economy Summer Study on Energy
Efficiency in Buildings. August 20-25, 2000. Pacific Grove, California, USA.
Hughes, P.J., Shonder, J.A., Sharp, T., and Madgett, M. (2003). Evaluation of Federal Energy
Savings Performance Contracting - Methodology for Comparing Processes and Costs of
ESPC and Appropriations-Funded Energy Projects. ORNL/TM-2002/150, Oak Ridge
National Laboratory. Oak Ridge, Tennessee, USA.
IBM (2014). SPSS Statistics (Version 22) [Software]. Available from http://www-01.ibm.com/
software/analytics/spss/products/statistics/.
ICF and NAESCO (2007). Introduction to Energy Performance Contracting. U.S. Environmental
Protection Agency, ENERGYStar Buildings Program. Washington, DC.
IFC (2011). IFC Energy Service Company Market Analysis. Ref. 5712. International Finance
Corporation – World Bank Group. Washington, DC, USA.
Imbeah, W. and Guikema, S. (2009). Managing construction projects using the advanced
programmatic risk analysis and management model. Journal of Construction Engineering
and Management 135: 772-781.
IPMVP Committee (2002). International Performance Measurement & Verification Protocol:
Concepts and Options for Determining Energy and Water Savings - Volume I. DOE/GO102002-1554. International Performance Measurement and Verification Protocol
Committee. Washington, DC, USA.

485

Irwin, J., Rhodes-Conway, S., White, S.L., and Rogers, J. (2011). Making MUSH Energy
Efficient: Energy Efficiency in the Governmental and Institutional Sector. Center on
Wisconsin Strategy. Madison, Wisconsin, USA.
Jackson, J. (2010). Promoting energy efficiency investments with risk management
decision tools. Energy Policy 38: 3865-3873.
Jairath, N. and Weinstein, J. (1994). The Delphi methodology (Part one): a useful administrative
approach. Canadian Journal of Nursing Administration 7(3): 29-42.
Jannadi, O.A. and Almishari, S. (2003). Risk assessment in construction. Journal of Construction
Engineering and Management 129: 492-500.
Jefferies, M. and Chen, E. (2012). Identifying risk factors of BOOT procurement: a case study of
Stadium Australasia. The Australasian Journal of Construction Economics and Building
4(1): 11-20.
Jennings, J. and Skumatz, L.A. (2006). Non-energy benefits (NEBs) from commissioning in
schools, prisons, and other public buildings. In Proceedings of the 2006 American
Council for an Energy-Efficient Economy Summer Study on Energy Efficiency in
Buildings. August 13-18, 2006. Pacific Grove, California, USA.
Jinrong, H. and Enyi, Z. (2011). Engineering risk management planning in energy performance
contracting in China. Systems Engineering Procedia 1: 195–205.
Jones, B. and Bogus, S.M. (2010). Decision Process for Energy Efficient Building Retrofits: The
Owner’s Perspective. Journal of Green Building 5(3): 131-146.
Juan, Y-K., Kim, J.H., Roper, K., and Castro-Lacouture, D. (2009). GA-based decision support
system for housing condition assessment and refurbishment strategies. Automation in
Construction 18: 394-401.
Kalogirou, S.A. (2009). Artificial neural networks and genetic algorithms in energy applications
in buildings. Advances in Building Energy Research 3(1): 83–120.
Kachelmeier, S.J. and Shehata, M. (1992).Examining risk preferences under high monetary
incentives: experimental evidence from the People’s Republic of China. The American
Economic Review 82(5): 1120-1141.
Kahneman, D., Knetsch, J.L., and Thaler, R.H. (1991). Anomalies: The endowment effect, loss
aversion, and status quo bias. The Journal of Economic Perspectives 5(1): 193-206.
Kahneman, D. and Tversky, A. (1979). Prospect theory: an analysis of decision under risk.
Econometrica 47(2): 263-291.

486

Kansas Corporation Commission, Energy Division (2014). Energy Savings Performance
Contracting in the FCIP. Downloaded from http://www.kcc.ks.gov/energy/fcip/espc.htm
on May 15, 2014.
Kaplan, S. and Garrick, B.J. (1981). On the quantitative definition of risk. Risk Analysis 1(1):
11-27.
Kapp, M.J. and Girmscheid, G. (2005). Risk based life cycle cost analysis model for comparable
life cycle project delivery decision taking. In Proceedings of the 3rd International
Structural Engineering and Construction Conference ISEC-03. September 20-23, 2005.
Shunan, Japan.
Kaufman, R.J. (1970). Life cycle costing-decision-making tool for capital equipment acquisition.
Cost and Management 44(2): 21-28.
Keeney, S., Hasson, F., and Mckenna, H. (2011). The Delphi Technique in Nursing and Health
Research. Wiley-Blackwell: Oxford, England, United Kingdom.
Keeney, S., Hasson, F., and Mckenna, H. (2006). Consulting the oracle: ten lessons from using
the Delphi technique in nursing research. Journal of Advanced Nursing 53(2): 205-212.
Keeney, R.L. and von Winterfeldt, D. (1991). Eliciting probabilities from experts in complex
technical problems. IEEE Transactions on Engineering Management 38(3): 191-201.
Kindinger, J.P. and Darby, J.L. (2000). Risk factor analysis- a new qualitative risk management
tool. In Proceedings of the Project Management Institute Annual Seminars &
Symposium. September 7–16, 2000. Houston, Texas, USA.
Kirkham, R.J., Boussabaine, A.H., and Awwad, B.H. (2002). Probability distributions of
facilities management costs for whole life cycle costing in acute care NHS hospital
buildings. Construction Management and Economics 20: 251-261.
Kishk, M., Al-Hajj, A., Pollock, R., Aouad, G., Bakis, N., and Sun, M. (2003). Whole Life
Costing in Construction: A State of the Art Review. RICS Foundation Paper Series,
Volume 4, Number 18. RICS Foundation. London, England, United Kingdom.
Klotz, L. (2011). Cognitive biases in energy decisions during the planning, design, and
construction of commercial buildings in the United States: an analytical framework and
research needs. Energy Efficiency 4:271–284.
Kmenta, S. (2002). Scenario-based FMEA - using expected cost: a new perspective on
evaluating risk in FMEA. Presented at the Institute of Industrial Engineers Workshop,
January 22, 2002.
Kmenta, S. and Ishii, K. (2004). Scenario-based failure modes and effects analysis using
expected cost. Journal of Mechanical Design 126: 1027-1035.
487

Kmenta, S. and Ishii, K. (2000). Scenario-based FMEA: a life cycle cost perspective. In
Proceedings of the 2000 ASME Design Engineering Technical Conferences. September
10-14, 2000. Baltimore, Maryland, USA.
Kolokotsa, D., Diakaki, C., Grigoroudis, E., Stavrakakis, G., and Kalaitzakis, K. (2009).
Decision support methodologies on the energy efficiency and energy management in
buildings. Advances in Building Energy Research 3(1): 121-146.
Kolokotsa, D., Tsoutsos, T., and Papantoniou, S. (2012). Energy conservation techniques for
hospital buildings. Advances in Building Energy Research 6(1): 159-172.
Kokkaew, N. and Chiara, N. (2010). Modelling completion risk using stochastic critical pathenvelope method: a BOT highway project application. Construction Management and
Economics 28: 1239-1254.
Korpi, E. and Ala-Risku, T. (2008). Life cycle costing: a review of published case studies.
Managerial Auditing Journal 23(3): 240-261.
Kwak, S-Y, Yoo, S-H, and Kwak, S-J. (2010). Valuing energy-saving measures in residential
buildings: A choice experiment study. Energy Policy 38(1): 673-677.
Kyckelhahn, T. (2012). State Corrections Expenditures, FY 1982-2010. NCJ 239672, Bureau of
Justice Statistics. Washington, DC, USA.
Laitner, J.A. (2013). Calculating the Nation's Annual Energy Efficiency Investments. Research
Report E133, American Council for an Energy Efficient Economy. Washington DC,
USA.
Larsen, P.H., Goldman, C.A., and Satchwell, A. (2012a). Evolution of the U.S. energy service
company industry: Market size and project performance from 1990–2008. Energy Policy
50: 802-820.
Larsen, P., Goldman, C., Gilligan, D., and Singer, T.E. (2012b). Incorporating non-energy
benefits into energy savings performance contracts. In Proceedings of the 2012 American
Council for an Energy-Efficient Economy Summer Study on Energy Efficiency in
Buildings. August 12-17, 2012. Pacific Grove, California, USA.
Laryea, S. and Hughes, W. (2011). Risk and price in the bidding process of contractors. Journal
of Construction Engineering and Management 137: 248-258.
Latino, R.J. (2004). Optimizing FMEA and RCA efforts in health care. Journal of Healthcare
Risk Management 24(3): 21-28.
Lawrence, S., and J. Travis. 2004. The new landscape of imprisonment: Mapping America’s
prison expansion. CPR04 0121, The Urban Institute – Justice Policy Center. Washington,
DC, USA.
488

Layzell, J. and Ledbetter, S. (1998). FMEA applied to cladding systems - reducing the risk
of failure. Building Research and Information 26(6): 351-357.
Lee, H.W., Tommelein, I.D., and Ballard, G. (2013). Target-setting practice for loans for
Commercial energy-retrofit projects. Journal of Management in Engineering.
http://dx.doi.org/10.1061/(ASCE)ME.1943-5479.0000245
Lindahl, M. (1999). E-FMEA - a new promising tool for efficient design for environment. In
Proceedings of the First International Symposium on Environmentally Conscious Design
and Inverse Manufacturing. February 1-2, 1999. Tokyo, Japan.
Linstone, H. and Turoff, M. (Eds.) (1975). The Delphi Method: Techniques and Applications.
Addison-Wesley: Reading, Massachusetts, USA.
Linstone, H.A. (1999). Decision Making for Technology Executives: Using Multiple
Perspectives to Improve Performance. Artech House: Norwood, Massachusetts, USA.
Liu, H-C., Liu, L., and Liu, N. (2013). Risk evaluation approaches in failure mode and effect
analysis: a literature review. Expert Systems with Applications 40: 828-838.
Liu, J., Martinez, L., Wang, H., Rodriguez, R., and Novozhilov, V. (2010). Computing with
words in risk assessment. International Journal of Computational Intelligence Systems
3(4): 396-419.
Lucuik, M. and Meil, J. (2004). Cost Evaluation of Commercial Wall Envelope Systems. In
Proceedings of Thermal Performance of Exterior Envelopes of Whole Buildings IX.
December 2004. Clearwater Beach, Florida, USA.
Ma, Z., Cooper, P., Daly, D., and Ledo, L. (2012). Existing building retrofits: methodology and
state-of-the-art. Energy and Buildings 55: 889-902.
Marenjak, S., El-Haram, M.A., and Horner, M.R.W. (2003). A generic approach to minimize
whole life costs in the building industry. In Proceedings of the 19th Annual ARCOM
Conference. September 3-5, 2003. Brighton, England, United Kingdom.
McCabe, M.J. (2011). High-Performance Buildings – Value, Messaging, Financial and
Policy Mechanisms. PNNL-20176, Pacific Northwest National Laboratory. Richland,
Washington, USA.
McDermott, R.E., Mikulak, R.J., and Beauregard, M.R. (2009). The Basics of FMEA, 2nd
Edition. CRC Press: New York, New York, USA.
McKim, R.A. (1992). Risk behavior of contractors: a Canadian study. Project Management
Journal 23(3): 51-55.
McMillan, S. (2010). HVAC for prisons. ASHRAE Journal 52(7): 30-34.
489

McNeil, B.J., Pauker, S.G., Sox, Jr., H.C., and Tversky, A. (1982). On the elicitation of
preferences for alternative therapies. New England Journal of Medicine 306: 1259-1262.
McQuade, D.P. and Piotrowski, R. (1993). Nine steps to a successful prison construction project.
Corrections Today 55(4): 90-92.
Mecca, S. and Masera, M. (1999). Technical risk analysis in construction by means of FMEA
methodology. In Proceedings of the 15th Annual ARCOM Conference. September 15-17
1999. Liverpool, England, United Kingdom.
Michigan Department of Corrections (2014). Parnall Correctional Facility (SMT). Downloaded
from http://michigan.gov/corrections/0,4551,7-119-1381_1385-5339--,00.html on May
15, 2014.
Michigan Department of Licensing and Regulatory Affairs (2011). DELEG Announces Energy
Efficiency Improvements at Three Correctional Facilities to Save State $1.7 Million A
Year. Downloaded from http://www.michigan.gov/lara/0,4601,7-154-10573_11472251540--,00.html on June 30, 2013.
Michigan Public Service Commission (2014). Michigan Energy Measures Database.
Downloaded from http://www.michigan.gov/mpsc/0,4639,7-159-52495_55129---,00.html
on March 14, 2014.
Miles, M.B., Huberman, A.M., and Saldana, J. (2014). Qualitative Data Analysis: A Methods
Sourcebook, 3rd ed. SAGE Publications, Inc.: Los Angeles, California, USA.
Mills, E., Kromer, S., Weiss, G., and Mathew, P.A. (2006). From volatility to value: analysing
and managing financial and performance risk in energy savings projects. Energy Policy
34: 188–199.
Mitroff, I.I. and Turoff, M. (1975). Philosophical and Methodological Foundations of Delphi. In
H. Linstone and M. Turoff (Eds.) The Delphi Method: Techniques and Applications.
Addison-Wesley: Reading, Massachusetts, USA.
Molenaar, K., Sobin, N., Gransberg, D., McCuen, T., Korkmaz, S., and Horman, M. (2009).
Sustainable, High Performance Projects and Project Delivery Methods: A State-ofPractice Report. Charles Pankow Foundation. Vancouver, Washington, USA.
Moon, J.H. and Kang, C.S. (1999). Use of fuzzy set theory in the aggregation of expert
judgments. Annals of Nuclear Energy 26: 461-469.
Moore, E.O. (1980). A Prison Environment: Its Effect On Healthcare Utilization. PhD
Dissertation, University of Michigan, Ann Arbor, Michigan, USA.
Morgan, L. (2010). Green job training in prisons benefits everyone. Corrections Today 73(2):
34-37.
490

Morganstern, M., Meyer, S., Whitten, B., and Reuer, M. (2008). The energy retrofit of a
building: a journey through Bloom's learning domains. Journal of College Science
Teaching 37(5): 16-22.
Morelli, M. and Svendsen, S. (2012). Investigation of interior post-insulated masonry walls with
wooden beam ends. Journal of Building Physics 36(3) 265–293.
Mosleh, A., Bier, V.M., and Apostolakis, G. (1988). A critique of current practice for the use of
expert opinions in probabilistic risk assessment. Reliability Engineering and System
Safety 20: 63-85.
Mozzo, M.A. (2001). Setting the energy baseline for performance contracts. Strategic Planning
for Energy and the Environment 21(1): 12-19.
Murphy, M., Heaney, G., and Perera, S. (2011). A methodology for evaluating construction
innovation constraints through project stakeholder competencies and FMEA.
Construction Innovation 11(4): 416-440.
Nalbone, J.T. (2004). Evaluation of Building and Occupant Response to Temperature and
Humidity: Non-Traditional Heat Stress Considerations – A Comparison of Different
Construction Types Used by the Texas Department of Criminal Justice. PhD Dissertation,
Texas A&M University, College Station, Texas, USA.
Nielsen, A. (2002). Use of FMEA – failure modes effects analysis on moisture problems in
buildings. In Proceedings of the Sixth Symposium on Building Physics in the Nordic
Countries. June 17-19, 2002. Trondheim, Norway.
Okay, N. and Akman, U. (2010). Analysis of ESCO activities using country indicators.
Renewable and Sustainable Energy Reviews 14: 2760–2771.
Oklahoma Department of Corrections (2012). Facility Construction Security Standards. OP150202, Oklahoma Department of Corrections. Oklahoma City, OK, USA. Downloaded
from http://www.doc.state.ok.us/offtech/op150202.pdf on April 15, 2013.
Oliver, P. (2006). Purposive Sampling. In V. Jupp (Ed.) The SAGE Dictionary of Social
Research Methods. SAGE Publications, Ltd: London, England, United Kingdom.
Onodera, K. (1997). Effective techniques of FMEA at each life-cycle stage. In Proceedings of
the Annual Reliability and Maintainability Symposium. January 13-16, 1997.
Philadelphia, Pennsylvania, USA.
Onwuegbuzie, A.J., Daniel, L., and Leech, N.L. (2007). Spearman’s Rho. In N.J. Salkind and K.
Rasmussen (Eds.) Encyclopedia of Measurement and Statistics. Sage Publications, Inc.:
Thousand Oaks, California, USA.
491

Oregon Department of Energy (2014). Energy Savings Performance Contracting (ESPC).
Downloaded from http://www.oregon.gov/energy/cons/espc/Pages/faq.aspx on May 15,
2014.
Pai, R.R., Kallepalli, V.R., Caudill, R.J., and Zhou, M.C. (2003). Methods toward supply chain
risk analysis. In Proceedings of the IEEE International Conference on Systems, Man and
Cybernetics, 2003. October 5-8, 2003. West Point, New York, USA.
Palady, P. (1995). Failure Modes and Effects Analysis: Predicting & Preventing Problems
Before They Occur. PT Publications, Inc.: West Palm Beach, Florida, USA.
Pallin, S. (2013). Risk Assessment of Hygrothermal Performance – Building Envelope Retrofit.
PhD Thesis, Chalmers University of Technology, Gothenburg, Sweden.
Pampanin, S. (2009). Alternative Performance-Based Retrofit Strategies and Solutions for
Existing RC Buildings. In A. Ilki et al. (Eds.) Seismic Risk Assessment and Retrofitting,
Geotechnical, Geological, and Earthquake Engineering. Springer: New York, New York,
USA.
Pan, W., Wu, G., Zhou, H., Wang, C., and Hou, Y. 2012. Risk-based life cycle cost analysis
method for transmission systems. In Proceedings of the 3rd IEEE PES Innovative Smart
Grid Technologies Europe Conference. October 14-17, 2012. Berlin, Germany.
Perera, B.A.K.S., Dhanasinghe, I., and Rameezdeen, R. (2009). Risk management in road
construction: the case of Sri Lanka. International Journal of Strategic Property
Management 13: 87-102.
Perry, J.G. and Hayes, R.W. (1985). Risk and its management in construction projects.
Proceedings of the Institution of Civil Engineers, Part I 78: 499-521.
Petersen, S. (2009). Evaluating how energy performance contracting impacts the business case
for investing in building energy efficiency improvement. Presented at TowardsNETZERO London. November 20, 2009. London, England, United Kingdom.
Phillips, T.S. and Griebel, M.A. (2003). Building Type Basics for Justice Facilities. Wiley:
Hoboken, New Jersey, USA.
Pill, J. (1971). The Delphi method: substance, context, a critique and an annotated bibliography.
Socio-Economic Planning Sciences 5: 57-71.
Powell, C. (2003). The Delphi technique: myths and realities. Methodological Issues in Nursing
Research. 41(4): 376-382.
Raftery, J., Csete, J., and Hui, S.K.-F. (2001). Are risk attitudes robust? Qualitative evidence
before and after a business cycle inflation. Construction Management and Economics 19:
155-164.
492

Raftery, J. (1994). Human aspects of project risk management: In Proceedings of the CIB
International Symposium. Hong Kong.
Rahimi, S.A., Jamshidi, A., Ait-Kadi, D., and Ruiz, A. (2013). Using Fuzzy Cost-Based FMEA,
GRA and Profitability Theory for Minimizing Failures at a Healthcare Diagnosis Service.
CIRRELT-2013-61. Interuniversity Research Centre on Enterprise Networks, Logistics
and Transportation. Montreal, Quebec, Canada.
Rajendran, S. and Gambatese, J.A. (2009). Development and initial validation of sustainable
construction safety and health rating system. Journal of Construction Engineering and
Management 135(10): 1067–1075.
Ranasinghe, M. and Russell, A.D. (1993). Elicitation of subjective probabilities for economic
risk analysis: an investigation. Construction Management and Economics 11: 326-340.
Rasmussen, N.C. (1981). The application of probabilistic risk assessment techniques to energy
technologies. Annual Review of Energy 6:123-138
Rauch, W. (1979). The decision Delphi. Technological Forecasting and Social Change 15(3):
159–169
Reigle, J.A. and Zaniewski, J.P. (2002). Risk-based life-cycle cost analysis for project-level
pavement management. Transportation Research Record 1816: 34-42.
Rhee, S.J. and Ishii, K. (2003). Life cost-based FMEA using empirical data. In Proceedings of
the 2003 ASME Design Engineering Technical Conferences. September 2-6, 2003.
Chicago, Illinois, USA.
Rhee, S.J. and Ishii, K. (2002). Life cost-based FMEA incorporating data uncertainty. In
Proceedings of the 2002 ASME Design Engineering Technical Conferences. September
29-October 2, 2002. Montreal, Quebec, Canada.
Rhee, S. and Spencer, C. (2009). Life Cost Based FMEA Manual: A Step by Step Guide to
Carrying out a Cost-Based Failure Modes and Effects Analysis. SLAC-TN-09-001.
SLAC National Accelerator Laboratory. Menlo Park, California, USA.
Romm, J.J. and Browning, W.D. (1994). Greening the Building and the Bottom Line –
Increasing Productivity Through Energy Efficient Design. D94-27. Rocky Mountain
Institute. Snowmass, Colorado, USA.
RS Means (2014). Facilities Maintenance & Repair 2014 Cost Data Book. RS Means: Norwell,
Massachusetts, USA.

493

Rushing, A.S., Kneifel, J.D., and Lippiatt, B.C. (2013). Energy Price Indices and Discount
Factors for Life-Cycle Cost Analysis – 2013: Annual Supplement to NIST Handbook 135
and NBS Special Publication 709. NISTIR 85-3273-28, National Institute of Standards
and Technology. Gaithersburg, Maryland, USA.
Russell, T. and Thaler, R. (1985). The relevance of quasi rationality in competitive markets. The
American Economic Review 75(5): 1071-1082.
Russo (2011). Green Outlook 2011: Green Trends Driving Growth. McGraw Hill
Construction: New York, New York, USA.
Sadler, G.R., Lee, H-C., Lim, R.S-H., and Fullerton, J. (2010). Recruiting hard-to-reach United
States population sub-groups via adaptations of snowball sampling strategy. Nursing &
Health Sciences 12(3): 369-374.
Samuel, S. (2010). Information Framework for Residential Energy Retrofit. Master’s Report,
Michigan State University, East Lansing, Michigan, USA.
Samuelson, W. and Zeckhauser, R. (1988). Status quo bias in decision making. Journal of Risk
and Uncertainty 1: 7-59.
Sankey, M.S. (2007). Performance contract failures – causes, cases, and remediation. In
Proceedings of the World Energy Engineering Congress 2007. August 15-17, 2007.
Atlanta, Georgia, USA.
Satchwell, A., Goldman, C., Larsen, P., Gilligan, D., and Singer, T. (2010). A Survey of the U.S.
ESCO Industry: Market Growth and Development from 2008 to 2011. LBNL-3479E,
Ernest Orlando Lawrence Berkeley National Laboratory. Berkeley, California, USA.
SBW (2003). Achieving Silver LEED: Preliminary Benefit-Cost Analysis for Two City of
Seattle Facilities. SBW Consulting, Inc. Bellevue, Washington, USA.
Schieg, M. (2007). Risk management in construction projects, selected methods for the
evaluation of risks. In Proceedings of the 9th International Conference on Modern
Building Materials, Structures and Techniques. May 16-18, 2007. Vilnius, Lithuania.
Schweitzer, M., Shonder, J.A., Hughes, P.J., and Schmoyer, R.L. (2000). Methodology for
Recalculating and Verifying Savings Achieved by the Super ESPC program.
ORNL/CON-496, Oak Ridge National Laboratory. Oak Ridge, Tennessee, USA.
Scott-Hayward, C.S. (2009). The Fiscal Crisis in Corrections: Rethinking Policies and Practices.
Vera Institute of Justice. New York, New York, USA.

494

Seeley, C.C. (2012). Innovative Financing Mechanisms for Energy Efficiency Retrofits in
Buildings – A Shift in the Traditional Paradigm. In Proceedings of EcoDesign 2011: The
7th International Symposium on Environmentally Conscious Design and Inverse
Manufacturing. November 30 -December 2, 2011. Kyoto, Japan.
Shang, T-C., and Zhang, L-R. (2008). Energy Performance Contracting Project Evaluation and
B-S Model. In Proceedings of the 4th International Conference on Wireless
Communications, Networking and Mobile Computing. October 12-14, 2008. Dalian,
China.
Sheldon, P. and Atherton, E. (2011). Greening Corrections Technology Guidebook. National
Law Enforcement and Corrections Technology Center Corrections Technology Center of
Excellence. Gaithersburg, Maryland, USA.
Sheldon, P.M., Klein, B., Jones, M., Atherton, E., Nadkarni, N., and Pacholke, D. (2011). Save
$1,000 per inmate: seven steps to more sustainable correctional facilities and programs.
Corrections Today73(2): 52-55.
Shonder, J.A. and Hughes, P.J. (2005). Seeing Savings from an ESPC Project in Fort Polk’s
Utility Bills. ORNL/TM 2004/294, Oak Ridge National Laboratory. Oak Ridge,
Tennessee, USA.
Shonder, J., Morofsky, E., Schmidt, F., Morck, O., and Himanen, M. (2010). Best Practice
Guidelines for Using Energy Performance Contracts to Improve Government Buildings.
IEA ECBCS Annex 46, International Energy Agency, Energy Conservation in Buildings
and Community Systems Programme. Paris, France.
Silberman, A.P. (2010). Disputes and litigation of energy savings performance contracts.
Construct! 19(1): 1, 10-13.
Simu, K. (2009). The Construction Site Manager’s Impact on Risk Management Performance.
PhD Dissertation, Lulea University of Technology, Lulea, Sweden.
Singh, A. (2009). Life Cycle Cost Analysis of Occupant Well-Being and Productivity Impacts in
LEED® Offices. MS Thesis, Michigan State University, East Lansing, Michigan, USA.
Skulmoski, G.J., Hartman, F.T., and Krahn, J. (2007). The Delphi method for graduate research.
Journal of Information Technology Education 6: 1-21.
Smith, D.O. and Ferber, J.J. (1996). Performance contracting with state and local governments.
Public Contract Law Journal 25(2): 393-422.
Society of Automotive Engineers (2001). Recommended Failure Modes and Effects Analysis
(FMEA) Practices for Non-Automobile Applications. ARP 5580, Society of Automotive
Engineers. Warrendale, Pennsylvania, USA.
495

Sorrell, S., Schleich, J., Scott, S., O’Malley, E., Trace, F., Boede, U., Ostertag, K., and Radgen,
P. (2000). Reducing barriers to energy efficiency in public and private organisations.
JOS3CT970022, University of Sussex. Brighton, England, United Kingdom.
Spencer, C.M. and Rhee, S.J. (2003). Cost based Failure Modes and Effects Analysis (FMEA)
for systems of accelerator magnets. In Proceedings of the IEEE 2003 Particle Accelerator
Conference. May 12-16, 2003. Portland, Oregon, USA.
Stamatis, D.H. (2003). Failure Mode and Effect Analysis: From Theory to Execution. American
Society for Quality, Quality Press: Milwaukee, Wisconsin, USA.
Stephan, J.J. (2008). Census of State and Federal Correctional Facilities, 2005. NCJ 222182,
Bureau of Justice Statistics – U.S. Department of Justice. Washington, DC, USA.
Stephan, J.J. (2004). State Prison Expenditures, 2001. NCJ 202949, Bureau of Justice Statistics –
U.S. Department of Justice. Washington, DC, USA.
Strand, J. (2011). Rebound Effects with Emphasis on Long‐Term Infrastructure Investments, and
Their Interaction with Absolute (Ordinary) Rebound Effects. The World Bank.
Washington, DC, USA.
Stuart, E., Larsen, P.H., Goldman, C.A., and Gilligan, D. (2013). Current size and remaining
market potential of the U.S. energy service company industry. LBNL-6300E, Ernest
Orlando Lawrence Berkeley National Laboratory. Berkeley, California, USA.
Stum, K. (2000). Commissioning in energy savings performance contracts. In Proceedings of the
2000 American Council for an Energy-Efficient Economy Summer Study on Energy
Efficiency in Buildings. August 20-25, 2000. Pacific Grove, California, USA.
Subramanyan, H., Sawant, P.H., and Bhatt, V. (2012). Construction project risk assessment:
development of model based on investigation of opinion of construction project experts
from India. Journal of Construction Engineering and Management 138: 409-421.
Syal, M., Duah, D., Samuel, S., Mazor, M., Mo, Y., and Cyr, T. (2014). Information framework
for intelligent decision support system for home energy retrofits. Journal of Construction
Engineering and Management. http://dx.doi.org/10.1061/(ASCE)CO.19437862.0000773.
Tah, J.H.M. and Carr, V. (2001). Knowledge-based approach to construction project risk
management. Journal of Computing in Civil Engineering 15: 170-177.
Tah, J.H.M. and Carr, V. (2000). Information modelling for a construction project risk
management system. Engineering, Construction and Architectural Management 7(2):
107-119.

496

Tah, J.H.M, Thorpe, A., and McCaffer, R. (1993). Contractor project risks contingency
allocation using linguistic approximation. Computing Systems in Engineering 4(2-3):
281-293.
Teddlie, C. and Yu, F. (2007). Mixed methods sampling: A typology with examples. Journal of
Mixed Methods Research 2007 1: 77
Teng, S-H. and Ho, S-Y. (1996). Failure mode and effects analysis: an integrated approach for
product design and process control. International Journal of Quality & Reliability
Management 13(5): 8-26.
Tetreault, T. and Regenthal, S. (2011). ESPC Overview: Cash Flows, Scenarios, and Associated
Diagrams for Energy Savings Performance Contracts. Technical Report NREL/ TP7A30-51398.
Thaheem, M.J. and De Marco, A. (2013). A survey on usage and diffusion of project risk
management techniques and software tools in the construction industry. World Academy
of Science, Engineering and Technology 78: 1090-1097.
Thaler, R. (1985). Mental accounting and consumer choice. Marketing Science 4(3): 199-214.
TIAA-CREF (2013). Asset Allocation Evaluator. Downloaded from https://ais4.tiaacref.org/asstallocguidance/nsjsp/dms.do on September 17, 2013.
Tongco, M.D.C. (2007). Purposive sampling as a tool for informant selection. Ethnobotany
Research & Applications 5: 147-158.
Toole, T.M. (2006). A primer on social science research methods in construction. In Proceedings
of the ASCE-CIB Joint Conference on Leadership in Construction. May 4-6, 2006.
Freeport, Bahamas.
Toole, T.M. (1998). Uncertainty and home builders’ adoption of technological innovations.
Journal of Construction Engineering and Management 124(4): 323-332.
Trochim, W.M.K. (2006). The Research Methods Knowledge Base, 2nd ed. Downloaded from
http://www.socialresearchmethods.net/kb/ on February 12, 2014.
Tse, R.Y.C. and Love, P.E.D. (2001). Quasi-rational behavior in the property and construction
market from an economic perspective. Journal of Construction Research 2: 109-114.
Tserng, H.P., Yin, S.Y.L., Dzeng, R.J., Wou, B., Tsai, M.D., and Chen, W.Y. (2009). A study of
ontology-based risk management framework of construction projects through project life
cycle. Automation in Construction 18: 994-1008.

497

Tso, B., Skumatz, L., and Cohan, D. (2003) Determining the Total Costs and Benefits of
Commissioning Public Buildings. In Proceedings of the 2003 International Energy
Program Evaluation Conference. August 20-22, 2003. Seattle, Washington, USA.
Turoff, M. (1975). The Policy Delphi. In H. Linstone and M. Turoff (Eds.) The Delphi Method:
Techniques and Applications. Addison-Wesley: Reading, Massachusetts, USA.
Turoff, M. (1970). The design of a policy Delphi. Technological Forecasting and Social Change
2(2): 149-171.
Tversky, A. and Kahneman, D. (1992). Advances in prospect theory: cumulative representation
of uncertainty. Journal of Risk and Uncertainty 5: 297-323.
Tversky, A. and Kahneman, D. (1986). Rational choice and the framing of decisions. The
Journal of Business 59(4): S251-S278.
Tversky, A. and Kahneman, D. (1981). The framing of decisions and the psychology of choice.
Science 211(4481): 453-458.
Tversky, A. and Kahneman, D. (1974). Judgment under uncertainty: heuristics and biases.
Science 185(4157): 1124-1131.
U.S. Department of Energy (2014a). Commercial Buildings Portal. Downloaded from
http://energy.gov/eere/buildings/improving-energy-efficiency-commercial-buildings on
March 14, 2014.
U.S. Department of Energy (2014b). Buildings Performance Database. Downloaded from
http://energy.gov/eere/buildings/buildings-performance-database on March 14, 2014.
U.S. Department of Energy (2014c). Buildings Energy Software Tools Directory. Downloaded
from http://apps1.eere.energy.gov/buildings/tools_directory/ on March 14, 2014.
U.S. Department of Energy (2013a). Energy Information Administration – Commercial
Buildings Energy Consumption Survey. Downloaded from http://www.eia.gov/
consumption/ commercial/ on May 24, 2013.
U.S. Department of Energy (2013b). Advanced Energy Retrofit Guide (AERG): Practical Ways
to Improve Energy Performance; Healthcare Facilities. BK-5500-57864. National
Renewable Energy Laboratory. Golden, Colorado, USA.
U.S. Department of Energy (2013c). Better Buildings Workforce Fact Sheet. U.S. Department of
Energy. Washington, DC, USA.
U.S. Department of Energy (2012). New Tool for Estimating O&M Costs in Financed Energy
Projects, November 1, 2002. Downloaded from https://www1.eere.energy.gov/femp/
news/news_detail.html?news_id=7309 on April 18, 2014.
498

U.S. Department of Energy (2011a). Buildings Energy Data Book, Chapter 1: Buildings
Sector. U.S. Department of Energy - Energy Efficiency & Renewable Energy.
Washington, DC, USA.
U.S. Department of Energy (2011b). Buildings Energy Data Book, Chapter 3: Commercial
Sector. U.S. Department of Energy - Energy Efficiency & Renewable Energy.
Washington, DC, USA.
U.S. Department of Energy (2008) High Performance Sustainable Buildings. DOE G 413.3-6.
U.S. Department of Energy. Washington, DC, USA.
U.S. Department of Housing and Urban Development (2014). Public Housing Assessment
System (PHAS). Downloaded from http://portal.hud.gov/hudportal/HUD?src=/program_
offices/public_indian_housing/reac/products/prodphas on April 24, 2014.
U.S. Environmental Protection Agency (2014). Energy Star Portfolio Manager. Downloaded
from http://www.energystar.gov/buildings/facility-owners-and-managers/existingbuildings/use-portfolio-manager on May 1, 2014.
U.S. Environmental Protection Agency (2001). Risk Assessment Guidance for Superfund
(RAGS) Volume III - Part A: Process for Conducting Probabilistic Risk Assessment.
Downloaded from http://www.epa.gov/oswer/riskassessment/rags3adt/ on May 31, 2014.
Vagias, W. M. (2006). Likert-type scale response anchors. Clemson University - Clemson
International Institute for Tourism & Research Development, Department of Parks,
Recreation and Tourism Management. Clemson, South Carolina, USA.
van Buiten, M. and Hartmann, A. (2013). Public-private partnerships: cognitive biases in the
field. In Proceedings of the Engineering Project Organization Conference. July 9-11,
2013. Tabernash, Colorado, USA.
Vanneste, J. (2010). Washington Department of Corrections Sustainability Report – 2010 Data
Collection. Washington Department of Corrections. Olympia, Washington, USA.
Vogt, W.P. (2005). Dictionary of Statistics & Methodology, 3rd Ed. SAGE Publications, Ltd:
London, England, United Kingdom.
von Ahsen, A. (2008). Cost-oriented failure mode and effects analysis. International Journal of
Quality & Reliability Management 25(5): 466-476.
Vose Software (2014). ModelRisk Industrial (Version 5) [Software]. Available from http://www.
vosesoftware.com.

499

Walker, A. and Dominick, J. (2000). Performance contracting of a parabolic trough system at the
Federal Correction Institution-Phoenix. In Proceedings of the 35th Intersociety Energy
Conversion Engineering Conference and Exhibit. July 24-28, 2000. Las Vegas, Nevada,
USA.
Wang, J. and Chen, X. (2008). Estimation of the risks of energy performance contract projects
based on the unascertained measure model. In Proceedings of the IEEE International
Conference on Automation and Logistics. September 1-3, 2008. Qingdao, China.
Ward, S.C. and Chapman, C.B. (1995). Risk-management perspective on the project lifecycle.
International Journal of Project Management 13(3): 145-149.
Waste Reduction Partners (2008). North Caroline Guide to Energy Performance Contracting for
K-12 Schools, Local Governments & Community Colleges. Waste Reduction Partners
Land of Sky Regional Council. Asheville, North Carolina, USA.
Webster, A. (2010). Environmental prison reform: lower costs and greener world. New England
Journal on Criminal and Civil Confinement 36: 175-195.
Wener, R. (2012). Evidence-based design & correctional architecture. Academy of Architecture
for Justice Journal. Winter 2012.
Whitestone Research (2009). Approximation of the asbestos cleanup liability for large real
property portfolios. Downloaded from http://www.whitestoneresearch.com/media/725/
estimateasbestosliability.pdf on May 31, 2014.
Willis, K. (2008). How the “dirty dozen” ECMs created a need for a risk assessment tool. In
Proceedings of the 2008 American Council for an Energy-Efficient Economy Summer
Study on Energy Efficiency in Buildings. August 17-22, 2008. Pacific Grove, California,
USA.
Wilson, C. and Dowlatabadi, H. (2007). Models of decision making and residential energy use.
Annual Review of Environment and Resources 32:169–203.
Wirba, E.N., Tah, J.H.M., and Howes, R. (1996). Risk interdependencies and natural language
computations. Engineering, Construction and Architectural Management 3(4): 251-269.
Wisconsin Department of Administration (2011). Energy Use in State Facilities, Fiscal Year
2010. Wisconsin Department of Administration, Division of State Facilities. Madison,
Wisconsin, USA.
Woodward, D. G. (1997). Life cycle costing--theory, information acquisition and
Application. International Journal of Project Management 15(6): 335-344.

500

Xiang, P., Zhou, J., Zhou, X., and Ye, K. (2012). Construction project risk management based on
the view of asymmetric information. Journal of Construction Engineering and
Management 138: 1303-1311.
Yang, T.A. and Lowe, D. (2011). Construction risk modeling and assessment: insights from a
literature review. The Built & Human Environment Review 4(Special Issue 1): 87-97.
Yin, R. K. (2014). Case Study Research: Design and Methods, 5th ed. SAGE Publications, Inc.:
Los Angeles, California, USA.
Yousuf, M.I. (2007). Using experts’ opinions through Delphi technique. Practical Assessment,
Research & Evaluation 12(4): 1-8.
Zavadskas, E. K., Turskis, Z., and Tamosaitiene, J. (2010). Construction risk assessment of small
scale objects by applying TOPSIS method with attributes values determined at intervals.
Computer Modelling and New Technologies 12(4): 38–43.
Zhu, Y., Tao, Y., and Reygan, R. (2012). A comparison of deterministic and probabilistic life
cycle cost analyses of ground source heat pump (GSHP) applications in hot and humid
climate. Energy and Buildings 55: 312–321.

501