Mun . 5-? If («list-t. ”in 'r 'ly;~§‘5:~* $544. ‘3‘“ ‘1 v-“ ‘r l‘ ‘ fit an: «a: 7. wwwgé -' ‘ r- J t g: , f ’5 17?; 53‘qu 1:4 “1:4; .=a w‘r- ‘ ‘ zgfisifllr‘zh egg: a... .5, .. 'r n. 5- gig?! 3:3“be _! . 3‘ 02.51%“: 1;; i,-—.!.‘ i‘Fsi‘f’Fgéiff" ' 1 a ’3 ,3 .x: a. m’ ”£335 3 again?! Tm! .75 t5 / oat-lanai? LIBRARIES CHIGAN STATE UNIVERSITY EIA‘ST LANSING, MICH 48824-1048 This is to certify that the dissertation entitled REFLECTIVE PRACTICE IN ACTION: A FRAMEWORK FOR PROFESSIONAL ONLINE LEARNING PhD. presented by NIJSIREE WAEOCHAN has been accepted towards fulfillment of the requirements for the degree in Human Environment: Design & Management .IJM,’ Al .51/“ {41 aor ro 9880 So Igna ure PA 5-177) , ' I I A I Date MSU is an Affin‘native Action/Equal Opportunity Institution PLACE IN RETURN BOX to remove this checkout from your record. TO AVOID FINES return on or before date due. MAY BE RECALLED with earlier due date if requested. DATE DUE DATE DUE DATE DUE APRIUW '1‘02‘5 «'7 6/01 c:/CIRC/Date0ue.p65-p. 1 5 REFLECTIVE PRACTICE IN ACTION: A FRAMEWORK FOR PROFESSIONAL ONLINE LEARNING By Nijsiree Waeochan A DISSERTATION Submitted to Michigan State University in partial fulfillment of the requirements for the degree of DOCTOR OF PHILOSOPHY Department of Human Environment and Design 2004 ABSTRACT REFLECTIVE PRACTICE IN ACTION: A FRAMEWORK FOR PROFESSIONAL ONLINE LEARNING By Nijsiree Waeochan This study identified the factors that are important in reflective practice (RP) implementation within professional online learning and explored how to prepare a learning environment to support RP behavior. In this study, reflective practice is defined as a desired behavior for any effective practitioners and involves learning processes or information processing consisting of deliberation using reason to test thoughts and responses in given situations or for particular practices. The RP process reflects upon existing knowledge and experience to move into a higher level of learning and/or working performance improvement. The Facility Management Online Program at the Michigan State University is used as the ease for the research. The study was conducted during summer of 2003 and 2004. There were 31 students who participated in the research. Both quantitative and qualitative approaches were used. It includes both triangulation of research methods and sources Of data. Grounded theory and logical reasoning were also used in the data analysis procedure. The analysis results were used to develop and reframe a RP framework for professional online learning. The triangulation of methods included an online questionnaire, DELPHI web discussions and a content analysis of literatures related to the topic. The literatures were reviewed and analyzed to generate a preliminary framework Of online learning and key findings Of variables related to the study. The data from questionnaires were used in statistic procedure for preliminary test of the RP learning model as well as identified panicipants’ learning behaviors. The statistical analyses include factor analyses, multiple regression, partial correlations, and best subsets regression. DELPHI discussions were used to clarify unanswered questions. DELPHI web discussions on the topic of research were organized during a four week period. The web discussions were conducted in three rounds. Each round was held for two weeks. Responses from the first round were analyzed and synthesized into themes for the second round discussion. Then, the third round discussion was used for in-depth discussion of the themes. The findings from these analyses were analyzed along with the statistical results and used in reframing the preliminary RP framework for online professional learning. The analysis results indicate that the reflective practice behavior has two dimensions in nature: process and content. The reflective practice process includes both reflective thinking and critical thinking behavior. It involves with knowledge, belief, values, and purposefully objectives of thinking and/or practicing. The analyses also indicate that, without an objective, the reflective practice process might not be generated and might not lead to any improvements. Four different learning patterns were revealed. Each learning pattern has a different effect on learning and wOrking performance improvement. Therefore, encouraging some learning techniques may lead into high learning performance and working performance improvement. As the result, there are two main recommendations are issued: 1) improving of the validity and reliability of the questionnaire by redesigning the questionnaire and test it using a larger sample size of online professional students, 2) further studies in different effects of online learning approaches to RP behavior. © Copyright by NIJSIREE WAEOCHAN 2004 DEDICATION This dissertation is dedicated to my sister, who struggled to get her dissertation done by the ending of her life. vi ACKNOWLEDGEMENTS The completion of this dissertation taught me the meaning of never-ending learning. It took a lot more time and effort than I thought and could not be completed with out the participation of many individuals. I especially want to acknowledge all my committee members: Dr. Susan Mireley, Dr. Dana Stewart, Dr. Roger Baldwin, and Dr. David Lawrence for the unflagging encouragement. I am also grateful to the following individuals and organizations for their valuable advice, time, and supporting: The Royal Thai Government, my Official sponsor, the staff of The School of Architecture, and Department of Interior Architecture, King Mongkut's Institute of Technology, Ladkrabang (KMIT,L), Dr. Won Song, Dr. Sally Helvenston, and Dr. Julia Miller. In addition, I would like to give special recognition to all my friends at Michigan State University who shared their time, gave me the afflation, and encouragement to keep going. Special thanks to Nong OOd: Sungwom Ngudgratoke, Nong Krerk: Krerk Piromsopa and P’ Heang: Radchada Settavongsin for their brilliant helps and support. Last and far from least is my parents to whom I extend my love and utmost appreciation. Even though the time difference between Bangkok and the US is about twelve hours, they were always willing to receive my calls. And to those who I unintentionally missed to mention, I beg their forgiveness. vii TABLE OF CONTENTS LIST OF TABLES ........................................................................... LIST OF FIGURES ......................................................................... KEY TO ABBREVIATIONS .............................................................. CHAPTER ONE INTRODUCTION ............................................................................ Purpose of the Study .................................................................... Research Questions ..................................................................... Assumptions ................................................................................ Significance and Limitations of the study ............................................ Operational Definitions ................................................................. CHAPTER TWO LITERATURE REVIEWS ................................................................. Reflective Practice Concept ............................................................ Definitions of Reflective Practice and Related Variables ..................... Reflective Practice Process ....................................................... Schéin and Argyris’s Double-loop Learning .............................. Kolb’s Experiential Learning ............................................... Boud’s Model of Learning from Experience .............................. Reflective Practice Techniques and its Applications ............................... Individual Techniques ............................................................. Group Techniques .................................................................. Multiple Techniques ............................................................... Problems and Concerns in Implementation of Reflective Practice Learning Techniques ............................................................... CHAPTER THREE METHODOLOGY ........................................................................... Procedure ................................................................................. Variables and Measures ................................................................. Participants ............................................................................... Instrument ................................................................................. Data Analyses ........................................................................... Quantitative Analysis ............................................................... F actor Analysis ................................................................ Structural Equation Modeling ............................................... Qualitative Analysis ................................................................ viii xii XV '-‘\O\OOOO\I—‘ 16 17 22 28 29 31 35 39 41 45 48 50 56 58 61 67 69 71 72 72 73 75 CHAPTER FOUR QUALITATIVE ANALYSES AND RESULTS .......................................... Preliminary Analysis ................................................................... Preliminary Findings ............................................................. Emerging Themes and Behavioral Analysis ......... . ......................... Focus Group Discussion ................................................................. CHAPTER FIVE QUANTITATIVE ANALYSES AND RESULTS ....................................... F actor Analyses .......................................................................... Learning Technique Utilization (TU) ............................................ Interaction to People and Information Sources (1) .............................. Thinking Behavior (T) .............................................................. Working Performance Improvement (WP) ...................................... Experience (E) ....................................................................... Previous Knowledge (K) ........................................................... Time Spent in Learning (TI) ...................................................... Partial Correlations and Multiple Regression ....................................... Structural Equation Modeling ......................................................... CHAPTER SIX DISCUSSION AND CONCLUSION ...................................................... Nature of Reflective Practice .......................................................... Influences of Individual Attributes to Reflective Practice Behavior. ............ Teaching Techniques for RP Online Learning ....................................... Recommendations for Further Studies ................................................ Reliability and Validity Improvement of the Instrument ...................... Tests of Critical RP Online Learning Model .................................... APPENDICES Appendix A: Consent Letter ............................................................ Appendix B: Questionnaire ............................................................. Appendix C: Relationships among Research Questions, Focus Group Web Discussion Questions, and Research Objectives ..................... Appendix D: Quantitative Data from Questionnaire ............................... Appendix E: Summary of Data from Questionnaire ............................... Appendix F: Example of Scale Development for Variable Measures ........... REFERENCES ............................................................................... ix 77 78 78 83 93 124 126 126 134 139 146 148 ' 146 150 152 162 173 173 176 179 181 181 182 185 187 201 206 220 236 238 Table 2.1 Table 4.1 Table 4.2 Table 4.3 Table 4.4 Table 4.5 Table 4.6 Table 4.7 Table 4.8 Table 4.9 Table 4.10 Table 4.11 Table 4.12 Table 4.13 Table 4.14 Table 5.1 Table 5.2 Table 5.3 Table 5.4 Table 5.5 Table 5.6 Table 5.7 Table 5.8 Table 5.9 LIST OF TABLES Definitions of Reflective Practice and Related Variables .............. Preliminary Findings and Emerged Themes 1 .............................. Preliminary Findings and Emerged Themes 2.. ............................ Preliminary Findings and Emerged Themes 3 .............................. Preliminary Findings and Emerged Themes 4 ............................. Preliminary Findings and Emerged Themes 5 ............................. Preliminary Analyses: Previous Knowledge and Experience versus Thinking Ability ............................................................. Preliminary Analyses: Experience versus Learning Performance. . . .. Preliminary Analyses: Deductive Approach versus Inductive Approach ...................................................................... Preliminary Analyses: Frequency of RP Techniques Utilization versus Thinking .............................................................. Preliminary Analyses: Age versus Learning Performance, Working Performance Improvement, and Thinking Ability ....................... Preliminary Analyses: Interaction versus Learning Performance. . Preliminary Analyses: Gender versus Thinking Ability. . . . . . . . . . . Preliminary Analyses: Improvement after Taking Class(es) ........... Preliminary Analyses: Time and Consequences of RP. . . .. ............ Descriptive Statistics of Learning Technique Utilization (TU) ........ Factor Analysis: Pattern Matrix Analysis of Learning Technique Utilization (TU) .......................................................................... Reliability Report of Learning Techniques Utilization (TU) Measurement Model with 13 TU Sub-scale Items ...................... Reliability Report of Learning Techniques Utilization (TU) Measurement Mode]: If Each Item Deleted from the Model .......... Reliability Report of Interaction (1) Measurement Model 1 ............ Reliability Report of Interaction (I): If Each Item Deleted from the Model 1 ........................................................................ Reliability Report of Interaction (1) Measurement Model 2 ............ Reliability Report of Interaction (I): If Each Item Deleted from the Model 2 ........................................................................ Reliability Report of the Interaction (1) Final Measurement Model (3) .............................................................................. 25 79 80 81 82 83 84 85 86 87 88 89 90 90 91 127 128 129 130 135 135 135 136 139 CHAPTER ONE Introduction In the past two decades, unprecedented developments have reshaped professional education especially at the level of higher education. We have seen the widespread development of digital processing and communication doubled using the Internet networking. This has opened up a broad set of teaching and learning opportunities, allowing a new emphasis on interaction and concept exploration. As universities and businesses move toward the use of technology for online education and training, there is a need to make this alternative both more attractive and viable. New learning concepts recently have emerged, particularly new instructional approaches such as action learning and e-leaming. These new instructional approaches provide opportunities to learn as well as compete in the professional educational market. In a competitive and constantly evolving knowledge based economy, universities must come up with strategies to improve their educational standards in order to survive. It is imperative that graduates are equipped to be flexible and versatile so that they are well- prepared to assume responsibility for their own learning in continuous life-long professional development. Therefore, one of the most important characteristics of a well- prepared graduate is the ability to transfer his/her professional knowledge practically and effectively into specific work situations. Unfortunately, not all graduates have this ability. Theory, as presented in school, is quite different from in professional practice or in the “real world”. One reason for this is because of the difficulty of finding opportunities for novices to apply and experiment with conceptual knowledge in a relatively safe or stable environment. In fact, many question graduated students/practitioners to think independently, function without data being provided, change their approaches in mid-stream, negotiate, and continually reflect and inquire after having completed only theory-based learning experiences (Reilly, 1982). It is generally accepted that when graduates enter professional practice, they are expected to demonstrate high levels of proficiency with respect to knowledge and highly developed skills such as problem solving, interpersonal communication, time management, team work, project management and communication and information technologies. Cognitive skills alone are insufficient to ensure survival in a competitive work market (Harvey et aL,1997) To ensure that students achieve self-efficacy and an ability to apply the knowledge and skills gained from their formal learning experiences, in addition to any disciplinary specific knowledge and understanding they have gained, educators must develop teaching and learning methodologies that actively encourage students to become autonomous, independent, and self motivated learners. The philosophy of teaching must change from “providing a body of knowledge and enforcing skills” to “promoting effective learning and encouraging continuous learning” throughout their professional lives. The standards of a professional education must be strengthened because not only do graduates need adequate knowledge in their professional disciplines but they also need a critically reflective thinking ability to enable them to continue learning and improving standards. A movement in professional higher education is inevitable for many reasons. Beyond the need to improve the quality of graduates, to be able to survive in an increasingly competitive educational business, a range of other forces are shaping university policy, such as budget cuts and the globalization of professional practice. One challenge for educational institutions is the need to transform these forces into opportunities. This transformation is not merely necessary because technology is available and supposedly cost and time effective, but also because it is capable of enhancing teaching and learning solutions and thus the quality of the profession in the future (The Virtual University Design and Technology at Michigan State University [VUDAT-MSU], 2002). In response to this challenge, many American higher education institutions have implemented new educational models based on and enabled by Intemettechnologies. According to the 2001 Campus Computing Survey, nearly 70% of US institutions of higher education are engaged in delivering instruction via the Web (Greene, 2001, p.7). The National Center for Education Statistics (N CES) reports that in 1997-1998 nearly 44% of all higher education institutions offered distance learning courses, representing a 33% increase over the previous three years (NCES, 1999), and the rate increased to 56% in 2000-2001( American City Business Journal, July 21, 2003). In 2002, NCES reported a continued growth of computer use in education with the proportion of college students using computers in their classes increasing fiom 63% in 1997 to 79% in 2001, with about 69% using computers at home to do their school work (NCES, 2002). As of 2000, nearly 15% of all American higher education students (2 million out of 14 million) were enrolled in online courses delivered via Web technology (Howell, 2002). The NCES also reports the prevalence of computers in the workplace. By 2001, the use of computers had become widespread with 54% of all employees using computers in their work. More frequent use of computers is associated with higher levels of education and higher incomes. Of those with master's degrees, 85% use computers at work and those employed in managerial or professional fields use computers more than persons in other fields (NCES, 2002). Because computers can be used as an effective tool for learning and the rate of computer use in the workplace for those in managerial and professional fields is significantly high, it makes sense that online learning will be an appropriate strategy for professional education. However, the question that arises is “how do we prepare the online learning environment for professionals” to enhance both their learning and working abilities? ‘ Michigan State University (MSU) is one of the higher learning institutions that has spent the last decade developing irmovative technology and related strategies for online learning (VUDAT-MSU, 2002). The university, in support of its stated missions, has its own virtual university teams to provide professional and innovative technology enhanced teaching and learning. There are many strategies that have been used both in asynchronous and synchronous (both communicating to each others at the same time and communicating to each others at the different time manner using the Internet tool) format. One of the concepts that has been applied to virtual classes at the university is “action learning.” This strategy has focused on the use of “action learning” in “professional higher education.” An example of such a professional MSU program is the facility management (FM) online certificate program (Thatcher et al., 2001). The Master’s Level Certificate in FM has been offered since 1998 through MSU’s Lifelong Education Program. It consists of four courses: Facility Management: Theory and Principles; Information Management for Facility Professionals; Achieving Facility Management Organizational Effectiveness; and Facility Real Estate and Building Economics. With the cooperation of the Department of Human Environment and Design and MSU’s Lifelong Learning, the first course of the program: Facility Management: Theory and Principles, was offered during the summer of 1998 by using 95% Intemet- based methods, with students meeting in person only on the first and last weekends of the course. The evaluation of that first course indicated that it was feasible to deliver the whole course via the Internet and, thus, it became a 100% Intemet-based course in spring 1999 and has been offered that way ever since. This program responds to the demands of FM professionals who want to continue their learning and professional development, while at the same time continue working. Students attend classes via the Internet and learn by working in both groups and individually. Most students taking the virtual classes are FM practitioners who already have experience in the FM field. Therefore, students learn fi'om each other’s experiences, as well as working together on simulations of real work situations. The facility management online program is the focus of this research. The goal of the research is to provide a better online learning environment for professionals taking the online FM courses and thus enhance the teaching and learning that occurs in these courses. This study is also designed to explore the learning behavior of online students within the context of reflective practice, and how it relates to action learning and reflective inquiry. While action learning concentrates on encouraging students to practice using real world situations, this research will move into a deeper understanding of “students’ behaviors” by testing “what” factors influence reflective practice behavior. Purpose of the Study According to a literature review related to professional education, most researchers are concerned with three fundamental issues involving learning. The first issue involves what people learn—the identification of knowledge and skills needed for a profession. The second issue involves the process of learning, such as, how do we learn and what activities could be used to facilitate learning? The third issue involves using technologies for learning, such as, designing and building learning environments or learning and teaching strategies to facilitate the learning process. This study concentrates on the last two issues: the process of learning within the concept of reflective practice and how to facilitate the learning behavior. Because this study presents research in an human environment and design major with a specialization in facility management, it is designed to enhance human environment effectiveness. The initial outline of this research focuses on how to use technology to enhance students’ learning performance. Then, the research fiamework has been extended to cover more specific details related to online learning environment and how to support reflective practice behaviors. In this study, “reflective practice” (RP) is defined as a desired behavior that involves learning processes or information processing that consists of deliberation using reason to test thoughts and responses in given situations or for particular practices. The RP process reflects upon existing knowledge and experience to move into a higher level of learning and/or working performance improvement. It is a conscious, systematic, deliberate process of framing and reframing practice as a result of the consequences of our learning and practice. It involves knowledge, beliefs, values and objectives in a physical, political, socio—eultural and historical context. Therefore, to equip new graduates with this desired behavior, the major purpose of this study is to find how to simulate a situation that is designed to create personal experiences for online learners and that serves to initiate their own processes of inquiry and understanding and to support them in achieving the personal performance improvement. As a result of this study, this researcher expects to fulfill two main objectives: 1. To provide further understanding of RP behavior and its factors as they relate to the online learning environment and instructional methods, and 2. To outline a preliminary RP learning model within the professional online learning environment. Furthermore, as an action research itself, this researcher expects that this study has brought some positive consequences for the participants within the study itself and, hopefully, lead into improvements of the online professional program. This researcher anticipates that, by bringing up the issue of being reflective and reflective practice in professional learning to participants’ considerations, participants will recognize the importance of reflective practice behavior and the application of the reflective practice concept to professional education and learning environment. To achieve these research objectives, three main research questions are set. Research Questions This research is exploratory in nature. It examines the following questions: 1. What is the nature of RP among online professional learners? 2. On what basis do students choose to reflect on their learning? 3. What kinds of instructional methods encourage students’ RP behaviors? These three questions are examined in detailed below: Question One: What is the nature of RP among online professional learners? What elements of RP are identified through the existing literature? What factors influence RP behavior? Is RP affected by students’ experiences, previous knowledge and gender? Do common RP patterns or processes occur in online learning? Question Two: On what basis do students choose to reflect on their learning? Does RP tend to occur in specific circumstances for specific reasons or need specific motivations? Is RP more likely to occur when students participate in group activities than when they study alone? Question Three: What kinds of instructional methods encourage students’ RP behavior? What kinds of instructional methods/assignments do students prefer? Does an instructional method foster RP behaviors? What kinds of course delivery methods foster reflecting (RP) on learning? To examine these questions, research assumptions are provided. The following assumptions are necessary for this research in order to be guideline for setting preliminary framework of the reflective practice concept and it possible to conceptualize and investigate the relationships among variables within the framework. Assumptions In this study, it is assumed that RP is a continuous process that follows an action learning model. Given the nature of action learning, numerous variables and constructs related to the behavior can be identified. These include constructs in both the learning V environment (such as learning styles) and in the working environment (such as working responsibilities). This research concentrates only on RP in the learning environment and instructional strategies that involves and leads to an improvement in students’ RP abilities. It is assumed that within the context of the action learning model, students and/or practitioners learn in the same way that they work, that is, they use the same thinking systems in both the learning and the working environments. Regardless of whether reflective thinking is a natural behavior or not, people in normal mental health are assumed to have reflective thinking abilities, and thus, it is assumed that an RP ability can be developed. Therefore, in an online RP environment, all students are assumed to have and use their RP abilities. The degree of use will vary according to various governing variables and related factors. Significance and Limitations of the Study This study is designed to provide insights to professional educators involved in online learning and who have a goal of helping their online students become “reflective practitioners”. The purpose of the research is to identify factors that might improve online education and determine how those factors can be integrated into the online professional learning environment in order to achieve improved reflective leaning behaviors. Thus, this study also has the potential to benefit a number of professionals as well as online learners in general. Despite the many potential benefits associated with this research, some limitations must be noted. Because this research is a case study of one particular online professional program, the findings are limited to an in-depth understanding of this program only. This FM online program is perhaps unique in its combination of field-based learning activities and experiences in FM practice. Thus, it is questionable as to whether the findings of this study are useful for other professional educational programs. However, this study of online learning strategies and environment addresses universal themes and, therefore, sheds some light on the dynamics of professional online learning. For case studies such as this research, the intent is not to generalize to other situations, per se, but rather to understand a particular situation in a greater detail than that found in broader studies. With this understanding, it is possible to enrich the comprehension of other similar situations, which must be considered also from their own perspectives and particular contexts. Assessment of the degree of online learning that occurs while students are taking courses and the extent of improved working performance that occurs as a result of taking the courses are important considerations in this study. Under these circumstances, the measurement of working performance improvement relies on the participants’ self evaluations. These considerations are problematic measures and thus provide another limitation to the research. The students, as F M practitioners, are in different physical environments and have different experiences and educational backgrounds. As a result, they may evaluate their learning and work performances using different standards. To 10 minimize this inconsistency, assessment of working and learning performance focuses on their self-reported assessment of improved performance and, more importantly, on the effects of RP on their performances. No attempt to predict future performance is attempted. . Moreover, this research was conducted using an online questionnaire and online focus group discussions and the participation recruitment was via e-mail. The number of alumni and current students’ e-mail addresses involved in the MSU virtual courses is about 120 addresses. Therefore, due to the small sample size and the voluntary nature of the response sample, triangulation of research methods and data sources were used for this research to minimize any bias that may occur and to enhance the reliability of the study. The main objective is not a solid reliability of the entire research process, but rather a preliminary test with recommendations for firrther studies to test the model in the future. In summary, limitations to the study are as follow: 1. This study is limited to online professional students. 2. Generalization of this study is limited to the case study itself. 3. The study is limited to perceptions expressed by a self-selecting group of participants. Operational Definitions Operational definitions for terms used throughout this study are provided below for use as a systematic reference. 11 Reflective Practice (RP) Reflective practice is a desired behavior for any effective practitioners and involves learning processes or information processing consisting of deliberation using reason to test thoughts and responses in given situations or fer particular practices. The RP process reflects upon existing knowledge and experience to move into a higher level of learning or performance improvement. It is a conscious, systematic, deliberate process of mm and reframing practice, as a result of the consequences of our learning and practice. It embraces knowledge, beliefs, values and objectives in physical, political, socio-cultural and historical context. The process incorporates a range of techniques, through which one acquires a deeper understanding of oneself and his/her interactions with others in the working and learning environment. In this study, the RP behavior is assumed to consist of three dimensions: (a) reflective thinking—refers to its process; (b) critical thinking—refers to the context and values related in the process; and (c) level of analytical thinking—refers to the quality of the process. These dimensions are further defined below. Reflective Thinking Behavior (R) Reflective thinking refers to the cyclical inquiry process within the reflective practice process. It mainly focuses on an individual’s psychological learning or working style, not the content within the process. To identify whether a practitioner is a reflective practitioner, we must look at this style of thinking. Critical Thinking Behavior (C) Critical thinking closely associates with reflective thinking as it refers to the contents within the reflective practice process. Critical thinking is not a stand-alone 12 construct; it is recognized as a purposeful reflective thinking behavior. It contains the evaluation process of contents and values related in each practice. Critical thinking differs from reflective thinking in that it focuses on commitments to content of values, beliefs, and any pre—set objective(s) to achieve a desirable result for each practice. The term “critical reflective behavior” is used in this research as it includes both dimensions of RP: process and content. Level of Analytical Thinking (L) Analytical thinking refers to the ability to think logically, breaks things down, and to recognize cause and effect. In this study, levels of analytical thinking refer to the quality of thinking used to analyze information at different levels from a surface comprehension to in—depth analyses. Reflective practice involves this dimension as it indicates the quality of the process and the differences of the reflective practice ability. Reflective Practice Technique The reflective practice technique is a method or means that can be used for stimulating the learning process and/or foster reflective thinking behavior. It refers to ways for helping learners or practitioners transform information received into knowledge and, in turn, apply this knowledge to specific tasks or situations in professional or work settings; finally, reflecting upon the knowledge gained from completing a task or activity to benefit learning or work in the future. Learning Style Learning style refers to a learner’s personal preference for a particular way to learn. A leaming style is built from various constructs that are incorporated into a particular learning process. These constructs can include psychological, physiological, l3 environmental, sociological, and/or emotional aspects. Within the pro-set research framework for this study, there are three variables assumed: (1) learning technique utilization; (2) interaction to people and information sources (refers to the openness to information); and (3) time spent in learning activities. 1 Interaction (1) Interaction refers to an individual’s behavior in order to connect with others and within the learning and work environments. In this study, interaction is specified as a behavior that links the owner to information sources within the online learning environment. It also indicates the level of openness of each practitioner to information and experiences of others. This construct is included in the pre-set RP learning model as it is to be tested to indicate whether it is a cause that influences RP behavior. Reflective Practice Framework The RP framework is a model for professional education that provides both online students and practitioners with the capability to transfer knowledge into a professional reality and, in turn, reflect upon the knowledge gained from that practice in order to improve their work or future learning and decision making abilities. These operational definitions are used as a reference in the following chapters of this study. Following this introductory chapter, chapter two reviews selected literature related to this study and a brief analysis of its contents. The results of the content analysis in the chapter are used for identifying related variables and developing a conceptual model for this study. Chapter three includes descriptions of the research methodology, population and participants, theoretical framework, procedure for RP model development using both qualitative and quantitative methods. The instruments used in this study are 14' described in this chapter as well as how they were developed and used. In chapter four, findings from qualitative parts of this study, which were gathered from a preliminary questionnaire and focus group discussions, are reported along with their detailed analyses. The qualitative analyses in the chapter include both inductive and deductive approaches using content analysis, quasi-statistics, behavioral analysis, and logical reasoning. Quantitative analyses and findings are presented in chapter five in detail. Finally, chapter six includes a discussion of the findings and alterative choices for the RP theoretical model. The evidences from both the qualitative and quantitative parts of this study are merged together and used for issuing a preliminarily best-fit model, and some recommendations for future studies are found in the conclusion of this preliminary study. 15’ CHAPTER TWO Literature Review To provide theoretical and empirical information on which this study is based, this chapter presents literature review on major topics that are related to the research. The research questions, as indicated in chapter one, lead to the selection of these major topics: RP concepts, reflective learning techniques and application of RP to the online learning environment. The general idea of this literature review is to explore the concept of RP, its nature, process, and possible techniques that can be used for supporting RP behaviors within an online learning environment. A content analysis of the literatures is also conducted along with this review to define operational definitions, variables within the preliminary RP framework, and approaches of the RP implementation. There are two main sections in this literature review. The first section presents a review of the various concepts and definitions of reflective practice and synthesizes the concepts. The synthesized part can be divided into two aspects: (a) its contents, related constructs and variables; and (b) the RP process. The content reviewed in the section is synthesized into details for a conceptual framework development, which will be used as a preliminary outline of this research (see Table 2.1 for synthesized details). The RP operational definition is defined as well as identification of its elements and dimensions to be included in the RP learning framework (Operational definitions are presented in 16. chapter one). The second section reviews techniques of reflective learning both for online learning and teaching in general in order to identify the possibilities of combining the concept of RP with the online teaching and learning. It examines and illustrates various ideas and approaches that can be used for fostering RP behaviors. Reflective Practice Concept There are many concepts related to RP, and many definitions available in research literature, beginning with the writing of Dewey (1933) in the earlier theories of teaching, continuing with the work of Schdn (1983, 1987, 1991) and Kolb’s experiential learning (1984). Although there are similarities across the definitions of RP, offered by these theorists, there are also important differences to be recognized as one becomes acquainted with the literature on the subject. While there is no standard definition for the concept “reflective practice”, the term of RP is often described by linking to concepts of reflective thought, experiential learning, constructivism, action learning and critical reflection. Among a number of materials related to RP, the most frequently mentioned is Sch'o'n’s theory of action. Sehdn introduces the concept of RP in his book, The reflective practitioner: How professional think in action, in 1983. The concept has been applied to many professional fields, such as architecture, design and nursing, as a part of professional development process. While SehOn is generally credited with initiating the term of “reflective practice ”, the concept of reflective practice is similar to the earlier idea of Dewey’s learning by experiences. 17’ SchOn (1983) initiates the concept of reflective practice within his theory of action. He explains that the reflective practice is evidenced by an individual’s actions. Schdn indicates that reflective practice involves the critical analysis of everyday working practices in order to improve competences and to promote professional development. He describes his own reflective practice as "a dialogue of thinking and doing through which I become more skilled" (1987, p.31). He maintains that lasting behavior changes are possible, only if accompanied with changes in personal “theories-of—action” and he advocates reflective practice as the viable means to change personal “theories-of-action” as well as organizational changes. In his work, SehOn (1983; 1987) argues the conventional (technical-rational) view of professional practice, which assumes that professionals operate by applying formally learned specialization or technical knowledge. He disputes that this is not the only way in which professionals go about solving problems. In reality, they use a form of tacit knowledge: knowledge linked to specific activities, which he calls “kn‘owing-in-action In addition, they develop “repertoires ” of solutions and learn how to re-frarne difficult problems into those they can deal with more readily. As a result, their professional practice can be seen more as a form of artistry than applied theory. He argues that, practitioners should view professional practice as an interactive art that is based on the need of inquiry into practice (1983). Schr'jn emphasizes that the first step toward reflection is a well defined problem. He states that: When we set the problem, we select what we treat as the “things” of the situation, we set the boundaries of our attention to it, and we impose upon it a coherence, which allows us to say what is wrong and in what directions the situation needs to be changed (SchOn 1983, p.40) 18' He simplifies his concept of reflective practice by making a distinction between two types of reflection; “reflection-in-aetion” and “reflection-on-action”. SchOn calls them the ”new epistemology " of professional practice and, in particular, the concept of the “reflective professional He equates critical practice with “reflection-in-action”, which focuses interactively on the outcomes of action, the action itself, and the intuitive knowing implicit in the action (1983, p.56). In other words, our thinking serves to reshape what we are doing while we are doing it (SchOn 1987, p. 26). He describes three salient features of “reflection-in-action”. Although reflection is often not vocal, it is conscious. It is critical and it enables on the spot of experimentation. Whereas “reflection-in-action” is spontaneous and occurs in an ”action present ”, “reflection-on- action” refers to the process of making sense of “an action after" it has occurred and as such is considered to be more passive and more deliberate. He defines the knowledge derived from the reflection-in-action as unstable and tacit. While the knowledge-in-action is only learned through practice and it does not get through the process of problem refrarning, the knowledge-on- action is a cognitive process that depends on retrospection and constructive learning or given data. However, Moon (1999b) criticized these two types of actions as somewhat ambiguous: whether the time of reflection may involve a "stop and think” or it is always smoothly embedded into performance. Schr‘in (1983) says that professionals must be reflective practitioners, more specifically, individuals who understand the inextricable link between ideas, values, and the work done day-to-day. He develops the idea of combining learning and practicing together in his later work (Educating the reflective practitioner, London: Jossey—Bass 19' Publishers, 1987). He referred to John Dewey who stated that a student cannot be taught what he needs to know, but he can be coached: He has to see on his own behalf and in his own way the relations between means and method employed and results achieved. Nobody else can see for him, and he can't see just by being ’told' although the right kind of telling may guide his seeing and thus help him see what he needs to see. (quoted in SchOn 1987) Dewey (1933) clearly states that while we cannot learn or be taught to think, we do have to learn how to think well, especially to acquire the general habit of reflecting. SchO‘n terms “reflection-in-action ", as thinking that is embedded in action, emphasizes several qualities. The qualities include: the uniqueness of decisions facing practitioners; the affective components of reflection and a critical function which allows the practitioner to question and challenge strategies instead of assuming previous professional knowledge. Applied to professional education, these qualities characterize an approach which suggests that the theory and practice relationship is not simply a matter of applying a specific body of knowledge but involves the negotiation of unique and complex situations. Schdn (1983) argues that the primary professional competence is “reflection”. In his view, reflection is the key to acquire all other competencies and to maintain a process of continuous improvement. Interestingly, however, SchOn does not offer a comprehensive model of professional competence, nor any detailed analysis of the types of competencies needed by professionals, but the comprehensive conceptual model of reflective practice that greatly influences many later learning theories, such as David A. Kolb’s experiential learning, Senge’s organizational learning, etc. All these theories involve the same process of reflecting on behavior and using such feedback to learn and to modify behavior influenced from SchOn’s reflective practice concept. 20‘ While Schdn’s idea is probably the most widespread, Morrison ( 1995) highlights the fact that Schdn’s work is preceded by that of both Dewey (1933) and Haberrnas (1972, 1974). Earlier, John Dewey mentions the similar concept of learning from experience that “We learn by doing if we reflect on what we do.” (1933, p.19) His explanation of learning became the fundamental concept of reflective practice and provided the bases for experiential learning later. Dewey considers “reflective thought " to be not only an intellectual endeavor but also one that is involved with the entire person, including emotions. He indicates that reflective thinking is distinct from other forms of thought because "it- involves a state of doubt, hesitation, perplexity, mental difficulty in which thinking originates, and an act of searching, hunting, inquiring to find material that will resolve the doubt, to settle and dispose of the perplexity." ( 1933, p. 12) However, the process, in which he is most interested, is conceived as essentially mono-logic (Dewey, 1933). Reflective practice, in his view, is a way of examining one’s actions and improving the quality of one’s work. He believes that learning takes place when students have the opportunity to try out new behaviors and reflect on them. Dewey views learning from experience as the process of “connecting the achievements of the past with the issue of the present within the experience” (1938a, p.23). He indicates that all genuine education comes about through experience and that education is a development within, by and for experience. Conversely, Haberrnas ( 1972) argues that reflective practice has a social aspect as well as a psychological basis. It does not stop at the individual but, instead, may be dialogical (interaction with others). As such, he lays out the foundations for the politicization of contemporary professional practice, by suggesting that RP is not neutral 21 but can serve as a range of interests. Haberrnas is more concerned with illuminating the purpose and outcomes of RP rather than the process, with which Dewey and later SchOn are concerned. Due to the varieties of concepts and concerns related to RP as well as definitions and elements that have been allocated in various ways in the different literatures, the following sections will specifically synthesize and concentrate on two main aspects: variables related to the RP concept and the RP process. Practically, concentration on these specific issues would help readers understand what part of this review would be used for the preliminary framework of this study. To identify variables related to the concept, definitions of RP are presented and synthesized using content analysis (see Table 2.1 for details). Definitions of Reflective Practice and Related Variables The earliest quoted definition of RP comes from Dewey (1933), the educational guru and most mentioned theorist. He defined reflective practice as “. .. the persistent, and careful consideration of any belief or supposed form of knowledge in the light of the grounds that support it and the firrther conclusions to which it tends.” (p.6). Dewey (1933) describes three attitudes that a practitioner (teacher) is assumed to have in order to practice (teach) reflectively: (a) open-mindedness--one’s ability to suspend judgment and to be open to new possibilities, (b) whole heartedness--the capacity to enter into the teaching event with all of one’s heart and attention and (c) responsibility--the moral and ethical implications inherent in the educational process. Dewey also suggests that a practitioner (teacher) must never uncritically accept suggestions or solutions, and that one 22 must always suspend judgment during the necessary period of inquiry, carefully weighing the evidence provided and the consequences of one’s actions before making instructional decisions. It is the link between reflection and improving practice (classroom instruction) that is the foundation for his vision of RP. Beyond engaging the good three attitudes to be opened to the RP learning, reflective learners are assumed to have the habit of the reflective behavior because reflective practice is considered a habit of mind that one develops within the context of one’s practice. Valli (1992) states that, “. . .reflection is the capacity to ‘notice oneself noticing’ that is, to step back and see one’s mind working in relation to its projects.” (p.99). Osterrnan (1990) describes reflection as being linked to thought processes and action, as well as developing one’s craft of practicing. Osterrnan (1990) defines that “ ...[RP] is the mindful consideration of one’s actions, specifically one’s professional actions... [and] is a challenging, focused and critical assessment of one’s behavior as a means towards developing one’s craftsmanship.” (p. 134) Many researchers in the educational field define RP as an information inquiry from existing experience. For example, Van Manen (1977) broadly defines RP as the systemic inquiry into one’s (teaching) practice and the deliberate attention to one’s experience. More specifically, RP is a conscious, systematic, deliberate process of framing and re-framing (classroom) practice, in light of the consequences of our actions, democratic principles, educational beliefs, values and preferred visions teachers bring to the teaching-learning event (Serafini, 2001). As introduced by Dewey (1933), RP is also described to be involved with attitudes as well as moral and social aspects. Carini (1979) states that “. . .[RP] shares 23' with these modes of thought, an attitude of concentration, focus and openness.” (p.26) Kemmis (1985) includes action and the social contexts of one’s actions with the meta- cognitive aspects of reflection. He elaborates that “. . .reflection is a dialectical process; it looks inward at our thoughts and thought processes and outward at the situation in which we find ourselves.” (p.57) From these definitions of RP as a social process, it can be inferred that RP does not occur as an individual activity, rather than reflection as a social activity, teaching and learning are always grounded in a particular physical, political, socio-cultural and historical context. From these literatures, RP can be briefly defined as a desired behavior for any practitioners, in which its owner is assumed to be more effective in working and learning than other people who work and learn without it. It is a conscious, systematic, deliberate process of framing and reframing practice, in the consequences of our learning and practicing. It involves knowledge, beliefs, values and objectives within physical, political, socio-cultural and historical context. It also includes the link‘between reflection and action: the propositions that problems are framed, unanswered-undiscovered parts of the practice are challenged. The literatures review indicates a number of candidant variables, which can be included in the theoretical model to be used in this study. Only some discrete variables are selected to be used as a testing ground in this study. Upon the analysis in Table 2.1, these variables are identified: experience, (previous) knowledge, thinking: critical, reflective, level of thinking, interaction (so'eial context), learning performance, working performance improvement (competence improvement, professional development). To provide more understanding about these variables, the next part of this review examines 24 how these variables are involved in the RP process and what the literatures say about implementation of RP. Table 2.1 Definitions of Reflective Practice and Related Variables . . . . Related Variables Authors Identification & Definitions & Key Findings Dewey “Reflective practice is a way of examining one’s (1933) . . . . , ” actions and rmprovrng the quality of one 5 work. “While we cannot learn or be taught to think, we do have to learn how to think well, especially acquire the general habit of reflecting.” “We learn by doing if we reflect on what we do” “Reflective thought does not relate to only an intellectual endeavor but also one that involves the whole person, including emotions. . .. it involves a state of doubt, hesitation, perplexity, mental difficulty in which thinking originates, and an act of searching, hunting, inquiring to find material that will resolve the doubt, to settle and dispose of the perplexity.” “Learning from experience is the process of “connecting the achievements of the past with the issue of the present within the experience.” “The persistent and careful consideration of any belief or supposed form of knowledge in the light of the grounds that support it and the further conclusions to which it tends.” - work quality improvement 0 habits of reflecting - practicing (doing) - Reflecting ° emotions - doubts ° thinking originating - searching - inquiry - experience 0 connecting ° thinking process(carefirl consideration) - knowledge 25' (table continues) Table 2.1 (continued) Definitions of Reflective Practice and Related Variables Related Variables Authors Identification & Definitions & Key Fin dings Haberrnas “Reflective practice has a social aspect as well as : Schgoalipigfl (1972) a psychological basis. It does not stop at the his: g individual but instead may be dialogical . interaction with (interaction with others)” others “[RP]. .. the systemic inquiry into one’s . systematic Van Manen, (teachin ) ractice and the deliberate attention to thinking (1977) , g P. ,, - deliberation one 5 experience. . - experience Carini “[RP]. .. shares with these modes of thought, an : 2233?; (1979) attitude of concentration, focus and openness.” - openness - working practice Schdn “Reflective practice involves the critical analysis 0 competence (1983,1987, 1991) of everyday working practices in order to improve competences and to promote professional development.” “[RP]. .. the conventional (technical-rational) view of professional practice, which assumes that professionals operate by applying formally learned specialization or technical knowledge.” “Professionals must be reflective practitioners, individuals who understand the inextricable link between ideas, values, and the work done day-to- day.” “Reflection is conscious although not often vocal, it is critical and it enables on the spot of experimentation.” improvement 0 critical analysis 0 professional development - technical knowledge ° ideas - values - practicing situations 0 conscious ° critical 26 (table continues) Table 2.1 (continued) Definitions of Reflective Practice and Related Variables Authors Identification & Definitions Ram" Va’mb’es & Key Findings Boyd and “Reflective learning is the process of internally Fales (1983) examining an issue of concern triggered by an 0 experience experience which creates and clarifies meaning in - values & beliefs terms of self, and which results in a changed (perspective) conceptual perspective.” Kemmis “Reflection is a dialectical process; it looks (1985) inward at our thoughts and thought processes and - thought process outward at the situation in which we find - (work) situation ourselves.” Irnel (1992) “Reflective practice is a crucial element of that art Lllirgvgéedge 0f form that integrates or links thought and action with reflection. . ..Engaging in reflective practice . awareness . . - professronal requires both knowledge of practice and . awareness of professional and personal philosophy philosophy.” . personal philosophy Valli (1992) “. . .reflection is the capacity to ‘notice oneself - mind noticing’ that is, to step back and see one’s mind 0 work practice working in relation to its projects.” (project) Jarvis “Reflective practice is commonly used by (1992) professionals as they meet new and different - practicing situations and challenges. ...Reflective practice is situation more than just thoughtful practice; it is the 0 learning situation process of turning thoughtful practice into a 0 reflective thought potential learning situation.” Atkins & “Reflection relates to a complex and deliberate Murphy process of thinking about and interpreting - deliberate process (1995) experience, either demanding or rewarding, in - experience order to learn from it.” (table continues) 27' Table 2.1 (continued) Definitions of Reflective Practice and Related Variables Authors Identification & Definitions Related Variables & Key Findings Caffarella & “. . .three major aspects associated with reflective Merriam practice: a commitment to problem finding and (1999) problem solving; the notion of making judgments about what actions will be taken—indicating an ethical dimensions; and the fact that some form of action will result from the process, even if it is the deliberate choice of in-action.” Serafini “. . .reflective practice is a conscious, systematic, (2001) deliberate process of framing and re-framing (classroom) practice, in light of the consequences of our actions, democratic principles, educational beliefs, values and preferred visions teachers bring to the teaching-learning event.” Van “Critical reflective working behavior seems to be Woerkom et a construct consisting of six dimensions, namely al. (2002) reflection on oneself in relation to the job, critical vision sharing, challenging group-think, asking for feedback, experimentation and awareness of employability.” - problems (in practice) - ethical issues - knowledge (facts of practice) - practicing situation 0 beliefs 0 values 0 learning events - self-reflection - vision sharing 0 group thinking 0 asking for feedback 0‘ experimentation ° awareness of employability Reflective Practice Process To determine the RP process for a preliminary outline of this research, there are three main learning models discussed in this section: (1) Sch'o'n and Argyris’s double- loop learning, (2) Kolb’s experiential learning, and (3) Boud’s learning from experience model. Some extensions from other theorists based on these models, will be also briefly discussed at the end of this section. 28' Scho'n and Argyris ’s Double-loop Learning. According to SchOn (1983), practitioners oflen say they believe in one theory, but their actions indicate another belief. Reflections on a person’s beliefs may add to individual understanding and knowledge base and how and 'why one makes decisions and solves problems. It is commonly presumed that these beliefs or “espoused theories” guide actions, but in reality, this is usually not the case (SchOn, 1983). Espoused theories are “theories of action” that we believe we follow and are different from the theories-in-use that would be inferred from our actual behavior. SchOn indicates that, typically, actions are guided by theories-in-use, which are implicit in patterns of spontaneous behavior with others (1983). In real life, we are aware of our espoused theories, but oflen unaware of our theories-in-use. The two are frequently inconsistent, notably in situations that trigger embarrassment or threat. We seem to be socialized in a particular theory-in-use, called model 1 or the unilateral control model, that comes out in these situations. Model I leads to low trust, low commitment, and limited learning. Argyris and SchOn (1978) describes RP process as an alternative theory-in-use, model 11. Model 11 is a theory of joint control and inquiry. Its underlying values are valid information, free and informed choice, and internal commitment. The primary strategies are to combine advocacy and inquiry, to make reasoning explicit and confrontable, and to encourage others to do the same. They refer to the process of “double-loop learning” (see figure 2.1) and advise that practitioners who find theories do not always fit with the uncertain realities of practice may be forced into becoming reflective. The consequences include an increasing capacity for learning not only to improve strategies for achieving existing goals (single-loop learning), but also to choose among competing norms, goals, 29 and values (double-loop learning). Most people readily espouse model 11, yet are unable to act consistently with it (Argyris, 1993). Learning to design model 11 action, moreover, is not simply a matter of learning new techniques. It also requires change in underlying values and assumptions that structure one's theory of practice. Developing competence in model 11 enables people to learn in the midst of difficult circumstances and to act as agents of organizational learning. Theories-in use is a useful theory in order to explain and use in guiding human behaviors. Governing _ Action L Consequences Variable ' Strategy 7 A I A L Single-loop Learning Double-loop Learning Figure 2.1. Single-loop and double-loop learning (Argyris & Schdn, 1978) While double-loop learning is defined as “learning that results in a change in the values of theory-in-use, as well as in its strategies and assumptions” , single-loop learning involves the learning of means to achieve given ends: confined to the way to achieve the ends (instrumental) and tactical issues concerning what is the best way to achieve overarching objectives represented by a framework of assumptions, values, goals or governing variables, while double-loop learning involves questioning overarching values, assumptions, goals and governing variables which function as the constraining framework (Argyris & SchOn, 1978). Therefore, double-loop learning involves 30' questioning the ends to which professional action is addressed, rather than the means of achieving given ends, and it is strategic and non-instrumental. Although single-loop learning is concerned with reflection upon detailed practice, double-loop learning is regarded as the more fundamental level of learning and the tone requiring deeper reflection because it has questions on the assumption framework, which determines action. As such, the promise is held not just for improving practice, but changing it in fundamental ways. K olb ’s Experiential Learning. David A. Kolb, another theorist whose work is associated with the concept of learning from experiences, demonstrates that all learning involves learning from experience and that learning from experience is the process whereby human development occurs. Kolb (1984) suggests that “learning is a process whereby knowledge is created through the transformation of experience. "(p.38) Kolb’s ideas build On and integrate idea from the foundational model of experiential learning of Dewey. He describes a learning model that is grounded in experience. In his view, students actively reflect on their experience to develop concepts and plan action by setting new goals and strategies for learning and the process is repeated itself. Kolb (1984) identifies characteristics of experiential learning, which are fundamental to his view of the learning process. He indicates that these propositions derive from the traditional learning processes (views of Dewey): 1. Learning is best conceived as a process, not in term of outcomes; 2. Learning is a continuous process grounded in experience; 3. The process of learning requires the solution of conflicts between dialectically opposed modes of adaptation to the world; 31’ 4. Learning is a holistic process of adaptation to the world; 5. Learning involves transactions between the person and the environment; and 6. Learning is the process of creating knowledge. (Kolb 1984, pp. 26-38) Kolb (1984) outlines his experiential learning model, which involves learner’s learning preferences located in one of four quadrants. The learner’s preferences are produced from features of personality. The degree of introversion/ extroversion, and preferences towards using the left or right hand side of the brain are influential in producing preferred styles of learning, which prioritize learning from the concrete or abstract and reflection or action. He defines four main styles of learning: 1. Concrete experience - feeling - having the experience, 2. Reflective observation - watching - reviewing the experience, 3. Abstract conceptualization - thinking - concluding from the experience, and 4. Active experimentation - doing - planning the next stage (Kolb 1984, p.41). Kolb (1984) conceptualizes learning as the fundamental of human behavior that is adapting to changing circumstances. He proposed two basic learning processes-the process of grasping information through the mode of either concrete experience (CE) or of abstract conceptualization (AC) and the process of transforming information through the mode of either active experimentation (AE) or of reflective observation (RO). In his view, learning requires engagement of all four of these behavior modes, and thus the entire learning process can be modeled as a cycle moving from a concrete experience, to reflective observation about the experience, to development of abstract conceptualizations about the experience, to active experimentation around the experience (see Figure 2.2). The theory maintains that each of us develops a preferred mode of carrying out each of the two basic learning processes and that individual differences exist 32 in the characteristics of our own learning cycles. Four basic learning styles-diverging, assimilating, converging, and accommodating-arise from combining the two modes of the two learning processes. Concrete Eirperience (CE) Comprehension Accommodating Diverging ' A cm e Intention Extension ReflectiVe Experimentation, Observation (AE) (R01 .:.Legend: . Prehensioerimension . Transiforr‘nation Dimension Figure 2.2. Kolb’s experiential learning model (adapted from Kolbl984, p.30) Kolb indicates two adaptive learning dimensions in his model: the prehension dimension and the transformation dimension. There are also dialectical tensions between the adaptive modes. The prehension dimension includes the apprehension — comprehension dialectic, while the transformation dimension includes the intension — extension dialectic. Kolb explains that “the prehension dimension describes the current 33 state of our knowledge of the world — the content of knowledge, if we will — whereas the transformation dimension describes the rates or processes by which that knowledge is changed” (Kolb 1984, pp. 101-102). Kolb argues for a hierarchy of development or learning sophistication, where the four basic styles are at the first order of development. The circle appearing in the lower portion of the diagram, in Figure 2.2, represents this first order of development. Relying on one of these basic learning styles means that an individual resolves the dialectics of each learning process by focusing on one mode at the expense of the other mode. This would manifest itself in relative inflexibility in one's responses to in the-moment events, which corresponds to Sehdn’s “reflection-in-action”. Thus, a learner who shows an assimilating style would always tend to respond to such events with abstract and reflective behaviors. In contrast, a learner who shows a converging style would tend to look for an immediate solution to what he or she perceives to be the problem. Learners may possess the ability to pass through all the stages of the cycle, but most people have a preferred learning style. The style is preferred or dominant but not exclusive. We each enter the circle at the point of our own preference and move according to our needs and circumstances. In some instances, the learner just learns in one style and sees no need to move. While Kolb (1984) refers to learning styles as individuals’ characteristics, which are influenced by different sides of brain dominances, Carl Jung (1927), expressed major differences in the way people perceived (sensation versus intuition), the way they made decisions (logical thinking versus imaginative feelings), and how active or reflective they 34' were while interacting (extroversion versus introversion) at the beginning of learning style theorizing. While Jung (1927) and Schdn (1983) have described reflective learning, in the same way as a cyclical process, in which knowledge built upon experiences, Kolb identified the experience as the starting point for learning. He does not treat reflection as a whole cycling process, but a point of learning process. Kolb also looks at “reflection” as “reflective observation”, during which students need to stop for observation before making their “abstract conceptualization”. Therefore, in Kolb’s view, students must “stop to think”, while Scho'n describes both “reflective-in-action” and “reflective-on-action”. There are also other theorists who have developed their models of learning from experience based upon SchOn and Kolb’s model, such as David Bond. David Bond’s Model of Learning from Experience. David Boud has been a significant contributor to the theory of experiential learning. He outlined a model of learning from experience, in which he emphasizes a “culturally-embedded nature” of learning (Bond 1994, p.53). Boud believes that his model will aid thinking about learning from experience and how it can be facilitated (p.49). He addresses three aspects of the experiential learning process or “the three stages of engagement in the learning event”: preparatory activities, activities--experience during the learning event and reflective process that occur after the learning event (see Figure 2.3 for the model). This three stages concept is compatible with Schdn’s concept of reflection-in-action and reflection-on-action. Bound outlines two assumptions on which the model is based: 35' “The basic assumption of the model is that learning is always rooted in prior experience and that any attempt to promote new learning must in some way take into account that experience. . .Learners bring with them to any event their personal foundation of experiences. . .The second assumptions behind the model is that the process of learning from experience is necessarily an active one, which they are a part. This engagement and intervention is with what is termed the learning milieu — i.e. the social, psychological, and material environment in which learner is situated.” (Bound 1994, p.50) Focus on: Return to Experience Reflection in Action Learner l Milieu . Personal Attend to Skill/Strategies Foundation Feelings 0f Experience Re-evaluation PREPARATION EXPERIENCE REFLECTIVE PROCESSES Figure 2.3. Boud’s model of learning from experience (adapted from Bond 1994, p.51) Prior to the learning event, Boud (1994) suggests three considerations: the learner, the learning milieu, and skills/strategies. With respect to the learner’s experience during the learning event, Boud suggests that “it is the learner’s engagement with the milieu, which constitutes the particular learning experience. Learners create a learning milieu through their presence and interaction with it. By noticing, intervening and reflection-in- action, they steer themselves through the milieu in accordance with their intents and what is available for them to use in his process (p.51). Following the learning event, he also 36 suggests that “much important learning can occur following an event as the distractions of the milieu and the lack of opportunity to stand aside from the dynamics of the action limit what it is possible to do at the time. Some aspects inevitably take time and the ability to view particular events in a wider context (p.52). He identifies that reflection after the event has three elements: (a) returning to the experience, (b) attending to feelings and (c) a re-evaluation of the experience. Influenced by Kolb’s experiential learning, Osterrnan & Kottkamp (1993) described RP as a cycle that begins with a concrete experience, continues with observation and analysis of the experience, develops abstract re-conceptualizations to generate different perspectives concerning the experience and ends with active experimentation where the practitioner puts something into action. Additionally, Ross (1989) describes a cycle that begins with: (1) recognizing the educational dilemma, (2) recognizing the similarities to other educational situations, (3) reframing the problem, (4) considering various consequences of the .various alternatives proposes, and (5) arriving at a judgment to act upon. Additional to the three main theories of professional learning demonstrated by Schdn, Kolb and Bond, social contexts have been brought to the RP process as discussed by Smyth (1989) and Kemmis (1985) later. Smyth (1989) reveals a more social perspective on the reflective process when he described RP as a process of: (a) describing, (b) informing, (c) confronting, and (d) reconstructing. (p.271) He suggests that reflective practice is a process of re-visualizing various experiences to understand the historical and political implications of the situation in order to make better decisions about the moral and the social ramifications of one’s (instructional) decisions. Both 37 Smyth (1992) and Kemmis (1985) consider RP as a social process that is embedded in the historical, political and social contexts of the educational event. Reflection is shaped by ideology, a relationship between thought and action in real historical situations in which teachers find themselves. Reflection is not value-free: rather it serves particular social and political interests. Kemmis (1985) states that, “. .. reflection is not a mechanical, technical process: it is a practice which expresses our power to reconstitute social life by the way we participate in communication, decision making and social action.” (p.149). Reflective practice is visualized as both an individual cognitive process, as well as a process that is inextricably embedded in the social, political and historical contexts in which it takes place. Obviously, all of the literatures reviewed describes a particular process one progresses through when reflecting. These steps or stages of the RP process generally include: (a) identifying the nature of the problem (practice), (b) generating possible solutions, and (c) analyzing the various alternatives to solve the problem. Though a number of literatures and theories are described in various ways, the consensus is that RP is a linear, step by step process with a predetermined sequence and objective(s) of practice. The literatures describe the same idea of process that begins with concrete experiences, which is informed by individual observation and analysis, followed by generating abstract generalizations about the specific experience in question, and ends with some form of active experimentation or decision. According to the literatures, a pro-assumed theoretical model is set. The variables within the model include: experience, (previous) knowledge, thinking-critical, reflective, and level of thinking, interaction (social context), learning performance, and working 38 performance improvement. To make the model testable and demonstrate causes and effects of each variable, the variables are arranged into clusters. There are two main clusters for independent variables: individual attribute and online learning behavior, which are assumed to have casual effects to three main dependent clusters: thinking behavior, learning performance and working performance improvement (see Figure 3.2 for the preliminary model). The preliminary model development is further discussed in chapter three. Reflective Practice Techniques and Applications The availability of increasingly web-based, networked technologies offers opportunities for creating and sustaining collaborative, reflective learning experiences for a professional education. When the technologies are combined with reflective learning techniques, they can be used to encourage students to look beyond their academic accomplishments and recognize that the depth and range of others’ transferable knowledge and working experience benefit them as well as the theory-based knowledge received from classes. Even if we accept the notion that RP is not something we can be taught to do, we must explore how we can cultivate RP behavior and what strategies are that we can use to foster the behavior using technologies. According to Schdn (1987), we can help practitioners engage in RP by observing, describing and trying “to illuminate the things practitioners actually say and do, by exploring the understanding revealed by the patterns of spontaneous activity that make up their practice. He goes on to indicate that the primary concern is to discover and help practitioners discover what they already 39' understand and know how to do” (p.5). Therefore, to design a practical online learning environment, we have to consider all those related variables involved in learning process as well as the teaching approach to support good RP behavior. Generally, an online learning environment is leamer-centered--there is a multidirectional flow of information. Course participants act on knowledge as they read course material and think about it. They formulate their own thoughts about the ideas they come across, share them with others, and in the process transform their mental maps (Odin, 1997). Students can be encouraged to obtain information from other on the web through participation in discussion groups, conferences, chat rooms, or via electronic mail. Given the nature of communication tools, students must communicate by writing. Therefore, students who have not had much opportunity to participate in regular traditional class or have language barriers are more likely to contribute in online discussion and have more time to express their ideas carefully. Online learning environments, by their nature, foster active participation and require learners to take an active role in the learning process. However, there are both group and individual techniques that can be used for fostering reflective learning behaviors and to foster the reflective learning behavior; we have to encourage “critical reflection” and cultivate a reflective learning culture among learners. To make it happen, the role of instructor/teacher is beyond providing knowledge or information; the important role is encouraging effective collaboration among learners while leading and guiding. In an effective online learning environment, the instructor plays an important role as “the content expert” to explain, clarify, direct, and help students learn from their own knowledge base (Odin, 1997). 40 To build theculture of RP, we need to encourage reflection. Boud and Knight (1996) point out the importance of introducing and establishing an effective climate for reflection illustrating that “All learning requires learners to actively engage with knowledge, but reflection makes such engagement an essential part of the process.” (p.28) They suggest eight concerns for introducing and establishing a climate of reflection as follows: 0 Articulating an educational rationale for the process 0 Introducing a simple exercise to illustrate reflection 0 Providing an opportunity for students to clarify their understanding of the idea 0 Introducing a framework or model to aid thinking about elements of reflection - Modeling a reflective approach in one’s own presentation of the idea 0 Identifying an area of the process that student can make their own 0 Providing time o Treating reflection as a normal activity (Boud & Knight 1996, p.28). There are many strategies that can be used to encourage critical reflection and RP, such as writing a journal, case analysis, and concept mapping. To illustrate different techniques of teaching and learning that foster reflective practice behaviors, the following review separates into to subsections: individual techniques, group techniques and multiple techniques. Individual Techniques Learning Journal. Moon (1999a) indicates that the most popular technique to promoting reflection is a “learning journal”. There are many different types: workbooks, diaries, logs, portfolio, progress files, and profiles. Boud (2001) indicates that: “We write a journal for different reasons prompted by many different purposes. We may want to 41 capture an experience, record an event, explore our feelings or make sense of what we know. We may want to narrate something of importance so that others can see what we saw in it (p.9), but the main purpose of a learning journal is individual focusing: for students to look back over their recent learning and order their thoughts in order to identify strengths, weaknesses, new levels of understanding learned and review of learning experiences. They are also used increasingly to record learning in situations like fieldwork and work placements. A reflective journal chronicles the "thoughts, feelings, successes, and frustrations that are a part of each student's journey as an emerging professional" (Banta et al., 1996, p. 114). The journal allows students to reflect on the activities in their own time. This process is similar to Schon's (1983) suggestion of "reflection-on-action", the development of a capacity beyond technical expertise which invites professionals to respond to uncertain and complex circumstances. McAlpine (1992) considers journal writing as a professional conversation, carried out in written form to provide opportunities for question and concerns related to experiences. In this view, descriptive and reflective writing can be used to relive feelings and emotions or to use these feeling and emotions to understand the impact of experience on practices. Journals may be confidential or open to share with a learning team as an assignment. For online students and instructors, if they have some, it provides an excellent record of a practitioner’s development over the length of a program and allows opportunities to identify exactly where a particular student may be struggling and requires further support. In addition, to the deeper benefit, the journal's potential can be a component of a broader professional development document. It is an excellent record of 42 that individual's experience and professional progression. The process of maintaining such a document is an important step in encouraging the graduate to take control of their professional development and career management as a life-long learner. Many different methods for reflective writing exist and. the content of different disciplines will largely govern the technique used. A description of contrasting approaches to keeping a journal are described by Moon (1999a), outlined below: 1. Diary and autobiographical writing, 2. Prescribed writing sessions at the end of exercises, I 3. The asking of questions to provide structure and to guide issues, 4. Use of a journal to accompany other learning (e. g. to run along-side a research dissertation), and 5. Built into Professional Development Profiles or Portfolios (which are broader in scope, including other documents relating to one's professional life). Learning Contract. Learning contracts are documents used to plan a learning project, negotiated between the learner and lecturer to outlinethat a particular activity will be undertaken to achieve specified learning outcomes. Their format usually includes consideration of the learning objectives of the project, the strategies and resources available, the evidence to be produced to prove achievement of the objectives, and criteria to be used for assessment (Anderson et al., 1996). Contracts are important prompts for reflection because they oblige the student to step—back and review their learning at various stages. The first stage occurs prior to submission of the initial draft contract where the student is required to reflect on past experiences to establish relevant 43 learning needs and how these may be met. Secondly, at the negotiation stage, where the proposal is subject to challenge by the tutor, usually requiring a re-evaluation and revision of the document. Thirdly, it is important upon completion of the project when the student needs to review the outcomes of their learning and ”how they can be presented to another person (Anderson et al., 1994). Critical Incident Analysis. Critical incident analysis is a technique used in disciplines such as health-care where there is a strong interaction between academic work and practical experience (Ghaye & Lillyman, 1997). Practitioners keep short records to document analyses of an experience or incident, which is then used to improve practice and apply scientific knowledge [the term critical incident can be misleading; it applies simply to any event that can be used as a learning experience, not a major “emergency” type incident as initially implied]. In practice this involves recording the incident and its analysis in a consistent format. The format may involve a sequence of set questions such as, what is a description of the incident, by whom it was handled, what learning occurred, what were the outcome of the incidents, how has this incident affected your practice, etc. The key point is that analysis focuses on “uh-picking” the event in order to understand the significance of the practitioners responses in order to recognize and document progression in their professional competence. Critical incident analysis technique differs from learning journals in that they are not used like diaries, but written at the discretion of the practitioner when a significant “incident” or learning experience is deemed worth noting. Critical incident analysis can be also used as a group technique. Brookfield ( 1992) describes the use of critical incident analysis as a group technique that, in critical decision debate, participants debate and support views that they personally agree on. After arguments have been made, participants then consider the extent to which their original positions have been modified. In crisis decision simulations, participants are placed in hypothetical crises where immediate decisions are pressing. When the decision is completed, participants are asked to reflect on the assumptions and reasoning that informed their choices. Brookfield believes that the activity brings uncovering assumptions “closer to the lived experience of the learners” (p.14). This approach alerts participants to their values and assumptions. More specifically, it focuses on the identification and analysis of events remembered for their emotional significance (Brookfield 1992, p.18). However participants draw on real life experiences to hear the insights concerning their assumptions from colleagues who have developed through other group of simulations. Group Techniques Learning Partners, Critical Friends, Group Support-Discussion. These techniques are effective methods for encouraging reflection. The strategy is to assign students partners with whom they can exchange ideas and discuss general aspects of their learning (Beardon, 1995). The partner is asked to comment deliberately on these ideas in order to deepen reflection. Boud & Knights (1996) point out that this strategy has important benefits for part-time students who have less opportunity to interact with their peers. Learning partners and critical friends methods may be used in action learning sets, which involve a group of students (a set) working together for a period of time to look 45 back over experiences, generate discussion of issues raised, place deliberate attention on the relationship between reflection and effective action, and propose a way forward. McGill and Beaty (1995) describe how a set may operate with three students. After confidentiality has been agreed, the first student acts as a presenter to describe an issue or concern that they have regarding their learning. The second student acts as an enabler to help the presenter think, through their “problem” and specify steps that can be taken to solve the problem. The third student has the role of observer, to listen and provide feedback on the effectiveness of the “enablers” comments and action proposed. This process is rotated so that each student occupies each role. Costa and Kallick (1993) specify using a critical friend as one path to reflection. They explain that a critical friend provides feedback to an individual about learning through the eyes of the learner. The learner first describes a practice and requests feedback from the critical friend by asking questions to understand the practice and clarify the context. Then the learner sets intended outcomes and the critical friend provides significant feedback. The critical fiiend continues to raise questions and critiques the work helping the learner see different perspectives. This technique requires both participants to reflect and write. The learner writes about the context of learning while the critical friend writes suggestions and advice for the learner. In this process, the learner does not have to respond or make decisions; merely reflect on the feedback without having to defend. Costa and Kallick (1993) recommend this strategy for people who find themselves isolated or too busy to reflect on their practice. Group discussion is another way to facilitate articulation of thoughts and argumentation among participants in group. Kanuka and Kreber identify that: 46 Instructors can use debates to enhance their learners' confidence and ability to express viewpoints as well as help them to develop coherent organization and precise expression of ideas structured in a manner that matches the speaker's (or writer's) purpose and intended audience. The desired learning outcome of a debate is to force learners to confront situations that result in contradictions that challenge the learner to acquire better understandings (Kanuka & Kreber, 1999). _ Group discussions also provide an opportunity for learners to share their knowledge, beliefs, values, and experiences within a small group of people. Generally, group discussions are spontaneously formed with guided tasks to be accomplished within a period of time. The tasks generally include: idea-generation, information gathering, question gathering, list-making, or problem solving, while the desired outcomes of group discussions are varied, but often include: provision of individual input in large classes, ice breakers, and assessing previous knowledge and experience (Renner, 1997). Practice-based Assignment, Project-based Learning and Problem-based Learning. Practice-based assignments are described by Doel and Shardlow (1989) and Evans (1991) as assessment methods which seek to elicit evidence of both the practice performance and also of the reflection on that performance. They describe that practice settings typically emphasize the details of practice performance, while academic settings seek to ensure reliability in assessing judgments and the accessibility and appropriate selection of evidence. Evans (1997) indicates that practice-based assignments can be both specific interaction and whole setting. Assignments may be designed to be focusing on a specific interaction between the student and one or more others. Whole setting assignments seek to elicit evidence of another key aspect of practice and understanding the nature of the work situation within which the student is practicing: its structures, policies, resources, procedures and consequence of the practice. 47 The concept of practice based assignment is similar to problem based learning as described by Bridges & Hallinger (1995): Problem based learning is grounded in the belief that learning involves both knowing and doing. In problem based learning prior knowledge is activated to incorporate new knowledge, opportunity is given to apply it and participants encode the new knowledge in the context of the practice setting (p.5). The problem based learning approach seeks to meet a range of goals including identifying problems, developing skills to solve problem, and seeking solutions and developing alternatives for effective solutions of working situations. Participants in problem based learning need to reflect about the problem and develop alternatives for the working situations. Brookfield (1992) discusses the way of developing solutions for working situation as scenario analysis. He indicates that scenario analysis is an exercise that explores actions of various fictional characters, usually at a critical juncture in decision making. Participants follow a four step process: 1. Individually, each person lists the assumptions that they think underlie the character’s choices and decisions. 2. In small groups, these assumptions are presented and the most common ones identified. 3. Group members explore how the most common assumptions could be checked out by the characters for their accuracy. 4. Group members generate an alternative interpretation of the scenario that could explain the character’s behaviors but that the learners judge would probably come as a surprise to those character (Brookfield 1992, pp. 13-14). Multiple Techniques Reflective Questioning. Reflective questioning, as described by Patricia (2000) is a three-in-one review strategy that uses self—questioning, four communication skills, and three levels of thinking plus a review of class notes, both individually and as a group. The 48’ seeds for comprehension are in the self-questioning and the written and oral exchange. When students stop questioning they stop thinking and comprehending. Keep students thinking and comprehending through the reflective questioning strategy. After all, comprehension and enjoyment are what reading is about (p.487). Reflective questioning is generally designed to require the learners to engage in more effective thought (Sanders, 1990). Questioning is considered as the "single most influential teaching act because of the ability of questions to influence the learning process" (Sanders 1990, p. 119). Desired learning outcomes of the question method often include: stimulating thinking, subject matter review, discussion leaders, and an ability for the learners to be active participants (Renner, 1997). Case Study and Case Analysis. Ashbaugh and Kasten (1993) describe a case study as “a philosophical critique of practice in which deliberative action is derived from empirical and interpretive modes of inquiry” (p. 158). Case studies can be used to help develop RP and transform experience. Ashbaugh and Kasten emphasize how case studies encourage dialogue, or internal conversation as the reader tries to understand the case and make inference. Understanding in the case studies will bring understanding in experiences to learners, providing vicarious experience of familiar events as well as providing an occasion for double loop learning. A case study provides information about a simulated (or sometimes real) situation (Marsick, 1990a); learners respond to predetermined questions or develop an action plan. If cases are developed so as to bring about a questioning of learner assumptions and if learners are also provided with the opportunity to examine those assumptions in 49' interactions with others, critical self-reflection will be fostered. Kanuka and Kreber ( 1999) indicate that case studies can be used to help student develop analytical skills, improve group thinking skills, transfer skills to operational settings, provide opportunities to learn another's point of view, provide active rather than passive learning and act as bridge between espoused knowledge and actual practice. However, there are some weaknesses in using case studies in teaching. Ashamalla and Crocitto (2001) indicate that the case study method may lead to “students' lack of identification with the case, poor preparation, and/or reluctance to participate in a large group setting” (p. 159). When we use case studies in discussion, the instructor may overly control the case discussion (Argyris, 1980), with students limiting their comments. Therefore, case analysis in group discussion may damage students' participation and become an end in itself. Another critique of the case method concerns transfer validity, in which some cases depict problems that may not represent the current complexities of organizations [work situations] and/or may have limited information available (Jennings, 1997). Therefore, student-developed cases may be more effective than hypothetical cases to connect theory with practice or real-life events and thus address some of the limitations. Problems and Concerns in Implementation of Reflective Learning Techniques Even though there are various techniques that we can use for fostering reflective learning and RP behaviors, there are some concerns for using those techniques due to lack of experience and misunderstandings of the nature of learning and reflection. 50’ For some learning models, learning activities typically force students to follow a sequence of steps of reflection and require them to reflect on demand. Elements of the learning activities became check lists, which students follow mechanically without regarding to their own learning. Boud and Walker ( 1998) call this kind of learning “recipe following”. While a check list of learning activities can be useful, it is also a risk of blocking reflective thinking. They indicate that “when combined with a teacher- [center approach] rather than a student-center approach to education, rule following turns ‘reflection’ into a process of memorized and applies unthinkingly.” (Boud & Walker 1998, p.192) There is a concern about what Boud & walker call “reflection without learning”. They indicate that while reflection is important, not all planned reflective processes lead to learning. Inadequate, inappropriate, or badly used reflective activities can become an obstacle (Boud & Walker 1998, p.193). An example of an inappropriate plan of reflective learning is leaving time for reflection but not using the time in a productive way and without direction. This can lead to desperate conclusions and learning outcomes may not meet. This example implies that planning a reflective learning process needs to be matched with learning objectives, rather than being a “recipe follower” without any plan. Bond and Walker (1998) specify that “there is inevitably a tension between guidance which leads to the problem of recipe following [identified above] and a lack of structure, which can lead to a loss of focus.” (p.193) Another concern when we prepare a reflective learning model is the belief that reflection can be easily contained. It is a misunderstanding that reflective thinking or reflection can he commanded or controlled by a teacher. Actually, it is not exactly. Bond 51 and Walker (1998) indicate that there are “very nature of reflective activities in such that they may lead to serious questioning and critical thinking, involving the learners in challenging the assumptions of teachers or the learning context in which they are operating.” (p.193) Those reflective activities may lead students to focus on personal distress, oppressive features of the learning environment, resources provided and so on. Therefore, teachers or facilitators of this form of learning need to be aware that the activity can tap into such issues and block reflection. There are also possibilities of mismatching between types of reflective activities and where it is used. Boud and Walker (1998) give examples of the mismatching. The first example is: a teacher asks students to explore their misconceptions in a situation and write what they do not understand so that the teacher can assess whether students gain understanding about a subject matter or not. Students will write what they know, not what they do not know: they may not know whether they have any misconceptions. This is a good example of mismatching between the assignment and the assessment purpose. The second example of mismatching between reflections and assessment is in the use of a writing journal, if a teacher assigns a reading for a reflection paper, and expects that students will read as assessors. When students engage in a subject matter and know that assessment of journal writing will be based on reflection skill on the subject matter, students may fail to engage with their feeling and learning but do focus on the subject matter and reflection skills. Therefore, a reflective writing should be judged in terms of criteria for the recognition of reflective writing, not in terms of standard academic writing (Hatton & Smith, 1994). 52' Bond and Walker (1998) mention that reflections are often involved with emotions and feeling. Reflection is also commonly treated as an “intellectual exercise”. In an educational setting, students may be afraid of expressing their emotions unless students are in conditions of trust and security and know that the expression of emotion is not likely to lead them to negative consequences. Therefore, a learning environment, which supports reflective learning, should allow emotional disclosure to let students reflect in all aspects in their practices. However, inappropriate disclosure may occur, such as confidential information, unethical behaviors, personal sensitive matter, etc. When students feel comfortable and are not aware of the constraint, they may raise issues that embarrass others in the reflective activities and lead to negative consequences later. As indicated by Bryant et al. (1996), experience cannot be separated from knowledge, it needs to be interpreted as a social practice: it is neither coherent, complete nor masterable. To encourage reflective behavior we should treat both knowledge and experience as issues for reflection and also process them as questionable issues, which are interpreted in social practice: not issues that will be damaged by questioning. If we accept experience and knowledge as theories to be believed in without questioning, the cycle of reflection will be limited. This will be a major concern in some cultures, where students will be placed as listeners and teachers will expect that students to believe whatever teachers say. However, we should keep in mind that generating new knowledge with a constructive approach needs to be built on previous knowledge or presuppositions and presuppositions are framed by previous theories. Without reflection, the process of learning and knowledge generating will not be possible. 53' According to the earlier mention of Dewey’s RP definition and attributes, Yost et a1. (2000) interpreted Dewey’s three important attributes of critical reflection and indicated the rules for instructors in RP learning as follows: Open-mindedness. as a desire to listen to more than one side of an issue, to give attention to alternative views, and to recognize that even the firmest beliefs may be questioned; Responsibility. as the desire to actively search for truth and apply information gained to problem situations; and Whole-heartedness. as the attitude whereby individuals can overcome fears and uncertainties to make meaningful change and to critically evaluate themselves, organizations, and society. The attributes of open-mindedness, responsibility, and wholeheartedness may be restricted in some individuals who have little sense of involvement in their own learning (Main, 1985). Thus, the role of the teacher/instructor in developing reflective leadership is to start with students' experiences rather than with the teacher's/instructor’s predefined agenda and then to emphasize the critical analysis and reformulation of that experience. The goal is to develop the level and quality of student observations to the point where students begin to engage in "reflection-in-action" and ultimately in the formation of new concepts (Kaagan, 1998). As important as designing the online learning system and preparing learning objectives and leadership, to encourage reflective learning in an online environment, teachers/instructors should be also aware of reflection scope in with the learning contents. The level of reflection can be both surface and in depth. While it may not be possible to 54 predict all factors in a given practice, to help students reflect about a subject matter, teacher/instructors should be well prepared for the subject in depth and understand the reflection process. Teachers should be aware of what they can and cannot handle. In the case where teachers could not handle the process, developing. networks of practitioners within the practice and information resources via the Internet would be a good idea. In brief, this chapter reviewed current discourses in three main major areas of the literature relating to this study: the reflective practice concept, RP learning processes and its applications to the online learning. Reflective practice is defined as a desired behavior for any practitioners, in which its owners are assumed to be more effective in working and learning than other people who work and learn without it. It is a conscious, systematic, deliberate process of framing and reframing practice, in consequence of our learning and practicing. It involves knowledge, beliefs, values and objectives within physical, political, socio-cultural and historical contexts. To encourage RP behavior, there are many techniques that can be used, both individual and group techniques. However, there are some concerns for implementation of these techniques due to a misunderstanding of the RP concept. Therefore, to design a practical online learning environment all these factors should be kept as major concerns. The synthesis of this literature provides initial information to identify the RP concept, its related variables and their relationships within the concept. A theoretical model is developed based upon this information and ready for a preliminary test in this study (see the model in Figure 3.2). The next chapter presents an overview of the research design and methodology used in testing this model. 55' CHAPTER THREE Methodology This study applies a hybrid of quantitative and qualitative research techniques, including a triangulation of data collecting and analysis methods. The data for this study was gathered using three different means: an online questionnaire, a series of focus group web discussions and a literature review. The data analysis methods are also triangulated using statistical, along with content analysis, and the logical reasoning of grounded theory development (a heuristic approach). The triangulation of data collecting includes the qualitative data from literature review in chapter two, quantitative and descriptive data fi'om an online questionnaire and analytically qualitative data from the DELPHI typed focus group web discussion. The qualitative data from the literature review is used for developing the assumptions, related operational definitions, pre-set RP conceptual fi'amework and pro-assumed relationships among variables within the framework. The data from the online questionnaire is used for validating the assumptions and relationships among the variables that were established from the literature review. The DELPHI-typed focus group web discussion that conducted in three rounds also validates the results from the quantitative data that derived from the online questionnaire. A mixed use of content analysis, quasi-statistical analysis and logical reasoning of behavioral analysis is another triangulation used in this study . (see Chapter 4 for the analyses). Content analysis method is used in the beginning of this 56' study to extract data from the literature review and identification of RP theoretical themes, while quasi-statistics from the first eighteen questionnaire responses indicate preliminary findings. The content analysis of data from literature review is also used to establish the pre-set RP conceptual framework. Then, the statistical analysis of the entire- received quantitative data and logical reasoning of analyses are used to validate the RP conceptual framework. The above approaches are used for several reasons. First, it allows an in-depth and open-ended analysis for model development and a preliminary test of the model. Secondly, the research is strengthened by the use of both rigorous statistical and heuristic evidence. And, third, because this research is limited to a small population frame, and due to the shortage of time for project implementation, it is more logical to employ a mixed method so that the quantitative part responds to the needs of the statistical tests for the initial model development and the qualitative part serves to recheck the model for reliability and face validity. In the quantitative part of this study, factor analytic data-reduction is used to provide insight into the initial formation of the RP learning model as well as to generate and recommend testable hypotheses and do preliminary tests of the RP structural equation model. Qualitative techniques are used in the initial phase of this study to indicate the pre-set variables for the initial model-development and to strengthen the findings from the quantitative data and to fill in any gaps within the model development process for which the quantitative method could not be used. 57 Procedure This research involves six main tasks: (1) outlining a preliminary RP online learning conceptual model from a literature review; (2) developing a questionnaire to measure the variables involved in the model; (3) pro-testing the questionnaire to confirm the validity of the identified variables; (4) conducting three rounds of DELPHI-type web- discussions to strengthen the model; (5) indicating themes that emerge in order to reframe the model; and (6) reframing the conceptual model and issuing recommendations for future study. PARALLEL TASKS WITHIN THE RESEARCH PROCEDURE Quantitative Method Qualitative Method Phase 2: Developing a Phase 1: Outlining a questionnaire to ‘— prelirninary conceptual measure the constructs framework with involved variables and V operational definitions Phase 3: Preliminary test of the . * RP conceptual model * using Statistics """" Phase 4: Conducting web- 6 based focus group discussions Phase 6: Refrarning the conceptual . * model and issuing ‘ . ‘ Phase 5: Indicating themes and recommendations for merging statistical ev1dence to firture studies reframing the conceptual model Figure 3.]. Parallel tasks within the research procedure Phase One: Prior to developing the questionnaire, a content analysis of existing literature was conducted to identify variables and their measurement constructs. These variables were candidate variables and were supposed to be included in the RP online learning 587 model. Operational definitions were also provided using the content analysis from the existing literature (see chapter one for the operational definitions). The operational definitions were used as a systematic reference as well as for keys to identify whether each variable and construct was soundly related to the model. This task was based on logical reasoning using an inductive approach to pre-assign the relationships that could be used for the measurement model of each variable among the alternative variables and their indicators. Phase Two: A questionnaire was developed to measure the variables indicated in phase one. To measure the variables, indicators used for the measurement model of each variable were pro-assigned. For example, in this study, working performance improvement (WP) was measured using four main indicators: (1) working faster, (2) working more cost effectively, (3) working better, and (4) receiving more appreciation from an employer/boss. These indicators are latent variables that were tested for reliability to determine if they should be included in the conceptual model. (Details of questionnaire development are further discussed in the instrument section of this chapter.) After the questions for each variable test were completely developed, the questions were reviewed by a research committee to enhance their content-related validity. These valid questions were then converted into FrontPage format and uploaded to the questionnaire webpage for a systematic check. Ten international students volunteered to complete this test (They are volunteers and not our target population). This practice test was also administered to ensure the readiness of the system, as well as the accessibility and proper installation of the webpage. 59 Phase Three: After the questionnaire webpage was ready, a consent letter was distributed via e-mail to the target population with an electronic link to the questionnaire web page (see Appendix C for the consent form). Participation in this online questionnaire was completely voluntary and the participants’ anonymity protected. Questionnaire responses did not contain any names or identifying characteristics. All participants were notified that completing and/or returning the questionnaire to the researcher via the Internet would indicate their consent to be a subject in this study. Both quantitative and descriptive data derived from the online questionnaire were decoded and processed using triangulation of data analyzing methods. The details of data analyses are firrther discussed in the analysis section of this chapter. Phase Four: A group of alumni within the target population was invited to participate in a series of (DELPHI-type) focus groups in online discussions. Focus group participants were recruited based on their course completion and involvement in the FM field as practitioners. The focus group discussions were conducted in three rounds using the DELPHI approach. The DELPHI technique is a constructive and systematic procedure using informed intuitive judgment. The procedure elicits and refines group judgment as it is based on the notion that a group of experts is better than one expert when exact knowledge is not available. After each round, the information is consolidated and edited. The nature of focus group discussion using DELPHI approach allows participants to respond to others participants both in (online) conversation and after each set of questions. Moreover, the inductive nature of the DELPHI approach allows the researcher 60 to get data in broader detail than the quantitative method. The questions used for these discussions were developed to fill in some unexplained gaps within the model left alter using the earlier online questionnaire results. The additional data derived from the earlier rounds of discussion were also analyzed to generate more questions for later rounds of discussion. Phase Five: The responses from focus group discussions were merged with statistical evidence from the online questionnaire to develop themes and validate the relationships among the variables. The tested model identified in phase three is discussed to validate its theoretical themes using additional data from these focus group discussions. Theoretical themes were anticipated for each alternative model to be compared to the pre-determined model using both statistically significant and logically reasoned evidence. Phase Six: Alternative models from phase five were finalized to determine which model was judged the best for maximum enhancement of reflective practice online behavior. Final recommendations were issued based on the best model to reframe the conceptual framework and future tests of the model. Variables and Measures Based on the literature review in chapter two, there are a number of variables that determine the reflective practice concept. These include social, psychological, sociological, environmental, political and emotional constructs, but this study specifically investigates variables related to how students learn online with the concept of RP. The 61 candidates for variables to be included in the RP online learning conceptual model were identified based on content analysis of the literature review. Reflective practice behavior is mainly related to psychological and sociological constructs, which are latent (unobserved) variables. In order to measure them, they must be defined clearly by operational definitions. Operational definitions are important for both model development and the clarification of this study. The process of model development contains tests of reliability and validity for each of the variables, as well as relationships among the variables. If variables are not clearly identified, it is very difficult to identify their indicators and the model is not reliable or valid because of a lack of precise indicators for its measurement. The literature review indicates two main clusters of independent variables: individual traits and online learning behavior. These independent variables are considered as causes that influence RP behavior and performance improvement. To test the causal effects of these variables as they pertain to reflective practice behavior, presumed relationships among the variables were indicated (see Figure 3.2 for preliminary conceptual framework and the relationships among the variables). Pre-set independent variables within an individual trait cluster include age, gender, previous knowledge and experience. Another independent variable cluster, which was also considered a dependent variable to RP behavior and performance improvement, was online learning behavior. The online learning behavior cluster includes learning styles related to learning technique utilization, interaction with people and information sources and time spent in learning. Since reflective practice behavior is identified in chapter one as a desired behavior, 62 which leads to the improvement of performance, learning performance and working performance improvement are included in the preliminary model as dependent variables. Another dependent variable is thinking behavior. Thinking behavior is included in the model as a cluster that contains three subscales: reflective thinking behavior, critical thinking behavior and level of analytical thinking. The exclusion of thinking behavior from online learning behavior was for the purpose of testing the causal effect of the learning behavior cluster on thinking behavior and clarification: whether learning behavior, which mainly involves sociological and environmental constructs, has an influence over psychological behavior like thinking. In summary, there are a total of ten main variables involved in this study: 1. Thinking behavior: thinking behavior was set as a dependent variable in the preliminary model. It is measured using three subscales: reflective thinking behavior, critical thinking behavior and level of analytical thinking. Reflective thinking behavior (R). The measurement scale for this construct was based on responses to a situation that indicates different levels of reflective thinking behavior from the highest to lowest level of “reflective.” Critical thinking behavior (C). The measurement scale for this variable is based on responses to a situation that indicates different levels of critical thinking behavior from the highest, “critical,” to the lowest level, in which students tend to be “irrational.” The critical thinking concept is identified and differentiate from reflective thinking as it more or less relates to personal values of the thinking context and its obj ective-oriented nature, while reflective thinking is a value free related behavior that implies only the process not the context within the process. 63' Level of analytical thinking (L). Level of analytical thinking is another construct that refers to a dimension of thinking. Both critical and reflective thinking relate to this construct as it indicates level of thinking ability. The measurement scale for this construct is designed to indicate levels of analytical thinking from the “in-depth” to the “surface” level. 2. Interaction (I): The interaction with people and information sources is set to be a candidacy variable for both independent and dependent variable status. It is included in the learning behavior cluster, which is measured based on degrees of group technique utilization and responses to specific situations. The measurement scale for this construct is designed to differentiate levels of openness or the connection between students to the people involved in learning and the information sources around them. The scale varies from “self-efficiency” to “broadly-opened to all available infOrmation”. 3. Time spent in learning ( T I): Time spent in learning is treated as an independent variable. It is measured based on the time used for learning (hours per week), by tracking the number of connections to the intemet for learning (times per week) and tabulating the numbers of times e-mail is checked per day. 4. Previous knowledge (K): Previous knowledge is identified using “levels of education” and “fields of study” as its indicators. A Likert scale is used to transfer descriptive data into quantitative data. 64 xeoaoEsE 3:38:00 meEBE .N.m. 33M: \111 mcotmszm on momconmom :o_~m:_m>m— sauna—Agm bow 3v wan—EFF 323.25. he .33 row was eozooom 320 "SE Eggomm—E CV “55.; .825 IV "Ed 55.50.25 Huge—omega EC “55.? oesocom @5523 @7520? newsman CEMZEH . EC .59 :mE .0 9.3830 90 SEC. £83 Lon 35E. £83 con maze: ”9.5 WEE—34 E Eon 25,—. 225m .5 2m: mecca—:5 "ADV Lou—30 8 moweommox new new: 027E203. 9380 Lo 03on . A: 83:90. :ozaELSE 4% Enema— :zt 5:228:— 1.. 22m new .33 ..,.. Av: once—>55— 2.2.5.5 mung—Each 95:8: 9 no“ 8: Co 35:35 , "5.5 Est-5:5 2.3—Econ. ”5:394 . . _ mozzfifieoamom ES 9.3% 1— mzmmhhhm UZ—ZMFVHJ mZ—AZO "av 3:020an daze—<5 gaze mow—.053 "3:05am: .503; REM—pace RAE—Egan.— .. 65 5. Experience (E): Experience is indicated using both a quantitative and qualitative approach. Number of “years” of experience is used as its measurement, while “positions” and “responsibilities” are used as descriptive indicators. The descriptive indicators are transferred into a score and analyzed using both quantitative and qualitative analyses. 6. Age (A): Age is directly measured in “years” using a five point Likert scale. 7. Gender (G): Gender was indicated as an independent variable within the individual trait cluster to indicate whether being “male” or “female” would have any effects on reflective practice learning behavior and performance improvement. 8. Learning techniques utilization (TU): Frequency and utilization of online learning techniques is used as the measurement for this variable. The leaming technique utilization variable was set to be in the online learning behavior cluster. It consists of 13 learning techniques, which have been used in the FM courses. This variable is analyzed to find out whether online students have any learning patterns in common and which learning patterns would affect their performance. 9. Learning performance (LP): Grades received from each class and self-evaluations are used as indicators to measure this construct. 10. Working performance improvement (WP): Self-evaluation is used to measure this construct. The details of the measurement method and scale development are discussed in 66 the instrument section of this chapter. Participants This study was conducted using alumni who graduated from the FM online master’s level program at MSU and students and/or FM practitioners who finished at least one of the four main FM classes from the summer of 1998 to spring of 2003. The total number of population for this study is approximately 120. Half of them graduated by taking all classes in the program and the other half of the population are currently taking one or more of those classes. Due to the small population frame, the research was carried out using a convenience sample of random voluntary participants. The online F M students are apparently the most reliable population to investigate the concept of reflective practice, due to the fact that most of them are F M practitioners working in different business industries, who are willing to update their knowledge via an online course. Even though a random sample is more desirable and would possibly enable the researcher to generalize the findings of the research to all FM online learners, participation in this study was dependent on the participants’ availability and willingness to participate and the generalization of research findings is within the case study itself. It is not intended that these findings be generalized for other online programs. Thirty one people participated in the online questionnaire for the preliminary test of the model and nine of them participated in focus group discussions as experts who have direct experience in the courses. The recruitment of focus group discussion participants was based on the significant success of their learning as well as recognition of their 67’ experience within the F M field. (see table 3.1 for the numbers of participants in each activity) Table 3.1 Numbers of Participants in the Study . . . Numbers of Total Activities Instrument Participants Population Research Note Setting Online First 18 responses from questions questionnaire the online questionnaire for focus 18 ~120 were used for setting the group questions and quasi- discussion statistical analysis Preliminary Online ~30% responded rate test of the questionnaire (The total number of RP 31 ~120 available e-mail conceptual addresses is 120, but model about 100 are currently active) Focus Questions Group were Discussion developed in the 9 ~120 12 people were invited consequence of the first 18 responses. It is important to recognize the group of subjects, as they are supposed to be more or less professionals in the field. Results likely would be different if other groups of online students were used. Therefore, it should be noted that this study was a preliminary study for model development and was based on a reflective practice concept. It was performed on only a few subjects in the field of facility management and, presumably, is only good for generalization within the case study itself and is not intended for use in cross disciplinary fields or prediction of other professional working performances. 68 Instruments An online-self administrative questionnaire is the main instrument used in this study. The questionnaire consists of five sections, which were designed to measure variables and constructs indicated from the literature review: reflective thinking behavior (R), critical thinking behavior (C), level of analytical thinking (L), interaction with people and information sources (1), learning techniques utilization (TU), previous knowledge (K), experience (E), learning performance (LP), working performance improvement (WP), and other descriptive data (see Appendix B for the instrument). The five sections of the questionnaire contain a total of 85 items that range from multiple-choice rating scales, true or false, to open-ended questions. With the exception of descriptive open-ended and true or false questions, most of the questions are based on a five point Likert scale. The number of items asked for each variable ranges from four to sixteen. Scales for construct measurement were developed depending upon a prior set standard for each variable, such as, working performance improvement is based on self evaluation. Thus, to measure working performance improvement, both working performance before and after finishing the classes are measured to indicate the improvement. For variables such as working performance improvement, a five point Likert scale is used for participants’ self evaluation and then interpreted from the change of responses before and after taking classes. Another way to measure working performance improvement is to ask participants to indicate their self evaluation via responses to questions that indicate improvement of performance, such as, whether they work better after taking the classes (see question 4.3 in Appendix B). It is important to note that different variables may need different scales of 69 measurement. In contrast to a self-evaluation based question, questions one through eight in section one of the instrument are based on responses to specific situations. Each multiple choice represents a level of ability or behavior that needs to be distinguished from other choices to indicate differences of scale. Thus, key indicators for each scale must be clarified (see examples of scale development in Appendix F). Data gathered from this instrument are analyzed and pro-tested to identify what relationships exist among the variables within the preliminary conceptual model. Some constructs that are not clearly significant or need further analysis were then considered for additional discussion in the focus group phase. Three sets of questions were generated based on the results of the quantitative analysis of the questionnaire and were used as another instrument in this study. The validities of the instruments were reviewed and tested both by using statistical procedures and a face validity audit. Statistical procedures were used for testing construct validity, as well as validity of the criteria, while content-related validity was recognized by a face validity audit. For each item asked, the variation of responses shows its content- related validity. Therefore, performing a face validity audit by looking at a summary of the data from response sheets roughly tells the ratio of data variation. In other words, responses for each variable should have a variation close to a normal distribution in order to have acceptable face validity. For example, any true or false question that has 99% of the responses as true, may be considered invalid and should be discarded from the measurement model. Also, a five point Likert scale question where all responses are in the high range of the scale would not represent a normal distribution of data and, therefore, would also be considered 70 invalid. The low variation of data within each measurement question indicates a lack of content-related validity, which may be caused by the language used or ambiguity of the question. (The issues of reliability and validity of the tests are further discussed in the analysis section.) In addition to this validity audit, it is impOrtant to note that this research is meant to be a preliminary study for the RP learning model development and uses only a small sample size. Therefore, reliability of the overall preliminary model is not expected to be high, but rather the test and identification of variables to be included in the model and the reliability of each variable included. Data Analyses The data analysis method used in this study could be identified as multiple techniques as it consists of both qualitative and quantitative methods, which are paralleled from the beginning to the end (see Figure 3.1 for the research procedure). The study starts by using content analyses to indicate candidacy variables and to determine relationships among the variables. The pre-set variables and the relationships among them were set as a model and analyzed using a quantitative approach to test the model. This test was a preliminary study conducted on a small sample size with the intention of being a pilot study for a full-scale test in a future study. The final goal of the analysis was not to have a perfect standardized model, but rather to create a forum to use for the brainstorming of ideas, as well as to use multiple approaches to determine a best alternative for the online professional learning model within the reflective practice concept. The data analysis procedure used in this study is not a linear process but rather consists of parallel tasks, which shift between qualitative and quantitative approaches. 71 The following section indicates details of quantitative analysis methods and procedures, which include factor analyses, structural equation modeling, partial correlation analyses, multiple regression and best sub-set analyses. Quantitative Analysis Factor Analysis. Factor analysis is used for two main purposes. The first one is called “exploratory factor analysis.” Exploratory factor analysis is a test for unidimensionality and multidimensionality of the pre—set constructs which are used as indicators of each variable. The process consists of tests of reliability of each construct one by one using Cronbach’s Alpha Statistics, pattern matrix analysis and rotation analysis. In other word, it mainly is used for testing of instrumental items (question of measurement of variables) for reliability of the instrument. Generally, exploratory factor analysis is conducted on a latent (unobservable) variable to explore what indicators can be used to measure the variable and whether the constructs are reliable indicators. Each indicant item is examined for its correlation to other items in its cluster, along with a reliability test (Cronbach’s Alpha), when included in a model as an indicator of that variable. If some indicant constructs within a measurement model have a low degree of correlation to other constructs within the model, the total reliability of the measurement model is lower. Therefore, these items are discarded from the model. In this study, five variables are considered as latent variables: learning technique utilization (TU), experience (E), Interaction with people and information sources (I), thinking ability (T) and working performance improvement (WP). Pre-set constructs 72 identified earlier from a literature review are used as indicators for these variables and are tested using exploratory factor analysis. Another use of factor analysis is called “confirmatory factor analysis”. Confinnatory factor analysis is used to rigorously test the validity of each construct measured. Construct validities are confirmed using this type of analysis. Items with low and insignificant factor loading are discarded from the measurement model for each variable. The rule of thumb for defining the significance of factor loading is that it is significant when the loading has “t-value” >1 .97 or contains a loading two times greater than its own standard error (SE). Structural Equation Modeling. Structural equation modeling (SEM) is a way to test a model of causal relationships among theoretical variables using covariance analysis. SEM determines whether a theoretical model successfully accounts for the actual relationships observed in sample data. The output of the analysis provides indices that demonstrate whether the model at a whole, fits the data, as well as significance tests for specific causal paths. As with most statistical procedures, SEM requires several necessary conditions be met to ensure the validity of the analysis. All variables should be either interval or ratio level data. Statistical tests conducted with the SEM-LISREL program assume a multivariate normal distribution. Relationships between variables should be linear and additive with the variables free of multi-collinearity. To test for goodness of fit, the initial causal model must be pre-identified. A pre-identified model is one that includes more equations than unknowns. Finally, there should be ratio of at least five subjects for each parameter (total parameters equal path coefficients plus variances and covariances) to be estimated 73 (Bentler & Bonnett, 1980). Some researchers argue that the SEM analysis is best conducted on samples greater than 200 (March et al., 1988), and some indicate its reliability is greater when the responded numbers for analysis are at 5 to 20 times bigger than the numbers of variables involved (Hair, et.al., 1998). I In this study, a LISREL program was used to confirm the structural equation model. It empirically tested the constructs and variables represented in the overall model of reflective practice online learning, as well as the path model that specifies causal relationships among latent (unobserved) variables. SPSS and Minitab programs were also used accordingly for partial correlation analyses, multiple regression analyses and best sub-set selection of the variables involved in the model. Partial correlation analyses were conducted to identify the subscale of learning technique utilization (TU): whether each technique could be used to support RP learning behavior, while multiple regression and best subset regression analysis were conducted to identify the regression equations of alternative models. To empirically test the preliminary model, four main criteria were identified to enhance criterion validity of the model. Items were candidates for deletion from the model if they: ( 1) displayed an insignificant regression weight (t—value >1.97.); (2) showed several large residuals when involved with other variables (recommended >2.58); (3) shared a large unexplainable variance due to error with other indicators or lurking variables; and (4) shared a common variance (multicollinearity) with multiple indicators of other construct(s). However, these criteria are not a standard for finalizing the model. It is important to consider both statistical evidence and theoretical issues from content analysis before any items are deleted, as well as to use logical reasoning for the whole 74 model. Qualitative Analysis The qualitative analyses in this study were mainly involved in phases one and six. The methods used were combinations of content analysis, logical reasoning and inductive method, which is referred to as a “grounded theory approach.” Grounded theory refers to theory that is developed inductively from a corpus of data (Borgatti, 2004). It takes a case or cases rather than statistical perspective. This means in part that the researcher takes different cases to be wholes, in which the variables interact as a unit to produce certain outcomes. A case-oriented perspective tends to assume that variables interact in complex ways, and is suspicious of possible models or theories. A content analysis of the existing literature as it relates to the reflective practice concept and online learning was conducted in the beginning of this research. The purpose of the analysis is to indicate candidacy variables and their possible relationships. A pre- determined model was set based upon the relationships of pre-set variables to be able to conduct and operate the testable reflective practice learning theory. Content analysis was also an important part of data reduction of quantitative analysis. As mentioned earlier, model development using structural equation modeling must be considered when deleting insignificant variables from the model one by one. Without logical reasoning and content analysis, a tested model may lack content-related validity. Therefore, we must analyze the whole model using both statistical evidence and logical reasoning regarding its validity. Content analysis techniques used in this study were various: from a tabulation of 75 terms and cluster categorizing to a comparison of alternatives. The logical reasoning part also includes behavioral analysis of idiosyncratic cases and themes and development of variable relationships. All emerged themes and possible additional variables derived from phase six of this study were merged with the statistically significant evidence from phase three in order to finalize a best conceptual model and issue recommendations for testing the model in a future study. In surrunary, this chapter presents the research methodology of the study. The methodology is identified as a mixed-method with parallel tasks that are developed in six non-linear orders. Participation in this study was voluntary. The subjects were online students and alumni of the MSU FM program. A population frame of 120 was used for this pilot study to establish a testable model and do preliminary testing of the model. After a pre-determined model of variables and their relationships were developed fi'om content analysis of existing literature, an online questionnaire was developed to measure the variables and their relationships. This questionnaire was used as an instrument to pre- test the model. By using both quantitative and qualitative techniques, alternative models were identified. The quantitative techniques used in this study consist of factor analysis, structural equation modeling, as well as partial correlation analyses and multiple- regression for best subset model alternatives. The qualitative part includes multiple techniques of content analysis and logical reasoning. Qualitative data were derived from three different sources: a literature review, an online questionnaire and a focus group Delphi-type discussion, as a triangulation of data. The quantitative analysis results are strengthened by additional evidence from the focus group discussions to identify the best alternatives for the RP online learning model and are presented in the next chapter. 76 CHAPTER FOUR Qualitative Analyses and Findings In this study, qualitative techniques are used mainly in phases one (content analysis of existing literature) and five (theme development using grounded theory). Content analysis of the existing literature, as it relates to the RP concept and definitions, is presented in chapter two. Preliminary findings from the content analysis are used for setting research assumptions, defining operational definitions of terms and variables and identifying relationships among variables. The content analysis is necessary for this research in order to be able to pre set an RP conceptual framework: variables within the RP concept and relationships among the pre-set variables. In other words, the results of content analysis informed the quantitative part of this study by establishing a pre-set RP framework: making it possible to be tested in the quantitative part of this study and validate the RP theoretical framework. While content analysis of literature is presented in chapter two and the operational definitions are in chapter one, this chapter presents the analyses and results of phase five of this research. The analyses include the identification of theoretical themes, qua-si statistical themes and behavioral analysis. The data for these analyses comes from two sources: an online questionnaire and focus group discussions. The questions for (DELPHI-typed) focus group discussions are set upon the responses of the online questionnaire. 77' Preliminary Analysis After the online questionnaire was made available to participants for two weeks, eighteen responses were received. These eighteen responses are analyzed using quasi- statistical estimation to develop preliminary themes. Thesepreliminary themes are needed for establishing relationships among variables, which were identified from the literature review in chapter two. The relationships are analyzed and integrated into findings from the content analysis and are used for presetting of the RP conceptual framework. This preliminary analysis is also needed for questions used in the DELPHI focus group discussion, which is another source for in-depth qualitative data. Preliminary Findings The findings from the first eighteen responses indicate five main themes of relationships among variables as follow: (1) Age (A) versus Learning Performance (LP), Thinking Ability (T) and Working Performance Improvement (WP), (2) RP Technique Utilization (TU) versus Learning Performance (LP), Working Performance Improvement (WP) and Thinking Ability (T), (3) Previous Knowledge or Education (K) and Experience (E) versus Thinking Ability (T), (4) Interconnection with People and Information Sources (1) versus Reflective Thinking (R), Critical Thinking (C) and Learning Performance (LP), and (5) Gender (G) versus Thinking Ability (T) It is important to note that these emerged themes are derived from a preliminary analysis of a small sample size and is only a preliminary setting of relationships among the 1 variables (see Figure 3.2.); is not meant to either confirm or test the relationships. 78’ However, the use of these emerged themes at this state can be validated by using triangulation of analysis techniques. Table 4.1 Preliminary Findings and Emerged Themes 1 # Findings Emerged Themes 1.1 There are five students who Theme 1: Age (A) versus Learning have perfect learning Performance (LP), Thinking Ability (T) and performance scores. Four of Working Performance Improvement (WP). them are 240 yrs and one of them is 36-40 (respondents # 1, Students 231 yrs tend to have higher 10, ll, 16, 18). learning performances (LP) than younger students. The findings show a very strong 1'2 Two ofthe students who have relationship between age and learning the highest thinking scores are performances. However, the relationship 240 yrs (respondents # 16’ between age and thinking abilities seems to be 12)- controversial. 1.3 Two of the students who are . . Z40 yrs have lower thinking In thrs case, even though the top two students abilities than students <40 for thinking abrlrty are >40 yrs, there are some (respondents # 9, 6). students >40 that get dramatically low thinking scores (T). Also, there are some students In the 1-4 Students aged 30'35 yrs have middle range (30-35) that have higher thinking higher thinking scores than scores than the older students. Therefore, it younger students and some appears that older students have higher older students (respondents # learning performances (LP) and younger 4, 8, 17)- students have lower thinking abilities (T) than students in the middle range (30—35). 1.5 Students <25 yrs and between 25-30 have lower thinking scores than most of the students >40. (respondents # l, 2, 5, 7,13,14,15). The findings have not shown a strong relationship between age and working performance. 79 Table 4.2 Preliminary Findings and Emerged Themes 2 # Findings Emerged Themes 2.1 Among 18 respondents, there is Theme 2: RP Technique Utilization (TU) only one respondent >40 yrs that versus Learning Performance (LP), Working has a devastatingly low thinking Performance (WP) and Thinking Ability (T). score even though he has strong educational background and A higher level of RP technique utilization experience in the F M field with a may lead to higher thinking abilities while near perfect score in learning there is not enough evidence to indicate that (respondent # 9). using more RP techniques will lead to better working and learning performances. 2.2 The top student, who has the highest thinking ability, a perfect The comparison between students who have score in learning performance the highest and lowest thinking scores shows and exhibited a distinguished that the students who get the highest thinking working performance, has the score also have the highest level of RP highest level OfRP technique technique utilization while the students with Utilization (respondent # 16)- the lowest scores have the lowest level of RP technique utilization. 2.3 The student who has the lowest rate 0f RP technique Utilization Even though it is clear that students who use 801 the lowest thinking score RP learning techniques more often have (respondents # 2)- higher thinking scores, using more RP learning techniques does not ensure that they 2.4 Some “A” students did not will have higher learning and working utilize RP learning techniques performances that much (respondents # 2, 8, 18). In addition, the preliminary data shows that there is a high possibility of emerging 2. 5 Most students, who have high relationships among the level of RP thinking abilities, have high rates techniques, learning performance and of RP techniques utilization working performance. (respondents # 3, 4, 12, 16, except # 8). 2.6 Among 18 respondents, there is only one “A” student who got a very high thinking score, but has not used RP learning techniques for her learning that often (respondent # 8). 80 Table 4.3 Preliminary Findings and Emerged Themes 3 # Findings Emerged Themes 3.1 The top five students in Theme 3: Previous Knowledge/Education (K) thinking ability do not have and Experience (E) versus Thinking Ability strong educational backgrounds (T) in the FM field (respondents # 3, 4, 8, 12, 16) and some did not Previous knowledge and experiences may have experience in the field at , not be the main factors that affect thinking all (respondents # 4, 8). ability. 3'2 ittvhiirhgvteun‘idlrifeijiesrtileiiireussin In this case, students who achieve a high the field having higher thinking thinking score are the people who do not have ability (see respondents # 6 9 the strongest educational background and 10) ° ’ ’ working experiences. The student who has the strongest educational background with some experiences in the field even has a lower 3'3 3::n5tzgtegéxgggfilthe thinking score. Also, the findings show that background in PM has a lower the highest three thinking scores belong to the thinking score than students students who do not have both educational . background and experiences in the field. who have not had educational . V background rn PM at all. These unexpected findings may be caused by (respondent # 17) , many other factors (lurking variables). 3'4 The student who has quite Therefore, this study should follow this strong educational background direction by arranging discussions about and experiences may have a possible factors that may affect thinking dramatically low thinking score. ability. (respondent # 9) 3.5 Students may have high thinking ability without experiences in the field at all. (respondents # 4, 8, 18) 3.6 Most students who have both educational background and experience in the field get lower thinking scores than students who do not have both. 81' Table 4.4 Preliminary Findings and Emerged Themes 4 # Findings Emerged Themes 4.1 Students who have a high Theme 4: Interconnection with People and reflective thinking score (R) Information Sources (1) versus Reflective also have a high level of Thinking (R), Critical Thinking (C) and connection with people and Learning Performance (LP). information sources. (respondents # 3, 4, 12, 16) Students who have more information and 4.2 Students who have a high connect to other people may not have higher critical thinking score (C) also reflective thinking abilities, but students who have a high level of have higher reflective thinking abilities connection with people and always have an open connection to others information sources. and to information sources. (respondents # 8, 16, 18) 4.3 Having a high level of connec- The preliminary data also show that students tion with people and with the highest critical thinking scores are not information sources does not the people who have the highest connection to ensure high critical thinking information sources, while the highest abilities (respondent # 3). reflective thinking score belongs to the person who has the highest level of connection to 4-4 Having a high level 0f connec- people and information sources. Also, tion With people and students who have more of a connection to information sources does not sources of information and others tend to ensure high reflective thinking have higher learning performances (LP). abilities. (respondents # 5, l7, 18) 4.5 Students who have more connections with sources of information and people get better learning results. (respondents # ll, l6, 17, 18) 82 Table 4.5 Preliminary Findings and Emerged Themes#5 # Findings Emerged Themes 5.1 For the top four students in Theme 5: Gender (G) versus Thinking Ability thinking ability, two are males (T). and two are females. . (respondents # 4, 8, 12, 16, According to the findings and the proportion based on 18 responses: 11 male, of male/ female respondents, female students 6 female, 1 unknown gender), tend to have a higher thinking ability than male students. However, the number of 5.2 The two lowest scores for respondents is not large enough to confirm the thinking ability belong to male students (respondent # 2, 9, based on 18 responses: 11 male, 6 female, 1 unknown gender), while the highest score belongs to a female student. (respondent # 16) statement using statistical significances. To validate these preliminarily emerged themes, triangulation of this quasi- statistical analysis, behavioral analysis and logical reasoning are integrated. These emerged themes are compared with theoretical themes (logical reasoning) and behavioral analysis. The comparison of themes and behavioral analyses are shown in Tables 4.6- 4.14. Similarities and idiosyncrasies of the cases are discussed in this analysis, as well as the possibility of other related factors that may cause a change in the themes. Analytical notes are issued at this state as a precaution for the use of these themes when they are combined with quantitative results at the end of this study. Emerging Themes and Behavioral Analysis follow: The detailed analyses of emerged themes and behavioral aspects are presented as 83 Table 4.6 Preliminary Analyses: Previous Knowledge and Experience versus Thinking Ability Tlfigggal Quasi-statistical Theme 81:12:?! Analytical Note Students Previous knowledge and More than half of These unexpected who have experience may not be the the students with findings may be prior main factors that affect a lower level of caused by many knowledge thinking ability. experience and other factors and more knowledge, but (lurking variables). experience In this study, students who who utilize more in the field achieve high thinking scores RP learning Recommendation: should have are the people who do not techniques, A follow up study better thinking scores. have the strongest education- al backgrounds or working experiences. The student who has the strongest educational background, with some experience in the field, has an even lower thinking score. Also, these findings show that the three highest thinking scores belong to students who do not have both an educational background and experience in the field. achieve better thinking scores than students with a high level of experience, but who did not utilize the RP techniques that much. is recommended to learn about possible factors that may affect thinking ability. 84 Table 4.7 Preliminary Analyses: Experience versus Learning Performance Theoretical Quasi-statistical Behavioral Analysis Analytical Note Themes Theme Students who Not all of the top Behavioral analysis of the best Is it possible have more five students in cases versus the worst case that we do not experience in learning (#16, 12, 8, versus #9) shows learn from the field performance have that while #9 and #16 have “experience”, should extensive almost the same level of RP but rather that achieve better experience in the technique utilization, student we learn from learning field. One (#18) #16 has a much lower range of “reflecting performances. has no experience experience in the field (#9 and experience”? and two (# 12, 9) have a long range of experience but still gained lower learning scores. #12 have more than 15 years, while #16 has 4-10 years and #8 has no experience at all) and a lower level of prior knowledge (#9 has a professional degree related to PM, while #16 has a BA in business, #12 and #8 have no education related to PM at all), student #9 achieved a lower learning performance score and also has the lowest thinking score (lowest reflec- tive thinking score). With no prior knowledge in the field, and lower levels of RP technique utilization than #16, but a lot more experience, #12 achieved the lowest learning performance score among all the respondents. Interestingly, case #8, who has a low rate of RP technique utilization, no experience at all and no solid education or prior knowledge in the field, but who has a high level of critical and reflective thinking scores, got a fairly high learning score. 85' Should we conclude that students will not achieve a better learning performance score if we force them to share their knowledge and experience with their classmates, without paying attention to “reflecting” that knowledge and experience or their “reflective ability”? Table 4.8 Preliminary Analyses: Deductive Approach versus Inductive Approach Theoretical Themes Quasi-statistical Theme Behavioral Analysis Analytical Note The nature of Students who Using both construc- Both “learning by “action pay more tive and objective objectives” and a learning” is a attention to learning approaches “constructive” constructive learning obj ec- will lead to a better approach are approach tives tend to gain learning performance “deductive” by nature, wherein a lower learning score than will using a while action learning knowledge is performance than constructive or takes an “inductive” built upon prior those who pay learning by an objec- approach. knowledge less attention to tive approach alone. and/or experi- learning obj ec- The concept of action ences. tives, but utilize learning is learning by other RP learning “doing” specific activities, such as “tasks,” by finding a reading, team solution for the tasks discussion, and (practice based getting more learning). but for the information via constructive approach the intemet. and learning by objec- tives, learners must acquire an amount of knowledge and/or skills to achieve the objectives and/or to apply them to the tasks using “generalizations” (theory-based learning). Even though consistency in the themes emerged, there is another precaution that should be brought into this analysis: inconsistent themes should not be eliminated, but should be compared with quantitative data to establish the relationships among the variables and for consideration for hypothesis testing in future studies. 86 Table 4.9 Preliminary Analyses: Frequency of RP Techniques Utilization versus Thinking Theoretical Quasi-statistical Behavioral Themes Theme Analysis A nalytrcal Note Regardless of The comparison If students who Even though the any other between students who utilize more RP quasi-stat indicates factors, if we have the highest and learning techniques the possibility that assume that: lowest thinking achieve higher “reflective thinking” 0 “reflective thinking” is a natural ability that everyone has (possibly at different ‘ levels), students who use more RP learning techniques should have higher thinking scores. 0 If “reflective thinking” is not a natural ability, can it be facilitated or motivated? o Reflective thinking is not a natural ability, and can be facilitated or motivated using more RP learning tech- niques that should lead to higher thinking scores. scores shows that students who get the highest thinking scores also have the highest levels of RP technique utilization while the students with the lowest scores have the lowest level of RP technique utilization. Higher levels of RP technique utilization may lead to higher thinking abilities while there is not enough evidence to indicate that using more RP techniques will lead to better working and learning performances. Even though it is clear that students who use RP learning techniques more often have higher thinking scores, using more RP learning techniques does not ensure that they will have higher learning and working performances. thinking scores, it can be assumed that the students’ behaviors were affected by using the RP learning techniques as facili- tation tools, and “reflective think- ing” can be motiva- ted (regardless of whether it is a natu- ral ability or not, it can be stimulated). The quasi-stat indi- cates a possibility that “reflective thinking” can be facilitated according to the relationship between RP tech- nique utilization and thinking scores. Recommendation: A statistical test, using an experi- mental setting with a control group, will validate the finding. (This is suggested as a future study.) 87 can be stimulated, the level of intervention (stimulation using RP learning techniques) is still a curious one. At this stage, it is indicated that the person who has the highest level of RP technique utilization achieves the highest thinking score. How about the person who gets to a “burn out” level? Would he/she benefit from interven- tion? “How much” is “too much” and even if reflective thinking happens only when students have been facilitated, would it happen when the students put it into practice in their real working situations? Is it a temporary behavior or habit? Table 4.10 Preliminary Analyses: Age versus Learning Performance, Working Performance Improvement, and Thinking Ability 1210mm“! Quasi-statistical Behavioral Analysis Analytical Note emes Theme - Older students Students 231 yrs Is it possible that the Possible factors with greater tend to have higher “weak case” (#9) was (lurking variables): levels of learning per-for. not forthcoming experience mance than about his learning 0 The time of #9’s demonstrate younger students. behavior? Did he questionnaire better leaming The findings show a utilize the RP response is 7:19 and working very strong techniques as much am, but he prefers performances relationship between as he stated? learning and class and thinking age and learning activities at night. abilities based performance, Are there any other on the However, it seems to factors that led him 0 His education constructive be controversial to a low thinking format is informal learning whether or not there score? (online/evening approach. is a relationship school and short between age and thinking abilities. Case #9: With the lowest thinking score, but the highest course training) 0 Does “Thinking in Even though the top age, experience and a action” (react two students in high RP technique immediately, while thinking ability are utilization level, he thinking) is a factor >40 yrs, there are reported a 10% that led him (#9) to a some students >40 who get dramatically lower thinking scores. Also, there are some students in the middle range 30- 35 yrs who have higher thinking scores than the older students. improvement in work performance but declined in the understanding of his work process after taking the class. ) low thinking score? 0 Does “attitude” related to the low score? 88 Table 4.11 Preliminary Analyses: Interaction versus Learning Performance Theoretical Themes Quasi-statistical Theme Behavioral Analysis Analytical Note Students who have more information, and are able to transfer the information to knowledge and apply the know- ledge to their learning situations will achieve a better learning performance. 0 Students who have a higher level of con- nection to others and information sources achieve better learning performance. 0 Students who have more information and connect to other people may not have higher reflective think- ing abilities, but students who have higher reflective thinking abilities always have an open connection to others and to information sources. 0 Students with the highest critical thinking scores are not the people who have the highest connection to information sources, while the highest reflective thinking scores belong to those who have the highest levels of con- nection to people and information sources. Also, students who have a greater connection to sources of information tend to have higher learning performance. The descriptive data shows that students who achieve better learning perfor- mance tend to use both group learning and independent learning techniques, while most students who achieve lower learning perfor- mances do not connect to others that much and rarely perform self evaluations. If a student achieves more information from his/her open- ness to the sources of information, would connecting to others help the student in “knowing how” to transfer and apply the knowledge to specific work situations by reflecting the knowledge with his/her learning group? Should we intervene in this process? Is it possible that encouraging students to learn with others and connect to informa- tion sources will facilitate “reflective thinking”, while “critical thinking” cannot be facilitated this way? Is it appropriate to encourage students to search for more information on their own rather than giving them all the information in the class materials and forcing them to contact others to accomplish each assignment? Could a “reflective writing” assignment be assigned to classmates? If connecting to others and informa- tion sources leads to a higher perfor- mance, should it be concluded that group learning is a better way for reflective learning to occur rather than learning in isolation. 89 Table 4.12 Preliminary Analyses: Gender versus Thinking Ability Theoretical Themes Within the same According to the Quasi-statistical Theme Behavioral Analysis Is it possible that other Analytical Note Should we consider environment, findings and the factors, such as “having a female team- males and ratio of male to culture, brain functions mate” as mandatory females should female or physical conditions, for any learning have the same respondents, affect gender-based group? level of female students behavior and link to thinking ability. tend to have a gender-based thinking higher thinking ability? ability than do male students. However, the number of respondents is not large enough to validate this finding. Table 4.13 Preliminary Analyses: Improvement after Taking Class(es) Theoretical Quasi-statistical Behavioral Analysis Analytical Note Themes Theme After taking All participants Is it possible that the Should we allow Class(es), report that they learning experience students to use students should achieved an he (#9) gained is not problems in their real achieve better improvement in their applicable to his work situation for working performance. working perfor- mances in some ways, while only one participant (#9) reported a decline in his understanding of work process. work situation and working process due to the nature of his organization? their learning projects? 90 Table 4.14 Preliminary Analyses: Time and Consequences of RP Theoretical Quasr- . . . Themes statistical Behavioral Analysts Analytical Note Theme 0 Reflective More than half If reflective thinking We should allow a thinking may occur of the partici- can occur both sufficient period of both “in action” pants reported immediately (at the time for students to (while performing reflective think- time of the action) complete their the action) and “on ing both “in and after the action, assignments to allow action” (after the action). 0 Reflective thinking occurs only when people gain information from a specific action and then stop and think before experiencing a reaction. action” and “on action” and students who reported the behavior in both ways tend to have a higher thinking score. is it possible that delaying making a decision and reflecting more on the action will be a better way, because more information can be obtained in order to make a better decision? them to consult with their teammates/instructor and to search for more information on their own. In this way, students will have an opportunity to reflect more on their learning. The triangulation analysis shows the following consistency themes: 1. Previous knowledge and experience may not be the main factors that affect thinking ability. 2. Students who pay more attention to learning objectives tend to achieve lower learning performances than those who pay less attention to learning objectives but utilize other RP learning activities, such as reading, team discussion and getting more information via the intemet. 91 3. Students who use RP learning techniques more often have higher thinking scores, but this does not ensure that they will have a higher rate of improvement in learning and working performances. 4. Students who have a higher level of connection with others and information sources achieve better learning performance. 5. Students 31 years old and older tend to have higher learning performances than younger students. 6. Students who have more information and connections with other people may not have higher reflective thinking abilities, but students who have higher reflective thinking abilities always have more open connections with others and information sources. 7. Even though the highest reflective thinking score belongs to the person who has the highest level of connection to maple and information sources, students with the highest critical thinking scores are not the ones who have the highest connection to information sources. 8. Students who have more connections to sources of information tend to have higher learning performance. 9. According to the findings and ratio of male to female respondents, female students tend to have higher levels of thinking ability than male students. However, the number of respondents is not large enough to validate this finding. 10. Students report working performance improvement in various ways. As indicated in the analytical notes, there are some luring variables that may be involved in the RP learning process. Therefore, in order to use these analyses in a focus 92 group question setting, these themes should be discussed in depth with open-ended questions (see Appendix C for the questions). Focus Group Discussion Qualitative analysis continues in phase five of this study using data from DELPHI focus group discussion. The DELPHI approach is a group technique, which involves bringing a small group of experts (generally 6-15) together and facilitating the discussion by encouraging all participants to put their thoughts on the table. With opportunities to express their opinions and respond to other participants’answers, the discussion can lead to a final consensus of themes. This method of theme development is known as a grounded theory approach. At the end of the focus group discussions, there are 14 coding categories of findings emerged within three main themes. The findings are discussed as follow: Coding Category 1: Previous Knowledge and Experience. ACcording to the discussion, students rely on their previous knowledge and working experience and integrate them into their learning (see Figure 4.1 for discussion script). The discussion also indicates that having different levels of knowledge and working experience affects learning performance, as well as accelerates reflection in the learning process. Even though experience was identified as a foundation of knowledge that leads to success in learning, the discussion indicates that having experience was not likely to aid in conceptualizing and understanding a theoretical approach. 93 Focus Group Discussion Script Emerged Theme One: Factors influencing RP online learning Coding Category 1: Previous Knowledge and Experience (“A student with a broad knowledge ‘( , base often accelerated his/her ,Wlthf’ut that experience, . learning experience by reflecting on investigating or understanding a past experiences.” (participant # 3, subject probably WOUId not have been so easy to understand or even round 1) . V \V/ complete an assrgnment “Work experience does not successfully. Definitely, a lot more necessarily aid in conceptualizing effort would have to go into and understanding a theoretical understanding a subject or solving a approach to a problem. This, I problem. It is important for think challenges the concept of students to build a foundation of academic teachings with real-life knowledge in order to take experience teachings.” advanced classes or tackle (participant #3, round 1) complicated issues.” (participant % 44 #5, round 2) . “I agreed with a perception that “I searched the 'Net for having different level of knowledge/background affect learning performance. For those students who have minimum information, and then I relied on personal experience.” (participant #8, round 1) experience/FM-related background, so \ much knowledge they can reflect on at “I think the more difficult the beginning.” (participant #7, assignments caused me to reflect round 1) more on my learning and “ _ N experiences. Those assignments Learmng can b? accelerated couldn't be accomplished by rote, using the experiences, automatic solutions. I really had to successes and failures of think about what I had learned and others. They become our best then how to apply that to the teachers. (participant #6’ problem/assignment at hand.” Kround l) jwartieipant #8, round 2) / Figure 4;]. Discussion script of coding category 1: Previous knowledge and experience Coding Category 2: Sharing Experience. While having personal experience related to learning topics is an advantage for individual learning, sharing experiences with team members is another valuable way to learn from others. Participants in the discussions agree that they were involved in the process of learning by sharing experiences, as well as 94 gaining benefits from the process as it applies to their working reality. However, there are some concerns, such as when a full-class is chatting (via the Internet), it may not be possible for students to share their opinions and experiences due to a time restriction and various other circumstances (see discussion script in Figure 4.2). Thus, a small group- setting is a better way to encourage online students to share experiences. A “pr0per mind set” is also mentioned as important in the process of sharing experiences. Therefore, to promote a culture of sharing experiences, we should pay attention to this psychological factor (mind-set), as well as to the arrangement of learning and teaching methods. Coding Category 3: Quality of Team. The quality of team members’ group learning has been identified as another important factor for online learning. Participants seem to agree that a good mix of team members with good channels of communication contributes to the (reflective) thinking process. According to focus group discussion, a collaborative team is needed as a foundation that helps to make group learning wOrk through constant communication, having responsibility for individual workloads, sharing their perspectives and helping each other in learning. The benefits of using a collaborative learning approach are pointed out by discussion participants; for example, “the more collaborative the team, the richer the learning. ” (see Figure 4.3). This finding indicates acceptance of the collaborative learning approach among online learners, as well as the importance of the quality of the team. 95‘ Focus Group Discussion Script Emerged Theme One: Factors influencing RP online learning Coding Category 2: Sharing Experience \ ml think experience does present an advantage in relating to course topics and assignments as well as with problem solving in general.” (participant #3, round k1) “I thought that the learning from the team experiences was every bit as valuable as the content of the courses.” (participant #2, round 2) “I agree with the perception that work experience helps in the FM class series. The team members, who have not had any work experience, were at a tremendous disadvantage. In fact, some of the people whose work experience was limited to very small facilities also seemed to have a disadvantage relating to those whose experience was with larger facilities.“ (participant #1, round 1) flalways think about previous knowledge,. .. At this point in my life, my brain just works that way. Any question, assignment or problem gets broken down into areas of understanding from previous experiences. How would I respond to the scenario-«putting myself into that position---walking through the building picturing my experiences-- -dealing with the questions of others, how would I react. Relative to the learning experience with the virtual university” (participant #1, round 2) “I think having more experience sharing opportunities in the small group setting can allow others to pick up "best practices" from fellow students and incorporate them into their learning experience. Sometimes it was harder to share experiences during the full class chats as “I concur with (garticinantfiil’s name) that a basic knowledge of facilities is extremely beneficial to success in these courses (or, in a more generic sense, some professional knowledge would be ' well, unless the proper mind-set is Qablished.” (participant #4, round 1) there were many people trying to express their opinions/experiences.” helpful in taking courses at this level, in any industry.) It is great that students can bring their experiences and best practices to share with the rest of the class, and that they have a 'reality' from which to draw insight in this process. However, it can be a handicap, as (participant #8, round 1) ( “I do believe that personal experiences tend to move online learners through the maze ....... This, of course, applies to any type of experience, and is not necessarily limited to online learning” Nanicipant #3, round fl Figure 4.2. Discussion script of coding category 2: Sharing Experience 96' Focus Group Discussion Script Emerged Theme One: Factors influencing RP online learning Coding Category 3: Quality of Team “Creating teams with a good mix of people contributes greatly to the learning experience. I don’t think this is a factor of number of team members, but rather quality of team members” (participant #1, round 1) “Students NOT towing be successful.” (participant #6, roundl) the load will NOT “Our team was in constant\ communication. We had great people to begin with. but he/she forced the advantage of our thought processes” (participant #1, round 1) “It has helped in the past t:< consult with other “I also think that by incorporating new technologies such as "conference cameras", an added level of "personal connectivity" might help make the team work all that much harder.” (participant #9, round 1) “Perhaps a part of certain tasks teammates regarding their views about the assignment. It is a benefit to have other opinions about an assignment so you’re not heading off in the wrong direction. (participant #5, \ round 2) or programs would require soliciting advice from a teammate, including individual assignments; or even inviting former MSU-VU participants to be available to dispense advice “For the classes in which I was grouped with dynamic people, the group projects were great learning experiences. I eventually did not want to be grouped with those who I knew from their experiences.” >’(participant #9, round 1) “Team mates could offer you ideas about where to acquire information to complete a paper. They might even have experience that you could tap into and document it as an interview.” (participant #5, round 2) round 2) “I was fortunate to be part of some very collaborative teams during a couple of my classes and learned much from them. The more collaborative the team, the richer the learning.” (participant #3, round 1) were "dead beats. "” (participant #1, “If teams had not been mandatory, I would still have sought out a support group to help figure out the assignments and share ideas. I think these classes would be very difficult to get through on your own. And like normal university classes, I think a good portion of the learning comes from interacting with your peers.” fiarticipanWS, round 3) Figure 4.3. Discussion script of coding category 3: Quality of team 97‘ Coding Category 4: Instructor Involvement. The role of the instructor and instructor involvement in class activities are indicated as important factors for collaborative online learning. Participants indicate that the instructor should provide the learning direction and must be very engaged in class activities to push students into a collaborative learning environment. The instructor also should be the one who has real-life experience in the topics he or she teaches (see Figure 4.4). This learning approach is also compared with professional practice in real-life situations where the instructor serves as the “boss or upper management” and as a practitioner; students should provide solutions to satisfy upper management in specific situations. This theme points out the philosophy that professional education has as one of its purposes to be applicable to real-life practice. Coding Category 5: Motivation. Motivation is another issue discussed among the participants. It was agreed that personal motivation cannot be overlooked. For students who are willing to learn, a well-prepared learning environment will help them in learning, but for students who are not inspired to learn, clearly defined expectations from the instructor may be used as a motivator for them. The willingness to learn is also indicated as a factor in the reflective thinking process. If students are not willing to learn, the process is not easy to trigger. The participants also agree that, without a willingness to learn, exquisite experiences or previous knowledge in the professional field will not help them to learn. The instructor also has an important role in motivating students, providing a proper learning environment, and in encouraging students to achieve the learning expectations. Therefore, a learning environment can be used to foster reflective learning is implied, just as is collaborative learning with motivation to achieve learning expectations (see discussion script in Figure 4.5). 98. Focus Group Discussion Script Emerged Theme One: Factors influencing RP online learning Coding Category 4: Instructor Involvement “. . .the instructor could point you in the right direction." k(participant #5, round 2) “I think that instructors must be veh engaged with class activities. (Instructor name) was a real “pain in the behind.” I say that with affection. f“ln my opinion, the best learning environment is one that is taught by someone who has personal experience in what he/she is teaching.” (participant #3, round 1) flee! the instructors should be more facilitators and collaborators in the course. I noted with humor (participant #1 ’5 name) comment on one of the instructors although in my case, I don't think I appreciated that particular teaching style!!! It also helps if an instructor is willing to learn along with the class. There isn't much that is more gratifying than having a teacher state they learned something from any particular insight by a student positive reinforcement, of a sort.” (participant #4, round 1) because he or she was also one of the best online instructors. He or she got in people’s faces with his or M online commentary. Whether you liked it or hated it, agreed or disagreed, he or she forced the team to work harder” (participant #1, round 1) / “I like instruction methods that engage the teacher and the student with enough individual or small group feedback to make a difference in how I might go about the next endeavor. . .. I think if more formal ways of having the instructor get into the discussions with the individual and teams, the students would benefit.” (participant #1, round 3) “On the other hand, an individual learning experience isn't much of a learning “The instructors serve as the boss' or upper management, and they are the ones that need be satisfied with analyses and recommendations on specific matters.” (participant #4, round 3) experience unless you have a very participatory instructor who is ready to give lots of feed back and critique. Generic comments don't cut it in that case.” (participant #1, round 2) Figure 4.4. Discussion script of coding category 4: Instructor involvement 99 Focus Group Discussion Script Emerged Theme One: Factors influencing RP online learning Coding Category 5: Motivation “Personal motivation cannot be overlooked in any environment, whether it is a learning environment, or the work environment... This assumption on my part takes us back to square one—personal motivation can/will get them through it.” (participant #3, round 1) \ >fl"he students must be engaged: ' \(whiiihgi:tthhfdzr$ 1.1.5.236 a Perhaps a statement, philosophical in willingness to learn is only one bent, explaining the needto want to of them. As artici ant#3’s learn, to put pre-conceptions aSide M stat e d,Ith e anIsw er t 0 what and to be open to new ideas would motivates the students (each as help. I also. think the instructors can an individual) would go a long take a pOSItive role in this process. way in ‘helping’ them learn. Again, at this level and dealing wrth . . . Either that, or they need to be presumably professronal, Willing motivated .. (participant #4 Qudents.” (participant #4, round 1% round 1) “I think in the first case we learn from our immediate environment. If we have opportunities and are driven enough, we learn from other environments.” (participant #6, round 1) “Some desire for further learning is triggered by the above process (reflective learning process), but some desire for learning is strictly spurred by my desire to constantly be on top of information in my profession.” (participant #1, round 1) “For people who actually want to learn, then I agree with the perception (having experience makes people learn better). For people who simply do not want to learn the experiences of others will not help. Neither will engaging group activities.” [Kararticipant #6, round 1) “For those who are not motivated, I think the expectations of the course administrators and fellow students must be very clear!” (participant #6, round 1) Figure 4. 5. Discussion script of coding category 5: Motivation 100 Coding Category 6: Learning Culture. Learning culture is another factor identified as important in an online learning environment. Participants seem to agree that a reflective learning culture is beneficial for learning as well as working situations. They also indicate that good work ethics in online learners need to be included in the desired learning culture (see Figure 4.6). The desired learning culture is one where team members have good work ethics and share their workloads, with positive attitudes in learning. To cultivate the learning environment, students should be notified that their roles consist of both group and individual learning tasks. The discussion indicates that students who have good learning (work) ethics and recognize the expectations of the learning culture are better prepared and respond to learning activities. Focus Group Discussion Script Emerged Theme One: Factors influencing RP online learning Coding Category 6: Learning culture “. . .it would be very beneficial to “As for team members as in any case promote a culture of reflective concerning your peer group some had more to offer than others. Some learning.” (participant #9~ I sought out for answers, others I did round 1) not. This was based on my , “In any event, the key is that everyone perception Of “WWW“ ethics an has to have a positive attitude, be a the “3'?” 0f the" Input. willing participant and want to learn >\(particrpant #1’ round 2) something new.” (participant #9, round I) “A strong work ethic applies in ’< “From experience, I feel that team dynamics are almost half the battle. If everyone "pulls their own weight" you will most likely succeed. If the team is dysfunctional, even with some talented members, there will be a less than should be expected outcome to the project.” (participant #9, round 2) / both situations. Someone with a strong work ethic—and that starts, usually, before the adult experience—will applies him or herself to whatever situation is present at the time.” “participant #3, round 1) Figure 4.6. Discussion script of coding category 6: Learning Culture 101' Coding Category 7: Cyclical Learning Process. Most participants tend to agree that their learning process is a cyclical one. It involves reflections on experiences, knowledge, observations of new information, and tests of assumptions for conceptualization and understanding of learning. The discussion shows that students link learning situations and problems at work together. They indicate that the reflective process helps them in learning. They solve problems (learning and work) by reflecting on their experience and thinking about how they could have done better and in what different ways. The reflective learning process is also compared with the cycle of the total quality improvement—plan, do, check, act, cycle—and is accepted as an improvement process. It helps students in making decisions using various options in both their learning and working situations. The words “. . .opened my mind” and “there is always room for improvement...” (see Figure 4.7), indicate that reflective practice leads to improvement (in learning and working performances). By comparing experiences (individual and sharing with others), students find better ways to do things. Even though most of the participants agree that reflective practice learning is a cyclical process of improvement, some participants identify that the steps within the process are not linear. They indicate that they may not occur in order and not even be a completed cycle. For some students, the reflective practice process involves only three out of the four steps (see Figure 2.2 for the experiential learning cycle). Some students define steps as going forwards and backwards within the cyclical process. These inconsequent steps within the process indicate the differences in thinking behaviors among students and the need for motivators or drivers to help students in getting through the continually improving process. The discussion also indicates a need for learners to be 102 open-minded and “reflective practitioners” and for previous knowledge and experiences to exist in order to be able to reflect about the context. Thus, RP learning is not a practical process without an existing foundation of knowledge and experiences. Focus Group Discussion Script Emerged Theme Two: RP Learning Process Coding Category 7: Cyclical Learning Process “These four phases read a lot like the Plan, Act, Study" process in Deming (Quality Management). I know there are many other names attributed to this, depending on what study of TQM people may have delved into, but they all follow the same cycle. I would also have to agree that, ideally, this is the way things do go.” (participant V4, round 1) “What I found, after my cynicism faded, was that the theory helped frame and form the basis for my reality. It opened my mind to different options and approaches to my job, and created my hunger for more education.” (participant Q4, round 1) “I looked back at experiences that would allow me to contribute or or outcome. Either as an individual or in a team format it is important to play the same. In the real working word that’s the approach I use. It’s important.” (participant #5, round 2) r “....they (learning processes) may not W produce the appropriate information‘ “As I studied the class activities and \ lessons I looked to ways that I could have done things differently in the past, or would in the future. There is always room for improvement. to refine the way you do things. We all wear blinders, of a sort, sometimes tied too strongly to tradition, and its good to investigate better ways of doing things. ” (participant #4, round 2) “As such, phase 4 (test the concept & knowledge in the specific situation) was not specifically something that I have done. I have, however, used my learning experience to help me better understand aspects of FM issues as we service our clients in our consulting practice. “(participant #1. round 1) / occur in that order nor would they be completed” (participant #6, round 1) “I have reflected upon previous experiences if it applied or somewhat applied to a particular class assignment or even a problem/issue at work.” (participant #5, round 2) / Figure 4. 7. Discussion script of coding category 7: Cyclical learning process 103~ Coding Category 8: Specific Milieu and Objective. Most participants report their learning process to be involved with previous knowledge, experience, work situations, and objectives for their learning and work. Some of them compare the learning process with the work process, which it must serve, or for solving a specific real life work situation. They try to apply their knowledge (theories and standards: see script in Figure 4.8) and experience to specific assignments to achieve their objective of learning (or working). While some view their learning objective as satisfaction of the instructor (or upper management), some learn that the purpose is to enhance their knowledge and use that new knowledge for their career development. Coding Category 9: Group Learning Activities versus Individual learning Activities. Participants seem to agree that group learning activities are beneficial to their learning (see Figure 4.9), while individual assignments also have advantages for specific reasons. While individual assignments allow students to concentrate on specific learning topics in depth, group learning activities reflect real work situations that require a team approach. Students indicate that they pay more attention to the content of assignments when they work alone. Even though the majority of the participants agree that there should be both individual and team assignments for the online courses, some students prefer to work alone (see Figure 4.10). They gave as reasons that they gain better quality of work and can reflect more on the learning content when they work alone. Some students imply that there is plenty of valuable time wasted in team effort while they work with "non- simpatico” teammates. In that case, they learn more from individual assignments. Some indicate that they have more creativity and freedom with less constraint in individual tasks. 104' Another group of participants indicates their preferences for group learning activities. They give different reasons for their preferences, such as sharing knowledge, gaining different perspectives, better quality from experienced team members, enhancing teamwork skills, reflecting more on others’ ideas and experiences, and diversity among them (see Figure 4.9-4.12 for discussion script). Focus Group Discussion Script Emerged Theme Two: RP Learning Process Coding Category 8: Specific Milieu and Objective “I tried to apply certain theories and standards that were proposed to see if they were valid when compared to real world situations. If you could not take a text book statement, theory. guideline, etc. and validate it with a real lifetime experience, then I think you would summarily reject it as being too ethereal.” (participant #9, round 2) “The way I approached these courses (especially at these levels), was to treat each course as if they were a special project assigned to me. In that respect, no matter how much I valued the mentoring or guidance provided me by an Instructor/1‘ A, I viewed them as upper management. They were looking for results or analysis, and my job was to supply them with the information they needed. As in the workplace, 1 would make sure I was clear on what was expected of me and deliver it in a professional way that eliminated as many questions as possible.” Qnicipant #1, round 2) “Circumstances that caused me to reflect\ on experiences or previous knowledge were generally driven by class assignments - a kind of "ok how do I approach this assignment". What parts am I comfortable with and what am I least knowledgeable about?” / “I think everyone going into a : (participant #2, round 2) learning situation has some sort of basic objectives they want to accomplish by taking the class. I think we all have these basic objectives in mind when we sign up for a class. It is helpful when the class publishes its objectives because then you can tell if it's a good match with what you want to learn in the glass.” (participant #7, round 3) ‘< “Learning is learning, and sometimes we need to go off in our own directions to follow some tangential thoughts. All in all though, being left to our own devices all the time would neither satisfy our instructors nor our managers.” (participant #4, round 3) Figure 4.8. Discussion script of coding category 8: Specific milieu and objective 105' Focus Group Discussion Script Emerged Theme Two: RP Learning Process Coding Category 9: Group Learning Activities versus Individual learning Activities these courses, so no, I would not opt to work alone completely. I liked the mix of both individual and team. I would opt to work with another student - or perhaps a "study group" where the assignments are discussed - if teams were not being Ktilized.” (participant #2, round 3) fl hen working alone I tended to focus more on the content of the assignment and how my own experiences could relate and help me with it. For the team assignments I focused more on the "team product" than the specific content quality and was concerned with making sure that I held up my end of the work and provided the team with good quality work.” Warticipant #2, round 2) “I think the team work was key to \Kltshould be a requirement for students to work as both individuals and in teams. . .. Choosing who was on the team was effective because we worked well together. We could rely on each other to accomplish a certain task. 1 know that choosing your own team is not always practical. Part of the learning experience is to work with other peers with different backgrounds. I think more can be done to assemble/deveIOp teams. I believe this would benefit all students.” (participant #5, round 3) >r “The individual assignment serves as an opportunity to address specific area of interest, and allows further investigation on a specific topic. On the other hand, the group assignment reflects a real world situation of working in team. It is the best A circumstance to develop and strengthen people skill.” (participant #7, round 2)) F igure4. 9. Discussion script of coding category 9: Group learning activities versus individual learning activities 106' Focus Group Discussion Script Emerged Theme Two: RP Learning Process Coding Category 9: Group Learning Activities versus Individual Learning Activities “In classes with "non-simpatico" “1 tended to reflect more about teammates, i.e. team members who what 1 was doing when working were marking time or not willing to on an individual assignment.” fully contribute, I absolutely learned (participant #3, round 2) more as an individual.” (participant #1, round 2) fllfnot mandatory, I probably “I felt that quality of my work would have liked working was better and was accomplished much easier when I worked alone.” (participant #2, round 2) alone----before I had experienced a team environment. The team groupings have a serious drawback m the Virtual . “I think the reflection and introspection\ univerSIty format because if . , was more prevalent when working on someone doesnt show up or . . . . ,, ,, . mdrvndual aSSIgnments. We could apply shows up by logging onto a . . , . our thought processes in more speculative chat but doesnt say anything, . . . . ways, playing With ideas and concepts. I there is no person-to-person . . . . . . . . found there were less constraint in those ability to get the indiVidual in . . . . . . tasks than in the team enVironment when line. It is a big time and effort . . ,, . . we were responsrble for only a piece of a waste. (participant #1, round . . 3 larger puzzle. Also, it was easrer to be ) proactive when working on single assignments.” (participant #1, round 2) “I found there was much more freedom and creativity allowed in the individual assignments. We were allowed to rise and fall on our own merits, and not be responsible for or reliant on any other players. To me that was more fun and more gratifying.” (participant (1, round 2) Figure 4.10. Discussion script of coding category 9: Group learning activities versus individual learning activities “Reading or doing tasks that require high levels of concentration are best done alone.” (participant #6, round 2) “Between group and individual assignments, I prefer the individual ones. This is because I found a tremendous amount of time was wasted while waiting on others to do their part of the team assignments.” (participant #3, round 2) 107 Focus Group Discussion Script Emerged Theme Two: RP Learning Process Coding Category 9: Group Learning Activities versus Individual Learning Activities “I found I would reflect more when working on a team assignment. When working alone I found myself relying too much on experience and standardized knowledge rather than thinking about new I and better ways to approach and solve problems.” (participant #9, round 2) “Working with another student or support group is a great exercise to enhance teamwork skill.” (participant #7, round 3) “I much preferred working in “I think the team brings a synergy to the finished product that sometimes is not there in an individual project.” (participant #6, round 2) “Working alone is NOT an option nor should it be an option for the entire Facility Management course. Communication with others, in my opinion, is a critical factor in the learning process. While working alone you might be able to learn everything, but in my opinion you will have a difficult time when you try Qapply that learning.” (participant {£5 round ‘1\ groups - team or class participation activities. . .. Also. team ideas could be bounced around and in my opinion the result, given that everyone was sharing and accepting, was always better than an individual's idea.” (participant #6, round 2) “I enjoyed all the assignments that involved 'participation' of other students and/or the instruction team. Through these interactions I think we all got a good idea of where everyone was - as far as knowing or understanding the material.” (participant #6, round 3) “The circumstance to “Between individual work and group work, I preferred group-«but only in the context of finding people who really wanted to be in the class and ' were ready to contribute.” (participant #1, round 2) reflect on learning experiences was group discussion either on specific topics or team assignments.” (participant #7, round 2) Figure 4.11. Discussion script of coding category 9: Group learning activities versus individual learning activities 108 Focus Group Discussion Script Emerged Theme Two: RP Learning Process Coding Category 9: Group Learning Activities versus Individual Learning Activities “In team activities on a corporate level. you usually have professionals from different disciplines taking part, which expands the scope of the project.” (participant #1, round 2) “l reflected more on my experience in the team environment. On team assignments, I could compare and contrast the different approaches we all had to the assignment.” (participant #1, round 2) “I guess the group activities were a prime factor for reflecting on my individual experience. As the team broke up to handle differing project needs, the assignments within the team many times were done on the basis of who knew the most about what.” (participant #1, round 2) “There are very few things in life that are done 'alone'. I think I reflected more on learning working on team assignments. Many brains are better than one??” (participant #6, round 2) “I quickly found that teams made a BIG DIFFERENCE in the quality of learning with others." (participant #1, d 2 roun ) “I saw the value of the \ teams, given the volume and diversity of the team assignments. Also, I was on a couple of teams where we all got along well and really complemented and supplemented each other. I “On the positive side, different insights and different solutions to problems are always beneficial to reaching the best resolution to a project .dealing with things alone tends to emphasrze and pornt out the found those experiences to be tunnel ViSion that everyone has and d' ,, limits peoples' effectiveness.” very rewar ing. (participant #4, round 3) Qartrcipant #1, round 2) j Figure 4.12. Discussion script of coding category 9: Group leaming activities versus individual learning activities 109 Focus Group Discussion Script Emerged Theme Two RP Teaching Methods Coding Category 9: Group Learning Activities versus Individual Learning Activities /.there were other cases Wham ran into (and witnessed) uncomfortable situations where personalities got in the way and good work was discarded due to personal opinion. As a minor note, there were some teams where members just didn't pull their own weight. This was easier to get around; we've all learned that a team is only as strong as its weakest link and experience has shown us how to deal with that.” (participant #1, round 2) “In most cases, real life, real situations are not like that and time is needed. Doing exercises or assignments on-line we sometimes tended to underestimate the time involved completing a task or portion of the work.” (participant #6, round 2) “Working within a team became more of a reactive process, adding another tier of approval and restrictions through which we had to travel to attain a final result.” (participant #1, round 2) “There are benefits in working in a team environment. Again, in our profession, teamwork is a necessity in most projects that FMS are involved in. This approach is especially effective when dealing with larger projects in a short timeframe. Also, it serves as practice (or a learning experience) as we have to accept the fact that a team is only as strong as its weakest link, forcing us to make adjustments and deal with intemecine problems and still realize success. This is a challenge in our “I like to research a topic inside- and-out and then write about it. Within the team environment the part about totally researching all avenues of a subject is somewhat lost.” (participant #5, round 2) A % “In general, I prefer a group assignment. However, there were a couple of teams that I was a member of that had some very reluctant participants and that made completing the assignment a very difficult challenge.” work as we try to manage or co- exist within a structure of diverse personalities.” (participant #4, (participant #9, round 2) round 3) Figure 4.13. Discussion script of coding category 9: Group learning activities versus individual learning activities 110’ Participants expressed several concerns about the group learning approach. Group activities can both accelerate the learning process, as well as lead to time and effort waste in the process. A large project may need a team approach to shorten the timeframe, while a group project may also struggle through an amount of time wasted in communication among team members. It may become a reactive process because of reluctant participants and other restrictions from team members, as well as the diverse personalities of the team members. Therefore, a team is only as strong as its weakest link (see Figure 4.13 for discussion script). Coding Category 10: Course Design for RP. Participants discussed teaching methods and had several recommendations for online course design. The recommendations related to the RP learning framework include: 1. To help build a culture of reflective learning, consider providing guidance on the thought process, such as a support group. 2. Provide web links with case studies, a course material list and other information over the Internet to encourage adequate information gathering and reduce the time wasted in searching for information. 3. Match different senior, more experienced members with others that have very limited experience; a team member with honed computer skills with a novice, etc. 4. To make the course more challenging and expand learning horizons, encourage web talk weekends for specific advanced learning topics and challenge a sharing of experiences among students by facilitators. lll 5. Provide more interactive activities among the instructor, teaching assistant, and students, including both in the beginning of the class and in follow up activities. Consider assignments that challenge sharing experiences and best practices. such as an interview assignment on a specific learning agenda. Provide voice teleconferencing for team meetings, which move along much smoother and probably more efficiently than live chat. 8. Include quizzes as a requirement of the course rather than examinations and tests, as quizzes will be the driver for students to catch up on class content by reading. Focus Group Discussion Script Emerged Theme Three: RP Teaching Methods Coding Category 10: Course Design for RP (to help build a culture of reflective 7/“1 think there's benefit in providing \ learning, consider providing guidance on the thought process such as decision guideline. For example, developing course materials that describe the relationship between business decision making and the implication on corporate facility to show how the lesson learned in class will help make good facility decision that meet business objectives.” (participant #7, round 1) some capability of voice teleconference team meetings if this is cost effective. I think a team meeting held as a conference call can move along much smoother and probably more efficiently kthan live chat.” (participant #2, round / f “I also like the idea of the instructor relaying web links to case studies, research information and other “I did feel that the team approach on certain projects definitely helped my learning experience. . .. completing an members with some that have very (participant #9, round 1) sources to communicate experiences.” analysis of the make up of team members, i.e. matching different senior experienced limited experience; a team member with honed computer skills with a novice, etc.” (participant #1, round 1) “For a student, the on-line instructor could include in the course materials a list of reputable on-line research sites (such as the Electric Library) in order to encourage adequate information gathering.” (participant #3, round 2) / Figure 4.14. Discussion script of coding category 10: Course design for RP 112' Focus Group Discussion Script Emerged Theme Three: RP Teaching Methods Coding Category 10: Course Design for RP play off of experiences helps. Encouraging class members to relate experience with specific issues, relaying web links, email addresses and other items that would spur communication might be good.” “Generating weekend web talk topics that “I think the courses need to be \ challenging... The purpose of education at this level must be to expand horizons and stretch thinking.” (participant #4, round 1) (participant #1, round 1) f \< “Maybe there could be an assignment that explored the background of other team members and how it relates to the course subject matter. Once this information is available the student could try to reflect on others past experiences/knowledge when working/completing assignments.” (participant #6, round 1) / “I enjoyed and appreciated the interaction processes that developed in the assignments.” (participant #6, round 3) “I will add comments ...... Encourage information sharing between students, faculty and others. Create arguments and have discussions led by facilitators. Afterwards have a follow-up program that includes continued discussion even after the course completion.” (participant #6, round 1) “I would like to see a team chat session at the beginning of the class with the \ instructor.” (participant #5, round 3) 1 (:One of the best learning experiences l that engaged people in the class was an interview assignment, in which we did either an online or a telephone interview with another classmate, not a team member. The discussion topic was pertinent to the class agenda.” \ (participant #1, round 1) / “I liked the case studies because they gave you a chance to apply what you learned. I also liked the web talk weekends because they were a forum to share ideas and best practices with a wide variety of facilities professionals that I don't get on a regular basis through my job.” warticipant #8, round 3) “I prefer a given topic, with an individual assignment in writing. I have no interest in exams. And I did enjoy the live chats,...l would like to see the use of live video, with the instructor dealing with whatever is the most challenging topic from the course. This could be coupled with an 800 number for call-in questions.” (participant #3, round 3) Figure 4.15. Discussion script of coding category 10: Course design for RP 113' Focus Group Discussion Script Emerged Theme Three: RP Teaching Methods Coding Category 10: Course Design for RP fllliked the use of case studies, analysis paper, ! “The team/ instructor and Web Talk weekends. Each assignment was chat 5355i“! would be of great benefit. A case study allowed for an in a 300d way to interact depth look and understanding of a particular and get to know each issue/subject. An analysis paper did the same. other. There 5'10“” be Web Talk Weekends allowed for open discussions some sort 0f format on a particular subject and normally motivated me presented prior to the to investigate the subject in detail. This was a good chat.” (participant #5, learning experience. It was interesting to read round 3) other students responses. Many ideas, experiences, and views became available.” “I have found the written assignments (participant #5, round 3) to be worth while in exploring various topics. I enjoyed doing the papers, short and longer. These gave me the opportunity to explore something in depth.” (participant #1, round 3) “Case studies/case analysis paper type of assignment is preferred since it demonstrates an understanding of the course materials and student's ability to apply the new knowledge. Written analysis of a situation or factors examinations could be beneficial for students to strengthen analytical skill.” (participant #7, round 3) “I found the case studies or analyses to be interesting in the team format. Quizzes force the student who doesn't take readings seriously to get with it. I found that when quizzes came up, I made a point to read materials! I don't think exams, per se, are of great value in the “I really liked the dialogue continuing education arena. Quizzes situations with the instructors. are enough. 1 like the final paper and These were generated primarily the final team project as the "final with the submittals of papers and L exam." (participant #1, round 3) projects. Most of the classes did not include enough instructor “The online courses didn't just rely feedback. I think that the posting on reading the lectures - there were and feedback are an important team assignments and class chats method of gaining information that gave us a chance to interact about what we, and other teams with other people in the class and might be doing.” (participant apply what we were learning.” wound 3) (participant #8, round 3) Figure 4.16. Discussion script of coding category 10: Course design for RP 114 Focus Group Discussion Script Emerged Theme Three: RP Teaching Methods Coding Category 10: Course Design for RP fl .exams or even quizzes would not “Examinations or tests are great fox be the best way to measure a student's progress. Those tend to be artificial and almost always deal with recall issues as opposed to intuitive periodic updates (reports) and status checks would be more appropriate; much like we have to do in the real world.” (participant #4, round 3) “I do think that the web talk is effective, as it supplies a medium for discussion and an exchange of results or 'reflective' learning. Rather. the instructors in that it would be able to help the instructors know where everyone fit on the learning curve, but from my personal point of view the self—tests. quizzes. web- talk, team meetings, etc. were good enough to let me know where I stood.” (participant #6, round 3) “Having a support group is a good idea for expanding the experience base of the group you contact.” (participant #1, round 3) ideas and best practices, again, girliiisilgliilgzlgoptfitethecregswle flOne exercise that brought out a lot of (participant #4, round 3) reflective experience was the Web-Talk k weekend relating to technical advancements in FM. Many people discussed their knowledge with others, including very basic information sources. . .. Credit could be given for posts associated with that section of web-talk.” (participant #1, round I) “Students today are not as dependent on that source since there is an abundance of information on the Internet.” (participant #3, round 2) “I would request some suggestions from the TA and probably the library for any additional sources of information they may be aware of. The intemet is great, but unless you know what you're looking for - you can waste a lot of precious time.” (participant #2, round 2) “ I would not prefer the inclusion of exams. . .. I liked case studies and also liked the web talk weekends. I also think that situation analysis assignments would be very useful.” (participant #2, round 3) Figure 4.1 7. Discussion script of coding category 10: Course design for RP Coding Category 11: Written Communication versus Oral Communication. Participants in the discussion agree that written communication has many advantages for 115. online learning. It allows students to think more about what they are communicating, as well as allows more response time for reflecting. This availability of time leads to a higher quality of thinking and clarification of content than regular oral communication does. Compared with oral communication (via the phone or in a regular class format), students have an equal voice when they communicate in writing via web talk due to the time restriction. It is also easier in writing for references and a retroactive process of reflection (see discussion script in Figure 4.18). Coding Category 12: Use of Assignment Examples. From the participants’ perspectives, examples of assignments posted for students are considered very helpful as “best in class.” They indicate that it points out the instructor’s expectations and criteria for the assignment. The participants seem to agree that it would be helpful only when it is accompanied by the instructor’s comments on the assignment: what is good and what is bad about the assignment. Participants also indicate that an example of a good assignment is another way to share experiences and best practices among students. It also helps students to measure the quality of their assignments, as well as encourages them to achieve a higher standard of learning and working. Good examples can also help new students organize the structure of their assignments, as well as provide ideas for their final products. Some participants agree that examples of were done assignments helps them in analyzing their own work. For this issue, one should be careful concerning the availability of the assignment examples as it is for the purpose of knowledge sharing and is meant to be used as a guideline for criteria and standards, but not for replication of contents or plagiarism of ideas. It should not be misused. 116 Focus Group Discussion Script Emerged Theme Three: RP Teaching Methods Coding Category 11: Written Communication versus Oral Communication “I know personally that I think XCommunicating in writing on—line can be\ more about what I am going to say a challenge in that the "tone" of before I post a message. I believe communication should not be intimidating that depending upon a person’s or reflect an unintended message. And. style/abilities they might think that yes. I do believe it forces students to online and oral is the same.” think more about what they say or (participant #5, round 3) write.” (participant #3, round 3) $1 . .except live chat, written : communication allows longer response time. Therefore one could think and reflect more before responding. Written communication also requires thoroughness to minimize interim corresponding due to clarification.” “I think by communicating in writing it does provide a better response. It gives you time to analyze what you're going to write and how to phrase it sensitively..... When everything's written you have to read it first and then rocess it and , , then give a response, wliich is (participant #7’ round 3) hopefully a little more thoughtful.” >\ (participant #4, round 3) “Sometimes in oral communications you get caught up in what you want to say next that you don't fully listen to what . . . ,, the speaker is saying. Or you blurt out Of Wins n communication. something in haste that you might not want (participant #2’ round 3) to have said.” (participant #8, round 3) “I definitely feel that writing online was 7 u - - - . N beneficial to this process. If the end result of ...m my opinion, putting th’ . th . l d . . t thoughts in writing is good is exercrse at we are mvo ve mis o for this type of course. Why, weigh the aspects of reflective leammg. then because verbal conversations this part of the sessnons should have lent the tend to be totally dominated most credence to it. When we write, we have by those who tend to be more the opptzrtpmty to choose ourtvryords and aggressive and like to hear ensure a we are conveying e message we their own voice. Everyone Intend. It takes much more deliberation and has an equal voice if thought than an oral discusswn, as we are everything is in writing. able to revrsrt what we said and Improve its Besides one could always quality. We can always have discussions after look back to see what was the fact for clarification but if we need to put written as opposed to trying to our best foot forward and showcase our figure out who said what and learning, writing is the way to do it.“ h n - - . rt t #6, Wticipant #4, round 3) w; 33;»! ICIPan J Figure 4.18. Discussion script of coding category 11: Written communication versus oral communication “. . .this is certainly an advantage 117’ Focus Group Discussion Script Emerged Theme Three: RP Teaching Methods Coding Category 12: Use of Assignment Examples fixamples of 'best in class' were not that beneficial to me regarding the assignments, per se. They may have been interesting or provided new insight, but I never felt that someone else would do 'better’ than me and or my team. If we all work with the same criteria and applied our knowledge and ability to the problem, final drafts were invariably different but never lacked in the quality of the recommendations. In the end, at this level, better or worse really tends to be subjective and no one can use that as a guide.” (participant #4, round 3) “Yes, the class can learn from the very good examples of assignments that there could be good solutions to a problem. In addition, these examples show the level of comprehensive thinking, innovation, and presentation skill.” (participant #7, round 3) Yua superior piece might help K \ point out what expectations were, but that is not the best way to inspire effort on the next assignment.” (participant #4, round 3) “Posting 'good' assignments is < always a positive thing for students. The assignments should be posted with a note from the instructors or evaluators as to why they are 'good'. One might even want to post some 'bad' assignment, ensuring they are anonymous of course.” \ (participant #1, round'3) \< “Posting of other teams or individual assignments was only good if you also had the teacher's comments about the work along with it. I don't think just posting does a whole lot, particularly if you don't think that the work is very good.” (participant #1, round 3) “I did find the posted assignments to be very useful in analyzing my own work.” (participant #3, round 3) “I found these (examples of paper) helpful and learned from their content and structure - especially as a new student. I still keep some of my teammates papers and refer to them now and then because of their applicable kcnntent.” (narticinant #2. round 3) Quad 3) “. . .the class examples were helpful in analyzing my work. I used the class examples to see how well my content measured up as well as if others shared my point of view on what I thought was important in the assignment. I think the examples also give a clearer picture of what the professor is after in the assignments.” (participant #8, ) Figure 4.19. Discussion script of coding category 12: Use of assignment examples 118’ Focus Group Discussion Script Emerged Theme Three: RP Teaching Methods Coding Category 13: Learning by Objective “Learning objectives should “Learning objective should be specific \ be provided by someone (to outline behavioral outcome of the knowledgeable in a particular course) and common (every student will subject. I guess with some be evaluated against the same set of research a person could criteria). Therefore, it would be best for investigate, recognize, and the instructor team to formulate the develop learning objectives. learning objectives.” (participant #7, because of the lack of experience, etc.” (participant #5, round 3) I >f“l like the instructor's identification of learning objectives. Again, I am going to develop the projects, I will be imposing my own objectives on the session any way. I think there needs to be a stated learning objective for the Cass.” (participant #7, round 3) Still, it would seem that areas of round 3) a subject would be over looked “I think it would be very difficult< to develop our own learning objectives for these classes. No matter what rung of the ladder we are on, we always need external objectives to determine where we are going. we have to satisfy the goals of those we working for; self directed goals tend to help no one but ourselves, even with the best intentions.” (participant #4) Nund 3) Figure 4.20. Discussion script of coding category 13: Learning by objective Coding Category 13: Learning by Objective. In evaluating the learning approach used among students, the issue of learning by objective is discussed. Even though no one discusses the importance of using learning objectives as his/her learning guideline, one participant indicates that the learning objectives should be very specific in describing a behavioral outcome (see discussion script in Figure 4.20). Only a few participants indicate that they developed their own learning objectives. Some urge that learning objectives should be developed by experts in the learning topics because it is very difficult to develop their own without direction, especially without experience in the FM 119 field. Thus, the discussion indicates a need for specific and preset learning objectives from the instructor to be used as behavioral and learning guidelines for the students. Focus Group Discussion Script Emerged Theme Three: RP Teaching Methods Coding Category 14: Technology Issues “I would support use of voice/vidmince this is 'high tech' stuff maybe conferencing as a tool to augment proponent of linking people with video, as long as the access to appropriate technology is there. I chat limits the person-to-person connection.” (participant #1, round 3) personalizing the sessions. I am a big telephone, voice conference or even think that strictly email, web talk or some more futuristic teaching methods could be used. Let's say video or interactive video lectures; virtual building tours or case studies; use of PowerPoint with audio for on- line lectures” (participant #6, round 3) >/ “Video conferencing would be a difficult technology to implement. . .Other technologies “Computer skills are also lacking. This makes online more challenging” (participant #3, round ,1) learning for students in that field 4 would be a greater benefit in the course like teleconferencing. It seemed easier to discuss issues in an open format than trying to respond in a chat session. . .. Generally, live chats are a good “Consider video conferencing since it allows real-time access to a common material (such as a presentation) as well as live discussion. It is desirable to offer education/training to students via technology currently used in the business community. However, the school should assure that adding the new technology would not be an economical burden to students.” (participant #7, round 3) way to communicate and that method should stay.” (participant #4, round 3) “I don't see any real advantage of video conferencing or streaming video. Everyone has to some level of technology available and I think these would be unavailable for most of us. Technology is great, but at some point we all have to pull out the books, read the lecture materials and apply what we learn. Too many bells and whistles can j fit in the way!” (participant #2, Figure 4.21. Discussion script of coding category 14: Technology Issues 120' Focus Group Discussion Script Emerged Theme Three: RP Teaching Methods Coding Category 14: Technology Issues flTechnical innovations are great if everyone can access them. . .. The dilemma is time at home with less technical capability. Voice conferences could be good to allow people to connect more personally. Live chats with the team are good, but with too many people are bad.” (participant #1, ound 3) “Video conferencing would have been nice so you could put a face with your teammates and professor's names. You develop relationships with these people over 16 weeks and yet you never know what they look like. I'm not saying this detracts from the learning experience; it'sjust a human thing, you want to see who l,” “One’s ease of using the keyboard can be a factor, particularly during Chan. written from and patience with those who are not as capable can be factors in participant #1, round 1) Typing speed. ability to put thoughts into how one “come off” to other teammates’ you're talking to. With the further refinement of DSL and cable modems, video shouldn't be such a hurdle from the technology end. However. not everyone in the class has access to the same level of equipment, which could make video conferencing difficult in the class 3 h} “After searching my own memory banks and notes, my second step would be to check in any books I had on hand, or lacking that, the Internet.” K(participant #1, round 2) setting.” (participant #8, round 3) Xproblem exists if you have a smart individual that can't type very well, or has a slow computer. Also, I am afraid that \< / somewhat of a purist and treating these utilizing cutting edge technology whe is not available in the workplace wou be a mistake. Some of us, also, were would have been a bit unfair.” (participant #4, round 3) “I think technology is great, and that we all as Facility Professionals should be aware ofwhat is available to us. However, being classes as if they were real world problems with solutions needed in real world terms, working with limited resources, especially at home. Offering these options or making us reliant on them to complete the course the trend is to use fractured English as a short hand, with mis-spelling and abbreviations, as the primary communication over the web line. Nuances that you would have in conversation are lost. Oflen the idea a person has can't be properly expressed. Some people may not be able to communicate thoughts online like they can by speaking.” (participant #1, round 3) n it Id Figure 4.22. Discussion script of coding ca tegory 14: Technology Issues 121' Coding Category I 4: Technology Issues. Participants generally support applying new technologies to be used online for the class. They request voice and video conferences to enhance the capacity of real-time interaction. Participants appear to be divided into two groups on this issue: one agrees with the use of a telephone call for the learning activities with the other disagreeing with the use of a telephone as a connecting media. While requesting voice and video conferencing, participants also express their concerns about using these types of technology. While the technology can be used effectively in a timely manner, some students may need extra computer skills for operating the machines. Besides the computer skills needed, not all of the students can access the technology. Therefore, using these technologies has both pros and cons, including all of these factors and cost effectiveness. In summary, this chapter presents findings from a qualitative perspective. The data from the first 18 questionnaire responses were analyzed using a triangulation method of quasi-statistic, logical reasoning, and behavioral analysis. The findings from this analysis were used in formulating questions for focus group discussions. The findings from the content analysis of the existing literature are presented separately in chapter two. There are five themes that emerge from the quasi-statistical analysis. The themes are amplified by comparing theoretical themes and behavioral themes to validate consistency relationships within the themes. After the questions were set based on these themes, the ground theory approach was used in fabricating coding categories for the data. By the end of the analysis, 14 coding categories emerged within the three divided themes: factors influencing RP online learning, RP learning processes, and RP teaching methods. In the 122 next chapter, these findings will be discussed, along with quantitative results, in order to identify a RP conceptual framework and to make recommendations for future studies. 123 CHAPTER FIVE Quantitative Analyses and Results This chapter discusses the quantitative analyses and results of this study in detail. The quantitative analyses mainly involve two analysis techniques of MANCOVA (Multivariate Analysis of Covariance): factor analysis and structural equation modeling. The results from exploratory and confirmatory factor analyses are used for identification of related constructs within the RP conceptual model, as well as validity and reliability tests for each variable’s measurement model. Structural equation modeling is used for preliminary identification of the relationships among the variables involved in the RP conceptual model. Partial correlation analysis, multiple regression, and basic statistical analyses are presented as supplementary evidence for clarification of the results. The following analytical presentation is divided into three sections. The first section presents factor analyses of measurement models for each of the variables involved in the RP preliminary framework. The second part contains partial correlation analyses, multiple regression analyses, and best subsets analysis of constructs related to the RP learning framework. The third section is an analysis of structural equation modeling for alternative conceptual models. A month after the consent letter was sent out to the target population (approxi- mately 100 people) via e-mail, a follow-up reminder was e-mailed. A total of 31 124 ‘ participants responded to the request. Allowing for a reasonable number of incorrect or outdated e-mail addresses, we estimate the actual total population to be 90 people. Using this number, the response rate is about 30%. The participants responded from various demographic locations, including Asia and Canada, but most are in the United States. The voluntary sample of 31 consists of 17 males, 13 females, and one of unknown gender. Most of the participants are currently working in the FM field. Five identify themselves as a “full-time student” and only two indicate their current working positions to be outside of the FM field of practice. The raw data from questionnaire responses are tabulated in quantitative format (see Appendix D). The first task of this quantitative analysis was to conduct a face validity audit by looking at the tabulated raw data. There are thirteen sub-scale items that were eliminated fiom the analysis: C3, L1, L2, WP1, WP2, WP3, WP4, WP5, WP8, LPl, LP2, LP3, and LP4. The exclusion of these sub-scale items is due to a low variation of data that shows a low level of content-related validity. In this case, the lowest edge of elimination is a 20% variation. The low level of content-related validity is due to the design of the questions, words or language used, as well as the interpretation of the questions. Another criterion for those that were eliminated is the rate of each questions’ response, where i.e., a low or no response. For example, sub-scale item L2 was eliminated due to a low response rate. This question asks participants to identify their behavior when they take an online examination, but only one of four classes has an examination online. Therefore, more than half of the participants did not respond to this question and the content-related validity of this scale item was extremely low. There are 125 i also construct-related and criterion validities that are tested using the factor analysis approach. These validity tests are presented in the following sections. Section One: Factor Analysis To determine how reliable each sub-scale item is in order to be used as an indicator of its involved variables, and whether it is valid to include it in the measurement model, the following factor analyses are provided for each variable that is identified in the preliminary RP learning model (see Figure 3.2 for the RP preliminary model). Learning Technique Utilization. (TU) There are 13 sub-scale items within the learning techniques utilization (TU) measurement model (see Table 5.1 for TU descriptive statistics). The descriptive statistics of TU show that online students still rely on reading as their major learning technique. Searching for information on the Internet and team discussion are more popular methods of learning for them than asking questions on web talk or from the instructor. However, statistics show a very low proportion of self evaluation activity, as well as the use of video and audio media (provided in some classes). Even though the statistics show almost the same ratio of asking questions on web talk and note-taking activity among students, the higher value of the standard deviation in note-taking indicates the differences in learning behaviors among students: the higher the standard deviation value, the more differences in that learning behavior. 126 Table 5.1 Descriptive Statistics of Learning Technique Utilization (TU) Descriptive Statistics Learning Technique Mean Standard N Utilization (Scale 1 to 5) Deviation TU 1: Reading 4.00 0.931 31 TU 2: Team discussion 3.29 0.739 31 Searching TU 3: information from 3.39 1.283 31 Internet TU 4_ Asking question on 2 84 1 293 31 ’ Web-Talk ‘ ' TU 5: Asking Instructor 2.03 1.224 31 TU 6: Note Taking 2.81 1.447 31 , Analyzing other’s TU 7. paper 2.26 1.237 31 TU 8' Reviewing Ieaming 2 03 1 169 31 ' objectives ' ' TU 9: Class discussion 1.94 1.124 , 31 TU 10: Self evaluation 0.84 0.820 31 , Reviewing TU 11. video/audio 0.94 1.181 31 _ Reviewing TU 12. discussion 2.19 1.078 31 , Reviewing own TU 13. paper 2.32 1.301 31 The differences in learning behaviors among students can also be identified using exploratory factor analysis with the principal component method and Oblimin rotation analyses (see Table 5.2). The pattern matrix analysis shows five different patterns of learning techniques are used among the online students: 127 i 0 pattern one includes TU8, TUS, TU7, TU4. TU] 3. and TUl. 0 pattern two includes TU13, TUZ, and TU12, 0 pattern three includes TUl, TU3, and TU6, - pattern four includes TU9 and TUIO, and 0 pattern five includes TU4, TU9, and TUl 1. Table 5.2 F actor Analysis: Pattern Matrix Analysis of Learning Technique Utilization (TU) Pattern Matrixa Component 3 T—US TU5 TU7 TU4 TU2 TU12 TU13 TU1 TU3 TU6 TU10 TU9 TU1 1 .865 .747 .643 .534 .421 -.416 .876 -.784 -.424 .913 .669 .467 -.873 -.823 .437 .413 .915 Extraction Method: Principal Component Analysis. Rotation Method: Oblimin with Kaiser Normalization. a. Rotation converged in 45 iterations. 128 Along with the exploratory factor analysis, reliability statistics of the TU measure- ment model are reported. The statistics show acceptable reliability levels, as well as possibilities for adjustment of the model to gain a better reliability level (see Tables 5.3 and 5.4). If TUl, TU 2, and TU 11 are deleted from the measurement model, it can be anticipated that the reliability of the measurement model will be very high. However, to readjust the measurement model, the construct-related validity of the learning patterns must be reassured within the measurement model as well. The sub-scale items (constructs) involved in each of the extracted patterns can be validated using confirmatory factor analysis as presented in Figures 5.1 to 5.3. Table 5.3 Reliability Report of Learning Techniques Utilization (T (0 Measurement Model with 13 TU Sub-scale Items Reliability Statistics Cronbach's Alpha N of Items .785 13 | The first confirmatory factor analysis indicates five insignificances of factor loading: TU13 and TUl in learning pattern one, TU2 in learning pattern two, and TU9 and TUll in learning pattern five. Multi-collinearity between TU12 and TU13 is also revealed. The learning techniques that contain insignificant factor loading are recommended for deletion from the measurement model. Learning pattern five should be also eliminated because it contains only one learning technique. 129 Table 5.4 Reliability Report of Learning Techniques Utilization (TU) Measurement Model: If Each Item Deleted from the Model Item-Total Statistics Scale Corrected Cronbach's Scale Mean if Variance if Item-Total Alpha if Item item Deleted Item Deleted CorrtLation Deleted TU1 26.87 62.383 -.009 .802 TU2 27.58 60.652 .167 .788 TU3 27.48 51.125 .564 .755 TU4 28.03 _ 52.232 .493 .762 TU5 28.84 52.540 .511 .761 TU6 28.06 50.396 .517 .759 TU7 28.61 52.178 .526 .759 TU8 28.84 54.073 .447 .767 TU9 28.94 54.796 .424 .770 TU10 30.03 58.766 .292 .780 TU11 29.94 56.462 .296 .782 TU12 28.68 55.359 .411 .771 TU13 28.55 50.723 .577 .753 130' . .2505: 5555:8332 ADC 585535.55: 255355.02 955an. .50 $93me 520555 50955555555ch .5 .m. susum 3.551.505 5655:. 55.... .0 $055800 .3. on 60 o v in. . . as. on $59,555 .55. on s 8. «on 5553255 5a 8.5.8.0 mm 552, 285.5550 3535583553 885.80 ..5.53.5 65.25.455.33 mssmsméx IL I“ o 5 D... 5.0.5290 ...wm , on. 5W. 55.55.5859... ...Mi. 9.55 55.053.58.55 55.350 x 555 555559 20.2 m3. 855. 59.5.5255. 5555555963 x. .c05mwsom5u 5555530305. ND... .c0_mm.30w_.U 5...me X 55 55.555.55.55. x m5: .5 50.5552... Eso 955250555. v: .5 5m .5 no>> co .m.e.l_c.m< 5D .5 6.55.53. M59550 m..5.5_~>_mc.<. 8. 5 5.58555... 0 N 5 D... «.556 55535955 . 85.5 «55.58%... lilV. av N 5555253 m: .5 52035595. 55.55.3535 w: .5 325830 55.55.355.55. TI on. 131 55.555. N .0555: 55555505305). “ADC 552535.55: 253555.85 $555553. .50 $3.528 58055.5 55855585555500 .Nw. 95%.: new". $52555 o 53 .5 5550.53.96 53w $.35. ”CgmmJOmi mam—0 5.5558595. 5.55.55 55.5.5201 55 555.585.: v3 .5 5.35.5553 :0 95.5.3. 55:350.. 132 Amv .2305: 555353555285: .95.... 55.55 55:53:35. 35.53.555.32 wEEmo. mo 2229.553 5923.5 50955555555550 .5..“ 33%.: . , 555.515.55.05 3855.5. 5...... 5.53 325.80 3.55.185 .v, 3min... 5,55 35.550. 3a 535. 35,35. 3.935 8 o- .. 335...”. ..E an .3 3.31.55.33.55? 35.558 3 5.3935350 3533.533 3.5538586 053 5 2055.33.93 5.3m 3:. .8." ".5 .5.... QDH .c0_wm...:m.vm_U wmm50. 353 .553 u 5.. ..v 335 5.5.5.9... 9532 .355... .5535». .35, .5.. n55 .5253 Eso 555.305.5553.". T. 3,. v35 .3._w5-no>> co Hmc.3.w< Trina 5 5:5 .239... 5.3.5350 ..mcfibmca. Tina. m... .5 5235555.... 55.5.3.2 a: .5 ..5.3.0>_3oo...n,o m:..2.o.>om. Time 5 133 After elimination of insignificant sub-scale items and pattern five, another confirmatory factor analysis is conducted. The second round test (see Figure 5.2) still shows an insignificant factor loading of TU1 in pattern three, but the multi-collinearity between TU12 and TU 13 is cleared out; thus, TU1 is removed from pattern three. As a result, the final measurement model for learning technique utilization (see Figure 5.3) contains four learning patterns Pattern one contains TUB, TU5, TU7, and TU4; pattern two contains TU12 and TU13; pattern three contains TU3 and TU6; and pattern four contains TU9 and TU10. The goodness of fit of this measurement model is significant at 29 degrees of freedom with a p_value of 0.77. (The p_value of 20.05 is considered to be significant for the test of fit). The goodness of fit index (GFI) of the measurement model is reported at 0.87 (87%). Interaction to People and Information Sources (1) Interaction to people and information sources (I) is another latent variable that cannot be observed directly. The questionnaire contains nine sub-scale items of the interaction measurement. Two of them are true or false questions that are used in the qualitative analysis part. Only seven items are used in this factor analysis and measurement model. The questions are designed using group learning techniques and by using responses to specific hypothetical learning situations as indicators. The fi'equency of group learning technique utilization and varying levels of responses to situations were transferred into a (Likert) scale ranging one to five. The reliability (Cronbach’s Alpha) of the measurement model when all seven sub-scales are included is 0.522 (52%) (see Table 5.5). 134 Table 5.5 Reliability Report of Interaction (I) Measurement Model 1 Reliability Reliability Statistics Cronbach’s Alpha N of Items .522 7 Table 5.6 Reliability Report of Interaction (I): If Each Item Deleted from the Model I Item-Total Statistics Scale Corrected Cronbach's Scale Mean if Variance if Item-Total Alpha if Item |t_em Deleted |t_em Deleted CornLation Dtieted l 1 17.94 14.262 .178 .514 l 2 17.35 14.903 .038 .577 | 5 18.39 15.245 .179 .512 l 6 18.29 11.213 .462 .382 l 7 18.84 12.806 .253 .486 l 8 19.52 11.525 .465 .386 l 9 19.74 13.465 .255 .485 Table 5.7 Reliability Report of Interaction (0 Measurement Model 2 Reliability Reliability Statistics Cronbach's Alpha N of Items .604 5 135 Table 5.8 Reliability Report of Interaction (I): If Each Item Deleted from the Model 2 Item-Total Statistics Scale . . Corrected Cronbach's Scalei Mean Varlltaerge If Item-Total Alpha if Item Item Deleted Deleted Correlation Deleted I5 10.32 10.43 .379 .559 l6 10.23 8.31 .381 .539 I7 10.77 7.98 .428 .510 |8 11.45 8.52 .393 .531 I9 11.68 9.69 .260 .599 The exploratory factor analyses indicate that the reliability of the measurement model can be increased (see Table 5.5 to 5.8). When I] and 12 are removed from the measurement model, the reliability improves from 55% to 60% (Cronbach’s Alpha from 0.552 to 0.604). However, confirmatory analyses of the measurement model verify only three significant sub-scales items (t_value >1.96) for the measurement, interaction via team discussion (15), using the Internet for gathering information (16), and using Web Talk for interaction (17) (see Figures 5.4 to 5.6). 136‘ Using team discussion Interaction 5 I 5 .22 —D Using the Internet for searching for information Interaction 6 l 6 1.91 —-> Using Web-Talk Interaction 7 l 7 174—. Asking Instructor Interaction 8 I 8 166—} Using Class E Discussion : Interaction 9 p I 9 : 1.45—> 28. t= 2.31 Interaction 1 .00 ‘ - Goodness of Fit Statistics: Chi-Square with 5 Degree of Freedom-=2. 13 (If 0. 83) ' : RMSEA= 0.00 p_ value for Test of Close Fit (RMSEA ‘< 0 05)}: 0.585 Goodness of Fit Index (GFI)= 0.97 Figure 5.4 Confirmatory factor analysis of interaction (1): Measurement model 1 After removing I9 (interaction via class discussion) from the measurement model, a second rounded test still shows the insignificant factor loading of 18 (interact with instructor). Thus, since a good measurement model is not supposed to contain less than three sub-scale indicators, an equal factor loading is assigned for each of the remaining subscales to test whether the factor loadings are significant. The analysis shows significance of all the remaining sub-scales, IS, 16, and 17, with influent levels of the measurement model at 81%, 9%, and 9%, respectively (see Figure 5.6). 137‘ .22 -V Using team discussion Interaction 5 I 5 1.88—> Using the Internet for searching for information Interaction 6 l 6 1.64—> Using Web-Talk Interaction 7 I 7 176—5 Asking Instructor l8 Interaction 8 ; I I .28. t= 2.29 Interaction (I) 1.00 Goodness of Fit Statistics: Chi-Square with 2 Degree of Freedom = 0.59 (p = 0.74) RMSEA = 0.00, p_value for Test of Close Fit (RMSEA < 0.05) = 0.76 Goodness of Fit Index (GFI) = 0.99 Figure 5.5 Confirmatory factor analysis of interaction (1): Measurement model 2 .06—’ Using team discussion Interaction 5 l 5 239-. Using the Internet for searching for information Interaction 6 I 6 49. t= 3.80. 81% ' . Interaction 2.48 -h Using Web-Talk Interaction 7 I 7 Goodness of Fit Statistics: Chi—Square with 2 Degree of Freedom = 2.96 (p = 0.23) RMSEA = 0.13, p_value for Test of Close Fit (RMSEA < 0.05) = 0.25 Goodness of Fit Index (GFI) = 0.94 Figure 5.6 Confirmatory factor analysis of interaction (I): Final measurement model (3) 138 Table 5.9 Reliability Report of the Interaction (I) Final Measurement Model (3) Reliability Statistics Crobach’s N of | Alpha Items | .574 3 | Table 5.10 Reliability Report of the Interaction (1) Final Measurement Model: If Each Item Is Deleted from the Model Item-Total Statistics Scale Scale Corrected Cronbach’s Mean if Variance if Item-Total Alpha if Item Item Correlation Item Deleted Deleted Deleted I5 6.226 4.580 .358 .551 I6 6.129 2.782 .428 .405 I7 6.677 2.759 .425 .412 Thinking Behavior (1) Thinking behavior is the most controversial issue to be discussed in this analysis since, to date, there is no standard approach specifically used for measuring reflective thinking. Due to the unobservable nature of this construct, a specific, operational definition is needed. The similar concept of “critical thinking” is often mentioned in the existing literature as the way to assess thinking behavior and ability. As defined in chapters one and two, critical thinking involves, as part of its purposeful nature, thinking 139’ contents, while reflective thinking refers to the process of thinking. Therefore, in this study, thinking behavior is designed to involve both dimensions: process and content. Also considered is the level of thinking, which is an indicator of differences in thinking abilities and is included in the measurement model. To identify thinking behavior (T), sub-scale items are set for each measurement of reflective thinking behavior (R), critical thinking behavior (C), and level of analytical thinking (L). The questions for the sub-scale items are designed with the assumption that students who perform more reflective thinking behavior also conduct more critical thinking behavior, have a higher level of analytical thinking and will have a higher reflective practice behavior (RP). Unfortunately, due to the complexity of the concept, it is anticipated that the number of sub-scale items necessary to make this measurement model reliable is about 40 or higher, especially with the limitations caused by having a small population. However, an effort to measure this construct is initiated in this study. There are six sub-scale items that are designed for each of the indicators of the variable. These sub-scale items are used for the preliminary test of the RP dimension and the items help to determine whether or not each dimension relates to the RP concept and whether each dimension can be used as an indicator of the RP measurement. But they are not anticipated to make a reliable measurement model for the variable. Even though six subscale items are set for each RP dimension, there are only two items for each one designed using an (Likert) interval scale. Therefore, only six sub-scale items can be used for the RP measurement model and, after a face validity audit, only five of them are verified for the model (L2 was eliminated in the face validity audit). 140 As expected, the reliability of this measurement model is very low and the statistics show a negative covariance among the items (see measurement model 1, Tables 5.11 and 5.12). The sub-scale items of critical thinking behavior (C) are separated from the reflective thinking behavior and level of analytical sub-scales by their negative covariance. While a good measurement model is not supposed to contain a negative value of correlation among variables, the negative value of correlation among the sub-scale items indicates that the sub-scale item does not belong to the component and that the sub- scale items should be eliminated from the measurement model (see measurement model 2, Tables 5.13 and 5.14). However, when we consider this finding, along with the pre- defined operational definitions, it is possible that these two separated components can be used as indicators and it is not necessary that they correlate with each other. Table 5.11 Reliability Report of Thinking Behavior (1) Measurement Model I Reliability Reliability Statistics Cronbach‘s Alpha N of Items .144 5 141 Table 5.12 Reliability Report of Thinking Behavior ( 1) Measurement Model: If Each Item Is Deleted from the Model Item-Total Statistics Scale Corrected Cronbach's Scale Mean if Variance if Item-Total Alpha if Item Item Deleted Item Deleted Correlation Deleted C 1 13.35 5.970 .304 -.0943 C 2 14.03 7.832 -.217 .458 L 1 13.26 5.798 .309 -.1143 R 1 14.42 6.185 .055 .127 R 2 14.23 5.914 .076 .101 a. The value is negative due to a negative average covariance among items. This violates reliability model assumptions. You may want to check item codings. Table 5.13 Reliability Report of Thinking Ability (7) Measurement Model 2 Reliability Statistics Cronbach's Alpha N of Items .479 3 Table 5.14 Reliability Report of Thinking Ability (7) Measurement Model 2: If Each Item Is Deleted from the Model Item-Total Statistics Scale Corrected Cronbach's Scale Mean if Variance if Item-Total Alpha if Item Item Deleted Item Deleted Correlation Deleted L 1 6.00 3.867 .405 .251 R 1 7.16 3.740 .196 .564 R 2 6.97 3.032 .334 .316 142~ To validate the components for the thinking behavior measurement, an exploratory factor analysis is conducted (see Table 5.15). The analysis indicates that two components exist within the conceptual model. The first component contains sub-scale items of reflective thinking behavior (R) and a level of analytical thinking (L). The second component contains the sub-scale items of critical thinking behavior (C). This finding implies that R and L may be the same conceptual component, and confirms that the concept of C is different than that of R or L. In other words, two dimensions exist within the concept. According to pre-defmed definitions, and these analyses, one assumes that reflective practice behavior (RP) consists of two main dimensions, process and content. (This finding is further discussed in chapter 6.) Table 5.15 Factor Analysis: Rotated Component Matrix of Thinking Behavior Measurement Model a Rotated Component Matrix Component R 2 .753 L 1 .671 R 1 .633 C 1 .824 C 2 -.428 .726 Extraction Method: Principal Component Analysis. Rotation Method: Varimax with Kaiser Normalization. a. Rotation converged in 3 iterations 143 Although the exploratory factor analysis of the reflective thinking behavior sub- scales indicates a low reliability level, there is a possibility that reliability can be improved by removing the sub-scale R1 from the model. However, confirmatory analysis of the measurement model does not assure the construct-related validity of the model (see Figure 5.7). While the reliability of the measurement model is considered to be problematic, there is another alternative for the measurement. As mentioned earlier, it is possible that the RP behavior construct contains both dimensions of reflective and critical thinking, which refer to its process and purposeful thinking contents. Therefore, these two dimensions may be valid when they are included in the measurement model and they do not have to correlate with each other (see Figure 5.8). ' ’ Reflective ‘ '1 ‘. ' ‘ ' . 12:-seep Thinking 1., 1.44,.t=1_.50,,14% . R2 ' “ " Thinking Behavior T ' ' Level of 12.33.... Analytical 2' _ _ * i» Thinking . ~ . ' _ ' L1 . 1.44.3150; 14% a "" , " 44W "n; ' Chr-Squarewith1DegreeofFreedom 7065" RMSEA“= 4152 Goodness ofFitIndex(GFI) 052 ._ Figure 5. 7. Confirmatory factor analysis of thinking behavior (T). Measurement model 1 144 i Reflective '7‘ I Thinkan R 1 1.52, t= 5.38, 76% Reflective Reflective -1.74 Thinking Thinking . .00 .74—N R 2 Behavior RT Level of 1.41, t= 2.26, 73% -.39 ~74-> Analytical Critical Thinking _ . L1 Thlnklng , 00 Behavior ' _ CT Critical Thinkin 1.96, t= 6.85, 88% .50 —> C 2 g Goodness of Fit Statistics: Chi- Square with 3 Degree of Freedom= 1. 34 (p= 0.72) RMSEA= CI. 00 Goodness of Fit Index (GFI)- - 0.98 Figure 5. 8. Confirmatory factor analysis of thinking behavior (T): Measurement model 2 Using confirmatory factor analysis, a preliminary test of the assumption that reflective practice behavior consists of two main dimensions is carried out and the results are significant (seeFigure 5.8). The sub-scale items for reflective thinking behavior become significant when critical thinking behavior is included in the model. However, there is a negative multiple-colinearity between the reflective thinking behavior subscale R1 and the level of analytical thinking sub-scale. This colinearity indicates that it is not a good measurement model and that the validity of the instrument for these subscales needs to be improved. Measurement model 4 (Figure 5.8) also shows a significant relationship between reflective thinking behavior (RT) and critical thinking behavior (CT). The relationship is negatively valid when the influent loading of the prediction is set equally for the RP 145' measurement model. At this stage, this finding logically implies that, even though reflective practice behavior (RP) contains these two dimensions, they are not the same thing. To establish a good measurement model for RP, unexplained factors involved within the concept must be explored and tested. Working Performance Improvement (WP) The measurement model of “working performance improvement” in this study is considered very reliable and valid. The reliability level is reported at 0.924 (Cronbach’s Alpha value) or about 92 % reliable (see Table 5.16). The validity of the model is also confirmed by the principal component matrix analysis and confirmatory factor analysis. The model is tested by including all five sub-scale items of WP and one sub-scale item of learning performance in the analysis (LPS). The results of these analyses reject the validity of LP as a WP indicator, as well as indicate that WPlS is invalid in this measurement model (see Table 5.17). The exploratory factor analysis shows a very strong validity for each of the sub-scale items: working more cost effectively, WP 12,; working better, WP13; working faster, WP14; and getting more appreciation from employer, WPl 5. Getting fewer complaints about work (WP15) is tested and found to be invalid as an indicator for WP (see Figure 5.9). Table 5.16 Reliability Report of Working Performance Improvement(WP) Measurement Model Reliability Statistics Cronbach's Alpha N of Items | .924 4 | 146. Table 5.17 Factor Analysis: Component Matrix of Working Performance Improvement (WP) Measurement Model Component Matrixa Component 1 WP 14 .920 WP 16 .911 WP 12 .899 WP 13 .886 LP 5 WP 15 Extraction Method Principal Component Analysis a. 1 component extracted. Rotated Component Matrixa a. Only one component was extracted. The solution cannot be rotated. Working more '14 "i cost effectively WP 12 .68. 1= 6.02. 77% Working better 22 ‘" WP 13 Working Performance Improvement WP 41.1: 6.10, 79% . ‘ Working faster '95 " WP 14 .62.t=6.16,80% . Getting more - ' . - - appreciation '10 --> from employer WP 16 Goodness of Fit‘Statistics: Chi-Square with 2 Degree of Freedom = 2.79 (p = 0.25) RMSEA = 0.11, p_value for Test of Close Fit (RMSEA < 0.05) = 0.27, Goodness of Fit Index (GFI) = 0.95 Figure 5. 9. Confirmatory factor analysis of working performance improvement (WP) 147 Experience (E) “Experience” is a latent variable that is always indicated in years and always intensifies with “responsibilities” during the years of experience. In this study, the measurement model is initiated with concerns about its dimensions. While using “years” and “responsibilities” as indicators for measuring the variable are acceptable, these indicators do not state the quality of those experiences. “Achievements” is another method of measuring experience that is used widely for recruiting personnel in the work market. This achievement measurement concept seems to be a practical one for the concept of professional learning for the real world. However, many limitations are involved, such as, a small sample size and the diversity of the nature of FM practicing. These make the measurement concept impossible at this state. Thus, experience in this study is measured using years and responsibilities as its indicators. The measurement model of experience is provided using three sub-scale indicators: number of years in PM field (E1), number of years in current position (E2), and responsibilities in PM field (E3) (see Figure 5.10). Since experience is reported in numerical format, a reliability test is not conducted. The confirmatory factor analysis supports the use of all three sub-scale items as indicators for a experience measurement model. The influent factor loadings of all indicators are significant, especially for “years in the field”, and the influent factor loading is as high as 92%. This confirmation indicates that all the indicators are fit to the measurement model, even though there is an unexplained factor value that is high for the “responsibilities” sub-scale item used as an indicator. 148 Number of Years in FM '55 —' Field E 1 2.45, l= 7.08, 92% Numberof . , . , Experience Years in Current E ‘55 —> Position E 2 Responsibilities Related 10 FM 26.59, 1: 5.42, 68% 335.94—D‘ E 3 Goodness of Fit Statistics: Chi-Square with 1 Degree of Freedom = 2.28 (p = 0.13) RMSEA = 0.15, p_value for Test of Close Fit (RMSEA < 0.05) = 0.15 Goodness of Fit Index (GFI) = 0.95 Figure 5.10. Confirmatory factor analysis of experience (E) Previous Knowledge (K) The measurement model for “previous knowledge” contains only two main indicators: level of education (K1, scale 1 to 5) and the field of study (K2, related (or not) to PM practices). The statistics show that most participants already have previous knowledge at the college level, or higher (see Table 5.18). However, most of them do not have a direct background in FM. This finding makes sense because the FM discipline did not exist until the last 20 years and the graduate level of FM education has only been offered for the last ten years. Table 5.18 Descriptive Statistics of Indicators within the Previous Knowledge (D Measurement Model Descriptive Statistics K1 K2 Mean 3.16 .3? Std. Deviation 1.003 .486 N 31 31 149 Because we use only two indicators for the measurement model of previous knowledge and only one indicator is designed using a standard scale, the reliability of the measurement model cannot be reported. The “level of knowledge” sub-scales also contain an unexplained factor loading of about 40% for each indicator. when an equal prediction loading for both indicators is assigned (see Figure 5.1 1). The total factor loading of both indicators confirms only 50% of the existing factors involved in this concept. Level of .40 —> EduKCimn .37,t=2.66, 25% Previous Knowledge K 1.00 FM Related .40 —‘> FIBId .37, t: 2.66, 25% K2 Goodness of Fit Statistics: Chi-Square with 1 Degree of Freedom = 57.89 (p = 0.00) RMSEA = 1.38, Goodness of Fit Index (GFI) = 0.54 Figure 5.11. Confirmatory factor analysis of previous knowledge (K) Time Spent in Learning (TI) “Time spent in learning” is measured using three sub-scale items: time logged onto the course web pages per week (T11); number of hours spent in learning per week (TI 2); and number of times e-mail is checked during class session (T13). Because this variable is not a latent variable and is reported numerically, a reliability test was not conducted. The confirmatory factor analysis (see Figure 5.12) supports the validity of using all three sub-scales as indicators for the measurement. The descriptive statistics of this variable (see table 5.19) indicate that, on average, students logged onto the class web pages more 150' than once a day and they check their e-mail more than twice a day during class session. It is interesting to note that, on average, students spend less than nine hours a week on their online course, as compared to ten to 15 hours per week when taught in a “regular” class format. The analysis also shows a large standard deviation of number of hours spent in learning per week (T12) and the standard deviation indicates a difference in learning behavior amongst students. That is, while many students spend a lot more than nine hours a week for learning, there are also many students who spend fewer hours studying. Table 5.19 Descriptive Statistics of Indicators within the Time Spent in Learning (TI) Measurement Model Descriptive Statistics TI1 TI2 Tl3 Mean 3.61 1.52 3.74 Std. Deviation .667 1.458 1 .125 N 31 31 31 When the factor loading of the three indicators is set to be equal, the analysis of the data set reveals significant validity for all indicators. The heavy influent loading is indicated for T1] with a very low unexplained factor (7%), while a lot larger and unexplained factor loading is revealed for T12 (number of hours spent in learning per week) and T3 (times checking e-mail per day during class session). Recommendations for validity improvement of this measurement model are discussed in chapter six. 151 ' .07 -—I Times of Logging into the course web pages per week .35. t= 2.70, 63% TI 1 ' 4.37 -D Number of Hours Spent in Learning per week TI 2 Time Spent in Learning TI .35, t= 2.70, 3% 1‘ 1.48—-> Times of .35. t= 2.70, 8% checking e-mail during courses T13 Goodness of Fit Statistics: Chi—Square with 2 Degree of Freedom = 0.012 (p = 0.99) RMSEA = 0.00, Goodness of Fit Index (GFI) = 1.00 Figure 5.12. Confirmatory factor analysis of time spent in learning (TI) Partial correlations and multiple regression analyses in this section are conducted to verify relationships of the sub-scale constructs that may have indirect effects on RP learning behavior. In the completed RP conceptual model, sub-scale indicators are hidden and the relationships among these sub-scale constructs are barely identified. Thus, conducting partial correlation analyses is necessary and helpful for issuing recommendations for implementation of the findings in detail. However, it should be noted that significant correlations between variables do not confirm the “cause and effect” relationships between them. The partial correlations are only used to determine whether the relationships really exist and whether they tend to be positive or negative. The partial correlation analysis of the entire data-set in this study reveals a total of 51 significant correlations among the variables and sub-scale constructs involved in the Partial Correlations and Multiple Regression 152‘ RP conceptual model. The correlations consist of 39 positive relationships and 12 negative relationships (see Table 5.20). These logical correlations are helpful for online educators in improving their teaching methods. The positive correlations between sub— scale leaming techniques (TU) and learning performance (LP), or working performance improvement (WP), indicate a possibility that encouraging RP behavior and enhancing RP ability uses those learning techniques. Negative correlations also can be used to generate precautions in learning techniques (fiirther discussed in chapter six). Even though the analysis does not confirm the cause and effect relationship, the significances of the partial correlations are used as a preliminary guideline for generating a hypothesis and outlining a teaching framework (see Figure 5.13). 153‘ Table 5.20 Partial Correlations among Variables Pearson Significances of Partial Correlations between Variables Correlation P- Value Learning technique utilization (TU) & ,,, 1 Thinking behavior (T) 0408 0'023 Learning technique utilization (TU) & * 2 Reflective thinking behavior (R) 0389 0031 Learning technique utilization (TU) & H 3 Level of analytical thinking (L) 0472 0007 4 Learning technique utilization (TU) & 0.804 0000“ Interaction to people and information sources (I) 5 Reading (TU1) & Learning performance (LP) 0.409 0.002M 6 Reading (TU1) & Working performance (WP) 0.478 0.009" 7 Reading (TU1) & Experience (E) 0.489 0.005M 8 Reading (TU1) & Age (A) 0.463 0.009" 9 Parnelpatmg in group online discussmn (TU2) & 0.608 0000“ Interaction to people and mformatlon sources (1) Searching information from the Internet (TU3) & u 10 Thinking ability (T) 0.485 0.006 Searching information fi'om the Internet (TU3) & M 11 Reflective thinking behavior (R) 0'500 0004 Searching information from the Internet (TU 3) & n 12 Level of analytical thinking (L) 0599 0'000 13 Searching information from the Internet (TU3) & 0.775 0000" Interaction to people and mformatlon sources (1) Asking questions via Web Talk (TU4) & u 14 Interaction to people and information sources (I) 0814 0000 Asking instructor questions (TU5) & * 15 Reflective thinking behavior (R) 0377 0037 16 Asking instructor questions (TU5) & Experience (E) -0.395 0.028* 17 Asking instructor questions (TU5) & 0_ 478 0.006“ Interaction to people and information sources (I) Note: * Significant at 95% Confidence Interval ** Significant at 99% Confidence Interval 154 (table continues) Table 5.20 (continued) Partial Correlations among Variables Pearson Significances of Partial Correlations between Variables Correlation P- Value 18 Asking instructor questions (TU5) & Gender (G: 0452 0012,, Female) 19 Note-taking (TU6) & Thinking ability (T) 0.547 0.001 ** 20 age-taking (TU6) & Reflective thinking behaVlor 0.508 0.004” 21 Note-taking (TU6) & Level of analytical thinking (L) 0.511 0.003" 22 Analyzing other’s papers/readings (TU7) & _0.3 69 0.041,, Learning performance (LP) 23 Analyzmg other 5 papers/readings.(TU7) & 0.380 0035,, Interaction to people and information sources (1) Reviewing class objectives (TU8) & * 24 Interaction to people and information sources (I) 0'400 0'026 25 Reviewing class objectives (TU8) & Age (A) -0.378 0.036* 26 Self evaluation (TU10) & Experience (E) 0.381 0035* 27 Self evaluation (TUIO) & Time spent in learning (TI) -O.399 0026* 28 Review own papers (r013) & Thinking ability ‘ 0.562 0.001" 29 Revrew own papers (TU13) & Reflective thinking 0.361 0046* behav1or (R) Review own papers (TU13) & Level of analytical ” 30 thinking (L) 0.488 0.005 31 fiesww own papers (TU 13) & Learning performance -0. 462 0.009“ 32 gegiew own papers (TU13) & Time spent in leammg .0377 0.03 6* Learning pattern 1 (Patteml) & u 33 Interaction to people and information sources (I) 0702 0000 34 Learning pattern 2 G’attemZ) & Thinking ability (T) 0.444 0012* Learning pattern 2 (Pattem2) & Level of analytical M 35 thinking (L) 0.473 0.007 36 Learning pattern 3 (Pattem3) & Thinking ability (T) 0.603 0.000" Note: * Significant at 95% Confidence Interval ** Significant at 99% Confidence Interval 155 (table continues) Table 5.20 (continued) Partial Correlations among Variables Pearson Significances of Partial Correlations between Variables Correlation P- Value 37 Learning pattern 3 (Pattem3) & Reflective thinking 0.587 0.001,", behavror (R) 38 Learning pattern 3 (Pattem3) & Level of analytical 0.643 0.000“ thinking (L) Learning pattern 3 (Pattem3) & H 39 Interaction to people and information sources (I) 0'634 00% 40 Learmng pattern 4 (Pattem4) & Learning performance _0.3 69 0.041,, (LP) 41 Thinking ability (T) & Time spent in learning (TI) -0.498 0.004** 42 Critical thinking behavior (C) & Experience (E) -0.464 0.009“ Reflective thinking behavior (R) & Level of analytical * 43 thinking (L) 0.376 0.037 44 Reflective thinking behavror (R) & Time spent in -0.385 0.033,, learmng (TI) 45 Level of analytical thinking (L) & Time spent in $543 0.002“ learning (TI) 46 Leammg performance (LP) & Working performance 0.388 0037,, (WP) 47 Learning performance (LP) & Previous knowledge (K) 0.389 0.031“ 48 hegmmg performance (LP) & Time spent in learning 0.454 0.010" 49 glarhng performance (WP) & Time spent in leammg 0.434 0.019,, 50 Experience (E) & Age (A) 0.516 0.003" 51 Experience (E) & Gender (G: Female) -0.406 0.026 Note: * Significant at 95% Confident Interval ** Significant at 99% Confident Interval 156' Significance of Partial Correlations as a Preliminary RP Teaching Framework ONLINE LEARNING BEHAVIOR .408’ _ .389‘ REFLECTIVE THINKING .377‘ INSTRUCTOR (TU 5) / 0.599” BEHAVIOR (R) 0.5H‘I GATHERING INFO. FROM THE INTERNET INTERACTION TO PEOPLE & INFORMATION SOURCES (I) 1:7 40.4"~ TIME SPENT IN LEARNING (TI) LEVEL OF ANALYTICAL THINKING (L) CRITICAL THINKING (C) 14m ‘ A .488.“ 0. 562'" newt ' Note: * Significant at95 % Confident Interval ** Significant at 99 % Confident Interval ' Figure 5.13. Significances of partial correlations as preliminary RP teaching framework 157' Regression Analysis: WP versus T_AB R2L1, E, The regression equation is WP = 4.89 - 0.262 T_AB R2L1 - 0.191 E - 0.361 K + 2.16 I + 1.13 TI - 0.688 A - 1.44 G — 2.29 Patternl + 0.462 Pattern2 + 0.940 Pattern3 — 0.589 Pattern4 + 1.24 LP 28 cases used, 3 cases contain missing values Predictor Coef SE Coef T P Constant 4.893 4.465 1.10 0.290 T_AB R2L1 —0.2618 0.6896 -0.38 0.710 E -0.1915 0.4666 -0.41 0.687 K -0.3607 0.4965 —O.73 0.479 I 2.1610 0.9648 2.24 0.041 TI 1.1281 0.8822 1.28 0.220 A -0.6881 0.4240 —1.62 0.125 G -1.438 1.006 —1.43 0.173 Patternl —2.2896 0.8347 -2.74 0.015 Pattern2 0.4624 0.5535 0.84 0.417 Pattern3 0.9397 0.7583 1.24 0.234 Pattern4 ‘0.5885 0.6565 -0.90 0.384 LP 1.245 1.020 1.22 0.241 ' s =- 2.25396 R-Sq = 68.0% R—Sqiadj) = 42.5% Analysis of variance Source ' _ DE‘ 53 MS F P Regression v» 12 162.224- 13.519 2.66 0.038 Residual Error 15 76.205 :45'080 Total 27' 239.429 p , Seq'ss U] I ._o 5'5 n m U Patteml. jHIJk&HWJPJHIayJHj§f+y-' c o w w 50.594 Patteth' ‘ 3 . 1 66 PatternB'TV " 4 .296 Patterntl'? 5 14.308 LP . 7.561 Unusual Observations T AB , ' " ' » Obs 321.1 . up r11: ' SE Fit Residual St Resid 5 3:50 - *» 12. 046 2.269 * * -x 12 5-00 15.000 10.504 1.396 4.496 2.5411 R denotes an observation with a large standardized residual. X denotes an observation whose x value gives it large influence. Figure 5.14. Multiple Regression Analysis 158 After a face validity audit is conducted and measurement models of each variable are assured, the valid and most reliable subcscale indicators are identified and used in a multiple regression analysis. The main purpose of these multiple regression analyses is not for prediction of the RP behavior, but rather to indicate the levels of reliability, when each variable is involved within the RP conceptual model, and for issuing recommendations for a better setting of the model. Although the main objective of conducting these multiple regression analyses is not to predict performance, variables involved in the analyses must be pre-set as independent or dependent variables. Due to the concept of RP and for purposes of this research, working performance improvement (WP) is set as a dependent variable for the analysis and the rest of the variables within the model are tested as indicators of this variable (see Figure 5.14 for this analysis). Residual Plots for WP Ibrmal Probabiity Plot of the Residuab Residuals Versus the fitted Values 99 . 5.6 . m 2.5 O E i C. . .. o . 8 so 2 cc . . . - . - 5 a e . e L . O .0. 10 2.5 e O 1 -5.0 -5.0 -25 0.0 25 5.0 9 12 15 18 Residual Flmd Value H'stogram of the Rsiduab Residuals Versus the Order of the Data is‘cNAA/r fX/ll -..v w W W 2 4 6 a 10 1214 1618 2022 2426 2830 Residual Observation Order Residual <3 <3 Figure 5.15. Residual plots of working performance improvement (WP) Show normal distribution of the data set. 159' The multiple regression analysis shows the significance of the regression model: p_value = 0.038. (p_value < 0.05 is considered significant) The regression square is - reported at 68%, with an adjusted value of 42.5%. The numbers indicate strong relationships among the variables, as well as its reliability, if this model is used for performance prediction. However, as previously stated, this analysis is conducted using a small sample size. Although the analysis shows a normal distribution, the model should be tested with a large sample size using a random sample procedure to confirm it reliability. According to the significant partial correlations of learning techniques in learning fix pattern three (derived from a factor analysis, pattern matrix analysis), and significant sub-scales of the reflective thinking behavior (RT) indicator, learning pattern three is assigned to all models of best sub-set analysis for the WP dependent regression model. The results of this best sub-sets regression analysis are presented in Figure 5.16. The best fit model is the model that has the highest regression square value (R-Sq), but also the smallest Mallow C-P value and standard error (S). There are two possible alternatives for the best sub-set regression model. They are bolded in Figure 5.16. Both alternatives contain almost the same variables, learning pattern three (Pattern 3), previous knowledge (K), interaction with people and information sources (I), time spent in learning (TI), age (A), gender (G), and learning. One of the models includes learning performance (LP) and has a better R-Sq and less of a standard error than another model that has a better C-p value (less C-p value). TherefOre both models should be analyzed, along with results from other analyses, to finalize a recommended RP conceptual model. 160 i The results of the best sub-sets analysis when the model excludes critical thinking behavior (CT) turns out to be irrational. Even though the best subset model contains an acceptably high level of R-Sq, it indicates a lack of effects from reflective thinking behavior (RT) and experience (E). These may be caused by three things: 1) small sample size, 2) poor construct-related validity of the instrument, and 3) the reliability of the instrument. Validities and reliabilities of these models are further discussed in chapter six. Best Subsets Regression: WP versus T_AB R2L1, E, Response is WP The following variables are included in all models: Pattern3 28 cases used, 3 cases contain missing values P p p a a a t t t t t t T e- e e . r " . .. r.-r*r-. -. Mallows . B T n n n.L Vars R—Sq, R—Sq(adj) , . ‘C-p‘. s e E K I-I A G 1.2 4-P 1 1 23.4v 17.2'*1' 14.0. 2.7034 - x*"1. ‘2 .~ 1 23.1 17.0 . ;14.1- 2.7078 ’ -x_ ”vsz3259 - ,-24;5; 4:;1145 245828 .fijX3qu' .. -*z~” ‘“ 2' 32.3 23.8 1’5 11.8 2.5936' 'x . f- x 3 45.7 36.2 7.5--2.3735 .x x: n-‘xa; *_ 3 .42.4 32:4..‘ '»9.0 -2,4438 x.;;x, .xes 4 ‘55.0 44.8,' " 5.1“ 2.2072 x X‘X: ,x; ' - 4 .53.3 42.7 .[529 _2.2493 x x y‘xaw .x‘ w 5 “58.7“ »* 46.9 1.5.4; 2.1663 x X'XI 'x: """ xx? 5 58 6. 46.7. » 5.4 .2.1688 x x x: .x- g 6 63.1 soazgi '.‘5.3' 2.0979 3.: a 1*1“ fi_. '6 ’62}? --49;7 ..‘7 5.5t*2.1082 x xxx X x: ' x'~ _ 7 65.7 5122fi3"-€6;1; 2.0755‘ . x~x x xix 03:, -‘17 64.8 *50.0.. ._;6;53 2.1010 x x x x.x‘x‘: ,x. 8 66.5 49.7‘ “7.7 2.1071 x x‘x x x x 8 66.2 49.3 » 7.9: 2.1156 x x x;x x x*- x: .9 67.2 47.9— .p*9:4 -2.1458 >x-x x*x x x ”x . 9 66.8 47.3 -9.6 2.1580 'x x x x.x x ~x 10 ‘67.7 45.5 11.1, 2.1928 x x x x x x x 'X '10 67.7 45.5 ' 11.2 2.1946 x x x X'X x x x 11 68.0 42.5 13.0 2.2540 x x x x x x x x x Figure 5.16. Best subset regression analysis of working performance improvement (WP) 161 Structural Equation Modeling Structural Equation Modeling (SEM) analyses are conducted in this section to explore a best fit RP conceptual model and gain more understanding of the variables involved in alternative models. Four data sets are arranged for the alternative models: (1) as shows in Figure 5.17, the model includes all significant variables within the RP conceptual model and reflective thinking behavior (RT) but excludes critical thinking behavior (CT); (2) as shows in Figure 5.18, the model includes all significant variables within the RP conceptual model, reflective thinking behavior (RT) and critical thinking behavior (CT); (3) as shows in Figure 5.19 - 5.22, the model includes only each learning pattern, significant variables within the RP conceptual model, reflective thinking behavior (RT) but excludes critical thinking behavior (CT); and (4) in the last analysis, as shown in Figure 5.23, the model includes only most positively significant learning pattern (pattern 3), significant variables within the RP conceptual model, and critical reflective thinking behavior, which includes both reflective thinking behavior (RT) and critical thinking behavior (CT). These data sets are provided according to prior content analyses of the existing literature, factor analyses of sub-scale indicators and the partial correlations results. The diagrams show only significant coefficient paths for independent variables but the coefficient paths for all dependent variables are presented. Each SEM analysis is presented, along with standardized path coefficients and their t_values, to indicate significant effects among variables. While thick lines indicate significant path coefficients, thin lines show insignificant coefficient paths. The solid lines indicate positive effects, whereas dotted lines show negative effects. Each diagram indicates only coefficient paths related to the RP concept. 162 wExEE :8an 5923 Bees. :5 HwEEeB 05.8 3.55835 .8 38:. :ocmsco .maaosbm .26 5.3%: v. 33365. 3039a _ . Es .. . ., . ,. . .. . . .coEosaé . ,nu.r...a.._ P 9.663 =2:on 95 _ wrnfl~ OF 1.1' OOCNEOtOE ...... .... .. . . .. .. . .. .... -...-.\ ---r 9:32. . . - .. - m- ... ’ I _ C050 50:: — on I I .. . .. . . . , «33.8. suing/I. m m Scouoaxw _ I) . I . . .. . 3.16.6... .. a: I. gnu-......opv... 22:0“. "0 .0950 l— ovml .3880 8:68.83". .I ........../W..u...... j . H5... , . 3:63 ..I.. I l/ ., .. < 6?. 4 «3.3.8.9.: , Ink .. .. . «man. :9. . m . , ...-\- . . e Eozma _ .-./- E bxx . . . a; Lars: - cosazom .. . .. m Egan. _ - 95.5.: aunts-.2. . . . - o>=oo=om . . N Eugen. _ 3mm reduced. ......, _ 163 BeoE :3 3:22:22 HwEEflx 05:5 3:26.833 mo Boos: song—co 3:585 .36 23%.: 0: 028.265. 25.55 _ L>> . EoEo>o..aE. , . . .. 9 nu. 2.. .. ones—Eaton .v...d..u.. fit. ill-V 95:25 .. . . .. . abuses... 5.5% as: _ {...-...... .... ... 2 S 5.682... .8 A... . I..! .. . . . . ...... ..Eoucotoexw .5.-«8 : 3.5..- . .. .. 3...: _..a.. . ...... .. . a... . . . I 9 n u“ 363nm 3.55.236. A . 93:..qu 33...... I r . 2. 2.2 . . .. 31.6....» .....Nmfi- 8.. 1.8.8» «as? 4 .. .n Prawn". _ . my. . . . . . . ....... .6323 $3.. , . . .. . War-l- 95:...» . , .. .. ..I own". mo 02.0033— fi. . ...: 2.... .355 . ...... 1 . . . I N526... . _ . 4 . rhesus. _ 164 As stated in chapter three, each data set’s covariance matrix was tested initially against the pre-set conceptual model (see Figure 3.2). All of the data sets provide perfect fit of the models (p_value = 1.00). In contrast to the regression analysis, in order to be considered as a good fit index, a p_value must be >005. In this case, all the data sets indicate perfect fit because of the small sample size. However, each model reveals different significances for the coefficients paths (t_value >1 .96 is considered to be significant). Data set one (Figure 5.17) represents the assumption that students use all of the learning techniques available, but at different frequencies, and keep performing reflective thinking behavior (RT) without concentrating on the objectives of the thinking process and contents of thinking. The SEM analysis for this data set shows a positively significant path-effect of learning pattern three and reflective thinking behavior (RT), but students who prefer using learning techniques in learning patterns 1 and 4 may see a negative impact on their learning and working performances. The analysis does not show any significant cause and effect relationships among the reflective thinking behavior (RT) and learning and working performances (LP and WP). Only two clear cause and effect paths are revealed: positive impacts of interaction (1) and time spent in learning (TI) on students" working performance improvement (WP). Data set two (Figure 5.18) represents the learning behavior of students who use all the learning techniques and concentrate on the learning objectives and contents within their reflective thinking process. For this learning behavior, previous knowledge (K) has a significantly positive impact on their learning performance. Their age may also have an indirect effect on their learning performance since age seems to be a significant cause in enhancing critical reflective practice behavior (RP) and the positive coefficients path 165 ' between RP and leaming performances contains a very heavy loading. Interaction with people and information sources (I) is another factor that has a positive impact on working performance improvement. To determine whether each learning pattern can be used for supporting students’ RP ability, data set three is analyzed separately for each learning pattern (see Figures 5.19-5.22). The analyses of data set four reveal positive coefficient paths only in the analysis of learning pattern three. The significant paths of learning pattern three to reflective thinking behavior (RT) and working performance improvement (WP) imply that if we encourage the use of learning techniques within pattern three (searching for more information for learning fiom the Internet (TU3) and note-taking (TU6)), students may have higher reflective thinking behavior (RT) and higher working performance. However, encouraging learning pattern three may not have a direct effect on students’ learning performance. The analyses show that the independent variables involved in the model have different impacts on the dependent variables when different learning patterns are encouraged. This finding significantly contributes to the way teaching methods and learning environments for the online students are prepared. Another benefit of conducting these SEM analyses and doing a preliminary test of cause and effect relationships among variables is to provide a better understanding of the RP concept. For example, the SEM analysis of data set four shows significant direct effects from learning pattern three to RP behavior, learning performance (LP), and working performance improvement (WP), when the critical thinking behavior (CT) is included in the model. This finding implies that, beyond the reflective thinking process, the concept of RP consists of other dimensions of critical thinking such as thinking objectives and content. 166 _ 522$ ”wEEeB 25:0 5.23805 «0 A2508 eocmzvo 3.382% .36. 6.5%.: mac-“Mn“..m—uawu, 9.032265. 2.0395 .....— A. .5 9:53.. _: 2.3m. oE_._. _ .. .. . . £2: a... $65.9. 22:03.55 A... 55083:. — . .. oocaetotom ..... . . . , . u. 95:25 A. . Am: ooeotoaxw m .8... m... ...8.. Age—:0". "0. .5050 a. ...? _ . w0>._~0w_n.u0 DEF—m2. UC.>>¢...>.0~A 63:25 958.30 .o a» ”MAI“ .EO. . . anal... . .585. n3 ooemantom 9:53.. . s ”Na-urea... .8. __ .8 E 8323.. x .. 5.5.5.: 6:86.61 at . 963.583 .938 mE~2mc< 02qu undo I 95:34 “.2635 K D: 3.- u u .mmd w. ..ofiEAwE mc2m< ”tobacco 9:20qu - 9:53.. oo_>..ow ”m a. an.” n. and IV 52.83 E 20:33 92:2 ”9:563 “.8636 s. a $9 .. . "r :3sz 167 N 50:9. E 3.22.3 9.25.: 9:58. o:..:0 13068.35: ..0 .0005 cognac .....Boabm SNw. 0.5M: .82. :30 9:30.31 "9 Pp 2.0.9.8 5.338.: 30.0 05.5 9.56.3”. "N F ...P 168 m 5039. ”9:500. 05.:0 3.8.80.9... .0 .0005 :050000 35.03% .NNW 05%: Av: 0muo.30:v_. 03030.5 _ .E 9:500. :. 50.5 0:: ... ES 505020.55. 00:05:23... @5502. 3.0 n. .3... 02” n. .3. 8...". .... .3. . Am. 00:0...0axm _. «I- m. .8. . n... 00:05:83: 55500.. A2050". .0. 50.500 and n. .8088 .. _. . _ .. . m 8...: ms N A 9:502 :0 9.0.0. 0.02 5 3h /. ...I .0525 0... E0: 0020502. and u. . 0.0:. ..o. 9.20.00m ”n D... ...m. .0323 um“: 95.5,; 2.32.3. A @5500. 0200:020w "n 5030.“. $8 169 «02003000.. 00005000.. 00000.5 000050.000 00.5004 E 332.0 0.5:...» «2.80.0 v 50:00 E5500. 00:00 .000.mm£05 ..0 .0005 0000000 .ansbm .NNw. 0.5%.: 00:03.05 =0m 5.. 3.. 00.00.0005 000.0 00.00 ”a 3... "v 5080... 170 5.0.0.... .0350 5.3 .0005 02005050 m 5000.. ”m5500. 05.00 300600.05. ..0 .0005 00000.00 .00505w .MNW 05M¢~ 0.. 0200.300: 00030.... — «00505005. 0000008000 0:002... .. 0.0.0.0. . 5...... ...... 00.500. 0. «0000 0.05 . x Ly.wlumn\..l.1' mi. .. .. , . 1.4! . .\. ... 50202:. 0.3.0 urge. ._ . . . . .. . . .0. 00000003 . 0.. .. 00005.0...00 @5500. .2050". ”0. 000006 .. 00.0 .1 0030.0. .<. ou< 3.0 u 0 .0050 @5500. 00 @020. 0.02 ”0 D... 00 ..0.>000m 00.0.0.5 208000 .3000 .1 .0505. 05 50.. 00000020. 0.05 00.. 00.00000w ”n PF %.n"~.gm . «0.00.»... 0. o. . , @5500. 02000.0...0m 0.0.00.1. . m .....m "m 5.020.... 171 The SEM analysis of data set four indicates that encouraging students to search for more information from the Internet and taking notes can be a good teaching method for online students to improve their RP abilities, as well as improve learning and working performances. This finding is supported by the positive, significant coefficient paths among learning pattern three, critical reflective thinking behavior (RP), learning performance (LP) and working performance improvement (WP). The teaching method in Figure 5.23 encourages students to use learning techniques in learning pattern three and to promote the importance of learning objectives, in addition to encouraging students to integrate learning objectives into their learning content. This learning method is an alternative learning model that will be compared with the preliminary RP learning fiamework and analyzed, along with all results from both qualitative and quantitative analyses, to generate final recommendations for further testing of the RP conceptual model. In summary, this chapter presents the quantitative data analyses and results for this study. It contains three sections of analyses: factor analyses, partial correlation and multiple regression analyses, and structural equation modeling. The results from these three sections provide quantitative evidence for a preliminary test of the RP learning conceptual framework, as well as indicate alternative teaching methods to support RP learning behavior. The quantitative results in this chapter will be further discussed in the next chapter, along with qualitative results from chapter four and theoretical guidelines from the literature review, to get final recommendations for fiiture studies. 172 ‘ CHAPTER SIX Discussion and Conclusions This chapter discusses the results and findings from both the qualitative and quantitative analyses in order to apply the research findings and adjust the RP conceptual model. Major findings from the analyses are restated with recommendations concerning the implications, as well as possibilities for further studies. This chapter is divided into three main sections discussing the research questions and objectives of the study: 1) the nature of reflective practice, 2) effects on reflective practice behavior, and 3) teaching techniques for reflective practice learning. Nature of Reflective Practice There are several major findings in this study concerning the nature of reflective practice. The first is identification of the elements of RP within the improvement process. The analyses support the reflective practice definition initially expressed by John Dewey (1933) and Schon (1983) that reflective practice is an improvement process that involves critical analysis of everyday working practices in order to improve competences and to promote professional development (see Table 2.1). However, the RP process can be significantly defined as an improvement process only when reflective practice behavior is included in dimensions related to critical thinking. These include objectively purposeful 173' contents and value-related aspects within the decision making process. This means that students may not show any improvement in their professional practice and working and learning performances if they are encouraged to think reflectively unless they concentrate on the objective of thinking and the values involved in the judgment necessary for effective learning and working. Thus, as the critical thinking concept differs from the concept of reflective thinking, the different causal effects of reflective practice behavior with and without the dimensions of critical thinking to learning and working performance improvement were discovered (see Figure 5.23). However, at this stage, the study shows that the effects of individual attributes and online learning behaviors on both learning and working performances, are not significantly obvious among students who haVe different learning patterns. While students who use learning techniques in learning pattern three more frequently than other learning patterns tend to show improvement in both their learning and working performance, the analyses show that using learning techniques in pattern one tends to lower students’ working performance improvement. Students who use learning techniques in pattern four also tend to demonstrate lower learning performance. Therefore, further studies about the effects of different learning patterns on RP behavior and working performance improvement are recommended (see recommendations for further studies at the end of this chapter). Another finding about the RP concept that needs to be confirmed is whether RP behavior is an individual instinct or must be cultivated? According to the finding that critical RP behavior is involved with objectives and values within its content, even if every student has this ability but there are no objective or standardized values for them to 174' use as a guideline for thinking. As such, it can be anticipated that the RP process will not be generated. To provide more insight into the nature of RP behavior, experimental research among treatment groups that do and do not use learning objectives as a guideline for learning, as well as defining a specific learning culture to be used as a standardized value, may be necessary. The nature of reflective practice as a process of learning from experience (E) is not significantly validated in this study due to the insignificant effects of experience on endogenous variables (further discussed in the next section). The result seems opposite to the assumption that experience is the major factor involved in the RP process, as well a Kolb’s fundamental ideas of learning from experience. While this result disaffiliates the assumption that experience is an important factor in the RP improvement process, discarding this variable from the RP model is not considered. In this case, qualitative analysis is essential to explain the importance of experience in the RP process. Due to the fact that F M is a relatively new professional field and there are not many students have direct experience in the field, participants in the study reports there experience not to relate to PM field. It is possible that participants are working in other fields and not reporting their working performance improvement because their experience does not relate to FM. Therefore, to provide more insight concerning this factor, experience should be a control variable in future studies to be more specific about their experiences, along with some independent variables discussed in following section. 175' Influences of Individual Attributes on Reflective Practice Behavior Two parts of quantitative analysis that imply the influences of individual attributes to reflective practice behavior: correlation analysis and SEM analysis. However, correlation analysis cannot assure causal relationships among the variables. It only shows a relationship between the two variables. It also indicates a possibility that the relationship is positive or negative at a significant level, while SEM analyses show both direct and indirect effects of individual attributes and online line learning behavior (exogenous) on the endogenous variables, RP behavior, learning performance, and working performance improvement. Therefore, the effects on RP will be discussed mainly based upon the SEM analyses. Correlation analysis and results from the qualitative data are used as additional evidence, as needed. Among all the SEM analyses conducted, the most significant alternative is the reflective practice process of learning pattern three when it includes the dimensions of critical thinking. Thus, the following discussion and interpretations are based on the SEM model 3 (see Figure 5.23). Influence of Experience to RP Surprisingly, experience is not significantly supported by the analysis results as an important aspect of the RP process nor are its effects on RP behavior or working performance improvement. As reflective practice is defined as an improvement cycle that relies on experience (Van Manen, 19771; Boyd and F ales, 1983; Irnel, 1992; and Atkins and Murphy, 1995), the results from all quantitative analyses do not significantly confirm that students who have more experience achieve more working performance improvement. Therefore, experience may not be a major factor in online professional 176' learning or in developing RP behavior. Another interpretation of this finding is that experience in a different subject matter does not influence an improvement in other specific situations that are not related to that experience, but may contribute to working performance improvement in its specific subject matter. Relationship between Previous Knowledge and RP It is interesting to note that previous knowledge (K) is not significantly validated as a major factor influencing RP behavior for students who mainly use learning techniques in pattern three, but there is a significantly direct effect of previous knowledge (K) on learning performance (LP). Previous knowledge clearly has a positive impact on learning performance, regardless of what learning pattern students use. However, it should be noted that, in this study, measurement of learning performance depends on self evaluation because the face validity audit indicated more than 20% of grade report data were missing. Therefore, the measurement model of learning performance contains a heavy loading of unexplainedfactors. (Factor analysis of LP was not conducted due to an insufficient number of sub-scale items). Influence of Gender to RP Pagezl77 [O]From quasi-statistical analysis and SEM analysis with learning pattern three, the result that female students tend to have more reflective behavior if they use the learning techniques in pattern three more frequently than other learning patterns is confirmed. Their learning and working performance improvement also tends to increase if they use 177 these learning techniques and concentrate on learning objectives within each lesson (indirect effects of RP to LP and WP are significant). However, there is concern about the implications of this finding. According to the findings from SEM, without critical thinking dimensions (see Figures 5.20 to 5.22), female students may not have more RP behavior. Also, there is a negative impact on their working performance improvement. Therefore, to equalize the RP learning environments for both genders, clear learning objectives should be provided along with the learning lessons. Influence of Age to RP Some evidence from the quantitative analyses indicate that older students tend to have more reflective practice behavior and tend to have better learning and working performances (indirect effects of age on endogenous variables). However, without the dimensions of critical thinking in the RP process, the effect of the age difference is not significant. Conversely, older students tend to gain a lower level of working performance improvement. It should be noted that even though the extremely significant correlation between age and experience is revealed, age has a significant effect on RP, whereas experience does not significantly influence RP. This can be interpreted in two ways: Pagezl78 [011) other variables may accompany age as lurking variables, or 2) non-FM related experience may cause the insignificant relationship of the variables. Additionally, the significance of the direct and indirect effects of individual attributes on RP, such as age, experience, and gender differences also correlate to a student’s preference for learning techniques. The older the students are, the more reading 178‘ (TU1) they do while doing less reviewing of objectives (TU8). While students with more experience tend to read (TU1) more and perform more self evaluation (TUlO), the more experience they have, the less they ask the instructor for advice (TU5). Moreover, female students tend to connect with their instructor more than male students. These significant correlations, as well as the positive effects of learning techniques discussed in the following section, should be kept in mind when considering the proper teaching techniques for encouraging RP learning behavior in online situations. Teaching Techniques for RP Online Learning With regard to the effects of the involved variables within the RP conceptual framework on RP, the most striking seem to be the direct effect of learning pattern three (pattern 3) on critical reflective thinking behavior (RP) and its indirect effects on learning performance (LP) and working performance improvement (WP). As discussed previously, using learning techniques in learning pattern three such as note-taking and searching for more information on the Internet, may enhance a student’s RP behavior, as well as learning performance and working performance improvement. Therefore, encouraging students to take notes and use the Internet for searching for information is a good idea if the learning objectives are stated clearly and guidelines for a good learning culture are implemented properly. The availability of various technologies also makes using many teaching techniques possible for encouraging RP online learning. In addition to learning pattern three and the learning techniques discussed in chapter two, web-talk availability and e- mail group technology are possibilities for use in encouraging sharing of experiences and 179 knowledge among the learning participants. Writing reflective notes on learning and posting them on web-talk or through e-mail to group members can be used to help students rethink the learning content, gain more understanding of the lesson and share their experiences and knowledge. These techniques are ways to stimulate interaction among the online learning participants. Even though the SEM analyses do not indicate a significant effect of interaction (1) on RP behavior, it may be a way to enhance working performance improvement (WP), as a significant effect on WP is revealed when students use mixed learning techniques. Time spent in learning (TI) is another factor that seems to be controversial for its effect on RP. Its positively significant effects are confirmed on working performance when students use learning techniques in pattern three more than other patterns, but it becomes a negative effect on reflective practice behavior when learning objectives and the values involved in the process are not stated clearly and applied to student behavior (see Figures 5.19 and 5.22). Negative correlations between time spent in learning and doing a self evaluation and time spent in learning and critical reflective thinking behavior are also significant. The interpretation of these findings is that if students do a self evaluation and have more critical RP behavior, they may spend less time in their learning. The may be due to the fact that they recognize that the learning objectives, as well as their own objectives for learning, are to be used as a guideline for self evaluation. However, according to qualitative analyses, encouraging self evaluation among students is not possible without a pre-set standard of evaluation and clear learning objectives. In this case, making the online learning experience attractive because it requires less time commitment is possible if learning objectives are provided clearly and the learning 180‘ culture is standardized for self evaluation. Therefore, providing clear learning objectives and a set of standards for a RP learning culture should be another recommendation for application of RP teaching methods. Recommendations for Further Studies In summary, there are two main recommendations for further studies subsequent to this research. This is a need for reliability and validity improvement of the instrument and for additional tests of the critical RP online learning model. Reliability and Validity Improvement of the Instrument Validity and reliability of measurement scales for each of the variables involved in the RP learning model are tested in this study and some measurement models show very high reliability levels. For example, the measurement model for working performance improvement (WP) contains as high as 0.924 of Cronbach’s Alpha value (about 92%). Generally, an acceptable reliability level that can be found in any peer- reviewed journal is 20.400. Therefore, the reliability of working performance improvement (WP) is considered very high but the reliability level of critical reflective thinking behavior and its sub-scale items are not reported as being at a satisfactory level. The reliability and validity of the learning performance (LP) measurement scale needs to be improved as self evaluation is used in this study and there is a lot of missing data due to Page: 181 [0]the fact that not all participants completed each class on the program. There are also very heavy unexplained factor loadings on the LP measurement. The factor loading 181‘ indicates the possibility of other lurking variables involved within the learning performance measurement. These lurking variables may relate to the assessment standards of the LP; this needs to be clarified. Therefore, the reliability and validity of each measurement scale should be improved by the testing of more sub-scale indicators and increasing the number of sub- scale items for each model. For an extremely latent variable such as critical reflective practice behavior, the number of sub-scale items should be especially higher than in this study in order to gain an acceptable reliability level. Tests of Critical RP Online Learning Model Even though the goodness of fit statistics of the SEM models indicate that the model’s fits are extremely good, goodness of fit may be an effect of small sample size. Therefore, to confirm casual effects among the variables, the model should be retested using a larger sample size. Generally, an acceptable sample size fOr SEM analysis is >200 or varies from five to twenty times bigger than the number of variables involved in the model. The recommended model for further testing is presented in Figure 6.1. Except for experience (E), individual attributes are removed from the model recommended for firrther studies as these are constant variables (cannot be improved). Alternative teaching and learning approaches should be added to the model and tested for their causal effects on RP. Conducting the SEM analyses for each learning approach will allow educators to gain more insight into RP and provide a proper learning environment for encouraging RP behavior. 182~ As the SEM analyses in this study are conducted on a small sample size, the significance of the conceptual model and testing hypotheses of causal relationships could not be confirmed and were not formally developed. It is recommended that conducting SEM and hypothesis testing using a larger sample size of professional online learners is needed to provide more insights. Fifteen recommended hypotheses for each learning approach are indicated in Figure 6.1. Conducting qualitative and quantitative studies in detail for each learning approach is also recommended. The studies may need to be detailed for each learning and teaching approach in order to find out the possibilities for applying the concept of RP to the learning approaches. Alternative Teaching and Learning Approaches . Self-directed Learning 5 Critical Reflective Thinking Behavior RP Learning Performance H15 [ Interaction (I) [ Time Spent in Learning (TI) 6 H/ | Experience (E) Figure 6.]. Adjusted alternative models with critical thinking dimensions for further Studies Working Performance Improvement WP 183 In conclusion, significant causal effects are revealed among the variables within the RP conceptual framework in this study. These cause and effect relationships bring more understanding to the RP learning process and its nature to the researcher, as well as indicate the possibilities of creating proper online learning environments that encourage RP learning behavior for online learners. However, due to the many limitations discussed in this study, there is a need for further studies to provide more insight into RP online learning. 184' APPENDICES APPENDIX A: CONSENT LETTER Reflective practice in action: A framework for professional online learning March 22, 2004 - Dear Participant: We would like to ask you for help in conducting a research project designed to aid in improving delivery of effective online education in facility management. The title of the research project is "Reflective practice in action: A framework for professional online learning". Nijsiree Waeochan, a graduate student at the Michigan State University and Susan Mireley, Program Director of the MSU Online Facility Management Program, are conducting the study. Its purpose is to identify essential factors that assist professionals in online learning, so they can be incorporated into online learning experiences and activities. All information that you share with us will be confidential and your privacy will be maintained throughout the study. . Your participation will involve four tasks, which will consist of completing one online questionnaire and participating in three web talk discussions (at times of your choice during given periods). The questionnaire can be completed in approximately 30 minutes. This link will take you to the questionnaire: (LINK). The web site address is: www.msg.edu/~waeochan/Professional%200nline.htm . The online discussion web site can be accessed at this site: http://hed-frn.vu.m_su.edu_/_c_:gi; bin/webgllg/system . The discussion format is the same that you used in Web Talk. Discussion sessions will be held in three rounds, with different questions discussed in each round. The topics discussed will be developed from your responses to the online questionnaire; they will be posted at the beginning of each round. Please post responses during the following dates: Round One from March 22, 2004 to March 31, 2004 Round Two from April 1, 2004 to April 10, 2004 and Round Three from April 11, 2004 to April 22, 2004 During each round, please visit the site as many times as you like, adding your responses or commenting upon other's comments. We hope you will remain involved in all three rounds. Please log into the discussion web site using your pilot ID. If you can't remember your pilot ID, please let us know by replying to this e-mail and completing the information requested in the attached form. You can also using this form to let us 185’ know whether you want to participate in this study or inform us of your updated e-mail address to enable us to communicate in the future. You will not need your pilot ID to enter the questionnaire web site. If you have problems accessing either web site, please contact us at: waeochan@msu.edu . There is minimal risk involved in participating in the study. Your responses will not affect your grades or your involvement in any future classes. As indicated, your individual responses will be kept confidential and neither your name nor identity will be referenced in any findings reported from the study. Your privacy will be protected to the maximum extent allowable. Finally, please recognize that your participation is completely voluntary. We will consider your completion of online questionnaire as your consent to participate in the study. We hope you will decide to become involved. Your efforts will help us improve the MSU online educational experience and significantly aid all firture participants involved in our programs. If you have any questions regarding your participation or wish to make suggestions to us, please contact: Researcher: Nij siree Waeochan PhD Candidate Facility Management Pro gram Department of Human Environment and Design, MSU E-mail: waeochan@_n_rsu. edu Phone: (517) 353- 6857 Research Investigator: Dr. Susan Mireley Department of Human Environment and Design, MSU E-mail: smirelev@msu.edu Phone: (517) 355-7687 ’ If you have any questions about being a subject of this research, please contact: Chair, UCRIHS: University Committee on Research Involving Human Subjects Dr. Peter Vasilenko University Committee on Research Involving Human Subjects, MSU E-mail: ucrihs@msu.edu Phone: (517) 355-2180 Sincerely, Nij siree Waeochan Susan Mireley 186' APPENDIX B PROFESSIONAL ONLINE LEARNING QUESTIONNAIRE INSTRUCTION: This questionnaire consists of five main sections. Questions in each section are developed to help us learn more about online learning and help you learn better in the future. Please feel free to give us your forthright responses and be assured that your responses will not have any effect on your grades or on any course(s) that you may take in the future. It will take from 30 minutes up to 45 minutes to finish this questionnaire. Please read and follow the instructions for each section carefully and submit your answers by clicking the "SUBMIT" button at the end of this questionnaire. Submission of your answers will be your consent to participate in this research. SECTION 1 Please read each question carefully and choose only one answer that would best describe your response to each situation. - 1.1 How would you handle the situation if your computer (or software) breaks down at a critical point near the deadline of a team assignment? r‘ I would get my computer fixed immediately and use another system in the meantime to meet the deadline.‘ r‘ I would contact my teammates via another device to let them know the situation and ask for help from them to meet the deadline. r‘ I would contact my teammates via another device to let them know the situation, while trying to get my computer fixed so that I could finish my part of the assignment and meet the deadline. r‘ I would expect that my teammates and the instructor would understand. I would solve the problem later. t‘ I would contact both my teammates and the instructor via another device, let them know the situation, while trying to get my computer fixed so that I could finish my part of the assignment and meet the deadline. 187’ 1.2 How did you learn and gain more understanding of the lessons from the online class(es)? t‘ I read the lectures and always understood it: there was no need to think about the lessons again. (‘ When I read the lectures, I always thought about the context of the issues, compared it with my experiences and understood it thoroughly. r“ Afler I read the lessons by myself and analyzed them on my own, class discussions helped me to clarify the lessons. r‘ I understood the lessons and learned more after I read the lectures a couple times and discussed them with others. r‘ I always gained more understanding after reading and class discussions, but there were always more issues about the lessons that I never completely understood. Therefore, I had to search for more information about the topics. 1.3 How would you handle the situation as the leader of a team assignment, if you found that one of your teammates uploaded exact contents from a paper, which you found on the Internet and he presented it as his work without any citation? r‘ I would respect him as an individual and put the contents into our assignment. However, I would add the citation into the final paper myself. r‘ I would respect the author of the original paper, contact the teammate and have him to add the citation. r: I would ask all teammates to submit citations for each part of their works without identifying the teammate who copied the paper from the Internet so that we could continue our work smoothly. r“ I would contact other teammates to let them know what that teammate did and ask them whether or not we should put the contents to our final paper. r‘ Plagiarism is unacceptable for me. I would report this matter to the instructor and ask to have the teammate who copied the paper from the Internet removed from our team. 1.4 How do you describe your online class discussion process? (‘ I always read the discussion topic and thought about what I would discuss before each session so that I could understand what they are going to talk about. r“ I always prepared what I would discuss and looked at classmates' messages over the Web Talk before each discussion session so that I could understand what they were thinking about the topic. 188’ t‘ I often had no time to prepare for each discussion session but always tried to read all lectures and post something from what I knew on the Web Talk. When I joined the discussion session, I gained more understanding about the topic. (‘ I always prepared for each discussion in advance. When we got into the discussion, we always exchanged a lot of ideas and learned more form each other. r“ Even though I prepared for each discussion session and exchanged lot of ideas, after each session ended, I always found something more to discuss. However, it was often too late. 1.5 What would you do if it were a couple weeks before an assignment due date and you were the team leader for a group assignment, where your teammates could not agree on the final solutions for a case study and team protocol seemed useless? (‘ I would follow my instincts and decide the final solution on my own. r‘ I would ask my teammates the reasons for their arguments and then make a decision based on the most logical reason. I would accept the solution from the most experienced teammate. r‘ I would ask my teammates the reasons for their arguments, prepare pros & cons of each argument, and then make the final solution based on that. r: I would choose two acceptable solutions from the options and let the teammates choose one. 1.6 Which one of the following statements is the best description for your online examination? (‘ I never took an online examination. r‘ While I was taking the exam, I could see answers as text or pictures that I had seen in the lectures. F To answer each question, I always analyzed all the given information by comparing it with the contents in lectures, my experiences, and my logic related to the question. P To find the right answer, I would eliminate each answer logically one-by-one and compare it with the contents in lectures. P To answer each question, I would try to think about what I learned from the lectures and reading assignments about the question. (‘ To find the right answer, I would logically analyze all the given information and compare it with what I learned from the class. 189' 1.7 What do you prefer— studying alone or in a team? I prefer studying alone and I will join group studies only when it is required. There is no difference. My teammates never contribute to my study and studying alone is so desperate. b There is no difference. I can work in both ways and get an acceptable job for each assignment. I enjoy participating in a team but I often do a better job when I work alone. I enjoy working alone but I found that I got a better job when working in a team. I enjoy both studying alone and in a team and always get a job well done. 1.8 If you got stuck while working on your individual assignment, what would you do? I would ask the instructor for clarification and more information. I would re-read all the lectures to understand more about the assignment. I would find more books and articles about the topic via the Internet. 333') I would contact my classmates who are working on the same topic and work together to get more information. '3 I would get more information from all resources that I could obtain and would ask the instructor and/or my teammates for their advice. SECTION 2 Please read the following statements carefully and indicate “True” (T) or “False” (F) for the contents in the statements. STATEMENT (‘ r‘ 1) 1 always have other thoughts related to the topics after joining class discussion, but then it is often too late. The session had already ended. r‘ (‘ 2) Most of the time I analyze information on my own to c0pe with my individual assignments. I rarely get help from others. r“ t‘ 3) I always re-think what I have already learned or done and find a better way to do my job. 190' 4) I find that sometimes ideas and solutions that I get from reading and chatting with my teammates are not exactly applicable to my real working situation. I have to learn from my own experience. 5) Often, ideas come to mind during conversations with others. 6) I prefer to have background music on while I’m concentrating, reading or studying. 7) Outside noise, music and other distractions tend to reduce my concentration. 8) If the room is absolutely quiet I have difficulties concentrating. 9) I can’t concentrate on my learning content in a room with low light. 10) When I learn something difficult I prefer to be in a room with direct sunlight or ample overhead lighting. 11) I always dress for cooler temperatures, even when I’m indoors. 12) If possible, I adjust the temperature control to a cooler setting while I read or study. 13) I find it hard to concentrate when I have to be in a formal environment- sitting upright at a desk or working table. 14) I often sit or he on the floor while I’m thinking, reading or concentrating. 15) I like to get up and log in to the course web page between 6am-8am or even earlier. 16) I prefer to have class discussion beginning after 9 am or during daytime, not in the afternoon or at night. 17) I like to study or work on difficult assignment at night. 18) I prefer to have class discussions after 9 pm. 19) I always complete assignments in advance and don't need any help or reminders from others. 20) I often postpone my work, hoping I won't have to finish projects, particularly when they are difficult. 21) It's important to me and to my family that I am successfirl in my education and do well in my studies. 22) I find that online learning is stimulating and interesting and always want to learn more. 191‘ r‘ r: 23) I always lose my concentration when I work on several projects at the same time. r‘ r‘ 24) When I studying or concentrating, I like to stop frequently or take breaks so that I can think about the context of the lesson. r‘ r“ 25) I felt familiar with using different kinds computer programs and the Internet before taking the Class(es). (‘ t‘ 26) I prefer familiar approaches to problem solving and function best with a pre-set study pattern. (‘ r‘ 27) I like to have an overview or know the reason and goals for something before I start. r: r‘ 28) I benefit most from analyzing information and study t0pics that move in a logical sequence and contain plenty of details. r“ r‘ 29) For better understanding I need to reflect on my thoughts and I prefer to consider all options before I make a decision. r‘ r‘ 30) Sometimes people tell me that I jump into the conclusion too quickly. r: r‘ 31) I always think about the consequences before take action. SECTION 3 3.1 In average, how often you used the following techniques for the online class(es)? FREQUENCY OF USE LEARNING TECHNIQUES Once Twrce Once Twice More than twice a week a Read lectures and reading assignments C C C C C l" b Participated in online team discussion (3— r‘ r‘ t“ r‘ (.. r“ _ 5 people) Searched for more information related to t‘ r‘ r‘ r“ r‘ (‘ lessons from the Intemet d Asfi‘ed questions by posting on the Web r: r‘ a Asked instructors for clarification and/or more information 192 |=‘|°° f Made short note on the lectures and discussion topics Analyzed of others' papers/reading assignments Reviewed class objectives I fl. Participated in online class discussion (more than 5 people) [... Performed self evaluation Reviewed video/audio clips of the class(es) |-:l’r Reviewed online class discussion sessions I B Reviewed my own papers 3.2 Where did you usually work from? (about or more than 50% of the time, please choose all applicable answers) 3333 3.3 What formal education did you have before taking the online class(es)? (please home/apartment personal office shared space/office computer lab other (please specify) I choose only highest degree you have received) *FM degree includes Architecture, Engineering, Interior Design, and Business Administration 33333 High School diploma Professional Certificate in a field related to FM* Professional Certificate in a field not related to FM* College Degree or equivalent in FM* College Degree or equivalent in a field not related to FM* 193‘ University Undergraduate Degree in PM University Undergraduate Degree in a field not related to FM* Master’s Degree in FM* Master’s Degree in a field not related to FM* 33333 other (please specify) I 3.4 What is/are the description(s) of your education? (Please choose all applicable answers) r‘ Regular course, but not an online (‘ Virtual school/Intemet based/Online course course r‘ Part-time study program/Evening r‘ Continuing study/ Self study program college (‘ Training program related to FM*(1-6 r“ Short course/Training related to FM*( 1 - months) 7 days) r‘ Others (please describe) I 3.5 Please indicate all of the job positions that you have held (choose all applicable answers) r‘ Educator F Administrator/Generalist (‘ Training Provider F Proj ect/Construction Manager F Researcher r‘ Architect/Engineer/Designer F Facility Manager F Business Administrator r‘ Space/Facilities Planner (‘ Consultant P Strategic Planner P Service Provider F Senior Management/CEO F Contractor r‘ r‘ Specialist (e.g., Technician) Other I 194’ 3.6 How long you have been in FM related position(s)**? “FM related positions includes any positions that are considered related to FM responsibilities for your organization. (‘ N/A F 4-10 years less than 1 year F 10-15 years 1-3 years (‘ more than 15 years 3.7 Please indicate your current position (choose all applicable answers) r F ull-time Student t“: Part-time Student r‘ Educator r Training Provider F Researcher P Facility Manager 1" Space/Facilities Planner r“ Strategic Planner P Senior Management/CEO P Administrator/Generalist ManZEEIect/Construction (A Architect/Engineer/Designer Business Administrator F Consultant Service Provider F Contractor F Specialist (e.g., Technician) P Free lance r‘ Full-time employment P Part-time employment t“ Not employed (‘ Other I 3.8 How long you have been in your current position? Less than 1 year F 1-3 years 4-10 years F 10-15 years (‘ more than 15 years 195' 3.9 What job responsibilities have you had? (please choose all applicable answers) r‘ Studying r: Operating and maintaining facilities (‘ Providing support services for building users r‘ Overseeing the acquisitions, installations, and allocations r“ Preparing business plan (‘ Evaluating changes and preparing contingency /emergency plans r: Managing and overseeing project t" Preparing and managing property portfolio r‘ Designing facilities, space, systems, etc. , r Overseeing all FM activities within my organization(s) r‘ Maintaining information and documentation (‘ Preparing technologies and services (‘ Conducting researches and studies (such as benchmarking, performance assessment) (‘ Communicating and negotiating with customers 1" Preparing education and training (‘ Preparing life-cycle/cost benefit analysis, financial scenario, and cost effective solutions r‘ Coordinating work performed within and outside organization r‘ Preparing facility plan: e. g. facilities, real estate, space planning, work environment, etc. t“ Analyzing situations and making decisions Other I 3.10 What were the reasons that you took the online class(es)? (please choose all applicable answers) r‘ I wanted a higher degree in FM. r r“ I wanted to earn more money. r‘ r‘ I needed to learn more about FM. (‘ I needed the degree for my career path. I wanted to update my knowledge. I wanted to take advantage of the availability of the classes and my spare time. Other (please specify) I SECTION 4 4.1 Please indicate the grades that you received for the following online classes? (in case of you have not finished the class yet, please indicate N/A) 196 CLASS N/A M C C+ B B+ A Facility Management Theory and Principle r‘ r: (‘ t‘ r‘ (‘ Facility Management Organizational Effectiveness r“ t‘ t‘ r: r: r: Facilities Real Estate and Building Economics t‘ r‘ r“ r“ r‘ r: Information Management for Facility Professionals r‘ r‘ r‘ r‘ r‘ r‘ 4.2 Please evaluate your ability to perform the following activities, both before and after taking the classes. Not ACTIVITIES Not at Poor Fair Good Excellence at Poor Fair Good Excellence all all r‘ r‘ r‘ r‘ r‘ a.) accomplish your r‘ r: c‘ (c r“ work r‘ r: r‘ r: r: b.) work with your C r: r“ r‘ r“ colleague(s) (a r“ p r: r‘ c.) meet your working r‘ r“ r‘ (a (a objectives r“ (c (.. r‘ (a d.) understand people (a (.4 r‘ r‘ (‘ around you r‘ r‘ (.. r‘ r‘ e.) meet your work r r‘ r‘ r‘ (e schedules C C C C C f.) make decisions C C C C C r r: r~ r r g.)analyze , , r‘ r r r‘ r: information/Situations (a r‘ p r‘ t‘ h.) perform your daily r“ p r‘ r‘ c“ routines i.) understand your work (a (.. r“ r‘ (a processes j.) deal with unexpected r‘ r“ r‘ r‘ r‘ Situations k.) present your ideas to others 197 4.3 Please indicate your opinions about the following statements. STATEMENT Strongly . Strongly Disagree Disagree Neutral Agree ‘ Agree ‘ a.) Knowledge that I acquired from the Class(es) helps me reduce my organization's C C C C C operating costs. b.) I work better after taking the class(es). C C C C C c.) I can finish my tasks faster after learning F F F F F about them from the class(es). d.) Before taking the Class(es), I received F F F F F more complaints about my work . e.) My boss seems to appreciate my work F F F F F more, after I took the class( es). SECTION 5 5.1 How would you describe your learning outcome(s) from the online courses? (please choose only one answer) r‘ I learned more and got through it fairly I did fine and learned a lot. well. I learned a lot and did quite well. I learned a lot, but dld not do really well. I did not learn much from the class, but I I did really well, but did not learn got through it. much. I did very well and enjoyed learning so much. 5.2 How regularly were you logged on to the Internet for working on the class(es)? Once a day P Once a week C At least twice a day r‘ Twice a week Several times a day, depending on circumstances 198' Several times a week, depending on circumstances 5.3 Generally, how much time did you spend studying for the online class(es)? . r‘ Less than an hour a day and not consrstent ‘ 5-9 hours a week (‘ 3-5 hours a day or more than 20 hours a week F 10-15 hours a week Not consistent, depending on my work schedule F 16-20 hours a week 5.4 Generally, how often did you check your e-mail during the course(s)? C Once a day r‘ Twrce a day More than twice a day t‘ A couple times a week, but r‘ I have had notifications whenever a new e-mail not constantly arrived and I checked it several times a days. 5.5 When I took the class(es), my age was ......... . Under25 " 25-30 C 31-35 F 36-40 C Over40 5.6 My gender is .............. P Male (‘ Female 5.7 Would you be interested in taking more online FM courses? P Yes (A No If no, can you give the major reason why you would not be interested in taking more courses? 199 engraved = 222.» mnmwwemmeamea 5.8 Would you be interested in taking more courses, if the program was extended into a full online FM master's degree program? F Yes P No 5.9 In two or three sentences, please comment on what might help you learn better in the online FM courses or suggestions for course improvement. 200 «8:538 636: $033 :32 $525 9o: 9 momasoo 8:5 85 36.8885 02 2:3 momwofibm “an? £25083 2: 5:5 8%.. 8: on so» .2 NN .....wEEe2 338:2: Mo 2330 a 2:5 as Eng 8 mow—=8 05:5 95 eBEOEOoE on 2:00 momwofibm .«0 mean. 323 dogma so.» .3 _.N ~533on £5 £5 ovum—w so» on "N 553:0 .3382 :05 8 35598 :05 beam can wEEmB 8m .38 “an? :o @3280 can 302 Sons x55 -2 can €23 058 Some .85 8:83 boron Fan— 2 use“ moocotoqxo mance? 0.6: on? 3:255 452865 3.. 20:3 Emo— 8595 20: 5 3:8 .38 35 986$ $020M can 350 >5 205 08 £258 So.» 5 3 53850 noun—.25. ”855%... 5:82.... .3 e38»: :03 O on 9 “EB 00:85 028292 moon— Au autos Emo— 3533 25:0 20: 2 max—SS e28 8:338 anew E wfiwwwao new wEEaB co meson—woe bang—2a $.35:— .oo=otoaxo :83 megs: 35 gonna : JESSE 8m “28::03820 c 22222— 33an ...—3383.3 .moocmficoton mew—.82 .3595 Bobs 282a >58 35 >638 258:2 Panacea “53:5 25:: m .N ._ 8365 2855526 Eegzoa 05 Eat momcenmom 2388: o :2: E889 05 2a 855 3 macaw 333.... wou—HES: wawmmxo 953:9. .3032 3038 smack”: 35:52 v.8 328.5 be 9:52— a... m ._ 2:38:— an 355% 33 .5526 $5 c8 Sun— 2382: o 338?: Me 35520 “as? 3 vi «2;.» .— % 8.53.36 mafiaumnfl aemnuaofifi 335% «3.3% $9590 SuQ gfiuumaq £3332 $333.30 «23.8% 38 zemumeaw =38:me fix: .muuaaem. Sufi .uaemauau {eacwmez MeeEu “Sizeuefifl U van—755.3. :. 2an 201 5000:...200 030.. .w:.00:00.5 0.0 03 E00... 0. 50.0088 00:050....0 .0 08:20.55 .0 00.5.0580 .30 5:0 00:050....0 w:.:.00. 5.0.. 50.9.0 0000... .0: .. Nd $.00... 05 :. 505.0005 0000.... 05 0. 50.09.50 w:.:..00. 5.3 000:0..00x0 .:0.0.....5 .0 .050 50.. :0» .0505: .0 50.5.0005 0000.... 0... 0.... 5.03.0 003 5032.0. :0» 00000... w:.:.00. 0:..:0 05 .0505: 0.0050. 0000... .00.. .. ..m ......005 0... ..0 0000.... . 05 30:0. 00:0..098 w:.E00. 0:..:0 .00.. 55 8 5.00.5 00000... w:.:.00. 0.50:5:00 0 .0 ..0: 0 0:5 5:0 w:....00. .05.... .0. 000.. 30: 0 08000.. 5:0 00:050....0 ..05 0. 550 08085098 000... .0 0.5.00. 0.. ... 0:050:50 0503 05.0000 ..05 :. 330.305. .0. ...00:00 05 .00. 0. .:0:...0..0.0 0.5500 ....0. 000.... :. 0 E00. >05 .05.... 5.3.0.055. 5:0 005033050 080500 .05 50.5000 0. .. .00.... 000.... :. 0 :0.00..00.5 .:0..0.:.0.:. 5:0 030.30.... 30: :0 .0050. 5:0 50>? 0 0>.00..0 ...B 0.:05:.0 .20.: E00. 0. .05 02.5000 03. 000.... 0 0.0305 ..5::0..w0.00.. 030.305. 0 00.. .0. 00:050....0 50.03.5500 E05850... 00.. .5530 b0>0 .05 5508300 05 5.3 0:.m0.. 0:0 000.... 0 ”050505025 ...w:.:.00. .0. 0000.... 05 ..0 .05.0 0.. .. 0000.... o 0:5:0 :. .0000 0000000... 50:. ...0. 8.0.50 00000... 0.... 00000... .0050? 0 0. w:.:.00. 3030. .0 0:.0..0.. 00500... 0 .... .. 00.00.55 .05 .0005 w:.:.00. .0:0.000..0... 0 0. 0.0.. ... 0.30.0... 0 03.0050. :0.:.:00 0n. .5 0 0:0...00zu 00.00000MQ 3.0.5 0.000% 000.00% 0.000000% 50.0000fl 00.....0036 5.5 . «.00: 25:00. 000.5030 £0.0000N 5:0 00.5.0.5 20.00300..Q 500. 600.50% 80¢ .0..0...00:U £0.0000fl 3.0.00 0350.20.00.0fl ..h 0.50... 202 3035800 030:» .:0::0 0:: :8: :050: 0:0 00:: :0» »:3 829:0 00003 o 05:08 :0» 0:0 0:0 :023 0808:8000 0:0:w 0:: 10:23:02 :003:0m .h 0:0:808 800: 80:: 00:0: :23 ::08:m:000 800: :0 82:03 0 0:0 :008:w:00: 1823:0880» :0 0:20 w:2:03 o . 8003:0m wm800 0:03 :0» :83 ::0:0 0:08 80:0: 0: 0:0: :0» 20 80800280 :023 5 .0 8:00:02: :03 0 3030: 00:20 »:0::0 »0:: :0:3 88: 00:23:00 0:0:m 8 0:082:00 080080 :23 .8000 0: 300:: 0:08 02:00:: m .N .— 0::08:m_000 0:0:w 2:: 3:23:08 50: 2:: :0» 08:30 92.5 0880 0:: 5 08:80:: 0 02:00—00: 0— 3 0000:2398 0:: 88:00— ::0» :0 80:0: 0: :0» 000:0: :8: 0008308320 0:: 0:03 :83 :0 0030305: 00020:: ::0» 2:: 000:2:098 ::0» 80:0 22:: 0: :0» 000:8 0008308320 :0 80000: 8:3 .000:0::00x0 88:00: ::0» 80: m .m .20 .080508 w8>_00 .0:0::0 32:00:00 828—00 85:00 0 :0: 82:08:00 00 :03 002080 20:: :0 :00_.:0: 0: 080080 000: 0:0000: »:0:: :05 0:022: 022:0 0:0_>0:m 0002:9508 0:288:88 0:08:02: 2.200% 85% :0» 0:0: 0: 0:0: 0: 2:00 :0; ~00: :0» 2:03 :23 5:0: 0 0:0—0800 8:00:00... 000: :0 000000: 328.00. 0: 8:08:08: 3:80 :0m :0: 2:00 :0» .z «i :03 0 050000 :0: 20:: 60:0 .<._. .:0:0:::02 80:: 688800: 805 20080: 02:: 02030:: 0008308320 :0 80:0: 838 :0» 280:: 0.000: 80:: 00:0: .:0 00::— 8:3 0:: :0000_ 0:: .:0 80800 8088208 050000 0: 0002—0 0:: _0>0::: :0» 0:0: 2:03 :83 80000. 0 2:80:08: :0: 20 :0» b 1:. ”0:882:85 8 8000 080050 ”8:00:08 »: 00:08:00: 0 0: 0:0: 02:00:: 0.: 0:00: 88:00: :0 0080880 :83 :020000 0: 20: 0: :0» 00:: 2:03 0:: 000020 3020: 0>::00_.:0: :83 n .N J 0880 ::0 0:2 :w:8:00_ 03:00:05. 0:08 0:800:88 0: 82880:: 0:: 0 3 .v 08:80:: 0 0000 A: :0 .N 0. 000.000.»:Q 020.0003Q :20§0&Q 5.0.5 800k 0.0080,: 0:5 82:00am £08000“ 30035800» 000.8036 £08000- 2:: =0.::00:@ =0~00=00R~ :0} 600.50% 50¢ .0:0.::00:U £08000- 3:023 0A30=0m8~0fl :5 050:1 203 «832.228 ~33» 26:26»: 9:08:33.“ 25c? 2: 350388 E wig—.82 .30» E25395 9 95% :39; .8 “co—yam .8528 55 go 2.2a bee—9:8 €03 9 3:035 023 :0» 2:03 gushing.— bofiv—BE a on 8: can 888 E wet—33 .= .3 $852: 808 8.: so» 33 236.: 33E «6 ~68: zoom 9 8:. 26; 2:03 no» 8:36:02 mcfioaou 55o 205 ou< 1a ocean—2: :05 3.598.“ 022 so» 283 $558— .50» 82:86 2 wow: 8: 203 :03. mm maosachm o fioaofl ho cages a «8:28 9 :88 So» .8 =9» wig—ma flawed—mama 5:55 o :83 E comm—5&0 0.63 83% 058% :23 353.33 ink no? 0 mafia—.0 o $30363 9:32 .850 3 3.38— .553 $533— o 038 film a :o 39$ 0 Ema—£8 omaohomvBm 030 0 T33 92‘ ~6on so» ow €53 .8:oE-w_mms mo womb wage—.8 05 .«O .a N .~ .3258 on“ E mam—=3— 80me 89:96 9 €92.me 05 3 new: 825 £552: 253$ uofiasonmu no? 0 $035 3525 av £335 bang—Ba ficoaamflmvoflofi bot—ow ”28526030 0 858 mo 95. uag Am «.8322. 3322.— 333:0.— 2.2.—5m 9:23.85 2253. 2352: fez—2. 8.53 a: 3:5 .25» .m m. 3.53.36 mafiuusau :Sumauuwfi $3.5 “Suck 823a. ~55 $3.335 «uuawnmfl $33538» muamBuEQ fiuguumufl 33 aemunuau £883me aux: .uuuxagm. Sufi 6.3.5an auguwnufl MEEB u$£a§£§~m¥ fin 035. 204 00:05:03: 030023: mEon .3 0053395 05 330 30:083.” 000055 2 .m $5500. 0:0 wEEnB 0:=:o 13230.35 :8 0.8305000 mEEaB 00:00:: 03:00—00: 0 0:230 9 .N 30:80:30 35 E 05500. 00:08: 03:00:00 080005 53 :03; 0050:. 3:28:55 0:0 0:085:35 w:._E00_ 0.5:: 05 9 00.0.0: £9000 0:0 05800. 00:08: 3:00—00: 00 0505300005. 03205 2 ._ 303.3036 :0§0n0- ammo—0:200: 05$“? toga sex 0: EB v:< ~20 w:_:.80_ 3.000: 0 03: 00>? .m— 32:00— 90: 020: 2:03 >05 33 5298 0:0 ©6000 0000—: .0050 o 00.8 0:3. 90>» 0.500: a 55:5 3030.. :8 @008: :5 003000: 0:03 0:200:86 8:0 034 a 0:8 :03 .8 30:0 03 9 @0095: 00 36:20.28 00mo> 0 52:00:00 w:o:8 32> win—«Bum a :33305 :03 0 H0088: 03:02.00: wibfim $5055.28 82> o flax—0:0 b05822: 0300;05— wfiofifig :8 $09»: :33 .8 .3 $00500 05 E 053035: 0:28:030 o 00:90—00 0: :00 0030—2300. M .N J Rama—0:50. 08:: 000 8 00:000.: 020: :0» 2:03 .3 3030.. 003800: a no 00:2 :33 :0 000200030 wEEaB :30 59A 02050 :0 0N_:woo0: :o» 2:95 6022:: 0:03 00300030 w:_:.80_ o: .2 .n— 0033209 00:00:: $563006 .98 8 50%—3:80 E 0>c00c2 6008 908:8 N J 0E3 .5 >8 >05 3:? 0:30 08:: x55 0:0 3000—0 0335:8500 5:830? :03 0 E023: 00:50 05:5 0Q A0 2 3:00:00. 00:8 0:2:0 wEEB 5 390258800 0000 .N— 0000:0t0qx0):20_> wfianm :5 5:08:85 wfiwfifioxu .wEEmB :o $83 :30 :8» 00 305.00% 06305 30:850.: 3:00:00 ”00:00:: 0380?: :5 23:05. 05 mining E 3.3—0: E05 gm :0» E: 60000: ”0.._0::o=m0:0 o :88.“ 00050:: E033: N J 0:03 uoow b0> 00.02050 3:08:33.“ no 00—0805 m< J— 3030: 0:533: o 00:50 :0 00:2 00:3 3 00:0»N0SQ 0:0.503Q =Su§00mn~ $309 0.300% 0 000.30% ~55. . . 0.30.03.03Q #050000“ 3000:2200» 00.50036 «0.308% 3:: :0Em0au =0mum=0M~Q :0} £00.30.» 30¢ .m:0.§00:® £000000- 93:3 0A..~m:0.=0~0- , fin 030:. 205 Quantitative Data from Questionnaire APPENDIX D Table 7.2 tonnaire t Quantitative Data from Ques amlalvas)9y (OJOI'.3SIDJJ0.uoo—n—o—OOv—a—nc—Ov—IOO (”14131095953 ' (0101:3SIDJ10wumv—n—u—n—no—uvu—mogKou—noo amlalnas)yy (040153570140oc—o—«o—aooooo—noo— amlalms)gy NVVSVNMMNN—‘Vlnm—v—V) (5‘ 01! 310959221 Nv—WVS—VMWNWNMNVNM (5 0: I aim) I a! (rm! A A :4).40,mm[ag . a .3 Sup/up“ heazbwwaeaaahwmax WWI/333 892$38$3$$EK$R33 (0“01"3SIDJ"0-—o—.—.o—-ooo——a—~o——-o mamas)” (owlwsm-‘Iwo—oooo~o~oc~oo—~c amlalvas)5g amlaloas)yg (0401.'asyag.lo__ __ ______________ anal 210.93) 93 A MMv-‘MW—‘F-‘WMMM-‘MMVW (£01131995')ZD WMMWMVVWMMVWMVVW (9 0: I 31099) I a (El/3 :%).logwqag 3up/ugtu Swesahbzwwaahaez Imam-9 tsessxxassss 3:33 #EISVD nmmvmehwasaztfiflfiflffi 206 (table continues) Table 7.2 (continued) tonnaire t Quantitative Data from Ques - (table continues) anal 3109s) 9» x (0401:3SIDJ'IO¢.oo—~—¢Qo¢¢—¢o~ amlaloos)5g - (0401:9slojao____o_ooo_o___‘o_ amlalvos)yy (oazlazsl’jd‘oe-coooo~ooooco—~— .l 9‘92! (£011 as)xN—NVMVVMNVVVNWW 3]” Z (go’IaDDS)IHva—NMMMMNVMVN—‘W I (fl/21A :%)«o.mqaaa v Bugugflmevvaascxooaxcxzeb: ‘(OJOI'QSIDJJO~o—oo—~ooo—-—-o~—-— amlaloos)93 ,l(0’lol’.aqvjw~~o~ooo~ooo~o~o 3124131095955) (oao[.-asloj.lo____oo__O__ooo_ amlaloos);3 '(oaol.:aslvj.lo_____________‘_____ 3’141911795')€3 (gallalvas)zaWWMWM~VIMVI~WMWMM (5.011210959I0MWMWMWWMWMVMWMW (SI/I) 1%)101401193 Bugugu2§oo§eb=oo3~o=u~=co3 ‘hooaxc—numvmehooaxc—u 207 Table 7.2 (continued) tonnaire 1 Quantitative Data from Ques (0 .to 1 :3qu .10 amlalvas‘)9£3"fi—‘oo—Oo—‘o—‘M—‘v—o“ (0 .40 1 25701 40 amlalva-S')Q£‘H~"""""‘—‘oo—¢—-o—-oo— (31/3) saouayad xa/Kuzzqzsuodsaa:eamaeeosesomnma 4011.7 (5010310959Z3NMMNNm—ovmmmm—NN (s 0103,09591gMMMOMWMOWV‘WWNNNM (sm¢%)£§:v:§:°£:§$2e2ez aauagladxyflfiéé-33365..ézéi~3~3:~335 mph—ewmcooeoovenmonm ( amasgaszsaasassz 00€/'I+¥+3~_~~N__~—~~~—~—~ -%)«f:mqV:aasaagagfiaszgaa gummulhmooaoco h hhfl©©Vl~ :04 (0’10,anO—Om—O—OOO~~~O~~O amlalvos)97 10:30.40 anqaloos)57 (olol'laslvjlomo—nu—nv—tv—n—nmc—A—n—d—uu—u—d—no—u amlaloos)y7 .to:aso.lo anuapafigj 6m ’0 $0 @0 60 60 §o M @o @o §o m N M §o $0 (9 01 I 3109‘) Z 'I VMWWWMMMMWWWVMMW - (5‘ 01 I 31999) I 7 (8/7 :%)3u.vtum a ”a” [Dagfilouyhvavhwvooméééeeve O. I. .0 .0 O. O. O. O. O. O. O. a foI’Aa'I'IFofiESBflfi'SS—n—leg’fib #HSVD uumvmehwexSZ'fiflzfl‘: 208 (table continues) (0 .10 [ :asloj .Io Quantitative Data from Questionnaire (continued) Table 7.2 amlaloos)agg o o T T o T F. d o (0 .10 1 :aslog .Io . amazon» 4:51 ° o T ° o T T T T (91/3) saauauad xa/mzqzwodm o L: m z :2 : oo 5: 2 2 2 ‘2 :2 m 901‘8‘5! (goxoazvas‘ng o —. m m m —- m m m (5‘ 010310959IHN MNV‘OWNNWVWWWN (HZ/3%) Nflnwsaaeezzzaao‘ estimating .. ..saazesaaaaewaaaaa bthFF©Vlhwbwwbm I assassasaazsegs :%)‘0!I!anae.:g.=aaza.ae'dazaaa gunnmuleem m©F©©hh©h©Q (0401-9570110~c~oo~ooo~~o~~~ amlaloos)97 (0.101 39.910de __ o _ _ o _. _ o __ ‘ and 2100s) 5 7 (04012570540 _ __ o o __ o o o _ amlaloas)y7 (0.101 $5701.10 h __ __ __ __ _ __ _ __ anuamoflf'] («3 ex. gx. ck. an e». e- fi- (‘4 (901131995927 ‘ m m <- m n In to tn tn (5 01 I 311793) I 7 (8/1 :%)3u.111um 0° [Dogfilvuyemmvmmheeojeebws @mfl7&ssassame§aaaae ”W9 22225323332238: 209 (table continues) Table 7.2 Quantitative Data from Questionnaire (continued) (0 .10 1 :asloj .10 anuaIvas)bgg ° "" —' —‘ —- o o — C (3n: 21;:3213; O "' ~ —- o o —~ o o (22:: 2,;ZzDi-J; O '-' o —- o o o o o mgiifiii‘; -— - _. _ o c c _. _ $1.110 9110:3131}; "‘ "" —' —' .... o —- o .— (galaljgfi-l; "‘ — ... —- o o o c o (ironic; 21230;; O "‘ '-‘ ~ .— o .... o -— (0331,3125?sz ~ ~ — o o o o o e: (353123555 —- - o — o o o o o $35331”; o o — _ c o .. o ._ taggzzvgg —— ._ o _ a o .. o o (‘inif’z’aziisiffié‘é — o — —- —~ o o o o ($32,355; o ~ o —— —- o — o —~ (21:: 2,30%; —' —~ :3 o o o o o o # as” m e a = s: :2 z a 2 210 (table continues) Table 7.2 Quan F (0 .10 1 :asloj .10 am 1 3100s) bf g (0 .10 1 3331111 .10 312.11 3100s) dg g (0 .10 [ :aslvj .10 am 1 3100s) 09 g (0 .10 [ :aslvj .10 am 1 2100s) as g (0 .10 [ :asloj .10 any 3109s) “If g (o .10 [ :asloj .10 am 1 2100s) If g (0 .10 1 .'as[0__.[ .10 am .1 2100s) 1]; g (0 .10 1 :aslog .10 am 1 3100s) [5 g (0 .10 I .0st .10 and 91099) .‘f 3 (0 .10 [ :asloj .10 am 1 3100s) 1]; g ire (continued) (0 .10 1 .0570; .10 212.1 1 3100s) 33‘ g tonna (0 .10 1 :aslo :1 .10 am 1 2100s) jg g (0 .10 [ .‘BS'IDJ .10 am; 3100s) of g (0 .10 1 53.910! .10 and 3109571”! 3 titative Data from Quest # HSVD 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 211 (table continues) «8:338 ~33» m m m o _ n v NM new m: "mm o m m 5m ~ _ @— N m m _ _ n v 3 3m R— "ev _ m .15 o c m— v m m o o _ v VN "me can "Nw _ v. m "mm o o v— v m V _ _ N v mN "he N2 "3. o m m 5m 9 o m— _ n v _ fl w v 9N "ch n: ”a... 0 ~ g u: o o N— m v m _ ~ m m 0N ”2. SN "Na _ v m "mm _ ~ n— m v m o o m N —N "hm $3 "No C m m 5m _ o S m v. m o _ m v MN "Nw N—N “I. ~ n v ”be c g a m m m o _ m N 9N uvm cN— 5v 9 m m “cm c o m m N v _ ~ m N 5N "Mb fig :3. o N N "mm o o b v v m o o m v ¢N 3m aa— "we 0 m .m 5m _ _ o v v m G ~ n v mN "be :2 5e o m m 5m _ _ m N m m o _ w v mN #9 as; "am o m m "ma 0 _ v m m m o _ m m wN “eh ww— umw c N N “mm ~ g m _ _ m o _ m N 3 “am can "8 o m m 5m d _ N v v m o _ m N NN "Ne :N 5h — v m "mm o m n I I I 0 I o I. I I S .1 I 3) l 0 o 3 o 3 u n n u... u.» H. ”mm.” 1%...” ....u m Mmmu “a. ”i. S S S 0 19 S S S 9 a W to . 9 H D )2 ..S/ M J D 9 a a .10 W9 9 a 30;. 0U0.3 IS a 0.240 I” W.” m 01 W 01 no 3W w. olsflm ,w n.” no a HP... “a as a a a :94. ..a a aoau +Va ..m. .4. Woom ..m. :01 S 0 0 0 ..l l .l I .l .I. .I... M. W X 0. w I. a I. \l a .I.. a I. 3 M .0. m w m m w m m. w m. + n m m. w. w /. m. m m. m a a a m a m a k. m M m a a m a m a Nnmazhzooy ugmuzafiumuam EEK Sufi uamwsmncuafl NH 2an 212 «332228 033» m v N _ o m m ”N "2. SN :3 _ v m "mm o o —m _ m m o _ w v NN "am N: "mm c N N "mm _ _ an N v m o _ m m mN "S SN "2. ~ v m "mm _ _ aN _ v v o o n v mN "Ne ch an... o m m "cm _ fl «N m N m _ _ n v NN "mm EN "5 o m m 5m _ _ «N v m m _ o m w NN ”am am— "3 o N N "an ~ ~ eN N m m _ o m m _N Km um. 5m o N N "mm o _ mN m N . v _ _ m N _Nfim ..E— "Ne _ v mama c _ ..N N m m _ c m m _N "hm ..MN "F _ v m "mm _ _ MN _ N m o _ m m NN "am 3N "Mb _ v m “mm _ _ NN m m m o o m m mN "Na 3— 3m o N N "mm ~ _ —N m N m _ _ w v MN "Ne mN— "Nv o N N ”mm o _ 9N g _ m o _ ~ N 2 5m 8— "G c m m an _ _ 3 _ _ N — _ w. v 3 "av ww— "Nm o m m “ma c o a— v m N o _ w v N "2. m2 “an _ m 9 xx: o o S u w. n 0n 0n u u 0..” 0000..” a 00m. 0” 0m .0 9 .s/ W) ma.» .0 .0 mam £..w J ) @mmq Js .01 m m m 0...». Wm m m .wm. wuww W.» x 00a m.» m...» m .I. I. l ma 0 no I .I. .I. I 0 ( a J mu m U U. D: m. 3 9 3 9 a a a ...w. .a a a an +Va ..I. W. nuuoo ..W ..m. S m 0 0 I. I .l I I. I M. M X 0. W .l a I. ..l a m .l a .I. a 7. 0 0. .0 m m m m 0. w m + m u. m w 0. my... 0. w ... m 0. IN. IN. ,0!) m a m G fin. nuu. l0. a Inn /0\ a m a @825 :8» usmuaafiunmau ESQ. Sufi mafiaiauaw NN 29S. 213 Table 7.2 Quantitative Data from Questionnaire (continued) (S' 0103100901911-om~~_m~~_~~mm_m Jozanasulgm “010310959v—vaq-Inmmmm—«VVNm 3119149114le ' 0 2100s)zau.131u1 ”OJ-1.0!"!v—anVVerx-rvvmmmmm Squwo $0.1 (901031999) uo-‘S‘n95'30mmmmmmkummmvvmmm “109117121 (903031093)mvmmmmmmmmmmMVer .3“!Pvaa{l[1.l '.%)uo-'wz!mneb—mmchamcewg—nm anbgqua‘lf'z—IVMMMMNMMMMMmNV co co on on co co .0 on o. no 00 o. no a o. o. 3.1.2.1400qu $882388§8$$$$e28 43171909 EEEEEe-‘HHEEEEEM‘HW 33VVNNWMNmNmmvmm—Nv—un (gNW—MMWIOMVMWMVMWW 0119109575'11 (g'MN—MMVOOMNOOv—m~v Q 01 0 31993) Z Ll (gmcnmmmvvvmmmmmmvv 01 1 3100s) [a1 :%)8uyaauu 000000041000022.0002002000000003'2 dwmeeaeeaessaasasss m—MMN~V~M—NM~V'—N (5 0: 0 312m) 6 I HOMNN—‘MNNfiNNmm—‘W (9 01 0 91099) 81 #EIS'VD .uvaunehooasazfiflz'fl‘fi 214 (table continues) Table 7.2 (continued) tonnaire t Quantitative Data from Ques (9 0: 0 0093) (11811 1010mm] 5‘ [u (; o: 0 2109s) J[I‘ll (13./14 f [11 01 0 21005) 1211431141 1110.11 'ofug ' 3“!”‘1199 £121 (9 01 0 31009) uogssnosgq ““731 Z 121 (9 010 31099) BuzpvaaI I In 3 3 4 3 5 4 5 4 4 5 5 4 5 2 3 3%) “0110011112 anbguqaal 3141114037 In 49: 32 21 14 14 28 32 21 43 49: 37 22 26 29 34 32 40: 45 52 49 45: 29 42 27 36 55 45: 29 65: 42 42121490 0 f m m f f f m f f m f m m m f 23;; V (9 01 I 0,000) 9 1.1 (9 01 0 aims) 2 LL (9 01 I 0100:) I Ll 3 3 4 (91/11 -'%) Same uuoo/suoyodgoymd 14.! wads awn 1.1 10 10 12 9 8 66 66: 80 60 53 66: 10 40 6 53: 53 12 12 8 7 80: 80 53: 47 5 6 33 40 (9 010 aims} 6 I 3 2 l l 1 l (5' 01 0 3109‘) 8 I 4 2 #EISVD 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 215 (table continues) (984Lfl .'%) Iuawaouoqua aauouuoflad 3141114044 0w (5' 010 3109-9) 5' J7 (9 01 o 31009) r 117 ' (9 01 0 0100:) 9 .1'1 (5' 01 0 31099) Z 11'] (9 0: 0 91093) I 117 ( 5' Z/J 7 :%) aauouuofxad 3“."14037 117 (g 01 0 3100s).13dod 31114131498 5‘! [u (ST 010 21009) uogssnasga 314.149.1423 Z1 {21 (9 01 0 WM) 0an 170919.14 II 12.1 (; 010 21009) uozwrtzma-fzas m {u (9 01 0 aims) uogssnosgq ssolg 6 [u (5' oz 0 3100s) saayoafqo 9501;) Bugmagaay g [u (9 010 91m) 30170321 Buimamar 2 12.1 (9 01 o 3109:) 31112101 21011 9 11.1 Quantitative Data from Questionnaire (continued) Table 7.2 1 # 3.9V.) 31 19 . 15 55:20 53:19 40: 10(25) 86. 67:24 44:16 53. 44:16 47:17 42 55: 20 53: 19 33:12 44:16 67:24 (table continues) 2 4 2 3 5 2 . 5 5 5 5 2 3 4 5 4 ? ? ? ? 5 ? S ? 5 5 S S 5 5 4 ? 5 5 5 2 (5) 2(5) . 25 7 (10) . 25 25 6 (10) 7 (10) 25 14 (15) 84: 21 96: 24 40 88: 22 40 100 70. 96: 24 100 100 60. 70. 100 90: 9 (10) 80: 8 (10) 93 2 4 3 3 l 3 2 3 4 3 2 l 3 10 11 12 13 14 15 16 Table 7.2 (continued) tonnatre t Quantitative Data from Ques (9911151 A G -%):uawaauvqua azageeaaazagg-g: ”WOW "as-52022059920. 311.131-‘0111 1“! °‘ 9 (gozoaloos)§.{7 e-mV-mem'Vva-Nva «01031099747 5. m m e m m m 5. e- m m m w v @- (g0,031035‘)£d7 V V) W 6 C‘- G‘- 'n C" W V1 V3 V V3 #0 C‘- (goxoaloas)z,17 cu m m e e. m m e. e. w v <2- m e. @- (501031093)IJ7 mmmme mmme-mmmme-m smvsggmssmmeves (ST/<17 SfiNSczNESfiNNNCC :Waauowxoflad 38:33:8‘37333333 Burma] ([7 °‘ °‘ 3. a °‘ " (g 010 3100.9) Jadod BuIM3!AaH£IHJ ~0~NONM~VMM~NMW (901 3“!M3!4321 Z I [M (90’031‘799)o~ocoo~ooooo~cN WP" V/OapiA I I 11.1 (5'01 503901121, (5‘01 5“”0602 (901021090 3“!M3!49¥8[Il , (9 Olgazmflwpm 0. ._ o N ... .. 0 0. v ... ... ... ... ...... WMWWIIH 3 (501031993) Noemowrmmvmvvvmm “PIUIWON9IH :easaaaaaaaaaa: #EISVD 217 (table continues) Table 7.2 Quantitative Data from Questionnaire (continued) in e. V M M V M M M (5‘ 01 0 3109579111111 —- e- -- —- N N m —- m (9 0: 0 31095951 M v a m m <- m m m <1- (5' 010 31093) II (Mi '0 e- v m <- v <- <2- <1- (9 0:0 31m) 91 M m e» va- n v v m m m (5' 01031099) Z1 1!.“ 40 3W1 31"”)!!11111 (01013570! ...-9. coco—~— .10 am 1 31009) 01 11M (0.101 .°as10_.1 _ a“ c __ _ o o o __ .10 am 1 31009) 6 J,“ (010135701 —- e. o c c c: o c o .10 and 31009) g ,1,“ (0101:”le ...5. oooooc— .10 an; 01009) 1 (1,11 (040’ my”)! —- e. o c: c c c o -— .10 and 31009) 9 am (0'10! "3570‘! — c o c c o o o 40 WI 3199‘) 5‘ (IA! (010! 3570‘! —- o o -— o c o -- .10 am 1 210.75) ' (1,11 4 (0‘0, 53910-1 .— o c o o o o —- .10 am 1 3100.9) 5‘ ,1,“ (0101 53570.21 _ c o c ... c c _. .10 and 3100.9) 5 J,” (04013357031 .... o o o -- o c o .10 312.11 31099) 1 J,“ v 2:22:22 #EISVD 218 (table continues) (9 010 31099) 9I 0m (5' 01 0 910959911111! (9 010 91099) n .w boNMV‘V (9 010 21099) 9I J/fl (9 010 31099) Z1 M (0 .10 1 :0910 .11 .10 3W 31993) I I M (0 .10 1 .0910; .10 am 1 01009) 01 (1,11 (0 .10 1 .0910! .10 am I 01009) 6 J,“ (0 .10 1 .0910; .10 an.1_1 01009) 9 (1,11 (0 .10 1 :091011 .10 012.1 1 01009) 1 J,“ (0 .10 1 .0910! .10 011.11 01009) 9 (1,11 (0 .10 1 .0910; .10 0011 01009) 5 J,“ (0 .0 z :3st .10 011.1 1 01009) 1 J,“ (0 .10 1 .0910; .10 0011 01009) g ,1,“ (0 .10 1 :3910_.1 .10 am 1 01009) 6 (1,11 (0 .10 1 :091011 .10 00.11 01009) 1 ,1,“ Quantitative Data from Questionnaire (continued) Table 7.2 17 18 19 20 21 #21920 219 (table continues) ? Missing data 22 23 24 25 26 27 28 29 30 31 Table 7.2 Note: * The respondents had not finished any classes yet by the time of responding. «00:50:00 03:: 08.0.0.0 M5063 0.0.0: :30 0050500 62. "806 0050 .800 0.0026 E0600 5 02:00:00 0.55 wee—5..... N coma: 3030M. 220 0:5 6300.305. .0080 60306 Saw... E50023: . 8:58:86 3030a ..N n 5. 0:0: :08 0605 6:0 090385 . ”0600.865 N n v. 8 80.80:: 600.8 05:000. :0. 5622.500 ...8 o E063.0>0.=0m N u m. 66 ..0>0: :5 65:000. 95% 0000:. 00: 06 I 0.003 . E58086 80.0 00> uw550506mc. 00.22.00 5. 6.— 0 2:0: 9m $06 0 00:0 00 a: 0.003 a 00:5. .238 . 80280.30 80.0 00> ”SE 0800.008 00> E0... .25: 0\0Nv “50503095 :3 N6 0008 60:08 5 050.. 52.. 500. "0.3m ”5:000.— N &:600: :0 00060.». 00> E55006 00.029. 00> E06583: 830$. 006 “.5, A608 ...8 6:0 05.:0 8.88.0 Esme 32:00..— ..6m 0 ”0.0: :05 00> E50005 00> ”:0.0.m.:.00<.0 $60 ”A... S... c 08008:. 00> E2: SE... 07. 50:083.... ..\06N E53563 5. 0. 60.0.0: 00: 00:»06 b.&0>.::\0w0..00 E03026”.— . 0:8 :03 00> ”0:00:02. 3.2.05.0 00> 800.28 :09.st ctn E0..00::00.§:. £00080: 6:0 00.3.0500. 5.3 .006 00: :5 .200» . ”00:. 00:08:. 00> ”M5856300AZ 0 Z 80528200....0 Am ”A .n a. .6 ”0. 6.-.. .5.. 03:05 00.000.00.300an 0 8 @6603... m E08386 500... 00> ”SEAS. 00> 80.5.60. 20.0 v. ”0008 M5355... SE .2: ...0500 a ma 000.00.02.08 :00» 2.6... v ”3600... 00> ”.5055mfi 00> ”00630:. 535025 cm-nN .N 0w< 0.88.0 v :0 0.00:... DH. .3— 00> E0... 3:090:50. 00> $6052 0:06:00 0.02 088.0 m5? 20.02. :30 00.00500 60.. "806 0050 E0600 :0 Q 5 03:00:00 "0.0m 9.6.5.; 6 ”00%.. 306000. 030.305. 6:0 0:05 000006 Saw... E02550: m 8:58:86 3030M on .I. 5. 00:55.00: w5E00_ my. :0 68: £5an. . ”0600.006; n M 0:00» 0000.. .0: 06... 0.003 a 00:0... 2-... 56 m E53505 ...0w 07. ”3.505.339. m u m. 0 0062 800. 8 050.. 50.6 5w0. "0.2m «5:000.— m E58086 80.0 02 ”SE 0800.003 02 En... .25: 00.52.00 5. Sn 80:00 :08 m 802000.30 80.0 00> E55006 00.029. 0 Z E06555: .305 $0N u:0.:0>0.&5. .5» 6:0 $6.8 ...8 05.:0 088.0 00.09: 32:00..— .560 N $5600.. :0 80:3. 00> E08095 00> E0...m.:.00<5 {one ”.52 g .05 22.. N H90: :08 02 as: 2:... 00> 2200000.... .08 ”.3 603.00 S... 5 00306 b.000>.::\0w0=00 “90.00056”.— . ...—80.50:. 02 ”5.0.600: 3.00055 00> 800.28 0:00:55 00mm E06365 my. 0.30.. m-. 00.. 002... mu 3.0.0031 v 0.63 00>» 00> 05005600002 07. 800.38200ha 2R E0.80::082:. 03.200 .05 05:5... Facaézugsés 0. ”es 059:. 00> N35:: 00> ”0.02.00 :06 .4 a . 0. a .0 no. 0050.05.00. 805083 5.3 .006 .0: 5.. m ”00.8086 500... 00> ”.5055"...— 00> ”00660:. 5850.25 6. ”0008 M565... .2: .m< .2... 20: 2... 5 000006090 .00: 2 -7 m 0508: 00> ”5.... 050.0500. 8:» ”€82 8-2 .N 00¢. 00000.0 v ...w 0.8:... D... .6: 2.88... 086.52. .202 0.00.00.20.95 809? 035.3% 8.00 :39? b02234 m... 030... 00.353.080au E0..\:§Q.\0 53523.6 "0 02:25:40 300:...200 03:.» 221 0080.0 56.0. 0.03.. .....0 3.00500 60.. "0.06. .350 m ”.0000 30.3.. 6m u A... :0..00 :0 Q 0. 50.. 0230.3. 3.3m 0:35.... n ”0:58:86 30.3.. m u v. .60.. 30.00 0060.305. E53252 n ”5600.006; 0 u m. 0505:5000 w5600. 600205. c E0..00.0>0 ...0w 6:0 05500. .0. :0...0:.0>0 ...8 66 .33: . .55: m E58086 80.0 07. 05505.06”... .0.>0..0.. my. Ev .0 w5E00. 0:0.» .03... 1 $06 0 00.3. ..050 0.0050 . 802.00....0 80.0 07. .2... 0003.006 02 E0... .00m... 0...”: ”50526.05. .5. 6:0 ..003 0 0.00.. 2-... $06 0 00:0 0. ..0 0.003 c 05600. :0 30.3.. 00> E0586 8.02... 02 E0..0..0m0: 000?. $8 ”.5, 3.. 0 005.. .03>8 050.. 50... :50. 3.3m «5:000.— n ”0.0: ..0..m 07. E50005 0Z E0...0.:..0<.0 o\oww ”.... 05.:0 0080.0 .0.:w0. ".05.0... .06”..— N ”.0.0E.0:. 07. E0... 2...... 07. 50.0805... $3 E0..0~....D .... 2.. 0. 60.0.0. .0: 00306 0.3.002 E53006”.- N u..0. 003 07. ”5.0....00 3.000....0 07. ”000.38 .3005... ... .0 E0..00::00.0.:. 0.00.. m. .0. ....w2 00000 6:0 58.0050 n ”0.5. .0505. 02 um:..0:6.000.Z 02 ”30.200200ZQ .m H. . 0 H... .00. $550.. m< 300.080. 00 0003.393 20.. .05.. n E58086 500... 07. ”20.2 07. ”00.....00. 20.0 NN ”0.08 36.5.... -..... M5650 .63... 2... 5 000053000 w5...03 0: I n 05600.. 07. 50.0005"... 00> U006.00... 50.5028 mm. m .m 0w< 0080.0 0. ..0 ..00...) DH. .5— . 0 Z E0... 3:0w.05m0. 00> $6022 5060.0 052...... 0.02 80.0 . 0... 050.0. 0.03.. .....0 3.00500 60.. “0.06 30.0 :0..00 :0 a. 5 .60.. 02.003. 3.3m 3.6—:6... 030.305. E53232 «m n my. ..0 .0 ....0 006. > 6:0 002.0030 0 ”300.. 30.3.. N .I. v. 80.0 6030.3. .33: 05500. 0:05 .03.... ... m ”0:58:86 30.3.. m u m. 5.00.0000 .0: 8.0 ...»06 0 00:0: ..000. .0: .00 0.003 0 ”0600.006; 0 005.. .0330 050.. 50.... :50. 3.3m ”:60004 N E0..00.0>0 ...0m 00> ”m5:.0...\.06m¢ 36.0.30 my. Sn >650 .28 n E58086 80.0 00> H2... 0003.003 00> E0... .00m... 60....50650 6:0 00.000 5.023 05..-..00 .0550 ".05.0..— .060 0 ”00230.30 80. 0 00> E50306 8.02... 00> E0..0..0w0: .000... 0000.03.05. .5» mu 2. 0 ”0503. .5 .803. 02 5.80.0 8: ”8005850 80082.5 H...... 0. 60.0.0. .0: 00.006 50.025.35.00 0:06.006”.— 0 ”0.0: :55 07. E0... 2...... 00> 0.0.088... 60....006.:: “a... 5.006 6:0 ”55.0... m 00.00.05. ”5.0.6.0.. 3.2.0....0 00> ”0006.30 ...0....0m.m $3 ”00.35:...— .0. €82.55: 5.3 .80 .2. .3 .08. 0. -.. 0 0:5 .03 8.. n0.0.0.0280? 8: H82:3...8...: 0:0. u0000.20.85... .0. 58.0050 0 00 55.031.08.058 6:0 .050... m ”0.5. 3:35. 00> ”20.2 00> ”00.....00. 20.0 An ”...... a. ... .0. 00000 0.00.28 2". .350... 00 0005.303 .00.. 0.-.? n E58086 500... 00> 20.0005"... 00> ”006.02. 50.5025 0. “0.08 w:...:...... 3565000. ..0 05.. 0... n ”05600.. 00> E0... 300905."..0. 00> ”>605? 9. 3.6 .m 0m< .... 80.0 0... 50.5. .0: 6.6. 80.0 05.:0 8.... 500...: D... .0. 5060.0 0.02 00000.35.“ 82:... 335.3% 8.00 «6002.080. 300...: 335.3% m... 0.00... «332.328 ~38» 8320 N n ma mfixs 2&2. =67. 359:8 was .233 .656 $2 comma 3053. N n v. .838 ... 338:2 83% 9.35.; m 6:23:85 3033— m n m 52. 388 £8.30 85:. "32.5.3.2 m 66.83%; wEEmo. c E28296 tom .5322. mm 9;. bob. corms—«3:8 Be .65.. :5 £335.08 war—Bo. _ ”:o_mm=8_v man—U 8> $5592.:va 3mm acoEo>En§ m3 .2 :5 v3: .23: 3 0:2“ wEEaB 5.93 $5.50. N $338.30 mun—U no> ”2m ou§>9m n5 :8..— .msmc {can ”A? 9.8» homo:— 8: cc .3003 a fine: 8-2 .998 a «2:: m $538 .8 805% a; ”.5663 832?— 92 ”55259: .39: $8. ”.3 ca .838 .350 Eng—on an Emo— 3_bw “£53 _ 68: team 9; E3805 9.5 “:oEchufiO $3 ”cowaflzaa .2 2:5 98 3&2 32:3..— .aem _ ”38:me a; ”cu—u 2&8 cz inhomoxfi o5. Eo:8=:8.§£ SE 8 c222 8: Eu: 05 E 6:02.2— ztuaen—AO 00> “mofitum conga—m3 Am "A .N um .98 88» o. -v 8.. SEE—n v ”8:— 85.: 50> ”matacmEooUAZ 8> ”fiqumEuoha 2 ”203. mac—:2... 8an v5 «as—3:8 £285.80 3 wan—83... m E06386 E3... 8> ”SEAS 8> ”82:8.“ 206 ow .26 .m 6w< .2.”— .SE 5 moocotoaxu an» 2 55 22: 26.. I m ”363% new ”ESSEES 9.5 UoEouE Eggs—Am Cog—ow BEEoBEé 8320 v =a x8 .7 DH .5— 8> ”5E zucowhoEm—O— a; item? 2.38m 253.3— $.29 05 mac—8 383 :3.» 32383 vs: "32. .350 .838 .3 258a». 8.3m uni—...; on H mm 53 823 demo“. 3:95 "32.9522 m ”893 3033— N u M 3:358. N ”323386 3033— £59.23 m n m wEEfl: $— =« v8: .33 a .33 mop—5493 m 668335 8:52...— uuzuzasnm C. 5 9.2.32 96..» .52.— 1 $3 a 335 .386 x85 _ E23396 tom 85 $5593.:va .5322. and En can goo? a mac: n _ A: $3 a 8:0 9 a: 333 N E23586 820 96> ”SE §>Qm 8> :33 £5: 35232: a 38: .826» 6:5: Eat Ewo— 8.3m «...—:34 m ”828030 3.20 3> ”.8383 3122 92 Ecuazowoc .305 3.880556. #3 ma 25:0 Snowy. 33.3.5 .53— N ”mama—W2 .8 80:3. :2 E9806 8> ”coEmEgiO coca—BEE. £3 .52 5 mm 2.22.2.3— m ”89. :25 ..Z :83 SEE 90> £03on5 Beacons—B ”.3 =a «a mics—n N £82.me oz 6:895.— ztoaen—AO az “moo; toga—5E o\evn ”53355 .2 ona 93 6:0th 3.2—ca £58330 .3238 c 3.8 so? :5 ”mg—336.822 mo> ”moomtomEouha 2 3 ”Staci—088:— .20 55 $3 8: :5 £8» m-_ you in 8 2.5-5.“ v ”£5 BESS 8> ”2&2 oz ”BE—mod .206 Am ”A . m ”a .w ”UV waste? ass—3:8 .EL 8 moocotoaxo 39» 2.7. m ”=o_mm=8_v 53h 8> ”Bazaar—3 90> ”3385 gains—Am 3 ”28m wan—EFF 32.3. Ea...— 85333. no 2:: 05 m ”mp—33x a Z ”53 AccowhoEmQ :2 Stem? cm-nN .m 6w< 3 man—u 95 5:6 8: 23 $20 05.5 Ea 38%... DP .5— ...ovau 0:54....— 932 auuzwtg 3:9? gm. 3.39 383538» Sufikw 535.3% ms 035. 222 $63538 3&3» momma—0 was—8 282. Exm .8an3 was 32:. .350 cm u my. cocoa ... oifiucfl 3.3m ”Eu—=3... _ Logan >333— N n v— 53 528 25:5:82 _ acofimsuflv 30.53— o n m .wEEmu. .5.. sac—2:95 $2 ”2238...; :um Ev 35: :5 .moscEsos mEEno. mm =u o ”cogs—agrom .5322. mm in wow: . £5 a San 0E: £ch 93 an... 53 33.568 _ Eofimng «ma—U ez UwEEw—{fiumfi 3mm uEEo>PEE_ A3 95% mic—:2. 3.35 $553. 985 .5qu .o: _ ”32.330 3.20 :2 ”2m 03558 :2 NEE .25: $3 ”A? 2.. £96 a 8:: =23 x85 Sagas". 8: :5 >2. _ ”9.63.. no aoocox :2 E2292. 8322 92 ”cocazowoc .305 $3 ”.3 «a a SE: .238 quz Eoc E wo_ "2.0m 9:534 M ”So: tozm oz :9ng c2 EociEtufiO gem €283.33 mm Esme. 32:3..— .=um N ”83:8:— 92 Edi Ema oz ”sohwvmuy—E c. R ”coauoEoobE— 2”. n 2.5 35 ..z ”25:8 €09.56 :2 ”$8.53“ again 3 a s um .2 Q o. 3:22 8: 8&2“ biutcgowozou 28:335— m 6.2: 652:. oz nucuafiEooUAZ oz ”801C828 ...R— mm “28m 9.35.; 252.3 wEEmoT m ”coamsofi—u 53,—. oz "2%: 92 633:8.“ 206 mm; m .n newt. .22.. 2..— ofi E muucutomxu PE: 8: 88 I m ”9%qu oz ago—aims oz ”838.: Essa—£3 ago—=5. 33% e __,w :87 E. .2 6z u...“... 559059 .3» 53w? 251.... 283 mmgu 05 was? 2&3 :Em ESQ—=8 on: "Sat .5an Roman x35 :28“ .3 3.58:2 ”015 uni-...; m Conan >530.”— om n AM 2E. auto—>65. .ophmov 5.32 32.95:: m $53386 30.58. m n v— .mosgcsoou wig. my. =a 3.: GE: M ”233035 m u m 23: can wEEoE 95 E 9:53. asp—m £3. Suva _ ”nouns—«>0 tom 02 ”w=_=_g\.:vmfi autism 96% mo .2 a Ev =5 .mEan. anew v E28388 awn—U :2 ”2m uomb>OAm 8> ”cu—m dam: 3.532. 5— SN human .0: on 5:528 8: :5 >2. a 0E3 .233 m ”32.3.30 93.0 :2 E0663 812? 06> "couazowoc .32: .\-a acoEo>895 m3 350 _a:o§\qu._ 89c Ewa— "oetm NEE—nod mum—.63.. :o ”coco: no> E9808 92 EoEwicofiO $8 AS» a o==:o\._a_=m2 canto."— .=ufl N 68: team 3> ”:2; Ema 8? ”seswoxa $2 “a: 2"— m uofiEfiE 8> 6:853 b.5356 no> ”3233. tonnamfi Quan E2335: .5— 8 v8.22 8: coho“. basic: two—Bo nae—«3:3— m u=8 no? 00> umzcufiEooUAZ oz ”mooEumEoohAQ 2R E28258»...— EQA a 56 v.8— 5.“ 555...,— ouanm ma wavy—.83 N ”85 SEE 8> ”En—A2 oz ”moE—Bfl 206 An N.— .o ”M .n ”UV £825.80 .23 25.559 conga bioexouwnm v ”cofimnumg 53,—. oz gas—haiku— ez ”co—<8:— ESEa—fim 3 ”28m wExEE. .5326» mm £8» 24. .5.. .2..— E moucutoaxu 9:2 1 m ”minnox mo> “cu—n 3592—50. :2 5..:quV 3.3 .N Hom< $20 28 85E: z DH .5— Eovam oEEuu augutuakm 35:6 bgh 930 «835289» :39? .9355.“ ms 2an 223 E63538 ~33» 8330 mix”: 222. :3... 659:8 :2. "3a.. .350 :28: .3 a. ... 52. 3.80:2 Stew "5..-...... N 5%.. >633. 224 2:: $39: 632305. 60%“... SEM... E23332 m on u my. 2.9 32> 3323: :96: :5 8:25.08 w:.E3. ”E2332: >633. m n v. my. ..a com: .23: 3 0:2: $5.53 5.0:. 133 c ”2258:; v n m a 2:9. a-m $2. a 3:0 8 a: .83 a 8:... .896... . E23286 ...om BE: .2022. .052. E2. Ewe. 8.5m «...—:34 . E23286 v.35 8> ”3.23.:va 332.2. .5. 2m >25 .23.. M ”828030 9.3.0 8> ”2". 322,03 :7. Ear. .msm: $o~ Eon—395E. m3 :5 .m:.:u>o 223:2. .o:._:o .5:on 32:3... ...—Km m $5.32 :o 82.3. :2 E228: 8.223. mo> E22285: .305 $3 ”A? a; 2... N 68: team 90> Emaooa o7. E22m.:vo<6 $8. HA... 2 33.2 .o: 02on 282:5. .0953 E23935 . :22.me 8> Eu... 2...... 8> E0883... $. m E2325: 5. EoEqunE €2.05 :5 .5... 30:52. £me 2“. n 3:3 no? :2 ”2.29:2. €2.2on :7. ”825m toaqsmm. c. R E280::ou.§:. 3.8296 .8238 46.3.0538 .20 .Eofisoov .22 v ”2:. 858:. 8> ”3.822892 :7. ”mootcomiooio Am 1. .o a. 6 ”UV 5.3 .8: .o: :5 .38» 2-.. 5.. SE... 8 €3.53... m E23586 Emu... 8> ”SEAS. :Z ”mozzfifl ZOO ... 6:03. w:.x:..F £228.. .805m .2“. :. 30559.0 .39» m... 1 n $233. a; “.m.u:a:.n.3 :7. ”0222:. 282323 913 .v Bw< 8.9.3.0 v ..a “.00.... D... .5— 8> E2: 35925.0. an; $2.52 E253. 0.3). am u my. momma—u wig—S 222. ...xm 5.2.88 22. "3a.. .350 v u v. .38 v.33. :28: 2 0282.2 31% "ESE—F m n m. 2:: 630.305. 683.. 3:3: E23235. m 22.2. 32>»... 832.2. A”. a." 8:25.08 MEan. .3. m .339». ..a :8: E... a :5 E: a Ban 2:: £9: :5 >2. 52. ”E2328... >633. .83 ..o w2vcsmcov§ 9:53. 95:» 8.3:. £33 a 2:9. 2-0. 5% a 023. m ”2328:; 8> ”9.22::va :. 35:52.2. 8E: 3:022. .082. ES. :. we. 8.5m 9:53 N E23296 ...om oz ”.2... 822,06 an; Ear. dam... 32.83 $3 $3: W. 853 :05 £558 c. 2.2:: m E2338... 820 an; E228: 833.3. 8> E23303: .303. 5583838. E5 £3.58 m:.:o>o 055-5: .o:..:o 3.8.3.5 .....M N ”8280.30 320 an; E3898 a; E22223<6 $8 ”$5 an s: m ”368. .5 soc»: 8> ”5:. 2:: a; ”50330:: $8 a: :. 6235:332qu b.m:o>.::\owu=3 2.2.3.5”.— m ”30: :25 a; ”2.295.. €2.8on an; ”8233. toqanmfi. $8 E28253 .5. 8.228: 2... N ”3833:. :7. $23228qu 8% ”82>:omEooZQ c. k. E280585E. 3838 Ba 2.35.280 as, .8.“ a: :5 £8: n 2.3 83 n5 U2.5,. 8> ”82.8. 206 .m A .v u: .e ”u. n... :2 2... ma wcctoaz E23260 $252: v "2:. «052:. 25 ”.a.o:a:.n.3 8> ”832:. 28:35.5 m. ”88.9. 33:2... 8an .2“. .2”. :. 80:32.5 80> n. 55 29:... m E2332: Ewe... a; Ea... 55925.3. 00> Scam; ow :26 .m Hom< 8820 o»: 3092 m $2.33. .525 2:3. DH .5— aucutwfixm 3.5.? Egan 989 38:538. SQQKE ASE-55 m... 29:... «00:50:00 030.. 00000.0 90.0.0. 000.000. :20 02:00:00 .00.. "0.0.. 0050 00.000 :0 Q ... 50.. 3:00:00 "0.5m nan—£5. 005 030.306. .000w00 02.»... ”00.50.00: 0006.00.02 3.500. .5. ..0 00: ,w:.E00. 0:00» 00.00:. 0.0.. 0 00.30 ..0E0 0.02.0 0:0 50:50:00 00: :5 $00 0 00:0 080.. 80c Emo— "0.bm “...-.0004 0.000 ..-. 00.500 00000 .05.00 308.3..— ...—E v £2.00. 3030M N ”05.00306 3030M $2 00.000.00.05 N ”00.00230 0.0m m ”00.000006 000.0 m ”002000.30 000.0 07. "2... 00002.08 :2 ”055580: 07. ”=0... 0.5.. emu cu mu and: 3:22. .2 0.0 $3 000530.58. m3 225 080...... .0050 ...w... "00.00003— v “90.0000 :0 000:3. 02 ”00.0.8.0 00.025. 02 ”00.00.0030 000$. {owe E3 N; 003.500 m ”000: tosm 00> ”5.0090 07. ”00.0.0.0.00fic $8 ”05 SE 00000>0 0:0 60.0.80 90.0.00: .00... b...00.. N 0000.500. 0.7. ”00.0 S. EE 00 > 2.000000%". foam “00:00:33 .5. .mEEEEooo .00... 3.0.00.5 $50.00 60.0 0000.02. . ”0:00 00>» 07. ”0:00:00. éoQEEAO 00 > ”000.200 roman-mm. o. R ”00.0000008000. £500.89: 6:80.00. 3000000 40005000 .000. n ”00:. 00000.0. 00 > ”9:80.030qu 0 Z ”0003000200....9 G 0.. ... ”a .0 ”U. 0030.05.03 5.3 .000 00: :5 0000» m. 00.. 008... v ”00.000006 E00 ... 07. ”SEGA 00 > ”00.....000 S56 NN ”0.000 90.0.02... 00 macro? ...SE. 0. 00.00.0096 000.. m. :05 00052 m ”$5000. 07. ”.0.0§=.EAA 07. ”0030.00. 5000.028 9. 00>0 .n ”0m... 000.0 000 0.00...) DE. .5— 00> ”=0... zogwbfimg 02 Scam? 000.030 08.0-0000 0.0.2 00000.0 wigs 000.00.. :30 00.00500 00... "000—. 0050 00.000 ..0 Q ... 50.. 03000:”: "0.5m «0.0.0.0.... 0300.305. "00.002002 00035.00. wear—00. my. ..0 .000: m ”02.0.. 3030,.— ..m u my. .800 00.0.0 Em... 00 3:500. 000..» 0000:. 6.0.0000 00 0 ”00200000... 30300. 0. u M :02: 00 ..080 0.02.0 50:80:00 00: .3 S00 0 0:0 . “0.0030005 m u m 00 a: 0.003 0 00:... .3030 .000000 0.30.. 0800 .000 . ”00:00.03 0.0m 07. EEENEdvmc. 0050 .0000000 0. 050... 800.. 0. m0. "0.5m 3.500.. N ”00.000006 000.0 00> ”SE 00002006 00> ”00E 0.5.. 00300.00. my. Sm 0500:. 0.. mat—.00 N ”003000.30 000.0 00> H00.0.8.0 8.02?— 00> ”00.00.0090 .0005 $0. ”000803008. 0.? :0. .000 £080 ...00 wig—.0000 $0.500 ".0:—.3..— .aufl N ”90.0000 :0 .00—«0... 00> ”swig—A0 00> ”00.0.0.3.00fi0 o\oNn 5.3 SE 0 ”0.0: team 00> ”0.0.0. SEE 00> 2.000000%... e\coo. ”A: 0. 00000500020300 3.000>.§\0w0..00 2.0.0320m N ”0900.000. 00> ”2.0.2.5.. .0000.ng 00> ”000.300 0.2.965 $3 ”00.30.33 5. w:.:.0b 0:0 m 0:00 0.03 00 > $5000.00000AZ 07. ”00330020020 o. \o. “00.00000000000. 00.3.0500. 5.3 .000 00: :5 .0000» ...-.. 00.. SSE v ”00:. 0050.... 00> USEAS. 00> H00.5.00. S56 3 n... .c J. ... ”U. 00 $0.831 $0.000... E80000 000 90.50... 00000 n ”00.00.0006 800... 00> ”3.000503. 00> "0030.0:— :.0E.0S..m a. ”00000 wag—...... .SE .m< .SE a. 000.00.02.08 000» m. :05 0008 I n @5003. 00> ”00.0 000092.50. 07. Scam? ov 00>0 .m 0m< 00000.0 0 ..0 0.00.7. DH. .5— 0:00:00 0.02 000:0...853 035.? 535.3% 0000 «00:50:00. 30ka 333.03% ms 0.00... «0.0322200 03:: 226 3% 2.. 95.3 0. u .2 0.0.30 :30. 5.00800 02.: .0: 0:0 "3.... 0050 v u M :0300 :0 Q E 500 02.00%: "0.5m ”50.5.; N £09.: 3059: N u m 3.500. 0:__:0 wcib 6300.305. 30.32.02 _ ”0:060:86 3030a 00:35.00. mEEfi: .2 :0. 000: :050 ES». 0.0: m ”06.50005 830:0: .3 in wow 30...: :5 $5.50. 9.0% 5.00:: £003 a 800.. _ 50.32.2623 $2 2.2 $00 a 0:0 050: 80.... Emo— .0_bw uni—3 v E06386 mun—U oz .mEEQxfifiQ ”E08383: m3 0:::0 £00020 ...—:3: 33:..0h ...—.m— m ”00>30030 mam—U 02 .2". oofluiw 02 ”:03 0.5: $3 53 2..— muwfivmo: :0 60:3— 02 50660: 8.022 00> ”:23003: .39: $3 ...: v; E 0935:020230 t_flo>_:=\0w0=00 ":0300avm m ”0.0: tonm oz EmfionAO 00> ”:0_._m_=_00 :03 SEE 02 ”nanomomfi otm .:0:00::00:0.:_ 0:: .2.— .S...” 5530:; gran—08 £050.30 5.? v ”0:8 903 02 ”0:02.00 EEAAO 00> ”0003.50 €25.68 3 J .m ”m K ”UV wfifizowo: 5:0:65000 0030.5 £0033... 0.0.5.6 m é:— 00E0.:~ 02 .mfififivsonxz 0 Z ”002300200KD 3 ”0.000 wExEFr .20 00.. 0.920%“: .30» _ :05 00.0— 5.. 005.. n E08386 E00... 00> “SEA—2 00> ”003.36 206 3-3 .N .0m< 8 wife: .2.— .m< .2.— :_ 00:05.05 :00» m4... v .3300... 02 “30:05.3. 02 ”00300:. 585025 0:003... mam—0 0:0 0.00,—.1 DH. .5— mo> ”:03 5:090:50— 0z ”.3an 050.5: 0.080..— mommso wExS 0:80: :0... 850800 00: .300 35¢ :03: :0 d ... 500 02.00%: "0.3m 9.25.; m p.09... 3030.”— mm H .2 0300.305— .»0:0E "00.32.02 _ ”0:23:02: 3030..— N u v— w:_:..00_ .8 530205.200. 20 :26: :5 £03.58. N “20.50005 N n m w:_:..00_ .2 =0 00.. .mEan. 000% 0000.... 50:50:00 o Bogs—go hum 02 .mEENEéva .0: 0:: >00 0 :00: :0 :05 $0. $00 a 00:0 _ “:0800006 mam—U 02 ”SE oombim 02 ”:03 0.5: 3320: .5— 9m 0080 3:00.50 5 0:8: Eat Ewo. "0.3m «5.2004 n ”003000.30 awn—U 02 60300.0 00.022 02 50.3603: .0002; $em 35830095 A? 0030.0 3.03: 30.50..— .30H m ”:63: :0 .oocom 02 “5100990 02 ”:02306330 $3 ”.5, n; 2,. v .30.. team 02 an... 2...... ..z 5838:... $05 ”.5 0. 020.0: .0: 8&0: £00035. Emu—.00 ":030053— n ”00.0.50:— 02 ”0:80.00 €000..on 02 ”002.com toaaamfim $wn 60:32:: .2 82:80 MEESESE v 0:8 00>» 0z ”3:26.822 02 aBEBESho 0:0 .:0..00::0eo.:. w:_08000 0:: 4:05:00: 60:05.85 35850:. n 0...: 00525 02 ”SEAS— 8> “fizz—Emu SCAD G ”A .m ”a 6 ”UV 3:0 :0.“ 03.30000: 5098.... :0.“ SE. 0.. v 5050.506 Each 02 ....BSEES 00> ”0030.05 £850.25 2 ”0.000 9.355. act—0032 £050.: 0:0 .2.— 00 80:05:08 :00» c. .7 m $5000... 02 “:2: Aofimbfimg 00> Scam? mm 50:: ._ now< 820 0:0 0.00.7. DH .5— EocBm 0.02 0.00.3003 83:... gm. 0.89 30035300» 509? 8.3.555 ms 030p. «$35300 030: 227 $20 05 wfixs 282. :03 559:8 02:. "3a: ..050 5:8 :0 d :_ 50: 038:2 8.3m 395...! mm n my. 05: 6300—305— ":02550—2 m coma 3033— NH v. wEF—ao. :0.“ guns—go row 20 :96: :5 £32508 _ 6:23:86 30:5."— m u m .2 :0 00m: .2535 “a 5338:. 95:» 5.3:: 00: _ ”06:88:; .3330— a:0._m 8.3.0 .0: £83 a 8:0: 3-2 in: c E23293 hum .0322. .3 2n 0 02:: .838 050: 50.: 5 m2 Seam «5:53 N E06336 mas—U 0z ”misgxsvmc. 3% 00.500 m:_:o>o 0534.3 65:0 3350..— ...3— m ”82.00.30 mam—U 00> ”SE £39m 02 E05 .35: u:qu>0.EE_ .5» 05 22:03:02 80525 E E0680: 00329— 99 Eozazomo: .393 $8 ”.52 88» m; :0.“ 803.33. BE... m: 93.831 m ”30: «Sam 02 E3890 02 Eofimigfiw $8. :3 .858: .038 as .5.: 0%: :2»... 0.2.8: n ”.3038— ..z ”5.: 2:8 :2 ”sessmc $8 a2§=5 .2 ..EwE Eocmé 5033200 $550: .5.: 30:63 n ”0:8 :03 02 6:0.an 3.500:on 00> ”303:8 €0.5sz 2 3 E0:00::082:_ 6:023:00: £0882 wag—0:58 55 .3: 00: n Eu:— BESE 00> $535300qu 0 z ”moomtomiooha Am ”A .w ”a .2 ”UV :5 $50800 gum—80:03 57.30.00 0 :0.“ 3303 m E08383 Each 00> ”2.52 00> ”82:2: 206 mm “203. wfixfifi. .805» SE :_ 0:558 0:03 m35toaxo .30.» 2 -7. n ”wfivuom 02 ”Basins 00> ”0038:— :55023 av :26 .muow< mama—o v :0 0.00,—.... DH. .3— uo> E20 5:090:50— 02 Scam? 0:008.“ 20th 3.20 05 wExS 2&2. :07. .2:an0 26: :30: .350 330 0.85 :058 ... 3.80:2 8:5 use—:2; _ coma: 30.53— pm n ma ago—30:: .8..on 85w:— E0ua>=02 Nnm:0_mm:8m: 3033— v u M 9:0 82> 003032 05 9:53. 5 gums—gum? 0 ”295000; N u m E: :26: $552 mEEmfl $qu .>_0:Sm:00 .0: c E23230 row 02 "MEENQEBC. 0:: an: a 50: :0 :05 m8. :5 $3 a 3:5 .333 _ E06386 man—U 02 ”2m 08:on 02 E05 .25: 8322. mm a: 0050 3:88: 5 2:0: 50: Emo— 815 ”...—:34 _ $2600.30 08—0 02 E0660: 8.029— 02 Eozuaowo: .039: $3 u:oEo>0EE_ A? 3500 3:5 ".0:—.5.,— Eam _ $5008 :0 “cocoa 00> E3956 02 EoEmE—uofio $8 ”m3 m; E". N ”22. :25 .2 as: 2:: ..z ”5.83%". $8 ”.5 5 08355020: b_m.o>_:=\omo=00 Eecuuacm _ £20.50:— 02 “0:032. €002.10 02 ”803:3 tan—awn.— .xfim E08333 mm .90 028:0: m ”0:8 00>» 00> UwEEEEORXZ 0Z ”moongooHAD 3 R E0500::088:_ 03:35: 09103 2:05—0:08 :5 82:8,: Ewan: m ”0.“:— 8585 02 ”SEAS— 02 ”33:20.: 206 Am J .m um .w "UV 3:0 :0: 03.80%?— .an> m.-— :0.. SEEP—wfioo m E28586 Each 02 ”365:53 02 ”0038:— 555225 E “0:03. wig—EFF :5 m: MSU—.531 EEEBEmo—u .0:. ma E3» m; 1 v unfit—Nod 02 E20 aostEmO— 02 Stem? mm 000:: ._ 6w< mam—o 0:0 0.00,—.1 DH 5— .5030 22:0... 03:02033 522.0 @3555 0.39 30035300» Sufixw ASSESM‘ ms Bank $035.28 038. 228 03.0 c 5%.. 3030... .m u my. 0:. 95.8 05.0.. :30 85:88 02:. "3a.. .056 . ”0:200:00... 30.?3. m u v. .080 0.003. 5.80 ... o>.:00..2 "0.3m mac—:3... . 6.23.8.0; 5.0.09 o u m. 08: 630.305. Sec-$.32 c Boga—.30 ...0m 5.08.3919... C. :03: :3: 0.: 33030.. .05 5.23.96 ...0w .0... .08: ~ ”5.33:: 0%.“. <2 ”3.503.:va 8.22.0: n.... 9R 0.2.0 02:: .05: $20 a 00.25 ..080 0.00:0 $5500. . ”00280.30 08.0 <2 ”2.... 08.303 <2 “:0... dam: o\oc ”30805.58. .5? anew 5.0:. 4.003 a 0:8... a-n $0: a 85.. .3260 . ”mp—.30: :o “cocoa <\Z £03.00: 8.03.3. <\z 60.8.83: .32... $3 £3 a. n 00E: 3:00:09 .089. ES... Ewe. “0.3m 9:53 908: team <2 ”@3008 <9. ”:0...m.:.00<5 $8. ...... 008:8 05.5 .05 5&0.— 3-83... ...—0... . £20.58. <2 ”:0... SEE <2 2.20000th .35 “5.30:5. my. 2.. 9 _ 0:3 £3 Sz 5:35.. E336 <2 ”82:8 08.556 2. a ”8:85.852. .0220. 8: 20¢ 2... :. 0030A. m{3.0.0.2 Sacco—...”. . H8:. «052:. $2 ”3.35.0.8qu <9. ”000.200.2820 Am”... .vny. d. ”U. £00» m. 8.. 5.00.0 5:85:05: £8.33 N ”coinage E3... <9. "SEA: <2 ”00.....00. 206 o. ”088 w:.0.:.:... 8:0 .m< 00 3.0.83 .2". :. 0005.330 0.6.. 53:1 m ”3.30M. <2 ”.a.0:u:....3 <\z ”00.0.0.5 £550.25 av :26 .m ”0w< 8mg: q .... 0.8:. a... ...: <2 an... 5505500. <2 ”:55? .52.: 25:3. 2:2 mm u mg. 08.0 . p.092. 3030“ n n v. 0:. m:.0.8 0.5.0: ...07. 559:8 02:. "80: .350 M ”0:23:00... 30.5... N u m. A008 0.003. 35:02:: u0.3m a............. $2 66330:; 0300.305. Eocgzoz c ”:o..a=.a>0 ...0m <2 ”€8.03...va 330:0: 5. S w:.:._00. 3.. 5.3203 ...3 2.0 :26: :5 6036.500. m ”coinsom... 08.0 <9. ”3.“. 08:30? <2 ”:07. .95: $3 u:0E0>o:.E. m3 .5. :0 .000: $5500. 9.2m 5.0:. $0.. a 00.30 m ”003.0030 man—U <2 £05.00: 8.03.3. <9. 50.8.83: .30... $2. “A3 :05 0.5:. ..080 0.00:0 4.003 a 052. 2A.. in: «amigo: :o 80:00. <2 ”$609.0 <2 ”:o...m.=.000.. "30.. 0059 N ”.0000. 3030.. oN n my. :280 :0 Q ... 50.. 03:00:00 "035 ”0.0.2.... N ”0:200:00... 3030.. N u v. 50.. 000000 630.305. 30.00.50: 0 02.000.00.05 N n m. .2200 0:0 002> . E28205 ...0w 3030: :30: :5 00:25.00. A... ..0 00: 62.00.30: . E20030... 000.0 02 $2202.21"... 0032.00 n.I.. 0.. ..0E-0 03:. =00:82:08.0 :0 9.200000 ... ”002.0030 000.0 02 ”2“. 0000303 07. E0... .03... $0 0:0—:3055. A.3 30.0 0 00:... .000>00 0050 3:00:00. .082. 800.. N ”@600: :0 60:0... 00 > E22000 00.029. 00 > E200..0w0: .000... $2. ”0.3 0N0 Emo. 60:80:58.0 :0 w:.0:000.0 >00 0 005.. .0:0>00 m. ”0.0: :25 0 Z ”5.03:0 02 E0...0.=.00<.O $00 ”0.. 0050 3:00:00. .052. :00... 2w0. "0.3m «20.304 m 0.0.0.505 00 > E0... 2...... 07. ”£00000om... $9 E2025... .5. 02.:0 30:20... ...—0”.— m 0:0. 003 0 2 ”2.0.0000 E00056 07. ”0003000 tonqsmm. c. . 0 E0..00::00:0E. Eu. 0. 0000.0.— .0: 00..»00 0>0.. E20025”... N ”2:. .0E0E. 00 > ”m:..0:..:000.z 07. ”002200200...5 30.. .03. .0. ”U. 000» 0 00.. 00:00.: 00000 00 w:.0.:03 m E20030... E00... 00> ”2...: 07. ”022.60.. 20.0 0. ”00000 30.2.... E3000 0200.2 0.0 3.0. 2 035.0098 000.. m. ... m ”@600“ 07. 0.0.2.0203. 07. ”00.0.0.5 20.2028 N 000:: .. ”0m< 000.0 . 0.00.». .... .... 07. E0... 30:0w002m0. 02 0.252 0:0—030 0.050... 000.0 0... m:_0.0. 000.2. :30 00.00800 0>0.. “000.0 0050 :0..00 :. 03.02.00 "0.5m «20.5.... 2 u my. 0:... . ”00:0: 3030... N n 0. 002.0 i 00:00.... 0... 03 00000.0 ..0 2:30:30 . ”0:200:00... 3030.. m u m. .50.. 000000 .0050: 02.»... 30.03.00: 0 ”2.50.0003 2.50 0:0 00.03 3030: :26: 00.0 o E20030>0 ...0m 00> ”m220bxavmfi 0030500. 0... 3m 0:0 :2.0:.0>0 ...00 0.0 00.00: $2.000.— :0 002.0.— 00>0: . E20030... 000.0 00> ”.2". 0000303 00> 2.0.0. 0.5.. fame .000: 00.08 00>0: .00>.b0 0. :23 2005020200500: . ”003000.30 000.0 00> H020.00.0 00.029. 00> E20289: .000... 35803092 :3 a... ..0E-0 0.00.. 0:0 £003 0 0.50... 0-0 0.0.0 0 00.3. .000. o ”3.000: :0 000.3... 00> "5.05.0 00> E0...0.=.00<.O {owe ”0.3 .0 00....0 000:0 . 020.. 200.. :30. "030m 3.500. c ”0.0: :25 00> E0... 20.3 07. 2.000000%: $00 ”.... 00000.0 02.:0 0:0 8.0m“: ”.0:—:0... ...—0m. . 0.0.0.505 00> ”2.0.0.2. 9008:.0 00> ”0003000 tongsmfi. o\o. N E20253 my. 2... 0. 00.0.0: .0: 0030.0 0 0.00.. E20320“... . ”0:00 0.03 00> ”w:.00:..:00HXZ 00> ”0003.50200KD c. \m E0..00::0000.:. 0:00.. . E..:. 00:08:. 00> ”2...... 00> U022.60.. 200 .04 £3. .20. 0.-.. 00.. BE". 00 9.0.003 30: £050... 03000.00 0:0 m E20052: E00 ... 00> H.0.0:0:.....\. 00> ”00.0.0.5 20.2023 N. ”00000 2.0.2:... 002.. .0000000200200 00 000.00.00.70 0:00» ...-.. ... v ”3.000.. 00> E0... >0:0w00:.m0. 00> ”avam?‘ 2”sz .N 0w< 00000.0 0 ..0 300...: D... .5— 0:00:00 0.0.). 000.0012?“ 3.5.? 335.3% 0000 300050200» 3093 b08234 m... 0.00.. 3832.228 ~33» $20 05 mac—S 882. :2» 559:8 32 ”3a.. .359 .528 .3 can ... 52. 0280:».— 33m wan—EFF N comma 3325 mm H mm 53 883 69.2265. dogma“. .23: Boasts—2 N ”mp—23:83 Bo_>om v u v— 23“ 32.3 >632 53: :5 833:5»: c 6633035 m u m .3 =~ 8: Saw a 923. 55 22: :37... :85 £83 _ ”cogs—gozum a Enos m2: cop—Sancho co wEvcoaov v.33 _ E23386 820 8> uwcmfigxacmfi 8322. .3 9? a 3:5 .833 0:8: Eat Emo— 335 "...-:34 _ ”8360.30 3.20 8> ”2m 8§>O$ no> ”5E .25: ..\oo acufio>oa§ A? NN& 93v Nu— ofiaoQ tonm _ ”ME—V3“ :0 80:3— no> :87.ch 832E 8% 62.583: .39: $3. #5» 63:8 @396 uEzéwq Jaime“ can—3.3 ...3— v ”Bo: team 8> E2306 8> ”:oEmEcofiO $8 ”A: E 8 33.2 8E3 3.83835 "aozaaum <\Z ”8.25m:— 8> ”ca—q 2km no> “nugomfi {ohm Barnum—:3 my. 33» _ 5:8 no? :2 628:3 €38on £5 60233 toaqsmfi o. \2 ”cozouccoeoE— 2-”. .8» 2m d 5533 3QO a ma 2:: :3 wife? N ”85 $585 8> ”$535.88qu :2 “moumtumEQoha And 6m .98 Bo: £2.53 0&3”? v5 5:529 08% .SE .35 E m ”coimaoflv Each mo> ”SEAS a; ”8:58.“ 206 2 ”203. chEE. .3836 .558. 3 805530 30> m. 55 22: 1 v ”95.8”— 92 sauna—HES cz UoE££ E85328 av ..26 .m Hom< momma—o m sop—.1 DH .5— 8> :83 hocowbEm—O— :2 55%? 2.0.53. 0:35,.— 320 05 mac—S 282. :3.“ 859:8 02:. "3a.. .850 5:3 .3 Q ... 52. 03832 8.5m Mai-...; ego—Bocéoohwo“. 552 33.3.3: o Conan 3033— ON H .2 06.5033 98 693 >632 _ ”mcoamsoflv 30301 N n M B>o= an :03 8 5633.6 320 E «592:3 9 663335 v u m ago: 98 .80: :05 SEE .55: :5 $3 a 336 _ ”scams—go tom 92 uw=_:_g\.=vmq 55 03:. 32:6 33:0 dogma—:86 co mic—Baum. c E23386 man—U 95 ”SE 03.553 8> "5E .95: .5323 mm 3v >3 a 88: 365m 350 .acoflon .oEo: Eat 32 ”82.330 «ma—U 90> ”scion—u 8329— 8> E23309: .323 {ca 3552,2qu m3 Ems .3:SmE:8_o co mE—Eonuv >3 a 3:5 .838 _ 9:32 .8 80:3— 8> ”58006 8> EoEmEco N53 SEE :2 ”50383:"— .xéo. ”a: 25:0 3.3.3.,— ...3— N ”58:me :2 528:2. €38on 8> ”mooEom tong—65 $3 E23353 my. 2..— 9 v8.28 8: 8&3. a 26.. 32.353— m 3:8 no? a; ”mataEEooUAZ a; ”m3_tom28.r5 o. \w Eoc8§882£ as» m ”as .0535 3» H2%,. E» ”82:8. 206 :3 .3 .98 m; .8 5533 08% a ma mac—53 ..mzflocomEE—um m ”cofimsomé 88,—. oz games—«Ems 90> ”838:. 53:33.8 2 ”208 95.55. 23 2m .SE E 380530 .89» 2-2 I m ”3:33— a; an... 559089! 62 ”33m? cw ~26 .n Hom< $20 _ 38,—... a. hy— Eo—zzu 25:0..— uwuavmuumxm Sufikw 3.3555. 360 333538» Sufikw 33.5.3.0 m6 Bank 230 $055200 038» 231 000.0 05 9.58 000000 :30 00000600 03.. 080—. 0050 _ ”00000 3033— mm H 5— 00:00 a. 9500:00 "030m uni-...; N ”002000006 30:03— 0. n v— 05: 00000000020 .00 o ”05030005 Nu m 3:30:90 630.305. .0230 005m... 2.0.3.502 _ ”song—«>0 .=0w 0:0—50003 30:6. N ”00.00.0006 0020 02 ”maggxgmc. 003200 .2 93 ~03: :5 0009502 my. :0 .000: $00 a 0035 :05 _ ”00300030 0005 02 ”2..— 00000>0Am 00> ”00E 0.0m: $o 000530000: .5» 000:. __0E-0 0.00:0 0:30:00 .00 :5 0.003 0 0035 m ”$000.00 :0 80:3— 00> ”02080.0 0302?: ”0030003: .0005 $3 "A? van 0050 3000000 0E0; ES.“ 0&2 "0.3m “...—:3 m ”80: :05 02 Ewauozv 02 ”002030310 $00 ”m..— 0000000 00:00 000 3.0%: 3.2—E0..— .56H m p.000—500— 00 > ”00.0 Ema 00> £00800MC $mv ”00:00:23 AM 2"— 0. 020.0. 00030 "00:00.5H m 0:8 003 02 ”0:00:00 3.0000on 02 ”0003000 0000963 2% ”00:00:59.2:— 0000» m; .0.“ 000020 030.800 N ”0.0:— 005005 02 ”$505900qu 02 ”0003000200.:D Ann‘— di .98 0 00 wfiutoi €830.50 000 0005.0 0&00030 0. ”02000005 800... 00 > ”2.52 02 ”00.0500.“ 206 E "00000 mac—EFF 50:03 00000 .000830 00 000020098 000» m4 ... v @5000". 00> ”3.055%; 02 ”0030.05 £85025 cmfim .N ”0w< 000—0 _ 300...... DH .5— 00> ”00.0 50090059 02 Exam? 000.500 0.080..— m ".0000 30301 9 H mm 0020 N ”002000006 3030M v u M 05 mac—S 000000 530 0050500 02:. "Sue ..05O _ 0062:0003 m u m 00:00 :0 0% a. 50.. 90:00:00 "0.5m use—03h. _ ”00:00:30 20m 02 ”ME—0.0.9039. 00090 00.32 000.0950: _ ”00.00.0006 0005 00> "2m 00000053 02 ”=05 dam: 003200 .3 2m 9:080 00 000:0. 026: :5 00:90:00. mm :0 _ ”003000.30 0005 00> ”0200000 00323— 8> ”005303: 000$.— $w ”0:080>0qu_ A? 0005.0 000: £00 a 00:0 .380 0.00:0 4:80:00 00: c ”95000. :0 000:0“ 02 “0900980 00> ”=0£0_=~00 6:8:00 €000..on 02 ”0003000 toga—swam— $ov ”00:03:03 AM 00000.0 00.030 canton ...—m.— N 0:5 003 00> EESEESUAZ 02 ”0003.000200PE 0:2 ”00:00::000005 .2... 8 002200 00030 a 30:00:3— m ”0.“:— 005005 00> ”20:: 00> ”005:0é 206 Ann.— .aum .2 ”UV 0000» 2-». 00¢. 00:0"— :0 m H0200000.“. 800% 00> Higgins 00> ”0030...: £85023 3 ”00000 mac—:2... 00 mac—003 .SE 5 000020098 80» m. :05 000.: I m ”mcfiaom 00> ”00.0 hocowbfimg 02 ”>005? 319” .v Hom< 00000—0 v =0 0.00.7. Du. a“ 0:0—=50 0.02 000.00.00.03 Sufixv ASE-50% 0000 300320300» 309? 335.05% mg. 030% «835:8.» 833» 232 m8.”U an H mm 05 main. 883 :63 859:8 96: "5... 8:5 macaw; 3033— N u M .538 ... 338:2 8.3m nae—EFF N 5:23:86 3033— n u m 52. 828 603338 8.32 32.3.32 9 ”3388‘; 23.33? c E28296 20m 8323 mm Sn 30:6. 28 conga—«>023. Ba .96: $3 _ E06386 man—U oz $55.53va .83 a 3:2 55 29: =~Eb :85 £83 a 952. ON N ”8280.30 man—U 8% ”SE 33053 8% Bar. .25: ”EoEoSEE m3 -2 8053586 no $52.33. 56 m 8E3 .233. m ”mics.— :o 80:0“ 8% Eofifiov 33.23 oz ”catacomo: .305 $3 ”m3 9N». 350 328.3 2:2. Soc Swo— "oEm 9:58; m ”32. team 8% ”@6005 8% ”:oEmScufiO $8. K5 0.5.8 8.5.3..— .25 _ ”33:me 8% ”5:. SEC 8% ”casemomc $3. EBEEED 5— SE 9 v2.22 8: 3..qu 23:833— v u=8 no? 8% ”2.0.25; hunger—AC 8% ”8238 tonaamfi c. R ”cages—:85:— anh m ”85 .0585 8% Hagan—615092 8% ”mooEomEouha 8— J .wum .YUV SJ. .8 2..— d. SE an acute? 2m .2“. £3.50 m ”58386 53,—. 8% ”2.52 . 8% ”85:66 :06 NN ”208 Qua—SE. ..Soabcoo ma mougtoqxo .80» n. 55 29: I n ”mp—68¢ 8% afiufiims 8% Boga:— ESEaS—Am av 85 .n ”ow< momma—o v =a sop—.1 DH. .5— 8% ”cu—n accumbfimO— 8% ”33m? «so—=59 «...—2 man—u v ”8%.. Begum mm H mm 05 wet—S 2&3 Exm 85988 26.. "8.6 52.5 v 6:23:86 BoFum N u M cocoa a. 258:8 8.3m uni-...; c ”2.539%; N u m— owvorsoE. "53.3.32 c ”cogs—go how 92 ”wsfiahxavmfi 293333 3032 van £23255 c E28585 man—U 92 ”SE 08855 92 Eu..— .mam: 31,23 .2 2n man—o £23295 :8 Eu 88: Saw a 8E: .5.—38 v ”82.330 man—U 8% E0633 $123— 8% E23263: 6.90:.— o\oc acofio>an_ m3 3 x85 23 5:358: :23 96: 4:838 .0: v ”mafia?— =o «oocum 8% E3806 8% EoEmEuofiO $8 ”m3 :5 .8538926 :o 2:50qu :83 a 8E: .805 v ”3o: tanm cz ”ca—a E3 92 ”genomema o\.. 2. “n3 mNu ooEo 82m 080: 59c ...—wo— 8_.9m ”...—:3 N £3252— 8% ”goutcm thump—EC 8% ”80358 tonaamAm €on 623533 .5— 3:55 o.— mEEah 32......“— ...3— N 5:3 no? 8% ”wanna—«E892 8% ”moorcomiooha o. \ o. ”cocoozcoBBE 2m 3 v8.28 8: ofiwuv 2.2.853— m ”85 8535 8% ”2m;— 8% ”moézofl 200 An”; ix .3qu 89A 28 55 m8— »8 6:20on 5:852:83 m 623386 53,—. oz gfioficis 8% 8033.: 58:33.8 3 ”23m mix—:5. a mu Mamie? .SE 5 moocotonxo 80> m; I v ”9.63% 8% ”cu—a >oaowuoEm—AM az ”vamSw cm-mN .N 8w< mad—u _ xoohz DH. 5— ...o—E.» 22.8..— 895....85.“ 8:98 gm. 8.39 «832228» 339% 83.555 ms 035,—. «00:22:00 03:: 820 0:: ”:98 80:0: :3... 50:9:00 05: "806 .056 7.0%: 3030.”— ov H 5— E0600 :0 Q a. £0: 068:0: "2.3m 9.35.; _ 8:28:86 3030”— N u v— 8.5.3.0. asazsz c 5.25805 n u m 063306; 3030: :26: 5: 8:06:00. mm :0 68: _ E0.:a:_u>0 .=0m $6 0 00:3 :05 80:. .380 0.00:0 4:08:00 .0: m ”58:86 820 00> ”w:_:_g\.:6mfi 830:0: mm 2n 5: 80:88:26 :0 9:65:06 >06 0 «2:: .838 _ 803.0030 80.0 00> ”2... 003303 00> E03 £5: $ 0 ”Eon—389:: LB 0050 3:080: 2:0: 50: 5&0. ”0.3m 36.304 _ ”$600.. :0 80:0,: 00> E08606 0:0-2E 00> E03283: .39: $8 ”m3 «Na 0560 6:0 .833: ".0:—.3..— .33— v ”0.0: t0:m 90> EmBuQAO 00> E0£m_:00<6 $8 ”.3 2m 3 630—0.. .0: 00%06 E05325 N ”00.038:— ¢0> E2: SEE 00> ”:8080ME $Nv E06865 ...—~— 98» m; 83 05 :0.“ 45:—8:00 2... a _ u=8 :03 02 0:0.qu gong—ARV 00> 8005.8 :3:ng o. \ a E0:00::00:0E_ an 31:03 .0228 65 8:52: 39.88 Emu—8:00 c é:— SEBE 00> ”3:36:0022 00> ”$0E8200H5 8— J is: .98 5:52: 0008 .2". .2: .0058: 8:80:0w v E28:86 500,—. 00> ”SEAS— 00> 80:600.: 206 MN ”0:08 235:5. £80060 8 80:05.58 80» n. :05 80:. ... v ”w:600m 00> ”ESSEES 00> ”006\0..:_ £85028 av :26 m ”0w< mamas v =a 0.87 :h a: 8> an... 58850:: 8> ”:58? .80.... 2.2 820 05 mama: 3030M mm u 5— wExS 0:80: :28 85:88 0.6: “:06 "3.6 .350 _ 8:28:86 3060»— N u M 8:8 ... 2:8:2 3.8 9.0.55. 0 56385 v n m 0:... 0:08 6:0 0:=:0 880.0 ..0 _ E23256 .20m 00> nwfifigfivmfi 3:50:02“ 630.305. .0896 8:3: E02950: _ E08386 820 00> ”SE 08:0.»08 00> E03 835 .0320: mu 2.. 06:0 6:: 0063 3030: :05: S: _ 80280.30 8:5 00> E08606 0:029— 00> Eocasowo: 8:03: $o ”5053895 .5» 8:06:00. my. :0 68: £6 a 00:5: 55 0.5:. :5: m “€600.— :0 «00:3— 00> E93090 00> E0£m_:_00<¢u $8 ”E5 -0 0.00:0 65 £003 a 950: 06 $6 a 3:6 _a._0>8 v0.0: t0:m 00> E20 SEC 02 Eouaomomfi $N0 ”.3 5N: 0050 03:8 080: 50.: ENE "0_bm "...-:3 N :08ng 02 U028:0: 3.2—2&0 00> ”80:68 9.0353 $3 E02365 .2 8820 5:9: 33:3,.— ..6H m 0:3 :03 n0> "m::0:6:000AZ 02 800E8200ha c. \ a E0:00::00.§:_ SE 9 60.20: .0: 00306 E03026H N Eu:— a050.:_ 00> ”2.5: 00> 80:600.: 200 35 4mm .0”qu 030% 2.3 .5.: :02". w: 35:03 5:53 000% m ”:0_8:86 :30... 00> ”66:05.:3 00> ”0030.?— 685023 NN ”0:08 Mimi—F 6:: SE .2.— .53060 m0 «00:0toax0 :00.» 2-2 I m ”€603. 00> E2: aufimng— 02 86:52 ov :26 .n ”0w< 8820 v ...N 0.00,—.1 DH. 5— ...065... 22:0,.— 00030303 §Q\.§ gm ~89 3:035:86 Safixs 3.53555 mg. 030,—. 233 «83538 3&8» 234 mam—o o5 wet—S 2&2. =37. 559:8 96: "8.6 356 manna 325% mm H. mm cocoa .3 an ... 5.5 359:8 8.55 was—£5. N ”82323.6 Begum N u M 25. 03% o as§83> m. u m was momma—o ..o bzfiazga dawn—325. "32.5.32 N E23296 hum Eva—{82> 3032 56: :5 8:35.08 .3 =a m E38386 820 «9% ”wfifinbxsvmfi 83an .2 SN com: £3 a 3:0 255-0 “.85 .3838 6: :5 «.33 N ”83.8.30 $20 a; ”EL 0855? a; ”53 £5: ..\oo acoEQSEE. E5 a 3:5 .238 0:8: Eat Emo— 3.5m «5534 m ”mica“: :o .oocom mo> E2286 BREE mo> “catacowo: .305 $8 ”A? can omo=oo 9:55 oficéan 8:83..— .25 n ”82. team :2 E3806 8% EoEvaofiO $8 #5 2..— 8 v2.22 8: 98on 32.3.6”.— N ”.83me :2 :83 SEE 8> ”5383:"— .xfiv E23533 mm 88> m-_ «m2 05 kc.“ _ 3:8 no? :2 5:898; EuaEmAO no> ”801:3 toga—am 2 \ a £283,582:— ucs—smcou ma macro? conga 8an 23 389250 m ”85 .0525 8> ”mag—.6500? mo> “mooEomEuuha Gw— 6E .93 .0333. SE 453.388 ..ofibflEEva 30:63 N E28286 88H n5 ”SEAS— 8> ”mug—Ba 206 2 ”28m mat—SE. .2..— .2m .852“ 8 3051098 :6» m. 55 EOE z N $533. 8> “36.35%; 8> ”8385 535328 ov ~96 .m new< mod—o _ xoohz DH. .5— 9; ”Sm—a accumuoEmO— 9; Exam? 2.0.5.» 2:: N “.692. Borfim mm H mm awn—o m uncoumsoflv 3933— v u M 2: 95.8 2&2. =67. 839:8 26: "3a.. .350 _ 6633815 n u m :28“ ... 038:2 8.3m use—5.; N £23296 tom cl $55.53qu 032305 Eocgtc—z m E23386 mam—U on; “SE oomb>OAm n; ”=2.— .mamc BEES AM 93. 333.202 “2 =a vow: N ”$380.30 mam—U 8% ”coafiov 8322 9; HcoEEowoc .32: .xé scoEo>an_ m3 5% a 335 55 958 :23 xouno 4:888 8: :5 m ”9.63.. so «3:3. 92 E3896 8> ”cop—flsaoio $8 ”m3 .woo=SmE:o.:o co NEE—ego“. x83 a 3:5 333. v ”So: roam 8> ”52a SEC no> usuaomomfi $3 ”.5 aNa 3E0 3.533 2.8; 50¢ Emo— 3_bm ”...—=34 _ ”58:sz a; 6:8:8 stun—8&0 96> ”moorcom :25st $3 E06355 .5— 25.5 d 5&8 53 sun—.8..— .33— m 3:8 903 8> $535950qu 92 ”803582319 etc. ”cocooccofloufi SE 9 “.822 Eon 05 E 3&3 32.3.6”.— v ”85 $585 8> ”2.52 8> ”836$ .206 Ann‘— Jxm .98 8:53 £3.83 was .8583 3an .SE .m< m Eofimammv Emu... 8> ”328:5: 8> ”830...: 555528 «A ”Boom 3355.. .ZA 3 moocotoaxo 85598 EPA m. :2: 0.85 I m ”9533— no> ”SW—q zocomuoEm—AM e Z SvBmfit 3. .53 .m ”ow< momma—o v =a xoohz DH. 5— .5.—=5... 2'2 gutwtvmam Sufixw $3555. 3&9 QuaEEou» SuQ‘? 3.3.5.39. ms 035. $.20 05 mac—S 0080a :20. 005958 03: "3.6 .359 .8300 .5 0300—30 “my—hm wee—£5. 05: 0.8% can mam—0 .«o £3256 .53 .0030 "nezgzez 8:35.00. mm :0 com: Sun a 00.8 .380 0.00:0 EB .3588 8: v5 .30 a Son 5 55 80. 0.003 a 0:5 .80>0m $355050; Eat Emo— aium 9:53?— 0==:o 0% 5&2 33:3,.— ...—0”.— 2m 3 @820. 00%“... E23031”..— mhnux m; .02 05 08 .0583 000% a ma 083:8 was—53 5:53 008m 8 805.0030 30> m; I 320 _ Jock: m ”0092— 3030M m ”macaw—08% 3030M N 66.38.05 _ ”cogs—30 tom m 62330.6 320 m ”82.8.30 $20 :2 ”.2”. 83268 ..z 5228.. 3:22 :2 05:32.30: :2 as: .05: 00> ”couscowoc .32: ..z ao£m38<6 .2» ”nosaomc 02 ”000308 0.6355 00> ”8030820020 02 ”82:05.0 206 00> ”838:— ESEm—fim ez scam? 000=0m005m~ 3." v" N" and: 3322. .2 as $2 u:0&0>895 A? $2“ :5 $8 5 x8 ”8:353 .2 c C c- ”GomuooS—Oohoucm AS .2». .de 2 ”28m 05:5 8-2 .~ no? 0:0—.5». 0.050..— 5% m ”ME—03. no «00:3— 02 ”56095 n ”Bo: tozm 00> ”SN—m SEE w ”0809.55 02 ”2399.8 300%:on m 0:8 90>» 02 35.0528qu v é:— SESE 02 ”2.52 m E23986 83H 02 ”Bazaar—3 m uwfivmum 02 :3:— 5:090:50— D.—. A“ Sanw SEER“. 30035390» 809? 8.3.555. mg. 03.3 0000 235 APPENDIX F EXAMPLE OF SCALE DEVELOPMENT FOR VARIABLE MEASURES Scale Development for Critical Thinking Behavior Measure Critical Thinking Operational Definition: Critical thinking is recognized as a purposeful reflective thinking behavior. It contains the evaluation process of contents and values related in each practice. Critical thinking differs from reflective . thinking in that it focuses on commitments to content of values, beliefs, and any pre- set objective(s) to achieve a desirable result for each practice. To measure a latent variable, we must set a hypothetical question by using key indicators involved in the specific operational definition of the variable. The hypothetical question for this variable is “How would you handle the situation if your computer (or software) breaks down at a critical point near the deadline of a team assignment?” The indicators for critical thinking behavior measurement include: 1. value: responsibility for the assignment 2. purposeful objective: getting the assignment done by the deadline 3. ability to break down choices and making decision The five multiple choices contain different levels of each key indicator to demonstrate the differences of scales. (see table 7.4 for details) On the scale one to five, scale level 1 shows the lowest level of critical thinking behavior and scale level 5 indicates the highest level of critical thinking behavior. 236 Table 7.4 Scale Development for Critical Thinking Behavior Measures Scale Answer Discrete I would expect that my teammates and : :3 giggiflllty 1 the instructor would understand. I ‘. do nth show ability to break would solve the problem later. down choices and effects I would contact my teammates via - no responsibility 2 another device to let them know the - objective is set situation and ask for help from them to - show low level of ability to break meet the deadline. down choices and effects I would get my computer fixed :2:v:c::§