.3. .. L sums? .. 1. . : V. n i... . «c . r. . .1??? .. . .« r i.‘ c , . . umwfu ufihw.7...rmwo‘. r . an . ... ; L. » . a... it" Q». ‘1.“ :l 3‘ .fi 2 ”9&3. him. at“... , . a 1 émwunéwrflu h1?-. la K); v. . .. my; 3 . 3h 2. :r}. .7 - .v,‘.l .- » Gaza“ 1. u". H.475. s. V. A r « t . .. A 3 .! tr wmhmxb.“ «3.6 3.11., . V. o. . :fi. #41 ! D 4. z... .34....» n. THESIS noel This is to certify that the thesis entitled Evaluation of Calibration for Optical See-Through Augmented Reality Systems presented by Erin Scott McGarrity has been accepted towards fulfillment of the requirements for Master's degreein Computer Science & Engineering ‘ é WM. Major professor Date 3’9 -O( 0—7639 MS U is an Affirmative Action/Equal Opportunity Institution LIBRARY Michigan State University PLACE IN RETURN Box to remove this checkout from your record. TO AVOID FINES return on or before date due. MAY BE RECALLED with earlier due date if requested. DATE DUE DATE DUE DATE DUE 6/01 cJClRC/Dataouopes-sz EVALUATION OF CALIBRATION FOR OPTICAL SEE-THROUGH AUGMENTED REALITY SYSTEMS By Erin Scott MC Garm'ty A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE Department of Computer Science and Engineering 2001 ABSTRACT EVALUATION OF CALIBRATION FOR OPTICAL SEE-THROUGH AUGMENTED REALITY SYSTEMS By Erin Scott McGarrz'ty This thesis describes a method for the metric evaluation of the registration accu- racy Of Optical see-through head-mounted displays. This method provides a quanti- tative estimate of the calibration error while the user is in situ. This method was implemented as a module which can be added to an existing AR system. The module works by superimposing a virtual object, 3.9. a small sphere, onto a known posi- tion in the workspace and asking the user to indicate where they perceive object to be located. The module then moves the Object to another location and the process repeats. Finally, after a sufficient number of points have been displayed and their po- sitions indicated, the module computes and displays the system errors by comparing the differences between the perceived and actual locations of the points. Detailed descriptions of two optical see-through augmented reality systems, along with their calibration requirements, are described. The first system uses magnetic tracking while the second uses infrared video-based tracking. Both systems have workspace volumes of approximately 1.0m x 1.0m x 1.0m. The results of the appli- cation of the method to both systems are shown. These results give a picture of the overall system error. They also Show that the the Observed error in an AR system is highly user dependent. © Copyright May 9, 2001 by Erin Scott McGarrity All Rights Reserved ACKNOWLEDGMENTS To begin, I would like to thank the people who worked with me on this project. First, there’s Dr. Charles Owen. He is a top notch advisor and editor. His thoughts and ideas about AR gave me a new way Of looking at it. Next, I want to thank Dr. Mihran Tuceryan. It was Mihran who got me into AR in the first place. He also pointed me in the directions of many excellent restaurants and provided plenty to talk about outside the realm of AR. Next, I give thanks to DrS. George Stockman and Frank Biocca. Their inputs made me think more about the human component of AR. Also, thanks to Yakup, Nassir, Benedicte, Frank, Xhang and Ali for the lively discussions and brainstorming sessions at Siemens. I would also like to thank four people who made indirect contributions and made my life much easier while I was working on my thesis. Let’s start with Dr. Anil Jain. He made the financial stuff happen with SCR. There’s also Ruth at Siemens who helped me find all those Obscure references. Dr. Charles MacCluer helped to hone my math skills along the way. He also gave me lots Of good advice about careers in academia. Finally, thanks to my wife, and best friend, Christine, for all her patience and support. She made the other parts of life (bills, car registrations, etc) go along smoothly. iv TABLE OF CONTENTS LIST OF TABLES LIST OF FIGURES 1 Introduction 2 AR System Description 2.1 Magnetically Tracked System .................. 2.2 Video Tracked System ...................... 3 Workspace Alignment 3.1 Rigid Alignment ......................... 3.2 Affine Alignment ......................... 3.3 Robust Alignment Computation ................. 3.4 Experimental Results ....................... 4 HMD Calibration 4.1 Camera Model .......................... 4.2 Estimation of the Display Parameters .............. 4.3 Calibration Procedure ...................... 5 Evaluating a Calibration 5.1 Evaluation Algorithm ....................... 5.2 Planar Calibration Validation .................. 5.3 Experimental Results for Planar Verification .......... 5.4 Analysis of Structured Errors .................. 5.5 Conclusions ............................ 6 Summary and Conclusions APPENDICES A User Error Field Plots BIBLIOGRAPHY vi vii K] 12 13 16 17 22 30 31 36 37 39 41 41 44 49 51 52 54 55 64 3.1 3.2 3.3 5.1 5.2 5.3 LIST OF TABLES Pointwise standard deviation measurements. ................ 18 Magnetic field distortion information ..................... 19 Error analysis results. ............................ 24 Results for a single user from multiple viewing zones. ........... 47 Average errors for different users from multiple viewing zones. ...... 47 Maximum errors for different users from multiple viewing zones. ..... 48 vi 1.1 1.2 2.1 2.2 2.3 3.1 3.2 3.3 3.4 3.5 3.6 4.1 4.2 5.1 5.2 5.3 5.4 5.5 5.6 5.7 A.1 A.2 A.3 A4 A5 A6 A7 A8 LIST OF FIGURES Optical see-through display. The real scene is mixed with graphics using a 1 /2-si1vered mirror ............................ Coordinate systems for a magnetically-tracked AR system. ........ A photograph of the magnetically tracked system .............. Magnetically tracked AR system. ...................... AR system with infrared video—based tracking ................ Pairwise inter-point differences for measured points ............. Distribution of inter-point distances is approximately normal. ...... Errors using the affine methods with control calibration data. ...... Errors using the affine methods with noisy calibration data. The outliers are highly visible in the robust procedure. ............... Errors using the rigid methods with control calibration data ........ Errors using the rigid methods with noisy calibration data ......... The geometry of a simple pinhole camera model for perspective projection. Coordinate transformations used for display calibration. ......... The optical see-through evaluation system .................. Error is measured in the plane of the evaluation board as seen from the eyepoint, E. The true location is pt, the observed location is p0 and the projection of the Observed point onto the board is p, ........ Error in the plane of the evaluation board as seen from the eye—points, E and E’. The error “p, — pb|| is larger than ||pt — pg“. ......... Error fields for an AR system which uses a magnetic tracker. These are the planar errors as observed from three zones in the workspace. . . . Viewing zones for an evaluation ........................ Error fields for an AR system which uses a infrared video tracking system. P0 indicates the calibration origin and T0 indicates the tracker origin. Reconstructed locations ........................... Three-view errors from a user ......................... Three-view errors from a user ......................... Three-view errors from a user ......................... Three-view errors from a user ......................... Three-view errors from a user ......................... Three-view errors from a user ......................... Three-view errors from a user ......................... Three-view errors from a user. The outliers in this figure are from the user moving outside the range of the tracker. ................ vii 27 28 29 31 34 42 42 43 45 46 48 49 56 57 58 59 60 61 62 63 Chapter 1 Introduction Augmented Reality (AR) is the process of superimposing computer graphics onto the real world. The intent of AR is to enhance the amount of information about a place or object while it is being Observed and interacted with. AR differs from Virtual Reality because it is not completely immersive. Instead, it is a composite of the real and the virtual. This paper provides a discussion about a particular branch of AR which is known as optical see-through AR. This branch uses a see-through head- mounted display (HMD) which mixes graphics with the scene using a 1/2-silvered mirror. Figure 1.1 shows a schematic for an Optical see-through display. There are many potential applications for AR. These include training, inspection and repair tasks. Examples include a circuit board inspection system [24] or a laser printer repair application [8]. It might also be possible to use an AR system in a telepresence application [6,18]. In this application, a technician at a remote location can be guided through a repair process by an expert in the home Office who is able to see what the technician sees using AR system. The expert can then provide 1 LCD Panel Eye Real Scene 1/2 Mirror Figure 1.1: Optical see-through display. The real scene is mixed with graphics using a 1/2-silvered mirror. multimedia feedback through the technician’s display. On another front, AR can be used for entertainment. There have been several applications in this vein such as “AR2 Hockey”, “AquaGauntlet”, and “Welbo” [31]. In the same vein there is the potential for installing AR systems in museums and historic sites. Imagine going to the courthouse in Boston and seeing an augmented re-creation of the “Taxation Without Representation” or going to Dover Castle and seeing William I and his army crossing the English Channel and landing on the Shores. Another ideal application for AR is in computer-assisted surgery. AR can provide a form of X-ray eyes, allowing the surgeon to see through tissue and allowing results from instrumentation such as PET and MRI to be projected directly onto affected tissue. If a surgeon could see a virtual overlay of a tumor in a patient, they would be able to minimize the amount of invasion to the patient. A prototype of an AR surgical microscope has been built to this end [5]. There is much work which needs 2 to be performed, however, before such a system comes into common use in the oper- ating room. Further exploration into tracking and registration methods needs to be performed. The validation of such a system iS also crucial Since there is no room for error in such an application. In order for an augmented reality system to be effective, the virtual Objects must be accurately positioned relative to the real Objects. This implies that certain mea- surements or calibrations need to be made. These calibrations determine the transfor- mations between the various components such as trackers, cameras and head-mounted displays (HMDS). The calibration requirements and the difficulties with their imple- mentation depend on the architecture of the particular system. Figure 1.2 Shows the transformations between the objects in a typical magnetically-tracked AR setup. The objective of calibration is to estimate the trans- formations between each Of these coordinate systems and a central reference coordi- nate system known as the Workspace Coordinate System (WCS). The WCS is gener- ally at a fixed location relative to the Operating environment. During the operation of a system, all of its components are referred to in the workspace coordinate system. The Tracker Coordinate System (TCS) is the coordinate system in which all measure- ments occur. The transformation between the TCS and WCS (W in the diagram) must be estimated since the origin of the TCS lies at some unknown position within the transmitter antenna. The Marker Coordinate System (MOS) is used to locate the user’s head at runtime. The transform from the TCS into the MOS (M from the diagram) is measured in real time by the tracking system. The Display Coordinate System (DCS) is the coordinate system that the graphics are rendered in. The trans- 3 (MCS) (DCS) (TCS) Figure 1.2: Coordinate systems for a magnetically-tracked AR system. form between the MCS and the DCS (G from the diagram) is estimated during HMD calibration. This paper is primarily concerned with HMD calibration and validation. HMD calibration is the practice of estimating the projective transformation which must be applied to the virtual objects in order to display them for the user. If the HMD is calibrated incorrectly, the virtual objects will not align properly with the real objects in the workspace. In principle HMD calibration is similar to camera calibration, yet, in practice there are many difficulties related to the tight coupling between the intrinsic and extrinsic parameters [11,27]. A particular problem associated with calibrating optical see-through HMDs is determining the accuracy of the system. Traditional image-based methods are of little use since there is no way to access the augmented image which is formed on the 4 user’s retina. Thus far, there have been two main approaches to accuracy verification for see-through displays. The first is to have the user calibrate the HMD and then report the qualitative accuracy of the alignment of a computer model to its real-world ” counterpart [33], i.e., “acceptable,” or “unacceptable. The second method uses a camera to replace the human eye and conducts image-based measurements to assess the accuracy of the alignment [27]. The former method is quick and easy, but it is subjective and gives no metric information. The latter method is performed by mounting a camera to the HMD so that it “sees” through the display panel. This is not very practical, nor does it guarantee that the measured accuracy will be Observed by the user Since a camera is only an approximation Of the eye. Furthermore, the method is very tedious and must be performed offline Since the camera is mounted where the user’s head would normally be. There are several general works which describe AR as a whole. Azuma classifies AR systems based on display types and levels of immersion [1]. Feiner, et at, and Kutulakos and Vallino discuss workspace Object representations in AR systems [8,21]. Klinker, et al., describe the merging of graphics and computer vision [20]. Fuchs and Rolland compare and contrast Optical and video see-through HMDS [10]. AR systems can be broadly categorized by the type of display they use. The first class is known as video see-through. Video see-through systems use a camera and display as an intermediary between the user and the scene. The video stream is captured by a camera, augmented and passed on to the user via a HMD or monitor. Much work has been done in this area. Tuceryan, et al., describe the calibration requirements for a video based AR system [32]. Janin, Mizell and Caudell developed a 5 monocular system for assembling aircraft wiring harnesses [17]. Kato and Billinghurst describe a combined camera/HMD calibration scheme for tracking and augmenting fiducial markers in a scene [18]. Neumann and Cho present a method for improving the registration of such a system [29]. Mellor describes a procedure to calibrate a camera with embedded intrinsic parameters [28]. This method is very Similar to the HMD calibration procedure described later in this thesis. Genc, et al., discuss the validation of a stereo video see-through system [12]. Other researchers have been concentrating on registration of 3D medical data to surgical patients [3,4,13,22,34]. The second class of AR systems are called Optical see-through systems. Tuceryan and Navab present a user-friendly calibration method for optical see-through HMDS [33]. Oisho and Tachi discuss the calibration of a rigidly mounted Optical see-through display [30]. Azuma and Holloway analyze sources of registration error in AR systems [2,14,15]. This paper takes a similar direction, but also provides an online method for cOmputing registration error. Previous work by this author in the realm Of Optical see-through calibration and validation can be found [11, 24—27]. The remainder of this paper is divided into four chapters. Chapter 2 describes the AR systems used for testing the verification method. Chapter 3 gives the details on the methods used to calibrate magnetic tracking systems. Chapter 4 describes the HMD calibration procedure used in both systems. Chapter 5 gives the particulars on the verification method and discusses results of its application to the systems described. Chapter 6 summarizes the results Of the work. Chapter 2 AR System Description Two different optical see-through AR systems were used for the experiments reported in this thesis. The first system is based on a magnetic tracker. The second system uses an infrared video-based tracker. Both systems have stereo capable HMDS, but all experiments were performed in monocular mode. Details for the magnetically tracked system are given in Section 2.1. An overview of the system which uses video-based tracking is given in Section 2.2. 2.1 Magnetically Tracked System The first system consists Of a SGI O2 workstation, a HMD and a magnetic tracker. A photograph of the system is shown in Figure 2.1. A fully detailed description of the calibration requirements and procedures for this system are given in the chapters that follow. Figure 2.2 a schematic for the setup. The HMD for this system is a Sony Glastron LD-100B. It has a resolution of 7 Figure 2.1: A photograph of the magnetically tracked system. \ Workspace / //: Stylus ransmitter Tracker Figure 2.2: Magnetically tracked AR system. 800 x 600 pixels per eye with a refresh rate of 60Hz. The stereo mode is frame- sequential for this display. The magnetic tracker is a Polhemus Isotrak II with a single transmitter and two receivers: a marker and a digitizing stylus. It is capable of sensing the position and orientation of the receivers at ranges Of up to one meter. The marker is rigidly attached to the HMD to track the user’s head motion during runtime. The stylus can be used to measure the coordinates of real Objects in the workspace at runtime. It can also be used to interact with virtual Objects in the workspace. 2.2 Video Tracked System The second system consists of a SGI 540 Visual Workstation, a HMD, a camera and a graphics tablet. A schematic for this system is shown in Figure 2.3. The details of the tracking system are unknown to this author, so they are not given in this paper [12,33,35]. The HMD for this system is a Virtual IO i-glassesTMfield-Sequential stereo display. It has stereo a resolution of 640 x 240 pixels per eye with a refresh rate of 60Hz. The monocular resolution for this display is 640 x 480. The video-based tracker uses fiducials with known geometry made of retro- reflective material. It uses a camera assembly which is mounted to the HMD. This assembly consists of a CCD camera with an infrared filter and a ring of infrared LEDS mounted around the barrel of the camera. This tracker uses a technique similar to that described in [23] to estimate the pose of the HMD with respect to the fiducials. 9 Camera /\ Workspace Q a. is...““. n ..... PE: “‘-‘.----l".‘, a. Figure 2.3: AR system with infrared video-based tracking. 10 The graphics tablet is a Wacom Intuos. It has a 15.25cm x 20.25cm writing area. The resolution for the tablet is 1000 dots per centimeter. There are fiducials mounted on the tablet so that it can be tracked by the camera at runtime. 11 Chapter 3 Workspace Alignment While working in a bench-top AR system it is important to have a reference coordinate system which the locations of the real and virtual objects can be specified in. In practice this coordinate system is set in a location which stays fixed during runtime such as a table. While the system is running the user’s head must be tracked in some way, e.g., magnetic tracker, video tracker, so that the virtual objects stay aligned with the real objects as the user moves around. Since this motion is measured in the tracker coordinate system (TCS) and the objects are in the workspace coordinate system (WCS), it is necessary to find a transformation between them so that the graphics can be rendered properly. To align the workspace to the magnetic tracking system, a series of measurements of a calibration Object with known position and dimensions is taken. The measure- ments can be used to find the transformations between the tracker coordinate system (TCS) and the workspace coordinate system (WCS). 12 The relationship between points in the WCS and the TCS is given by PW = SRPT + T, C“) where pw and pT are the workspace and Object points, respectively, 3 is a scale factor, R is a 3 x 3 rotation matrix and T is the translation vector between the two coordinate system origins. Given a set of points, pw,,-, whose WCS locations are known, along with their measured locations in the TCS, p73,, it is possible to find the transformation, (3, R, and T), with a least squares regression, i.e. n minimize Z(pw¢ — stm — T)”. (3.2) i=1 The remainder of this section is dedicated to the detailed analysis of workspace alignment using a magnetic tracker. In sections 3.1 and 3.2, the implementation details for two calibration methods which are based on this model are discussed. Next, a method for removing outliers from the measured data is described in 3.3. Finally, the results Of an experimental comparison between the two methods is shown. 3. 1 Rigid Alignment The first alignment method treats the two point clouds as bundles of rays emanating from their centroids. This method makes the assumption that the transform between the two clouds is completely rigid. In order to enforce this rigidity, an orthonormality 13 constraint must be imposed on R, that is, RTR = I. (3.3) First, the centroids Of both sets of points must be found. They are computed using 1 Ti. p, = 5 2p“, k e {T, W}. (3.4) Next, both sets of points are converted into bundles Of rays about their centroids. This transformation is written as rk,i = pk,i — Pk: ’9 E {T, W} (3-5) From the ray bundles, the scale difference between the two coordinate systems is computed by finding the ratio of the sums of the squared lengths of the individual rays. (This number should be close to 1 if the measured units of the clouds are the same.) It is written compactly as n _2 .s2 = %,=—1%’i7. (3.6) i=1 T,i Next, the rotation matrix can be recovered. As previously stated in Equation 3.3, this matrix is constrained to be orthonormal. TO begin, the sum of the coordinate- wise products of the corresponding point sets is calculated. This 3 x 3 matrix, M, is 14 computed using 11 M = Z(rw,)(r7~,)T. (3.7) Let the matrix N be defined as N = MTM. (3.8) Using the definition for N, the rotation matrix, R, can be expressed as R = MN-W. (3.9) To find N‘1/2, note that N can be expressed in terms of its eigenvalues, A), and eigenvectors, v,, as N = Alvlvf + Agvgvg + A3v3vg. (3.10) After using a standard method to find A, and v,-, N"’1/2 can be expressed by writing 1 l 1 N‘l/2 = MTM "”2 = ——v1vT + ——v2vT + —v3vT. (3.11) 1 2 3 x/7\T JAE WAT; Finally, T can be found by using R, s and the centroids of the point sets. To compute T, use T 2 [3w — SRflT. (3.12) Fiirther details of this method are given in [16]. 15 3.2 Affine Alignment For the affine alignment the orthonormality constraint on R is relaxed. Formally, it is written as R=s(Ro+Ra), (3.13) where R0 and R0 are the orthonormal and affine components of the matrix R. This form builds in a linear approximation to any systematic errors in the tracker mea- surements. To find the transformation parameters, R and T, a least squares method is used. After collecting the point cloud data for pw and pT, the block form of the solution can be written as where R1, R2 and R3 are the rows of R. 16 PW,1 pW,n (3.14) 3.3 Robust Alignment Computation Since both calibration methods are based on least squares, they use all of the point- cloud data in their solutions. Due to noise and inconsistencies in the environment, some of these points may have large errors. These errors reduce the accuracy of the transformation computation considerably if we use traditional least-squares methods, especially in the rigid alignment procedure. These errors, in turn, propagate into the HMD calibration procedure. To detect and remove points which have large errors, the Random Sample Con- sensus (RANSAC) method described in [9] is used. The RAN SAC algorithm is shown in Algorithm 1. The inputs to the algorithm are the data, the number of subsets to try, k, the minimum subset size, t, and the error threshold, 6. This section describes the choices made for each in detail. 1. Select a subset of size t or larger from the input data. 2. Solve the system and evaluate the error e = ||Ax — bl]. 3. If the error e is less than some prescribed tolerance, use x as the solution and terminate. 4. If the number of subsets evaluated is larger than some threshold, terminate and report failure, otherwise goto step 1. Algorithm 1: RANSAC Algorithm TO begin, an error tolerance must be determined for points coming from the tracker. There are two main characteristics which need to be quantified. The first is the overall tracker measurement variance. This quantity describes the amount of variance expected when measuring the same point multiple times. It is determined 17 Table 3.1: Pointwise standard deviation measurements. Point Control a ( cm) Warped o ( cm) 1 0.04 0.03 2 0.04 0.05 3 0.06 0.03 4 0.12 0.03 5 0.06 0.04 6 0.05 0.05 7 0.03 0.08 8 0.07 0.04 9 0.06 0.06 10 0.03 0.15 11 0.05 0.05 12 0.05 0.05 Mean: 0.05 0.06 by repeatedly measuring the calibration jig calculating the standard deviation at each point. The results for these measurements are tabulated in Table 3.1. The left hand column shows the standard deviations for measurements at 12 locations on the calibration fixture in a magnetically “clean” environment. The right hand column shows the standard deviations for measurements of the same points taken with a large metal object in the workspace to distort the magnetic field. Notice that the two columns are essentially the same. This indicates that static metallic objects induce a fixed amount of warpage to the field. Most of the measurements fall within the manufacturer specified standard error Of 1mm. While the pointwise standard deviation is interesting, it does not capture the warpage of the field. In order to see the distortions in the workspace it is necessary to examine the relationships between the points. This is accomplished by computing the distances 18 Table 3.2: Magnetic field distortion information. Data Set ppm, ( cm) opos ( cm) Control 0.32 0.15 Noisy 1.02 2.33 between all of the pairs of measured points (on the calibration fixture) and comparing them to their known values. This will give estimates Of the errors in length which in turn can be used to Show distortions in the magnetic field. The length errors from the previously described sets of data are shown in Figure 3.1. The average errors along with their standard deviations provide an estimate of the expected measurement error for the points used during a calibration. These values are shown in Table 3.2. The distribution of the errors is shown in Figure 3.2. Assuming a normal error distribution, the value for e is given by the mean error plus 1 standard deviation, i.e., c = 0.47cm. These results confirm that distortions from metallic objects are constant and could be calibrated out Of a system [19]. To perform such a calibration, it would be necessary to create a polynomial map for the antenna, i.e., calibrate the field. This map could be used to warp the measured positions of points during calibration and runtime. The second free parameter for the RANSAC procedure is k, the maximum num- ber of attempts at finding a consensus set. Let to represent the probability that a measured point is within the error tolerance e. The expected number of trials, k, tO select n acceptable data points can be written E(k) = w" + 2(1— w")w" + 3(1— w")2w" + - - - + 2(1 — w")’-1w" + - --, (3.15) 19 Inter-point Distance Errors. 1 4 I l I l I -e- Control Data 0 —e— Noisy Data 12 ~ ~ 10 - - a - i 6 - n -. s " " r 4 - .. " ‘ g LU -6 I 1 l I I l Point pair Figure 3.1: Pairwise inter-point differences for measured points. which can be Simplified to E(k) = w"(1+ 2(1— 111") + 3(1— w")2 + - ° - + i(1 — 112")"1 + - - ). (3.16) It is possible to simplify the series in Equation 3.16 by using the identity for the sum of a geometric series =T+72+T3+"'+Ti+°"- (3.17) Inter-Point Distance Error Distribution. 9 r r r it Points -3 -2 -1 0 1 2 3 Error (cm) Figure 3.2: Distribution of inter-point distances is approximately normal. Differentiating this series with respect to r produces 1 . 2==1+2r+3fi+n-~+n“1+-~. @1& (1-7‘) Substituting 1 — w" for r in the above gives (10")2 = 1+ 2(1— w") + 3(1— w")2 + . . . +i(1_ w")‘—1 + . .. , (3.19) therefore, E(k) = -1—. (3.20) wn Assuming that the measurement errors are normally distributed, a point will be within 21 the error tolerance 68% Of the time. Thus, if at least n = 4 points are needed to find an alignment, then E (k) 2: 1/(0.68)4 = 5. Three times this value will remove outliers about 99.6% of the time. The final parameter which needs to be specified is the minimum size of the con- sensus set, t. If p is the probability that a data point is within the error tolerance of an incorrect model, then 12“" is the probability that a consensus set has converged on an incorrect solution for the model. Thus, the value of 19“" should be small. Even though there is no good way to establish p a priori, it is reasonable to assume that p < 11). Setting p = 0.5 and requiring a minimum confidence level of 95% forces t — n to be at least 5. This means that t = 9 and the probability of selecting an incorrect solution is 3.12%. 3.4 Experimental Results To test the two algorithms, two sets of data, consisting of 120 points per set, were generated by measuring 12 points on the calibration object ten times each. The first set, which Shall be referred to as the control set, was taken in a magnetically clean environment. The second set, which will be called the noisy set, was taken with a large aluminum block placed near the center Of the workspace in order to distort the magnetic field. The test data sets were used to perform two groups of experiments comparing the least-squares and robust versions rigid and affine methods described in the previous sections of this chapter. The results are Shown in Table 3.3. For each of the experiments, two subsets Of size 12 were randomly selected and 22 used for calibration. The resulting matrix was then used to transform the entire data set. The transformed points were compared to the actual locations and error statistics were computed. This procedure was repeated five times for each of the combinations Of the clean and noisy data sets, e. g., calibrate with clean data, transform clean data, calibrate with clean data, transform noisy data, etc. Table 3.3 summarizes the results of the 16 experiments. In this table, the a column represents the average error produced by the standard least-squares calibration method, the Robust it represents the average error from the robust procedure and the o and Robust 0 columns show the standard deviations of the two procedures, respectively. There are four important points to note from Table 3.3. The first is that the affine methods tend to have better error statistics than the rigid methods. This comes from the lack of constraints on the affine calibration matrix which allows for a linear approximation to the distortion in the magnetic field. The second noteworthy point is that the least—squares and robust methods are nearly identical for the experiments which were calibrated using clean data. This is to be expected since there were no outliers in the data. On the other hand, the robust methods worked Significantly better when noisy data was used for calibration. Again, this is to be expected since the outliers will be removed by the robust procedures. Finally, the robust methods showed higher variance on average. This is because the outliers were not removed from the re-projected data. Figures 3.3—3.6 Show comparisons between the least-squares and robust methods for all combinations of the data sets. Figure 3.3 Shows the errors for the affine calibra- tion procedures. In this case, the errors for the least-squares and robust procedures 23 Table 3.3: Error analysis results. Statistics Experiment Mcm) Robust ,u ( cIn) 0(cm) Robust o ( cm) Affine Control Calibration Control Data 0.13 0.13 0.05 0.06 Affine Control Calibration Noisy Data 2.58 2.58 2.60 2.60 Affine Noisy Calibration Control Data 2.72 2.03 1.26 1.21 Affine Noisy Calibration Noisy Data 1.94 1.90 1.30 2.27 Rigid Control Calibration Control Data 0.15 0.15 0.07 0.07 Rigid Control Calibration Noisy Data 2.55 2.55 2.60 2.60 Rigid Noisy Calibration Control Data 1.54 1.20 0.66 0.55 Rigid Noisy Calibration Noisy Data 2.34 2.19 2.14 2.51 24 are nearly identical. Figure 3.4 shows the results for the affine calibration with noisy data. Note that the robust procedure has generally smaller errors, but greatly magnifies the errors from the outliers. Figure 3.5 shows the results of the rigid calibration method using clean data. Again, there is not a significant difference between the least-squares and robust pro- cedures. Figure 3.6 Shows the results from the rigid calibration performed with noisy data. The rigid calibration scheme does quite well when transforming the clean data set, but magnifies the outliers considerably when transforming the noisy set. 25 Pointwise Errors for Affine Control Calibration/Control Data 12 r r r I I — Affine - - Robust Affine Error (cm) owwmmmmw O 100 200 300 400 500 600 Data Points Pointwise Errors for Affine Control Calibration/Noisy Data 12 I r r r I —— Allin. - - Robust/Affine 10 - « 8 - . E" ‘ u: 4 - -( 2 - . ti, iu' i' V U o l l L 1 l 0 100 200 300 400 500 600 Data Points Figure 3.3: Errors using the affine methods with control calibration data. 26 Pointwise Errors for Affine Noisy CalhrationlControl Data l I — Affine - - Robust Affine 12 f I I I". I ll . I I , I I ‘I I l " V 4""! \‘ || I Id 3!!) 600 DataPoints Pointwise Errors for Affine Noisy CalibrationlNoisy Data 12 I T l T l — Affine - - RobustAffine 10” I | : .- , i I ' ll l' l l f i p. , 3 :4 I . s» I] ] .3 ] ,I I]. [g [.1 1" ll I" I “I '1 '11 d ’.I II 'I. " 'I‘I llI ’ II .I A i .[ I" it [I It. i f" ] g 6_ [.l‘ n. 'II |II I[I :] I” :f _ E ’ 'l” [’ : [" i I' '[ m i ': . I l ' ’ I ' ' 4- ] l - ' - l. [ ‘ ‘1. I I ' I I" [I I I b v if " ,. | a 2 ., 7 . ' : ’ ' ' I I i I ' I ' 1 '~ ~ | ['4’ ’ ’ I "l e I. ‘ ' i): i\ 111.5%”: 1 i [J L’ .1 in“ ‘1“: 1: [All :I ll 00 100 200 300 400 500 600 DataPoints Figure 3.4: Errors using the affine methods with noisy calibration data. The outliers are highly visible in the robust procedure. 27 Pointwise Errors for Rigid Consol Calibration/Control Data 12 I I I f 1 —Rigid - - RobustRigid Io» « 8* J E“ ’ LU 0 100 200 300 400 500 600 Data Points Pointwise Errors for Rigid Control CaibrationlNoisy Data 12 I T I I l - - RobustFtloid i 8 " -I III 4 - . 2 '- q t 't lU' t' t' ti, 0 1 1 l l l o 100 200 300 400 500 600 Data Points Figure 3.5: Errors using the rigid methods with control calibration data. 28 Pointwise Errors for Rigid Noisy Calibration/Control Data 1 2 l I l I l — Rigid - - Robust Rigid 10— - 8 T -I 22' 6 - _ E III 4 _ - 2 ‘ "I - T" I. '1 ‘ i“ i 1 ‘ I II \ l/' N ‘i4'\ I o i“ 1. i" "I, , ' I I i " "‘ ’ ..| I h I”I " Ila.’ II»h H ‘ ’v‘ I. 'r ‘ II 1 “'1 I o l l i l l 0 100 200 300 400 500 600 Data Points Pointwise Errors for Rigid Noisy Calibration/Noisy Data 12 I I I I l — Rigid — — Robust Rigid 10- 8 I- : 6~ E In “I [Li I 2 | ‘J I ' I, V: Ji 0 l 0 200 Data Points Figure 3.6: Errors using the rigid methods with noisy calibration data. 29 Chapter 4 HMD Calibration This section gives the particulars for the calibration Of an optical see-through head- mounted display (HMD). The primary assumption behind the method is that the eye/HMD system is a pinhole camera. There are several well known algorithms for calibrating pinhole cameras, but most of them will not work in the context of HMDS because they require the camera to be rigidly mounted. The method given herein has no such requirement. The rest of this chapter is divided into three sections. The first section describes the camera model. The second section gives a numerical solution to the calibration problem. The final section tells of the implementation details for the system which is currently in place. 30 v y A \ Image Plane Figure 4.1: The geometry of a simple pinhole camera model for perspective projection. 4. 1 Camera Model A simple pinhole camera (Figure 4.1) is used as a model for the HMD [7]. This model defines the basic projective imaging geometry with which 3D Objects in the system are projected onto the 2D display surface. This is an idealized model commonly used in computer graphics and computer vision to capture imaging geometry. It does not account for non-linear optical effects (such as radial distortions) that are Often properties Of real cameras. The model uses a right-handed coordinate system with the center of projection at the origin and the image plane at a distance of f (focal length) from the origin. Views through the center of projection are assumed to be views down the —z axis. The image of a visible point in the camera coordinate system is formed by passing 31 a ray through the point, p0, and the center of projection, 00, and computing the intersection of this ray with the image plane, pD. If p0 = [:I:c,yc,zc,1]T and pp 2 [u, v, 1]T then using similar triangles the display coordinates can be written, u : fsumc/Zc + “'0: v = fsvyc/zc+v0, (4.1) where (no, '00) are the coordinates of the image plane origin in the camera coordinate system, and 3,, and 3,, account for the pixel Sizes of the display. Let fu = 3,, f and fv 2* 3,, f. Placing these values into matrix form produces a transformation P which is used to transform 3-dimensional homogeneous coordinates, (pg), into 2-dimensional homogeneous display coordinates, (p D). P is given by i' . fu 0 “0 O P: 0 fv ’00 0 ’ (4'2) 0 0 1 0 In practice, the u and v axes may not be perpendicular to each other. To correct for this, a skew parameter must be added to P to form a new matrix Q, such that, fu 7' U0 0 Q : 0 ft) ’00 0 I (4'3) 0 0 1 0 where T represents the skew between the u and v axes. Note that since Q is projective, 32 it is defined only up to a scale factor 11}. Putting everything together, a point’s display coordinates can be found by - . - - 550 um f“ 7' no 0 310 um = 0 fv ’Uo 0 I (44) 20 w 0 0 1 0 ' ‘ ' ‘ 1 . 1 where w is the perspective scale parameter related to the depth (to = zC). In general, it is not possible to locate the HMD coordinate system directly. In- stead, there is usually a tracking system rigidly attached to the HMD whose workspace coordinates can be measured in real time. To account for this, an additional trans- formation between the tracking system and the HMD must be estimated. This trans- formation is rigid and can be written T11 7'21 7'31 1‘12 7‘22 7‘32 7‘13 7'23 7‘33 (4.5) where the r,,- are the elements of a 3 x 3 rotation matrix and the t,- are the translational components. Multiplying Q by T produces the transformation which must be applied to a point to get it from the tracker coordinate system into the HMD coordinate system and 33 Figure 4.2: Coordinate transformations used for display calibration. (TCS) onto the image plane. This is the transformation G in Figure 4.2. It is The individual components of G are 911 = fuT11+ 7"721+ “0731, 912 = fu7‘12 + ”22 + “0732, 913 = fuT13 + 7723 + 710733, 914 = fut1+ Tt2 + Hots, 911 912 913 914 921 922 923 924 931 932 933 934 921 = fuT21 + 90731, 922 = fvT22 + 710732, 923 = fvT23 + 710733, 924 = fvt2 + 710153, 931 = 1"31, 932 = 7‘32, 933 = 7‘33, 934 = t3- Figure 4.2 shows the transformations a point in the workspace coordinate system 34 must undergo to get into the display coordinate system. First, points are transformed from the WCS into the TCS using the transformation W found during the tracker calibration step described in Section 3. Then they go from the TCS into the MCS via the M transformation which is updated by the tracker at each timestep of the system. The entire transformation from the WCS to the MCS is given by pM = Mpr, (4.8) where pM is a point in the tracker mark coordinate system and pw is a point in the workspace coordinate system. Next, the point pM needs to pass through the transformation from the tracker coordinate system into the HMD coordinate system. This transformation is written DD = GPM- (4-9) It is this transformation, G, which must be estimated during calibration. Ex- panding Equation 4.9 and dividing by the homogeneous scale factor 11) produces the pixel coordinates for any visible point: Ud : if : 911$m + 9129711 + 913%: + 914 w 931$m + 9323/m + 933Zm + 934, vd : 3 2 921mm + 922%: + 923Zm + 924, (4.10) 11) 931137;: + 932’9m + 933%: + 934 where ad and 1),, are the pixel coordinates of the point. 35 4.2 Estimation of the Display Parameters By rearranging Equation 4.10 and it is possible to write two constraints on the G transformation, (91113171 + 9129m + 913%: + 914) - “4(93133m + 9323/m + 933Zm + 934) = 0, (921%. + 9229m + 923% + 924) — Ud(931$m + 9323/m + 933% + 934) = 0- (4-11) Since there are 12 parameters to solve for, at least 6 sets of 2D / 3D point correspon- dences must be collected by the user. If the it” tuple of correspondences is defined as ($m,i, ymy, 2%,, “4,4, ’Udfl') then the solution for G can be found by estimating a solution to the homogeneous equation Bg = 0, (4.12) where g is a vector made by concatenating the transposed rows of G i.e. g = [911, 912, 913, 914, 921, 922, 923, 924, 931, 932, 933, 934lT, (4-13) and P7341 0 ‘uddpfm 0 13711144 "'W,1Pff4,1 pM,n 0 — ud,np7]fll,n T T 0 pM,n _vd,an,n 36 Solving Equation 4.12 will produce the G matrix. Because this equation is homo- geneous, there must be another constraint to avoid a null solution. This constraint can be made by remembering that G is only defined up to a scale factor. This means that it is possible to solve for the g vector using the eigensystem of BTB. Formally, it is written minimizellBgH2 subject to||g|| = 1. (4.15) The g vector corresponds to the eigenvector of BTB with the smallest associated eigenvalue [7]. Once this solution has been found it can be placed into G. If the Sign of the determinant of G(1 : 3, 1 : 3) is greater than zero, then it is necessary to negate G [33]. This is due to a second solution which exists behind the camera. 4.3 Calibration Procedure Practicality is an important feature of any HMD calibration algorithm. It needs to be simple to use and accurate. A first attempt at a collection algorithm would ask the user to align crosshairs in the image plane with points in the workspace in some predetermined order. While correct, it can be confusing for the user as they must remember the sequence, or someone must direct them. With a small simplification, the user is freed of this burden. Instead of being asked to align different points in the workspace, they are asked to align the same point repeatedly from different po- sitions. This technique is known as Single Point Active Alignment Matrix (SPAAM) calibration [33]. A compact description of the procedure is given in Algorithm 2. 37 . A single point is selected in the workspace coordinate system. . The user is presented with a crosshair in the display plane. . The user moves their head until the crosshair is aligned with the point in the workspace. . The user clicks the mouse to indicate they are aligned and the point’s location in the marker coordinate system is computed and recorded. . The cursor is moved to a different location on the display plane and the process is repeated starting with step 2 until enough points have been collected. . The calibration matrix is estimated and the process terminates. Algorithm 2: SPAAM algorithm used for calibration of the HMD. 38 Chapter 5 Evaluating a Calibration When designing a scheme to evaluate an AR system, there are many factors to con- sider. First, the procedure must be on-line. An AR system is not complete without its human user. Schemes which replace the user with a camera and a rigid mOunting fixture do not reflect the situation properly. In fact, much of the error observed in an AR system is from the user. During HMD calibration, for instance, if the user raises their eyebrows, the display moves with respect to the eye and error is introduced into the system. Another important observation has to do with the system complexity. The amount Of error a user introduces into the system is reflected in the difficulty of the calibration and evaluation tasks. Several of the users tested remarked that it was difficult to keep the crosshair on the alignment point during the HMD calibration (see Section 4.3). A third feature an evaluation scheme must exhibit is precision. It must provide consistent quantitative results from trial to trial. These results must give a clear indication of the accuracy that a user will perceive. It must be able to communicate 39 this accuracy to the user through some feedback mechanism. With this and the two points above in mind, the evaluation scheme can be laid out. Consider an augmented reality system with an optical see-through HMD and a real-time tracking system. Assuming that the user has calibrated the system along with the HMD, graphics can be overlayed onto the real workspace with some unknown accuracy. The aim is to measure the registration accuracy between the virtual Objects and the real objects. The results from these measurements can be used in two different ways. First, calibration accuracy information can be Obtained. This information can be used to determine the expected registration accuracy at a given distance. For instance, the system could report, “At 70 cm away from the target, the user can ,1 correctly identify items which are 1 cm apart. This is useful in determining if the calibration is good enough for a particular application. Second, structured errors in the tracking system can be identified and possibly corrected. Structured errors are errors in the system which have a regular form, 6.9., linear, quadratic, etc. For instance, problem areas in a magnetic tracker can be found. The rest of this chapter describes a method for evaluating the accuracy of an Optical see-through AR system at runtime. The algorithm is given in Section 5.1. Section 5.2 discusses the error metric used and its implications. Experimental results are presented and discussed in Section 5.3. An analysis of the structure of the errors is presented in Section 5.4. Conclusions about the system evaluated are drawn in Section 5.5. 40 5.1 Evaluation Algorithm The evaluation module is comprised Of an Optical see-through AR system and an input device capable of reporting positions in the workspace coordinate system. For a video-based tracking system, this device may be a graphics tablet with fiducials on it so it can be tracked. In the case of a magnetic tracking system, the tracker’s stylus can be used. Figure 5.1 shows a schematic for a system with a video based tracker. Any time after system calibration has been performed, the user may invoke the evaluation module. The evaluation module performs the steps shown in Algorithm 3. Note that since the system is running during the process, the user’s head is being tracked so they are free to move around during the validation process. 1. Superimpose a virtual Object, e. g. a small sphere, onto a known position on the board. 2. Ask the user to touch the board where they perceive the Object. 3. Move the Object to another location and goto step 1 until the area of the board is evenly covered. Once the area has been covered goto step 4. 4. Compute and display the system error. Algorithm 3: Evaluation module algorithm. 5.2 Planar Calibration Validation While an AR system is running, it may be important for the user to get an estimate of their current calibration accuracy. For instance, if the HMD is moved with respect to the user’s eye, then the calibration could be invalidated. If this situation occurs, 41 Evaluation Board Figure 5.1: The optical see-through evaluation system. the module presented in Section 5.1 can be invoked. ”pf -po” pb Board p0 Figure 5.2: Error is measured in the plane of the evaluation board as seen from the eyepoint, E. The true location is pt, the Observed location is p, and the projection of the observed point onto the board is pb. For validation, there are three points Of interest pt, p0, and pb. The first point, pt, is the theoretical location of the virtual point in the workspace coordinate system. This point Should be unknown to the user so there is no bias in the results. The theoretical point is rendered in the display using the calibration parameters of the system. This produces the second point, p0. p0 is the location where the user 42 perceives the point to be in the workspace. In principle, it is impossible for the user to tell the 3-dimensional location of the point p0. If the HMD and tracker calibrations are exact, then pt and p0 will coincide. AS this is not usually the case, a final point, pb must be introduced. p), is the projection of pa into the plane of the evaluation board. This point is selected by the user when they touch the board during system validation. Figure 5.2 shows the geometry of the 3 points. It is important to note that the planar error metric completely depends on the user’s viewpoint. Figure 5.3 illustrates this point. The observed error from the view- point E is larger than that of E’. Also notice that from multiple views it is possible to estimate the location of pa through the use of point reconstruction techniques. This issue is addressed in Section 5.4 E E, R. P; Eval Plane R p. ’ “ ”pt _pb” IIDI-Dbll —~ + Figure 5.3: Error in the plane Of the evaluation board as seen from the eye-points, E and E’. The error HP: - pbll is larger than llpt — p[,||. Once the data has been gathered, the evaluation module computes the Euclidean distances between each of the theoretical points and their projected Observed loca- 43 tions, i.e., ”Pt - Pb“: as shown in Figure 5.2. The statistics from these computed distances can be used as estimates of tolerances for the current calibration. The er- rors could also be used to give more direct feedback, such as drawing spheres at each Of the picked locations with colors and / or radii corresponding to the amount Of error. 5.3 Experimental Results for Planar Verification In this section, results from two different AR systems are presented. The first system uses a magnetic tracker; the second uses an infrared video-based tracker. Both systems have monocular displays which were calibrated with the method described in Chapter 4. After calibration, the users invoked the evaluation module described in Section 5.1. The results from a typical user of the magnetically tracked system are shown in Figure 5.4. This plot shows the theoretical locations as circles along with their observed locations from three different zones in the workspace. These zones are defined as the 3—dimensional regions to which the users’ heads were confined when the observations were made. The range of head motion was restricted because the users were seated during the evaluation procedures. Figure 5.5 shows an example of the three dimensional viewing zones for an evaluation run. The theoretical points were placed in the plane 2 = —0.63cm. The users recorded Observations by clicking the Observed location with the stylus from the magnetic tracker. The observations Of the points from the three viewing zones are indicated on the plot by plusses, stars and triangles. The mean and maximum errors for each zone and the global errors are tabulated in Table 5.1. Tables 5.2 and 5.3 Show the mean 44 Planar Observation Errors from Three Views 1:33:13<<<.: TNKKKSK- i 353K”: Kris- Marxxfié<~ :: “TSXSSS ._ mexflflakj X Loc (cm) Figure 5.4: Error fields for an AR system which uses a magnetic tracker. These are the planar errors as Observed from three zones in the workspace. 45 Z (cm) Viewing Zones 14\l'.......... X (cm) Y (cm) Figure 5.5: Viewing zones for an evaluation. 46 Table 5.1: Results for a single user from multiple viewing zones. Observation Zone 1 (cm) Zone 2 (cm) Zone 3 (cm) Global (cm) Mean Error 0.66 1.05 0.30 0.67 Max Error 1.20 1.75 0.50 1.75 Table 5.2: Average errors for different users from multiple viewing zones. Zone 1 (cm) Zone 2 (cm) Zone 3 (cm) Global (cm) 0.66 1.05 0.30 0.67 0.71 0.97 0.64 0.77 0.38 0.79 1.19 0.79 1.21 1.52 1.08 1.27 0.77 0.71 0.85 0.78 1.21 1.68 0.52 1.14 1.16 1.63 0.81 1.20 1.00 0.72 0.82 0.85 errors and maximum errors for eight different user sessions. For these eight sessions there were six users. The errors shown in rows 1 and 2 and rows 3 and 4 are from the same users during two different sessions. From the tables it can be seen that the overall average system accuracy is 0.93cm. The worst error Observed was 4.730m, but this was caused by the user moving outside the effective range of the tracker. If that point is treated as an outlier then the worst Observed error was 3.32cm. The results for the video-based system are shown in Figure 5.6. In this plot, the + symbols represent the actual locations of the points provided by the validation modules and the x symbols Show the Observed locations. The circles provide visual feedback of the errors at each location. 47 Table 5.3: Maximum errors for different users from multiple viewing zones. Zone 1 (cm) Zone 2 (cm) Zone 3 (cm) Global (cm) 1.20 1.75 0.50 1.75 0.90 1.25 0.90 1.25 0.82 1.26 2.02 2.02 3.23 2.39 1.85 3.23 1.72 1.13 1.21 1.72 2.35 2.76 1.22 2.76 1.92 2.54 1.15 2.54 4.73 3.19 3.02 4.73 Error Field for AR System using Video Tracking @693 @166 35- ‘ 30~ @6969 663(1) _ 25— @®®.®®@ 20- a case @636) 15r - 401 I i Y (cm) 10F - o_ -5 ~ + True Locations 6 x Observed Locations — Error Estimate _10 l l 1 -1O 0 1O 20 30 4O 50 X (cm) e696) @6963 - ECG Figure 5.6: Error fields for an AR system which uses a infrared video tracking system. P0 indicates the calibration origin and T0 indicates the tracker origin. 48 Reconstructed Points + Theoretical Location 0 Reconstructed Location 25 10 5 Y (cm) 0 X (cm) Figure 5.7: Reconstructed locations 5.4 Analysis of Structured Errors Given multiple observations of a point from different viewpoints it is possible to estimate its 3—dimensional location. If all of the points from the evaluation task are estimated they can be looked at collectively as an error field. Figure 5.7 shows an example of such an error field. In this plot, the plus symbols represent the theoretical locations of the points and the circles represent the reconstructed locations. Notice that the reconstructed points are roughly coplanar. This implies that a correction can be performed upon the calibration. 49 Given two or more image plane observations of the same point, and the render matrix for each observation its 3-dimensional location p can be estimated by solving where and Ap=h 911,1 912,1 921,1 922,1 911,2 912,2 A = 921,2 922,2 911m 912m 921,n 922,n 912,1 922,1 912,2 922,2 912,1; 922,n UI(931,1 + 932,1 + 933,1 + 934,1) — 914,1 UI(931,1 + 932,1 + 933,1 + 934,1) — 914,1 “1(9312 + 932,2 + 933,2 + 934,2) — 914,2 ”01(9312 + 932,2 + 933,2 + 934,2) — 914,2 Un(931,n + 932,11 + 933,1: + 934,11) — 914,n vn(931,n + 932,11 + 933,11 + 934,n) — 914,n 50 d (5.1) From the solutions it can be seen that there is some structure to the error. Using this structure it may be possible to make adjustments to the various calibrations in the system. This is left for future work. 5.5 Conclusions From the results, it is now possible to describe applications which can be built using the AR system which was tested. Given that the system tested has an average error on the order Of 1cm and a maximum error of around 3cm, it would be possible to build several useful applications with it. In fact, it would be possible to implement most of the applications listed in Chapter 1. Games and entertainment, e. g., Hunt the Wumpus, Tic-Tac-Toe, are quite practical, provided the game objects are large enough (6cm or larger) to correct for interaction errors. A maintenence or inspection task could also be implemented, again, given that the objects that need to be worked with are larger than the system tolerance. For instance, an application to look for missing components on a circuit board would be feasible, an application which required the user to touch a probe to a contact would not. This sort of application would require millimeter accuracy. Finally, surgery would be completely out Of the question. In such an application, the accuracy demands would be sub-millimeter. Deeper study of Optical see-through calibration methods and more accurate tracking systems will be needed to achieve this goal. Also, robust real-time registration systems for the medical data must be integrated into the system. The author will be exploring the effects Of a better tracking system as future work. 51 Chapter 6 Summary and Conclusions This thesis presents a general method to objectively evaluate an optical see-through augmented reality system. This method differs from previous methods in three ways. First, it allows a user to measure the error of the system while it is running. Second, it produces quantitative measurement of the error in registration of the real and the virtual. Finally, this error estimate includes the user as part of the system. With this method it is possible to estimate the accuracy limits of an application and detect structured errors in the tracking system. The method also gives insight which can be used to design new calibration algorithms. Several conclusions can be drawn from the experimental results. First, an AR sys— tem must be evaluated as a whole. While the errors of individual system components are both useful and meaningful to an implementor, they would be of little value to a maintenance technician using an AR system to perform repairs. Users need concise, simple feedback on the current system calibration accuracy. Second, during an evaluation Of an AR system, the user must be factored in. 52 Since all of the calibration procedures for the system components involve a manual step, there is a possibility that the user can introduce errors into the system. This is especially true if the user is ignorant of actions which may cause errors, such as facial muscle contractions during the calibration, e.g., talking, raising the brow, etc. Also, simplicity is a must for any calibration algorithm. If the user has to perform some difficult action, it is likely they will inject an error into the system. Other factors may cause errors during calibration and evaluation. For instance, poor lighting which can make alignment of the stylus to the Observed point less accurate since the user has difficulty seeing the stylus tip through the display. Third, since the observed errors are dependent on the viewpoint, the verification Should be performed from all zones that the user intends to work in. Also, multi- ple planes may be evaluated throughout the workspace. From multiple views of the same plane the 3-dimensional error can be estimated. This estimation uses a tech- nique from computer vision known as point reconstruction [7,16]. Figure 5.7 shows the reconstructed locations Of the points from a user. It may be possible to make corrections to a calibration using this information. 53 APPENDICES 54 Appendix A User Error Field Plots This appendix contains the error field plots from eight trials of the magnetically- tracked Optical see-through system. These six users had varying amounts of experi- ence with the system. Figures A.1 and A2 and Figures A.3 and AA were generated from the two primary investigators. The rest were volunteers. Careful study of these figures shows that the accuracy limit for the system is between 1 and 2 centimeters for a typical user. 55 I: T t» “t it “t “1 3. ‘87 “t ’t “t ‘3. “t. it I 3: 1+ 31 3 3 1 3. t: 3: 1+ t a. 3, 3, 8 :4 3 3 3. a. 4‘1. <1 N 3 a 3 4, a. at 22 i 2». to xi: (cm) #4 1'6 is 2'0 2‘2 Figure A.1: Three-view errors from a user. 56 8318.dat l :: AI: ”I I» I I I A: AIA‘Ié'IéI’IIIkI I AIAIAIAIATA'IAI AIAIAIAIIIIIII : “IIIAIAIAIA’IAI 4_ KIL‘IIIA—IA—IWW Figure A.2: Three-view errors from a user. 57 22 20 18 16 14 Y Loc (cm) a 10 — _q .1 — «— c— —i True Location View 1 View 2 View 3 6 8 1O 12 14 XLoc(cm) Figure A.3: Three-view errors from a user. 58 22 20 18 16 14 Y Loc (cm) R; 10 9318.dat f [Q‘D/fp/fp/‘TV‘T ff‘t/Iv/IP/IVR l I I l l T l I True Location View 1 View 2 ~ View 3 I>*+O d KEN WP- WW‘KIN Wfi$ MIN. wwv’x I\ MNKGXK L l l l l l l l l l 6 8 10 12 14 16 18 20 22 X Loc (cm) Figure A.4: Three-view errors from a user. 59 Y Loc (cm) 22 20- 4186.dat Figure A.5: Three-view errors from a. user. 60 ’ inkfoA/IA : kkAA/ka/Id - IkAXA’XKIAa/ia/fi - fiWflA/‘IKIA/Ia/E Afifi-WWA/IMM FA§+A$IIWWMM 2 é a 110 xi; (cm) 1‘4 1‘6 1'8 2‘0 2‘2 Y Loc (cm) 672 1 .dat Z IIIIII- MIMI- w-IIIIIII- j-IIIIIII; ;IIIIIII. NIKKI$o%_ Figure A.6: Three-view errors from a. user. 61 Y Loc (cm) I: }\%7‘\f\%\%\f\ WYVMWMR I ‘2 IIIIIIA ‘:_ IIIIIIA ._ IIIA/WIA’II ._ aidfififiA/IKI Figure A.7: Three—view errors from a user. 62 8381 .dai 22 I I I I I T I fl L I True Location View 1 View 2 ~ View 3 IVM‘W/ ‘ N O I >¥+O YLo:(cm)_‘ I I I M by 1o_ firO 8“ 4% Ant-o $.53 :50 7) ‘ 6.— 4% i E *0 $0 +41”) m 4- _ 2 1 l l l l l l l l l 2 4 6 8 10 12 14 16 18 20 22 X Loc (cm) Figure A.8: Three-view errors from a user. The outliers in this figure are from the user moving outside the range of the tracker. 63 BIBLIOGRAPHY 64 [ll [2] [3] [4] [6] [7] [8] [9] [10] Bibliography R. Azuma. A survey of augmented reality. Presence: Teleoperators and Virtual Environments, 6(4):355—385, August 1997. R. Azuma and G. Bishop. Improving static and dynamic registration in an optical see-through display. Computer Graphics, pages 194—204, July 1994. M. Bajura, H. Fuchs, and Ohbuchi. Merging virtual objects with the real world: Seeing ultrasound imagery within the patient. Computer Graphics, pages 203—— 210, July 1992. M. Bajura and U. Neumann. Dynamic registration correction in augmented- reality systems. In Proceedings of the Virtual Reality Annual International Sym- posium (VRAIS ’95), pages 189—196, 1995. W. Birkfellner, K. Huber, F. Watzinger, M. Figl, F. Wanschitz, R. Hanel, D. Rafolt, R. Ewers, and H. Bergmann. A see-through hmd for computer-aided surgery. In 3rd International Workshop on Augmented Reality (I WAR 2000), pages 54—59, Munich, Germany, October 2000. A. Butz, T. Hollerer, S. Feiner, B. MacIntyre, and C. Beshers. Enve10ping users and components in a collaborative 3d augmented reality. In 2nd International Workshop on Augmented Reality (I WAR ’99), pages 35—44, San Francisco, CA, October 1999. O. Faugeras. T hree-dimensional Computer Vision: A Geometric Viewpoint. MIT Press, Cambridge, MA, 1993. S. Feiner, B. MacIntyre, and D. Seligmann. Knowledge-based augmented reality. Communications of the ACM, 36(2):53—62, 1993. M. A. Fischler and R. C. Bolles. Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM, 24(6):381—395, 1981. H. Fuchs and J. P. Rolland. Optical versus video see-through head-mounted displays in medical visualization. Presence, 9(3):287—309, 2000. 65 [11] [12] [13] [14] [15] [16] [17] [18] [19] [20] [21] [22] Y. Genc, N. Navab, M. Tuceryan, and E. McGarrity. Evaluation of optical see-through systems. Technical Report SCR-Ol-TR-688, Siemens Corporate Re— search, January 2001. Y. Genc, F. Sauer, F. Wenzel, M. Tuceryan, and N. Navab. Stereo optical see-through HMD calibration: A one step method validated via a video see- through system. In International Symposium for Augmented Reality, pages 165— 174, Munich, Germany, October 2000. W.E.L. Grimson, G. J. Ettinger, S.J. White, P.L. Gleason, T. Lozano-Perez, III W. M. Wells, and R. Kikinis. Evaluating and validating an automated reg- istration system for enhanced reality visualization in surgery. In Proc. of the IEEE Conference on Computer Vision, Virtual Reality and Robotics in Medicine ( CVRMed ’95), pages 3—12, 1995. R. Holloway. An Analysis of Registration Errors in a See- Through Head-Mounted Display System for Craniofacial Surgery Planning. PhD thesis, University of North Carolina at Chapel Hill, 1994. R. L. Holloway. Registration error analysis for augmented reality. Presence: Teleoperators and Virtual Environments, 6(4):413—432, August 1997. Ramesh Jain, Rangachar Kasturi, and Brian G. Schunck. Machine Vision. McGraw—Hill, New York, 1995. A. Janin, D. Mizell, and T. Caudell. Calibration of head-mounted displays for augmented reality applications. In Proc. of the Virtual Reality Annual Interna- tional Symposium (VRAIS’93), pages 246—255, 1993. H. Kato and M. Billinghurst. Marker tracking and HMD calibration for a video- based augmented reality conferencing system. In Proceedings of the 2nd IEEE and ACM International Workshop on Augmented Reality ’99, pages 85—94, San Francisco, CA, October 20—21 1999. V. Kindratenko. Calibration of electromagnetic tracking devices. Virtual Reality: Research, Development, and Applications, 4:139—150, 1999. G. J. Klinker, K. H. Ahlers, D. E. Breen, P.-Y. Chevalier, C. Crampton, D. S. Greer, D. Koller, A. Kramer, E. Rose, M. Tuceryan, and R. T. Whitaker. Conflu- ence of computer vision and interactive graphics for augmented reality. Presence: Teleoperators and Virtual Environments, 6(4):433—~451, August 1997. K. N. Kutulakos and J. R. Vallino. Affine object representations for calibration- free augmented reality. In Proceedings of the IEEE Virtual Reality Annual In- ternational Symposium, 1996. W. Lorensen, H. Cline, C. Nafis, R. Kikinis, D. Altobelli, and L. Gleason. En- hancing reality in the operating room. In Proceedings of the IEEE Conference on Visualization, pages 410—415, 1993. 66 [23] [24] [25] [26] [27] [28] [29] [30] [31] [32] [33] [34] D. G. Lowe. Robust model-based motion tracking through the integration of search and estimation. International Journal of Computer Vision, 8(2):113—122, August 1992. E. McGarrity and C. B. Owen. Aris: Augmented reality inspection system. Technical report, Michigan State University Manufacturers Research Consor- tium, East Lansing, Michigan, 2000. E. McGarrity and M. Tuceryan. A method for calibrating see-through head- mounted displays for augmented reality. Technical Report TR—CIS—0499-13, Indiana University Purdue University Indianapolis, April 1999. E. McGarrity, M. Tuceryan, C. Owen, Y. Genc, and N. Navab. Evaluation of optical see-through systems. In EUROIMAGE International Conference on Augmented, Virtual Environments and Three Dimensional Imaging, Mikanos, Greece, June 2001. Erin McGarrity and Mihran Tuceryan. A method for calibrating see-through head-mounted displays for AR. In 2nd International Workshop on Augmented Reality (I WAR ’99), pages 75—84, San Francisco, CA, October 1999. J. P. Mellor. Real-time camera calibration for enhanced reality visualizations. In Proc. of the IEEE Conference on Computer Vision Virtual Reality and Robotics in Medicine ( CVRMed ’95), pages 471—475, 1995. U. Neumann and Y. Cho. A self-tracking augmented reality system. In Proceed- ings of the ACM Symposium on Virtual Reality and Applications, pages 109—115, 1996. T. Oishi and S. Tachi. Methods to calibrate projection transformation parameters for see-through head-mounted displays. Presence, 5(1):122—135, 1996. H. Tamura. What happens at the border between real and virtual worlds: The mr project and other research activities in japan. In 3rd International Workshop on Augmented Reality (I WAR 2000), pages xii-xv, Munich, Germany, October 2000. M. Tuceryan, D. Greer, R. Whitaker, D. Breen, C. Crampton, E. Rose, and K. Ahlers. Calibration requirements and procedures for a monitor-based aug- mented reality system. IEEE Transactions on Visualization and Computer Graphics, 1(3):255—273, September 1995. M. Tuceryan and N. Navab. Single point active alignment method (SPAAM) for optical see-through HMD calibration for AR. In International Symposium for Augmented Reality, pages 149—158, Munich, Germany, October 2000. M. Uenohara and T. Kanade. Vision-based object registration for real-time image overlay. In Proc. of the IEEE Conference on Computer Vision, Virtual Reality and Robotics in Medicine ( CVRMed ’95), pages 13—22, 1995. 67 [35] Z. Zhang. A flexible new technique for camera calibration. Technical Report MSR-TR—98-71, Microsoft Research, March 1999. 68 IIIIjIIgIIIIIIIIII