DEVELOPMENT OF STRUCTURED ILLUMINATION REFLECTANCE IMAGING TECHNIQUE AS A NEW MODALITY FOR ENHANCED DEFECT DETECTION OF APPLES By Yuzhen Lu A DISSERTATION Submitted to Michigan State University in partial fulfilment of the requirements for the degree of Biosystems Engineering—Doctor of Philosophy 2018     DEVELOPMENT OF STRUCTURED ILLUMINATION REFLECTANCE IMAGING TECHNIQUE AS A NEW MODALITY FOR ENHANCED DEFECT DETECTION OF APPLES By ABSTRACT Yuzhen Lu Computer vision technology coupled with uniform illumination is still unsatisfactory in detecting some defects of apples, e.g., subsurface tissue bruising. Structured illumination (SI) provides an alternative illumination scheme, which is advantageous in obtaining high-contrast/resolution images and controlling light penetration for depth-resolved imaging. The objective of this research was therefore to develop a new imaging modality based on SI, which is hereinafter referred to as structured-illumination reflectance imaging (SIRI), to enhance defect detection of apples. A SIRI platform, capable of acquiring broadband and multispectral images, was constructed by using a digital micro-mirror device based digital light projector. Using sinusoidally-modulated illumination patterns, SIRI acquires phase-shifted pattern images from samples, which are then demodulated into three sets of images, direct component (DC) and amplitude component (AC), and phase, each of which provides different image features. An image formation model and a general demodulation methodology for SIRI were proposed for assessing the features of the technique regarding image contrast and resolution and light penetration. Experiments for standard targets and apple samples showed that AC resulted in higher image resolution and contrast but lower light penetration than DC, and that its effectiveness in defect detection depended on defect type, fruit variety or surface morphology, and spatial frequency of illumination patterns. Demodulation is a critical step in obtaining DC and AC images, and conventional approaches require a minimum of three phase-shifted pattern images. New demodulation approaches based on spiral phase transform (SPT) and Gram-Schmidt orthonormalization, which required only two     phase-shifted pattern images, were proposed to improve the SIRI image acquisition speed and lay a foundation for future real-time implementation of the technique. In addition, a new approach, based on solving a linear system composed of phase-shifted pattern images, was also proposed for demodulating composite sinusoidal patterns that contain more than one frequency components. Furthermore, a phase analysis methodology, based upon phase demodulation, phase unwrapping, phase-to-height conversion and in-plane calibrations, was proposed for reconstructing the three- dimensional (3-D) geometry of apples from acquired SIRI pattern images to aid in detection of surface concavities (i.e., stem/calyx regions) and enhance defect detection. Experiments were conducted on detection of subsurface bruising in apples of four varieties by using a multispectral SIRI system, in conjunction with the SPT demodulation approach. The six wavebands between 710 and 810 nm yielded the overall detection errors of 11.7-14.2%. Further experiments were conducted on detection of general surface and subsurface defects for apples of two varieties by using SIRI images at 730 nm, aided with machine learning algorithms. Compared to DC and AC images used individually, their combinations and or plus RT (ratio, the division of AC by DC) images achieved significantly better classification results, and convolutional neural network (CNN) resulted in the highest overall accuracies of 98% for both varieties of apples. Finally, an exploratory study was conducted on using SIRI, based on the two-phase SPT demodulation, to acquire pattern images from moving samples to evaluate the potential of the technique for future real-time applications, which accentuated the need for a high-speed camera for SIRI to operate at a practically acceptable sample moving speed. This research has demonstrated the feasibility and potential of SIRI as a new versatile and effective tool for enhancing defect detection of apples. More research, however, is needed to improve the instrumentation and software of SIRI for real-time, practical applications.     This dissertation is dedicated to my wife, parents and grandparents. Thank you all for always supporting me. iv   ACKNOWLEDGEMENTS I gratefully acknowledge the support and assistance from many people, which have made possible the completion of the PhD research. First, I would like to express my deepest gratitude to Dr. Renfu Lu, my major advisor and guidance committee chairman, for his consistent support, unfailing encouragement, professional guidance, and above all, his patience and friendship throughout the course of this study and research. All have helped shaped me into the person I am today. My profound gratitude also goes to other guidance committee members, Dr. Daniel Guyer, Dr. Randolph Beaudry, Dr. Lalita Udpa and Dr. Xiaoming Liu, for their time, support, and timely advice. Special thanks to Dr. Guyer for his kindly help to improve my teaching skill and meticulous proofreading of my dissertation. I wish to acknowledge the support and friendship from the current and former lab colleagues: Dr. Zhao Zhang, Dr. Anand Kumar Pothula, Dr. Haiyan Cen, Dr. Fernando Mendoza, Dr. Wei Lu, Mr. Richard Li, Mr. Dong Hu, Mr. Aichen Wang, Ms. Yuping Huang, Ms. Ye Sun and Mr. Yifei Xu. My appreciation is also extended to the office staff in the Department of Biosystems and Agricultural Engineering, who are always courteous, professional and willing to offer help when requested. I also acknowledge the financial support provided to me during my study, including the graduate research assistantship, the Galen and Ann Brown Scholarship by the Department of Biosystems and Agricultural Engineering and the Dissertation Completion Fellowship by the College of Agriculture and Natural Resources. Last but not least, I am indebted to my beloved wife Fen Liu, my parents and grandparents. You all are my greatest supporters, and obviously essential to the completion of my study and research at Michigan State University.   v TABLE OF CONTENTS LIST OF TABLES ....................................................................................................................................... ix  LIST OF FIGURES ...................................................................................................................................... x  KEY TO ABBREVIATIONS AND SYMBOLS ..................................................................................... xvii  1. INTRODUCTION .................................................................................................................................... 1  1.1. BACKGROUND ............................................................................................................................... 1  1.2. SIGNIFICANCE OF RESEARCH .................................................................................................... 9  1.3. OBJECTIVES AND HYPOTHESIS ................................................................................................. 9  2. LITERATURE REVIEW OF NON-DESTRUCTIVE DEFECT DETECTION OF APPLES .............. 11  2.1. DEFECTS ........................................................................................................................................ 11  2.1.1. Physiological Disorders ............................................................................................................ 11  2.1.2. Mechanical Damage .................................................................................................................. 15  2.1.3. Pathological Disorders and Contamination ............................................................................... 17  2.2. NON-DESTRUCTIVE TECHNIQUES AND APPLICATIONS ................................................... 19  2.2.1. NIR Spectroscopy ..................................................................................................................... 20  2.2.2. Time-resolved and spatially resolved spectroscopic techniques ............................................... 22  2.2.3. Fluorescence Techniques .......................................................................................................... 24  2.2.4. Machine Vision ......................................................................................................................... 28  2.2.5. Hyperspectral and Multispectral Imaging ................................................................................. 31  2.2.6. X-ray Imaging ........................................................................................................................... 35  2.2.7. MRI ........................................................................................................................................... 37  2.2.8. Thermal Imaging ....................................................................................................................... 41  2.2.9. Raman Techniques .................................................................................................................... 43  2.2.10. Biospeckle Imaging ................................................................................................................ 45  2.3. SUMMARY ..................................................................................................................................... 46  3. CONSTRUCTION OF A SIRI SYSTEM AND EXPERIMENTAL ANALYSIS OF ITS FUNDAMENTAL FEATURES ................................................................................................................. 49  3.1. INTRODUCTION TO STRUCTURED ILLUMINATION REFLECTANCE IMAGING ............ 49  3.2. CONSTRUCTION OF A SIRI PLATFORM .................................................................................. 56  3.3. DEMODULATION ......................................................................................................................... 59  3.4. PRELIMINARY TESTS .................................................................................................................. 60  3.5. ANALYSIS OF SIRI FEATURES .................................................................................................. 65  3.5.1. Theoretical Framework ............................................................................................................. 65  3.5.1.1. Image Formation ................................................................................................................ 65  3.5.1.2. Demodulation ..................................................................................................................... 68  3.5.1.3. Image Resolution and Contrast .......................................................................................... 68  3.5.1.4. Light Penetration ................................................................................................................ 70  3.5.2. Experimental Validation ........................................................................................................... 73  3.5.2.1. Four Experiments ............................................................................................................... 73  3.5.2.2. Results and Discussion ....................................................................................................... 76  3.5.3. Conclusions ............................................................................................................................... 86  4. IMAGE DEMODULATION APPROACHES ....................................................................................... 88    vi 4.1. INTRODUCTION ........................................................................................................................... 88  4.2. SINGLE-FREQUENCY DEMODULATION ................................................................................. 91  4.2.1. Demodulation Approaches ........................................................................................................ 91  4.2.1.1. SPT ..................................................................................................................................... 91  4.2.1.2. GSO .................................................................................................................................... 96  4.2.2. Performance Evaluation ............................................................................................................ 98  4.2.2.1. SPT-based Demodulation .................................................................................................. 98  4.2.2.2. GSO-based Demodulation ............................................................................................... 105  4.2.3. Discussion ............................................................................................................................... 109  4.3. COMPOSITE PATTERN DEMODULATION ............................................................................. 111  4.3.1. Demodulation .......................................................................................................................... 111  4.3.2. Numerical Simulations ............................................................................................................ 114  4.3.3. Experiment on Bruise Detection ............................................................................................. 116  4.3.3.1. Conventional Single-frequency Demodulation ................................................................ 117  4.3.3.2. Dual-frequency Demodulation ......................................................................................... 118  4.3.3.3. Triple-frequency Demodulation ....................................................................................... 121  4.3.4. Discussion ............................................................................................................................... 123  4.4. CONCLUSIONS ............................................................................................................................ 124  5. THREE-DIMENSIONAL GEOMETRY RECONSTRUCTION ......................................................... 126  5.1. INTRODUCTION ......................................................................................................................... 126  5.2. PHASE ANALYSIS FOR SURFACE PROFIELING .................................................................. 128  5.2.1. Phase-height Relationship Formulation .................................................................................. 128  5.2.2. Phase Demodulation and Unwrapping .................................................................................... 131  5.2.3. Absolute Phase Map ............................................................................................................... 134  5.2.4. Phase-to-height Calibration ..................................................................................................... 135  5.3. IN-PLANE CALIBRATION ......................................................................................................... 137  5.4. EXPERIMENTAL TESTS ON SYNTHETIC AND APPLE SAMPLES ..................................... 139  5.4.1. SIRI System ............................................................................................................................ 139  5.4.2. Samples ................................................................................................................................... 140  5.4.3. Image Acquisition and Processing .......................................................................................... 141  5.4.4. Performance Evaluation .......................................................................................................... 143  5.5. RESULTS AND DISCUSSION .................................................................................................... 144  5.5.1. System Calibrations ................................................................................................................ 144  5.5.1.1. Phase-to-height Calibration .............................................................................................. 144  5.5.1.2. In-plane Calibration ......................................................................................................... 146  5.5.2. Tests on Apples ....................................................................................................................... 148  5.5.3. Discussion ............................................................................................................................... 152  5.6. CONCLUSIONS ............................................................................................................................ 154  6. SUBSURFACE AND SURFACE DEFECT DETECTION ................................................................. 156  6.1. INTRODUCTION ......................................................................................................................... 156  6.2. DETECTION OF FRESH SUBSURFACE BRUISING ............................................................... 158  6.2.1. Material and Methods ............................................................................................................. 158  6.2.1.1. Apple Samples ................................................................................................................. 158  6.2.1.2. Image Acquisition ............................................................................................................ 159  6.2.1.3. Image Processing ............................................................................................................. 160  6.2.2. Results and Discussion............................................................................................................ 166  6.2.2.1. Demodulated Images ........................................................................................................ 166  6.2.2.2. PCA .................................................................................................................................. 168  6.2.2.3. Performance Evaluation ................................................................................................... 169    vii 7. EXPLORATION OF SIRI FOR DETECTION OF MOVING SAMPLES ......................................... 194  7.1. INTRODUCTION ......................................................................................................................... 194  7.2. SYSTEM SETUP ........................................................................................................................... 195  7.3. EXPERIMENTAL TESTS ............................................................................................................ 197  7.4. APPROACHES TO OBJECT ALIGNMENT ............................................................................... 198  7.5. IMAGE PROCESSING ................................................................................................................. 199  7.6. IMAGE QUALITY ASSESSMENT ............................................................................................. 203  7.7. RESULTS AND DICUSSION ...................................................................................................... 204  7.8. CONCLUSIONS ............................................................................................................................ 211  8. CONCLUSIONS AND FUTURE WORK ........................................................................................... 213  APPENDICES .......................................................................................................................................... 219  APPENDIX A: MODIFIED OTSU THRESHOLDING ..................................................................... 220  APPENDIX B: UNIMODAL THRESHOLDING ................................................................................ 223  APPENDIX C: BI-DIMENSIONAL EMPIRICAL MODE DECOMPOSITION ............................... 224  BIBLIOGRAPHY ..................................................................................................................................... 227  6.2.2.4. Discussion ........................................................................................................................ 171  6.3. DETECTION OF SURFACE AND SUBSURFACE DEFECTS .................................................. 172  6.3.1. Materials and Methods ............................................................................................................ 174  6.3.1.1. Apple Samples ................................................................................................................. 174  6.3.1.2. Image Acquisition ............................................................................................................ 175  6.3.1.3. Image Preprocessing ........................................................................................................ 176  6.3.1.4. Classification by RF and SVM ........................................................................................ 177  6.3.1.5. Classification by CNN ..................................................................................................... 180  6.3.1.6. Performance Evaluation ................................................................................................... 183  6.3.2. Results and Discussion............................................................................................................ 184  6.3.2.1. Image Enhancement by BEMD ....................................................................................... 184  6.3.2.2. Enhanced DC, AC and RT Images .................................................................................. 185  6.3.2.3. Classification Results ....................................................................................................... 186  6.3.2.4. Discussion ........................................................................................................................ 190  6.4. Conclusions .................................................................................................................................... 191    viii LIST OF TABLES Table 4.1. Comparison of four different approaches to amplitude demodulation ........................................ 100  Table 4.2. Contrast indexes for the four demodulation approaches for two apples...................................... 104  Table 4.3. Root mean square errors and processing times for the three demodulation approaches ........... 105  Table 4.4. Contrast indexes calculated from the amplitude images of apples for the four demodulation approaches……………… ..................................................................................................................................... 109  Table 4.5. Relative demodulation errors (in percent) for three demodulation approaches for dual-frequency and triple-frequency patterns................................................................................................................................. 116  Table 4.6. Contrast indexes (CIs) for the three demodulation methods for demodulating dual-frequency composite patterns………………… .................................................................................................................... 121  Table 4.7. Contrast indexes (CIs) for different methods of demodulating triple-frequency composite patterns……… ........................................................................................................................................................ 123  Table 5.1. Overall root-mean-square error (RMSE) for each calibration method. ....................................... 144  Table 5.2. Calibration residuals for eight pairs of control points in the calibration object. ........................ 147  Table 6.1. Results for detecting fresh bruises in four varieties of apples. ..................................................... 169  Table 6.2. Number of apples with different types of defects. .......................................................................... 174  Table 7.1. The quality index (QI) for the direct component (DC) and amplitude component (AC) images (shown in Figure 7.9 and Figure 7.10) obtained without (case 1) and with (case 2) object alignment for image demodulation, for the different sample displacements in acquiring two phase-shifted pattern images……… ......................................................................................................................................................... 209  ix   LIST OF FIGURES Figure 1.1. Schematic of uniform illumination (left) and structured illumination (right) system designs. .. 8  Figure 3.1. Schematic of three types of optical property measurement techniques: (a) spatially-resolved continuous wave, (b) time-domain and (c) temporal frequency-domain, where solid and dashed lines denote input and output (detected) signals, respectively. ................................................................................................ 51  Figure 3.2. Schematic of spatial frequency domain imaging (SFDI) for optical property measurement (modified from Fig. 3 in Cuccia et al., 2009). Spatially-modulated light is impinged on tissue, and it decays exponentially with depth z, and the remitted reflectance that is measured at multiple spatial frequencies is used to inversely estimate tissue optical properties. ............................................................................................ 53  Figure 3.3. Schematic (left) and photograph (right) of a broadband structured-illumination reflectance imaging (SIRI) system………… ............................................................................................................................ 57  Figure 3.4. Two versions of a multispectral structured-illumination reflectance imaging (SIRI) system. . 58  Figure 3.5. The graphical user interface (GUI) for structured-illumination reflectance imaging (SIRI). ... 59  Figure 3.6. Examples of three phase-shifted pattern images of an apple, obtained under the illumination of sinusoidal pattern with phase offsets of -2π/3, 0 and 2π/3 (from left to right), respectively. ........................ 60  Figure 3.7. Schematic (left) and photo (right) of a scattering nylon block with ink inclusions. .................. 61  Figure 3.8. Photographs of the impact pendulum (left) and bruised apples (right). ....................................... 61  Figure 3.9. Direct component (DC) and amplitude component (AC) images of the Nylon slab at the spatial frequencies of 0.02, 0.04, 0.05, 0.06, 0.08, 0.10 and 0.12 cycles/mm. ............................................................. 63  Figure 3.10. Direct component (DC) and amplitude component (AC) images of ‘Gala’ (a) and ‘Golden Delicious’ apples at the spatial frequencies of 0.03, 0.05, 0.08, 0.10, 0.15, 0.18, 0.20 and 0.25 cycles/mm……. ........................................................................................................................................................ 64  Figure 3.11. Direct component (DC) and ratio images of ‘Gala’ (a) and ‘Golden Delicious’ apples at the spatial frequencies of 0.03, 0.05, 0.08, 0.10, 0.15, 0.18, 0.20 and 0.25 cycles/mm. ....................................... 65  Figure 3.12. (a) A simulated modulation transfer function (MTF) profile for a structured-illumination reflectance imaging (SIRI) system, and (b) visualization of the spectral shifting of a SIRI pattern image in the Fourier domain [Note: the notations in the figure comply with those in Eq. (3.12)]. .............................. 69  Figure 3.13. The simulated modulation transfer function (MTF) profiles of apple tissues obtained through Monte-Carlo (MC) simulation and diffusion approximation (Cuccia et al., 2009), assuming a typical pair of optical properties of absorption ( ) coefficients. In addition to the optical properties, other settings for the MC simulation include 5 million photons, reflective index of 1.35, anisotropy factor of 0.9 and thickness of 10 cm for the (slab-geometry) tissue. .................... 72  ) and reduced scattering ( 1  0.2 cm 1  10 cm  a  s   x Figure 3.14. Schematic of light scattering within a turbid medium for reflectance measurements, showing three types of backscattered photons, i.e., ballistic, weakly scattered and diffusive (or multiply scattered), with corresponding light penetration regimes. ..................................................................................................... 72  Figure 3.15. Photograph of a ‘Delicious’ apple (left) with invisible subsurface bruising, a ‘Red Rome’ apple (center) with lenticel dark spots, and a ‘Delicious’ apple (right) with skin russeting. ......................... 76  Figure 3.16. (a) Spectral responses of direct component (DC) and amplitude component (AC) images at different spatial frequencies, where the line profiles, which have been scaled by dividing the maximum value of DC, correspond to individual spatial frequencies ranging between 0.01 and 1.0 cycles/mm, and the downward arrow indicates the direction of increasing spatial frequencies (fx); (b) frequency responses of the DC and AC images and the noise level at 730 nm, where each line profile has been normalized to the maximum of the DC image. .................................................................................................................................... 77  Figure 3.17. The direct component (DC) (left) and amplitude component (AC) (middle) images of the USAF 1951 target at the spatial frequency of 0.15 cycles/mm, and the close-up view (right) of the AC image focusing on the groups 0 and 1 elements of the USAF 1951 target. ..................................................... 78  Figure 3.18. The close-up view of direct component (DC) and the amplitude component (AC) images at different spatial frequencies (cycles/mm) as indicted in the name of individual sub-images, for the bars of groups 0 and 1 of the USAF 1951 target. .............................................................................................................. 79  Figure 3.19. The close-up image and pixel profiles for the corresponding lines in the DC and AC images: (top pane) vertically orientated illumination patterns, (middle pane) horizontally orientated illumination patterns, and (bottom pane) two-frequency grid illumination patterns. ............................................................ 80  Figure 3.20. The curves of contrast values for the full-size (left) and close-up (right) direct component (DC) and amplitude component (AC) images of the USAF 1951 target at different spatial frequencies. ............. 81  Figure 3.21. Uniform illumination images of the USAF 1951 paper target placed beneath 1-6 (from left to right) sheets of blank printing paper. ..................................................................................................................... 81  Figure 3.22. The direct component (DC) and amplitude component (AC) images for two paper targets for different spatial frequencies: (top three rows) the Snellen target is placed on top of the USAF 1951, and (bottom three rows) one sheet of blank paper is laid on top of the Snellen target that is on top of the USAF 1951 target…. ............................................................................................................................................................ 82  Figure 3.23. The direct component (DC) image (left), amplitude component (AC) image at 0.30 cycles/mm (AC_F0.3) (second from left), and the reconstructed images (three from right) by subtracting the AC image multiplied by a constant (i.e., 1, 1.5 and 2, respectively) from the DC image. It should be noted that the heuristics for deciding an optimal constant is open to question. ........................................................................ 83  Figure 3.24. The direct component (DC) and amplitude component (AC) images at different spatial frequencies (cycles/mm) for a ‘Delicious’ apple with subsurface bruising. .................................................... 84  Figure 3.25. The direct component (DC) and amplitude component (AC) images at different spatial frequencies (cycles/mm) for a ‘Red Rome’ apple with lenticel breakdown (top three rows) and a ‘Delicious’ apple with surface russeting (bottom three rows). ............................................................................................... 85  Figure 4.1. Real and imaginary parts of the spiral phase function (left and middle), and the distribution of its phase angle (right) where only principal values in the range ±π are shown. .............................................. 93    xi Figure 4.2. Simulation images: (a) direct component (DC), and fringe pattern images with phase offset - 2π/3, 0 and 2π/3 [(b), (c) and (d), respectively]. .................................................................................................. 99  Figure 4.3. Amplitude images obtained by the conventional three-phase demodulation (TPD) and three spiral phase transform (SPT) based approaches. ................................................................................................ 100  Figure 4.4. The effect of noise variance on the relative demodulation error (RDE) for the conventional three-phase demodulation (TPD) and three spiral phase transform (SPT) based approaches. .................... 101  Figure 4.5. The direct component (DC) and three phase-shifted pattern images from left to right for “Gala” (top) and “Golden Delicious” (bottom) apples. .................................................................................................. 102  Figure 4.6. The amplitude images obtained by conventional three-phase demoduation (TPD) and three spiral phase transform (SPT) based appraoches (i.e., “2-phase SPT’ and “2-image SPT” and “1-phase SPT”) from left to right for “Gala” (top) and “Golden Decious” (bottom) apples. The defect regions correspond to the areas enclosed by red solid lines in the amplitude images obtained by “2-phase SPT”. ....................... 103  Figure 4.7. Direct component (DC) and amplitude images obtained by spiral phase transform (SPT) based on two phase-shifted pattern images at the spatial frequencies of 0.05, 0.10, 0.15 and 0.20 cycles/mm (from left to right) for the mildly (top) and severely (bottom) bruised “Jonagold” apples. .................................... 105  Figure 4.8. Root-mean-square error (RMSE) of amplitude versus different noise levels of images without (a) and with (b) filtering preprocessing. SPT, GSO and TPD represent spiral phase transform, Gram- Schmidt orthonormalization and three-phase demodulation, respectively. .................................................... 106  Figure 4.9. (a) Root mean square error (RMSE) of amplitude versus phase difference for GSO1 and GSO2 methods, and (b) RMSE of amplitude versus frequency for GSO1 and GSO2 methods, where GSO represents Gram-Schmidt orthonormalization. ................................................................................................... 107  Figure 4.10. The direct component images of the bruised apples recovered by the three-phase demodulation (TPD) (a) and the spiral phase function (SPF) method. .................................................................................... 108  Figure 4.11. Amplitude images of bruised ‘Gala’ (a) and ‘Jonagold’ (b) apples obtained by SPT, GSO1, GSO2 and TPD methods at the spatial frequencies of 0.08 cycles/mm (top) and 0.15 cycles/mm (bottom). The red arrows and circled curves indicate the presence of the boundary artifacts and the bruised regions of apples, respectively. ............................................................................................................................................... 108  Figure 4.12. Simulation composite patterns: dual-frequency fringe pattern, dual-frequency grid pattern, triple-frequency fringe and triple-frequency grid pattern (from left to right). ............................................... 115  Figure 4.13. Direct component (DC, top row), amplitude component (AC, mid row) and ratio of AC to DC (bottom row) images of ‘Golden Delicious’ (a, left) and ‘Gala’ (b, right) apples, obtained by using single- frequency patterns at the spatial frequencies of 0.05, 0.10, and 0.20 cycles/mm, respectively. ................. 118  Figure 4.14. Amplitude images of ‘Gold Delicious’ (a) and ‘Gala’ (b) apples obtained by ‘Phase shifting’ for fringe composite patterns (top) and grid composite patterns (bottom) with dual spatial frequencies of 0.05 and 0.20 cycles/mm……. ............................................................................................................................. 119  Figure 4.15. Amplitude images of ‘Golden Delicious’ (a) and ‘Gala’ (b) apples obtained by ‘Phase shifting with SPT’ using fringe composite patterns (top) and grid composite patterns (bottom) with dual spatial frequencies of 0.05 and 0.20 cycles/mm. ............................................................................................................ 120    xii Figure 4.16. Amplitude images of ‘Golden Delicious’ (a) and ‘Gala’ apples (b) obtained by ‘Filtering with SPT’ using fringe composite patterns (top) and grid composite patterns (bottom) with dual spatial frequencies of 0.05 and 0.20 cycles/mm…….. .................................................................................................. 120  Figure 4.17. Amplitude images of ‘Golden Delicious’ (a) and ‘Gala’ apples, obtained by ‘Phase shifting’ using fringe composite patterns (top) and grid composite patterns (bottom) with triple frequencies of 0.05, 0.10 and 0.20 cycles/mm…… .............................................................................................................................. 121  Figure 4.18. Amplitude images of ‘Golden Delicious’ (a) and ‘Gala’ apples (b) obtained by ‘Phase shifting with SPT’ using fringe composite patterns (top) and grid composite patterns (bottom) with triple frequencies of 0.05, 0.10 and 0.20 cycles/mm. .................................................................................................. 122  Figure 4.19. Amplitude images of ‘Golden Delicious’ (a) and ‘Gala’ apples (b) obtained by ‘Filtering with SPT’ using fringe composite patterns (top) and grid composite patterns (bottom) with triple frequencies of 0.05, 0.10 and 0.20 cycles/mm…… ..................................................................................................................... 123  Figure 5.1. Geometry of the structured-illumination reflectance imaging (SIRI) system for height measurement.. ......................................................................................................................................................... 129  Figure 5.2. Examples of a raw fringe pattern (a) and a normalized fringe pattern (b) where the red line represents the detected center line of the left-most white fringe stripe. .......................................................... 135  Figure 5.3. Principle of the phase-to-height calibration. .................................................................................. 136  Figure 5.4. Perspective imaging model relating the world and image coordinate systems, where ξ and H denote the focal length of the lens and the distance between the lens center and the origin of the world coordinate system, respectively. The front image u-v plane is geometrically equivalent to the camera’s actual image plane, which is assumed to be parallel to the x-y plane of the world coordinate system. ..... 138  Figure 5.5. A photograph of the structured-illumination reflectance imaging system. ............................... 139  Figure 5.6. Two verification blocks used for phase-to-height and in-plane calibrations: rectangular Teflon block (left) and trapezoidal Nylon block (right). ............................................................................................... 140  Figure 5.7. Four ‘Gala’ apples in two different orientations, after having been subjected to impact bruising damage. The two halves in each photo constitute one apple. ........................................................................... 141  Figure 5.8. A calibration object (left), and its SIRI image (right) under planer illumination where eight points indicated by markers and letters are used as control points for in-plane calibrations. ...................... 142  Figure 5.9. (a) Curves of height versus phase difference, and (b) root-mean-square errors (RMSEs) at different heights obtained by different calibration methods for the reference plane. ................................... 144  Figure 5.10. Reconstructed height profiles of the rectangular (a) and trapezoidal (b) blocks, and the reconstructed height profile of the rectangular block (c) with 2π compensations. ........................................ 146  Figure 5.11. Reconstructed three-dimensional shape for the rectangular (a) and trapezoidal (b) blocks. 147  Figure 5.12. Raw pattern, normalized pattern, wrapped phase, unwrapped phase, phase difference, amplitude component (AC), direct component (DC) and ratio images for a normal (a) and a bruised (b) apple sample… ........................................................................................................................................................ 149    xiii Figure 5.13. Phase difference and ratio images for normal (left two columns) and bruised (right two columns) apples, where AC and DC denote amplitude and direct components, respectively. .................... 150  Figure 5.14. Reconstructed height profiles for all the apple samples, where the left two columns correspond to intact, normal apples, while the right two columns correspond to half, bruised apples. .......................... 151  Figure 5.15. Reconstructed three-dimensional shape for all the apple samples (expressed in real world dimensions for the three coordinates), where the left two columns correspond to intact, normal apples, while the right two columns correspond to half, bruised apples. ................................................................................ 152  Figure 5.16. Example of reconstructed height profiles for apple samples (the left two columns represent intact apples while the right two column for half apples) with using SPT for phase demodulation (2π phase compensations have been considered for the intact apples). ............................................................................ 153  Figure 6.1. Photograph of bruised ‘Delicious’, ‘Gala’, ‘Golden Delicious’ and ‘Granny Smith’ (from left to right) apples………… ....................................................................................................................................... 159  Figure 6.2. Schematic of a liquid-crystal tunable filter (LCTF) based multispectral structured-illumination reflectance imaging (SIRI) system. ...................................................................................................................... 159  Figure 6.3. Flowchart of principal component analysis (PCA) applied to multispectral images. .............. 162  Figure 6.4. Threshold selection for an artificial histogram (a) and a typical histogram (b) of a bruised apple, where Totsu, Tm1, Tm2 and Tm3 are obtained by the classic Otsu thresholding, and other three modified methods as described in Eqs. (6.3), (6.4) and (6.5), respectively. ................................................................... 165  Figure 6.5. Flow chart of the procedures of bruise segmentation. .................................................................. 166  Figure 6.6. Uniform illumination images of ‘Gala’, ‘Granny Smith’, ‘Golden Delicious’ and ‘Delicious’ (from right to left) apples… .................................................................................................................................. 167  Figure 6.7. Amplitude images of ‘Gala’ (a) and ‘Delicious’ (b) apples at the wavelengths from 650 to 950 nm………….. .......................................................................................................................................................... 167  Figure 6.8. Examples of the first principal component (PC1) score images for ‘Gala’, ‘Granny Smith’, ‘Golden Delicious’ and ‘Delicious’ (from right to left) apples. ....................................................................... 168  Figure 6.9. Loadings of the first principal component for four varieties of apples. ..................................... 169  Figure 6.10. Bruise segmentation by the threshold (T) obtained by the proposed method and the optimal threshold (Topt) by manual selection based on trials and errors. ...................................................................... 170  Figure 6.11. Photograph of a ‘Golden Delicious’ apple with bitter pit (left) and a ‘Delicious’ apple with frost ring (right)………… ..................................................................................................................................... 174  Figure 6.12. Schematic of a multispectral structured-illumination reflectance imaging system. ............... 175  Figure 6.13. Flow chart of image preprocessing procedures, where DC and AC represent direct and amplitude components, respectively. ................................................................................................................... 176    xiv Figure 6.14. (a) The out-of-bag error against the number of grown trees, and (b) the test error against the number of selected features, using the image input of the ensemble of direct component (DC), amplitude component (AC) and ratio images for two varieties of apples. ........................................................................ 179  Figure 6.15. The architecture of a convolutional neural network (CNN). ..................................................... 181  Figure 6.16. (a) Five augmented images generated by geometrical transformations of an original image (top left) and (b) Test accuracy and the number of learnable parameters versus the number of convolution- pooling layers in the convolutional neural networks using the ensemble of direct component, amplitude component and ratio images for two varieties of apples. .................................................................................. 182  Figure 6.17. Bi-dimensional empirical mode decomposition of a direct component (DC) image into six mode images and one residual image. ................................................................................................................. 184  Figure 6.18. Comparison of the original and reconstructed direct component (DC) and amplitude component (AC) images……….. ......................................................................................................................... 185  Figure 6.19. Example of direct component (DC, top row), amplitude component (AC, middle row) and ratio (RT, bottom row) images of ‘Delicious’ (the left three columns) and ‘Golden Delicious’ (the right three columns) apples with different types of defects. ...................................................................................... 185  Figure 6.20. Classification accuracies (in percent) obtained by random forest (top pair), support vector machine (middle) and convolution neural network (bottom) for the five forms of image input for the test group of ‘Delicious’ (left pane) and ‘Golden Delicious’ (right pane) apples, where ACC, PRE and REC denote overall accuracy, precision and recall, respectively; DC, AC and RT denote direct component, amplitude component and ratio images, respectively; and DC-AC and DC-AC-RT represent the two ensembles of two or three types of images. ........................................................................................................ 186  Figure 6.21. Comparisons of overall accuracies (in percent) by the three classifiers of random forest (RF), support vector machine (SVM) and convolutional neural network (CNN) for the five forms of image inputs for ‘Delicious’ and ‘Golden Delicious’ apples. DC, AC and RT denote direct component, amplitude component and ratio images, respectively, and DC-AC and DC-AC-RT represent two ensembles of DC and AC images and DC, AC, and RT images, respectively. Classification results in the same group with different letters are different at the 5% significance level………….. ............................................................. 187  Figure 6.22. Comparisons of random forest (RF), support vector machine (SVM) and convolutional neural network (CNN) in terms of overall test accuracies (in percent) for the five different image inputs for ‘Delicious’ and ‘Golden Delicious’ apples. DC, AC and RT denote direct component, amplitude component and ratio images, respectively, and DC-AC and DC-AC-RT represent the two ensembles of DC and AC and DC, AC, and RT, respectively. Classification results in the same groups with different letters are different at the 5% significance level. ................................................................................................................. 189  Figure 6.23. The direct component (DC, top row), amplitude component (AC, middle row) and ratio (RT, bottom row) images for the two defective ‘Delicious’ (the left two columns) and ‘Golden Delicious’ (the right two columns) apples which are misclassified by CNN using the ensemble of DC-AC-RT image input…………......................................................................................................................................................... 189  Figure 7.1. A structured-illumination reflectance imaging (SIRI) system with a motorized stage (left) and a graphical user interface for controlling the movement of the stage (right). ................................................ 196    xv Figure 7.2. Two consecutively acquired pattern images from a moving ‘Gala’ apple without (a) and with (a) a white reference maker. .................................................................................................................................. 199  Figure 7.3. An example of bi-dimensional empirical mode decomposition (BEMD) of a pattern image of a bruised ‘Gala’ apple……. ..................................................................................................................................... 200  Figure 7.4. Spiral phase transform (SPT) results of the first four intrinsic mode function (IMF) images shown in Figure 7.3 (from the left to the right). ................................................................................................. 201  Figure 7.5. The flowchart of image processing procedures for retrieving direct component (DC) and amplitude component (AC) images from two pattern images acquired from a moving sample. SPT and IMF denote spiral phase transform and intrinsic mode function. ............................................................................. 202  Figure 7.6. The direct component (DC) and amplitude component (AC) images for a stationary ‘Gala’ apple with subsurface bruising. ............................................................................................................................ 204  Figure 7.7. The direct component (DC) and amplitude component (AC) images, obtained by using the reference based approach (a) and based on automatic image processing (b) for object alignment, for a moving ‘Gala’ apple with subsurface bruising, at the moving speeds of 4, 8, 16, 24 and 32 mm/s (from left to right). The resultant quality index (QI) for each image was calculated, and all the images were resized to the region focusing on the fruit object based on unimodal thresholding of the DC image. ......................... 205  Figure 7.8. The direct component (DC) and amplitude component (AC) images, obtained by using the reference based approach (a) and based on automatic image processing (b) for object alignment, for four ‘Gala’ apples with subsurface bruising, at a moving speed of 24 mm/s. The resultant quality index (QI) for each image is shown above each image, and all the images were resized to the region focusing on the fruit object based on unimodal thresholding of the corresponding DC images. .................................................... 206  Figure 7.9. The direct component (DC) and amplitude component (AC) images for a bruised ‘Gala’ apple under the illumination of two phase-shifted patterns: (a) no sample displacement and (b) different sample displacement (D) ranging from 0.1 to 6.4 mm, for the two phase-shifted pattern images. .......................... 207  Figure 7.10. The direct component (DC) (top) and amplitude component (AC) (bottom) images for a bruised ‘Gala’ apple, with object alignment performed prior to image demodulation, for the sample displacement (D) ranging from 0.1 mm to 6.4 mm between the two phase-shifted pattern images. .......... 208 Figure A.1. Gray-scale histograms represented by probability distribution functions…………………..220 Figure A.2. Gray-scale histograms where s1, s2 and s0 are the areas of the corresponding shaded regions……………………………………………………………………………………………………221  Figure B.1. A unimodal thresholding method for determining a threshold from a histogram with a single dominant peaks located at lower end (left) and upper end (right) of the histogram…….………………..223            xvi KEY TO ABBREVIATIONS AND SYMBOLS Amplitude or alternating component Accuracy Artificial neural network Bi-dimensional empirical mode decomposition Controlled atmosphere Charge-coupled device Cumulative distribution function Contrast index Convolutional neural network Computed tomography Continuous wave Direct component Discrete cosine transform Digital fringe projection Digital light projector Digital micro-mirror device Diffuse optical tomography Empirical mode decomposition Electron-multiplying charge-coupled device Frequency-domain photon migration Fourier transform Field of view False positive xvii AC ACC ANN BEMD CA CCD CDF CI CNN CT CW DC DCT DFP DLP DMD DOI EMD EMCCD FDFM FT FOV FP     FN FST GSO GUI HT IFT IMF IR LBP LCTF LED LLE LS LSD MC MFP MI MLP MRI MSE MTF NEM NIR NMR PC False negative Fruit surface temperature Gram-Schmidt orthonormalization Graphical user interface Hilbert transform Inverse Fourier transform Intrinsic mode function Infrared Local binary pattern Liquid crystal tunable filter Light-emitting-diode Locally linear embedding Least squares Least significance difference Monte Carlo Mean free path Modulated imaging Multi-layer perception Magnetic resonance imaging Mean squared error Modulation transform function Noise-equivalent modulation Near-infrared Nuclear magnetic resonance Principal component   xviii PCA PD PLS Principal component analysis Photon density Partial least squares PLS-DA Partial least squares-discriminant analaysis PRE PS PSF QDA QI QTH RDE REC ReLU RF RGB RMSE RMSEP ROI RT SB SERS SFDI SI SIRI SN Precision Photo-oxidative sunburn Point spread function Quadratic discriminant analysis Quality index Quartz-tungsten halogen Relative demodulation error Recall Rectified linear unit Random forest Red-green-blue Root-mean-square error Root-mean-square error of prediction Region of interest Ratio image Sunburn browning Surface-enhanced Raman spectroscopy Spatial frequency domain imaging Structured illumination Structured-illumination reflectance imaging Sunburn necrosis   xix SNR SOM SPF SPT SRS SSC SVM SWIR TMFP TN TOF TP TPD TRS UI UV VIP Vis xf ACI DCI a s s   Signal-to-noise ratio Self-organization map Spiral phase function Spiral phase transform Spatial-resolved reflectance spectroscopy Soluble solids content Support vector machine Short wavelength infrared Transport mean free path True negative Time of flight True positive Three-phase demodulation Time-resoled reflectance spectroscopy Uniform illumination Ultraviolet Variable importance in projection Visible Spatial frequency along the x-axis Amplitude component image Direct component image Absorption coefficient Scattering coefficient Reduced scattering coefficient xx 1. INTRODUCTION 1.1. BACKGROUND Apple is one of the most economically important fruits, which is appreciated by consumers around the world due to its texture, flavor, visual appeal and nutritive values. In the U.S., apple production ranks third following grape (including wine, table and raisin grapes) and orange. In 2016, the U.S. produced 5.1 million tons of apples, including fresh and processing, which were valued at about $3.5 billion dollars at the farm gate (USDA-NASS, 2017). Apples are grown commercially in 29 states in the U.S. Washington state, in particular, produces about 65% of the apples in the U.S., and other leading states include New York, Michigan, Pennsylvania, California and Virginia (USDA-NASS, 2017). High-quality fruit are highly valued by consumers, and, hence, the major goal that apple producers, packinghouses and processors are striving to achieve. The quality of a horticultural commodity is often defined as a combination of attributes or characteristics that give value to the commodity, including appearance (size, shape, color, defects, etc.), texture (firmness, crispness, etc.), flavor (sweetness, sourness, astringency, etc.), nutritional value and safety (Kader, 2002). The importance of these quality factors vary with the intended use and target market of the product. The quality can also be defined in terms of consumer acceptability (Shewfelt, 1999), and it can be divided into purchase quality that is related to buying decisions of consumers, and consumption or eating quality that is related to flavor and mouthfeel of the product (Shewfelt, 2014; Walsh, 2014). Despite the two viewpoints of quality, it is generally agreed that appearance is among the most important factors in pricing fruit on the market. In particular, defects on the surface of fruit are a negative quality factor that normally discourages consumers from making a purchase. Defects can   1 also occur beneath the fruit surface or deep inside the fruit, which are detrimental to consumer loyalty and affect repeated purchasing decisions. Maintaining high- and consistent-quality individual fruits is of great importance for the apple industry, which requires evaluating or inspecting individual fruits for various quality attributes and sorting and grading them into different quality grades (e.g., U.S. Extra Fancy, Fancy, No. 1 or Utility). Inspecting fruits during postharvest handling would offer quality assurance to consumers, add value to the fruits and eliminate defective or unmarketable products, which will lead to improved consumer satisfaction, reduced management cost and thus industry profitability. Quality inspection would also assist packinghouses or warehouses in implementing appropriate handling regimes to maintain product quality and help retailers develop better marketing strategies for meeting the needs of different customer groups. Sorting fruit can also be done at harvest (Mizushima and Lu, 2011), which would reduce the cost of transportation, postharvest storage and packing, and enhance postharvest storage management as well. Manual inspection is traditionally used in the apple industry, which is generally done by means of naked eyes and sensory evaluations, while instrumental techniques have become popular for internal quality evaluation. Except for surface defects, manual inspection for appearance attributes such as shape and color is no longer practiced in the modern packinghouses, because it is tedious and labor intensive, and inspection results vary with inspectors over time due to human fatigue. Sensory evaluation is commonly used to evaluate fruits for flavor and textural attributes, in which a panel of expert or consumer judges are trained to assign scores to a list of predefined quality descriptors through tasting and touching tests (Meilgaard et al., 2006). The advantage of sensory evaluation is that it can predict the likely preferences of the ultimate consumers and is suitable for characterizing subtle quality attributes such as mealiness. However, sensory evaluation is prone to   2 variability due to the nature of subjective judgment. Furthermore, the evaluation procedure is slow, and the cost for selecting and training panel members is high. Physical or chemical instruments are commonly used to inspect quality attributes that are quantifiable or measurable. For instance, the Magness-Taylor pressure tester and Brix refractometer have been used as standard methods in the apple industry for determining firmness and soluble solids content (SSC), respectively; gas or liquid chromatography is widely used for measuring chemical compounds in fruits such as sugars, acids, pigments and volatiles. Compared to sensory evaluation, instrumental techniques are able to deliver objective, consistent and reliable results, and also reduce labor costs. However, they generally involve destructive sampling and time-consuming analysis procedures, and are limited to small quantities of samples. Therefore, technologies for rapid, non-destructive, and objective quality inspection of fruits are needed, so as to improve postharvest handling and marketing practices, maintain product quality, reduce postharvest loss, assure consumer satisfaction, and ultimately enhance industry profitability. Over the past several decades, considerable efforts have been devoted to the development of non-destructive sensing technologies for rapid, automated quality inspection of agricultural and food products (Abbott, 1999; Chen and Sun, 1991; Gunasekaran, 1996; Gunasekaran et al., 1985; Nicolai et al., 2014; Ruiz-Altisent et al., 2010). Most non-destructive sensing technologies rely on measuring electromagnetic radiation (involving optical, magnetic or thermal properties) of a product for its spectral and/or spatial features, which are directly or indirectly related with quality attributes of interest for the product, and they are usually fast, objective, non-contact, non-invasive, and thus well suited for on-line real-time quality inspection of horticultural products. The electromagnetic spectrum encompasses, from shortest to longest wavelengths, gamma-ray (less than 0.01 nm), x-ray (0.01-10 nm), ultraviolet (UV, 10-400 nm), visible (Vis, 400-700 nm),   3 infrared (IR, 700 nm-1 mm), microwave (1 mm-1 m) and radiowave (1 m-1 km). In most studies, optical techniques are referred to as those based on the interaction of matter with light from UV to near infrared (NIR, 700-2500 nm) region. Being extensively used for food quality assessment, optical techniques mainly include NIR spectroscopy, time/spatially-resolved spectroscopy, monochromatic and color or RGB (red-green-blue) imaging (they are often known as conventional machine vision), multispectral and hyperspectral imaging, and fluorescence spectroscopy and imaging (Abbott, 1999; Nicolai et al., 2014). X-ray imaging and magnetic resonance imaging (MRI) are based on the interaction of matter with the radiation in the x-ray and radio-frequency ranges, respectively. Based on the detection of IR radiation emitted from matter, thermal imaging has been also explored for food quality assessment. In recent years, Raman (spectroscopic and imaging) techniques and biospeckle imaging have emerged as new non-destructive sensing tools and found applications in food quality and safety inspection. All non-destructive techniques, relying on computer-based image acquisition and processing, can be loosely referred to as computer vision technologies. NIR spectroscopy is perhaps one of the earliest non-destructive sensing techniques, which has been utilized for quality assessment of agricultural products since 1960s. Early studies reported on detecting internal defects of apples based on light reflectance and transmittance measurements (Birth and Olsen, 1964; Francis et al., 1965). Since 1980s, it has found wide applications in the agriculture and food industry (Williams and Norris, 1987), owing to great improvements in both instrument hardware (e.g., fast and inexpensive spectrometers) and software (e.g., advances in chemometrics and computer processing power). However, NIR spectroscopy is point measurement, which is incapable of, or inadequate for, characterizing the spatial variability of a sample, e.g., for detecting skin defects that involve localized or specific areas on the fruit surface. Machine vision   4 developed rapidly during the 1980s and 1990s (Davenel et al., 1988; Lu and Lu, 2017a; Miller and Delwiche, 1991; Rehkugler and Throop, 1986). It mimics human vision to perform inspection tasks, implemented in broadband monochromatic or RGB color imaging mode. Automated inspection systems with machine vision are now widely adopted in the industry for sorting and grading fruits for such appearance characteristics as size, shape and color. However, they still struggle with detecting many types of defects, especially those that are invisible from the fruit surface (and they are also not suitable for detecting internal quality attributes due to the lack of sufficient spectral information). Since the late 1990s, hyperspectral imaging has emerged as a powerful inspection tool in the agricultural and food domain (Lu and Chen, 1999; Martinsen and Schaare, 1998), which integrates spectroscopy and imaging technologies to acquire both spectral and spatial information from samples simultaneously, thus allowing to detect both external and internal quality attributes. The last two decades have witnessed an exponential growth in research on hyperspectral imaging for food quality and safety assessment (Gowen et al., 2007; Lorente et al., 2012; Park and Lu, 2015; Sun, 2010). Due to the need of acquiring and processing large volumes of image data, hyperspectral imaging is primarily used to determine a few key wavelengths as a precursor for multispectral imaging applications (ElMasry et al., 2008; Huang et al., 2015; Kim et al., 2011; Lee et al., 2008; Yang et al., 2011). Most hyperspectral/multispectral imaging systems are based on reflectance measurement. An alternative is fluorescence imaging based on the detection of fluorescence (mostly chlorophyll fluorescence) emitted from horticultural products, which provides as a sensitive tool for inspecting some physiological disorders (DeEll and Toivonen, 2003) and fecal contamination (Kim et al., 2011) in apples. X-ray imaging and MRI, which are commonly used as medical diagnosis equipment, provide superior tools for inspecting interior features of a sample. Research into x-ray imaging and MRI   5 for detecting internal or hidden defects of fruits has continued for many years (Chen et al., 1989; Mathanker et al., 2013; Schatzki et al., 1997; Wang et al., 1988). However, limited by high instrumentation cost and long inspection time, they both still remain at the research stage. For x- ray imaging, in particular, radiation shielding is needed to protect operators against x-ray radiation, and there also exist some safety concerns about the irradiated foods. In spite of these, their uses continue to expand in the agricultural and food industry, because in some cases they are the only viable options for quality inspection. Thermal imaging, also known as thermography, relies on the detection of surface temperature of samples for quality inspection, which has been used to detect defects, such as bruise (Varith et al., 2003) and watercore (Baranowski et al., 2008) of apples, which alter thermal properties of fruit tissues. This technique, however, usually requires a lengthy heating or cooling process and well- controlled ambient temperature, which can be difficult to implement in practical situations. Raman techniques provide an effective tool for inspecting food safety, such as microbial contamination and chemical residuals. Recent research has been reported on the detection of pesticide residual on the surface of apples (Dhakal et al., 2014; Zhai et al., 2016). Biospeckle imaging is an emerging technique for food quality and safety inspection, based on the detection of the dynamics of speckle patterns (images) from agricultural materials (Zdunek et al., 2014). However, its potential and applications have yet to be explored. Over the past two decades, great progress has been made on the development and applications of non-destructive sensing technologies for automated quality inspection of fruit. However, these technologies are still short of meeting the industrial need for inspecting multiple quality attributes of fruit. Among the most successful are NIR spectroscopy and machine vision, which have been adopted in industrial sorting lines but are limited to inspecting fruit for certain quality attributes   6 like SSC, and size, shape and color. The inspection of fruit for defects still remains a challenging task (a review of defect detection of apples by non-destructive sensing technologies is given in Chapter 2). The difficulty is mainly due to the fact that there is a wide range of defects, involving various disorders, mechanical damage, contaminations and so on. The problem is further complicated by the fruit skin that may have great variations in color and texture, and the presence of surface concavities (i.e., stem/calyx regions) that can cause confusion with true defects if not properly identified. In particular, those defects that are invisible on the fruit surface, like bruise and internal defects, are presumably among the most difficult to detect. Therefore, research is still needed to develop a new, effective technique for enhancing defect detection of apples. Illumination or lighting is of significant importance for an automated inspection system. A well-designed illumination system produces high-fidelity/-contrast images (Gunasekaran, 1996), which helps improve the accuracy of image analysis and simplify image processing procedures that otherwise would be needed. Uniform or diffuse illumination is predominantly employed in the imaging inspection systems for food and agricultural products. The light is uniformly distributed throughout the scene or shone onto the object while minimizing unwanted shadows or glare, which can be achieved, as shown in Figure 1(a), by mounting the light source in a hemispherical diffusing chamber or multiple light sources symmetrically around the object (Cubero et al., 2011). An alternative is structured illumination (SI) (See also Section 3.1.), in which, as shown in Figure 1(b), the light shone over an object is modulated according to a specific pattern via a pattern generating device (e.g., projector). There exist a myriad of light pattern options, depending on applications (Heintzmann, 2006; Salvi et al., 2010), among which sinusoidal patterns are commonly used because of their well-understood harmonic properties that simplify the   7 subsequent image demodulation procedure. A spatially-modulated sinusoidal pattern can be mathematically expressed as follows: ( , ) I x y  I ( , ) x y  I AC DC ( , )cos(2 x y ) f x   x (1.1) where (x, y) are the spatial coordinates; DC( , ) x y , AC( , ) x y , I I xf and  represent background or direct component (DC), amplitude component (AC), spatial frequency (along the x-axis) and phase offset, respectively; and the part 2 xf x represents the spatially-varying phase. Figure 1.1. Schematic of uniform illumination (left) and structured illumination (right) system designs. SI with sinusoidal patterns enables both modulated imaging of biological tissues (Cuccia et al., 2009; Cuccia et al., 2005) and three-dimensional (3-D) geometry reconstruction of objects (Zhang, 2010), by extracting intensity (i.e., AC and DC) and phase components from acquired reflectance pattern images, respectively, which is referred to as demodulation. Previous studies in microscopy and diffuse optics have shown that SI with sinusoidal patterns is capable of enhancing image resolution and contrast compared to uniform illumination (UI) (Bassi et al., 2009; Gao et al., 2013; Gustafsson, 2000; Mazhar et al., 2010; Neil et al., 1997), and also enables depth-resolved imaging of heterogeneity of turbid media (Konecky et al., 2009) by varying the spatial frequency of patterns. Effective detection of defects requires acquiring high-quality images for defect segmentation. In view of its major features, SI can be advantageous to enhance defect detection of apples. Compared to UI, SI enables better control of the light penetration within fruit tissues, which would, in particular, enhance the detection of some depth-specific defects of apples, such as bruises.   8 Moreover, the ability of SI to reconstruct the 3-D geometry of fruit would be also of benefit to defect detection, because the reconstructed geometry can help discriminate surface defects from surface concavities. Hence, SI would provide new means and opportunities to improve the existing imaging inspection systems for enhanced defect detection of apples. 1.2. SIGNIFICANCE OF RESEARCH Defect detection of apples remains one of major hurdles to the full automation of fruit sorting and grading. All the existing imaging inspection systems, which rely on UI, have limited or mixed performance in detecting many types of defects of apples (Lu et al., 2016). By utilizing SI in place of UI, this research is aimed to develop a new imaging modality for defect detection of apples. The SI imaging technique has the potential to deliver higher-contrast, higher-definition images than those obtained under UI, thus helping enhance or improve defect detection. Therefore, this research can be expected to benefit the apple industry by providing a new, effective imaging technique for quality inspection. This research represents a first significant endeavor in exploiting SI imaging techniques for food quality assessment. The techniques developed in this research should have the potential to be used for inspecting other agricultural and food products. 1.3. OBJECTIVES AND HYPOTHESIS Defect detection is of great importance for the apple industry, but it is still a challenging task due to the complexity of the problem. SI opens up the possibility of enhancing defect detection because of its potential advantages in image contrast/resolution enhancement, depth-resolved imaging and 3-D geometry reconstruction. This research was built upon the hypothesis that SI can be used in place of UI in broadband or multispectral imaging mode to achieve enhanced performance in defect detection of apples.   9 The overall objective of this research was therefore to develop a new imaging technique based on SI for enhanced defect detection of apples. Specific objectives of the research were to: 1) Develop a SI reflectance imaging (SIRI) system or platform, operated in both broadband and multispectral imaging modes, and examine the fundamental features of SIRI; 2) Propose and evaluate new demodulation approaches to retrieving intensity (i.e., AC and DC) images from the acquired SIRI pattern images in order for fast implementation of SIRI technique; 3) Develop image processing algorithms and procedures for the reconstruction of fruit 3-D geometry based on phase images to assist in the detection of surface concavities of fruit; 4) Develop image processing algorithms for the detection of subsurface bruising and surface defects of apples based on demodulated DC and AC intensity images; and, 5) Explore the potential of acquiring SIRI images from moving samples towards future real- time implementation of the technique for food quality and safety evaluation. 10   2. LITERATURE REVIEW OF NON-DESTRUCTIVE DEFECT DETECTION OF 2.1. DEFECTS APPLES Apples are susceptible to a wide range of defects, which may occur before and at harvest, and during post-harvest operations such as transportation, storage and retail distribution. Most of these defects, according to their causes and nature, fall into four categories: physiological disorders, mechanical damage, pathological disorders and contamination. 2.1.1. Physiological Disorders Due to adverse pre- and post-harvest conditions, apples can be affected by various physiological disorders. Physiological disorders are characterized by abnormal tissue breakdown due to external or internal stresses such as nutritional deficiency, extreme temperature, suppressed respiration and senescence (Watkins, 2003; Wills and Golding, 2016). Hundreds of physiological disorders in apples have been reported. Bitter pit, sunburn, superficial scald, mealiness, watercore and browning are among the most common. However, the physiological causes and symptoms for many of these disorders are still not fully understood. Bitter pit is a disorder associated with calcium (Ca) deficiency in the fruit (Ferguson and Watkins, 1989). The symptoms are small, discrete, bitter tasting spots of the cortical flesh of fruit, becoming desiccated with time. These spots are predominantly located in the outer cortex of the fruit. Skin depressions are usually visual due to the collapse of the outermost cell walls. Bitter pit can also occur deep into the flesh, and become visible only when the fruit is cut open. Bitter pit is usually initiated when the fruit is on the tree, but it progresses rapidly during storage. The cause for bitter pit is mainly due to the low level of Ca in the fruit flesh, which impairs the permeability of cell membranes, resulting in cell injury and necrosis (Meheriuk et al., 1994). Within the fruit,   11 bitter pit is most common in zones with the decreased Ca concentration, although the pitted tissues per se do not show a low Ca content (Jackson, 2003). Bitter pit can affect a wide range of apple varieties regardless of geographic location, and the severity varies with climate condition and orchard management practice (Ferguson and Watkins, 1989; Meheriuk et al., 1994). Sunburn is caused by over-heating of the fruit surface due to excessive solar radiation in the field (Racsko and Schrader, 2012). The symptoms can range from bleached or bronzed patches to dark-brown burned spots on the sun-exposed surface, depending on varietal and environmental conditions. Sunburn downgrades the fruit quality, because of not only the disorder itself but also some sun-burn related disorders, like bitter pit and watercore (Schrader et al., 2003). Three types of sunburn are identified in apples: sunburn necrosis (SN), sunburn browning (SB) and photo- oxidative sunburn (PS) (Felicetti and Schrader, 2008; Schrader et al., 2001). SN is characterized by visible dark brown or black necrotic spots on the fruit surface, due to the lethal damage of epidermal tissue. SN can be triggered when the fruit surface temperature (FST) reaches 52 °C for only 10 min (Schrader et al., 2001). SB, which is characterized by a yellow, brown, bronze or dark tan spot on the apple surface, is the most common and of the most economic importance. SB can be induced by the FST in the range of 46-49 °C in combination with high UV-B radiation, but it is a sub-lethal event relative to SN (Racsko and Schrader, 2012). PS is distinct from SN and SB in that it is neither temperature nor ultraviolet radiation dependent. It happens to the apples that are suddenly exposed to full sunlight, with a bleached and white patch occurring on the previously shaded side of fruit surface. The occurrence of sunburn is variety specific. Sensitive varieties include ‘Granny Smith’, ‘Jonagold’, ‘Fuji’, ‘Golden Delicious’, ‘Braeburn’, ‘Boskoop’, and ‘Delicious’(Racsko and Schrader, 2012).   12 Superficial scald is manifested as irregular brown or shiny-black patches on the fruit skin (Golding and Jobling, 2012; Lurie and Watkins, 2012). The cause of superficial scald is thought to be oxidation processes of α-farnesene, a natural product accumulated in the fruit wax. The disorder occurs normally after several months of cold storage, and develops progressively after the fruit has been removed from cold storage to room temperature, although it may be visible during cold storage. Hence the disorder is seen as a type of chilling injury (Watkins et al., 1995). The incidence of superficial scald is affected by varietal, orchard and storage factors. Sensitive varieties include ‘Granny Smith’, ‘Cortland’, ‘Rome Beauty’, ‘Delicious’ and ‘Winesap’ (Meheriuk et al., 1994). The development of the disorder is favored by the hot, dry weather before harvest, immature fruit, high nitrogen and low Ca concentrations, and inadequate ventilation in storage rooms. Mealiness is caused by deterioration of fruit texture during storage. The disorder, normally free of visual symptoms, is perceived as a dryish mouthfeel during chewing apples. The development of mealiness is related to changes in the strength of cell walls and middle lamellae (Harker and Hallett, 1992). In fresh apples, the middle lamellae are strong enough to provide adhesion between neighboring cells. During storage, cells of over-mature fruits gradually lose cellular adhesion but maintain the strength of cell walls. When a load, like chewing, is applied, cells separate from each other instead of being ruptured (Harker and Sutherland, 1993); individual cells preserve their contents, resulting in a non-juicy sensation. Mealiness is a complex texture attribute, which cannot be described by a single sensory descriptor, but can be related to some measurable mechanical parameters for quantifying mealiness (Barreiro et al., 1998). The occurrence of mealiness varies with fruit variety, harvest date, and storage condition (Arefi et al., 2015). Nara et al. (2001) reported that among eight apple varieties, mealiness was the highest in ‘Starking Delicious’ (83%) and the lowest in ‘Golden Delicious’ (0%) after 20°C for 8 weeks of storage. Mealiness is seen as   13 a senescence-related disorder as it is associated with harvesting over-mature fruit or over-storing fruit (Wills and Golding, 2016). Mature apples are more prone to mealy texture than immature ones during storage. Watercore is a disorder that occurs on the tree, characterized by the liquid accumulation in the intercellular air spaces of the flesh (Itai, 2015). In mild cases, watercore is mainly localized near the core and the primary vascular bundles, manifested as radial or coalesced symptoms (Clark et al., 1998); in severe cases, it can affect the entire core area and the cortex, and even extend out to the fruit surface, resulting in translucent appearance. Watercore is usually seen as a quality defect, with one exception, ‘Fuji’ apples that are selected by Japanese breeders for its high tolerance for watercore. Watercore-inflicted apples are described as having a sweet flavor since the accumulated liquid is rich in sorbitol (Marlow and Loescher, 1984), which is a primary photosynthetic product in the phloem. In the normal tissues, the photosynthate is translocated from the vascular phloem cells throughout the fruit, and are then converted into fructose and glucose. In symptomatic tissues, however, the sorbitol metabolism is inhibited, leading to the accumulation of sorbitol liquids in the intercellular space. The liquid reduces the light-scattering ability of the tissue, causing it to be translucent. Mild watercore often recovers during storage. In some cases, it undergoes anaerobiosis to generate alcoholic flavors, or suffers from flesh browning, probably due to the decreased gas permeability of affected tissues (Argenta et al., 2002). The incidence of watercore in apples depends on a number of factors, among which maturity seems to be the most influential (Harker et al., 1999; Marlow and Loescher, 1984) Browning is a generic term of internal disorders related to storage, characterized by brown patches, possibly with cavities, within fruit tissues. There are at least three types of browning disorders according to specific causes and manifestations (Meheriuk et al., 1994; Watkins, 2003).   14 One is flesh browning, often referred to as internal browning, which happens to the outer portion of the cortex tissue or the most of the flesh in advanced cases; the second is core browning, also called core flush, brown core or brown heart, mainly occurs in the core area; and the third is internal breakdown, which differs from flesh or core browning in that the affected tissues are mealy or soft (referred to as senescent breakdown) or they are moist and markedly separated from normal tissues (referred to as low-temperature breakdown). Flesh browning and core browning are often related to the elevated CO2 and decreased O2 concentrations during controlled atmosphere (CA) storage, while for internal breakdown, the senescent breakdown and the low-temperature breakdown are related to the aging of tissues, which happens to over-mature or over-stored fruits, and low temperatures during cold storage, respectively. Browning disorders have been reported in many varieties such as ‘Fuji’, ‘Braeburn’, ‘Elstar’, ‘Cripps Pink’ and ‘Empire’. During CA storage, the high concentration of CO2 may trigger the production of stress-related radicals, such as reactive oxygen species, leading to cellular membrane degradation followed by a series of browning reactions. Volz et al. (1998) induced flesh browning in ‘Fuji’ apples by exposing fruit to 20 kPa CO2 at 20°C for three days. ‘Braeburn’ is particularly sensitive to internal browning, referred to as ‘Braeburn’ browning disorder (Elgar et al., 1998), which can be aggravated by increasing CO2 and decreasing O2 concentrations. ‘Cripps Pink’ is reported to be susceptible to flesh browning with different manifestations (James and Jobling, 2009), such as radial browning characterized by browning of the vascular tissue with the cortex tissue remaining unaffected, and diffuse browning with the complete opposite pattern. 2.1.2. Mechanical Damage Mechanical damage of agricultural products is referred to as tissue failure or damage caused by external forces under static or dynamic conditions (Mohsenin, 1986). Apples are susceptible to   15 mechanical damage while still on the plant. During growth, apples may be damaged due to the contact with other fruits or branches of the plant, the invasion of wild animals or adverse weather conditions such as wind and hail. Apart from pre-harvest damage, which is sporadic and not easily controlled, the majority of mechanic damage inflicted to apples results from human handling during numerous harvest and postharvest operations (Martinez-Romero et al., 2004). Apples are subjected to a series of operations from the orchard to packinghouse and ultimately to the consumer, which include harvest, transport, washing, waxing, sorting, grading, packing, cooling, storing, distribution, and so on. Mechanical damage can be induced during any of these operations, manifested in the form of bruises, punctures, cuts, abrasions, cracks, splits, etc. Bruising is the most common mechanical damage (Knee and Miller, 2002; Opara and Pathare, 2014; Van Zeebroeck et al., 2007b), which causes a direct cosmetic downgrade and also tissue rotting due to bacterial and fungal growth, thus resulting in economic loss. It is reported that the incidence of bruising in apples can be up to 50%, with a typical range of 10-25% (Studman et al., 1997) and that the apple industry in the U.S. loses over one hundred of million dollars annually due to bruising (Gonzalez, 2009). Given the great economic implications, prevention and/or detection of apple bruising has been an active research area over several decades. Some important findings or observations related to apple bruising are described in the following. Bruising is the damage due to external forces that cause physical change in texture and eventual chemical alteration without breaking the skin of fresh produce (Mohsenin, 1986). A bruise normally develops beneath the fruit skin, and the damaged tissues become darkened and brownish within a few hours due to the enzymatic or chemical oxidation of phenolic compounds, which may be visible or invisible from the surface, depending on the severity of damage. Accompanied with the discoloration of bruised tissues is water loss due to the evaporation or the absorption into   16 surrounding healthy tissues, thus causing the bruised tissue to desiccate eventually. The bruising damage increases ethylene production and respiration (Knee and Miller, 2002), which, together with microbial infection, accelerate produce deterioration. There are three types of forces that are responsible for bruising, i.e., impact, compression and vibration (Knee and Miller, 2002). Impact is the most common source of bruising to apples (Pang et al., 1992; Van Zeebroeck et al., 2007b), which occurs when apples impact hard surfaces or other apples. The impact damage is directly related to impact or absorbed energy (Brusewitz and Bartsch, 1989; Pang et al., 1992), which can be measured in terms of bruise volume. Bruising susceptibility is defined as the ratio of bruising volume to impact energy (or absorbed energy). It should be noted that bruises are visually inspected based on their size rather than volume in practice. There are a myriad of factors that affect bruising susceptibility of apples, including fruit size, curvature, turgidity, firmness, density, cultivar, orchard cultural practice, harvest, orchard transport and packing facilities, etc. (Baritelle and Hyde, 2001; Brown et al., 1989; Garcia et al., 1995; Timm et al., 1998; Van Zeebroeck et al., 2007a). Compression bruising is due to static or quasi-static forces when applied over a long time of period, which typically occurs during transport, packing lines or storage where the package or container is not strong enough to carry stacking loads. But it is normally less severe than impact bruising. Movement of fruit in the package during transport can cause vibration bruising to apples, but to which apples are less susceptible than impact and compression bruising (Wills and Golding, 2016). 2.1.3. Pathological Disorders and Contamination Pathological disorders and contamination usually represent more severe defects than those described above, against which fruit inspection is performed using a higher and stricter standard. Pathological disorders are referred to as the decay or spoilage of fruit tissues due to the infection   17 of microorganisms such as bacteria, fungi and virus (Kader, 2002; Wills and Golding, 2016), while contamination refers to that fruits are contaminated by harmful or toxic foreign materials, such as animal feces, soils, chemical residuals and microbial pathogens as well. For fruits like apples, due to the acidic pH of fruit tissues, fungi are the main culprit of pathological disorders (Wills and Golding, 2016). Apples are susceptible to a broad spectrum of pathological disorders caused by fungi, among which, gray mold, blue mold, mucor rot and core rot are the most common (Jijakli and Lepoivre, 2004; Kader, 2002; Kupferman and Waelti, 1993). Blue mold (Penicillum expansum), which is probably the most common and destructive pathological disorder found on apples in transit, in storage and on the market, is normally initiated at wound sites, such as cuts and stem punctures, and also develops through lenticels on unbroken skins at bruising sites (Kupferman and Waelti, 1993; Pierson et al., 1971). Typical symptoms are soft, watery spots that are sharply separated from healthy flesh. Gray mold (Botrytis cinerea) is primarily a storage disease. As in blue mold, the pathogens infect fruit through wound sites such as punctures and bruises, resulting in lesions characterized by firm, pale, translucent areas without sharp margins (Pierson et al., 1971). In advanced cases, the affected apples may appear baked and turn softer than those in the early stage. Mucor rot, caused primarily by Mucor piriformis that is soilborne fungus, is less frequently encountered than blue mode and gray mold, but can be highly destructive (Jijakli and Lepoivre, 2004), for varieties like ‘Granny Smith’ and ‘Fuji’. This disorder produces a very soft rot that can liquify fruit tissue, resulting in dripping spores onto healthy fruit in the bin and thereby spreading the disease in storage. Core rot can be due to one or more of a series of fungi, such as Penicillum expansum, Mucor piriformis and Alternaria alternata, which usually enter the fruit through an opening in the calyx and attack the core region and the adjacent flesh (Shtienberg, 2012). Hence, apples with a large calyx opening, such as ‘Delicious’ and   18 ‘Golden Delicious’, are particularly susceptible to core rot. Unlike other pathological disorders, core rot usually goes unnoticed until the fruit is cut open, unless very severe. Another closely related disorder is moldy core, characterized by dry, spongy brown lesions in the core area, which, distinct from core rot, does not progress into the fruit flesh. Depending on the infecting fungi, moldy core in some cases can turn into core rot where the infection extends to the fruit flesh. Fruit can be contaminated by pathogenic microorganisms or toxic chemical residuals. A wide variety of microorganisms including bacteria, parasites and viruses, can contaminate fresh produce and have been linked to the outbreak of foodborne illness (Kader, 2002). These microbial pathogens can originate from soils, water, decayed plant, fecal matter from insects, birds, cows, or other wild animals, and or human handling. Animal feces, which have been shown to transfer foodborne pathogens, like E. coli O157:H, are one of the major sources of microbial contamination in apples. Moreover, the survival of foodborne pathogens in the fruit can be greatly enhanced due to mechanical damage and tissue decay, which provide an ecological niche for the growth and multiplication of pathogens. Chemical residues contaminate fruit during growth, at harvest and during post-harvest handling, including insecticides, fungicides, heavy metal, machine lubricants from forklifts or packing lines, industrial toxins, etc. Evaluation of the risk of these chemical residuals should comply with specific standards against allowable levels of contaminants. 2.2. NON-DESTRUCTIVE TECHNIQUES AND APPLICATIONS In the past several decades, a number of non-destructive sensing techniques have been developed to facilitate quality and safety assessment of agricultural and food products, which mainly include NIR spectroscopy, time-resolved and spatially-resolved spectroscopic techniques, fluorescence techniques, monochromatic and color imaging, hyperspectral and multispectral imaging, x-ray imaging (including computed tomography), MRI and thermal imaging (Chen and Sun, 1991;   19 Gunasekaran et al., 1985; Nicolai et al., 2014). Raman techniques and biospeckle imaging have emerged recently for food quality and safety evaluation. Presented below is a review of these techniques and their applications for defect detection of apples, with emphasis on those research reported in the past 20 years. 2.2.1. NIR Spectroscopy In NIR spectroscopy, the product is irradiated with NIR radiation (700-2,500 nm), and the acquired spectrum is related to the chemical composition and physical properties of the product (Nicolai et al., 2007). NIR spectroscopy is mainly implemented in reflectance and transmission modes, which measure the reflected and transmitted radiation, respectively. Transmission measurement has been used for the detection of internal defects in apples, while reflectance measurement for surface or subsurface defects. Upchurch et al. (1997) examined light transmittance characteristics of ‘Delicious’ apples in the spectral region of 650-950 nm, observing that the light absorption at wavelengths of 720-750 nm decreased with the severity of internal browning. The direct use of the ratio of transmittance at 720 and 810 nm was able to differentiate defective apples from sound ones with detection errors of 6.3-12.0%. The transmittance measurement required a long integration time of 5 s, and apples were oriented with light entering the calyx end and the detection fiber probe placed in contact with the cheek midway between the stem and calyx. Clark et al. (2003) investigated the effects of source- detector arrangement and fruit orientation on browning detection in ‘Braeburn’ apples. The best accuracy was achieved by partial least squares (PLS) modeling when the stem-calyx end was horizontal and the light source and detector were located at right-angles to one another at the fruit equator, with a root-mean-square error of prediction (RMSEP) of 7.9% for the proportion of affected area. The brown tissues were not evenly distributed in the fruit, which partly contributed   20 to the detection error. Thus, the authors suggested multiple measurements from different positions. McGlone et al. (2005) expanded the study by developing two on-line NIR transmission prototypes with a spectral range of 650-950 nm, which were based on a time-delayed integration spectrometer and a large aperture spectrometer, respectively. Both the two systems operated at a speed of about five apples per second, and the latter gave superior accuracy with a RMSEP of 4.1%. However, the two prototypes needed be further improved and tested to address the issue due to the asymmetrical distribution of browning tissue. Recently, Khatiwada et al. (2016) compared three NIR spectroscopy instruments with different sensing configurations (interactance, partial transmission and full transmission) for detecting internal flesh browning in apples. It was found that the instrument with partial or full transmittance measurement outperformed that with interactance measurement, resulting in overall classification accuracies of more than 95%. Shenderey et al. (2010) demonstrated the capability of NIR transmission spectroscopy for detecting moldy core in apples. The authors developed a rotating table based system equipped with an off-the-shelf miniature spectrometer, to simulate inline detection. PLS combined with canonical discriminant analysis was used to classify apples into four classes according to the decay level, achieving accuracies of 86-100%. The system required scanning apples with different orientations, which were manually loaded and thus not yet suitable for practical applications. Kafle et al (2016) and Jarolmasjed et al. (2017) used NIR reflectance spectroscopy for detecting bitter pit in apples, by unitizing NIR spectral features combined with chemometric techniques to identify apples as healthy and bitter pit affected. However, NIR reflectance spectroscopy showed limitations in detecting bitter pit at an early stage when noticeable symptoms were absent, and it also required data sampling from multiple positions.   21 In recent years, NIR transmission spectroscopy systems for detecting internal defects in apples, such as watercore and internal browning, are commercially available. Examples of commercial products include “Inscan” provided by Aweta (http://www.inscan-iqa.com/), “iFA” by Greefa (http://www.greefa.nl/UK/), “IDD” and “InSight” by MAF Industries (https://mafindustries.com/), and “Inspectra” by Compact (http://www.compacsort.com/en/inspectra/). However, no studies have been reported of the efficacy of these commercial equipment. 2.2.2. Time-resolved and spatially resolved spectroscopic techniques The optical properties of biological tissues are mainly characterized by the absorption ( a) and reduced scattering ( s) coefficients. It is hypothesized that a is related to chemical constituents, such as water and pigments in tissues, while s is due to cellular structure, which is related to the composition and structural properties of tissues. The presence or development of defects like bruising in fruit normally leads to alteration in both a and s of tissues. Hence, knowledge of these two optical properties can provide a means for defect detection. Time-resolved reflectance spectroscopy (TRS) and spatially-resolved reflectance spectroscopy (SRS), both of which are capable of measuring tissue optical properties, have been explored for ascertaining defects in, or evaluating physiological condition of, apples. In TRS, a short monochromatic pulse is injected into a turbid media like fruit tissue, and the temporal distribution of the remitted light, which is attenuated, broadened and delayed due to light absorption and scattering, is measured and fitted with a proper model of photon migration to estimate optical properties of the media (Torricelli et al., 2008). Valero et al. (2005) reported on the use of TRS to measure the optical properties of ‘Golden Delicious’ and ‘Cox’ apples, which were then used as explanatory variables for mealiness detection. The discriminant model for mealy and non-mealy apples was built using 15 variables, mostly a and some s at 670, 818, 900, 930,   22 940, 960 and 980 nm, which yielded an overall classification accuracy of 80%. In a later study, Rizzolo et al. (2010) used s at 630, 670, 750 and 780 nm and a at 630 nm measured by TRS to classify ‘Jonagored’ apples into three classes based on mealiness severity level, resulting in an overall accuracy of 72%. Vanoli et al. (2010) used TRS to examine the changes of optical properties due to physiological disorders. The authors evaluated internal browning in ‘Granny Smith’, mealiness in ‘Braeburn’ and watercore in ‘Fuji’ apples, and found significant differences in the optical properties at 750, 790 and 912 nm between disordered apples and healthy ones. These physiological disorders could be detected by setting proper thresholds for optical properties. In later studies on detecting internal browning in ‘Braeburn’ apples, Vanoli et al. (2011, 2014) found that a in the range of 670-940 nm increased with internal browning severity while the contrary pattern was observed for s at 780 nm. The a and s at 780 nm were found to be the most relevant to internal browning, based on which 90% healthy fruit and 71% affected ones could be recognized. However, the classification deteriorated if the internal browning position was not predetermined, because TRS only covered a small portion of fruit tissue. The major advantage of TRS is that it is capable of interrogating tissues with deep penetration of about 1-2 cm (Rizzolo and Vanoli, 2016), which enables the detection of internal defects of fruit. However, the technique requires expensive and sophisticated instrumentation, and also close sample-detector contact, which may cause safety concerns and is also problematic in practical implementation for fruit quality inspection. Additionally, TRS requires long integration time and multiple-point measurements, and has limited wavelength options. All these shortcomings should be addressed before the technique comes into practical use. SRS offers another approach to optical property measurement, which collects remitted photons at multiple source-detector distances by using an array of fiber probes connected with a   23 spectrometer or a camera as detectors (Cen et al., 2016). Compared to TRS, SRS is cost-effective in instrumentation and efficient in implementation, and it supports a broad spectral range, which is thus more suitable for agricultural and food products. Researchers with the U.S. Department of Agriculture (USDA) Agricultural Research Service (ARS) at East Lansing, Michigan developed a hyperspectral imaging-based SRS technique for fast and effective measurement of optical properties of horticultural products (Cen and Lu, 2009, 2010; Qin and Lu, 2007). Lu et al. (2010) used this technique to examine the effect of bruising on optical properties of apples. The optical properties of normal and bruised apples of ‘Golden Delicious’ and ‘Delicious’ were measured for the spectral range of 500-1,000 nm. While no consistent trend of changes in the a spectra was observed between bruised and normal apples, the values of s for normal apples were considerably larger than those for bruised apples, and they decreased consistently over time for a period of up to 23 days after the apples had been bruised. This study showed that bruises in apples can be detected based on measuring the scattering characteristics of tissues. More research is still needed to explore SRS for ascertaining other defects of apples. It should be noted that SRS, whose measurement is usually confined to the superficial region of tissues, may not be adequate for detecting defects deep inside the tissues. While both TRS and SRS enables the detection of defects of apples, they are not viable for practical applications due to slow speed in data acquisition and processing. Moreover, it is still a challenging task to accurately measure optical properties of tissues. More research is needed to improve the efficiency and accuracy for optical property measurement. 2.2.3. Fluorescence Techniques Fluorescence techniques are based on the measurement of the fluorescence emitted from the object after it is excited by a beam of light, usually in the UV region of 340-360 nm. The UV illumination   24 produces two types of fluorescence in plant tissues, i.e., blue-green fluorescence in about 400-600 nm and chlorophyll fluorescence in about 650-850 nm, which mainly originates from cinnamic acid and chlorophylls, respectively. During the last two decades, fluorescence techniques have advanced rapidly as non-invasive tools for detecting plant stresses as well as food quality and safety (Gorbe et al., 2012; Karoui and Blecker, 2011; Zhang et al., 2012). Chlorophylls are the major fluorophores in plant tissues, whose photosynthetic activities are sensitive to stress factors such as low temperature and excessive heat. Hence, chlorophyll fluorescence is a potential tool for detecting superficial scald and sunburn in apples. DeEll et al (1996) explored chlorophyll fluorescence for evaluating the development of superficial scald. The variable fluorescence (Fv) measured at harvest, was found to be positively correlated with the scald severity in ‘Sturdeespur’, a strain of ‘Delicious’ apples during cold storage. The apples with high Fv (>18) were most likely to develop severe superficial scald, while those with low Fv (<10) were least likely to develop superficial scald. Mir et al. (1998) observed that the maximal fluorescence (Fm) and quantum yield (Fv/Fm) declined steadily in 'Cortland' and 'Redchief Delicious' apples during cold storage before they presented superficial scald symptoms. Such decline patterns were not observed in ‘Empire’ apples that are scald-resistant. The development of superficial scald in apples was associated with chlorophyll degradation. In detecting sunburn in ‘Braeburn’ apples, Wunsche et al. (2001) reported a sharp decline in the momentary value of fluorescence yield (Ft) with visual sunburn. The authors identified a possibility of using the chlorophyll fluorescence to sort apples in packinghouses, although no such evidence was provided. Seo et al. (2008) reported that Fv/Fm in ‘Fuji’ apples decreased significantly with the development of sunburn due to the increase in FST. The presence of sunburn is normally accompanied by chlorophyll breakdown in the sunburned sides of fruit. Tartachnyk et al. (2012) suggested using fluorescence parameters   25 F690 and F740 for sunburn detection, which decreased consistently in the sunburn affected areas due to the loss of chlorophylls. Although these studies have shown the possibility of fluorescence techniques for detecting superficial scald and sunburn, more statistical evidence is needed for warranting further applications. Mealiness that is associated with fruit ripening and senescence also contributes to the changes in chlorophyll fluorescence. Moshou et al. (2003) reported on using chlorophyll fluorescence to detect mealiness in ‘Jonagold’ and ‘Cox’ apples. Two levels of mealiness were induced by storing fruits for different periods at 20°C and 95% relative humidity. The decline in fluorescence values was observed in both varieties with the mealiness severity. Different discriminant models were built by using quadratic discriminant analysis (QDA), multi-layer perception (MLP) and self- organization map (SOM), among which SOM achieved the best performance with accuracies of 82.5-97.5% and 77.5-95% for ‘Jonagold’ and ‘Cox’, respectively. Later, Moshou et al. (2005) simplified the classification algorithm by fitting fluorescence curves with a tenth-order polynomial. The regression coefficients of the polynomial were used as the input of QDA for classification, resulting in an overall accuracy of 85%. However, both studies did not consider the effect of fruit aging, which cannot be neglected in practice. Until recently, most applications of fluorescence techniques rely on fluorescence spectroscopy for point measurement as with NIR spectroscopy. Fluorescence imaging uses a camera to acquire 2-D information representing the spatial heterogeneity of fluorescence signatures over an object, and it can be implemented in either hyperspectral or multispectral mode (the hyperspectral and multispectral imaging in reflectance mode are discussed in the next section). Fluorescence imaging is generally regarded as a sensitive, effective tool for detecting such defects as fecal contamination, because fecal matter emits fluorescence, like plant materials, when excited with UV radiation.   26 Researchers at the USDA-ARS in Beltsville, Maryland, initiated the studies of hyperspectral fluorescence imaging for detecting fecal contamination of apples (Kim et al., 2003; Kim et al., 2004; Kim et al., 2002). A hyperspectral imaging system with UV-A (315-400 nm) excitation was developed to acquire fluorescence images in the spectral range of 425-752 nm. Single waveband images were found inadequate for reliable detection of fecal contamination, while single two-band ratios, which reduced the variations in normal apple surfaces while accentuating image contrast, provided a simple yet effective means for fecal contamination. A laser-induced multispectral fluorescence imaging system with four wavebands of 450, 550, 678 and 730 nm was developed (Kim et al., 2003) and tested on the automated detection of fecal contamination of apples (Lefcourt et al., 2003, 2005). Based on the band ratio approach, the system was able to detect the 1:2 and 1:20 dilutions of animal feces applied on the surfaces of apples, with accuracies of nearly 100%, and the 1:200 dilutions with accuracies greater than 80% (Lefcourt, 2003). However, the detection deteriorated when the feces were applied to the apple calyx (Lefcourt, 2005). Apart from band ratio approaches, other image processing methods, such as universal power transformation (Lefcourt and Kim, 2006) and differences of two band images (Lefcourt et al., 2006), were also explored to enhance the detection of fecal contamination towards fast implementation. In addition, hyperspectral fluorescence measurement based on light-emitting-diode (LED) excitation was also investigated for fecal contamination of apples (Yang et al., 2011). Hyperspectral fluorescence imaging has also been utilized for detecting other defects of apples. Lotze et al. (2006) examined the ability of fluorescence imaging to detect bitter bit in apples. A transportable prototype was used to acquire low light intensity and high light intensity fluorescence images from ‘Golden Delicious’ apples. The cumulative distribution functions (CDFs) of pixel intensities of fluorescence images allowed for the separation of fruits with and without bit pit. A   27 stepwise analysis was performed to select key variables describing the CDFs, which were used for QDA, resulting in classification accuracies of 75-100%. However, the classification of healthy apples resulted in low classification accuracies of 13.5-50.5%, which were attributed to the environmental and physiological factors that affected fluorescence behavior in fruits. Recently, Chiu et al. (2015) developed a LED-induced fluorescence imaging system to acquire chlorophyll fluorescence images for bruise detection. Bruises were detected by applying adaptive thresholding, followed by morphological processing, resulting in recognition rates of 86.7% and 100% at 30 min and 1 h after bruising was incurred. However, the detection relied on accumulating 25 images per sample to account for the variation of fluorescence intensities over time. Fluorescence techniques, especially fluorescence imaging, are sensitive, effective tools for detecting defects such as fecal contamination of apples. They also hold promise for detecting stress-related physiological disorders, but substantial work is still needed to further elucidate the relationship of fluorescence responses with the development of physiological disorders in apples, and advance possible applications, before fluorescence techniques are of any practical use. 2.2.4. Machine Vision Machine vision discussed here is limited to visible light-based broadband monochromatic and color or RGB imaging for quality inspection of agricultural products. Machine vision relies on image processing algorithms to detect quality characteristics such as shape, size, color and surface defects that can be visually recognized. While machine vision is generally not used for detection of internal quality such as chemical components of products, it has great potential to become a practical tool for quality inspection in agricultural and food industry, because of its simplicity, low cost and rapid inspection rate (Chen et al., 2002; Li et al., 2015).   28 Early studies reported on using monochromatic imaging with a band-pass or long-pass filter for defect detection of apples. Defects were detected by applying simple thresholding to acquired gray-scale images (Davenel et al., 1988; Rehkugler and Throop, 1986) and computing shape features of segmented components (Rehkugler and Throop, 1989; Throop et al., 1995). Simple thresholding, however, usually cannot yield satisfactory segmentation, since the curvature of fruit causes intensity distortion of images whose gray level decreases from the center towards their edges. Surface defects on apples can vary in size, shape and texture, which also make simple thresholding problematic in defect segmentation. Some researchers performed spherical transform of images to correct intensity distortion before applying simple thresholding (Tao, 1996; Tao and Wen, 1999). Besides image correction, multi-thresholding and adaptive thresholding techniques can be used to improve defect segmentation (Bennedsen and Peterson, 2005; ElMasry et al., 2008), but at the cost of intensive computation. An alternative to thresholding is region-based segmentation, which finds coherent, homogeneous regions based on a similarity criterion. Yang (1994) proposed a flooding algorithm that is region-based to detect patch-like defects (e.g., bruises) on apples by treating apple images as geographic landscapes and the patch-like defects as catchment basins in the landscapes. This algorithm required no priori information about the size and shape of defects, which could thus be applied to different types of defects on fruit surfaces. Further improvement was made by using active contour models to closely surround the defects (Yang and Marchant, 1996). However, such methods are more computationally expensive than thresholding techniques. Since color attributes can provide useful information for defect detection, later studies focused on using color imaging for defect detection of apples. Heinemann et al. (1995) reported on the detection of russeting (surface defects appearing in a net-like pattern) on ‘Golden Delicious’ apples.   29 The hue feature of RGB images, combined with linear discriminant analysis (LDA), distinguished between russetted and good apples with an accuracy of 82.5%. Leemans et al. (1998) detected defects on ‘Golden Delicious’ apples by computing the Mahalanobis distance between a color image pixel and the Gaussian color model of healthy apples and then applying a threshold to the distance. This method achieved good segmentation for well-contrasted surface defects, but was not satisfactory in segmenting slight russeting and bruises and also revealed a lot of small spots. In another study on ‘Jonagold’ apples, Leemans et al. (1999) used a Bayesian classification method for defect detection on a pixel level, which could successfully segment many surface defects but had difficult segmenting russeting. Later on, Leemans et al. (2002, 2004) (Leemans and Destain, 2004; Leemans et al., 2002) used color imaging for online grading of ‘Golden Delicious’ and ‘Jonagold’ apples for defects, resulting in classification rates of 72%-78%. It was found that bruises, russeting and stem/calyx areas contributed largely to the errors. In a study by Zou et al. (2010), a color imaging system with three color cameras was used for online grading of ‘Fuji’ apples. The flooding algorithm and the active contour model proposed by Yang and Marchant (1996) were used for defect segmentation, achieving an overall error of 11%, with a false positive error of 15.1%, rejecting a high portion of good apples. Instrument cost, inspection speed and accuracy are three major concerns in adopting an automated inspection system for agricultural produce. Compared to most other non-destructive sensing techniques, machine vision has more potential to be implemented for real-time fruit sorting and grading due to its low cost and rapid inspection speed. Currently, automated machine vision systems are widely used for visual sizing and color grading in packinghouses. However, there are very few such systems for inspecting fruits for defects, because they still cannot achieve desired accuracy, especially for detecting defects that present weak contrasts or are externally invisible.   30 One way to enhance defect detection of apples is to introduce the spectral information to machine vision, especially in the NIR region of 700-900 nm (a spectral window with little absorption of water and pigments) in which light penetrates deep in biological tissues, thus leading to spectral imaging, which is discussed in next subsection. 2.2.5. Hyperspectral and Multispectral Imaging Hyperspectral imaging combines imaging and spectroscopy to obtain 3-D image data, which provides abundant information relevant to chemical constituents and external characteristics of samples. Research in hyperspectral imaging for food quality and safety inspection has expanded tremendously over the last 20 years (Lu et al., 2017; Park and Lu, 2015). With large volumes of image data, however, hyperspectral imaging is being faced with significant challenges in practical real-time implementation. Much research is done on using hyperspectral imaging to determine effective wavelengths for multispectral imaging implementation. Hyperspectral and multispectral imaging can be collectively called as spectral imaging, both of which have been extensively explored for detecting defects of apples. Bruise is one of major defects in apples, which has been intensively studied. Lu (2003) initiated the study on using hyperspectral imaging for detecting bruises in apples. Hyperspectral images in the spectral region of 900-1,700 nm were acquired from ‘Delicious’ and ‘Golden Delicious’ apples with bruises of different ages. The spectral region of 1,000-1,340 nm was found to be the most relevant to bruise detection, resulting in 62% to 88% detection rates for ‘Delicious’ and 59% to 94% for ‘Golden Delicious’. Xing and De Baerdemaeker (2005) used a hyperspectral imaging system in the wavelength range of 400-1,000 nm to detect bruises in ‘Jonagold’. Bruise detection was performed on the first principal component analysis (PCA) score images by detecting the indented part based on surface contour analysis, resulting in a detection accuracy of 78%. However,   31 the bruise detection algorithm can be problematic for detecting slight bruising that only introduces minor changes in fruit shape. In another study on detecting bruises in ‘Golden Delicious’, Xing et al. (2005) identified four key wavebands of 558, 678, 728 and 892 nm. A detection accuracy of 86% was achieved by applying moments thresholding to PCA score images of the four wavebands. Compared to hyperspectral imaging, multispectral imaging offers more practical value due to fast image acquisition and processing. Huang et al. (2015) recently developed a multispectral imaging prototype for real-time detection of bruises. Three effective wavebands of 780, 850, and 960 nm were selected by a hyperspectral imaging system and were implemented in the multispectral imaging system, resulting in a detection rate of 75% for bruised ‘Fuji’ apples in an online test with an imaging speed of three apples per second. Detection of fresh bruises is particularly challenging duet to the absence of external symptoms. ElMasry et al. (2008) investigated hyperspectral imaging for detecting bruises (from one hour to three days) on ‘McIntosh’ apples. Three wavebands at 750, 820, and 960 nm were selected by ranking variable importance in projection (VIP) scores of PLS on the mean spectra for the regions of interest (ROIs). Bruised regions were successfully detected by applying adaptive thresholding to the average of spectral images at the three wavebands. Baranowski et al. (2012) investigated hyperspectral imaging for detecting bruises for five varieties of apple one hour after bruising. Hyperspectral images were acquired in the Vis-NIR (400-1,000 nm) and short wavelength infrared (SWIR, 1,000-2,500 nm) regions. Several supervised classifiers were used to detect bruises of varied ages, resulting in detection rates ranging from 86% to 97%, depending on classifiers and bruise severity. It was further found that hyperspectral imaging could achieve more than 90% accuracies when used for distinguishing the bruising days of apples over a period of two weeks (Baranowski et al., 2013; Siedliska et al., 2014). In detecting bruises of varied ages from 0 h to 9   32 days, Ferrari et al. (2015) proposed a new approach to efficiently handle large amounts of hyperspectral data, called hyperspectrogram, which reduced a 3-D hypercube into a 1-D signal, thus allowing to use classic chemomeric techniques for modeling. Classification accuracies of 91% and 98% were achieved by partial least squares discriminant analysis (PLS-DA) for detecting bruises of 0-3 days and 6-9 days after bruising, respectively. The hyperspectrogram approach, however, has a major drawback in that it cannot be used to locate the position of defects in the image due to the loss of spatial information. Recently, Keresztes et al. (2016) explored pixel-based classification for detecting bruises of 2 h after bruising, and achieved a classification accuracy of up to 98% by optimizing diffuse illumination and accounting for effect of glossiness. Multispectral and hyperspectral imaging have been used to detect many other types of defects of apples. Kavdir and Guyer (2002) reported on classifying apples into 2 and 5 classes based on the surface quality conditions of fruit, using a monochrometer controlled light source based multispectral imaging system. The classification accuracies of 89.2% to 100% were obtained by using a multi-layer neural network fed with image pixel grayscale values and textural features. In a later study (Kavdir and Guyer, 2004), the authors compared four types of pattern classifiers with the input of different statistical and textural features, for apple classifications. Ariana et al. (2006) reported on the pixel-based classifications for various defects on apples with a multispectral imaging system integrated with both reflectance and fluorescence measurements. The integrated imaging approach, however, did not consistently achieve improved classifications, compared to using either reflectance or fluorescence imaging alone, depending on fruit variety. Nicolai et al. (2006) reported on the detection of bitter pit in apples by using a hyperspectral imaging system in the spectral range of 900-1,700 nm. The classification model based on PLS-DA successfully differentiated bitter pit lesions even in cases of no visual symptoms. The classification,   33 however, was based on selected pixels and only examined two bitter pit affected apples. ElMasry et al. (2009) reported on detecting chilling injury in ‘Delicious’ by using hyperspectral imaging in the spectral range of 400-1,000 nm. A full-spectrum artificial neural network (ANN) model was built based on average spectra for ROIs, which resulted in 100% classification success. The model was simplified by selecting five effective wavebands at 717, 751, 875, 960 and 980 nm, which led to 98.4% detection accuracy. Huang and Lu (2010) reported on using hyperspectral scattering imaging in the spectral range of 600-1,000 nm for mealiness detection of ‘Delicious’ apples. Mealiness was assessed based on hardness and juiciness, which were measured by destructive compression tests. Mean spectra extracted from hyperspectral scattering images were utilized for PLS-DA to classify apples into ‘mealy’ and ‘non-mealy’, resulting in overall accuracies between 74.6% and 86.7%. The accuracy declined dramatically when more classes were considered. The authors attributed the decline to small variability of mealiness, inadequate mealiness judgement criterion based on hardness and juiciness, and fruit shape effect. In a later study, Huang et al. (2012) used an unsupervised locally linear embedding (LLE) algorithm to extract features from 1-D hyperspectral spatial profiles. The LLE was able to map data from high-dimensional into low- dimensional embedding manifold, while preserving its local similarities. Classification models by PLS-DA yielded improved classification accuracies of 80.4% and 82.5%, respectively. Owing to its powerful detection capability, hyperspectral imaging has received great attention for food quality inspection. More work still remains to be done to explore its ability for detecting defects of apples. The technique, since its advent for agricultural and food research, has long been faced with the issues of lengthy acquisition and processing of a vast volume of image data. Further research efforts should be directed towards addressing the issues by either advancing hardware or improving data handling strategies, before the technique becomes a viable commercial reality.   34 2.2.6. X-ray Imaging X-ray is able to penetrate deep through most agricultural and food products. The transmitted energy is directly associated with density and thickness of the product. X-ray imaging can produce 2-D and 3-D images, by imaging from only one or multiple angles depending on its setups, which in the latter case is specifically referred to as computed tomography (CT) (Haff and Toyofuku, 2008). Due to deep penetration, x-ray imaging is well suited for detection of internal defects in apples, like watercore, bruising and internal browning, which usually involve changes in tissue density due to water accumulation or dissipation or the presence of internal voids. The majority of x-ray imaging systems in the food and agricultural sector employ a line-scan configuration, which is so called x-ray line-scan imaging or radiography, because of its suitability for continuous scanning in one motion direction, which enables on-line food inspection. The line- scanning imaging system employs low-energy (generally between 20 and 200 KeV) x-rays, often known as soft x-rays, for food inspection, which are less harmful to operators than the high-energy x-ray used in medical diagnostics. Schazki et al. (1997) first reported on using x-ray imaging for detecting watercore in apples by means of visual inspection of x-ray images. Shahin et al. (1999) analyzed the features of x-ray images for ‘Delicious’ apples with different levels of watercore severity. With stepwise selection, spatial features and discrete cosine transform (DCT) features were used for a linear Bayesian classifier, resulting in a detection rate of 79%. Further detection based on fuzzy logic and neural network classifiers achieved success rates of 80% and 88%, respectively (Shahin et al., 2001). Kim and Schazki (2000) reported on the on-line detection of watercore in ‘Delicious’ apples by using x-ray imaging. A set of features relevant to watercore extracted from x-ray images were fed into an ANN classifier, which resulted in low detection rates of 48-75% for apples. Moreover, consistent fruit orientation was required for the x-ray imaging.   35 In another study on bruise detection, Shahin et al. (2002) used spatial edge features extracted by Robert’s edge detector, combined with DCT features, which were fed into ANN to classify apples for both old (one month) and new (24 h) bruises. The accuracy on old bruises was 90% and 83% for ‘Delicious’ and ‘Golden Delicious’, respectively, but only 60% on new bruises for ‘Delicious’ (no test on the other variety). In recent years, CT has been used for inspecting internal defects in apples. Compared to x-ray line-scanning imaging, CT produces higher-resolution and contrast images, which is able to locate defects more precisely, and particularly useful for studying the dynamic behaviors of physiological disorder inside the fruit. Herremans et al. (2013) used CT to examine the dynamics of internal browning in ‘Braeburn’ apples under unfavorable CA storage conditions. Microstructural analysis showed that the development of internal browning was associated with cellular breakdown and cavity formation. The morphometric parameters of CT images were utilized by PLS-DA for the discrimination between healthy and disordered tissues, resulting in a correction rate of 97%. Later, Herremans et al. (2014b) quantified the microstructural changes of tissues during the course of internal browning by computing gas diffusivities in apple tissues based on a multiscale model constructed by coupling the microscale and macroscale models. The oxygen diffusivity in the affected tissue decreased with the development of fermentative breakdown, until the brown tissue was dehydrated and cavities were formed. In using CT for watercore detection, Herremans et al. (2014a) observed that watercore-affected tissues were recognized as the water-soaked regions with higher density in CT images around the major vascular bundles, and the spatial pattern of the disorder varied with apple variety. Affected apples were identified using the mean and variance of CT intensity histograms aided with a global threshold, which resulted in 89% accuracy.   36 X-ray imaging has also been used to detect pitter bit in apples. Jarolmasjed et al. (2016) reported on using CT for detecting bitter pit in ‘Honeycrisp’ apples. The development of bitter pit was examined by acquiring CT images from apples between 0 and 63 days after harvest, which showed that this disorder tended to progress during storage and that the majority of bitter pits occurred around the fruit calyx end. Apples were classified as healthy and bitter pit affected by using logistic regression with a threshold of 0.5, resulting in accuracies of 96% and 70% for apples collected from two field sites. Si and Sankaran (2016) furthered this study by developing image processing algorithms to quantify the number of bitter pits in apples. Bitter pits both on the fruit surface and inside the fruit were detected based on morphological processing and thresholding. About 42-66% of bitter pits in the affected apples were found to be present inside the fruit and the majority of the newly developed bitter pits in healthy apples appeared inside the fruit as well, which probably indicated that bitter pits are initiated from inside the fruit. X-ray imaging is a very powerful tool for detecting internal defects in apples. X-ray line- scanning holds potential for inline inspection due to faster speed, while CT has the advantage of examining the dynamics of the physiological disorders during storage, which is, however, still too costly in instrumentation and too lengthy in image acquisition. In addition, due care should be taken to protect against the potential health hazards in using x-ray imaging equipment. 2.2.7. MRI MRI uses radio-frequency waves that interact with magnetic moments of nuclei, predominantly 1H, to produce 2-D nuclear magnetic resonance (NMR) signals due to excitation and relaxation processes. Relaxation is generally a function of T1 (spin-lattice or longitudinal relaxation time) and T2 (spin-spin or transverse relaxation time), which are sample-dependent parameters, apart from proton density (PD), contributing to the contrast of MRI images (Clark, Hockings, Joyce, &   37 Mazucco, 1997). MRI is able to visualize the spatial distribution of proton in the sample, allowing detection of water-related disorders such as watercore, internal browning and mealiness. Watercore is a disorder associated with elevated water content, which can be readily detected by MRI. The watercore affected tissue in MRI images usually appears as areas of high image intensities (Wang et al., 1988). This disorder can dissipate during storage for ‘Fuji’ apples (Clark et al., 1998). Herremans et al. (2014a) used MRI to detect watercore in apples of different varieties. The greyscale histograms of MRI images were used to discriminate affected fruits from healthy ones, resulting in an overall recognition rate of 79%. The low accuracy was partly because some mild symptoms had been cured by the time when visual evaluation was performed. High equipment cost and low speed are major hurdles for the adoption of MRI for practical applications. Another issue lies in the high magnetic field typical of MRI, which requires avoiding metallic objects in the vicinity of the instrument. Low-field (<0.5 T or 21.3 MHz), non-imaging MR sensing that is lower in cost and requires less signal processing, has been explored as a possible solution. Marigheto and Hill (2005) built a low-field of about 0.1 T MRI prototype for detecting watercored apples. Tests were performed on the damaged region of apples with standard spin-echo imaging sequence, and watercore was readily identified based on the image contrast. Cho et al. (2008) employed a non-imaging low frequency (5.4 MHz) proton MR sensor to evaluate the watercore severity. The parameters of the T2 curve that was fitted to a three-term exponential model, were able to distinguish watercored from sound apples. The authors noted that signal attenuation was a concern for the low-field MR sensor due to the movement of apples through the magnet, for which pre-magnetization was suggested. Internal browning is associated with altered water distribution within tissues, which can be detected by MRI. Clark and Burmeister (1999) reported on using MRI to monitor the development   38 of internal browning in ‘Braeburn’ apples. MRI images were collected by a standard multi-slice imaging sequence at 1.5 T. Brown tissues were recognized as discrete areas of high intensity in the MRI images, indicating increased water content, and those regions enlarged and coalesced over the period of storage. Gonzalez et al. (2001) used MRI with a lower field of 0.6 T to detect brown tissues in ‘Fuji’ apples. The light brown tissues had lower signal intensity than normal tissues due to the shorter T2 and lower PD, while dark brown tissues had higher signal intensity due to longer T2. The decrease in PD in brown tissues was related to the presence of cavities, since they were devoid of water and of little effect on signal intensity. Defraeye et al. (2013) used a high-field and high-resolution MRI system to evaluate browning of apples. PD, T2 and self-diffusion coefficient were related to the dynamics of tissue browning in ‘Braeburn’ apples. Initially, high values were observed due to the elevated content of free water resulting from degradation of cellular integrity in the brown tissue; later on, the water diffused away and evaporated, resulting in cavities and thus low values of these (functional) parameters. Attempts have been made for on-line detection of internal browning in apples. Chayaprasert and Stroshine (2005) developed a low-field of 5.55 MHz non-imaging MR sensor with a Carr-Purcell-Meiboom-Gill pulse sequence. At a low conveyor speed of 50 mm/s, the detection errors were 12% and 0% for ‘Rome’ and ‘Delicious’ apples, respectively; but the errors increased to 31% for ‘Rome’ and 28% for ‘Delicious’ when the speed increased to 250 mm/s. Later, Cho et al. (2008) employed the same MR sensor to examine the use of T2 values for detecting internal browning. The T2 curves were fitted to a three-term exponential model, whose parameters were correlated with the severity of internal browning with r2 = 0.7-0.9. However, this study did not involve moving apples. The development of mealiness in apples is also associated with water diffusion in fruit tissue, affording the possibility of detecting mealiness by MRI. Barreiro et al. (1999) first reported a   39 correlation between MRI image features and mealiness in ‘Top Red’ apples. The authors observed that the minimum T2 was significantly lower for mealy than for non-mealy apples, and T2 values for mealy apples had a skewed histogram, in contrast to non-mealy apples. A stepwise discriminant model based on the features resulted in 87.5% classification for mealy apples (Barreiro et al., 2000). However, the MRI measurements were performed at a high field of 4.65 T, and required 20 spin- echoes images each time. Barreiro et al. (2002) further explored the use of non-spatially-resolved MRI at a lower field of 2.32 T. T2 values decreased consistently with the degree of mealiness, thus allowing mealiness detection. However, the field was still high for practical implementation. Marigheto et al. (2008) developed a low-cost MRI system operating at a low filed of 0.1 T, which was tested on detecting mealiness in ‘Delicious’ apples. The correlation between mealiness and the 1-D T2 distribution shown at high fields (Barreiro et al., 1999, 2002), vanished at the low field. The 2-D correlation measurements were performed to obtain T1-T2 correlation spectra. The mean values of T1 and T2 for most peaks were longer, although not statistically significant, in mealy apples than those in sound ones, and particularly, the T1 of the peak related with the cell wall was significantly longer in mealy apples. However, the T1-T2 correlation spectra were still insufficient for mealiness detection, and the correlation measurement was time-consuming. MRI is able to produce high-contrast images of internal structure of fruit tissues for discerning internal defects in apples. Among various parameters, T2 relaxation times are generally more correlated with the presence or development of defects that modify the distribution, mobility and diffusivity of water within fruit, based on which internal defects can be detected. Like CT, MRI provides a valuable research tool for examining the dynamics or mechanisms of physiological disorders, but currently it is still not viable for practical applications due to the equipment cost and inspection speed issues.   40 2.2.8. Thermal Imaging Thermal imaging, also called thermography or infrared thermography, is a non-invasive, non- contact IR sensing technique for measuring the surface temperature of materials, which has been recently utilized in food and agricultural domain (Ishimwe et al., 2014; Vadivambal and Jayas, 2011). In thermal imaging, the thermal radiation emitted by the object is detected by an IR detector, and transformed into an image called a thermogram that represents the temperature distribution of an object’s surface. Thermal imaging generally comes with two configurations, i.e., passive and active (Baranowski et al., 2009). In the former case, the measurement is performed under natural conditions between the object being detected and surrounding air, while the latter involves using an external heating source to heat the object under study. Unlike other imaging techniques, thermal imaging requires no illumination source, and it is also advantageous due to equipment portability and simple operational procedures. Varith et al. (2003) applied thermal imaging to detect bruises on apples. Under a heating treatment with forced convection in ambient air at 26 ºC for about 3 min, the bruised tissues were found 1-2 ºC cooler than sound tissues for ‘Fuji’ and ‘McIntosh’ apples, enabling 100% detection accuracies for the two varieties of apples. The same heating treatment, however, only achieved 67% accuracy for ‘Delicious’. The temperature difference was attributed to the change in tissue thermal diffusivity caused by the loss of water in bruised areas. Baranowski et al. (2009) explored pulsed- phase thermography for detecting bruises in apples. With heating treatments by a thermal pulse, temperature differences of 0.9-2.1ºC between bruised and sound tissues were observed, which allowed the bruised areas to be distinguished but were not sufficient to distinguish deeper (4 mm) and shallower (1 mm) bruises. Further analysis of amplitude and phase images, which were derived from a sequence of 500 thermal images, enabled to differentiate the depth-varied bruises, based on   41 the frequency components of the heat response. In a recent study, Doosti-Irani et al. (2016) used thermal imaging to estimate the depth of bruises in ‘Golden Delicious’ apples. It was found that the impact energy for bruising had a significant effect on the temperature of fruit surface that was measured by thermal imaging. Further, a multiple regression model based on the measured temperature allowed to predict the bruise depth, which, however, remained to be validated through more research. Thermal imaging provides a means for detecting watercore in apples. Watercore-affected apples accumulate liquid in the intercellular air space of flesh tissues. The presence of watercore alters the thermal diffusivity, mass density and heat capacity of the tissues, which can be detected by thermal imaging. Baranowski et al. (2008) used thermal imaging to detect watercore in ‘Gloster’ apples. Apples were stored at 1.5 ºC prior to the experiment, and then placed in the thermostatted measurement site with an ambient temperature of 20 ºC. Eight sequential images were collected every 10 minutes during the heating by air, to record the temperature responses with time. The derivative of the temperature curve for the apple surface was taken as the measure of the heating rate. The heating rates for watercored apples were found considerably lower than those for sound apples, which allowed the detection of affected apples. Thermal imaging in this study, however, required a heating process of about 70 min, and precise control of temperature gradient between the fruit surface and surrounding air, which is not suitable for practical implementation. Thermal imaging shows the utility for detecting such defects as watercore and bruising, which alter the thermal properties of fruit tissues. In thermal imaging, the measured temperature is a key parameter to ascertain the differences between defective and normal fruit. However, this technique requires well-controlled measuring conditions, especially ambient temperature, and most often   42 involve heating or cooling treatments over a rather long period of time. These requirements pose a great challenge to implement the technique in practical situations. 2.2.9. Raman Techniques Raman techniques discussed here are referred to as Raman spectroscopy and Raman chemical imaging (or Raman imaging for brevity), which are based on the Raman scattering (the scattering process involving energy transfer between scattered light and molecules in the media where light travels). Raman scattering is divided into Stokes scattering and anti-Stokes scattering, depending on the transfer of energy from or to the scattered photons, respectively. Usually, Raman scattering is recoded at only the low-energy side to give Stokes scattering (Smith and Den, 2005), as Stokes scattering is much stronger than anti-Stokes scatterings. Readers are referred to Smith and Den (2005) for more understanding of Raman scattering. Raman spectroscopy, like IR spectroscopy, records a spectrum of Raman scattering intensity of the incident light versus the Raman shift (i.e., the frequency shift relative to the excitation source, expressed as wavenumber cm-1), which can be implemented with several configurations including backscattering (traditional), transmission, surface enhancement, spatial offset, etc. (Qin et al., 2016), depending on applications. Compared to an IR spectrum, the Raman spectrum exhibits narrower, sharper peaks characteristic of specific molecules, which can be used to analyze chemical structure and composition of samples. Raman imaging integrates Raman spectroscopy and imaging techniques to provide spectral and spatial information simultaneously, which expands the application range of Raman techniques. Due to high specificity, low sensitivity to water and little sample preparation, Raman techniques have emerged as an attractive, powerful analytical tool in the fields of pharmaceuticals and biomedicine (Matousek and Morris, 2010; Smith and Dent,   43 2005; Stewart et al., 2012), and recently found applications in the field of food quality and safety inspection (Qin et al., 2010; Yang and Ying, 2011; Yaseen et al., 2017). Raman spectroscopy has long been used as a complement to IR spectroscopy for characterizing structure and composition of chemical compounds. In recent years, its potential for inspecting food safety has been demonstrated. There are many studies on the use of Raman scattering for detecting pesticide contamination on the apple surface. Li et al. (2012) used traditional Raman spectroscopy to detected chlorpyrifos on apple surface. Three Raman spectral peaks were identified, among which the most prominent one at 632 cm-1 was used to detect pesticide concentrations, resulting in the detection limit of 48 ppm (parts per million). Liu et al. (2013) used surface-enhanced Raman spectroscopy (SERS) to detect three types of pesticides, carbaryl, phosmet and azinphos-methyl, on the surface of apples. PLS was used to predict pesticide concentrations based measured Raman spectra, resulting in detection limits of 4.51, 6.51 and 6.66 ppm, respectively. Zhai et al. (2016) reported on the detection of carbendazim residue in apples by SERS. Three relevant Raman peaks were identified at 631.8, 1231.3 and 1521.7 cm-1, based on which a detection limit of 0.014 ppm was obtained. Despite greatly improved detection sensitivity, these SERS measurements required relatively complex sample preparation, which often involved destructive sampling. To achieve fast and non-destructive detection, Dhakal et al. (2014) developed a Raman spectroscopy hardware prototype with 785 nm laser excitation source. The Raman peak at 677 cm-1 was utilized for chlorpyrifos (a type of pesticide) detection on the apple surface, which was capable of detecting a minimum pesticide residue of 6.69 ppm within less than 4 s. Despite the potential advantages, applications of Raman imaging to defect detection of apples are still lacking. More research is needed to explore the potential of Raman techniques, especially Raman imaging, for quality and safety inspection of apples.   44 2.2.10. Biospeckle Imaging When a rough object is illuminated with coherent light (e.g., laser light), the scattered light results in intensity distribution consisting of randomly distributed dark and bright spots distributed in a random manner. Such intensity distribution is a speckle pattern, which is called biospeckle pattern when the illuminated object is a biological sample. The biospeckle pattern is not static but dynamic, which evolves and fluctuates with time due to the presence of physical and chemical activities occurring inside the sample. The root sources responsible for the biospeckle activity, include biological processes such as cytoplasmic streaming, organelle movement and cell growth, and the Brownian motions (Braga et al., 2009; Zdunek et al., 2016). The activity of biospeckle patterns is affected by a myriad of factors, such as physiological disorders, bruising and pathogen infections, relating to food quality and safety. Hence, biospeckle imaging, which acquires biospeckle patterns from biological materials, offers a non-destructive method for detecting and monitoring defects and diseases of fruits and vegetables (Zdunek et al., 2014). Compared to other imaging techniques, biospeckle imaging is simple and cost-effective to implement, which requires a laser source coupled with a beam expander, a camera detector with an objective lens and a computer. Biospeckle imaging collects a series of images (i.e., biospeckle movie), from which the biospeckle activity of samples is computed using various mathematical methods (Zdeunek et al., 2016). In an early study on the use of biospeckle imaging for assessing bruising in apples, Pajuel et al. (2003) observed the diminished biospeckle activity after impact bruising to apples and that the bruised tissue could be distinguished from the surrounding normal ones in activity images obtained by weighted generalized differences and other methods. Later, Passoni et al. (2005) proposed a wavelet entropy-based method to characterize the dynamics of biospeckle patterns, which allowed recognizing the bruised region in apples. In studying various   45 mathematical methods for characterizing biospeckle patterns, Kumari and Nirala (2016) analyzed the dynamics of biospeckle activities of bruised apples over a period of 9 days. The biospeckle activity in the bruised region were found consistently lower than that in the normal region of apples, thus enabling the differentiation between bruised and normal regions of apples. Biospeckle imaging has also been used for detecting other defects in apples. Adamiak et al. (2012) explored biospeckle imaging for monitoring the development of the bull’s eye rot (caused by species of Neofabraea) in ‘Pinova’ and ‘Topaz’ apples during storage. The fluctuation of biospeckle activity was observed, which first decreased before visible symptoms appeared, due to starch granule degradation, then increased rapidly due to pathogen infections, and finally declined due to tissue senescence. Arefi et al. (2016) applied biospeckle imaging to detect mealiness in ‘Delicious’ apples. Biospeckle patterns were acquired at the wavelengths of 680 and 780 nm, and statistical features extracted from these patterns were used for classifying apples into three classes, i.e., fresh, semi-mealy and non-mealy, by ANN models, resulting in accuracies of 81.7%, 70.9% and 77.3%, respectively. Biospeckle imaging is an emerging non-destructive sensing technique for food quality and safety inspection. More fundamental research is needed for a better understanding of biospeckle phenomena in relation to food quality and safety. It should be noted that the technique requires collecting hundreds of frames of images to evaluate the biospeckle activity, which may be not suitable for fast measurement, and also that it is sensitive to external vibrations, which poses a challenge for practical implementation. 2.3. SUMMARY Defect is among the most important factors in fruit sorting and grading in packinghouses and pricing on the market. Because of some unfavorable or adverse conditions, apples can develop or   46 be affected by many types of defects before, at harvest or during postharvest handling. Effective detection of these defects is critical for providing wholesome, high-/consistent-quality fruit for consumers, while enhancing profitability for the apple industry. In the past few years, a number of spectroscopic and imaging techniques have been researched on the defect detection of apples. NIR transmission spectroscopy is effective for detecting internal defects, and it holds great potential for on-line applications due to low instrument cost and rapid inspection speed. TRS and SRS offer a means for defect detection by measuring optical properties of fruit tissues, which still need much improvement in measurement accuracy and efficiency. Fluorescence techniques are a sensitive and less expensive tool for detection of physiological disorders and fecal contamination of apples; but more research is required to explore their applications to defect detection. Machine vision provides a low-cost and fast quality inspection tool, but it is not very effective for defect detection. Hyperspectral imaging has been extensively researched for food quality and safety inspection, but it is currently too slow in image acquisition and processing and thus generally used as a precursor to multispectral imaging that has more potential for practical use. X-ray imaging and MRI are effective in ascertaining internal defects in apples, but slow imaging speed and high equipment cost keep them from practical applications. Thermal imaging, Raman techniques and biospeckle imaging have shown good promise for food quality assessment, but their potential and applications have yet to be further investigated. Defect detection is still a challenging task due to the complexity of the problem, which requires more improvements to current non-destructive sensing techniques. In general, compared to spectroscopic techniques that provide no spatial information, imaging techniques are more suitable for detecting defects that are localized or spatially distributed, which thus deserve more attention in further research. Machine vision and multispectral imaging are perhaps the two techniques that   47 hold the most potential for real-time online defect detection since both are simple, cost-effective and fast in inspection speed. More research is needed to enhance their detection capability by acquiring high-fidelity, high-contrast and discriminative images and developing efficient and effective image processing algorithms, but the improvement in imaging hardware is, in practice, more preferable as it would obviate the need for sophisticated algorithms of data interrogation.   48 3. CONSTRUCTION OF A SIRI SYSTEM AND EXPERIMENTAL ANALYSIS OF ITS FUNDAMENTAL FEATURES 3.1. INTRODUCTION TO STRUCTURED ILLUMINATION REFLECTANCE IMAGING Structured illumination (SI), often known as structured lighting, illuminates objects with spatially structured or patterned light, as a counterpart to uniform illumination (UI) that shines uniformly distributed light onto objects. It should be noted that the two terms, i.e., ‘structured illumination’ and ‘structured lighting’, are often used in different contexts in literature. In optical imaging of biological tissues, ‘structured illumination’ is predominantly used, while in optical metrology, ‘structured lighting’ is more commonly used. Herein, we use the term ‘structured illumination’ to conform the practice in optical imaging. The SI pattern can come in different forms, such as pseudorandom array, binary, N-ary (i.e., more than two intensity levels), sinusoidal pattern, etc., depending on applications and pattern generating devices, and among various SI patterns, sinusoidal pattern is presumably the most extensively used in SI imaging systems. In the rest of the dissertation, SI is specifically referred to as that with sinusoidal patterns unless otherwise stated. The use of SI for imaging biological samples has been researched in biomedical fields, across spatial scales ranging from micrometers in microscopy (Gustafsson, 2000; Heintzmann, 2006; Neil et al., 1997) to centimeters in (macroscopic) diffuse optics (Cuccia et al., 2009; Cuccia et al., 2005; Mazhar et al., 2010). Compared to UI, SI has shown a number of advantages or merits, including image contrast and resolution enhancement in microscopic imaging, and optical mapping and tomography of biological tissues, and it also provides a means of shape measurement of objects. In microscopy where image resolution and contrast are of primary concern, conventional wide- field microscopes employing UI, are limited in acquiring high-fidelity optically sectioned images   49 due to blurring by light emitted from out-of-focus regions of samples. Using SI in place of UI enables optical sectioning to extract the in-focus information while rejecting the unwanted out-of- focus light (Neil et al., 1997; Wilson, 2011). Given a spatially-modulated sinusoidal pattern of illumination: ( , ) I x y  I 0 [1  m cos(2 x )] f x   (3.1) where 0I , m, xf and  are the illumination source intensity, modulation depth, spatial frequency and phase offset, respectively, an optically sectioned image can be obtained by illuminating a sample with three phase-shifted sinusoidal patterns at different phase offsets, aided with an image demodulation approach as follows (Neil et al., 1997): I section  2 ( 3 I 1  I 2 2 )  ( I 1  2 ) I 3  ( I 2  2 ) I 3 (3.2) where 1I , 2I , and 3I are the three acquired pattern images, which correspond to the phase offset θ = 0, 2π/3, 4π/3, respectively. Meanwhile, a separate image, corresponding to that obtained by using a conventional wide-field microscope under uniform illumination, can be obtained by simply taking the average of the three images. Resolution and contrast enhancement can be also achieved through super-solution SI imaging. Because of light diffraction, image resolution is governed by min d ) NA (2 where λ and NA are the wavelength of light and the numerical aperture of the microscope objective, respectively, which is known as the Abbe diffraction limit (Abrahamsson et al., 2017). Hence, the highest resolution is 200 nm for visible light. However, SI can result in image resolution well beyond the fundamental limit, with the maximum achievable resolution enhancement by a factor of two (Gustafsson, 2000; Heintzmann and Cremer, 1999). Different from the approach as described in Eq. (3.2), super- resolution SI imaging requires more phase-shifted sinusoidal patterns in different orientations and   50 at much higher spatial frequencies, and also more complicated image processing procedures due to image deconvolution and reconstruction needed (Gustafsson, 2000; Lal et al., 2016). In tissue optics, absorption coefficient ( a ) and reduced scattering coefficient ( s ) represent two basic optical properties of tissues, which relate to the structural features of tissues and thus provide opportunities for diagnostic assessment. At the core of the optical imaging techniques for quantitative tissue characterization (i.e., diffuse optical imaging, DOI), is quantification of the two optical parameters. DOI, which, according to light source types and measurement principles, falls in two general classes including time-independent [also known as spatially-resolved continuous wave (CW), Figure 3.1a] and time-dependent [i.e., time-domain (Figure 3.1b) and (temporal) frequency-domain (Figure 3.1c)] methods (Biogio and Fantini, 2016; Mazhar et al., 2014; O'Sullivan et al., 2012). Figure 3.1. Schematic of three types of optical property measurement techniques: (a) spatially-resolved continuous wave, (b) time-domain and (c) temporal frequency-domain, where solid and dashed lines denote input and output (detected) signals, respectively.   51 The spatially-resolved CW method measures reflectance at multiple distances from a well- defined source location, which can be detected by using either an array of detecting fibers (Dam et al., 2001; Doornbos et al., 1999) or a camera (Kienle et al., 1996; Qin and Lu, 2007), the measured reflectance as a function of distance is used to inversely estimate optical properties based on a diffusion model (Farrell et al., 1992; Kienle et al., 1996). This method is less expensive in instrumentation and fast in measurement, but it has limitations in interrogating deep tissues. With time-domain (or time-resolved) techniques, short pulses of light (in the picosecond scale) are launched into tissues and time-gated or photon-counting detectors are used to measure the temporal dispersion of the pulse (i.e., temporal point-spread function, t-PSF), which is used for model-based optical property estimation (Pattern et al., 1989). The depth information is intrinsically stored in the acquired temporal data; later arriving photons that have longer travelling paths can be used to assess deeper tissues. This method, however, requires expensive and sophisticated instrumentation. Frequency-domain photon migration (FDPM) techniques provide an alternative to measuring the temporal dispersion of light propagation in tissues (Patterson et al., 1991; Tromberg et al., 1997). FDPM launches temporally modulated (MHz-GHz) light into tissues, and the resulting photon density wave is diminished in amplitude and delayed in phase, and the changes in amplitude and phase are detected at multiple (temporal) frequencies, which can be used to fit a photon migration model for optical property estimation (Patterson and Pogue, 1994). The time-domain and frequency-domain methods share an exact Fourier transform equivalency; but compared to the former, the latter is usually simpler and more economic. In addition, both temporal methods can measure optical properties by using a single source-detector pair. All the three methods use a point light source (i.e., the light source is a pencil beam incident onto the sample) to illuminate the sample at a specific location, and hence their measurements are confined to a relatively small area.   52 To obtain 2-D maps of optical properties would require multiple sources and detectors or raster scanning over the tissue surface. Moreover, these methods often employ fibers as detectors that need to be in close contact with the sample. The spatially-modulated SI provides a means to address the aforementioned shortcomings by enabling wide-field, non-contact imaging of tissue optical properties, thus leading to a new DOI technique, referred to as spatial frequency domain imaging (SFDI) or modulated imaging (MI) (Cuccia et al., 2009; Cuccia et al., 2005). For SFDI, a spatially-modulated source term is introduced in solving the CW diffusion equation, which results in the diffuse reflectance that is not only dependent on depends on a and s but also on the spatial frequency xf , as illustrated in Figure 3.2. Therefore, the optical properties can be estimated by fitting an analytical light transport model to the diffuse reflectance at different spatial frequencies (Cuccia et al., 2009). Figure 3.2. Schematic of spatial frequency domain imaging (SFDI) for optical property measurement (modified from Fig. 3 in Cuccia et al., 2009). Spatially-modulated light is impinged on tissue, and it decays exponentially with depth z, and the remitted reflectance that is measured at multiple spatial frequencies is used to inversely estimate For a given set of optical properties, the diffuse reflectance specifies the frequency responses tissue optical properties. of the media due to absorption and scattering, i.e., modulation transfer function (MTF). The MTF curve is inversely related to xf , as shown in Figure 3.2, which describes the low-pass filtering property of tissues in the steady-state diffuse regime. Light with low spatial frequencies attenuates less than at higher spatial frequencies, and thus penetrates more deeply into tissues, which provides   53 the basis for depth-resolved imaging of tissue heterogeneity (Konecky et al., 2009; Lukic et al., 2009). Compared to other DOI techniques, SFDI has shallower light penetration into biological tissues, whose detection sensitivity is usually limited to a depth of several millimeter in the reflectance geometry (O'Sullivan et al., 2012). SFDI is well suited for subsurface imaging of tissue structures. SFDI shows the features of enhanced resolution and contrast (Bassi et al., 2009; Mazhar et al., 2010), depending on the spatial frequency of illumination patterns. Increasing the spatial frequency has the potential to obtain macroscopic images with microscopic-level resolution and contrast (Bodenschatz et al., 2015; Krishnaswamy et al., 2014), but the measurements are to be confined to a very superficial layer of tissues. Hence, there is a trade-off between image resolution/contrast and detection depth. SFDI has been applied for quantifying optical properties of human tissues (Mazhar et al., 2014), and, most of the reported studies were focused on the characterization of composition, structure and function of skin, such as volar forearm (Gioux et al., 2009; Weber et al., 2009), tattooed skin (Ayers et al., 2009), breast (Laughney et al., 2013) and ovary (Nandy et al., 2016). Since a is associated with tissue chemical composition, it can be used for mapping tissue chromophores such as hemoglobin and malanin (Saager et al., 2010), while s is correlated with the size, density and distribution of tissue scattering components and hence tissue structure (Kanick et al., 2014; Mazhar et al., 2014). Another important applications of SFDI is the detection of tissue inhomogeneities or abnormalities (e.g., burn and tumor) to enhance diagnostics and monitor diseased tissues. In examining tissue simulating phantom samples, Bassi et al. (2009) reported that the inhomogeneities in the sample affected the amplitude and phase of modulated light and suggested the possibility of locating inhomogeneities by detecting the phase change in a transmittance   54 imaging geometry. Konecky et al. (2009) first applied SFDI for tomographic imaging of phantoms with buried inclusions. Images of the absorption contrast were reconstructed, which provided improved resolution in localizing the inclusions at different depths, compared to those created by fitting for optical properties. Tomographic imaging under SI has also been done to reconstruct arbitrarily oriented structures of absorbing inclusions (D'Andrea et al., 2010). Although SI has been successfully applied to biomedical optics, there are limited applications for fruit quality evaluation. Early attempts were made to use laser-scanning based SI for detecting surface concavities (i.e., stem/calyx regions) of apples (Crowe and Delwiche, 1996a, b; Penman, 2001; Yang, 1993). Under the assumption that surface defects occurred only on the convex surfaces, surface concavities were identified by analyzing morphological features of fringe stripes (e.g., curvature) generated by SI, while defects on convex surfaces were detected by UI. Anderson et al. (2007) first reported on the use of a SFDI technique for bruise detection of ‘Golden Delicious’ apples. Optical properties of apples were determined through the demodulation of acquired pattern images and then inverse parameter estimation using a light propagation model. It was found that bruised tissues had considerably lower values of s compared to surrounding normal tissues, thus making bruises readily discernable in the s maps; however no obvious difference was observed in the a values between bruised and non-bruised tissues, which suggested that bruising was primarily associated with structural change in fruit tissues. Recently, Hu et al. (2016) used SFDI to measure the optical properties of ‘Redstar’ apples affected by internal browning. The affected fruit tissue was found to have much higher a values than those for the heathy tissue. While effective for defect detection, SFDI is an inverse approach that requires intensive computation for optical property estimations. Moreover, implementation of the technique also requires the   55 acquisition of images at multiple spatial frequencies and a reference-based calibration, which makes it too slow for practical use. 3.2. CONSTRUCTION OF A SIRI PLATFORM Based on the concept of modulated imaging with SI for biological tissues, a new imaging system in reflectance mode, hereinafter called structured-illumination reflectance imaging (SIRI), was built for defect detection of apples in this study. Compared to conventional imaging inspection systems for agricultural and food products, SIRI includes a special component, i.e., digital micro- mirror device (DMD) based digital light projector (DLP), in the illumination path between a light source and the sample being detected, which is used to generate spatially-modulated light patterns. The DMD is a semiconductor-based array of fast, reflective digital light switches, and due to the digital pattern generation nature, it provides flexible, easy and quick control over the illumination pattern pixel by pixel, and thus can generate any light patterns of interest. Other components of the SIRI system are similar to those in a conventional imaging system, including optical fiber, charge-coupled device (CCD) camera, lens, filter, etc. A broadband SIRI system, as shown in Figure 3.3, was constructed, which consists of a desktop computer (Dell Precision T7610, Dell Inc., Round Rock, TX, USA), a broadband DC- regulated 250-W quartz-tungsten-halogen (QTH) light source (Oriel Instruments, Stratford, CT, USA) along with a radiometric power supply controller, a DMD-based DLP (CEL5500-Fiber Light Engine, Digital Light Innovations, Austin, TX, USA) with a resolution of 768×1024 pixels, and a 12-bit high-performance CCD camera (SensiCam QE, Cooke Corporation, Auburn Hills, MI, USA) with a resolution of 1376×1040 pixels, which is coupled with a C-mount zoom lens (Navitar TV Zoom 7000, Tokyo, Japan). In particular, the projector utilizes the TI (Texas Instruments, Dallas, TX, USA) 0.55-inch XGA (extended graphics array) chipset based DMD for   56 spatial light modulation, which enables either broadband or monochromatic, high-brightness and high-speed (with a pattern rate of up to 500 Hz for 8-bit grayscale and 5000 Hz for 1-bit binary patterns) light projection, and is well suited for fast imaging applications. Figure 3.3. Schematic (left) and photograph (right) of a broadband structured-illumination reflectance imaging       (SIRI) system. The camera is slightly angled at 15 degrees relative to the vertical axis to mitigate the image distortion. The camera and projector are attached to adjustable fixtures, which allow for both translational and rotational adjustments. A pair of cross-linear imaging polarizers (Edmund Optics, Barrington, NJ, USA) is placed in front of the projector and the lens of the camera to eliminate specular reflectance. A fiber optic cable channels the output of the QTH lamp into the projector, and sinusoidally-modulated patterns in 8-bit gray-scale bitmap format, which are created in Matlab (The Mathworks, Inc., Natick, MA, USA), are uploaded to the projector’s bundled control software for pattern projection. The software allows users to individually switch light patterns from a timeline controller buffer during image acquisition. Image acquisitions are done using the bundle software of the camera. This system is operated in an enclosed dark chamber. The above SIRI system only enables the acquisition of broadband black/white images. To expand its ability to acquire spectral images, a liquid crystal tunable filter (LCTF) (Varispec, Cambridge Research and Instrumentation, Inc., Woburn, MA, USA) is used to assemble a multispectral SIRI system. The LCTF is electronically tunable for rapid selection of any waveband in the red visible light and NIR range of 650-1,000 nm, and its aperture size and bandwidth are 20   57 mm and 10 nm, respectively. With the available hardware resources, two versions of multispectral SIRI system are assembled, as shown in Figure 3.4. Figure 3.4. Two versions of a multispectral structured-illumination reflectance imaging (SIRI) system. The multispectral SIRI system in Figure 3.4 (left), referred to as the first version, has the same setup for the projector and camera as the broadband system in Figure 3.3 (left), except that the original zoom lens is replaced with a focusing lens (25 mm F/1.4 macro lens, Electrophysics,  Fairfield, NJ, USA), placed between the camera and the LCTF. Since the LCTF itself acts as a polarizer, the polarizer that was originally attached to the front of the camera lens is unnecessary and thus removed. The second version of multispectral system, as shown in Figure 3.4 (right), is equipped with a back-illuminated electron-multiplying (EM) CCD camera (PhotonMAX: 1024B, air-cooled, Princeton Instruments, Trenton, NJ, USA) with a resolution of 1024×1024 pixels. In this system, the EMCCD camera is vertically set up while the projector is slightly angled with respect to the camera. The EMCCD camera has a large sensor format, with a pixel size of 13×13 μm2, which is four times as large as that of the SensiCam CCD camera (Cooke Corporation, Auburn Hills, MI, USA) used in the first system, and it has a higher quantum efficiency in the NIR region and a wider dynamic range than the CCD camera, which is thus more suited for NIR (and low-light) imaging applications. Moreover, the EMCCD camera uses a high-performance thermoelectric cooler that can cool the CCD array down to -55 °C to greatly suppress system noise. A different focusing lens   58 (35 mm, Edmund Optics Inc., Barrington, NJ) is used in the system, which is selected to be compatible with the EMCCD camera and the LCTF to reduce illumination vignetting. A graphical user interface (GUI), as shown in Figure 3.5, was developed in LabVIEW 2016 (National Instruments, Austin, TX, USA) to control and synchronize the camera and LCTF for image acquisition. Figure 3.5. The graphical user interface (GUI) for structured-illumination reflectance imaging (SIRI). 3.3. DEMODULATION In SIRI, the raw reflectance images acquired from samples are 2-D fringe patterns, as shown in Figure 3.6. These pattern images need to be processed to remove their fringes to obtain DC, AC and phase images, which is referred to as demodulation (such concept comes from the fields of communications and signal processing). The DC and AC are intensity images, which provide a measure of light attenuation within tissues due to absorption and scattering and thus can be used for defect detection of apple, while phase images reflect the topography of an object, which can be used for 3-D geometry reconstruction, which will be discussed in Chapter 5.   59 Figure 3.6. Examples of three phase-shifted pattern images of an apple, obtained under the illumination of sinusoidal pattern with phase offsets of -2π/3, 0 and 2π/3 (from left to right), respectively. Presented below is a conventional phase-shifting approach, herein referred to as to three-phase demodulation (TPD), to retrieving DC and AC images from three phase-shifted images, which has been widely adopted in optical-sectioning SI microscopy and optical mapping of biological samples (Neil et al., 1997; Cuccia et al., 2009). 1 ( 3 I 1  I DC  I 2  I 3 ) I AC  2 ( 3 I 1  I 2 2 )  ( I 1  I 3 2 )  ( I 2  I 3 2 ) (3.3) (3.4) Where 1I , 2I and 3I represent three acquired phase-shifted images with a phase difference of 2π/3 among each other [See also Eq. (3.2)]. The requirement of three image acquisitions pose limits the approach for fast, real-time imaging applications. In chapter 4, new approaches are proposed to achieve the demodulation using fewer images. 3.4. PRELIMINARY TESTS Preliminary tests were performed to examine the applicability of SIRI for apple defect detection by imaging a highly-scattering nylon slab with foreign inclusions and apple samples with fresh tissue bruising. The broadband SIRI system as shown in Figure 3.3 was used for the tests. Experiments were first conducted on the scattering nylon slab drilled with four cylindrical holes running parallel with the surface. As shown in Figure 3.7, the four holes of 5 mm, 6.5 mm and 12 mm in diameter were centered at depths of 5 mm, 7 mm and 11 mm, respectively, and the two 5- mm-diameter holes were 10 mm spaced apart from center. All the holes were filled with 1000   60 times diluted India ink (Higgins) as absorbers for optical property contrast with the bulk material, and then sealed with adhesive tape. Figure 3.7. Schematic (left) and photo (right) of a scattering nylon block with ink inclusions. Further, separate experiments were conducted on two ‘Golden Delicious’ and ‘Gala’ apples, which were purchased from a local grocery store and free of visual defects. Bruise was induced by impacting each apple at its equatorial area using a pendulum device equipped with a wooden ball, as shown in Figure 3.8(left). Each apple was cut into halves, each of which was placed on the platform of the pendulum, and the wooden ball was released freely from a predefined position with the impact energy for the apple estimated to be 1.0 J. The impact level caused no visible damage for each apple right after the impact test, as shown in Figure 3.8(right). A detailed description of the impact test device is given in Zhu et al. (2016). The apples were imaged immediately after the impact test. ‘Golden Delicious’ ‘Gala’ Figure 3.8. Photographs of the impact pendulum (left) and bruised apples (right).   In the preliminary tests, two sets of spatial frequencies, 0, 0.02, 0.04, 0.050, 0.06, 0.08, 0.10 and 0.12 cycles/mm, and 0, 0.05, 0.08, 0.10, 0.15, 0.180, 0.20 and 0.25 cycles/mm were used for   61 the nylon slab and the apple samples, respectively. The light pattern at 0 cycles/mm corresponded to UI without light modulation. The distances from the sample to the projector was about 25 cm, which corresponded to the field of view (FOV) of 10×12 cm2 for the projector. The output of the QTH light source was set to 150 W, and the camera was operated in automatic exposure mode to ensure high intensity for the acquired images. For the nylon sample, the orientation of the hidden holes was kept parallel with the pattern stripes. A Spectralon panel (Labsphere, Inc., Sutton, NH, USA) with the reflectance rate of 98% or higher was imaged under UI to correct the non-uniformity of the source illumination, and a dark image was collected by turning off the light source to correct the dark current effect of the camera. The corrected images were calculated using the following equation (such correction is known as flat-field correction): I corr  I  I white I dark I  dark I white (3.5) Where I , I dark and white I are images acquired from the sample, white reference and dark background, respectively; and white I are the mean value of the reference image. Figure 3.9 shows the demodulated images of the nylon slab. The filled ink solution caused strong light absorption, making the hidden holes relatively dark in the images. The DC image, equivalent to that obtained under UI, remained almost the same at each spatial frequency, in which the right hole was easily recognized, however the middle (deepest) hole appeared obscure and the two left (shallowest) holes were not clearly resolved. The AC image at the lowest frequency of 0.02 cycles/mm was similar to the DC image. As the frequency increased from 0.04 to 0.10 cycles/mm, the two left holes became spaced apart from each other, and the right hole remained visible, while the middle hole faded and disappeared. As the spatial frequency increased further,   62 the images revealed more apparent surface texture of the nylon sample. At the highest spatial frequency of 0.12 cycles/mm, all the holes became blurry and indistinguishable, which was because the light penetration was mostly confined to the superficial region above the holes. Figure 3.9. Direct component (DC) and amplitude component (AC) images of the Nylon slab at the spatial frequencies of 0.02, 0.04, 0.05, 0.06, 0.08, 0.10 and 0.12 cycles/mm. These observations suggested that higher frequency resulted in shallower light interrogation but improved resolution of the image details. Similar findings were also reported in the biomedical research using SI in transmittance and fluorescence imaging modes (Bassi et al., 2009; Mazhar et al., 2010). Thus, SIRI was able to detect depth-specific defects within sub-surface regions by choosing a proper spatial frequency; however, there is a trade-off between detection resolution and depth depending on applications, which should be considered in specific applications. Figure 3.10 shows the DC and AC images of ‘Golden Delicious’ and ‘Gala’ apples. The DC images failed to reveal the bruises, which was also true of the AC images at the low frequencies of 0.03 and 0.05 cycles/mm. As the frequency increased, the AC images began to reveal the bruises with strong contrast. Since the DC images were comparable to the results obtained by a broadband machine vision system under UI, the findings from AC images demonstrated that SIRI had enhanced capabilities of detecting fresh bruises, compared to conventional UI-based machine vision. It was also noted that, as the frequency increased to 0.25 cycles/mm, the non-defect surface features of apples, like lenticels, became more resolved, due to the decreased light penetration into   63 the tissues, which, may add difficulty to bruise segmentation. Overall, the frequencies of 0.08-0.20 cycles gave relatively good clarities of bruises for the two apples. Figure 3.10. Direct component (DC) and amplitude component (AC) images of ‘Gala’ (a) and ‘Golden Delicious’ apples at the spatial frequencies of 0.03, 0.05, 0.08, 0.10, 0.15, 0.18, 0.20 and 0.25 cycles/mm. Further effort was made to improve the contrast of AC images by dividing AC by DC [i.e., I AC I DC , referred to as ratio (dimensionless) image]. As shown in Figure 3.11, the ratio images significantly enhanced the visibility of the bruises, particularly at the low frequencies of 0.03 and 0.05 cycles/mm, and the frequencies of 0.08, 0.10 and 0.15 cycles/mm appeared to be the best in terms of the overall image contrast. In addition, the ratio images also had a positive effect on the correction of intensity distortions due to the apple’s surface curvature.   64 Figure 3.11. Direct component (DC) and ratio images of ‘Gala’ (a) and ‘Golden Delicious’ apples at the spatial frequencies of 0.03, 0.05, 0.08, 0.10, 0.15, 0.18, 0.20 and 0.25 cycles/mm. The preliminary tests clearly demonstrated the merits and enhanced capability of SIRI, in comparison to UI based-imaging, for the detection of sub-surface defects in turbid media like apples, and also revealed important features of the techniques regarding image resolution, contrast and detection depth that depended on the frequency of light patterns. To better assess the ability of SIRI for apple defect detection and help optimize imaging hardware, a further theoretical and experimental analysis was conducted to examine these features, which are described next. 3.5. ANALYSIS OF SIRI FEATURES 3.5.1. Theoretical Framework 3.5.1.1. Image Formation The image formation in SIRI comes down to two basic steps: 1) the incident light interacts with the sample through absorption and scattering, which are responsible for the decay and broadening of the light, respectively, and 2) the light remitted from the sample travels through a series of optical devices (e.g., lens, camera) of the imaging system, and eventually form a digital image. By treating SIRI as a linear, space-invariant system, to which the transfer function theory for an optical   65 imaging system can be applied (Goodman, 2005; Saleh, 2011), the image formation can be modeled as follows: ( , ) I x y n  ( , ) p x y S x y H x y H x y n ( , ) ( , ) ( , )   s i (3.6) where ( , )x y are the image coordinates; nI x y and ( , ) np x y are the nth acquired and projected ( , ) pattern images, respectively; ( , S x y represents the sample information, more specifically, the ) absorption contrast; sH x y and ( , ) iH x y represent the point spread function (PSF) due to light ( , ) scattering in the sample and the imaging process, respectively; and  denotes the convolution operation. The convolution operations describe how light scattering and the imaging process negatively affect resolution and contrast of resulting images. Scattering events broaden the light beam, thus blurring acquired images. The imaging process, in essence, acts as a low-pass filter, due to imperfect focusing of the lens, wave diffraction, the discrete sampling process of camera and other possible optical aberrations. All these factors contribute to the resolution and contrast loss of acquired images. For mathematical convenience, we combine the PSFs of the light scattering and imaging process into a single term ( , ) H x y H x y H x y ( , ) ( , )   s i , albeit the two processes are independent of each other. Logically, H x y has the effect of low-pass filtering on the product of the illumination pattern ( , ) and the sample information. In Eq. (3.6), the illumination pattern np x y is generated in a ( , ) sinusoidal form as follows [see also Eq. (3.1)]: ( , ) P x y n  1 1 cos(2 2    ) f x  n  x (3.7) where xf is the spatial frequency along the x-axis; and n is the phase offset. Eq. (3.7) can be rewritten in a complex exponential form as follows:   66 ( , ) P x y n  1 1  2 4 )exp( 2 [exp( i i   f x x ) exp( )exp( 2 i i      )] f x x Substituting Eq. (3.8) into Eq. (3.6) yields: where, ( , ) nI x y  I DC ( , )+ exp( ) i x y  ( , ) I x y SI 1 2  1 2 exp( ) ( , ) i I x y    SI and I I DC ( , ) x y  1 2 [ ( , ) ( , )] S x y H x y  ( , ) I x y SI  1 2 [exp(i2  ( , )] f x S x y H x y x ) ( , )  (3.8) (3.9) (3.10) (3.11) SI( , * x y is the complex conjugate of SI( , ) I ) x y . Further, Eq. (3.9) can be expressed in the Fourier domain as follows:  ( , ) I u v n  1 2  [ ( , ) S u v H uv  ( , )] + exp( ) ( ( , ) i S u f v H u v  , )    x 1 4  1 4  ) ( exp( ( , ) i S u f v H x y   , )   x (3.12) where the tilde denotes the Fourier transformed function and ( , )u v are frequency coordinates. Eq. (3.9) or (3.12) lays the mathematical basis for analyzing the SIRI system. According to the equations, the acquired pattern image is composed of three parts. The first part is (DC), i.e., DCI , corresponding to the conventional image obtained by a UI-based imaging system, with the Fourier spectrum centered at the origin. The other two parts are oscillatory or modulated components, complex conjugates of each other, with the Fourier spectra shifted by xf and xf , and hence they share the same amplitude distribution. The sum of their amplitude distribution is represented as ACI , i.e., the AC image, which is equal to | I SI ( , ) | x y [see Eq. (3.11)].   67 3.5.1.2. Demodulation Demodulation is to recover DCI and ACI from the acquired pattern images. Although Eqs. (3.10) and (3.11) describe the makeup of the two images, they cannot be directly used for the demodulation, since the PSFs of the tissue scattering and the imaging system are unknown. By revisiting Eq. (3.9), a general method to achieve this can be developed by forming and solving a linear system consisting of different phase-shifted pattern images. Since there are three unknowns in Eq. (3.9), we need to acquire three phase-shifted pattern images from a sample, so as to establish a well-posed linear system as follows: where, ( , ) I x y  1  ( , ) I x y  2 ( , ) I x y   3       I  DC  M I  SI I    SI ( , ) x y ( , ) x y ( , ) x y      M  1 2 exp( ) ) 2 exp( i i    1 1 2 exp( ) ) exp( i i    2 2 ) 2 exp( ) exp( i i    3 3           (3.13) (3.14) Then, one can solve for DCI and SII [and further, AC I ( , ) x y |  I ( , ) | x y SI ], by inverting the matrix M and pre-multiplying it to the pattern image matrix. In a special case where the phase offsets of illumination patterns are 0, 2π/3 and 4π/3, it can be readily proved that the expression given in Eq. (3.13) is reduced to the classic TPD approach as given in Eqs. (3.3) and (3.4). 3.5.1.3. Image Resolution and Contrast Resolution, contrast and light penetration or detection depth features are the most critical for fruit defect detection and, are of primary concern for the demodulated images. The resolution and contrast features can be understood by examining the image formation model as described in Eq. (3.12). A separate discussion on detection depth and its relationship with image resolution and contrast is given in the following subsection.   68 As aforementioned, H x y has a low-pass filtering property. So the MTF of the imaging ( , ) system due to the light-medium interactions and imaging process, which is equal to the magnitude of  ( , ) H x y , should be a bell-shaped function like the one depicted in Figure 3.12 (a). The MTF profile describes how the imaging system transfers the sample information (i.e., contrast) at a given spatial frequency (i.e., resolution), and it has the lower and upper cut-off frequencies that define the passband of the system. According to Eq. (3.12), ACI contains the sample information whose Fourier spectrum has been shifted by xf , compared to that in DCI . It is due to the spectral shifting that a certain amount of higher-frequency sample information, which is absent in DCI due to the limited passband of the system, is down-modulated into the lower-frequency passband and becomes resolvable in ACI , as schematically illustrated in Figure 3.12(b). Hence, ACI is expected to have higher image resolution and contrast than DCI . Figure 3.12. (a) A simulated modulation transfer function (MTF) profile for a structured-illumination reflectance imaging (SIRI) system, and (b) visualization of the spectral shifting of a SIRI pattern image in the Fourier domain [Note: the notations in the figure comply with those in Eq. (3.12)]. It is important to note that the down-modulating of higher-frequency information is accompanied by up-modulating of lower-frequency information, which indicates that ACI would have lower intensity than DCI ; higher frequency would result in the lower intensity, due to the decreasing MTF profile with the increasing frequency. It follows that, assuming a shot-noise   69 limited imaging system, for which the signal-to-noise (SNR) of the system is positively correlated to the square root of the signal level, higher frequencies would result in a lower SNR for ACI . Therefore, there exists a trade-off between image resolution/contrast and SNR. An implicit assumption underlying the mathematical understanding of the resolution and contrast enhancement, is that the sample information ( , S x y ) and the PSF ( , sH x y ) , due to light scattering, remain unchanged in varying spatial frequencies. While this assumption leads to simplification of the analysis, one should be aware that different spatial frequencies would result in different light penetrations, and probably a change in ( , S x y ) and ( , sH x y ) due to the heterogeneity/anisotropy of real samples. 3.5.1.4. Light Penetration In diffuse optics, under the illumination of a narrow pencil beam (e.g., laser light), the light penetration depth in biological tissues is generally defined as the reciprocal of the effective light attenuation coefficient [i.e., 1 3    s a ( a ) ], which is equal to the travelling distance of light corresponding to a decay in power rate by a factor of 1/e (~37%) (Wilson and Jacques, 1990). It follows that, the light penetration depth is estimated to be 2-6 mm for apple tissues with typical absorption ( a ) and reduced scattering ( s ) coefficients of 0.1-0.5 cm-1 and 9.0-17.0 cm-1, respectively, over the spectral region of 500-1000 nm (Lu and Cen, 2015). This depth, however, does not indicate the actually detectable region for an imaging system under spatially-extended wide-field illumination, such as SIRI. In such cases, the detected signal is mostly due to the photons backscattered close to the illumination source [i.e., the decreased source (light entry point)-detector (camera pixel) separations] (Bigio and Fantini, 2016; Dognitz and Wagnieres, 1998), which correspond to a far more superficial depth of tissue interrogation than that derived from diffuse light attenuation. This is especially important in the reflectance measurement   70 geometry, since the remitted light intensity due to the point source decays by many orders of magnitude within millimeters. It has been shown that, through Monte-Carlo modelling on light transport inside the human skin (Binzoni et al., 2008), the optical information in the image captured under wide-field uniform planar-wave illumination, primarily originates from a small, conic like volume of tissue situated vertically under image pixel, with a maximum detection depth of about 2 mm in the wavelength range of 600-1000 nm. Since fruit tissues such as apples normally have more strong light scattering ability (i.e., larger s ) than human skin (Jacques, 2013), the corresponding detection depth is presumably even smaller than that for human skin, albeit the actual depth depends on specific sample and imaging hardware. Back to DCI and ACI images, the former, corresponding to conventional (wide-field) UI-based imaging, has a relatively fixed depth of tissue interrogation, while the latter, due to the use of SI, has the ability to control the detection depth by varying the spatial frequency of illumination patterns. As shown in Figure 3.13 on the simulated MTF of apple tissues, due to scattering and absorption, fruit tissues act as a low-pass filter, which attenuates the reflectance more strongly as the spatial frequency increases (Cuccia et al., 2005). As a result, the low-frequency light is more sensitive to a deeper region of tissue, while the high-frequency light is more sensitive to shallower tissues, and DCI represents an asymptotic case of ACI when the spatial frequency is approaching zero. Hence, DCI can image deeper regions of the tissue than ACI , but the latter enables depth- resolved imaging of tissue, which the former is incapable of.   71 Figure 3.13. The simulated modulation transfer function (MTF) profiles of apple tissues obtained through Monte- Carlo (MC) simulation and diffusion approximation (Cuccia et al., 2009), assuming a typical pair of optical properties of absorption ( 1  0.2 cm  a ) and reduced scattering ( 1  10 cm  s ) coefficients. In addition to the optical properties, other settings for the MC simulation include 5 million photons, reflective index of 1.35, anisotropy factor of 0.9 and thickness of 10 cm for the (slab-geometry) tissue. Furthermore, examination of the composition of photons backscattered from a turbid medium will provide an intuitive, qualitative insight into the relationship between light penetration and image contrast/resolution. The backscattered photons, as schematically illustrated in Figure 3.14, comprise three types, i.e., ballistic, weakly scattered and diffusive or multiply-scattered (Wang et al., 1991), corresponding to three levels of light interrogation in the medium. Figure 3.14. Schematic of light scattering within a turbid medium for reflectance measurements, showing three types of backscattered photons, i.e., ballistic, weakly scattered and diffusive (or multiply scattered), with corresponding light penetration regimes. The ballistic photons experience one backward and one forward scattering event before being detected, and due to the shortest travelling path, they suffer from minimal scattering and thus can deliver images of superior resolution and contrast. However, the ballistic photons yield information on the very superficial layer of the medium, of about one mean free path (MFP) of a   72 photon (Ntziachristos, 2010), usually less than 100 μm for fruit tissues like apple (assuming a scattering coefficient of s  100 cm 1  ). The weakly scattered photons provide information on deeper, sub-surface tissues, and they are still capable of forming well-resolved images due to limited scattering events. One transport mean free path (TMFP), usually less than 1 mm for fruit tissues, represents an upper bound of light penetration, beyond which the photons become completely diffusive (Ntziachristos, 2010), significantly degrading the resolution and contrast of resulting images. Therefore, in principle, the detection depth decreases with the increasing image resolution and contrast (Gigan, 2017; Ntziachristos, 2010). The DCI which corresponds to a deeper light penetration into tissues, would include a larger contribution from diffusive photons, thus degrading the resulting resolution and contrast; while the ACI , which probes a shallower depth, would contain more ballistic and weakly-scattered photons, thus producing better image resolution and contrast. Higher spatial frequency is more likely to enhance image resolution and contrast. This general relationship agrees with the mathematical interpretation of resolution and contrast enhancement due to frequency shifting in the Fourier domain (see Section 3.5.1.3). In case of very high-frequency illumination, only ballistic or quasi-ballistic photons are captured by the sensor, and the imaging process becomes the limiting factor for image resolution and contrast, as encountered in microscopy. Presented in the following sections are several experiments that were conducted to demonstrate the image resolution and contrast enhancement and light penetration features of SIRI. 3.5.2. Experimental Validation 3.5.2.1. Four Experiments The first experiment was to examine the MTF profile of the imaging process in SIRI. Images were acquired from a Spectralon reference panel (Labsphere, Inc., Sutton, NH, USA) with a reflectance   73 rate of 98% or higher. By imaging the white panel, it could be reasonably assumed that the reflectance attenuation was only due to the instrument hardware. A set of sinusoidal patterns, covering 21 spatial frequencies of 0.01, 0.03, 0.05, 0.08, 0.10, 0.12, 0.15, 0.18, 0.20, 0.22, 0.25, 0.30, 0.35, 0.40, 0.45, 0.50, 0.60, 0.70, 0.80, 0.90 and 1.00 cycles/mm, were used for sample illumination. Three phase-shifted pattern images were acquired at each spatial frequency with the phase offsets of 0, 2π/3 and 4π/3, respectively, and were demodulated into DC and AC using the TPD approach [see Eqs. (3.3) and (3.4)]. The average intensities at different frequencies gave the frequency response (i.e., MTF) profiles of the system. Besides, images were also acquired from a dark background with the light source turned off, representing the image noise due to the dark current of the CCD sensor. Other noise sources such as photon-shot and readout noise, were not considered due to their small, negligible contributions. Knowledge of the noise level allow determination of the limiting (upper-bound) spatial frequency of the system, which is defined as the frequency at which the MTF curve fell below the noise- equivalent modulation (NEM) (Boreman, 2001). The second experiment was to demonstrate the resolution and contrast enhancement features of SIRI. A negative USAF 1951 optical target, consisting of different groups of black horizontal and vertical bars at various spacings on a white background, was chosen for this experiment. This target was printed on regular white paper and illuminated with the same set of patterns as in the first experiment, excluding those high frequencies above 0.5 cycles/mm. The resulting DC and AC images were compared in terms of resolution and contrast. Image resolution was visually evaluated based on the ability to resolve the fine details of the USAF 1951 target, while image contrast was evaluated using the Michelson contrast metric (CM) (Bex and Makous, 2002):   MC  ( I max  I min ) ( I max  I min ) (3.15) 74 where max I and minI denote the maximum and minimum intensities of an image, respectively. For the sake of reliability, it is a common practice to replace the maximum and minimum with the mean of a number of the highest and lowest intensities (Thomas et al., 2013). Here, a third of image pixels of the highest and lowest intensities were used to calculate the image contrast. The third experiment was to demonstrate the depth-resolved imaging feature of SIRI. Firstly, the detection depth of SIRI was explored by imaging the USAF 1951 paper target under UI, which was placed beneath a number of sheets of blank printing paper. The thickness of each sheet of paper was estimated to be 100 μm. Afterwards, two targets, including the USAF 1951 and another Snellen eye chart that consists of black letters of various size printed on white printing paper, were put together and imaged by SIRI with the same set of illumination patterns as in the second experiment. Two scenarios were considered in arranging the two targets for image acquisition: 1) the Snellen target was placed on top of the USAF 1951 target, and 2) a sheet of blank printing paper was put on top of the Snellen target that was placed on top of the USAF 1951 target. In both cases, the two targets were placed on top of a stack of blank paper of the same size and about 1 cm in thickness, which served as an opaque background. The fourth experiment was used to further demonstrate the ability of SIRI for defect detection using three defective apple samples, two ‘Delicious’ and one ‘Red Rome’, as shown in Figure 3.15. One of the two ‘Delicious’ apples, purchased from a local grocery store, was inflicted with subsurface tissue bruising by impacting its equatorial area with impact energy of about 0.5 J, which could not be identified visually. The other two apples were hand-picked from an orchard at the Horticultural Teaching and Research Center of Michigan State University in Holt, Michigan, USA. While still on the tree, these apples had been affected with physiological disorders, i.e., circular dark pits in the fruit skin centered on the lenticels for the ‘Red Rome’, and skin russeting for the   75 ‘Delicious’, appearing as a brown, corky and rough surface. SIRI images were acquired from the three apples using the same set of illumination patterns as in the second and third experiments, and image acquisitions for the ‘Delicious’ apple with subsurface bruise were done immediately after it had been damaged by impact. Figure 3.15. Photograph of a ‘Delicious’ apple (left) with invisible subsurface bruising, a ‘Red Rome’ apple (center) with lenticel dark spots, and a ‘Delicious’ apple (right) with skin russeting. The multispectral SIRI system, as shown in Figure 3.4 (right), was used for the above four experiments. Spectral images were acquired in the wavelength range of 650-830 nm with 20 nm increments, with the camera’s exposure time of 300 ms and on-chip 2×2 binning operations, and  the fringe of illumination patterns was perpendicular to the plane spanned by the optical axes of the camera and the projector. 3.5.2.2. Results and Discussion Figure 3.16(a) shows the spectral curves for DC and AC images acquired from the white panel. These spectra at different spatial frequencies were due to the joint effect of the LCTF and camera, which peaked around 730 nm. Figure 3.16(b) shows the frequency response curves for DC and AC (i.e., the MTF profile due to the imaging process of SIRI) images at the wavelength of 730 nm. As expected, the intensity of DC remained basically unchanged, while the intensity of AC was lower than that of DC (even at the lowest spatial frequency), and decreased steadily with the increase in the spatial frequency, which was true of other wavelengths as well. The decreasing MTF profile confirmed the low-pass filtering property of the imaging process, which would wash out the high-frequency sample information during imaging acquisition.   76 Figure 3.16. (a) Spectral responses of direct component (DC) and amplitude component (AC) images at different spatial frequencies, where the line profiles, which have been scaled by dividing the maximum value of DC, correspond to individual spatial frequencies ranging between 0.01 and 1.0 cycles/mm, and the downward arrow indicates the direction of increasing spatial frequencies (fx); (b) frequency responses of the DC and AC images and the noise level at 730 nm, where each line profile has been normalized to the maximum of the DC image. The MTF profile was well modeled by a Gaussian function, ( ) f x  a exp(  2 ) ( x b  2 2 c , where a, b ) and c are parameters to be estimated, with the determination coefficient being larger than 0.999 for all the wavelengths. The limiting frequency was then determined by setting the function value to the noise intensity. As indicated in Figure 3.16(b), the limiting spatial frequency was 0.57 cycles/mm at the wavelength 730 nm, where the intensity of AC decreased to less than one tenth of that of DC. Similarly, the limiting spatial frequency for the other wavelengths was in the range of 0.41-0.59 cycles/mm. It is important to stress that these obtained limiting spatial frequencies do not represent the maximum that SIRI is able to support, since it largely depends on the imaging hardware and configurations. For example, one can increase the limiting frequency by setting a longer camera exposure time without causing signal saturation, especially for those wavelengths at which the imaging system has a low spectral sensitivity [See Figure 3.16(a)]. However, based on this experiment, it is recommended that the spatial frequency used for SIRI should not exceed 0.6 cycles/mm. Since the wavelength of 730 nm gave the highest image intensity and also relatively large limiting spatial frequency, only spectral images acquired at this wavelength were used in further analysis. Figure 3.17 shows an example of the resulting DC and AC images for the USAF 1951   77 target. All the images could resolve well the black bars for the groups -1 and -2, but not for the groups 0 and 1, which represented even finer details, as shown in the close-up sub-image. So comparisons of image resolution were made based on resolving the bars for the groups of 0 and 1. Figure 3.17. The direct component (DC) (left) and amplitude component (AC) (middle) images of the USAF 1951 target at the spatial frequency of 0.15 cycles/mm, and the close-up view (right) of the AC image focusing on the groups 0 and 1 elements of the USAF 1951 target.       Figure 3.18 shows one DC image and all AC images at different spatial frequencies for the close-up view of the optical target. It should be pointed out that each spatial frequency produced a DC image but all the DC images remained basically the same. Hence only the first DC image (i.e., at 0.01 cycles/mm) was presented for comparison. Visually, as the spatial frequency increased, some vertical bars (e.g., those bars in the lower right corner, with the actual spacing of bars of about 0.5 mm) became more and more resolved, while the resolution for the horizontal bars was not noticeably improved. This is due to the fact that the fringes of illumination patterns were parallel to the vertical bars, so that the resolution enhancement mainly occurred in the horizontal direction, that is, the direction in which the frequency components of illumination patterns are situated in the Fourier domain, thus making the vertical bars more resolved.   78   Figure 3.18. The close-up view of direct component (DC) and the amplitude component (AC) images at different spatial frequencies (cycles/mm) as indicted in the name of individual sub-images, for the bars of groups 0 and 1 of the USAF 1951 target. As a further illustration, the pixel profiles for four representative groups of vertical and horizontal bars were analyzed for the DC image and the AC images at 0.5 cycles/mm. As shown in Figure 3.19, with using vertically orientated illumination patterns, the AC image gave much more resolved pixel profiles than the DC image for the vertical bars, while the AC and DC images gave similar pixel profiles for the horizontal bars. Moreover, the horizontally orientated illumination yielded opposite results, and the grid illumination patterns, which integrated both vertical and horizontal frequency components, produced more isotropic resolution enhancement at the cost of increased image acquisitions (five phase-shifted illumination patterns are needed for recovering both DC and AC images).   79       Although resolution enhancement depends on the orientation of illumination patterns, it may Figure 3.19. The close-up image and pixel profiles for the corresponding lines in the DC and AC images: (top pane) vertically orientated illumination patterns, (middle pane) horizontally orientated illumination patterns, and (bottom pane) two-frequency grid illumination patterns. not make a marked difference in fruit defect detection, since most defects are non-directional in their spatial distribution. Hence the selection of illumination direction (i.e., vertical, horizontal or grid patterns) depends on the application and system setup, to ensure minimal pattern distortion and high-quality demodulated images. Besides, the imaging speed is also a potential factor, given that grid patterns requires more image acquisitions than vertical and horizontal patterns for demodulation. The following discussion is focused on the vertical illumination patterns only. Figure 3.20 shows the contrast curves for DC and AC images of the USAF 1951 target at different spatial frequencies. For both full-size and close-up images, DC had almost a constant contrast value since it was independent of spatial frequency, while AC had much higher contrast values than DC and the contrast increased steadily with the spatial frequency.   80 Figure 3.20. The curves of contrast values for the full-size (left) and close-up (right) direct component (DC) and amplitude component (AC) images of the USAF 1951 target at different spatial frequencies.        All these findings demonstrated that AC images enhanced resolution and contrast compared to DC images, and hence the superiority of SIRI over conventional UI based-imaging technique in enhancing image features. Figure 3.21 shows the UI images of the USAF 1951 target at different depths, defined by the number of blank printing paper sheets stacked on top of the target. The UI image, corresponding to the lowest (i.e., 0) frequency illumination patterns, draws an upper bound of the detection depth that can be reached by SIRI. Clearly, at the depth of more than three sheets of paper, direct imaging of the target was impossible due to substantially deteriorated resolution and contrast. Thus, for the present imaging system, the maximum detection depth with acceptable resolution and contrast was confined to no more than 3 sheets of paper (or less than 400 μm). Although the detection depth for fruit samples would be different, this experiment revealed the superficial sampling nature of reflectance imaging under wide-field uniform illumination. Figure 3.21. Uniform illumination images of the USAF 1951 paper target placed beneath 1-6 (from left to right) sheets of blank printing paper. Figure 3.22 shows the resulting DC and AC images for two paper targets, i.e., the Snellen eye   target and the USAF 1951 target.   81 Figure 3.22. The direct component (DC) and amplitude component (AC) images for two paper targets for different spatial frequencies: (top three rows) the Snellen target is placed on top of the USAF 1951, and (bottom three rows) one sheet of blank paper is laid on top of the Snellen target that is on top of the USAF 1951 target. For the Snellen target directly exposed to the camera, the DC image, which corresponded to the maximum detection depth, contained features on both targets; and the lower-frequency AC images were similar to the DC image in revealing the features of both targets. However, as the spatial frequency increased, the AC images were increasingly focused on the surface Snellen target, while the features of the USAF 1951 target almost disappeared completely at the highest spatial frequency of 0.5 cycles/mm. A similar pattern was also observed for the second scenario, in which   82 one sheet of paper was placed on top of the Snellen target. These findings confirmed that the detection depth of AC images decreased with the spatial frequency and thus depth-resolved imaging of the sample can be achieved by controlling the spatial frequency. It is important to note that although AC images for different spatial frequencies would have different detection depths, they are not truly sectioned or tomographic to provide information exclusively on certain depth as in conventional tomographic imaging. All the images, including DC, are still dominated by the shallower target information. An intuitive approach to obtaining a better depth-sectioned image is to subtract from the DC or a lower-frequency AC image a higher- frequency AC image multiplied by a varying constant, as illustrated in Figure 3.23. Such simple subtractions, however, cannot completely eliminate the information of the shallower target, due to the nonlinear relationship between the demodulated images and detection depth. Using inverse image reconstructions in diffuse optical tomography (Konecky, 2012; Lukic et al., 2009), it may be possible to obtain more depth-specific images at much greater depths. Figure 3.23. The direct component (DC) image (left), amplitude component (AC) image at 0.30 cycles/mm (AC_F0.3) (second from left), and the reconstructed images (three from right) by subtracting the AC image multiplied by a constant (i.e., 1, 1.5 and 2, respectively) from the DC image. It should be noted that the heuristics for deciding an optimal constant is open to question. Figure 3.24 shows the DC and AC images for the bruised ‘Delicious’ apple. As demonstrated in the preliminary tests, the subsurface bruising was invisible in the DC image, which was consistent with visual inspection, and it still could not be observed clearly in the lower-frequency (i.e., less than 0.05 cycles/mm) AC images. However, as the spatial frequency increased, the bruise region became well resolved with enhanced contrast between bruised and normal tissues. In the meantime, due to improved image resolution and contrast at higher frequencies, the surface   83 lenticels of the fruit became more distinct. Above a certain spatial frequency (e.g., 0.4 cycles/mm), the bruise started to fade out, and the image became greatly darkened due to strong signal attenuation. Hence, the spatial frequency for effective bruising detection would be confined to the range of 0.05-0.40 cycles/mm. Figure 3.24. The direct component (DC) and amplitude component (AC) images at different spatial frequencies (cycles/mm) for a ‘Delicious’ apple with subsurface bruising. The DC and AC images for the two other apples with surface defects are shown in Figure 3.25. The ‘Red Rome’ apple with lenticel spots gave a quite different picture. The lenticel spots could be readily identified in the DC and low-frequency (e.g., less than 0.10 cycles/mm) AC images, and also resolved in the color photo (Figure 3.15). However, as the spatial frequency increased, the lenticel spots became less obvious and even disappeared. This was because the lenticel spots, although visible from the fruit surface, were not fully exposed to air, but instead they occurred in the outer cortex and lower epidermis covered with thin, intact cuticle (Brook, 1968). The high- frequency AC images were mainly focused on the very surface features, thus overpassing the lenticel spots. The ‘Red Rome’ did not have as many noticeable lenticels as the bruised ‘Delicious’, because of the difference in the skin morphology of the two varieties. Likewise, the intensity of the AC images of ‘Red Rome’ decreased with the increasing spatial frequency.   84 Figure 3.25. The direct component (DC) and amplitude component (AC) images at different spatial frequencies (cycles/mm) for a ‘Red Rome’ apple with lenticel breakdown (top three rows) and a ‘Delicious’ apple with surface russeting (bottom three rows). The third apple of ‘Delicious’ variety exhibited skin russeting, which, relative to the lenticel spots, is more of a surface defect. Skin russeting results from microscopic cracks, which probably occurs during early fruit development, followed by the formation of a periderm in the hypodermal cell layer, appearing at the fruit surface after the epidermis and cuticle are shed (Faust and Shear, 1972; Khanal et al., 2013). As shown in Figure 3.25 (bottom three rows), all DC and AC images showed the russeting defect, and the defect became more distinct as the spatial frequency increased, except for some intermediate frequencies (i.e., 0.1-0.2 cycles/mm) which had relatively low   85 contrast. The spatial frequencies above 0.2 cycles/mm seemed to be more appropriate for detecting the defect (on a side note, attention should be paid to the enhanced non-defect surface features and decreased image SNR in using high-frequency patterns). Apart from skin russeting, one also discerned another circular and brighter defect in the AC images at spatial frequencies of 0.03-0.25 cycles/mm, which was attributed to slight, old-aged subsurface tissue damage, probably resulting from finger pressure during apple picking. The above findings suggest that while AC images are effective in enhancing certain features at specific spatial frequencies, they are not always superior over the DC image in fruit defect detection. The actual performance of AC images would depend on a number of factors, including defect type, and inherent optical contrast, fruit variety, spatial frequency of light pattern, and the wavelength selected. In essence, it depends on whether adequate image resolution and contrast can be obtained at the probing depth where the defect of interest is located. Although AC images have been proven effective in detecting subsurface bruising and skin russeting, their effectiveness or potential in detecting other subsurface and surface defects, remains to be fully explored. Moreover, in high-frequency AC images, surface lenticels become more apparent, accompanied with a significantly diminished image SNR, which may pose more challenges for accurate and reliable defect segmentation and detection. 3.5.3. Conclusions An image formation model and a general demodulation methodology are proposed for SIRI, providing a basis for analyzing the features of this technique. The imaging process of SIRI has a low-pass filtering property, which, together with the sample absorption and scattering, accounts for contrast and resolution degradation of resulting images. AC images, which represent the hallmark of SIRI, compared to DC images that correspond to conventional UI-based imaging,   86 possess enhanced image resolution and contrast and also have a depth-resolving ability. The resolution and contrast enhancement can be explained by down-modulating of high-frequency sample information in the Fourier domain and suppressed light scattering in the sample, but they are convolved with a decreased light penetration or detection depth as the spatial frequency of illumination patterns increases. The effectiveness of AC images for apple defect detection would depend on a number of factors, such as defect type, fruit variety and spatial frequency of light patterns. Without using inverse computations, SIRI is only able to interrogate a superficial region of fruit samples, and it should thus be positioned as a subsurface-surface imaging technique for thin-skinned fruits, such as apple. 87   4. IMAGE DEMODULATION APPROACHES 4.1. INTRODUCTION In SIRI, the acquired pattern images need to be processed to remove the fringes or stripes to obtain intensity (i.e., DC and AC) images and/or even phase images, depending on the application need, the process of which is referred to as image demodulation. The image demodulation can be achieved using a conventional multi-step phase-shifting technique (Lu et al., 2016), which requires a minimum of three phase-shifted pattern images. However, the requirement for acquiring multiple images at the specific phase offsets limits the speed of image acquisition of the techniques for practical applications. Hence, new demodulation approaches are needed in order to move SIRI towards real-time practical implementation. Many attempts have been made to address the issue of multiple image acquisitions for rapid imaging. Some research focuses on optimizing pattern-coding strategies and or instrument hardware. For instance, color coding that integrates three patterns into individual channels of a color pattern, enables optical imaging by acquiring a single-shot image (Krzewina and Kim, 2006; Su, 2007). However, this method requires a color-sensitive camera and a projector that is able to shine color stripes, and it also suffers from the issues of color crosstalk and color absorption, thus restricting its use. Polarization coding provides another means for single-shot imaging (Wicker and Heintzmann, 2010), which encodes three patterns into polarization coded light. This approach, however, is restricted to samples with polarization-maintaining properties. Recently, researchers proposed a two-pulse illumination technique for instantaneous imaging of rapidly occurring events such as water sprays (Kristensson et al., 2014; Storch et al., 2016). In this approach, the sample was illuminated with two phase-mismatched modulated laser beams successively, and a SI image with suppressed residuals was extracted by judiciously selecting a high spatial frequency. This   88 technique requires sophisticated optical arrangements and remains to be explored for characterizing biological tissues. Considerable research, on the other hand, has been focused on the algorithmic improvement or optimization. A single-shot method based on the 1-D Hilbert transform (HT) was investigated for retrieving amplitude images for optical property measurement of biological tissues (van de Giessen et al., 2015; Vervandier and Gioux, 2013). In this method, the background intensity, i.e., DC, is first eliminated by using a high-pass filter in the Fourier domain and then HT is applied to the DC-free image line by line to obtain the AC image. However, the resulting AC images using this method are often of low, deteriorated image quality due to imperfections in the filtering processes, and also could lose information that is originally encoded in the correlation between lines. Further, a two-phased based HT method was proposed in which DC was removed by subtraction of the two phase-shifted pattern images (Zhou et al., 2015). Spiral phase transform (SPT), which acts as a 2-D extension to conventional 1-D HT (Larkin, 2014; Larkin et al., 2001), provides an alternative for amplitude retrieval. SPT has recently been applied for optical property measurement of biological tissues by SFDI (Nadeau et al., 2014), which is done by obtaining the AC image from a single pattern image plus a UI image. While this demodulation method holds potential for real-time applications, its accuracy is limited by the fact that only one SI image data is involved. Orthogonalization is able to produce a pair of in-quadrature images when applied to vectorized DC-free or -filtered sinusoidal pattern images, which thus provides another method for amplitude demodulation. Vector orthogonalization is a well-established method that can be implemented through Gram-Schmidt orthonormalization (GSO), Householder reflections, or Givens rotation, among which, GSO is simple, popular and fast to use (Strang, 2009). In recent years, GSO has   89 been used for phase retrieval in 3-D object shape measurement (Niu et al., 2015; Vargas et al., 2012), in which GSO extracts a phase map from two phase-shifted pattern images. This method can be applied to two arbitrarily phase-shifted pattern images, thus eliminating the need for precise phase control, and it is a spatial-domain based method, without considering filtering preprocessing, compared to others like HT and SPT that are frequency-domain based. Single-frequency sinusoidal patterns are predominantly used in SI-based imaging techniques because of easy retrieval of amplitude and/or phase information. As shown in chapter 3, light penetration in biological tissues decays with spatial frequency of illumination patterns, which provides the basis for depth-resolved imaging of tissue heterogeneity by using a set of different spatial frequencies. Single-frequency sinusoidal patterns are thus sub-optimal in interrogating the tissue at varied depths, unless one uses many such patterns at different spatial frequencies. A composite sinusoidal pattern that integrates multiple frequencies of interest, on the other hand, has the potential for providing an effective and efficient means for depth-resolving tissue imaging. For fruit defect detection, composite sinusoidal patterns are more likely than single-frequency ones to ascertain possible defects because of the ability of depth-varied interrogation of tissues. Recently, composite sinusoidal patterns have been investigated for phase retrieval in 3-D shape measurement of objects (Garcia-Isais and Alcala Ochoa, 2014; Guan et al., 2003; Ochoa and Garcia-Isais, 2015; Wu et al., 2012). But, demodulating composite sinusoidal patterns is not as straightforward as in single-frequency patterns since different frequency components are interwoven together in the spatial domain. In previous studies on phase retrieval, researchers used filters to separate different frequency components. However, the filtering process is prone to errors, especially when it comes to the frequency components that are close to one another, and moreover, selection of an appropriate filter often requires manual intervention. It is thus desirable and also necessary to   90 develop an effective means for demodulating composite sinusoidal patterns for SIRI, so as to expand the capacity of the technique for fruit defect detection. This research was therefore aimed to develop approaches to demodulating both single- frequency and composite SIRI pattern images. Two specific objectives were to: 1) develop SPT- and GSO-based approaches to recovering DC and AC images from two phase-shifted images, and evaluate these approaches through numerical simulations and experiments; 2) develop a Fourier transform (FT) based approach to recovering AC images from phase-shifted composite pattern images, and also evaluate the approach through simulations and experiments. 4.2. SINGLE-FREQUENCY DEMODULATION 4.2.1. Demodulation Approaches 4.2.1.1. SPT SPT, which was originally proposed by Larkin et al (2001) for demodulating naturally occurring fringe patterns such as human fingerprints, provides an isotropic, elegant extension of conventional 1-D HT to 2-D domain. Before introducing SPT, we will first show how its relative HT is used for amplitude retrieval. HT is an important tool for constructing analytical signals in signal processing, which is defined through an improper integral as follows (Bracewell, 2000; Hahn, 1996): HT f x ( )] [  ( ) g x  1    ( f x  ) 1 x  x dx  = 1 x   ( ) f x (4.1) where HT and  denote operators for HT and convolution, respectively. Based on the convolution theorem, we can take the Fourier transform (FT) of Eq. (4.1): [ ( )] FT g x  ( ) G u    ( )exp( 2 g x   iux dx ) i   sgn( ) ( ) u F u (4.2) where FT is the Fourier transform operator; u and i denote the frequency-domain coordinate and imaginary unit, respectively; sgn(u) is a signum function [i.e., sgn(u) = 1 if u>0 and sgn(u) = -1 if u<0]; and ( ) F u    ( )] FT f x [ x . 91 It can be seen from Eq. (4.2) that HT has no effect on the amplitude of Fourier components, but shifts their phases by π/2, positively or negatively, depending on the sign of u, which thus transforms the cosine components in a function to sines and vice versa. Such transformations are referred to as quadrature transform. Hence, given a DC-free harmonic signal, its amplitude can be obtained using HT as follows: ( ) A x |  ( ) f x  iHT f x ( )]| [ (4.3) To process a 2-D fringe pattern, HT needs to be implemented row by row and column by column, which is inefficient and may cause loss of information encoded in the correlation between lines and also is problematical if the phase of the pattern is not strictly monotonic. SPT provides a natural generalization of 1-D HT to 2-D domain. The key to SPT is to replace the 1-D linear signum function in Eq. (4.2) with a 2-D revolutionary signum function, known as spiral phase function (SPF) or spiral phasor, which is defined as follows (Larkin et al., 2001): ( , ) S u v  u iv  u v 2  2 = exp[ ( , )] i u v  (4.4) where u and v are frequency coordinates in the Fourier domain, and  is the phase angle. It is noted that the SPF is not well defined at the origin, which is however unimportant since it is continuous everywhere else. By convention, one can set the function value at this singular point to a constant such as zero. Figure 4.1 shows the real and imaginary parts of SPF and the distribution of principal values of its phase. The spiral nature of the phase distribution gives rise to the names of SPF and SPT. Compared to the 1-D sign function, SPF shares two commonalities, i.e., unit magnitude and odd symmetry, but it has no preferred direction, in contrast to the directional step shape in the former, which lays the basis for achieving isotropic 2-D quadrature transform (Larkin, 2014).   92 Figure 4.1. Real and imaginary parts of the spiral phase function (left and middle), and the distribution of its phase angle (right) where only principal values in the range ±π are shown. Described below is the implementation of SPT for amplitude demodulation. The DC retrieval is discussed later since it cannot be directly achieved by SPT. SPT, defined using the operator $ , involves three operations, including 2-D Fourier transform, spiral phasor multiplication in the Fourier domain and inverse 2-D Fourier transform, which can be expressed in terms of operator notation as follows (Larkin, 2014): ${[ ]}  IFT S u v FT { ( , )  {[ ]}} (4.5) where IFT and  denote inverse Fourier transform and element-wise multiplication, respectively. Presented below is a simple derivation of how SPT recovers the amplitude from a DC-free fringe pattern as follows: ( , I x y 0 )  I AC ( , x y )cos[ ( , x y  )]  I ( , x y )cos[2 (  f x x  f y y )  ]  AC (4.6) where xf and yf denote the spatial frequency along x-axis and y-axis, respectively, and  is phase offset. Multiplying the Fourier transform of Eq. (4.6) by Eq. (4.4) yields: ( , ) S u v FT I x y   ( , )  0  u iv  u v 2 2  [ ( S f = I AC 2  I AC 2 [ ( - , - )exp( i) + ( + , + )exp(- i)]   u f v f u f v f   x y x y (4.7) , f y x ) ( - , - )exp( ) i   u f v f y x  ( S f , x f y ) (  u f v f   , x )exp(- )] i  y where  is the Dirac function and its sifting property is used in the above derivation. Further, taking the inverse Fourier transform of Eq. (4.7) yields:   ( , IFT S u v FT I x y { ( , )  [ 0 )]}  iI ( , ) ( x y S f , x f y ) ( , I x y  0 ) AC (4.8) 93 where 0 ( , I x y )  I AC sin[2 (  f x x  f y y )  ]  . Therefore, based on Eqs. (4.5), (4.6) and (4.8) and noting | ( S f , x f y  , the amplitude ACI ) | 1 can be obtained from the following equation: I AC ( , x y )  I 2 0 ( , x y )  | $[ I ( , x y )] | 2 0 (4.9) where the symbol ‘  ’ instead of ‘=’ is used in view of the fact that there exist numerical errors resulting from the finite discrete Fourier transform in practice. From the above derivation, SPT is faced with a need to eliminate DC from the pattern, which cannot be well addressed by the general filtering methods. Here a simple and effective alternative is to subtract two-phase shifted sinusoidal patterns, thus leading to the two-phase based SPT we intend to have. For simplicity, two vertical sinusoidal patterns (i.e., yf  ) are illustrated here, 0 which are expressed as follows: ( , ) I x y 1  I DC  I AC ( , ) I x y 2  I DC  I AC cos(2 ) f x  1  x cos(2 f x  2  x ) Then, the difference difI of the two patterns is defined as: I dif ( , ) x y  ( , ) I x y 1  I x y 2 ( , ) 2  I AC sin( )sin(2 )     f x x (4.10) (4.11) (4.12) where =    2 1  and 2  = 1+   2 2 . According to Eq. (4.8), the amplitude is obtained as follows: I AC  [2sin( )] 1   I  | $( I dif ) | dif (4.13) The angle  in Eq. (4.13) is required to be known a priori, which, however, is unimportant in reality since we are mostly concerned about the amplitude contrast in fruit defect detection. That means that SPT is able to retrieve the amplitude image from two arbitrarily phase-shifted pattern images. Even so, it is preferable to have 2 1      , such that the largest amplitude is obtained   94 and meanwhile the DC image, DCI , which can be used to obtain ratio images (i.e., the ratio of ACI to DCI ), can be readily obtained by taking the average of the two pattern images. For convenience, the two-phase shifted SPT based approach is denoted as “2-phase SPT”. One special case for the “2-phase SPT” is when one of the two phase images contains no AC term, that is, only uniform illumination. Then the amplitude can be obtained as follows: I AC   I 1  I 2 2   $  I 1  I 2  2 (4.14) In the very simplest case, one can use a single pattern image, in conjunction with a preprocessing (i.e., filtering) for DC removal, for amplitude demodulation. The two cases of SPT are denoted as “2-image SPT” and “1-phase SPT”, respectively, which are to be used in comparison with “2- phase SPT” in the following sections. Utilizing the spiral phase formalism, we can also give a general expression of the DC image derived from the two phase-shifted patterns. Define the sum sumI of the two patterns: I sum ( , ) x y  ( , ) I x y 1  I x y 2 ( , ) 2[  I DC  I AC cos( )cos(2 )]     f x x (4.15) Similar to retrieving ACI , multiplying the Fourier transform of difI by , S u v ( ) yields: , S u v ( )  [ F T I ( , x y )] dif )[ iI   = iI  ( ( , sin u S u v    A C ) exp( , ( f i v u     x , sin ) exp( ( [ v u f    A C x ( ) exp( , i v f u     f )] )] x , v ) exp( i  ) x i  ) (4.16) It can be seen that the sine function described in Eq. (4.12) is transformed into a cosine function described in Eq. (4.16). Then, DCI can be obtained as follows:   I DC ( , ) x y  1 2 [ IFT FT I { sum ( , )] x y  i 95 cot  ( , ) [ S u v FT I ( , )]} x y dif (4.17) In Eq. (4.17), assuming α is unknown, based on the linearity property of Fourier transform, cot can be estimated as follows: cot  sgn[  ( I sum  ( I dif f f ,0) x ,0) x ]  ( I sum  ( I dif f f ,0) x ,0) x (4.18) where ‘sgn’ is a signum function responsible for addressing the sign ambiguity; and su mI and I dif denote the Fourier transformations of sumI and difI , respectively. 4.2.1.2. GSO GSO provides an alternative to SPT for amplitude retrieval when applied to DC-free sinusoidal patterns, and it is advantageous in terms of fast computation and clean resulting images free of ripple artifacts encountered with Fourier transform. Given two DC-free sinusoidal patterns:  I 1  I 1 ( , x y )  I cos(2 f x x   1 ) AC ( , x y )  I cos(2 f x x   2 ) AC (4.19) (4.20) Prior to applying GSO, the two pattern images need to be reshaped into vectors, and then GSO begins with normalizing one of the two vectorized patterns: q 1   I  1 I 1  Subtracting the projection of 2I in the direction of 1q and normalizing the result: ˆ I 2   I 2  , q q I 2 1 1  q 2  ˆ I 2 ˆ I 2 (4.21) (4.22) (4.23) Note that in Eqs. (4.20)-(4.22), ,  and represent the inner products of two vectors and the normal of a vector, respectively, and the coordinates (x, y) are dropped for simplicity (also in the   96 following derivations). Here 1q and 2q form a pair of orthonormalized vectors. Presented next is how to use the two vectors to retrieve the amplitude image, which is found  to be related to the norm of 1I  or 2I  . Without loss of generality, take for example 1I . The square of its normal is:  I 1  N  1 i  [ I 2 AC 2 cos (2  f x x i   1 )] = N I 2 AC 2  I 2 AC 2 N  1 i  cos(4  f x x i  2 )  1 (4.24)  where N is the total number of sampling points. Assuming 1I has more than one complete cycle of fringes (i.e., 1xf  for a unit dimension), it follows that the second term of Eq. (4.24) is negligible because N  | N  1 i  cos(4 f x x i  2 ) |  1 (the inequality can be readily verified by graphing the ratio of its right-hand operand to left-hand operand), which leads to an important fact: ACI   I 1 2 N (4.25)  Then, given the phase change of 2I after orthonormalization, 1q and 2q can be expressed as: Further, q q 1 , 2  1 N q 1  2 N cos(2 f x x   1 ) q 2  1 N cos(2 f  2  x ) cos(    2  1 )  1 N N  1 i  cos(4  f x x i     2  1 (4.26) (4.27) ) (4.28) Similarly, under the assumption that the fringe pattern has more than one complete period, the second term of Eq. (4.28) is approximated to be zero, which leads to cos(   1 2 ) 0  . Here comes the major conclusion that 1q and 2q obtained from GSO are in quadrature (with a phase difference of an integer multiple of 2 ). Then, the amplitude can be readily obtained as:   97 I AC   I 1 q 2 1  q 2 2 or  I 2 q 2 1  q 2 2 (4.29) For convenience and clarity, the whole procedure for retrieving ACI from two DC-free sinusoidal patterns is denoted as GSO (  I 1 ,  I 2 ) . As in SPF, DC removal is needed for GSO. In practice, when DCI is known or has been retrieved, ACI can be obtained by GSO( I 1  I DC , I 2  I DC ) ; alternatively, one can use the difference and sum (i.e., difI and sumI ) of the two patterns to calculate ACI : I AC  GSO( I , I dif sum ) 1 2 ( I dif 2  ( I sum  2 I DC ) 2 ) (4.30) Both options to GSO for amplitude retrieval require simple computations and no prior information of phase shifts of the patterns. It is noted that when the phase difference of the two patterns is close to an integer multiple of π, the GSO methods will be seriously inaccurate because of the failure in orthonormalizing two almost parallel patterns, which is shown in the following section. In practical implementation, loosely speaking, Eq. (4.30), which requires the DC removal from only one pattern image, may lead to better accuracy than the first option, especially when a filter is used for DC removal. Hereinafter, GSO1 and GSO2 are used to represent GSO( I 1  I DC , I 2  I DC ) and the option described in Eq. (4.30), respectively. 4.2.2. Performance Evaluation 4.2.2.1. SPT-based Demodulation Numerical Simulations Three phase-shifted sinusoidal patterns, as shown in Figure 4.2, were created in Matlab to simulate real pattern images captured by SIRI. These images were eight-bit gray-scale, with the dimensions of 520 × 688 pixels and the phase offsets of 0, 2π/3, 4π/3 respectively. The parameters DCI and ACI were both set to (255/2); and xf was set to 0.15 cycles/mm. A DC image of the same size was   98 created as well by setting all the values to DCI to simulate the uniform illumination. All these images were contaminated with Gaussian noise whose variance was set to 1. All image processing throughout this section was performed by Matlab R2014a software (The Mathworks, Inc., Natick, MA, USA) on a 3.7-GHz desktop PC. Figure 4.2. Simulation images: (a) direct component (DC), and fringe pattern images with phase offset -2π/3, 0 and 2π/3 [(b), (c) and (d), respectively]. To obtain the amplitude images, without loss of generality, “2-phase SPT” was conducted on the first and second pattern images, “2-image SPT” on the DC and the first pattern images, and “1- phase SPT” on the first pattern image. In implementing “1-phase SPT”, the DC background present in the images was eliminated by applying a high-pass Gaussian filter to the two pattern images, and the filter size was decided by trial and error, which might be not optimal. Besides, the conventional TPD was performed on all three pattern images. To quantitatively evaluate the performance of these demodulation methods, the relative demodulation error (RDE) was calculated, which is defined as the ratio of the root-mean-squares error (RMSE) between the theoretical and obtained amplitude to the theoretical one (i.e., ACI ). Moreover, the processing time of each method was also taken into account. Both the RMSE and processing time were obtained by averaging the results for 50 consecutive runs. Figure 4.3 shows the amplitude images obtained by the different demodulation approaches. Visually, the four approaches gave very similar results. The resulting RDEs and processing times for these approaches are summarized in Table 4.1. The conventional TPD method was the most   99 accurate and fastest in computation speed. The fast speed of TPD was expected since it involves simple algebraic operations that are quite easy and fast to implement in Matlab. However, it was noted that TPD is the most time-consuming in practice due to its longer time for acquisition of three images. The proposed “2-phase SPT” showed its advantages; it achieved high accuracy and fast speed only after TPD. Both “2-image SPT” and “1-phase SPT” had similar accuracy, but the former, which required no frequency-domain filtering, was thus much faster than the latter. Overall, the “2-phase SPT” showed good performance in both accuracy and speed. Figure 4.3. Amplitude images obtained by the conventional three-phase demodulation (TPD) and three spiral phase transform (SPT) based approaches. Table 4.1. Comparison of four different approaches to amplitude demodulation Method RDE Time (s) TPD 0.006 0.005 “2-phase SPT” 0.007 0.054 “2-image SPT” 0.012 0.057 “1-phase SPT” 0.013 0.164 Note: TPD, SPT and RDE represent the three-phase demodulation, spiral phase transform, relative demodulation error, respectively. The noise level in images is a critical factor that affects the demodulation accuracy. A good method should be robust to image noise. The effect of noise on the demodulation accuracy was examined by varying the variance of noise from 0.001 to 100. Figure 4.4 shows the curves of RDE versus noise variance for the four approaches. Overall, the methods of TPD and “2-phase SPT” were less susceptible to noise than the methods of “2-image SPT” and “1-phase SPT”, which indicated that subtraction of two phase-shifted images could be effective for noise suppression. The “2-phase SPT” was more accurate than TPD at high noise levels. TPD, compared to “2-phase SPT”, used one more image for amplitude demodulation, which could induce more noise. In addition, the “1-phase SPT” outperformed “2-image SPT” when the noise variance exceeded a   100 certain level, which was probably due to the fact that the frequency-domain filtering for DC removal implemented in “1-phase SPT” exerted a certain degree of the effect of image de-noising. Given the noise effect, image preprocessing to reduce the noise is recommended before using these methods, which also is a common practice in image analysis. Figure 4.4. The effect of noise variance on the relative demodulation error (RDE) for the conventional three-phase demodulation (TPD) and three spiral phase transform (SPT) based approaches.   Experimental Validation The broadband SIRI system as described in Section 3.2 was used for the experiment on bruise detection of apples. Two apples, one “Gala” and the other “Golden Delicious”, free of visual defect, were used for bruise detection. Bruises were induced on the two apples through impact tests using a pendulum with a wooden ball attached to one end, with the impact energy of about 1.0 J. In addition, impact tests were also carried out on two ‘Jonagold’ apples, each of which was subjected to two levels of bruising, with impact energy of about 0.5 and 0.8 J, respectively. A further detailed description of the impact test is given in Lu et al. (2010) and Zhu et al. (2016). Image acquisitions for the apples were done immediately after the bruising tests. The spatial frequency of 0.15 cycles/mm was used for the “Gala” and “Golden Delicious” apples, and the four spatial frequencies of 0.05, 0.10, 0.15 and 0.25 cycles/mm were used for the two “Jonagold” apples. The samples were illumination under three phase-shifted sinusoidal patterns with the phase   101 offsets of -2π/3, 0 and 2π/3 at each spatial frequency. Moreover, uniform illumination was also performed to collect DC images of the apples. All the demodulation methods were used and compared for processing the images of the “Gala” and “Golden Delicious” apples, while only the “two-phase SPT” was used for the two “Jonagold” apples. Figure 4.5 shows the acquired DC and three phase-shifted pattern images for “Gala” and “Golden Delicious” apples. These images had been preprocessed for correction of non-uniformity of the source illumination, and high-frequency noise removal. Amplitude demodulation was done on these images by using TPD and the three SPT approaches described above. Due to the enhanced visibility of the bruise defect, only the ratio images, obtained by dividing AC by the corresponding DC images, are presented. Figure 4.5. The direct component (DC) and three phase-shifted pattern images from left to right for “Gala” (top) and “Golden Delicious” (bottom) apples. Figure 4.6 shows the resulting ratio images. For the two apples, the amplitude images revealed the bruise to varying degrees depending on the demodulation approach applied. By visual inspection, the TPD and “2-phase SPT” methods achieved comparable and strong contrast between the defect and normal tissues for both apples, followed by the “2-image SPT” that outperformed “1-phase SPT”. Further, to quantitatively evaluate the image contrast for the four demodulation approaches, the apple in each image was assumed to be composed of two regions, one for defect and the other for normal tissue. The defect region was decided by thresholding the amplitude   102 images obtained by “2-phase SPT” due to the strong visual contrast. A contrast index (CI) was defined as the ratio of the between-class variance to the total variance of pixel intensities, following the principle of Otsu’s thresholding (Krishnaswamy et al., 2014), which was calculated as follows: ( xN x CI  2  ( yN y 2  z ) ( z i 2  z ) ) z  zN  1 i  (4.29) where xN , yN and zN are the pixel number of the defect, non-defect and the entire regions, respectively ; x , y and z are the average intensities of the corresponding regions. The CI value, which ranges from 0 to 1, provided a statistically quantitative measure of the separability of the bruise region from its background; the higher CI, the better performance of the corresponding demodulation method. Figure 4.6. The amplitude images obtained by conventional three-phase demoduation (TPD) and three spiral phase transform (SPT) based appraoches (i.e., “2-phase SPT’ and “2-image SPT” and “1-phase SPT”) from left to right for “Gala” (top) and “Golden Decious” (bottom) apples. The defect regions correspond to the areas enclosed by red solid lines in the amplitude images obtained by “2-phase SPT”. Table 4.2 summarizes the CI values for the four demodulation approaches. The “2-phase SPT” method achieved a comparable or higher CI than TPD, followed by the “2-image SPT”, and the “1-phase SPT” gave the lowest CI values. These results are in agreement with visual inspection. The “1-phase SPT” suffered from residual patterns in the demodulated images, which was due to the fact that it required high-pass frequency-domain filtering, which often could not completely   103 eliminate DC from real-scene patterns. On the contrary, the other three methods effectively removed DC only through subtractions, and thus obtained good demodulated images. Table 4.2. Contrast indexes for the four demodulation approaches for two apples 2-phase SPT 0.244 0.216 2-image SPT 0.208 0.187 1-phase SPT 0.061 0.038 Method “Gala” “Golden Delicious” TPD 0.236 0.216 Note: TPD and SPT represent the three-phase demodulation and spiral phase transform, respectively. In addition, it was noted that, compared to TPD, all the three SPT based approaches had various degrees of demodulation artifacts around the object boundaries. These artifacts are typical issues encountered with the discrete Fourier transform applied on a finite dimension, which generates the Gibbs oscillations (also known as ripple artifacts) that occur around the signal or image boundary (Bodenschatz et al., 2015). Furthermore, the efficacy of “2-phase SPT” was tested by processing the pattern images of the two “Jonagold” apples at the spatial frequencies of 0.05, 0.10, 0.15 and 0.20 cycles/mm. As shown in Figure 4.7, the resultant amplitude images revealed the defect regions that were invisible in the DC images, and the image contrast varied with spatial frequency and bruising severity as well. These observations were in agreement with the preliminary tests reported in Chapter 3, and also confirmed the validity of “2-phase SPT”. Moreover, it was important to note that more obvious boundary artifacts occurred for the images at the low spatial frequency of 0.05 cycles/mm, compared to the other higher spatial frequencies of 0.10 cycles/mm and above. However, these artifacts had little effects on bruise detection since that they occur only around the boundaries.   104 Figure 4.7. Direct component (DC) and amplitude images obtained by spiral phase transform (SPT) based on two phase-shifted pattern images at the spatial frequencies of 0.05, 0.10, 0.15 and 0.20 cycles/mm (from left to right) for the mildly (top) and severely (bottom) bruised “Jonagold” apples. 4.2.2.2. GSO-based Demodulation Numerical Simulations The same phase-shifted sinusoidal patterns as in the simulations for SPT based approaches were used to test the performance of the GSO based approaches, expect that the spatial frequency xf was reduced to 0.10 cycles/mm and the noise variance to 0.01. Comparisons were made among the conventional TPD, “two-phase SPT”, and the two GSO approaches (i.e., GSO1 and GSO2). Table 4.3 summarizes the demodulation RMSEs. The “two-phase SPT” gave the largest RMSE, but it was still very accurate with the demodulation error (RDE) of less than 0.4%. The two GSO approaches yielded comparable accuracies, both of which were superior to SPT but inferior to TPD. In processing times, TPD was the fastest, followed by GSO1 and GSO2, while SPT was the most computationally expensive. However, implementing GSO1, GSO2 and SPT in real situations would be more efficient than TPD, since the latter needs capturing one more image. Overall, the two GSO methods achieved a compromise between imaging speed and accuracy. Table 4.3. Root mean square errors and processing times for the three demodulation approaches Method RMSE Time (s) SPT 0.442 0.061 GSO GSO1 0.1992 0.0217 GSO2 0.1951 0.0237 TPD 0.0816 0.0044 Note: RMSE denotes root-mean-square error; SPT, GSO and TPD represent spiral phase transform, Gram-Schmidt orthonormalization and three-phase demodulation, respectively.   105 Noise affects vector orthonormalization and thus the demodulation accuracy of the GSO approaches. As shown in Figure 4.8 (a), compared to SPT and TPD, the two GSO approaches behaved in almost the same manner, which yielded rapidly increased errors as the noise variance increased, indicating that they were more susceptible to noise present in the images. After application of a low-pass filter to denoise the images, the GSO approaches were still inferior to TPD and SPT at high noise levels [Figure 4.8(b)], although the resultant RMSEs had been dramatically reduced. Further reductions in the demodulation error can be made by optimizing the filtering method. Figure 4.8. Root-mean-square error (RMSE) of amplitude versus different noise levels of images without (a) and with (b) filtering preprocessing. SPT, GSO and TPD represent spiral phase transform, Gram-Schmidt orthonormalization and three-phase demodulation, respectively. Deriving the GSO approaches is built upon an important hypothesis on the phase difference, which is not equal to a multiple of integer of π, and the two phase-shifted patterns that should contain more than one complete fringe. Further investigations were then made on the effect of phase difference and spatial frequency of the two pattern images on the performance of the GSO approaches, with the noise variance fixed at 0.01. As shown in Figure 4.9(a), the demodulation RMSE remained at a low level, except that it jumped near the phase difference of 0, π and 2π. Hence, such a phase difference should be avoided, although GSO offered much freedom to the setup of phase offsets. Figure 4.9(b) shows the RMSE against the spatial frequency of patterns (i.e., number of frequency cycles of the image here). The RMSE fell dramatically when the number of fringe cycles increased from 0 to 1, and then leveled off at a low value after the number of fringe   106 cycles exceeded 5, which also confirmed the hypothesis. There was no noticeable difference between the two GSO approaches according to Figure 4.9. Figure 4.9. (a) Root mean square error (RMSE) of amplitude versus phase difference for GSO1 and GSO2 methods, and (b) RMSE of amplitude versus frequency for GSO1 and GSO2 methods, where GSO represents Gram-Schmidt orthonormalization. Experimental Validation Likewise, the GSO approaches were further tested on the bruise detection of apples. Two ‘Gala’ and ‘Jonagold’ apples, free of any visual defects, were inflicted with subsurface bruising through impact tests. Immediately after bruising, the two apples were imaged by the broadband SIRI system. Two sets of phase-shifted sinusoidal patterns (with the same phase offsets of 0, 2π/3, 4π/3) at two spatial frequencies of 0.08 and 0.15 cycles/mm, were used for sample illumination individually. Figure 4.10 shows the DC images of ‘Gala’ and ‘Jonagold’ obtained by TPD and the spiral phase function (SPF) method described in Eq. (4.16), respectively, from the phase-shifted images at the spatial frequency of 0.08 cycles/mm, which were almost the same as those at the spatial frequency of 0.15 cycles/mm (not presented). The two methods produced almost the same results, but the SPF method used only two phase-shifted image, thus proving the validity and advantage. Next, the recovered DC images by SPF were used for DC removal in amplitude retrieval by GSO methods.   107 Figure 4.10. The direct component images of the bruised apples recovered by the three-phase demodulation (TPD) (a) and the spiral phase function (SPF) method. Figure 4.11 shows the AC images obtained by different demodulation approaches. All these images clearly revealed bruise defects for the two apples and the image contrast depended on spatial frequency; and they also exhibited an enhanced image resolution and contrast, compared to the DC images. By visual inspection, the two GSO approaches and TPD yielded quite similar results, with strong image contrast; while SPT was slightly different due to the ripple artifacts occurring around the object’s boundary, which were more obvious at the lower spatial frequency, as indicated by the red arrows in Figure 4. 11. Figure 4.11. Amplitude images of bruised ‘Gala’ (a) and ‘Jonagold’ (b) apples obtained by SPT, GSO1, GSO2 and TPD methods at the spatial frequencies of 0.08 cycles/mm (top) and 0.15 cycles/mm (bottom). The red arrows and circled curves indicate the presence of the boundary artifacts and the bruised regions of apples, respectively. Furthermore, the contrast index (CI) was calculated for all the AC images using Eq. (4.29) based on partitioning the image into two parts, defect and non-defect. In calculating CI values, the defect regions for the two apples, which were the areas enclosed by the red curves in Figure 4.11, were segmented by manually thresholding the AC images obtained by TPD at the spatial frequency 0.15 cycles/mm. Table 4.4 summarizes the resultant CI values. The two GSO approaches and TPD resulted in almost the same image contrasts, which, however, were slightly inferior to SPT except   108 for ‘Gala’ at the spatial frequency of 0.08 cycles/mm. Overall, the GSO methods exhibited a strong similarity with TPD and suffered from no noticeable ripple artifacts around the boundary of the object, thus attesting the validity of GSO approaches. Table 4.4. Contrast indexes calculated from the amplitude images of apples for the four demodulation approaches. Variety Spatial Frequency (cycles/mm) 0.08 ‘Gala’ 0.15 ‘Jonagold’ 0.08 0.15 Contrast index SPT GSO1 GSO2 0.0549 0.0603 0.0603 0.0275 0.0213 0.0214 0.0569 0.0520 0.0519 0.0704 0.0661 0.0661 TPD 0.0601 0.0215 0.0523 0.0663 4.2.3. Discussion Three approaches based on SPT (i.e., “2-phase SPT”, “2-image SPT” and “1-phase SPT”) were first proposed for amplitude demodulation. The “1-phase SPT”, despite its poorest accuracy, could be useful for quick detection of certain obvious defects. The “2-image SPT” gives improved accuracy, and is beneficial to demodulating numerous images of sequential frequencies since all the pattern images can use one common uniform illumination image for DC elimination. However, when both demodulation performance and speed are desired, the “2-phase SPT’ should be the most preferred choice. Also, the “2-phase SPT” is more robust to image noise than the other two methods due to the subtraction of two phase-shifted pattern images. It is noted that the SPT based approaches suffer from boundary artifacts, especially at low spatial frequencies (e.g., less than 0.10 cycles/mm), which are due to the use of finite, discrete Fourier transform. While such issue could be avoided if continuous Fourier transform and its inverse are applied over an infinite domain, it is difficult to achieve in reality due to the infinite image size. In principle, given a sinusoidal pattern, whose DC has been subtracted or filtered out, transforming the pattern from the Fourier domain into the spatial domain is equivalent to convolution with a certain sinc function in the spatial domain, thus generating ripple artifacts   109 around the edges of the resultant image. However, the effect of ripple artifacts on defect identification would be marginal, if the image size is relatively large. A possible solution to the artifact issue is to apply a window function such as a Hamming filter (Bracewell, 2000; Chitode, 2008), to the amplitude image during image post-processing. Compared to SPT, the GSO approaches are straightforward and spatial-domain-based, given two DC-subtracted or filtered pattern images. Although SPF was used for DC retrieval in implementing GSO approaches, the resultant AC images suffered from no noticeable boundary artifacts that are observed in the SPT based demodulation. The GSO demodulation, like SPT, is also applicable to arbitrary phase shifts, but it requires that the phase difference of the two pattern images is away from an integer multiple of π. It also requires the pattern images to be processed have more than one (5 and above recommended) complete fringes. Moreover, the GSO is more sensitive to image noise, so noise suppression is needed prior to GSO processing, which can be done by using such methods as low-pass filtering, smoothing or wavelet transform. Removing the DC term from pattern images plays a critical role in the demodulation accuracy and speed. DC removal is conventionally achieved by applying a high-pass filter. This filtering operation is applicable to the images whose DC is approximately uniformly distributed throughout the region of interest and well isolated from the image component of interest in the Fourier space. However this is not common occurrence for the images of real objects (e.g., a curve-shaped apple with defects). Trial-and-error efforts are usually needed to determine suitable filters, including their type and size. Even so, a satisfactory result may not be necessarily achieved. Recently, bi- dimensional empirical mode decomposition (BEMD) has been utilized for DC removal (Trusiak et al., 2012; Zhou et al., 2012). However these methods require iteration computations for extracting individual intrinsic mode functions, which are much more computationally intensive   110 than the subtraction operation. Compared to the filtering or BEMD, the subtraction operations offer a simpler yet still effective method, which can be achieved by subtracting two phase-shifted pattern images or one uniform illumination image from one pattern images, as done in the SPT based demodulation, or subtracting a DC image recovered through the proposed SPF method, as done in the GSO demodulation. 4.3. COMPOSITE PATTERN DEMODULATION 4.3.1. Demodulation Amplitude demodulation of composite sinusoidal patterns can be achieved by generalizing the conventional phase-shifting methods into the Fourier space, in which the amplitude is recovered by solving a linear system. To illustrate the concept of the method, consider a 2-D composite sinusoidal pattern: I p N ( , ) x y  I DC  N  1 n  I n AC cos(2  n f x x  p 2   n n f y y  ) (4.31) where NI p x y is the pth phase-shifted image with N frequency components; DCI and AC ( , nI ) are the DC and AC of the nth frequency, respectively; xf n and yf n are the nth frequency along x and y directions, respectively; and p n is the pth phase offset of the nth frequency component. To simplify the notation, we will drop off the coordinates (x, y) in the following derivations. Applying 2-D FT to Eq. (4.31) yields:   ( FT I p N )  F DC  N  1 n  n F AC p exp( i  n )   n F AC N  1 n  p exp( i   n ) n F AC  1 2 I ( u f   AC n x , v  f n y )  n F AC  1 2 I ( u  AC  f n x , v  f n y ) 111 (4.32) (4.33) (4.34) where FT denotes Fourier transform defined as ( , ) F u v        ( , )exp[ 2 ( f x y  i  ux vy dxdy )]  ; i is the imaginary unit; δ is the Dirac Delta function; and u and v are 2-D frequency coordinates. Note that AC nF nF and AC are centrosymmetric. Assuming the number of phase-shifted images is P, it follows from Eq. (4.32) that: where F AX (4.35) A 1  1 exp( i  1  2 1 exp( i   1  . .    . .  P 1 exp( i    1 ) ) ) 1 exp( i   1 2 exp( i   1 . .  exp( P i  1 ) ) ) . . . . . 1 . exp( i  N 2 . exp( i  N . . . . P . exp( i  N ) ) ) exp( exp( exp(   . .  1 i  N 2 i  N ) )          P i   N P ) X  F dc F 1 ac ~ F 1 ac . . F N ac ~ F N ac                       2( N 1)1  and, F           ( FT I ( FT I . . ( FT I 1 N 2 N ) ) N N )          N 1  (4.36) (2 N 1)  (4.37) To solve the linear system, if A is not rank-deficient, one can simply invert the matrix A and pre-multiplying it to the matrix F. Then, the amplitude can be obtained by applying inverse Fourier transform as follows: n acI  2 ( IFT F n AC ) or n acI  2  n ( IFT F AC ) (4.38) where IFT denotes inverse FT, defined as ( , ) f x y        ( , )exp[ 2 ( F u v  i ux vy dudv )]  . It is noted that two amplitude images can be obtained at each frequency, which result from the positive and   112 negative parts of the same Fourier component. The two images are mirror images of each other, so either of them can be used for analysis. From Eq. (4.35), the number of phase-shifted images P should be equal to or larger than 2N+1 so as to yield a stable and unique solution. Hence the method requires multiple phase-shifted images for amplitude demodulation, which may pose a speed issue of fast image acquisition. However, the method is still advantageous relative to conventional spatial-domain phase shifting methods that need a minimum of 3N images for each frequency, especially when multiple spatial frequencies are needed or preferred. Furthermore, simplification of the method can be achieved when the phase offsets of the composite patterns are set up in a special way. If it is assumed exp( ) i   exp( ) i   , according to the Euler’s formulae, the two exponential values are equal to 1 or -1 when  is an even or odd number of π respectively, and then the two terms AC nF nF and AC in Eq. (4.32) can be combined as a whole. Then we set the phase offsets for all the frequency components to 0 in the first pattern image, and let them to be π in turn for other pattern images, such that the linear system in Eq. (4.35) is transformed into as follows: where, B 1   1  .    .   .  1   1 1  1 . . 1 . 1 1  1 . 1 F BY (4.39) . . . . . . . . 1 1 1 .  . 1 1 . . . . 1   . 1  . 1   . 1   . .  . P N .   (  1)  , and Y            F dc ~ 1 F  ac . .   ~ N F ac ~ N F ac           1  ( N )1  (4.40) 1 F ac 1  N F ac N F ac In order for Eq. (4.39) to have a solution, P should be equal to or larger than N+1. Assuming that P = N+1 and B is a full-rank matrix that is invertible, then Y can be readily obtained by   113 Y B F 1 , where B-1 is the inverse matrix of B. With the Y being known, the amplitude at each frequency can be obtained by using SPT as described in Section 4.2. I n ac    ( n F ac  )~ n F ac 2  $( n F ac  2 )~ n F ac 1 2   (4.41) With the aid of SPT, the number of required phase-shifted images is reduced to N+1 from 2N+1, which represents a great reduction in the time of image acquisition. Further reduction of pattern images is also possible by applying proper band-pass filters. Blocking filters are applied in the Fourier domain to extract each frequency component, which cut off the components outside the rectangle and leave unchanged the components within the rectangle. The extracted components are then subjected to the processing of SPT to obtain the corresponding amplitude image using Eq. (4.40). With the filtering-based method, amplitude images at different frequencies can be obtained using only one pattern image. In implementing the method, the size of filters needs to be well adjusted so as to mitigate the frequency crosstalk or overlapping between neighboring components. For convenience, ‘Phase shifting’, ‘Phase shifting with SPT’ and ‘Filtering with SPT’ are used to denote the three methods, respectively, in the following sections. 4.3.2. Numerical Simulations This study was only focused on dual-frequency and triple-frequency composite patterns. Eight-bit composite patterns were generated in Matlab according to Eq. (4.31), in which the value DCI was set to (255/2), and ACI to (255/2) and (255/3) for dual-frequency and triple-frequency patterns, respectively. The spatial frequencies of 0.05 and 0.20 cycles/mm were used for the dual-frequency pattern, and 0.05, 0.10 and 0.20 cycles/mm for the triple-frequency pattern. Those frequencies were chosen because they could achieve a reasonable range of depths for light penetration into   114 apple tissues according to preliminary studies described in Chapter 3. The number of phase-shifted images to be created depended on the demodulation method. For ‘Phase shifting’ and ‘Phase shifting with SPT’ and ‘Filtering with SPT’, five and three pattern images were used for dual- frequency demodulation, respectively, and seven and four for triple-frequency demodulation, respectively; while ‘Filtering with SPT’ only used one pattern image for both dual-frequency and triple-frequency demodulation. The phase offsets were decided following the requirement of the corresponding methods as described in Section 4.3.1. For ‘Phase shifting’, the phase offsets were arbitrarily set as long as the matrix A in Eq. (4.35) was not rank-deficient (i.e., a matrix does not have a full column or row rank) to ensure an accurate and stable amplitude solution; while for ‘Phase shifting with SPT’, a phase difference of π relative to the first pattern, was successively set up at each frequency component. ‘Filtering with SPT’ was conducted on the first composite pattern image of ‘Phase shifting with SPT’. Moreover, two pattern orientations, parallel and orthogonal, were considered in both dual-frequency and triple-frequency patterns, leading to fringe and grid composite patterns, respectively, as shown in Figure 4.12. In the triple-frequency grid pattern, the two frequency components at 0.05 and 0.20 cycles/mm were set in the same direction, while the third one at 0.10 cycles/mm was in the direction perpendicular to them. Before the demodulation, these raw patterns were corrupted by Gaussian additive noise (with the SNR of 30 dB). Figure 4.12. Simulation composite patterns: dual-frequency fringe pattern, dual-frequency grid pattern, triple- frequency fringe and triple-frequency grid pattern (from left to right). Table 4.5 summarizes the relative demodulation error (RDEs). Overall, all the methods gave very small RDEs, mostly less than 1%, which suggested their potential utility for processing real   115 triple-frequency patterns Dual-frequency Fringe pattern Grid pattern 0.47 0.60 0.83 0.47 0.61 0.67 Triple-frequency Fringe pattern Grid pattern 0.82 0.77 1.28 0.80 0.76 1.12 Demodulation methods ‘Phase shifting’ ‘Phase shifting with SPT’ ‘Filtering with SPT’ Note: SPT denotes spiral phase transform. composite patterns. The RDE increased consistently from the dual-frequency pattern to triple- frequency pattern. That may be explained by the fact that the amplitude at each frequency diminished with the addition of more frequency components. Table 4.5. Relative demodulation errors (in percent) for three demodulation approaches for dual-frequency and For the dual-frequency pattern, ‘Phase shifting’ achieved the best accuracy, followed by ‘Phase shifting with SPT’ and then ‘Filtering with SPT’; and for the triple-frequency pattern, ‘Phase shifting with SPT’ was slightly better than ‘Phase shifting’ that was superior to ‘Filtering with SPT’. ‘Filtering with SPT’ was the worst for both situations. ‘Phase shifting’ and ‘Phase shifting with SPT’ were more accurate since they gave analytical solutions for amplitude, while ‘Filtering with SPT’ obtained approximate results due to the imperfection of the filtering processing that glosses over the interactions between different frequency components. Furthermore, relative to fringe patterns, grid patterns yielded improved accuracy overall, especially for ‘Filtering with SPT’. In a grid pattern, the distances among the frequency components in the Fourier space were enlarged, which had the effect of alleviating the issue of frequency overlapping and thus enabled resolving each frequency component. 4.3.3. Experiment on Bruise Detection As in demodulating single-frequency patterns, experiments were performed on bruise detection of apples using the broadband SIRI system to evaluate the composite pattern demodulation approaches. Two ‘Golden Delicious’ and ‘Gala’ apples, free of visual defects, were inflicted with subsurface bruising through impact tests (with the impact energy of ~0.8 J). Immediately after   116 bruising, the apples were imaged by the broadband SIRI system. For comparison, conventional single-frequency sinusoidal patterns at the spatial frequencies of 0.50, 0.10 and 0.20 cycles/mm with a relative phase shift by 2π/3 were also used in addition to their composite patterns. Prior to demodulation processing, the acquired pattern images were first subjected to flat-field corrections to correct the non-uniformity of source illumination and high-pass Gaussian filtering for noise suppression. Likewise, the contrast index (CI) of the resultant AC images were calculated to compare the performance of the different demodulation approaches. 4.3.3.1. Conventional Single-frequency Demodulation The single-frequency patterns were demodulated using the conventional TPD. As shown in Figure 4.13, the bruises of the two apples could be readily identified in the AC images, with varying image contrast (and resolution) depending on spatial frequency, but not in the DC images that corresponded to uniform illumination. Compared to the raw AC images (Figure 4.13, mid row), the ratio images (Figure 4.13, bottom row), obtained by dividing the AC image by the DC image of the same fruit, greatly improved image contrast and the uniformity of background intensity as well. Bruises, whose areas are enclosed by the red curves in Figure 4.13, were segmented by manually thresholding the ratio images at the spatial frequency of 0.10 cycles/mm, based on which the CIs were calculated for all the ratio images. The resultant CIs at the spatial frequencies of 0.05, 0.10 and 0.20 cycles/mm were 0.412, 0.415 and 0.273, respectively, for ‘Golden Delicious’, and 0.335, 0.337 and 0.230, respectively, for ‘Gala’. In the following discussion, only the ratio images are presented and used for comparison.   117 DC DC AC_F0.05 AC_F0.10 AC_F0.20 AC_F0.05 AC_F0.10 AC_F0.20 AC_F0.05 AC_F0.10 AC_F0.20 AC_F0.05 AC_F0.10 AC_F0.20 (a) (b) Figure 4.13. Direct component (DC, top row), amplitude component (AC, mid row) and ratio of AC to DC (bottom row) images of ‘Golden Delicious’ (a, left) and ‘Gala’ (b, right) apples, obtained by using single-frequency patterns at the spatial frequencies of 0.05, 0.10, and 0.20 cycles/mm, respectively. 4.3.3.2. Dual-frequency Demodulation Figure 4.14 shows the amplitude images obtained by ‘Phase shifting’, and the CIs of these images are summarized in Table 4.6. Compared to the single-frequency demodulation, ‘Phase shifting’ resulted in decreased CIs at the spatial frequency of 0.20 cycles/mm for both fringe and grid composite patterns, but increased CIs at 0.05 cycles/mm for the grid composite patterns. In comparison to the fringe composite patterns, the grid composite patterns gave higher CIs at 0.05 cycles/mm but lower CIs at 0.20 cycles/mm; however, from visual inspection, the former led to noticeable image artifacts, especially for the ‘Gala’ apple. These artifacts were mainly due to the incomplete separation of different frequency components for the fringe composite patterns. Hence, the grid composite patterns were more preferred than the fringe composite patterns overall.   118 AC_F0.05 AC_F0.20 AC_F0.05 AC_F0.20 AC_F0.05 AC_F0.20 AC_F0.05 AC_F0.20 (a) (b) Figure 4.14. Amplitude images of ‘Gold Delicious’ (a) and ‘Gala’ (b) apples obtained by ‘Phase shifting’ for fringe composite patterns (top) and grid composite patterns (bottom) with dual spatial frequencies of 0.05 and 0.20 cycles/mm. Figure 4.15 shows the demodulation results by ‘Phase shifting with SPT’. Overall, the resultant images exhibited reduced image contrast compared to those obtained by ‘Phase shifting’, except that the images of ‘Gala’ at 0.20 cycles/mm had relatively higher CIs (Table 4.6). The AC images at the low spatial frequency of 0.05 cycles/mm suffered from obvious ripple artifacts near the object edges, which were due to the use of SPT. Despite the drawback, ‘Phase shifting with SPT’ saved almost half of the images needed for amplitude demodulation of composite patterns relative to ‘Phase shifting’, and the bruise defect remained discernable in all the images. Likewise, the grid composite patterns at 0.20 cycles/mm, resulted in better images with reduced residual stripes compared to fringe composite patterns.   119 AC_F0.05 AC_F0.20 AC_F0.05 AC_F0.20 AC_F0.05 AC_F0.20 AC_F0.05 AC_F0.20 (a) (b) Figure 4.15. Amplitude images of ‘Golden Delicious’ (a) and ‘Gala’ (b) apples obtained by ‘Phase shifting with SPT’ using fringe composite patterns (top) and grid composite patterns (bottom) with dual spatial frequencies of 0.05 and 0.20 cycles/mm. The amplitude images obtained by ‘Filtering with SPT’ are shown Figure 4.16. Compared to ‘Phase shifting with SPT’, the approach further reduced the image contrast, making it difficult to ascertain the bruise defect. The deteriorated image contrast was because of the imperfection of the filtering processing, to which grid composite patterns were less susceptible and gave higher CIs than fringe composite patterns (Table 4.6). Boundary artifacts also appeared in the images at the spatial frequency of 0.05 cycles/mm, which were associated with the side effect of SPT as well as (band-pass) filters applied. AC_F0.05 AC_F0.20 AC_F0.05 AC_F0.20 AC_F0.05 AC_F0.20 AC_F0.05 AC_F0.20 (a) (b) Figure 4.16. Amplitude images of ‘Golden Delicious’ (a) and ‘Gala’ apples (b) obtained by ‘Filtering with SPT’ using fringe composite patterns (top) and grid composite patterns (bottom) with dual spatial frequencies of 0.05 and 0.20 cycles/mm.       120 Table 4.6. Contrast indexes (CIs) for the three demodulation methods for demodulating dual-frequency composite Demodulation Method ‘Phase shifting’ ‘Phase shifting with SPT’ ‘Filtering with SPT’ patterns ‘Golden Delicious’ Fringe Pattern F0.05 F0.20 0.254 0.417 0.232 0.208 0.076 0.154 Grid Pattern F0.05 F0.20 0.217 0.423 0.217 0.211 0.101 0.211 ‘Gala’ Fringe Pattern F0.05 F0.20 F0.05 0.337 0.200 0.097 0.268 0.189 0.068 0.182 0.219 0.146 Grid Pattern F0.20 0.150 0.183 0.147 Note: SPT denotes spiral phase transform; F0.05 and F0.20 denote the spatial frequencies of 0.05 and 0.20 cycles/mm, respectively. 4.3.3.3. Triple-frequency Demodulation Figure 4.17 shows the demodulated images by ‘Phase shifting’. All resultant images exhibited a consistent decrease in image contrast relative to single-frequency demodulation. Compared to ‘Phase-shifting’ in the dual-frequency demodulation cases, the low-frequency pattern at 0.05 cycles/mm had lower CIs, but the opposite happened to the high-frequency patterns at 0.20 cycles/mm (Table 4.7). The grid composite patterns produced CIs higher than or comparable to those by fringe composite patterns, except at 0.10 cycles/mm for the ‘Gala’ apple; and the latter had more obvious pattern residuals. The new amplitude images at 0.10 cycles/mm could also locate the defect regions with strong image contrast. Additionally, although to a lesser degree, the issue of pattern residuals seemed to occur in the grid composite patterns as well. AC_F0.05 AC_F0.10 AC_F0.20 AC_F0.05 AC_F0.10 AC_F0.20 AC_F0.05 AC_F0.10 AC_F0.20 AC_F0.05 AC_F0.10 AC_F0.20 (a) (b) Figure 4.17. Amplitude images of ‘Golden Delicious’ (a) and ‘Gala’ apples, obtained by ‘Phase shifting’ using fringe composite patterns (top) and grid composite patterns (bottom) with triple frequencies of 0.05, 0.10 and 0.20 cycles/mm. Figure 4.18 shows the demodulated images by ‘Phase shifting with SPT’. Similarly, all the resultant images had decreased CIs compared to those in the single-frequency demodulation; and   121 compared to dual-frequency demodulation, the amplitude images at 0.05 and 0.20/cycles exhibited decreased and increased CIs, respectively. The fringe composite patterns suffered from more evident pattern residuals than the grid composite patterns, and the edge artifacts remained in the images at 0.05 cycles/mm, leading to the dramatically diminished visibility of the bruise defect for both apples. Moreover, ‘Phase shifting’, although having achieved strong image contrast for all the frequencies, did not consistently outperform ‘Phase shifting with SPT’; in comparison to the former, the latter produced slightly improved CIs for the grid composite patterns at 0.10 and 0.20 cycles/mm for the ‘Gala’ apple, and comparable results for the same grid patterns for the ‘Golden Delicious’ (Table 4.7). AC_F0.05 AC_F0.10 AC_F0.20 AC_F0.05 AC_F0.10 AC_F0.20 AC_F0.05 AC_F0.10 AC_F0.20 AC_F0.05 AC_F0.10 AC_F0.20 (a) (b) Figure 4.18. Amplitude images of ‘Golden Delicious’ (a) and ‘Gala’ apples (b) obtained by ‘Phase shifting with SPT’ using fringe composite patterns (top) and grid composite patterns (bottom) with triple frequencies of 0.05, 0.10 and 0.20 cycles/mm. Figure 4.19 shows the results by ‘Filtering with SPT’. The approach resulted in consistently degraded image contrast, compared to ‘Phase shifting’, ‘Phase shifting with SPT’, and that in the dual-frequency demodulation. The amplitude images at the spatial frequency of 0.05 cycles/mm, were no longer useful due to extremely low contrast. More frequencies increased the possibility of interactions or cross-talks among different frequency components, and thus the difficulty of separating them through simple filtering operations.   122 AC_F0.05 AC_F0.10 AC_F0.20 AC_F0.05 AC_F0.10 AC_F0.20 AC_F0.05 AC_F0.10 AC_F0.20 AC_F0.05 AC_F0.10 AC_F0.20 (a) (b) Figure 4.19. Amplitude images of ‘Golden Delicious’ (a) and ‘Gala’ apples (b) obtained by ‘Filtering with SPT’ using fringe composite patterns (top) and grid composite patterns (bottom) with triple frequencies of 0.05, 0.10 and 0.20 cycles/mm. Table 4.7. Contrast indexes (CIs) for different methods of demodulating triple-frequency composite patterns ‘Golden Delicious’ A B C Fringe Pattern Grid Pattern F0.05 0.384 0.122 0.045 F0.10 0.362 0.279 0.028 F0.20 0.262 0.243 0.143 F0.05 0.406 0.124 0.072 F0.10 0.361 0.353 0.258 F0.20 0.267 0.269 0.146 “Gala” Fringe Pattern Grid Pattern F0.05 0.286 0.117 0.032 F0.10 0.304 0.264 0.024 F0.20 0.204 0.223 0.057 F0.05 0.322 0.119 0.072 F0.10 0.224 0.272 1.260 F0.20 0.219 0.227 0.099 Note: A, B and C represent ‘Phase shifting’, ‘Phase shifting with SPT’ and ‘Filtering with SPT’, respectively; SPT denotes spiral phase transform; F0.05, F0.10 and F0.20 represent the spatial frequencies of 0.05, 0.10 and 0.20 cycles/mm, respectively. 4.3.4. Discussion As an initial exploration of demodulating composite sinusoidal patterns in SIRI for food quality detection, three approaches, i.e., ‘Phase shifting’, ‘Phase shifting with SPT’ and ‘Filtering with SPT’, were proposed based on Fourier transform and its inverse to retrieve amplitude images. ‘Phase shifting’ was implemented by solving a linear system composed of multiple phase-shifted pattern images; ‘Phase shifting with SPT’ simplified the ‘Phase shifting’ by using SPT, which saved almost half of the images needed; and ‘Filtering with SPT’ retrieved amplitude images by applying SPT to the frequency components that were isolated by band-pass filters. All the proposed methods were viable in theory due to small relative demodulation errors achieved. For bruise detection of apples, however, the use of composite patterns, in comparison to conventional single-frequency patterns, tended to degrade the image contrast, to a certain degree depending on demodulation approach, number of frequency components and pattern orientation. Among the three approaches, ‘Phase shifting’ and ‘Phase shifting with SPT’ could obtain decent   123 results with strong image contrast, which greatly outperformed ‘Filtering with SPT’, as the first two approaches derived amplitude analytically, while the last one only gave approximated results. On the other hand, ‘Filtering with SPT’, although needing further improvement, would be the most efficient in image acquisition because of only one image required. These approaches can be used for demodulating general composite sinusoidal patterns, but their performance would deteriorate when more frequency components are involved because of the shrunk amplitude at each frequency and the increased frequency crosstalk. In addition, grid composite patterns generally gave better- quality images with less pattern residuals than fringe composite patterns, since the former had larger distances between different frequency components in the Fourier space, thus alleviating the issue of frequency crosstalk. Hence, the dual-frequency grid composite patterns, combined with ‘Phase shifting with SPT’ or ‘Phase shifting’ provided a relatively practical choice in terms of imaging speed and accuracy in the framework of this study. 4.4. CONCLUSIONS Conventional phase shifting approaches requires acquisition of at least three phase-shifted pattern images, which limit the speed of image acquisition for SIRI. SPT and GSO are proven as viable solutions for fast demodulation by using fewer pattern images. SPT, which acts as a 2-D quadrature transform operator, can be applied flexibly to two phase-shifted pattern images, one pattern image plus one uniform illumination image or a single pattern image, but it produces the most satisfactory result when used for two phase-shifted pattern images, in terms of demodulation accuracy and visual contrast of resultant images. Moreover, the approach requires no prior knowledge of phase offsets of illumination patterns, which is practically useful for processing arbitrarily phase-shifted pattern images. GSO provides a simple, fast, and spatial-domain based approach to demodulating two phase-shifted pattern images. In comparison to SPT, GSO suffers from no obvious ripple   124 artifacts caused by finite, discrete Fourier transform, but it is sensitive to image noise, requires more than one complete fringe present in the patterns and the phase difference that should be away from a multiple integer of π. For the two approaches, DC removal is a critical preprocessing step for successful amplitude demodulation, which can be effectively done by subtracting another pattern or uniform illumination image, or the DC image separately recovered from two phase- shifted pattern images. Composite sinusoidal patterns, compared to conventional single-frequency sinusoidal patterns, provide the possibility for multi-frequency amplitude in an efficient way. New approaches are proposed for demodulating those patterns by solving a linear system of phase-shifted pattern images in the Fourier domain, which greatly reduces image acquisitions required in the single- frequency demodulation, at the cost of decreased image contrast. In addition, band-pass filtering enables the demodulation using a single pattern image, but results in greatly deteriorated image quality. More frequency components present in the composite patterns would generally deteriorate the performance of these approaches, and grid composite patterns are superior over the fringe ones due to reduced interactions between different frequency components. Since image demodulation is the first critical step in using SIRI for fruit defect detection, these new demodulation approaches provide valuable tools for expanding the applicability and capacity of SIRI, and will help move forward the technique for fast, practical imaging applications.   125 5. THREE-DIMENSIONAL GEOMETRY RECONSTRUCTION 5.1. INTRODUCTION Three-dimensional (3-D) geometry reconstruction allows acquiring surface range (height or depth) information of objects, and it is useful for non-destructive inspection of agricultural commodities by imaging techniques. For fruits such as apples, 3-D geometric information provides a means for determining fruit shape or size (Danckaers et al., 2015; Moreda et al., 2009), which is one of the most important quality metrics for apple sorting and grading. Currently, commercial imaging inspection systems for defect sorting are still faced with the challenge to recognize stem/calyx regions that are normal parts of fruit in the image, which, otherwise, may cause confusion with true defects. Since stem/calyx regions are located around the concave area of fruit, they can be readily recognized through the reconstructed 3-D surface contour of fruit (Jiang et al., 2009; Zhu et al., 2007). In addition, the 3-D geometry of an object can be also used for image corrections for intensity distortions due to the fruit curvature (Gomez-Sanchis et al., 2008b), thus helping enhance the detection of fruit defect (Gomez-Sanchis et al., 2008a). Numerous optical or imaging techniques have been developed for 3-D shape measurement, among which the most prevalent are time-of-flight (TOF), stereo vision and structured lighting (Chen et al., 2000; Seitz et al., 2006; Zhang, 2016). TOF uses a single camera to measure the distance between the object and the light source by detecting the time delay of modulated light pulses, which has been commercially used, with measurement errors ranging from several millimeters to centimeters (Blais, 2004). Stereo vision is a multi-camera based technique, which reconstructs the 3-D shape from a number of images acquired from different viewing angles based on the triangulation principle. In stereo vision, establishing correspondences between two stereo images, however, is extremely difficult, which restricts the ability of this technique to reconstruct   126 objects with smooth surface. Structured lighting is similar to stereo vison, except that one camera is replaced with a projector, which shines structured or patterned light to an object, and the 3-D geometry of the object is recovered by analyzing how the pattern is deformed by the object. Thanks to the active light pattern projection, structured lighting technique obviates the need in stereo vision for the surface texture variation of an object to find corresponding points, while enabling accurate 3-D reconstruction, which is closely related to its pattern coding strategies (Salvi et al., 2010; Salvi et al., 2004). In the past two decades, structured lighting technique, benefiting from advancements in digital imaging and light projecting technologies, has become increasingly popular in 3-D shape measurement in a wide range of fields, such as manufacturing, computer science, entertainment, et al. (Bell et al., 2016). In recent years, structured lighting technique has spread rapidly into the agricultural field for 3-D imaging applications (Rosell-Polo et al., 2015; Vazquez-Arellano et al., 2016), ranging from plant phenotyping (Chene et al., 2012; Jiang et al., 2016; Paulus et al., 2014) and image segmentation of plant leaves (Xia et al., 2015) to monitoring of the body conditions of livestock (Rosell-Polo et al., 2015). In most of these applications, 3-D images were obtained by using a low- cost consumer-grade depth camera, e.g., Microsoft Kinect. The Kinect is a structured lighting technique using a random pattern coding strategy, in which an infrared laser, combined with a diffractive optical element, is used to generate a unique pattern consisting of a large number of random dots (Zhang, 2012), and 3-D information is then derived from the captured patterns by triangulation. This technique, however, is sensitive to noise and has limited spatial resolution. Among the existing structured lighting techniques, a digital fringe projection (DFP) technique, originated from laser interferometry, which uses sinusoidally-modulated fringe patterns (i.e., sinusoidal or fringe patterns for brevity), has been widely studied in 3-D shape measurement in   127 last decade (Lu et al., 2017; Zhang, 2016), due to its ability to achieve high-precision measurements at the pixel level. In DFP, height or depth information is computed using a phase shifting method from a phase or phase difference map, rather than intensity images that are commonly used in many other structured lighting techniques, which is typically more robust to noise. Since the computation is performed for each pixel, a dense 3-D reconstruction can be obtained with high measurement accuracy, if calibrations have been properly done to establish the phase-to-height relationship (Vo et al., 2012). Despite its great potential, the DFP technique, however, has yet to be explored in agricultural applications. By using the demodulation approaches as described in Chapter 4, SIRI is able to obtain DC and AC images, both of which are intensity images associated with light attenuation due to sample absorption and scattering. Due to the use of sinusoidal patterns for sample illumination as in DFP, SIRI can, additionally, provide phase images that encode the geometry information of samples, and the phase images can be exploited to reconstruct the 3-D geometry of the sample. Hence, this study was aimed at exploring the feasibility of SIRI for 3-D geometry reconstruction of apples based on phase analysis, so as to expand the technique for fruit quality evaluation. Specific objectives of this work were two-fold: (1) to develop phase analysis techniques, including phase demodulation (also known as phase retrieval), phase unwrapping and system calibrations, for 3-D shape measurement by SIRI; (2) to test and evaluate the techniques through the reconstruction of 3-D surface contours for reference and apple samples. 5.2. PHASE ANALYSIS FOR SURFACE PROFIELING 5.2.1. Phase-height Relationship Formulation A 2-D sinusoidal pattern can be mathematically described as follows:   ( , ) I x y  I ( , ) x y  I AC DC cos[ ( , ) x y  ]  (5.1) 128 where ( , )x y are the spatial coordinates, DC( , x y and AC( , ) I I x y represent the intensities of DC ) and AC, respectively, and ( , x y ) is the phase map, which is usually linearly modulated as  ( , ) 2 x y f  where x xf denotes the spatial frequency along the x-axis, and δ is the phase offset. Now let us look at how the phase map can be used for height measurement in SIRI. The SIRI system consists of a digital micro-mirror device (DMD) based projector and a charge-coupled device (CCD) camera, whose geometry is schematically shown in Figure 5.1.  A C (B )  Figure 5.1. Geometry of the structured-illumination reflectance imaging (SIRI) system for height measurement. The camera is vertically set up at a distance H from the horizontal reference plane, while the projector is tilted at an angle 2 relative to the vertical axis. Points O1 and O2, spaced apart at a distance d, are the optical centers of the projector and camera lenses, and the line passing through the two points intersects the horizontal axis at an angle 1. Then, given a point C on the sample, its height can be readily calculated as follows: h  AB AB [sin( )tan( d   2  1 ) cos( )]   1 H (5.2) where AB denotes the length of the line segment AB. Simple manipulation of Eq. (5.2) yields as follows:   129 1 h  c 1  c 2 AB where 1c and 2c are two constants, equal to 1 H and d (5.3) ) cos( )]  1 H , respectively. [sin( )tan(   1 2 Further, it is noted that AB is related to the phase difference between points A and B in the reference plane. Since O2, C and A points are on the same light ray emitting from the projector, the point A in the reference plane has the same phase values as the point C in the sample, which corresponds to the phase value of the point C in the image plane; the phase value of the point B in the reference plane corresponds to that of the point B (note that C and B correspond to the same image pixel). Thus, AB is actually dependent on the phase difference between points C and B , the same pixels in the image plane for the object and reference plane, respectively: AB     B    C   A B     (5.4) where A and B represent phase values of points A and B in the reference plane, respectively; C and B represent phase values of points C and B in the image plane, corresponding to points C and B in the real scene, respectively. With Eqs. (5.3) and (5.4), a general relationship between height and phase can be obtained as follows: 1 h  c 1  c 3   or h       c 1 c 3 (5.5) where 3c is a constant that incorporates 2c . And, if H is far (e.g., more than ten times) larger than the object height, Eq. (5.5) can be further reduced to a proportional relationship as follows: h c  3 (5.6) Eq. (5.5) or Eq. (5.6) provides a simple reference plane-based method for measuring the height of an object by determining the phase difference between the phase map of the object and the   130 reference plane. To do this would require two steps: 1) obtaining the phase maps for the reference plane and the object, which is to be achieved by phase demodulation and unwrapping, and 2) calibrating the relationship between phase difference and height, both of which are detailed in the following subsections. 5.2.2. Phase Demodulation and Unwrapping As in amplitude demodulation, phase-shifting approaches are the most widely used for phase demodulation due to their simplicity and high accuracy. Conventionally, a three-phase based approach is used for phase demodulation, Given the phase-shifted sinusoidal patterns with the phase offset = -2π/3, 0 and 2π/3, as used in Chapter 3, phase demodulation can be achieved by the following equation:  ( , x y )  arctan[ 3 ( , I x y 1 2 ( , ) I x y  2 ( , ) I x y  3 ( , ) I x y  1 ) ( , I x y 3 ] ) (5.7) From Chapter 4, with the use of SPT, two DC-free sinusoidal patterns that are in quadrature can be obtained from two generally phase-shifted pattern images, which can be also used for phase demodulation. Given two phase-shifted sinusoidal patterns as described in Eqs. (4.10) and (4.11) and their difference described in Eq. (4.12), the phase image can be obtained as follows:  ( , x y )  arctan[ I $( i diff I diff )  ]  (5.8) where $ denotes the operator for SPT, and diffI is the difference of two phase-shifted images, and  is a constant representing the average phase offset of the two phase-shifted patterns. As a proof of concept, this study was mainly focused on the conventional demodulation approach. It is important to point out that the arctangent expression in Eqs. (5.7) and (5.8), however, only provides principal phase values that are restricted to the range of –π to π, with discontinuities or jumps near 2π, appearing in the phase distribution, which is known as wrapped phases. The   131 wrapped phase values need be restored to the underlying continuous phase by detecting and removing the 2π jumps, which is referred to as phase unwrapping, before height information can be deduced from the phase distribution. The relationship between the unwrapped phase, ( , x y ) , and the wrapped phase can be formulated as: ( , x y  )   ( , x y ) 2   ( , k x y ) (5.9) where k(x, y) is an integer number at a given pixel. Phase unwrapping is to determine the k value, which is a difficult problem due to multiple sources for discontinuities in the wrapped phase map from real fringe patterns, such as noise and phase aliasing (Ghiglia and Pritt, 1998). Currently, a plethora of algorithms have been developed for phase unwrapping (Burton, 2014; Ghiglia and Pritt, 1998; Judge and Bryanston-Cross, 1994). These algorithms basically fall into two categories: spatial and temporal phase unwrapping (Zhang, 2016). Spatial phase unwrapping, to which most of the existing algorithms belong, is based on the spatial comparison of phase values at neighboring pixels to detect 2π jumps; while temporal phase unwrapping is performed in the temporal domain, which requires a sequence of phase maps at different spatial frequencies or wavelengths and thus is infeasible for high-speed applications. Since processing speed is critical for quality inspection of food products, only spatial phase unwrapping algorithms that are fast in execution and robust in performance are considered in this study. The simplest spatial phase unwrapping algorithm is the direct integration of the difference of wrapped phases, which was proposed by Itoh (Itoh, 1982). This algorithm, however, only works in extremely simple cases where the data is well sampled (i.e., no phase aliasing) and noise free, and thus it is of little practical value. Inspired by the Itoh’s work, researchers have been seeking to develop more robust and effective algorithms, among which quality-guided phase unwrapping is straightforward and usually able to yield desired results in a fast way (Su and Chen, 2004; Zappa   132 and Busca, 2008). The quality-guided phase unwrapping uses a quality map, an array of values defining the quality or goodness of each phase value, to guide the phase-unwrapping path, which starts from high-quality pixels to low-quality pixels until all pixels are unwrapped (Ghiglia and Pritt, 1998). The quality map that can be constructed using different criteria is vital to the success of phase unwrapping. Herraez et al. (2002) proposed using a reliability function, derived from second differences, for guiding phase unwrapping, which is performed following a non-continuous path based on sorting by reliability. In this approach, the reliability of the pixel is defined as the reciprocal of the second difference at this pixel with respect to its eight neighbors in a 3×3 pixel block. Given a wrapped phase value ( , )x y , the reliability of the pixel is calculated as follows: where D i ( , ) j  2 H i ( , ) j V i ( , ) j D i ( , ) j D i ( , ) j    2 2 1 2 2 H i ( , ) [ ( j W i    1, ) j   ( , )] i [ ( , ) j W i j   V i ( , ) [ ( , j W i   j 1)    ( , )] i [ ( , ) j W i j    ( i   1, )] j  ( , i  j  1)] [ ( 1, 1( , ) D i j W i    j [ ( 1, 2( , ) D i j W i    j 1)    [ ( , ) ( , )] i j W i j   1)    [ ( , ) ( , )] i j W i j   ( i    1, j  1)]   ( 1, i  j  1)] (5.10) (5.11) (5.12) (5.13) (5.14) where W denotes the wrapping operator, i.e., modulo 2π. With the reliability of each pixel, the reliability of an edge is defined as the summation of the reliabilities of two adjacent pixels. Then phase unwrapping is performed onto the edges, following a path defined by their reliabilities, from high-reliability edges to low-reliability ones, until all the pixels have been unwrapped. This method has been proved robust and efficient in practice, and hence was adopted in this study. More detailed implementation of the method is given in Herraez et al. (2002).   133 5.2.3. Absolute Phase Map The phase unwrapping methods described above only give a relative phase map, which is relative to a certain point that is unwrapped first during phase unwrapping, which may vary with different unwrapped phase maps. The relative phase map needs to be converted to an absolute one that is relative to a common, predefined point in the phase map, before it is converted to a height map by calibration. A typical way to obtain an absolute phase map is to create a distinctive marker on the raw fringe pattern, serving as a common reference point (Lu et al., 2017). Then, the absolute phase map can be obtained by simply subtracting the phase of the marker from the relative phase. Designing a proper marker and identifying it before phase unwrapping, however, is not a trivial task. This study proposed a simpler method based on fringe pattern normalization and automatic fringe detection for obtaining the absolute phase. First, two intensity images, DC and AC, are obtained using the demodulation approaches as described in Chapter 4 or 3 (depending on the pattern images available). Then, a normalized fringe pattern, which is referred to as a DC-free and amplitude-normalized pattern (Quiroga and Servin, 2003), is obtained as follows: NI  ( I  I ) I AC DC (5.15) where I can be any of acquired phase-shifted pattern images. In this study, only the first pattern was used for further analysis. A normalized fringe pattern, as shown in Figure 5.2, exhibits significantly enhanced fringe contrast compared to the raw pattern. Based on the normalized fringe pattern, the center line of its left-most white fringe stripe is then detected, as shown in Figure 5.2(b), which can be readily achieved either by examining local maxima row by row as done in this study or by thresholding   134 the fringe pattern image followed by morphological erosion. Thereafter, an absolute phase map is obtained as follows:  abs ( , ) x y  ( , ) x y    ref ( , ) x y (5.16) where ref ( , x y ) is the average (a constant value) of phase values in the detected line in the relative phase map obtained from phase unwrapping. It should be noted that this method requires the left- most white fringe to be kept in the camera’s field of view. Figure 5.2. Examples of a raw fringe pattern (a) and a normalized fringe pattern (b) where the red line represents the detected center line of the left-most white fringe stripe. 5.2.4. Phase-to-height Calibration Phase-to-height calibration is to establish the mapping relationship between the phase difference and height, which is also frequently referred to as out-of-plane calibration since it is performed along the vertical axis. Calibration of the phase-to-height relationship consists in estimating the parameters relating to the system geometry, as given in Figure 5.1. Direct, accurate measurements of these parameters, however, are tedious and often difficult as well. To circumvent the problem, researchers have developed a number of calibration techniques to map the relationship between the phase difference and height, without explicitly determining these parameters (Guo et al., 2005; Huang et al., 2010; Jia et al., 2007; Liu et al., 2003; Merner et al., 2013). Least-squares (LS) curve fitting provides a straightforward and effective method (Guo et al., 2005; Huang et al., 2010), which can be performed either in a linear manner given in Eq. (5.6), or in a nonlinear way according to Eq. (5.5). The latter is preferred since it will yield more accurate   135 results. It should be noted that Eqs. (5.5) and (5.6) are not perfect representation of the true phase- to-height relationship, since they do not account for the optical parameters, such as camera focal length and lens distortion, which may affect the calibration accuracy. Another way is to perform linear LS curve fitting to a polynomial, which is often utilized to approximate the phase-to-height relationship (Merner et al., 2013; Saldner and Huntley, 1997; Sitnik et al., 2002; Xu et al., 2011b), or to perform simple linear interpolation without involving LS calculations. In this study, nonlinear LS curve fitting, polynomial curve fitting and linear interpolation were examined for phase-to- height calibrations. To establish the phase-to-height mapping relationship, a series of phase maps with known ground-truth height information need to be acquired, which can be done by imaging a flat reference plane attached to a vertical translation stage. As shown in Figure 5.3, the reference plane, also called calibration plane, is initially positioned at height z = 0, and its phase map is first measured. Next, the reference plane is moved up with a height increment ∆h, and the corresponding phase map is measured again. The procedure is repeated until the desired number of phase maps are required. The increment and number of phase maps should be properly determined, depending on the accuracy requirement and the size of objects to be measured. With all phase maps acquired, the calibration is performed at the pixel level by using aforementioned methods. Figure 5.3. Principle of the phase-to-height calibration.   136 5.3. IN-PLANE CALIBRATION In-plane calibrations are to convert the image’s pixel coordinates to the dimensions with physical units (e.g., mm), to obtain complete coordinates (x, y, z) in the real world, which can be done by using camera calibration techniques (Shapiro and Stockman, 2001; Tsai, 1987; Zhang, 2000). The camera calibrations involve estimating the camera’s intrinsic parameters, such as focal length, principal point, distortion, etc., and the external parameters related with affine transformations (e.g., translation, rotation and scaling), which, although accurate, however, require rigorous and complicated procedures. In this study, given the geometry of the SIRI imaging system as shown in Figure 5.1, a simple perspective imaging (Hartley and Zisserman, 2003) based method was proposed for in-plane calibration, which was able to achieve acceptable accuracy. Given a perspective imaging model, as schematically shown in Figure 5.4, and that the x-y plane of the world or workbench coordinate system overlaps with the reference plane as shown in Figure 5.1, a point P(xw, yw, zw) in a calibration object in the world coordinate system and its corresponding point P'(vi, ui) in the image plane are mathematically related as follows: x w  w H z   ( v i  v 0 ) y w  w H z   ( u i  u 0 ) (5.17) (5.18) where H and  denote the distance between the lens center and the reference plane, and the lens focal length, respectively; zw is the distance from the reference plane; and (v0, u0) is the coordinates of the image principal point (i.e., the intersection of the optical axis with the image plane).   137 Figure 5.4. Perspective imaging model relating the world and image coordinate systems, where ξ and H denote the focal length of the lens and the distance between the lens center and the origin of the world coordinate system, respectively. The front image u-v plane is geometrically equivalent to the camera’s actual image plane, which is assumed to be parallel to the x-y plane of the world coordinate system. Thus, the in-plane calibration is reduced to estimating the four parameters of H, , v0, and u0. Direct estimation of these parameters, however, entail intensive iterative computations. A simpler alternative is to reformulate the two equations into a linear system. Let us assume two points P1 (x1, y1, z1) and P2 (x2, y2, z2) in the world coordinate system correspond to the two image data points P1' (v1, u1) and P2' (v2, u2). It follows that, x 1  x 2  ( v 1  ) v k 2 1  ( v z 1 1  ) v z k 2 2 2  ( z 2  ) z k 1 3 y 1  y 2  ( u 1  ) u k 2 1  ( u z 1 1  ) u z k 2 2 2  ( z 2  ) z k 1 4 The two equations can be expressed in a matrix form as follows: x 1 y 1      x 2 y 2     v 1 u 1      v 2 u 2 v z 1 1 u z 1 1   v z 2 2 u z 2 2 z 2 z 1  0 0  z 1 z 2          2 k 1 k k k 3 4 (5.19) (5.20) (5.21)       where 1k H  k , 2   1 k , 3 v   and 0 k 4 0   u  are the four unknown parameters to be solved for by calibrations. Compared to Eqs. (5.17) and (5.18), it is trivial to solve Eq. (5.21) by using the LS approximation. Given the four parameters, solving the calibration problem requires at least three control points representing those distinctive feature points in the calibration object to establish the correspondence between the world and image coordinate systems. It is noted that we only need to measure the spatial offsets of these points in x and y directions, not their specific   138 coordinates. Once the calibrations are completed, the world coordinates xw and yw, can be obtained by the following equations: x w  k v 1 i  k z v 2 w i  k z 3 w  y w  k u 1 i  k z u 2 w i  k z 4 w  k k 1 3 k 2 k k 1 4 k 2 (5.22) (5.23) 5.4. EXPERIMENTAL TESTS ON SYNTHETIC AND APPLE SAMPLES 5.4.1. SIRI System Figure 5.5 shows the SIRI system used in this study. It was implemented in broadband imaging mode, and in particular, a linear translation stage (Edmund Optics, Barrington, NJ, USA) with a travelling accuracy of 0.102-0.279 mm (or 0.004-0.011 inches), and a flat plate acting as a reference plane attached to the translation stage, which is adjustable along the vertical axis, were used for the calibration purpose. A detailed description of other major instrument components of the system is given in Section 3.2. Camera Digital light projector Fiber optic cable Lens Linear translation stage Reference plane Figure 5.5. A photograph of the structured-illumination reflectance imaging system.   Three phase-shifted sinusoidal patterns, in 8-bit image format, with phase offsets of -2π/3, 0 and 2π/3, generated in Matlab (The Mathworks, Inc., Natick, MA, USA), were uploaded to the projector’s control software for light projection. The fringe direction of projected patterns were perpendicular to the plane spanned by the optical axes of the projector and the camera. The projected area on the reference plane at z = 0 was approximately an isosceles trapezoid due to the   139 tilting of the projector, with two parallel sides of 180 mm and 210 mm, and the width of 170 mm. The spatial frequency of the fringe patterns was about 0.15 cycles/mm-1 (equivalent to 15 complete fringes over a 10 cm width, and this frequency was proven effective for defect detection of apples from Chapters 3 and 4). 5.4.2. Samples Two standard samples, including one rectangular block made of Teflon and one trapezoidal block made of Nylon, as shown in Figure 5.6, were used for the phase-to-height calibration, and the trapezoidal block was also used as a gauge block for in-plane calibration. The rectangular and trapezoidal blocks are 25.95 mm and 25.34 mm in height (from their bottom surface to the top surface), respectively. Figure 5.6. Two verification blocks used for phase-to-height and in-plane calibrations: rectangular Teflon block (left) and trapezoidal Nylon block (right). Further, separate tests were conducted on four ‘Gala’ apples. The four apples had been kept in cold storage for about four months. They were free of any visual blemishes or defects, with the equatorial diameters of 65-80 mm. Images were first acquired from the intact apples, and each apple was imaged for two orientations, one with its equatorial (convex) area facing towards the camera and the other with its stem or calyx (concave) region facing towards the camera. Then, these apples were cut into halves, resulting in eight samples, for bruising tests, one of which was cut in the direction of the stem-calyx axis, while three others in the equatorial axis with different orientations, as shown in Figure 5.7. Each of the apple halves was subjected to bruising damage and then immediately imaged by SIRI. Bruises were induced by impacting the apple halves using   140 a pendulum device with a wooden ball falling freely from a rest position, with the impact energy of about 0.4 J, without causing noticeable damage. Figure 5.7. Four ‘Gala’ apples in two different orientations, after having been subjected to impact bruising damage. The two halves in each photo constitute one apple. 5.4.3. Image Acquisition and Processing Images for the reference plane were acquired under illumination of three-phase shifted sinusoidal patterns for the phase-to-height calibration. First, the reference plane was positioned at z = 0 and three pattern images were acquired, and then the reference plane was manually moved upwards to 18 different positions in about 5.08 mm (0.2 inch) increments, covering a measurement range of 0-91.44 mm, which is suitable for measuring fruits like apples. It should be mentioned that due to the limited depth of focus of the camera, one is not free to adjust the height, which, otherwise, would cause a severe out-of-focus issue. Thereafter, the reference plane was restored to its original position, and the two standard blocks and apple samples were placed on the reference plane and were imaged individually. In particular, the trapezoidal block was also imaged under planar illumination for in-plane calibration. For the two standard blocks, their long sides were kept parallel to the fringe stripes of the illumination patterns during image acquisition. Image acquisitions were performed using an in-house-developed program with the LabVIEW (National Instruments, Austin, TX, USA) based GUI, with the exposure time of the camera set to 100 ms and 2×2 image binning operations, which resulted in the images of 512×512 pixels. The acquired images were first processed by applying a low-pass Gaussian filter to suppress noise.   141 Phase demodulation was then done, by using the convention three-phase based approach, to obtain wrapped phase maps, followed by phase unwrapping and post-processing to yield continuous absolute phase maps. It is noted that for apple samples, apart from phase maps, AC images were also obtained to ascertain the bruise damage. Phase-to-height calibrations and height measurements for an object were performed based on the phase difference that was calculated by simply subtracting the absolute phase of a targeted object from that of the reference plane at z = 0. Phase-to-height calibrations were done using three types of methods, including non-linear LS curve fitting, linear LS (polynomial based) curve fitting and linear interpolation, as described in Section 5.2.4. For the LS curve fitting, five different polynomials of order 2-6 were studied. Further in-plane calibrations were done with the trapezoidal block as a calibration object, which are further described in the following section. The method described in Section 5.3 was used for in-plane calibrations. Eight corners of the trapezoidal block, as shown in Figure 5.8(right), were selected as control points, whose pixel coordinates in the SIRI image were manually detected by means of the imtool (an image viewer tool) of Matlab (The MathWorks, Inc., Natick, MA, USA). The correspondence between the eight control points in the SIRI image and their counterparts in the real scene was defined through visual inspection. The eight pairs of control points, that is, (A, B), (B, C), (C, D), (D, E), (E, F), (F, G), (G, H) and (H, A) were used for calibration. Figure 5.8. A calibration object (left), and its SIRI image (right) under planer illumination where eight points indicated by markers and letters are used as control points for in-plane calibrations.   142 To calculate the spatial offsets between two points, the x and y axes needed to be defined first, which, in principle, can be any two perpendicular lines passing through the point where the optical axis intersect with the reference plane. For convenience of calculations, the line passing through points A and B (or E and F) was assumed to be parallel with the y-axis, while the line passing through points A and F (or B and E) is parallel with the x-axis. Then the spatial offsets for a pair of points were obtained by measuring the geometry of the block with a high-precision digital caliper. The heights of these points were measured with the caliper as well, which could also be obtained from the phase-to-height calibrations. Once the spatial offsets, heights and pixel coordinates of these control points were known, the LS estimates of the four calibration parameters were computed from Eq. (5.21). 5.4.4. Performance Evaluation For the phase-to-height calibration, different methods were compared in terms of root-mean- square error (RMSE) of the difference between estimated and true heights for the reference plane at different positions, and also the overall RMSE was calculated as the average of mean-squared RMSEs for all the reference planes. The optimal calibration method, in terms of both RMSE and computation complexity, was further tested by measuring the height of the two plastic blocks. The in-plane calibration was evaluated in terms of residuals between the estimated and true spatial offsets along x and y axes for the eight pairs of control points, and RMSE of these residuals as well. The best phase-to-height calibration method combined with in-plane calibration was chosen for recovering the surface of apple samples.   143 5.5. RESULTS AND DISCUSSION 5.5.1. System Calibrations 5.5.1.1. Phase-to-height Calibration Figure 5.9(a) shows the curves of height versus phase difference for the reference plane at different heights. These curves are smooth, and vary monotonically and nonlinearly with phase difference, indicating that the curve fitting approaches could be used to calibrate the phase-to-height relationship. Moreover, a linear relationship could be approximated for a small line segment, thus justifying the use of linear interpolation for the phase-to-height calibration. Figure 5.9(b) shows the curves of RMSE versus height for the seven calibration methods. All the methods have yielded reasonably high accuracies with RMSEs of less than 0.4 mm. Figure 5.9. (a) Curves of height versus phase difference, and (b) root-mean-square errors (RMSEs) at different heights obtained by different calibration methods for the reference plane. Table 5.1 summarizes the overall RMSEs of the seven methods. The polynomials of order 3-6 and the non-linear calibration performed similarly, among which the 6th-order polynomial had the lowest RMSE of 0.027 mm, while the 2nd-order polynomial and linear interpolation performed much worse than the other methods [Figure 5.9(b)]. Table 5.1. Overall root-mean-square error (RMSE) for each calibration method.  Calibration method Overall RMSE (mm) Linear interpolation 0.188 Linear least-squares* A 0.212 0.033 0.030 C B Nonlinear least- squares E D 0.028 0.027 0.038 * A-E denote polynomials of order 2-6.   144 The nonlinear LS curve fitting, based on an analytically derived equation, only yielded an intermediate accuracy, because factors, such as lens distortion which affects the phase-to-height calibration, were not considered in the equation. Moreover, the method was the most computationally expensive because of intensive iteration operations. The polynomial based approach, by contrast, demonstrated good performance, although their parameters had no direct physical meaning as those in the non-linear LS calibration. Increasing the polynomial’s order from 2 to 3 resulted in a large decrease in RMSE; however, the improvement became marginal beyond the order 3. Moreover, it was noted that a higher-order polynomial potentially required more phase maps for calibration, and also incurred increased computation time and complexity. Therefore, the 3rd-order polynomial was chosen for the remaining analysis. Figure 5.10 shows the reconstructed height profiles for the rectangular and trapezoidal blocks by the 3rd-order polynomial. The reconstructed height for the rectangular block was much smaller than the true value of 25.95 mm, while for the trapezoidal block the height was accurately reconstructed with RMSE = 0.959 mm, with a relative error of 3.75%. The failure in height reconstruction for the rectangular block is a well-recognized problem due to sharp discontinuities of an object itself (Burton, 2014; Burton et al., 1995), which is frequently encountered by spatial phase unwrapping algorithms. Spatial phase unwrapping is carried out onto the wrapped phase by detecting the phase change at the neighboring pixels. In the presence of a sharp change in the height of the real scene, e.g., sharp edges of the rectangular block, the true phase change between neighboring pixels may exceed 2π, which, unfortunately, cannot be correctly resolved or detected by spatial phase unwrapping. As a result, the obtained unwrapped phase for discontinuous objects may suffer from large phase errors for multiples of 2π, which occur at those sharp discontinuities and can propagate across the image depending on the path of phase unwrapping.   145 Compared to the rectangular block with sharp edges, the trapezoidal block has gradually changed edges (parallel to fringe stripes during image acquisition), which had no abrupt changes in phase, and thus could be accurately reconstructed. Re-examination of Figure 5.10(a) reveals a height error about 20 mm, which corresponds to a 2π phase error according to Figure 5.9(a). Hence a quick remedy for the reconstruction failure is to add 2π to the unwrapped phase of the rectangular block, which is then converted to the height profile by the phase-to-height calibration. Figure 5.10(c) shows the reconstructed result for the rectangular block. It can be seen that, with the 2π compensation, the height of the rectangular block is fairly well reconstructed with RMSE = 1.080 mm, corresponding to a relative error of 4.16%. It is noted that errors for the two blocks were far greater than those in calibration, which was because the two blocks were not precisely machined, and their surface was not perfectly flat. Figure 5.10. Reconstructed height profiles of the rectangular (a) and trapezoidal (b) blocks, and the reconstructed height profile of the rectangular block (c) with 2π compensations. 5.5.1.2. In-plane Calibration Table 5.2 summarizes the calibration residuals for the in-plane calibration of the trapezoidal block, with the overall RMSE of 0.311 mm. The accuracy is fairly good, confirming the validity of the proposed calibration method. But it was still not comparable to the accuracies in the phase-to- height calibration, which was mainly attributed to the imperfections of the calibration model and the calibration object.   146 Residual x-axis residual (mm) y-axis residual (mm) (A, B) -0.803 -0.264 * These control points are shown in Figure 5.8 (right). (B, C) -0.181 -0.242 (C, D) 0.007 0.000 Control point pair* (D, E) 0.150 -0.159 (E, F) 0.401 -0.137 (F, G) 0.195 0.443 (G, H) 0.366 0.000 (H, A) -0.136 0.360 Table 5.2. Calibration residuals for eight pairs of control points in the calibration object. It should be emphasized that the present calibration was built upon a simple form of perspective imaging, which assumed no lens distortion and that the image plane was parallel to the reference plane (i.e., the world frame x-y plane, indicating no rotation and translation). Further improvement in the calibration accuracy would require to use a more adequate model and a more precisely machined calibration object. Applying the in-plane calibration to the height profiles obtained from the phase-to-height calibration resulted in the 3-D shape in the world coordinate system. The 3-D geometry of the rectangular block was reconstructed from the height profile with 2π phase compensations. Figure 5.11 shows the reconstructed 3-D shape (i.e., dense 3-D point cloud) for the two blocks. The reconstructed 3-D results look similar to reconstructed height profiles shown above, except that in-plane coordinates in pixel in the latter are converted to physical units of real space. Compared to the reconstructions shown in Figure 5.11, the reconstructions presented in Figure 5.10, strictly speaking, are not 3-D but 2.5-D, which contain only height or depth information per pixel. Despite this, the height profile, which captures the surface variations of an object, is sufficient in many applications, e.g., identification of surface concavities of fruit, which will be shown later. Figure 5.11. Reconstructed three-dimensional shape for the rectangular (a) and trapezoidal (b) blocks.   147 5.5.2. Tests on Apples For each apple sample, a number of images could be derived from the acquired raw fringe patterns, including normalized fringe patterns, wrapped phase, unwrapped phase, phase difference relative to the phase of a reference plane, AC, DC and ratio (i.e., AC/DC) images. Each of these images has different implications or values for fruit quality assessment. Figure 5.12 shows the resultant images for one normal apple and the other bruised apple. Compared to the raw fringe pattern, the normalized pattern significantly enhanced the fringe contrast, thus making more obvious the pattern distortion due to the fruit surface concavities, i.e., stem/calyx regions. Analysis of changes in the curvature of these fringes would help locate stem/calyx regions. Three phase maps, i.e., wrapped phase, unwrapped phase and phase difference, were mainly used for 3-D shape reconstruction, among which the phase difference is directly related to the surface contour of an object. As indicated in Figure 5.12, the concave (i.e., stem/calyx) regions were clearly revealed in the phase difference image, which appeared as a relatively dark spot or patch due to the decreased height of these regions. Different from the phase maps, AC, DC and ratio (i.e., AC/DC) images are intensity-based; and relative to DC images, AC and ratio images exhibited enhanced contrast, which are more effective for bruise detection.   148 Figure 5.12. Raw pattern, normalized pattern, wrapped phase, unwrapped phase, phase difference, amplitude component (AC), direct component (DC) and ratio images for a normal (a) and a bruised (b) apple sample. Figure 5.13 presents a comprehensive comparison of phase difference and ratio images for all the apple samples, both intact and half, normal and bruised apples. All the phase difference maps indicated the concave regions very well, especially when these regions were not around the far edges of the fruit; but they did not reveal the bruises that were visible in the ratio images. When both bruises and concave regions were imaged, only the concave regions were present in the phase difference maps. This is because such defects as bruises usually incur no significant changes in the fruit shape. Hence, it is logical and feasible to discriminate bruises from the concave regions by using both ratio or AC and phase difference images, which may warrant further study.   149 Figure 5.13. Phase difference and ratio images for normal (left two columns) and bruised (right two columns) apples, where AC and DC denote amplitude and direct components, respectively. Further, the height profile and 3-D shape of these apples in the world coordinate system were reconstructed based on the phase-to-height and in-plane calibrations. As shown in Figure 5.14 and Figure 5.15, the basic geometry of apples was well reconstructed. However, the reconstructed   150 height for the intact apples deviated significantly from the actual sizes ranging from 65 to 80 mm, due to the object’s sharp edges, which is the same problem as with reconstruction of the height profile of the rectangular block. The edge or boundary of an intact apple in the image may have a large distance (e.g., more than 20 mm) from the reference plane, which will cause an abrupt phase change exceeding 2π. Thus, the resultant height profile only represents the relative height of the upper part of the apple seen by the camera. Despite this, the reconstructed height profiles or shapes, as with the phase difference maps, still allowed to locate the concave regions of apples. Compared to Figure 5.14, Figure 5.15 gives complete information about the 3-D geometry of apples, with three coordinates represented in real world units (i.e., mm), which can be used for measuring the size and volume of fruit. Figure 5.14. Reconstructed height profiles for all the apple samples, where the left two columns correspond to intact, normal apples, while the right two columns correspond to half, bruised apples.   151 Figure 5.15. Reconstructed three-dimensional shape for all the apple samples (expressed in real world dimensions for the three coordinates), where the left two columns correspond to intact, normal apples, while the right two columns correspond to half, bruised apples. 5.5.3. Discussion Phase demodulation is the first critical step for phase-based 3-D geometry reconstruction of an object. Although the present work employed the conventional three-phase based demodulation approach, the proposed two-phase SPT based demodulation approach, as described in Eq. (5.8), was also attempted for height profiling of apples (Lu and Lu, 2017b), since it was more attractive practically. Using a 3rd-order polynomial based curve fitting for the phase-to-height calibrations, resulted in relative measurement errors, for the rectangular and trapezoidal blocks, of 7.46% and 6.67%, respectively (2π phase compensations were done for the rectangular block). The errors were larger than those obtained with using the conventional three-phase based demodulation approach. Further tests on the apple samples also allowed for the high profile reconstructions, except that there existed slight ring artifacts due to the use of finite Fourier transform in SPT and   152 boundary noise, as illustrated in Figure 5.16. Hence, there is a trade-off between imaging speed and the reconstruction performance or accuracy for using SPT for phase demodulation. apples while the right two column for half apples) with using SPT for phase demodulation (2π phase compensations Figure 5.16. Example of reconstructed height profiles for apple samples (the left two columns represent intact have been considered for the intact apples). In the overall phase analysis procedures, the importance of phase unwrapping cannot be overestimated, since it will determine the success or failure of the ultimate height or shape reconstruction and the computation load or time as well. In order for rapid inspection, spatial phase unwrapping is more preferable than temporal phase unwrapping, since the former requires only a single wrapped phase map rather than a sequence of wrapped phase maps for the latter. But, spatial phase unwrapping has a major drawback: it cannot resolve the ambiguity when the phase jump exceeds 2π due to object discontinuities based on a single phase map (this is an inherently ill-posed problem), as demonstrated by tests on reconstructing a rectangular block and intact apple samples in this study. Hence, the SIRI system, based on spatial phase unwrapping, may not achieve accurate 3-D shape reconstruction for the objects with sudden changes in height. Despite this shortcoming, SIRI is capable of capturing the geometry variations of objects, with or without the aid of phase compensations of multiples of 2π. The resultant phase difference map, the height profile and the 3-D shape, as verified by the test results for apple samples, still allowed detecting the concavities   153 of fruit. Moreover, a possible remedial solution to the phase unwrapping issue is to decrease the spatial frequency of illumination patterns, at the price of decreased height sensitivity. The preliminary test on apple samples only shows the potential of SIRI for discrimination of bruises from surface concavities of apples. An in-depth, extensive study is still needed for detect both surface concavities and defects by combining the DC and AC and phase-derived images (e.g., phase difference, reconstructed height or 3-D geometry profiles). Moreover, normalized fringe patterns may also aid in identifying surface concavities due to enhanced fringe contrasts. 5.6. CONCLUSIONS This study proposed the phase analysis methodology for SIRI to reconstruct the 3-D geometry of fruit. The underlying principle is that the phase of sinusoidal patterns encodes information about the geometry of fruit in the distorted pattern stripes of the captured pattern images, and that the geometric profile can be reconstructed by means of phase-based calibrations. The whole procedure consists of four major steps: (1) the acquired pattern images are demodulated into a phase map, referred to as wrapped phase; (2) the wrapped phase is converted to a continuous, relative phase map through phase unwrapping, which is further converted to a phase difference map with respect to the absolute phase of a reference plane; (3) the phase difference map is then transformed to a height profile based on a calibrated phase-to-height relationship; and (4) the 3-D shape in a world coordinate system is obtained based on in-plane calibrations. Phase demodulation can be done by using the conventional three-phase based approach or by the two-phase based SPT, despite that the latter can result in decreased accuracies and ring artifacts for the reconstructed contours. Phase unwrapping plays a pivotal role in attaining high-quality phase maps for subsequent calibrations. One possible pitfall involved in the present phase unwrapping algorithm is that it cannot resolve the phase ambiguity in case of phase jumps larger   154 than 2π caused by sudden changes or discontinuities of an object. Consequently, the reconstructed profiles for apples, most often, represent the upper surface of fruit imaged by the camera. In calibrating the phase-to-height relationship, 3rd-order and higher polynomial based curve fitting performs better than curve fitting based on an analytically derived model and linear interpolation. Further, an in-plane calibration method based on the simplified perspective imaging is proposed to obtain the whole set of 3-D information in the world coordinate system. This study has laid a foundation for using SIRI for the reconstruction of 3-D geometry of fruit, and thus expanded the capability and applications of the technique for fruit quality evaluation. Further research is needed on detecting the surface concavities and defects of apples based on the phase difference map, reconstructed height profile or 3-D shape, together with the intensity images (i.e., DC and AC). Moreover, further optimization is needed regarding the phase demodulation and unwrapping algorithms for rapid detection.   155 6. SUBSURFACE AND SURFACE DEFECT DETECTION 6.1. INTRODUCTION Subsurface tissue bruising is one of the most common defects encountered during postharvest handling of apples (Opara and Pathare, 2014; Van Zeebroeck et al., 2007b). Detection of bruises in apples is not a trivial task due to a diversity of factors including bruise type, severity, time, and fruit pre- and post-harvest conditions (Upchurch et al., 1994). The bruises that occur within a few hours or even minutes after bruising damage, have not been fully developed, and are generally absent of visible external symptoms and significant chemical alterations of fruit tissues. Fresh bruises can be more difficult to detect than old ones (Martinsen et al., 2014). While hyperspectral and multispectral imaging have been researched for bruise detection of apples, the research has been largely focused on the detection of bruises of a few hours to days (Baranowski et al., 2012; ElMasry et al., 2008; Lu, 2003; Xing et al., 2005 ; Xing and De Baerdemaeker, 2005). As reviewed in Chapter 2, apples are also susceptible to various surface defects or disorders, and considerable effort has been devoted to using computer imaging technologies for defect detection of apples in the past three decades. The most common modalities for detecting surface defects on apples include monochromatic or black/white imaging (Davenel et al., 1988; Lu and Lu, 2017a), color vision (Barreiro et al., 1998; Chen and Sun, 1991), and spectral imaging, including hyperspectral (Ariana et al., 2006; Mehl et al., 2004) and multispectral (Bennedsen et al., 2007; Kleynen et al., 2005; Unay et al., 2011). The performance of these imaging modalities, which use diffuse, uniform illumination, still falls short of the industry’s expectations because some surface and subsurface defects are either not visible or can be confused with the normal tissue during image processing and classification.   156 SIRI looks promising for enhancing defect detection of apples, compared to conventional UI- based modalities, since it is able to provide two sets of intensity images, i.e., DC and AC, corresponding to UI and SI, respectively. According to Chapter 3, AC images have superior spatial resolutions and image contrasts over DC images, while also possessing depth-resolving features, depending on the spatial frequency of light patterns. The preliminary study described in Chapter 3 and the work done by Li (2016) have shown that AC images can effectively detect subsurface bruising in apples at early stage, which otherwise could not be achieved by DC images. In detecting fresh subsurface bruising in ‘Delicious’ and ‘Golden Delicious’ apples, AC images consistently outperformed DC images, resulting in overall detection rates ranging from 70% to 100% (Li, 2016). However, it was also found that AC images might be inferior to DC images for detection of surface defects on apples, which would depend on type of defect as well as fruit variety (Li, 2016). This can be explained by the fact that AC globally enhances fruit surface features, including normal lenticels and other skin textures, which could confound with the appearance of defects, and also diminish the image signal-to-noise ratio (SNR), especially at high spatial frequencies. Since DC and AC images provide different features pertaining to defects, it is expected that proper combination of DC and AC images would improve the detection accuracy than using them individually. Built upon and expanded from the preliminary work on defect detection of apples using SIRI, this study was aimed to carry out more extensive, in-depth experiments to evaluate the ability of SIRI for defect detection of apples. In this study, defect detection of apples was performed by using a multispectral SIRI system, coupled with the two-phase based demodulation approach described in Chapter 4. Given the importance of bruising defects for apples, the first experiment was focused on detection of fresh subsurface bruises in apples, which were artificially generated   157 through impact tests; and a second, separate experiment was conducted on detection of naturally occurring surface and subsurface defects of apples. 6.2. DETECTION OF FRESH SUBSURFACE BRUISING The main objective of this study was to demonstrate the capacity for detection of fresh subsurface bruising in apples by using a multispectral SIRI system with two phase-shifted sinusoidal patterns of illumination. Three specific objectives were to: 1) acquire spectral images in the wavelength range of 650 to 950 nm for four varieties of apples, under sinusoidal illumination at a spatial frequency of 0.10 cycles/mm, 2) apply a two-phase based demodulation approach to retrieve AC and DC images, and 3) implement a modified Otsu thresholding method for bruise segmentation and a simple shape factor based algorithm for fast bruise detection. 6.2.1. Material and Methods 6.2.1.1. Apple Samples A total of 60 apples, 15 apples for each of four varieties, ‘Delicious’, ‘Gala’, Golden Delicious’ and ‘Granny Smith’, were used for bruise detection. These apples were purchased from a local grocery store, and were all free of visual blemishes or defects. ‘Golden Delicious’ and ‘Granny Smith’ apples have uniformly, bright green-colored skin, which are susceptible to bruising; while ‘Gala’ and ‘Delicious’ have dark colored skin with a red peel, for which bruises are generally more difficult to recognize. These apples were cut into halves in the direction of the stem-calyx axis for the experiment, resulting in a total of 120 apple samples. Eighty samples, 20 for each variety, were subjected to bruising tests, while the remaining 40 samples were not bruised. Bruising tests were performed by impacting an apple at its equatorial area with a wooden ball falling freely from the rest position. The impact energy for bruising was estimated to be 0.6 J, and the resultant bruising damage (about 10 mm in diameter beneath the fruit skin) for these apple samples could not be   158 clearly observed immediately after the impact test, as shown in Figure 6.1. A detailed description of the impact test is given in Zhu et al. (2016) . Figure 6.1. Photograph of bruised ‘Delicious’, ‘Gala’, ‘Golden Delicious’ and ‘Granny Smith’ (from left to right)   6.2.1.2. Image Acquisition apples. As schematically illustrated in Figure 6.2, a LCTF-based multispectral SIRI system was used in this experiment. This system was equipped with a 12-bit CCD camera with 1376×1040 effective pixels (SensiCam QE, Cooke Corp., Auburn Hills, Mich.), attached with a C-mount focusing lens (Electrophysics 25 mm F/1.4 macro lens), and an LCTF (VariSpec, Cambridge Research and Instrumentation, Inc., Woburn, Mass.). A description of other instrument components is given in Section 3.2, and a photograph of the SIRI system is shown in Figure 3.4 (left). Figure 6.2. Schematic of a liquid-crystal tunable filter (LCTF) based multispectral structured-illumination reflectance imaging (SIRI) system. Considering the low spectral response of the camera above 950 nm and the bandwidth (i.e., 10 nm) of the LCTF, the effective wavelength range was adjusted between 650 and 950 nm with 20 nm increments, thus resulting in a total of 16 spectral images acquired for each sample. Two phase-   159 shifted sinusoidal patterns with phase offsets of 0 and 2π/3, respectively, at the spatial frequency of 0.10 cycles/mm, were used for the experiment. As comparison, uniform illumination was also performed at each wavelength. The distances from the sample to the projector and to the lens of the camera were set at approximately 15 cm and 26 cm, respectively. The fringe stripe direction of projected patterns was perpendicular to the plane spanned by the optical axes of the camera and the projector. All the bruised apples were imaged immediately after they had been subjected to impact bruising, and meanwhile the unbruised apple samples were also imaged by the system. During imaging, the apple samples were placed on a sample holder with a vertically adjustable stage, and the exposure time of the camera was set to 250 ms for all the wavelengths with 2×2 binning operations (resulting in acquired images of 688×520 pixels in size). 6.2.1.3. Image Processing Preprocessing The acquired raw images suffered from non-uniformity and noise artifacts. The non-uniformity is also known as the vignetting effect (i.e., edges are darker than the center in the image), which is caused by multiple factors including the camera, lens, filter and the curvature of samples. Image de-noising was first performed by applying a low-pass Gaussian filter to the raw images. Then flat-field corrections, as described in Eq. (3.5), were done by acquiring images from a Spectralon panel (Labsphere, Inc., Sutton, NH, USA) with the reflectance rate of 98% or higher, and dark background with the light source turned off, which would correct the image non-uniformity and the dark current effect of the CCD camera as well. Image Demodulation The two-phase based demodulation approaches described in Chapter 4, were utilized for retrieval   160 of both AC and DC images. The AC image was obtained by applying SPT to two phase-shifted pattern images, according to Eq. (4.13), while DC was recovered by Eq. (4.17). Further, the ratio image was obtained by dividing AC by the corresponding DC image. Subsequent image analysis was mainly performed on the ratio images due to its improved image uniformity and contrast between bruise and non-bruise regions. Background Removal The demodulated images contain information on both apple and background. The background needs to be removed so that further analysis would be focused on the region of interest (ROI), i.e., apple alone. Since the background usually has much lower intensity than the apple, a global threshold can be chosen and applied to a demodulated image for background removal. Here, the demodulated DC image at 730 nm, which exhibited good contrast between the fruit and the background, was subjected to thresholding to generate a mask for each apple sample. A simple threshold could erode or remove the bruise region that also had low intensities. Thus, further morphological operations were performed to fill the possible holes due to thresholding. The obtained mask was then applied to the demodulated images at other wavelengths for background removal for the apple sample. Principal Component Analysis Principal component analysis (PCA) is a widely used method for dimensionality reduction, by projecting high-dimensional data into a low-dimensional space spanned by a few major principal components (PCs) (Gemperline, 2006). These PCs are a linear combination of the original data weighted by orthogonal basis vectors (eigenvectors) that are known as loadings or eigen-spectra. The PC loadings indicate the relative contribution from each variable in forming a specific PC, which can be used to identify variables, i.e., wavelengths, representing large contributions. In this   161 study, PCA was used to identify key wavelengths for bruise detection of apples. Figure 6.3 shows the process of applying PCA to the AC images at different wavelengths. The spectral images at different wavelengths, after a mean-centered pretreatment, were stacked into a 3-D image cube, where m and n are pixel numbers of the image in the row and column directions respectively, and k is the number of wavelengths. The image cube was then transformed (reshaped) into a 2-D matrix A, where each column represents one spectral image. Eigenvector decomposition was applied to the variance-covariance matrix (ATA) to obtain its eigenvectors (V) (i.e., PC loadings), and PCs were derived from PCk = AVk. The resultant PCs were then reshaped into 2-D matrices to obtain PC score images. It was found that, as shown in Figure 6.3, only the first PC image provides the most meaningful information about the bruise defect. Therefore, only the first PC image was extracted and utilized for bruise segmentation and its loading plot was used for identifying the key wavelengths. Figure 6.3. Flowchart of principal component analysis (PCA) applied to multispectral images. Image Segmentation Image segmentation is a critical step in processing AC images for defect detection. Numerous   162 image segmentation techniques exist, among which thresholding is perhaps the most widely used to segment gray-scale images (Sezgin and Sankur, 2004). Otsu thresholding is a simple and popular technique for automatic image segmentation, which selects an optimal global threshold based on maximizing the between-class variance derived from the image histogram (Otsu, 1979). However, the between-class variance based Otsu thresholding technique tends to cause over segmentation in the case where the histogram is close to unimodal with too dissimilar peaks, or there is a large difference in size between the targeted object and background (Ng, 2006; Xu et al., 2011a), e.g., for detection of defects that are usually much smaller relative to the entire object. In this research, modifications to the Otsu technique were proposed to alleviate this issue by introducing a within- class variance for valley-emphasized thresholding. Let us consider a gray-scale image C with N pixels and intensity levels ranging from 0 to L, and assume it to be partitioned into two regions C1 and C2 with N1 and N2 pixels, respectively, by a threshold T. The intensities for C, C1 and C2 at pixel i are denoted by iC , 1iC and 2iC , respectively; and the average intensities for C and C1 and C2 are C , 1C and 2C , respectively. It follows that, the variance of the pixel intensities across the entire image, 2 , can be decomposed into two within-class and between-class portions 2 w and C 1 i C 1 i i ( ( (  = N  1 i  N 1  1 i  1= N 1 N 1 N 1  N 1= N 2    B N 1  1 i     N  1    1 i   N 1  1 i  2 W ( ( 2    C C  2 )  2 C )  1 N N 2  2  i ( C 2 i  2 C )  C C C 1   1 C 1 i  2 C 1 )  2 )  1 N N C C  ( 1 1 C 2  ) 2 ( N 2  1 i        ) 2  i 1 N 1 N 163 2 B as follows: C 2  C 2  2 C ) (6.1) N 2  1 i     ( C 2 i  C 2 2 )  N C ( 2  2 C ) 2      C 1 i  2 C 1 )  N 2  1 i  ( C 2 i  C 2  N C C  ( 1 1 2 )  N C ( 2  2 C ) 2   In the conventional Otsu thresholding, the best threshold is decided by maximizing 2 B , or equivalently minimizing 2 w . However, 2 w in Eq. (6.1) is a biased representation of the actual within-class variance (it accounts for the prior probabilities for the two objects), which, instead, should be expressed as follows: 2  W   1 N 1 1 N 1 N 1  1 i  N 1  1 i  ( C 1 i  2 C 1 )  2 C 1 i  2 C 1  1 N 2 N 2  1 i  1 N 2 N 2  1 i  ( C 2 i  C 2 2 ) 2 C 2 i  2 C 2 (6.2) Then, with the new within-class variance, a modified Otsu thresholding method is proposed to seek an optimal threshold as follows: T m1  Arg min 0 T L     ( ) T 2 W (6.3) For a histogram that has two unequal peaks in size, the modified method is capable of obtaining a proper threshold that corresponds to the valley position of the histogram (see APPENDIX A for detailed proof of the modified thresholding method). Another two variants can be obtained by taking into account both the within-class and between-class variances as follows: T m2  Arg max 0 T L    2   W ( T 2 B ) ( T  ) T m 3  T opt1 m ax{ m ax{ 2  B 2  B ( T ( T T )} opt 2 )} m in{   m in{ 2  W )} ( T 2  W ( T )} (6.4) (6.5) Figure 6.4 shows two examples of threshold selection by the classic Otsu thresholding and the other three modified methods as described in Eqs. (6.3)-(6.5). These thresholding methods are applied to an artificially generated bimodal histogram, which has a dominant peak representing the normal tissue and a secondary peak residing in the left part corresponding to the bruised tissue, and a typical histogram of a bruised apple. It can be seen that, the three modified methods, especially the two methods given in Eqs. (6.3) and (6.5), produce more reasonable thresholds, from   164 which the classic Otsu thresholding remarkably deviates. For more reliable defect segmentation, the optimal threshold was determined by averaging the results obtained by Eqs. (6.3) and (6.5) in this study. Figure 6.4. Threshold selection for an artificial histogram (a) and a typical histogram (b) of a bruised apple, where Totsu, Tm1, Tm2 and Tm3 are obtained by the classic Otsu thresholding, and other three modified methods as described in Eqs. (6.3), (6.4) and (6.5), respectively. Before image segmentation, a 10×10 median filter was applied to the spectral images or PC score images to eliminate lenticel spots on the fruit surface and other small irregularities, which otherwise could contribute to false positives. After the initial segmentation, morphological filling operations were done to removal holes caused by thresholding. Furthermore, the segmented objects were evaluated based on their shape features including perimeter (P) and area (S), and either dismissed or counted as bruises. Since the shape of a bruise region is close to circular or elliptical, the circularity [C = P2/(4πS)] ranging from 0.7 to 1.3 was used to select the bruise area. The circularity range was chosen based on literature (Clark et al., 1998) and experimental observations. Finally, a cleaning operation was applied to remove very small (smaller than 0.5% of ROI) objects based on their area feature. The flowchart in Figure 6.5 demonstrates the whole procedure of image segmentation for bruise detection. The initial segmentation after thresholding gave a binary image with the normal fruit tissue set to zero (black) and possible bruise regions set to 1 (white); morphological operations removed very small regions, and further refinement was done to eliminate the elongated edges based on circularity; and finally the bruise region was segregated, which accorded well with the   165 actual bruise region. It is noted that initial segmentation sometimes revealed the fruit edges, which are due to the effect of fruit curvature and can be falsely recognized as bruises. Figure 6.5. Flow chart of the procedures of bruise segmentation. The image segmentation procedures described above were developed based on the pre-analysis of AC images for four typical apple samples, one for each variety, and would be applied to the rest of apple samples for bruise detection. If there existed no non-zero pixels in the final segmented binary image, the apple was considered normal, otherwise bruised. The detection performance was evaluated in terms of the percentage rates of false positive (FP), false negative (FN) and overall error that were calculated for the 120 apple samples. 6.2.2. Results and Discussion 6.2.2.1. Demodulated Images Figure 6.6 shows examples of uniform illumination images acquired for four apples, one for each of the four varieties, with fresh bruises at 730 nm. No bruises could be ascertained in these images, and also in uniform illumination images at other wavelengths (not presented). On the contrary, as shown in Figure 6.7 (only the amplitude images for two apple samples are presented), the amplitude images obtained by the multispectral SIRI system clearly revealed the bruise, which verified the superior capability of the SIRI technique, as in the case of broadband black/white imaging, for detection of such subtle defects as fresh bruises in apples. The success of amplitude images could be attributed to the use of sinusoidal illumination patterns that is able to produce   166 enhanced-contrast/-resolution and depth-resolved images, compared to uninform illumination. The contrast variation in the amplitude images was associated with wavelength as light at different wavelengths had different interactions with fruit tissues. Every single spectral amplitude images seemed to be able to identify the bruise in the apple, but overall, the amplitude images over the wavelengths from 690 to 810 nm gave relatively strong image contrast for bruise detection. Figure 6.6. Uniform illumination images of ‘Gala’, ‘Granny Smith’, ‘Golden Delicious’ and ‘Delicious’ (from right to left) apples. It was important to note that the spectral amplitude images above 850 nm suffered from specular reflectance artifacts, due to the polarizer in the SIRI imaging system that was not very effective over the NIR region. Moreover, these spectral images had low signal-to-noise ratios since the camera used had rather low spectral responses for the wavebands above 850 nm. Given these factors, PCA was applied to the spectral amplitude images excluding those above 850 nm. Figure 6.7. Amplitude images of ‘Gala’ (a) and ‘Delicious’ (b) apples at the wavelengths from 650 to 950 nm.   167 6.2.2.2. PCA Figure 6.8 shows the first principal component (PC1) score images for four apples. These PC1 images represented the most significant portion (more than 80%) of the total variation information from multiple wavelengths, and they looked similar to the original amplitude images and exhibited strong image contrasts between bruised and normal tissues. Other PC score images (not presented) revealed the information about apple conditions, such as surface lenticels, but they provided little information about bruises. Figure 6.8. Examples of the first principal component (PC1) score images for ‘Gala’, ‘Granny Smith’, ‘Golden Delicious’ and ‘Delicious’ (from right to left) apples. Figure 6.9 shows the loading plot of the PC1 score images for the four varieties of apples. The loadings are the weighting coefficients in forming PC1, indicating the relative importance of each wavelength. The loadings had lower values at the wavebands of 650, 670 and 690 nm, which were susceptible to color influences of fruit skin, reached a plateau above 710 nm and then decreased after 830 nm. The two greenish varieties, i.e., ‘Granny Smith’ and ‘Golden Delicious’ exhibited a valley at 670 nm, which was absent for the other two reddish apple varieties, due to the absorption of chlorophylls at the waveband. Overall, the seven wavebands between 710-830 nm had consistently higher contributions to the PC1 score images, which accorded well with visual observation of the amplitude images. It had been found that either the spectral amplitude or PC1 score images presented strong contrasts that enabled bruise detection by applying a simple threshold to these images. Next, the spectral images at the seven wavelengths, together with the PC1 score images, were subjected to the aforementioned thresholding based image segmentation individually to evaluate their performance in bruise detection.     168 Figure 6.9. Loadings of the first principal component for four varieties of apples. 6.2.2.3. Performance Evaluation Bruise detection was performed by applying the image segmentation algorithm described in Section 6.2.1.3 to the seven spectral and PC1 score images individually for 120 apple samples. Table 6.1 summarizes the detection results including FP, FN and overall error rates. Table 6.1. Results for detecting fresh bruises in four varieties of apples. 710 nm Error 15.4 FN (%) [a] 7.1 FP (%) [b] Overall error 12.5 810 nm 15.4 4.8 11.7 750 nm 16.7 7.1 12.5 770 nm 16.7 4.8 12.5 790 nm 19.2 4.8 14.2 730 nm 16.7 7.1 13.3 830 nm 39.7 2.4 27.5 PC1 19.2 9.5 15.8 [a], [b] FN and FP represent false negative and false positive, respectively. All six wavelengths from 710 to 810 nm achieved decent accuracies with overall error rates ranging between 11.7-14.2%; the wavelength 810 nm resulted in the lowest error, but the accuracy difference was only marginal compared with the other five wavelengths. Similar, good accuracies obtained for the six wavelengths could be due to the fact that apple tissues generally exhibit similar optical properties, including absorption and scattering coefficients, over this wavelength range; however, bruising would have a more significant effect on the scattering coefficient than on the absorption coefficient (Anderson et al., 2007; Martinsen et al., 2014), which contributes to image contrasts between bruised and sound tissues. The wavelength of 830 nm gave the highest overall error of 27.5% because of the deteriorated bruise contrasts. In addition, the PC1 score images, which incorporated information from all the wavelengths, were only superior to the wavelength of 830 nm with an overall error of 15.8%. The relatively larger error for the PC1 score images was   169 probably due to the decomposition of variance-covariance matrix, which contained negative values that may have impaired the image contrast and affected the bruise detection. Besides, it was noted that the FN rates were much higher than the FP rates, indicating that more apples with bruises were misclassified as normal. The misclassification was partly explained by the fact that the amplitude images for some apples, e.g., ‘Delicious’ that appeared to be more bruising resistant, presented weak image contrast for bruises. Imperfection of the thresholding algorithm, on the other hand, also contributed to the higher FN rates as well, since the misclassification occurred even when the bruise region had been clearly revealed in the amplitude image. As illustrated in Figure 6.10, the misclassification of a bruised apple was primarily caused by the improper threshold. The histogram of that apple was multi-modally distributed due to a high degree of variations in fruit color. A relatively dark stripe of fruit skin existed along the SC axis of the apple. The proposed thresholding method missed the optimal threshold and counted the dark stripe as the bruised area, and thus made a wrong decision based on the shape factor criterion. Figure 6.10. Bruise segmentation by the threshold (T) obtained by the proposed method and the optimal threshold Possible solutions to the multi-modal histogram issue may be obtained by incorporating more (Topt) by manual selection based on trials and errors. shape features of a histogram, such as valley, peak and curvature, constructing a 2-D histogram or introducing multi-level or adaptive thresholds, in the thresholding algorithm, which will be the subject of future work. In addition, supervised image segmentation methods, such as discriminant analysis, neural networks and support vector machine, are also worthy of trial, although they are   170 more complicated due to model training required. 6.2.2.4. Discussion The bruise detection accuracies obtained by the multispectral SIRI are comparable to, or better than, those reported on detection of the old bruises (>12 h) in apples by using uniform illumination based hyperspectral or multispectral imaging. Lu et al. (2003) reported the detection rates of 69.6% for ‘Delicious’ and 62.8% for ‘Golden Delicious’ apples, for bruises of 12 h after impact bruising. Xing et al. (2005) reported a detection accuracy of 86% for 1-day old bruises in ‘Golden Delicious’ apples. Some recent studies reported accuracies of over 90% for detecting both new and old bruises in apples by hyperspectral imaging (Baranowski et al., 2013; Huang et al., 2015; Keresztes et al., 2016). Direct comparisons with these studies are difficult because of different experimental conditions such as apple variety, bruise age and severity, as well as detection algorithms. It should be pointed out that all these studies including the present work were focused on the detection of bruises that were artificially created under controlled conditions, which may not be representative of the bruises occurring under natural conditions like fruit transportation and sorting. Naturally occurred bruises are usually far more varied in size, shape and severity than the controlled bruises, and thus more challenging to detect (Li, 2016). Bruise detection is of great importance but still a challenging issue in automatic sorting and grading of fruit. SIRI is capable of obtaining high-resolution and -contrast images and enabling depth-resolved characterization of tissues by shining sinusoidally-modulated light, which has great potential for enhancing bruise detection of apples. This study confirmed the superior performance of the multispectral SIRI system in detecting fresh bruises in apples. Wavelengths at 710, 730, 750, 770, 790 and 810 nm achieved similar, good detection accuracies, which suggests the possibility of rapid detection of bruises in apples by monochromatic or dichromatic imaging by using one or   171 two bandpass filters or monochromatic lighting sources. The proposed thresholding method based on the within-class variance is applicable to segmenting other types of defects other than bruises, under a realistic assumption that the image histogram has two well-resolved peaks corresponding to defective and normal tissues. Further improvement of the thresholding method is still needed to deal with the segmentation issue caused by multimodal histograms. In this study, the demodulation for AC and DC images was accomplished by using a new two- phase based approach instead of the conventional three-step phase-shifting technique, thus saving one third of the time for image acquisition, and only single spatial frequency at 0.10 cycles/mm was used to further speed up image acquisition. But it should be noted that the current SIRI system is only applicable to stationary samples. Future research is needed on the improvement of instrumental hardware and image processing algorithms to detect moving samples in real-time applications. Moreover, further research is also needed to develop a more general algorithm for defect detection. 6.3. DETECTION OF SURFACE AND SUBSURFACE DEFECTS Although AC images are far more effective than DC images for detecting fresh bruises, they may not have the same prowess in detecting some surface defects that are visible to the naked eyes. It seems that, according to experimental observations, either AC or DC images are unlikely to give the best detection for general subsurface and surface defects of apples. In the above study on detecting subsurface bruising in apples, only a simple rule-based algorithm was applied due to the relatively regular shape of bruises, which, however, could fail for detecting general defects that vary greatly in their morphological characteristics (i.e., shape, size and surface texture) and contrasts. Hence, a more effective image analysis algorithm needs to be developed, from image preprocessing to decision making, for detection of general subsurface and surface defects of apples.   172 Image enhancement is critical to detection of fruit defects by imaging techniques. Vignetting (a phenomenon characterized by a grading fading out of the image intensity towards its periphery) and noise are common artifacts, which obscure the features of interest and degrade image quality. Bi-dimensional empirical mode decomposition (BEMD), as an extension of 1-D empirical mode decomposition (EMD) to 2-D domain, can adaptively decompose an image into components of varied frequency scales (Linderhed, 2009; Nunes et al., 2003). BEMD provides a new means for image enhancement by selectively discarding unwanted image components, such as those resulting from noise and illumination vignetting, which can thus be used to aid in defect detection of apples. In recent years, machine learning has emerged rapidly as a new field of artificial intelligence for data analysis and image processing, and provides a plethora of learning algorithms or pattern classifiers (Marsland, 2015), for addressing object classification and detection problems. In particular, random forest (RF), support vector machine (SVM) and convolutional neural network (CNN) are among the most successful algorithms in many classification tasks, which have the potential for being used for enhanced defect detection of apples. The main objective of this study was therefore to explore the ensemble or combination of DC and AC images for detecting both surface and subsurface defects for apples. Specific objectives were to: 1) acquire DC and AC images for two varieties of apples with different types of defects by using a multispectral SIRI system coupled with a two-phase based image demodulation approach; 2) enhance the DC and AC images, prior to features extraction and classification, by using a new BEMD image preprocessing method; and 3) develop classifiers for DC, AC and ratio (i.e., AC divided by DC) images and their combinations, by utilizing three machine learning algorithms, including RF, SVM, and CNN, to classify apples into defective and normal classes, and compare their performance for defect detection.   173 6.3.1. Materials and Methods 6.3.1.1. Apple Samples Apples of ‘Golden Delicious’ and ‘Delicious’ varieties were hand-picked from an orchard of Michigan State University’s Horticultural Teaching and Research Center in Holt, Michigan in September 2017. Many of the harvested apples had been inflicted with different types of defects while they were still on the tree. Each apple was visually inspected and recorded for specific physiological disorders and/or mechanical or natural (i.e., bruising, frost) damage. The list of defects observed for the two varieties of apples are summarized in Table 6.2. Table 6.2. Number of apples with different types of defects. Russeting or scab Lenticel spots or breakdown Mechanical injury Insect damage Rot or decay Foreign matter Mixed defects 37 28 40 6 28 7 3 1 1 20 35 Variety Bitter pit 64 DL GD Frost ring 76 Note: DL and GD represent the varieties of ‘Delicious’ and ‘Golden Delicious’, respectively, and the mixed type indicates a combination of two or more types of defects, usually including mechanical injury. Among the different types of defects, frost ring and bitter pit, as shown in Figure 6.11, represented the main defects for the ‘Delicious’ and ‘Golden Delicious’ apples, respectively. Frost ring, a special type of skin russeting, was caused by frost when apples were in bloom, while bitter pit developed beneath the fruit skin but had become externally visible. Mechanical injury, including subsurface bruising, was observed for both varieties, which could have been induced by the fruit-to-fruit impact during transport of apples and stacking loads during storage. The collected apples had been kept in cold storage at 4 °C for about 50 days prior to image acquisition. In total, there were 190 defective and 128 normal apples for ‘Delicious’ and 156 defective and 94 normal apples for ‘Golden Delicious’. Figure 6.11. Photograph of a ‘Golden Delicious’ apple with bitter pit (left) and a ‘Delicious’ apple with frost ring (right).   174 6.3.1.2. Image Acquisition A multispectral SIRI system, as schematically shown in Figure 6.12, was used to acquire images from apples. Compared with the multispectral SIRI system used in Section 6.2 above, this system used a back-illuminated EMCCD camera (PhotonMax: 1024B, air-cooled, Princeton Instruments, Trenton, NJ, USA) with a resolution of 1024×1024 pixels, attached with a different focusing lens (35 mm, Edmund Optics Inc., Barrington, NJ, USA). The camera was set up vertically while the projector was slightly (about 15 degrees) tilted with respect to the vertical axis. Two phase-shifted sinusoidal patterns (with phase offsets of 0 and 2π/3, respectively), were used for sample illumination. Image acquisitions were performed using a custom-developed GUI program in LabVIEW 2016 (National Instruments, Austin, TX, USA). The exposure time was set to 300 ms, and on-chip 2×2 binning operations were implemented. A more detailed description of the system is given in Section 3.2.   Figure 6.12. Schematic of a multispectral structured-illumination reflectance imaging system. Selection of proper spatial frequencies for the illumination patterns is critical for achieving optimal performance. According to Chapter 3 and the work done by Li (2016), higher frequency is more conducive to enhancing surface features of fruit, including normal lenticels, and meantime decreasing the SNRs of resulting images. Based on a preliminary test on a range of spatial frequencies of 0.05-0.3 cycles/mm, the frequency of 0.15 cycles/mm was chosen in this study,   175 which would enhance both surface and subsurface features, while suppressing the lenticel features. Waveband selection is also important for effective defect detection. According to Section 6.2, the spectral images above 850 nm suffered strongly from specular reflectance and were thus not considered. A preliminary study on comparing 10 wavebands in the range of 650-830 nm with 20 nm increments, showed that the NIR bands at 710 nm and above, which were insensitive to color variations of fruit skin, had similar performance in recognition of many types of defects. However, the waveband at 730 nm was selected in this study, because it had the highest spectral response, which is also in agreement with many previous studies using 740 nm for apple defect detection (Bennedsen and Peterson, 2005; Bennedsen et al., 2007; Throop et al., 2005). 6.3.1.3. Image Preprocessing Image preprocessing for the raw SIRI images consisted of three steps, as shown in Figure 6.13. First, the acquired pattern images were demodulated into DC and AC images, using the two-phase demodulation approaches as done in Section 6.2. Next, object masking was performed on the DC images alone using a unimodal histogram based automatic thresholding technique. The unimodal thresholding is to find a threshold based on maximizing the distance to a straight line drawn from the histogram peak to its last non-empty bin position, which is described in detail in Appendix B. Finally, since the derived DC and AC images still suffered from the artifacts of noise and vignetting (i.e., an uneven illumination phenomenon featured by gradual fading out of the image intensity towards its periphery), image enhancement was performed through BEMD and subsequent image reconstruction to remove or minimize these artifacts. Figure 6.13. Flow chart of image preprocessing procedures, where DC and AC represent direct and amplitude components, respectively. BEMD is a 2-D extension of the empirical mode decomposition (EMD) method, which was 176   originally proposed by Huang et al. (1998) for nonlinear and non-stationary time series analysis. Different from classic Fourier and wavelet analysis, BEMD requires no a priori basis for data decomposition, and it is empirical, adaptive and fully data-driven (Huang and Shen, 2014). In this study, BEMD was utilized to decompose the demodulated DC and AC images through an iterative sifting process into a number of intrinsic mode function (IMF) images plus a residual (the number of IMFs was limited to 6 in this study). The resulting IMFs and the residual images provided a representation of the original image at varied frequency scales. By selectively discarding the first IMF and the residual from the raw image, an enhanced image with vignetting and noise removed could be reconstructed, which will be demonstrated later. A detailed description of the principle and implementation of BEMD is given in Appendix C and Lu and Lu (2018) . According to Section 6.2, the ratio (RT) images, which are obtained by dividing AC by DC, were useful for bruise detection of apples, because they enhanced image contrast for bruises and also had the effect on the correction of illumination vignetting. Here, the ratio image was derived from the original DC and AC images with their first IMF eliminated, which was equivalent to that calculated from the original images followed by image de-noising. In addition, the ensemble of DC and AC images, and the ensemble of DC, AC and ratio images were also exploited as inputs for image classification. 6.3.1.4. Classification by RF and SVM Features Extraction Image classification with RF and SVM requires the images to be represented by a number of discriminative features. In many studies for fruit defect detection, textural features were used for image classification (Kavdir and Guyer, 2008; Unay et al., 2011). Hence in this study, three types of textural features were extracted, which included 28 Haralick textural features, 59 local binary   177 pattern (LBP) features and 67 Gabor features from each image. In addition, 6 basic intensity features and 7 Hu-moments features were also extracted, which resulted in a complete set of 167 features for each image. The Haralick features were calculated for the mean and range of 14 different textural measures based on the co-occurrence matrix with a 3-pixel distance over 4 directions defining co-occurrence (Haralick et al., 1979). The LBP features were calculated from 58 uniform LBPs and one non- uniform LBP (Pietikainen et al., 2011), in which the uniformity of LBPs is defined based on the occurrence of at most 2 bitwise transitions when circularly sampled in a neighborhood of 3×3 pixels (Ojala et al., 2002). For the Gabor features, the image was filtered with a bank of Gabor filters generated with 8 dilations and 8 rotations as a spatial mask of 21×21 pixels, and the average magnitudes of the filtered images were taken as the Gabor features, in addition to the minimum, maximum and normalized difference of the magnitude (Kumar and Pang, 2002). The basic intensity features included mean, standard deviation, kurtosis, skewness and mean gradient and mean Laplacian (Nixon and Aguado, 2012). The Hu-moments features used are invariant shape descriptors (Hu, 1962). A more detailed description of all these features and their computations can be found in Nixon & Aguado (2012) and references therein. The extracted features were concatenated in tandem for each image to form a features vector. In the case of two ensembles of input images (i.e. DC-AC and DC-AC-RT), the features extracted from each channel (i.e., DC, AC or RT image) were concatenated as well. For the ensemble of DC and AC images, for instance, the length of the feature vector was 334 (i.e., 2×167). Since each type of features had different scales, feature-wise normalization was done so that all normalized features had a zero mean and a unit variance.   178 Classification The image classification was performed for the two varieties of apples, separately. For each variety, the dataset was randomly partitioned into training and test sets according to a ratio of 7:3, resulting in 223 training and 95 test samples for ‘Delicious’ apples and 175 training and 75 test samples for ‘Golden Delicious’ apples. RF is a learning algorithm that uses an ensemble of unpruned decision trees, and each of the tree is built based on a subset (i.e., 2/3) of training data created through bootstrap sampling with replacement [known as bootstrap aggregating or bagging, and the left-out samples are called out- of-bag (oob) data] using a subset of randomly selected variables or features (Breiman, 2001). Implementation of RF would require to tune two parameters, i.e., the number of grown trees and the number of randomly selected variables. The preliminary tests, as exemplified in Figure 6.14, suggested that the classification error decreased rapidly and then leveled off without overfitting, when the number of grown trees exceeded a certain value, and it also was insensitive to the number of features used. Hence, fine-tuning of the two parameters was not necessary. In the subsequent analysis, the number of trees was chosen to be 200, and all the features were used in all training cases (so, the RF algorithm in this study was different from the standard version with a subset of randomly selected features). Figure 6.14. (a) The out-of-bag error against the number of grown trees, and (b) the test error against the number of selected features, using the image input of the ensemble of direct component (DC), amplitude component (AC) and ratio images for two varieties of apples.   179 SVM comes in many different versions with or without using kernel tricks to solve linear or nonlinear classification problems. In this study, a linear soft-margin SVM (Cortes and Vapnik, 1995) was chosen for the image classification, given that the two-class (defect versus normal) classification could be reasonably treated as a linear problem. In the SVM, only one parameter needed to be tuned, i.e., the regularization parameter that controls the trade-off between maximizing the margin and minimizing the misclassification. Ten-fold cross validations were performed here to decide the parameter, which was sampled over a wide range of 0.001 to 1000 based on the minimum classification error. Given the randomness of data partition, each of the above training cases was repeated 30 times. In each round of training for each apple variety, the sample indices for the training and test sets were shared in training the 5 different image inputs (i.e., three sets of DC, AC and RT images and two combination sets) with the three different classifiers, including the CNN that is presented next, to ensure fair comparison of the results for the different image inputs and the three classifiers. The above described procedures for image preprocessing, features extraction and model training with RF and SVM were performed in Matlab (The Mathworks, Inc., Natick, MA, USA). 6.3.1.5. Classification by CNN CNN, which was originally introduced by LeCun and the colleagues (LeCun et al., 1989; LeCun et al., 1998) for handwritten digit recognition, has a specially designed network architecture for image classification. A typical CNN architecture embodies three major design characteristics, including local receptive field, shared weights and pooling (or sub-sampling), contributing to the sensitivity to local features and robustness to deformations and shifts, and also reducing learnable parameters as well. As illustrated in Figure 6.15, a convolution operation is applied to a local receptive field of 3×3 pixels in size (i.e., the convolution kernel size) in the input image, which is   180 connected to a hidden neuron of the first hidden layer, and sliding the local receptive field across the entire image step by step (by one pixel here) builds up the first hidden layer, which is defined by a number of feature maps. Then maximum pooling operations follow, which downsize the input feature maps by outputting the maximum in a local neighborhood of 2×2 pixels. In performing convolutions, all weights (and bias) in the convolution kernel are shared for building each of the feature maps. A CNN usually has more than one convolution-pooling layer, followed by one or more conventionally fully-connected layers before the final output. Figure 6.15. The architecture of a convolutional neural network (CNN). Like other deep neural networks, CNN, which contains a large number of learnable parameters, needs a large set of images for model training so as to avoid overtraining. Preliminary attempts showed that the present image data were inadequate for the CNN to produce satisfactory accuracy. To address the problem, data augmentation was performed to increase the number of training images by applying various geometric transformations to the original image, which is common practice for training CNNs with limited data (Lu, 2016; Simard et al., 2003). The geometric transformations included translation, rotation, shear, zooming and flipping, among which the first four were done by uniformly applying a random displacement within a preset range, and flipping (both horizontal and vertical) was randomly performed. Figure 6.16(a) shows an example of five images generated through these transformations. As a result, the training samples were augmented for ‘Delicious’ apples by five times to 1,115 images and for ‘Golden Delicious’ to 875 images. All   181 augmented images were resized to a standard size of 320×320 pixels based on the fruit object size, which were then normalized to have zero mean and unit variance before training. Figure 6.16. (a) Five augmented images generated by geometrical transformations of an original image (top left) and (b) Test accuracy and the number of learnable parameters versus the number of convolution-pooling layers in the convolutional neural networks using the ensemble of direct component, amplitude component and ratio images for two varieties of apples. Training a CNN model requires much effort and skill to configure numerous hyper-parameters, including learning rate, training epoch, batch size, activation nonlinearity, number of hidden layers, convolution kernel size, pooling size, dropout, etc. (Bengio, 2012). This study was not intended to optimize all these parameters, but instead, it focused on the number of convolution-pooling layers (relating to the depth of a CNN), which plays a crucial role in model performance and also affects the model complexity in terms of the number of learnable parameters. For each convolution- pooling layer, 32 convolution filters (resulting in 32 features maps) of size 3×3 pixels with a stride step of 1 pixel were used, followed by 2×2 max-pooling, as shown in Figure 6.15. A rectified linear unit (ReLU) was used as the activation function for all the hidden layers, and ‘softmax’ for the output (dense or fully-connected) layer with ‘categorical_crossentropy’ used as the corresponding loss function. The optimizer for model training was ‘adadelta’ (Zeile, 2012) with a starting learning of 1, and a dropout trick was used following the last pooling layer and the output layer, with dropout rates of 0.25 and 0.5, respectively. A learning rate updating scheme was implemented, which reduced the learning rate by half when the test accuracy did not improve for 5 successive   182 epochs. The training process would be terminated if there was no accuracy improvement for 15 successive epochs. The maximum number of training epochs was fixed to 50, which was found to be sufficient in this study. Based on these settings, 1-6 convolution-pooling (i.e., hidden) layers were explored for CNN modeling. As shown in Figure 6.16(b), accuracies of more than 90% were obtained by using 3 or more hidden layers, and the accuracies started to converge or remain stable when the number of hidden layers exceeded 5. It was noted that increasing the hidden layers greatly reduced model parameters, and 5 hidden layers gave the minimum parameters (~ 40k without considering neuron dropout), which were therefore chosen for the CNN modeling for the remaining analysis. The above described image classification procedures with CNN were performed by using Python with the deep learning packages of Keras (Chollet, 2015) and Theano (Bergstra et al., 2010) in a desktop computer. 6.3.1.6. Performance Evaluation Image classification results for the test dataset were evaluated using three metrics, i.e., precision (PRE) and recall (REC) and overall accuracy (ACC) (Marsland, 2015): PRE  REC  # TP TP #  # FP # TP # TP  # FN  100%  100% ACC  # TP # TN   # # TN FP # TP   # FN  100% (6.6) (6.7) (6.8) where # means ‘number of’, TP and TN represent true positives (i.e., defective samples that were classified as defective) and true negatives (i.e., normal samples that were classified normal), respectively, and FP and FN represent false positives and false negatives, respectively. PRE is the percentage of TP samples of all the samples classified as positive, where REC is the percentage of   183 TP samples of all the positive samples (equivalent to the true positive rate). The three metrics were calculated from the 30 training replicates. Moreover, statistical comparisons were conducted on ACCs for the different image inputs and classifiers, using the Fisher’s least significance difference (LSD) procedure, coupled with the ANOVA test, at the 5% significance level. 6.3.2. Results and Discussion 6.3.2.1. Image Enhancement by BEMD Figure 6.17 shows an example of the decomposition of a single DC image into six IMF images and a residual image by using BEMD. The first several IMFs captured well the high-frequency features of the original image, while the last IMFs and the residual embodied its low-frequency features and overall trend. In particular, the first IMF could be seen as a noise component, which carried no meaningful information, while the residual basically corresponded to the illumination vignetting. Therefore, an enhanced DC image could be reconstructed by simply eliminating the first IMF image and the residual image from the original one (i.e., summation of the IMF images in between). The enhancement was applicable to AC images as well. Figure 6.17. Bi-dimensional empirical mode decomposition of a direct component (DC) image into six mode images and one residual image. As shown in Figure 6.18, compared with the raw DC and AC images, their reconstructed counterparts possessed greatly improved uniformity and contrast, thus enhancing the clarity of   184 image features of interest, such as defects. Hence, the BEMD-enhanced DC and AC images were used instead of the raw images in further image analysis for all three classifiers. Figure 6.18. Comparison of the original and reconstructed direct component (DC) and amplitude component (AC)   6.3.2.2. Enhanced DC, AC and RT Images images. Figure 6.19 shows examples of DC, AC and RT images, after the BEMD enhancement, for several defective ‘Delicious’ and ‘Golden Delicious’ apples. The DC and AC images revealed different types of defect features for these apples, while AC and RT images looked similar. The defects, such as frost ring and scab, occur on the surface, while bitter pit that develops beneath the fruit surface but is externally visible, and these surface defects could be well revealed in the DC images, On the other hand, the AC images were more effective in ascertaining subsurface bruising/injury and rot, and certain russeting defect as shown in Figure 6.18. Figure 6.19. Example of direct component (DC, top row), amplitude component (AC, middle row) and ratio (RT, bottom row) images of ‘Delicious’ (the left three columns) and ‘Golden Delicious’ (the right three columns) apples with different types of defects. It should be noted that, unlike fresh bruising which would appear as darkened areas, the bruises 185   encountered in this study had been in existence for a prolonged period of time, resulting in brighter appearances due to the desiccation of the bruised tissue. Many defective apples had more than one type of defect, some of which were undetected in DC or AC images alone. Hence, the ensemble of DC, AC and RT images could have great potential to increase discriminative features and thus classification accuracies. 6.3.2.3. Classification Results Figure 6.20 shows the results of PRE, REC and ACC by the three classifiers of RF, SVM and CNN for all five forms of image inputs (i.e., DC, AC, RT, DC-AC and DC-AC-RT) for ‘Delicious’ and ‘Golden Delicious’ apples. ‘Delicious’ ACC PRE REC DC AC RT DC-AC DC-AC-RT ACC PRE REC DC AC RT DC-AC DC-AC-RT ACC PRE REC 97 93 89 85 81 97 93 89 85 81 97 93 89 85 81 97 93 89 85 81 97 93 89 85 81 97 93 89 85 81 ‘Golden Delicious’ ACC PRE REC DC AC RT DC-AC DC-AC-RT ACC PRE REC DC AC RT DC-AC DC-AC-RT ACC PRE REC DC AC RT DC-AC DC-AC-RT DC AC RT DC-AC DC-AC-RT Figure 6.20. Classification accuracies (in percent) obtained by random forest (top pair), support vector machine (middle) and convolution neural network (bottom) for the five forms of image input for the test group of ‘Delicious’ (left pane) and ‘Golden Delicious’ (right pane) apples, where ACC, PRE and REC denote overall accuracy, precision and recall, respectively; DC, AC and RT denote direct component, amplitude component and ratio images, respectively; and DC-AC and DC-AC-RT represent the two ensembles of two or three types of images.   186 Detection accuracies, as measured by PRE, REC and ACC, varied with image input and classifier. The three performance metrics had similar trends of change with the image input. Among the three metrics, ACC had the most conservative results, which indicated no accuracy paradox (i.e., good accuracy but bad predictive power with lower REC and PRE) in the classification problem. Hence it is safe to use only ACC for the model performance comparisons in the following discussion. Figure 6.21 shows statistical comparisons of the classification results (in terms of ACC) by the three classifiers for the five forms of image inputs. DC and AC images did not consistently outperform each other, although DC had better results in 4 out of the six cases. These results are in general agreement with the study of Li (2016) that AC did not offer a distinct advantage over DC for surface defect detection of apples. RT images, however, compared favorably to both DC and AC images, except for one case when using RF for classifying ‘Delicious’ apples. ‘Delicious’ DC AC RT DC-AC DC-AC-RT a b bc b b c a ab c d b c c a a RF SVM CNN 97 93 89 85 81 97 93 89 85 81 ‘Golden Delicious’ DC AC RT DC-AC DC-AC-RT a a a a b b c ab a b b b c d c RF SVM CNN   Figure 6.21. Comparisons of overall accuracies (in percent) by the three classifiers of random forest (RF), support vector machine (SVM) and convolutional neural network (CNN) for the five forms of image inputs for ‘Delicious’ and ‘Golden Delicious’ apples. DC, AC and RT denote direct component, amplitude component and ratio images, respectively, and DC-AC and DC-AC-RT represent two ensembles of DC and AC images and DC, AC, and RT images, respectively. Classification results in the same group with different letters are different at the 5% significance level. Further, combining DC and AC images, i.e., concatenating the relevant features for the three classifiers of RF, SVM and CNN, improved the classification over using either DC or AC alone, statistically significantly in 3 out of the six cases. The overall improvements ranged between 2%-   187 5%, which thus confirmed the hypothesis that the combination of AC and DC images provides additional discriminative features for enhanced defect detection. The combination of DC and AC images also compared favorably to RT images, resulting in significant improvements in 4 of the six cases. Furthermore, the ensemble of the three sets of images (i.e., DC-AC-RT) for the three classifiers achieved significant improvements in all cases for both varieties, when compared with AC, DC and RT alone, and it also significantly outperformed the ensemble of DC-AC images when using SVM for classifying ‘Delicious’ apples. Further comparisons for the three classifiers are shown in Figure 6.22. Overall, CNN was the best classifier; it was significantly better than RF for all 10 groups of comparisons, and also significantly better than SVM for 6 out of the 10 groups. When the ensemble of DC, AC, and RT images was used, CNN achieved overall accuracies of 98% for both varieties of apples. SVM performed significantly better than RF in all cases, except for the DC of ‘Delicious’ apples. SVM achieved the highest accuracies of 94.7% and 97.5% for ‘Delicious’ and ‘Golden Delicious’ apples, respectively, when using DC-AC-RT. The plausible reasons for the poorer performance of RF could be due to the relatively intuitively linear two-class classification and a smaller training dataset used in the study, which worked better for a linear SVM than for an RF that is intrinsically suited for multi-class problems involving a large number of training instances. It was also noted that RF gave relatively larger standard errors (i.e., longer error bars in the figure) than those obtained by SVM and CNN, which could be due to the limited training data for implementing the randomization in the RF algorithm.   188 ‘Delicious’ RF SVM CNN a b b 97 93 89 85 a a b a b c a b c a b c ‘Golden Delicious’ RF SVM CNN a a b a b c 97 93 89 85 a b c b a a a a b DC AC RT 81   Figure 6.22. Comparisons of random forest (RF), support vector machine (SVM) and convolutional neural network (CNN) in terms of overall test accuracies (in percent) for the five different image inputs for ‘Delicious’ and ‘Golden Delicious’ apples. DC, AC and RT denote direct component, amplitude component and ratio images, respectively, DC-AC DC-AC-RT DC-AC DC-AC-RT and DC-AC and DC-AC-RT represent the two ensembles of DC and AC and DC, AC, and RT, respectively. Classification results in the same groups with different letters are different at the 5% significance level. AC RT DC 81 With the input of the ensemble of DC-AC-RT images, CNN gave RECs of 97.6% and 97.4% for ‘Delicious’ and ‘Golden Delicious’ apples, respectively, indicating less than 3% of (i.e., at most 2 on overage) defective apples misclassified as normal. Figure 6.23 shows DC, AC and RT images for the two most frequently misclassified apples for each variety, in the 30 training replications with random data partitioning. Figure 6.23. The direct component (DC, top row), amplitude component (AC, middle row) and ratio (RT, bottom row) images for the two defective ‘Delicious’ (the left two columns) and ‘Golden Delicious’ (the right two columns) apples which are misclassified by CNN using the ensemble of DC-AC-RT image input.   It could be seen that these apples contained only slight defects (i.e., old bruise, russeting and   bitter pit) and presented weak image contrast for the DC, AC and RT images, thus resulting in   189 their misclassifications. It should be mentioned that the wavelength used in the study (i.e., 730 nm) was not optimized for detecting the types of defects observed in the apple samples, and hence it may not represent the optimum wavelength for enhanced detection. 6.3.2.4. Discussion The classification results for the two varieties of apples, obtained by the three machine learning classifiers, demonstrated that using the ensembles of DC-AC and DC-AC-RT resulted in significant improvements in the overall detection accuracy, compared with using them separately. While AC images have been shown to be much more effective in detecting subsurface defects such as bruising, which are externally invisible, DC images are generally better in detecting surface defects of apples. This further confirmed that DC and AC images provide different image features and hence are complementary and synergistic for defect detection. The classification results from this research also compare favorably with those reported in previous studies using conventional color or NIR imaging techniques. Unay et al. (2011) reported an overall classification accuracy of 93.5% for the two-class classification of ‘Jonagold’ apples using SVM combined with features selection. In classifying ‘Fuji’ apples, Zhang et al. (2015) obtained an accuracy of 95.6% using weighted relevance vector machine with features selection. Recently, Moallem et al. (2017) reported an accuracy of 92.5% for ‘Golden Delicious’ apples using SVM. It should be noted that in our study, features selection was not implemented in using RF and SVM, which could further boost the accuracy. Hence, SIRI can be advantageous for fruit defect detection over conventional imaging technique under uniform illumination. CNN provides a potent tool for image classification given its superior performance and unique advantage of integrating features extraction and classification. However, training a CNN model is by no means an easy task, because of its network architecture and many hyper-parameters that   190 need considerable tuning effort and long training time. In this study, simplifications were made to focus on the number of convolution-pooling layers so as to speed up computations and thus ease the training task. Unlike SVM (and/or RF), CNN generally requires much more training samples per class, which may be problematic in preparing datasets since it can be difficult to collect sufficient samples for all types of defects (with varied severity). This was also the main reason that only two-class classification was performed in this research. Although this issue could be alleviated by artificial data augmentation, as done in this study, it is highly desirable and also beneficial to collect more real samples to realize the full potential of CNNs. As aforementioned, this study only dealt with a two-class classification problem. In practical applications, it would be more desirable and valuable to perform multi-class (e.g., multiple levels of defect severity), even defect-specific classifications and defect segmentation, which should be investigated in future research. In utilizing DC, AC and RT images, there are also other ways for integrating them for enhanced detection. For example, one may build up models using them separately, and make classifications based on certain voting rules as done in ensemble learning. 6.4. CONCLUSIONS This chapter first presents the application of a multispectral SIRI system for detection of fresh subsurface bruising in apples of four varieties. Immediately after impact bruising, apples were imaged by the system in the visible and NIR range of 650-950 nm with 20 nm increments, under the illumination of two phase-shifted sinusoidal patterns at a spatial frequency of 0.10 cycles/mm. The obtained amplitude images clearly revealed bruises, while the conventional uniform illumination images could not have done so. The application of PCA to the multispectral images showed that the seven wavelengths between 710 and 830 nm were more relevant to bruise detection. The spectral images for 710-810 nm in 20 nm increments, achieved good, similar   191 detection results with overall detection errors of 11.7% to 14.2%, when a modified Otsu thresholding algorithm was applied. This study clearly demonstrated the superior performance of SIRI in detecting fresh bruises in apples, which is usually a difficult task for conventional uniform illumination based modalities. It also provides a basis for using a multispectral SIRI system for detecting fruit defects and shows the feasibility of detecting fresh bruises in apples by implementing monochromatic SIRI at wavelengths of 710 to 810 nm. Further research will focus on improvement of the image segmentation for more reliable and accurate bruise detection, the development of hardware and image processing algorithms for detecting bruises of varied ages and other types of defects for apples. The second application of SIRI was focused on detection of general defects for apples. Two varieties of apples with various surface and subsurface defects were imaged by SIRI with two phase-shifted sinusoidal patterns at 0.15 cycles/mm, at a single NIR waveband of 730 nm. BEMD was proposed to preprocess the demodulated DC and AC images for image enhancement by discarding their components associated with noise and illumination vignetting, and the enhanced images were utilized by the three pattern classifiers for classifying defective apples. While AC and DC images showed their respective advantages for detecting certain types of defects, they resulted in lower overall defect classification accuracies when used separately; however, by combining DC and AC and or their ratio images, the three classifiers yielded improved classification accuracies, with improvements of up to 5%. Among the three employed classifiers, CNN, which was trained with artificially augmented images, resulted in the best classification, with the highest overall detection accuracies of 98% for both varieties of apples, followed by SVM and RF that were both trained with hand-crafted image features. Since SIRI enables acquiring both DC and AC images, which provide different, complementary discriminative features, the technique, coupled with an   192 appropriate machine learning algorithm, has potential to be a new, effective, and versatile modality for defect detection of apples and other fruits.   193 7. EXPLORATION OF SIRI FOR DETECTION OF MOVING SAMPLES 7.1. INTRODUCTION While the studies described in the previous chapters have shown the effectiveness and versatility of SIRI for defect detection of apples, they were conducted when the apples were kept stationary during image acquisition. Further research is thus needed on detection of moving samples to evaluate the potential of SIRI for practical applications. Real-time implementation of SIRI, compared to uniform illumination based modalities, requires more dedicated instrumentation and analysis to address the need of acquiring more than one pattern image per sample and pixel-based image demodulation. Our studies showed that it is preferable to acquire two phase-shifted patterns from each sample so as to balance the imaging speed and demodulation accuracy. However, a problem may arise for moving samples, due to the mismatch of the fruit object in two acquired pattern images, which could result in demodulation artifacts, depending on the speed of image acquisition and sample movement. There are several possible approaches to real-time implementation of SIRI. The most intuitive approach is to perform a ‘go-stop-go’ running mode, in which the sample is continuously moving unless it enters the field of view of interest. The approach is, however, problematic in practical situations. A second approach is to have the sample move without a stop, fix the illumination pattern and acquire pattern images from the moving sample. In this approach, the phase shift is introduced to the acquired patterns, from the perspective of the sample, due to sample movement; and ad-hoc image processing is needed to align the regions of interest in (two) acquired pattern images for image demodulation. There is another way for real-time SIRI by synchronizing the digital light projector (DLP) and camera, both of which run at a sufficiently high speed. In this case, the issue of sample mismatch between two consecutively acquired images may be negligible,   194 when the sample moving speed is relatively slow, compared to the imaging speed. The DLP in the present SIRI system supports a pattern switching rate of up to 500 Hz for 8-bit grayscale patterns, which would be fast enough for real-time defect detection of apples [a more advanced DLP (e.g., DLP7000 by Texas Instruments) enables an even faster pattern speed than at 1,900 Hz or higher for 8-bit gray]. However, the camera employed for SIRI, either the SensiCam CCD (Cooke Corporation, Auburn Hills, MI, USA) or the EMCCD (PhotonMAX: 1024B, air-cooled, Princeton Instruments, Trenton, NJ, USA), had a frame rate of no more than 10 frames per second (fps) for full-resolution image acquisition, which excluded the choice of this approach in this study. This study represents a proof-of-concept exploration of SIRI for detection of moving samples. Specific objectives of the study were three fold: 1) to improve the software platform for controlling the sample movement by using a motorized sample stage; 2) to explore the feasibility of using a single fixed sinusoidal pattern for detection of moving samples by SIRI, in which two phase- shifted patterns due to the movement of the sample are acquired and used for image demodulation; and 3) to explore the effect of sample motion, under illumination of the two phase-based sinusoidal patterns, on the image demodulation. Comparisons were made between the demodulated DC and AC images for moving and stationary samples. 7.2. SYSTEM SETUP As shown in Figure 7.1(left), a sample positioning unit was installed, in place of the original static stage, for SIRI for imaging a moving sample. This unit is composed of a belt-driven linear translation stage (LinTech 1846018-CP1 series, LinTech, Monrovia, CA, USA), with a loading capacity of 3810 kg and a top speed of 3000 mm/s, which is coupled with a gearhead (VC series, VC-W-065-003S-[N23-V61]-S006, GAM, Mount Prospect, IL, USA). The stage is driven by a stepper motor (Lexium MDrive LMDCM571, Schneider Electric Motion, Marlborough, CT,   195 USA), integrated with a fully programmable motion controller, which has the maximum velocity of 50 revolutions/s, corresponding to a translation speed of about 1500 mm/s for the stage. A USB to RS485/422 series interface converter was used to connect the motor to a computer. A graphic user interface (GUI) in LabVIEW 2016 (National Instruments, Austin, TX, USA), as shown in Figure 7.1(right), was developed to control the movement of the stage, which was incorporated into the original GUI for image acquisition. The GUI allows to define a starting position of the sample and its moving speed and moving manner (i.e., relatively with respect to the last position or absolutely to the zero positon), dynamically monitor the moving status, including velocity and positon of the sample, plot its travelled position curve, and so on. Figure 7.1. A structured-illumination reflectance imaging (SIRI) system with a motorized stage (left) and a graphical user interface for controlling the movement of the stage (right). In the following experiments, the SIRI system was operated in broadband mode without using         a filter device. The camera’s exposure time was set to 30 ms, with a relatively longer delay time of 180 ms for saving the acquired image to the computer’s memory (limited by the data readout speed of the camera). Sinusoidal patterns at the spatial frequency of 0.15 cycles/mm were used for sample illumination.   196 7.3. EXPERIMENTAL TESTS Six apples of ‘Gala’, purchased from a local grocery store and free of visual defects, were used for the experiments. Prior to image acquisitions, these apples were cut into halves along their stem- calyx axes, each of which was subjected to impact bruising. Bruising was inflicted onto the apple by impacting their equatorial areas using a wooden-ball equipped pendulum device, as described in Section 3.4, with the impact energy of about 0.6 J. The bruised apples were imaged by SIRI immediately after the impact test. Experiments were first conducted on five of the bruised ‘Gala’ apples to evaluate the feasibility of continuously acquiring two pattern images from the sample in motion by the SIRI under a single sinusoidal pattern. Image acquisitions were first performed for stationary apple samples under the illumination of two phase-shifted sinusoidal patterns with phase offsets of 0 and 2π/3, respectively, which provided benchmark results for comparison. Thereafter, image acquisitions were performed for moving samples at each of the five moving speeds of 4, 8, 16, 24 and 32 mm/s. It should be stressed that the moving speeds are still far too slow to meet the industrial requirements (e.g., 0.5 m/s or higher), due to the constraint in the frame rate of the camera. In addition, experiments were also conducted on one bruised ‘Gala’ apple to examine the effect of sample motion under the illumination of two phase-shifted sinusoidal patterns on image demodulation, which is important for real-time SIRI by synchronizing a high-speed projector and a comparable camera. Since the synchronization of the two devices was still not realized in the current SIRI system, simulations on their synchronization were attempted by acquiring two phase- shifted pattern images from the apple sample with different sample displacements, which were controlled by the motor. A series of sample displacements, i.e., 0, 0.1, 0.2, 0.4, 0.8, 1.6, 3.2 and 6.4 mm, were individually set for image acquisitions.   197 7.4. APPROACHES TO OBJECT ALIGNMENT In imaging a moving object, the motion would result in a degree of misalignment of the object in the two consecutively acquired pattern images. Object (i.e., fruit) alignment is therefore needed prior to image demodulation, which is performed on a pixel basis. In the current configuration of SIRI, the movement occurred in a horizontal direction. So, one only needed to estimate the horizontal displacement (in pixels) of the sample, based on which object alignment can be readily achieved by applying translation transform to either of the two pattern images. There are several possible approaches to estimating the sample displacement. Given the camera exposure time ( exposure t ), delay ( delay t ) and velocity (v) of sample moving, the sample displacement can be approximated to the product of the velocity and total elapsed time, i.e., v (2 t exposure t delay , ) which is then converted to the dimension in pixels based on the pre-predetermined image spatial resolution (~0.263 mm/pixel in the experiment). This approach, although very straightforward, is prone to errors related to timing, velocity variation and spatial resolution. Another approach is to image a reference beforehand and compare pixel coordinates between key points of interest in the reference, which can be easily identified in the two collected pattern images, to directly obtain pixel displacements under different imaging conditions. This approach, referred to as reference based, serves the purpose of calibration and is done before image acquisition for samples. In this study, the reference based approach was examined for estimation of sample displacement. A ruler of 15 cm in length, put onto the sample stage and kept parallel to the horizontal axis, was used for obtaining the sample displacement at five moving speeds in the first experiment. The reason for using a ruler as a reference is because it contains many distinctive points, in case some of the points may appear in the dark areas of the light pattern.   198 The reference based approach, however, is rigid and largely dependent on imaging setup. Following this line of thought, another approach was proposed and tested in this study, based on detection of a reference marker, which is automatic and requires no prior knowledge of imaging parameters. It can be difficult to directly estimate the sample displacement from the pattern images themselves, as shown in Figure 7.2 (a), since they generally contain no distinctive points. Figure 7.2(b) shows the pattern images acquired from a moving apple, with a white reference maker in the background. The marker was a small piece of adhesive tape, of about 3×3 mm2 in size, which was temporarily attached to the stage for the experiment. With such a reference marker, the sample displacement can be obtained by estimating the displacement of the marker based on image processing techniques, which is described in detail in the following subsection. Figure 7.2. Two consecutively acquired pattern images from a moving ‘Gala’ apple without (a) and with (a) a white reference maker. 7.5. IMAGE PROCESSING Preprocessing was done for the two acquired pattern images for a moving sample for image demodulation to retrieve both DC and AC images using the approaches described in Chapter 4. The most critical step was to estimate pixel displacement, based on which translation transform would be followed for object alignment. Estimation of pixel displacement is trivial in the first two approaches described previously, while the third approach, which relies on detecting the reference marker from two pattern images, required some special efforts, because part of the marker could be overshadowed by dark fringes of the light pattern.   199 In this study, marker detection was done by applying a series of image processing procedures to the two acquired pattern images for each sample, which included bi-dimensional empirical mode decomposition (BEMD), amplitude retrieval and thresholding. As illustrated in Figure 7.3, a fast BEMD algorithm introduced in Section 6.3 for defect detection of apples, was used to decompose a single pattern image into six intrinsic mode function (IMF) images plus one residual image. Figure 7.3. An example of bi-dimensional empirical mode decomposition (BEMD) of a pattern image of a bruised ‘Gala’ apple.   The first four IMFs carried higher-frequency information on the original image and exhibited clearer object contours. But instead of resorting to contour detection, spiral phase transform (SPT) was applied to the first four IMF images, so as to remove their pattern fringes and obtain corresponding amplitude images, as shown in Figure 7.4. These amplitude images provided more effective discrimination between the marker and its background, than the original pattern image and the corresponding IMFs. Thereafter, marker detection was accomplished by applying a thresholding technique to the amplitude images.       200 Figure 7.4. Spiral phase transform (SPT) results of the first four intrinsic mode function (IMF) images shown in Figure 7.3 (from the left to the right). It was noted that the contour of the square marker in the recovered amplitude images became more blurred than that in the corresponding IMF images, which is due to the fact that the fringe patterns in the IMF images are not strictly sinusoidal. It was also noted that the bruise defect in the apple also appeared in the recovered amplitude images, with weak contrast. Such amplitude images were much inferior in overall image quality, to those obtained by demodulation of two phase- shifted pattern images, and hence were not suited for defect detection applications. The amplitude image derived from the second IMF was chosen for further analysis. Given the fact that the reference marker in the acquired images had an almost fixed vertical position, a simplification was made by confining the area, which would be subjected to thresholding, to the bottom part of the image, excluding the apple region. It followed that the resulting image histogram for the reduced area was predominantly unimodal (due to the dissimilar size between the marker and surrounding darker background), and that one could easily segregate the marker via a unimodal thresholding method as described in Appendix B. The pixel displacement of the marker was estimated to be the difference of the horizontal coordinates of its centroids in the two pattern images, and then, one of the two pattern images was translated accordingly, to be lined up with the other in terms of the fruit object. Finally, SPT was applied to the updated pattern images, to retrieve both DC and AC images as done in Chapter 4. Figure 7.5 presents a complete flowchart of the proposed image processing procedures.   201 It should be mentioned that the phase shift for the resultant pattern images is unknown. The AC retrieval was performed by using Eq. (4.13), which is applicable to two pattern images of arbitrary phase shifts, while the DC retrieval, according to Eq. (4.17), requires knowledge of phase information that is related to the phase difference of the two patterns, which was estimated by using Eq. (4.18). Figure 7.5. The flowchart of image processing procedures for retrieving direct component (DC) and amplitude component (AC) images from two pattern images acquired from a moving sample. SPT and IMF denote spiral phase transform and intrinsic mode function.   202 7.6. IMAGE QUALITY ASSESSMENT To evaluate the feasibility of the proposed approaches for detection of moving samples, comparisons were made of the resultant DC and AC images for the same sample imaged under both stationary and moving conditions. With the images for the stationary sample used as a reference, a universal image quality index (QI) (Wang and Bovik, 2002) was introduced for image quality assessment and comparison between the stationary and the moving samples. The QI is mathematically defined as follows: QI  4 x y  xy )[( ) ( x 2 2   y  2 x  ( ) ] y 2 (7.1) where x and y represent the reference and the test (corresponding to the moving sample) images, respectively; and x and y represent the mean values for the reference and test images, respectively; and 2 x and 2 y represent the variances for the reference and test images, respectively; and xy represents covariance of the reference and the test images. This quality index models various image distortions as a combination of three different factors, including loss of correlation, luminance distortion and contrast distortion, which, compared to conventional mean squared error (MSE) based quality metrics, has a strong ability in measuring information loss and correlates very well with the subjective visual inspection (Wang and Bovik, 2002). The QI, has a dynamic range of [0, 1] (excluding a case where the two images under study are negatively related). A higher QI value indicates a better image quality (assuming the reference image is a perfect one). Given that image quality is often space variant, the QI was calculated locally and then combined together as follows (Wang and Bovik, 2002):   QI  1 M  M  1 j QI j 203 (7.1) where a total of M local quality jQI was calculated and averaged as a single overall quality value. The jQI for local regions was obtained by applying a sliding window (of 32×32 pixels in size in this study) that moved pixel by pixel horizontally and vertically across the whole image. 7.7. RESULTS AND DICUSSION Figure 7.6 shows an example of the DC and AC images obtained from two phase-shifted pattern images for a stationary ‘Gala’ apple with subsurface bruising. As demonstrated previously, the DC image failed to reveal the bruise, which, however, was well resolved in the AC image. DC AC Bruise Figure 7.6. The direct component (DC) and amplitude component (AC) images for a stationary ‘Gala’ apple with subsurface bruising. Figure 7.7 shows the DC and AC images for the same apple imaged at different moving speeds. The object alignment was done prior to image demodulation, by using the (ruler) reference-based and automatic marker detection based approaches to estimating the pixel displacement. Both approaches gave acceptable and comparable images in quality in all the cases, affirming their validity, except that the AC images deteriorated with noticeable fringe residuals at the highest moving speed of 32 mm/s. The QIs were determined for these images, with the corresponding DC and AC images (as shown in Figure 7.6) for the stationary sample used as references. Overall, the quality of images deteriorated with the increase in the sample moving speed, except that the DC images at 32 mm/s reversed the trend; and the AC images deteriorated more rapidly than the DC images. The sample moving speed at 32 mm/s resulted in AC images with the lowest QIs, which accorded with their visually low-quality appearances, while all other lower moving speeds gave decent images with QIs exceeding 0.75 for both DC and AC images. Similar results were also   204 observed for other apples. Given a relatively large drop of QIs for the AC images at the speed of 24 to 32 mm/s, an upper-bound sample moving speed, for the current hardware setup of SIRI, was presumably within this speed range, without greatly sacrificing the quality of resultant images. Figure 7.7. The direct component (DC) and amplitude component (AC) images, obtained by using the reference based approach (a) and based on automatic image processing (b) for object alignment, for a moving ‘Gala’ apple with subsurface bruising, at the moving speeds of 4, 8, 16, 24 and 32 mm/s (from left to right). The resultant quality index (QI) for each image was calculated, and all the images were resized to the region focusing on the fruit object Further, Figure 7.8 shows the resultant DC and AC images for the other four bruised ‘Gala’ based on unimodal thresholding of the DC image. apples at a moving speed of 24 mm/s (assumed to be the highest, acceptable speed in this study). Likewise, the DC images had consistently higher QIs than the corresponding AC images, which, however, enabled clear detection of bruised regions that were invisible in the former. And the two proposed approaches to pixel displacement estimation resulted in similar images.   205 Figure 7.8. The direct component (DC) and amplitude component (AC) images, obtained by using the reference based approach (a) and based on automatic image processing (b) for object alignment, for four ‘Gala’ apples with subsurface bruising, at a moving speed of 24 mm/s. The resultant quality index (QI) for each image is shown above each image, and all the images were resized to the region focusing on the fruit object based on unimodal All these findings have shown that it is viable to detect moving samples by SIRI by acquiring thresholding of the corresponding DC images. two pattern images under the illumination of a single fixed pattern, and confirmed the validity of the proposed approaches to determining pixel displacement for object alignment. Given the limited moving speed, these results only represent proof of the concept. It should, however, be noted that the sample moving speed depends on the image acquisition speed. The increase in the latter would produce a proportional increase in the former. For instance, if the image acquisition is completed within 10 ms (including exposure and delay times) for a sample (240 ms in the current imaging setup), it is possible to run the SIRI system at a speed of 0.5 m/s (assuming 24 mm/s represents the upper-bound moving speed in this study), which is a reasonable speed for many practical   206 applications. In light of this, the concept of implementing SIRI under a single illumination pattern has the potential for practical applications, if it is used with a higher-speed camera. In addition to using a single fixed sinusoidal pattern, another possible way for detecting moving samples by SIRI is to project two phase-shifted patterns sequentially and synchronously acquire each of the two pattern images. Direct demodulation of the acquired pattern images, however, would incur artifacts for the resulting DC and AC images, due to sample motion. Figure 7.9 shows the demodulation results in the second experiment for the bruised ‘Gala’ apples imaged without and with sample displacements. In the case of no and slight (e.g., no more than 0.8 mm, corresponding to three pixels here) sample displacements, one could obtain decent DC and AC images, without performing object alignment (for the motion correction); while on the other hand, as the sample displacement increased further, the pattern residuals became more and more serious, rendering bruise detection impossible. DC AC (a) DC AC (b) Figure 7.9. The direct component (DC) and amplitude component (AC) images for a bruised ‘Gala’ apple under the illumination of two phase-shifted patterns: (a) no sample displacement and (b) different sample displacement (D) ranging from 0.1 to 6.4 mm, for the two phase-shifted pattern images. It can be seen that, the motion artifacts were negligible when the sample displacement was sufficiently small (less than or equal to 0.8 mm in this experiment), which implies that real-time SIRI can be achieved under the illumination of two phase-shifted patterns, by means of   207 synchronizing pattern projection and image acquisition. However, with small sample displacements for achieving acceptable demodulation results, high pattern projection and image acquisition rates are required, if a fast sample moving speed is desired. For instance, assuming a maximum sample displacement of 1 mm (based on the above results) and a sample moving speed of 0.5 m/s (acceptable for practical applications), then the image acquisition needs to be done within 2 ms (for acquiring two pattern images for each sample), corresponding to a pattern projection or image acquisition rate of at least 1000 Hz, which is within the reach of the state-of- the-art technology. Furthermore, the motion artifacts can be reduced or mitigated to allow for a larger sample displacement, by performing object alignment before image demodulation, as done in the first experiment by using a single fixed illumination pattern. Figure 7.10 shows the demodulation results after object alignment was performed onto the raw pattern images. Here, the automatic marker detection based approach (as described in Figure 7.5) was utilized to estimate the pixel displacement for object alignment. Visually, the DC and AC images for the larger sample displacements of 0.8-6.4 mm were improved compared to their counterparts shown in Figure 7.9, especially for AC images for the two largest displacements of 3.2 and 6.4 mm, making the bruise region more apparent.   Figure 7.10. The direct component (DC) (top) and amplitude component (AC) (bottom) images for a bruised ‘Gala’ apple, with object alignment performed prior to image demodulation, for the sample displacement (D) ranging from 0.1 mm to 6.4 mm between the two phase-shifted pattern images.   208 Table 7.1 summarizes the QIs of the resultant images for different sample displacements in   Figure 7.9 and Figure 7.10, with the corresponding DC and AC images (of the sample with 0 mm displacement) used as references. For the lower displacements of less than 0.8 mm, there was not much difference in QIs for the images obtained without and with object alignment; while for the larger displacements, the QIs were improved by a large margin due to object alignment, especially for the AC images when the displacements were 3.2 and 6.4 mm, which was also in agreement with the visual inspection. These results are practically encouraging, since a larger allowable sample displacement implies a smaller pattern projection and image acquisition rate required for the given sample moving speed, which would be much easier to realize. Table 7.1. The quality index (QI) for the direct component (DC) and amplitude component (AC) images (shown in Figure 7.9 and Figure 7.10) obtained without (case 1) and with (case 2) object alignment for image demodulation, for the different sample displacements in acquiring two phase-shifted pattern images. The displacement (D) of a sample in two acquired phase-shifted pattern images QI Case 1 DC AC Case 2 DC AC 0.1 mm 0.9980 0.9919 0.9980 0.9914 0.2 mm 0.9989 0.9928 0.9991 0.9915 0.4 mm 0.9959 0.9600 0.9988 0.9958 0.8 mm 0.9848 0.8535 0.9966 0.9825 1.6 mm 0.9613 0.6817 0.9888 0.9602 3.2 mm 0.8973 0.4584 0.9777 0.8839 6.4 mm 0.7597 0.2618 0.9050 0.6998 The above experiments on bruise detection of apples have validated the feasibility of real-time SIRI for detecting moving samples by using either a single fixed sinusoidal pattern or two phase- shifted sinusoidal patterns. Compared to detection of stationary samples, detection of moving samples required an additional image processing step to align the fruit object in two pattern images, due to sample movement or displacement, prior to image demodulation. Given the simple moving configuration, the object alignment was nothing but a translation transform based on estimated pixel displacement between the two pattern images in terms of the fruit object’s pixel coordinates. Two approaches were proposed in this study to estimate the pixel displacement, 1) by using a ruler reference for directly inspecting the pixel displacement (beforehand) for a specific imaging   209 configuration, or 2) based on automatic image processing to detect the pixel displacement of a distinctive marker imaged along with the sample. Although more flexible and adaptive, the second approach requires more computations, which involves BEMD, SPT and thresholding. This study accentuates the need for a high-speed camera for SIRI to operate at a practically acceptable sample moving speed. The current sample moving speed, which was only on the order of millimeters per second, was greatly limited by the low frame rate of the camera. In many high- speed structured light systems for 3-D measurement applications (Zhang, 2018), researchers use a high-speed camera (with a frame rate of generally more than 1000 Hz), which is comparable and in synchronization with a high-speed projector, for fast image acquisition for detecting moving samples, having achieved measurement rates at a few hundred Hz (Hyun and Zhang, 2017; Liu et al., 2014; Wang et al., 2013). In a seminal study on real-time 3-D measurement (Zhang and Huang, 2006), Zhang and colleagues synchronized a high-speed camera with a projector to acquire pattern images at a frame rate of 120 Hz, and, based on three-step phase shifting, achieved a measurement rate of 40 Hz. According to the experiment results in this study, this concept of synchronizing two high-speed devices provides a promising solution to extending SIRI to real-time applications. However, a concern may arise out of the quality of images acquired with extremely short exposure times, which is also related to the imaging sensor, for the purpose of fruit defect detection instead of 3-D measurement, which remains to be explored in future work. The proposed approaches to real-time SIRI in this study required two image acquisitions and object alignment. Real-time SIRI may be also realized through acquiring a single wavelength- multiplexed fringe pattern image. Although a general single fringe pattern is inadequate, ill-posed for obtaining decent DC and AC images for defect detection, one may obtain two phase-shifted pattern images through a single-frame image acquisition. One potential way to achieve this is to   210 acquire a dual-wavelength-multiplexed pattern image, which consists of phase-shifted patterns at two different wavelengths superposed together. The key point, conceptually, is to generate such two patterns, combine them as a whole, image the projected pattern at the two wavelengths all at once, and finally extract the two pattern images at respective wavelengths from the acquired dual- wavelength-multiplexed image. Such a single-frame based approach is most attractive due to its time efficiency, and should be the subject of future exploration. Further research should be directed toward developing a hardware prototype of SIRI for real- time applications. More dedicated hardware and software are needed to address many critical challenges that lie ahead, e.g., real-time image acquisition and processing, effective algorithms that accommodate complex moving conditions of samples (in practical settings, apples are rotated along the inspection line for being imaged for different areas of the fruit surface), inspecting the whole surface for defect detection, fast inspection rates to meet the industrial requirements. SIRI will primarily remain as a research tool before these challenges have been successfully resolved. 7.8. CONCLUSIONS Explorations of SIRI for detection of moving samples are essential for practical implementation of this technique into a practically useful tool for fruit defect detection. A programmable, motorized stage based SIRI platform was constructed for moving sample detection. Experimental tests on bruise detection of apples clearly demonstrated the feasibility and potential of SIRI for detecting moving samples, which could be achieved by means of sequentially acquiring two pattern images from moving samples or synchronizing two high-speed devices (i.e., camera and projector) to acquire two pattern images under the illumination of two phase-shifted patterns, respectively. Limited by the current hardware and configurations, the present study was only conducted at low sample moving speeds (e.g., 0-32 mm/s), and imaged samples at different   211 displacements (e.g., 0-6.4 mm), which provided the initial proof-of-concept work rather than practical implementation. Sample movement or displacement between two acquired pattern images requires object alignment prior to pixel-based demodulation. Two approaches were proposed, in light of the simplicity of sample moving (only translation assumed) in this study, to estimate the pixel displacement for object alignment. The first approach was to image a ruler reference beforehand to directly determine the pixel displacement for specific imaging setup, while the second was based on automatic detection of a white reference marker using image processing techniques. The two approaches performed similarly in retrieving DC and AC images, in terms of the quantified image quality compared to the images obtained for a stationary sample. It was found that a higher sample moving speed or larger sample displacement tended to deteriorate the quality of resultant images even after object alignment was done, and that, on the other side, decent images could be obtained for smaller sample moving speeds or displacements, even without object alignment. The findings highlighted the need to use a high-speed camera for detecting moving samples so as to maintain a practically acceptable sample moving speed. Further research is needed to explore other potential solutions to efficient implementation of SIRI and develop a hardware prototype for real-time defect detection of apples. Further research should also be directed at addressing some critical challenges, including inspecting the whole fruit surface, real-time image analysis, effective algorithms for addressing complex moving conditions for object alignment, and fast inspection rates, before SIRI can ultimately become a practically viable tool for fruit defect detection.   212 8. CONCLUSIONS AND FUTURE WORK This dissertation research has explored the feasibility and potential of SIRI as a new modality for enhanced defect detection of apples, and made advances on implementation of the technique for practical applications. Built upon the concept of modulated imaging, SIRI uses (sinusoidally- modulated) structured illumination, in place of conventional uniform illumination for sample imaging. Owning to structured illumination, SIRI possesses a number of unique, valuable features or advantages, compared to traditional imaging inspection techniques for food products; and these features make SIRI a promising, versatile modality for enhanced defect detection of apples. A SIRI platform was constructed, with a DLP for structured illumination, for image acquisitions in either broadband or multispectral mode. A series of studies were conducted on analysis of fundamental features of SIRI, demodulation approaches, 3-D geometry reconstruction, development of imaging processing/classification algorithms for effective detection of subsurface bruising and surface defects of apples, and explorations of the technique for detecting moving samples towards real- time implementation. The major findings and conclusions drawn from this research are summarized as follows: 1) The preliminary tests on a high-scattering Nylon block with embedded inhomogeneities and apple samples with subsurface bruising demonstrated the capability of SIRI to detect depth- specific defects with enhanced image resolution and contrast by adjusting the spatial frequency of illumination patterns. An image formation model for SIRI and a general demodulation methodology for obtaining AC and DC images were proposed, which provided a basis for understanding and analyzing the fundamental features of this technique regarding light penetration and image resolution and contrast. Compared to DC, AC, which is unique to SIRI, possessed enhanced image resolution and contrast and had a depth-resolving ability; however,   213 the effectiveness of AC images for defect detection of apples would depend on defect type, fruit variety and spatial frequency of light patterns. 2) Demodulation approaches based on SPT, which is a 2-D extension to Hilbert transform, were formulated and experimentally proven effective for demodulating SIRI pattern images. In comparison to the conventional phase shifting techniques that requires three phase-shifted pattern images, SPT required one or two pattern images of arbitrary phase shifts for each sample, thus speeding up the image acquisition process. Among the different implementations of SPT, a two-phase based SPT was the best in terms of demodulation accuracy and contrast of resultant images, except that it could incur slight boundary artifacts due to the Fourier transformations involved. In addition, based on an assumption that there existed more than one complete fringe, GSO was also proven to be a simple and effective demodulation approach for two phase-shifted SIRI pattern images; however the technique was sensitive to image noise and required DC removal for successful amplitude demodulation. Furthermore, simulations and experiments were conducted on using composite sinusoidal patterns that combined two and three frequency components to acquire augmented frequency information efficiently from samples. Demodulation approaches for the composite patterns were proposed by solving a linear system of phase-shifted pattern images in the Fourier domain, without or with the use of SPT. With two or three frequency components, grid composite patterns gave high-quality amplitude images than the fringe composite patterns. 3) The phase information in the acquired SIRI pattern images allowed for reconstruction of the 3-D geometry of fruit, which would be beneficial for detecting surface concavities of fruit to enhance general defect detection. Phase analysis methodologies were proposed for reconstructing the 3-D geometry of apples based on a reference plane, which involved phase   214 demodulation, phase wrapping, the calculation of phase difference with respect to the phase of the reference plane, phase-to-height and in-plane calibrations. In particular, the phase-to-height calibrations were realized by using low-order polynomial based curve fitting, while the in- plane calibration was based on a simplified perspective imaging model. Experimental tests on two rectangular and trapezoidal verification blocks resulted in relative height measurement errors of 4.16% and 3.75%, respectively, with a conventional three-step phase shifting approach for phase demodulation, compared with 7.46% and 6.67% when SPT was used for phase demodulation of two pattern images. Further tests on apple samples showed that surface concavities could be readily recognized from the reconstructed 3-D shape, 2-D height profiles, and also phase difference maps and normalized fringe patterns, while the bruising defects only visible in the intensity images (i.e., AC or ratio images) were absent in the phase-related images. This study on phase analysis of SIRI pattern images laid a foundation for using the technique for reconstructing 3-D geometry of fruit, and thus expanded the capability and applications of the technique for fruit quality evaluation. 4) The capability of SIRI for enhanced defect detection was further demonstrated by applying the technique, in conjunction with the two-phase based SPT demodulation approach, to detection of both subsurface bruising and surface defects of apples. In detecting fresh subsurface bruising in apples of four varieties by a multispectral SIRI system in the wavelength range of 650-950 nm, AC and ratio images were able to reveal the bruises, which, otherwise, could not be achieved by the DC images. The ratio images at the six NIR wavebands from 710 to 810 nm in 20 nm increments produced similar detection results with overall detection errors of 11.7- 14.2%, when modified Otsu thresholding was used coupled with a shape-based bruise detection algorithm. In detection of surface and subsurface defects for apples of two varieties   215 by SIRI at a single NIR wavelength of 730 nm, machine learning algorithms were used to evaluate the efficacy of the ensemble of DC and AC images for enhanced defect detection. BEMD was proposed and found to be effective for enhancing DC and AC images by removing vignetting and noise artifacts. In classifying apples into defective and normal classes, the combination of DC and AC images resulted in significantly enhanced classification accuracies than using them separately. Among the three classifiers (i.e., RF, SVM and CNN), CNN, which was trained with artificially augmented images, achieved the best results, with overall detection accuracies of 98% for both varieties of apples. This study showed that SIRI, coupled with machine learning, can be a versatile and effective modality for apple defect detection. 5) In a proof-of-concept study, SIRI was explored for detecting moving samples by setting up a programmable, motorized stage for motion control of samples. Experimental tests on bruise detection of apples demonstrated the feasibility and potential of SIRI for detecting moving samples, which was achieved by means of sequentially acquiring two pattern images from moving samples or synchronizing two high-speed devices (i.e., camera and projector) to acquire two pattern images under the illumination of two phase-shifted patterns, respectively. Limited by the current hardware and configurations, the present study was run at low sample moving speeds (e.g., 0-32 mm/s), with simulations being carried out for the synchronization of the projector and the camera by imaging samples at different displacements. While the dissertation research has demonstrated the unique capabilities and potential of SIRI for enhanced defect detection of apples, more research is needed before SIRI becomes a practically useful inspection tool. The following recommendations are offered for future research:   216 1) Further studies should be focused on moving sample detection at high speed by developing appropriate hardware for real-time defect detection of apples and other fruits and food products. Limited by the image acquisition speed of the camera in the current SIRI system, the sample moving speeds was too low for practical implementation for defect sorting of apples. Hence, further research should consider using a high-speed camera and a compataible projector to improve the sample moving speed. Synchronization of a high-speed camera and a projector may provide a viable solution for SIRI to detect moving samples, which should warrant further investigations. In addition, a dual-wavelength-multiplexed approach, conceptually, enables the most efficient implementation of SIRI by means of obtaining two phase-shifted pattern images from a single-frame image acquisition, which should be explored in future research. Other challenges, which are critical for practical implementation of SIRI, include inspection of whole fruit surface, object alignment under practical moving conditions for image demodulation, real- time image acquisition and analysis, and fast inspection rate, all of which would require more dedicated instrumentation and software. 2) Fundamental studies should be carried out for a better understanding and utilization of SIRI for food quality and safety inspection. The research has showed the limited detection depth of SIRI (e.g., sub-millimeters in the experiment on detecting paper targets), which would restrict its applications to detection of surface and subsurface defects of apples. Further research is hence needed to achieve deeper light penetration and detection depths, by improving the imaging hardware (e.g., light source) or even resorting to inverse approaches for diffuse optical tomography. Although SPT is proven an effective demodulation approach, further improvements can be made to minimize or reduce the associated Gibbs ripple artifacts near the object boundaries, which should be explored in future work. While the dissertation research   217 has demonstrated SIRI is capable of enhancing the classification of apples into defective and normal classes by exploiting the features from both AC and DC images, more effective defect detection requires localizing the defects within an image (i.e., defect segmentation) so as to grade apples into multiple classes according to the defect severity. In addition to defects on fruit, the AC images also contain enhanced features pertaining to normal skin texture and lenticels, which may interfere with defect segmentation. Hence attention should be also paid to developing or deploying effective image analysis and machine learning techniques to address the complexities of defect segmentation. The proposed phase analysis based approach allowed recovery of the fruit geometry to aid in the detection of surface concavities; however it still needs to address the issues related to phase unwrapping, which is a rate-limiting and error-prone step. Further research is also needed to improve the phase analysis procedures, especially the phase analysis algorithm, for more efficient and accurate height profiling or 3- D reconstruction of fruit, and further incorporate the geometric information for enhancing defect detection. Moreover, research should also be considered on expanding SIRI for quality and safety inspection of other horticultural commodities and food products. Further research on SIRI should be explored in conjunction with other sensing modes, such as fluorescence imaging, which is sensitive to philological disorders of fruit.               218 APPENDICES   219 APPENDIX A: MODIFIED OTSU THRESHOLDING   Given a normalized bimodal histogram that is represented by a probability distribution function f(t), as shown in Figure A1, and that the function f(t) is seen as the sum of two functions f1(t) and f2(t) corresponding to the two distinct objects. Let the histogram be portioned into two parts by a given threshold T, and, without loss of generality, T ≥Topt where Topt corresponds to the valley of f(t) or the intersection of f1(t) and f2(t) (here it is tacitly assumed that the two objects have equal prior probabilities, so to say not to consider priors for thresholding). In the following derivations, the variable t is omitted for convenience in the probability distribution functions.   Figure A. 1. Gray-scale histograms represented by probability distribution functions. According to probability theory and Eq. (6.2), the within-class variance 2 W for the histogram can be formulated as follows: T    0 2 W 2 t  f 1   f dt 2    T  0 tf dt 1  f dt 1  tf dt 1 2  f dt 2  tf dt 2 2 2 t t L  0 f dt 1 L  0     L    0 K K 1  t 2     L  0   L  0 K 3  2 2    2     t T     0 2    f 1   f dt 2 tf dt 1 2   L  T 2    T 2  0  L  T 2 t  f 1   f dt 2  L     T t  f 1  f 2 tf dt tf dt 1 2 T  0  L  0 tf dt 2  T  0 tf dt 2  2      L  T  2  tf dt 2 2   L 2  T tf dt tf dt 1 2 L  T  2 T     0 tf dt tf dt 1 1 L  T T   0 tf dt tf dt 1 2 T  0    (A1)  2 T     0 tf dt tf dt 2 2 L  T L   T tf dt tf dt 1 2 L  T    where K1 and K2 and are constants representing the variances of f1(t) and f2(t), respectively, which are expressed as follows:   220 K 1  K 2  L  0 L  0 2 t dt f 1      L  0 dt tf 1 2     2 t f 2 dt      L  0 tf dt 2 2     and K3 is a variable controlled by the threshold T, which is expressed as follows: K 3  T  0     dt tf 1  L  T tf 2 dt         T  0 tf 2 dt  L  T dt tf 1     (A2) (A3) (A4) In geometric terms, K3 is mathematically equal to the multiplication of the areas of two regions, which is rewritten as follows: K 3  ( s 1  s 0 )( s 2  s 0 ) (A5) where s1, s2, and s0 are the areas of three regions in the histogram, as shown in figure A2. Note that s1 and s2 are constants, and s0 varies with the threshold. Figure A. 2. Gray-scale histograms where s1, s2 and s0 are the areas of the corresponding shaded regions.   Further, Eq. (A5) can be expanded as follows:   K 3  s s 1 2  ( s 1   s 2 ) s s 0 0 (A6) Because (s1  s2) > s0 ≥ 0, K3 has a unique minimizer s0 = 0, equivalent to T = Topt. Therefore, the modified Otsu thresholding described in Eq. (6.3) results in a threshold as follows: T m1    Arg Arg T opt   ( T )] min[ 2  W 0 T L    min K K 0 T L    1  2 K  3 (A7) 2 221 According to Eq. (A7), the optimal threshold that corresponds to the valley position of a bimodal histogram can be attained through minimizing the within-class variance of the histogram. This completes the proof.   222 APPENDIX B: UNIMODAL THRESHOLDING   A unimodal histogram is the one that has one dominant peak, with or without secondary, small peaks. Such a histogram is frequently encountered in detection of fruit defects, such as bruises, which are generally much smaller in size than the surrounding normal tissue. Traditional bimodal thresholding techniques are not applicable to a unimodal histogram for image segmentation. To address the issue, Rosin (2001) proposed a simple shape-based unimodal thresholding technique. In this method, an image is assumed to have a single domain peak located at the lower part of its histogram, as shown in Figure B1(left), which has a detectable corner at the base of the histogram. To find the optimal threshold, a straight line is drawn from the peak to the first empty bin of the histogram closely following the non-empty bin, and the optimal threshold is selected at the point that maximizes the distance between the histogram and the straight line. The original unimodal thresholding is to locate the region of interest (ROI) that corresponds to the upper part of the histogram. However, for fruit defects detection, the ROI includes defects and dark background, which normally correspond to the lower part of a histogram. Hence, the original unimodal thresholding is extended here to the cases where the dominant peak is situated at the upper part of the histogram, as shown in Figure B1(right), which would enable the segmentation of dark background from the fruit object, and the defect region from the normal or healthy region after the background has been removed (Lu and Lu, 2017a). Figure B. 1. A unimodal thresholding method for determining a threshold from a histogram with a single dominant peaks located at lower end (left) and upper end (right) of the histogram.   223 APPENDIX C: BI-DIMENSIONAL EMPIRICAL MODE DECOMPOSITION   Bi-dimensional empirical mode decomposition (BEMD) is an extension of 1-D empirical mode decomposition (EMD) to 2-D domain, which is primarily used for image analysis. The principle of BEMD is to decompose an input image, in the spirit of the Fourier or wavelet decomposition, into a number of intrinsic components of varying frequency scales (Linderhed, 2009; Nunes et al., 2003). For a given image ( , I x y , BEMD gives a representation of the image as a sum of intrinsic ) mode function (IMF) images plus a residual image as follows: ( , I x y ) n  1 i  IMF x y ( , i )  Res x y ( , ) (C1) To find the individual IMFs and the residual images, the input image is decomposed through an iterative sifting process, which, in general, can be summarized as follows: (1) Find all the local extrema (i.e., maxima and minima) of the image ( , ) I x y ; (2) Create the upper and lower envelopes of the image by surface interpolation of its maxima and minima, which are denoted as ( , ) E x y  and ( , ) E x y  , respectively; (3) Calculate the mean of the upper and lower envelopes, which is denoted as ) ( , E x y ; (4) Update the image: ( , ) I x y  ( , ) ( , ) I x y E x y  ; (5) Steps (1)-(4) represent one complete cycle of the sifting process, and continue the iteration until certain stop criterion is met (e.g., a fixed number of sifting cycles). (6) When the stop criterion is met, the sifting is terminated and the first IMF are calculated to be last result of the step (4). (7) Repeat steps (1-6) to extract more IMFs with the previously obtained residual image as the input, until the resultant residual becomes a monotonic function or a predefined number of IMFs are obtained, and the final residual is taken as   224 Res x y . ( , ) In the above procedures, the first two steps are particularly computationally intensive and time- consuming, which keep BEMD for fast, practical applications. To address the computation issue, Bhuiyan et al. (2008) proposed a fast, adaptive version of BEMD (called FABEMD), which used two order-statistic filters, i.e., MAX and MIN filters, to estimate the upper and lower envelopes, respectively, followed by smoothing operations, instead of traditional surface interpolation, which significantly reduced the computation load. The author further reduced the decomposition time by limiting the number iterative sifting cycles to one for extracting each IMF. In implement the new BEMD for extracting IMFs, the size for the two order-statistic filters was adaptively determined based on the distance of adjacent extrema points in the image, and the average of the smoothed upper and lower envelopes was taken as an IMF. With the new FABEMD, decomposing an image of 512×512 pixels in size can be completed within minutes. However, it is still not sufficiently fast for many applications such as fruit defect detection. The key rating-limiting step in FABEMD is to calculate the distance of adjacent extrema points for updating the size for the order-statistics and smoothing filters. Recently, Trusiak et al (2014) proposed an enhanced version of FABEMD, called as EFEMD, which is much faster since it uses an approximated distance that is calculated as followed : D  M N K  (C2) Where M×N is the image size and K is the average number of maxima and minima points in the image. Besides, this method implemented morphological (erosion and dilation) operations instead of the order-statistics filters for envelope estimation, which further speeded up the computation. The EFEMD method enables the decomposition of a general image within one second. Our experience in running BEMD to decompose DC and AC images has shown that, simply setting the distance value in a predefined geometric progression (e.g., with an initial value of 1 and   225 a common ratio of 2), instead of resorting to Eq. (C2) or the approach used in FABEMD, can also yield descent decomposition results, which thus makes BEMD even faster to use and have the potential for real-time applications (Lu and Lu, 2018). In this dissertation, the fastest BEMD was used for image enhancement of DC and AC images for defect detection of apples, by selectively discarding the resultant first IMF and the residual images, which corresponded to noise and illumination effects, respectively, which is described in Chapter 6.                                 226 BIBLIOGRAPHY                               227   BIBLIOGRAPHY Abbott, J.A., 1999. Quality measurement of fruits and vegetables. Postharvest Biology and Technology 15, 207-225. Abrahamsson, S., Ball, G., Wicker, K., Heintzmann, R., Schermelleh, L., 2017. Structured illumination microscopy, In: Diaspro, A., van Zandvoort, M.A.M.J. (Eds.), Super-resolution imaging in biomedicine. CRC Press, Boca Raton, FL, USA. Adamiak, A., Zdunek, A., Kurenda, A., Rutkowski, K., 2012. Application of the biospeckle method for monitoring bull's eye rot development and quality changes of apples subjected to various storage sethods - preliminary studies. Sensors 12, 3215-3227. Anderson, E.R., Cuccia, D.J., Durkin, A.J., 2007. Detection of bruises on golden delicious apples using spatial-frequency-domain imaging, Proceedings Volume 6430, Advanced Biomedical and Clinical Diagnostic Systems V, San Jose, CA. Arefi, A., Ahmadi Moghaddam, P., Hassanpour, A., Mollazade, K., Modagh, A.M., 2016. Non- destructive identification of mealy apples using biospeckle imaging. Postharvest Biology and Technology 112, 266-276. Arefi, A., Moghaddam, P.A., Mollazade, K., Hassanpour, A., Valero, C., Gowen, A., 2015. Mealiness detection in agricultural crops: destructive and nondestructive tests: a review. Comprehensive Reviews in Food Science and Food Safety 14, 657-680. Argenta, L., Fan, X.T., Mattheis, J., 2002. Impact of watercore on gas permeance and incidence of internal disorders in 'Fuji' apples. Postharvest Biology and Technology 24, 113-122. Ariana, D., Guyer, D.E., Shrestha, B., 2006. Integrating multispectral reflectance and fluorescence imaging for defect detection on apples. Computers and Electronics in Agriculture 50, 148-161. Ayers, F.R., Cuccia, D.J., Kelly, K.M., Durkin, A.J., 2009. Wide-field spatial mapping of In vivo tattoo skin optical properties using modulated imaging. Lasers in Surgery and Medicine 41, 442- 453. Baranowski, P., Lipecki, J., Mazurek, W., Walczak, R.T., 2008. Detection of watercore in 'Gloster' apples using thermography. Postharvest Biology and Technology 47, 358-366. Baranowski, P., Mazurek, W., Pastuszka-Wozniak, J., 2013. Supervised classification of bruised apples with respect to the time after bruising on the basis of hyperspectral imaging data. Postharvest Biology and Technology 86, 249-258. Baranowski, P., Mazurek, W., Witkowska-Walczak, B., Slawinski, C., 2009. Detection of early apple bruises using pulsed-phase thermography. Postharvest Biology and Technology 53, 91-100. Baranowski, P., Mazurek, W., Wozniak, J., Majewska, U., 2012. Detection of early bruises in apples using hyperspectral data and thermal imaging. Journal of Food Engineering 110, 345-355.   228 Baritelle, A.L., Hyde, G.M., 2001. Commodity conditioning to reduce impact bruising. Postharvest Biology and Technology 21, 331-339. Barreiro, P., Moya, A., Correa, E., Ruiz-Altisent, M., Fernandez-Valle, M., Peirs, A., Wright, K.M., Hills, B.P., 2002. Prospects for the rapid detection of mealiness in apples by nondestructive NMR relaxometry. Applied Magnetic Resonance 22, 387-400. Barreiro, P., Ortiz, C., Ruiz-Altisent, M., De Smedt, V., Schotte, S., Andani, Z., Wakeling, I., Beyts, P.K., 1998. Comparison between sensory and instrumental measurements for mealiness assessment in apples. Journal of Texture Studies 29, 509-525. Barreiro, P., Ortiz, C., Ruiz-Altisent, M., Ruiz-Cabello, J., Fernandez-Valle, M.E., Recasens, I., Asensio, M., 2000. Mealiness assessment in apples and peaches using MRI techniques. Magnetic Resonance Imaging 18, 1175-1181. Barreiro, P., Ruiz-Cabello, J., Fernandez-Valle, M.E., Ortiz, C., Ruiz-Altisent, M., 1999. Mealiness assessment in apples using MRI techniques. Magnetic Resonance Imaging 17, 275-281. Bassi, A., D'Andrea, C., Valentini, G., Cubeddu, R., Arridege, S., 2009. Detection of inhomogeneities in diffusive media using spatially modulated light. Optics Letters 34, 2156-2158. Bell, T., Li, B., Zhang, S., 2016. Structured light techniques and applications, In: Webster, J. (Ed.), Wiley Encyclopedia of Eletrocical and Electronics Engineering. John Wiley & Sons, Inc. Bengio, Y., 2012. Practical recommendations for gradient-based training of deep architectures. Springer, Berlin. Bennedsen, B.S., Peterson, D.L., 2005. Performance of a system for apple surface defect identification in near-infrared images. Biosystems Engineering 90, 419-431. Bennedsen, B.S., Peterson, D.L., Tabb, A., 2007. Identifying apple surface defects using principle component analysis and artificial neural networks. Transcation of the ASABE 50, 2257-2265. Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., Turian, J., Warde- Farley, D., Bengio, Y., 2010. Theano: a cpu and gpu math complier in python, Proceedings of the Python for Scientific Computing Conference (SciPy). Bex, P.J., Makous, W., 2002. Spatial frequency, phase, and the contrast of natural images Journal of the Optical Society of America A 19, 1096-1106. Bhuiyan, S.M.A., Adhami, R.R., Khan, J.F., 2008. Fast and adaptive bidimensional empirical mode decomposition using order-statistics filter based envelope estimation. EURASIP Journal on Advances in Signal Processing. Bigio, I.J., Fantini, S., 2016. Quantitative biomedical optics: theory, methods, and applications. Cambridge University Press, Cambridge, UK.   229 Binzoni, T., Vogel, A., Gandjbakhche, A.H., Marchesini, R., 2008. Detection limits of multi- spectral optical imaging under skin surface. Physics in Medicine & Biology 53, 617-636. Biogio, I.J., Fantini, S., 2016. Quantitative biomedical optics: theory, methods, and applications. Cambridge University Press, Cambridge, UK. Birth, G.S., Olsen, K.L., 1964. Nondestructive detection of water core in Delicious apples. Proceedings of the American Society for Horticultural Science 85, 74-84. Blais, F., 2004. Review of 20 years of range sensor development. Journal of electronic imaging 13, 231-240. Bodenschatz, N., Kruter, P., Nothelfer, S., Foschum, F., Bergman, F., Liemert, A., Kienle, A., 2015. Detecting structural information of scatterers using spatial frequency domain imaging. Journal of Biomedical Optics 201, 116006. Boreman, G.D., 2001. Modulation transfer function in optical and electro-optical systems. SPIE, Bellingham, WA. Bracewell, R.N., 2000. The Fourier transform and its applications McGraw Hill Education, New York, USA. Braga, R.A., Dupuy, L., Pasqual, M., Cardoso, R.R., 2009. Live biospeckle laser imaging of root tissues. European Biophysics Journal 38, 379-686. Breiman, L., 2001. Random Forests. Machine Learning 45, 5-32. Brook, P.J., 1968. Pre-havest lenticel spot of apples. New Zealand Journal of Agricultural Research 11, 237-244. Brown, G.K., Burton, C.L., Sargent, S.A., Schulte Pason, N.L., Timm, E.J., Marshall, D.E., 1989. Assessment of apple damage on packing lines. Applied Engineering in Agriculture 5, 475-484. Brusewitz, G.H., Bartsch, J.A., 1989. Impact parameters related to post harvest bruising of apples. Transactions of the ASAE 32, 953-957. Burton, D.R., 2014. Phase unwrapping, In: Pramod, R. (Ed.), Phase estimation in otpical interferometry. CRC Press, Boca Raton, FL, USA, pp. 273-290. Burton, D.R., Goodall, A.J., Atkinson, J.T., Lalor, M.J., 1995. The use of carrier frequency shifting for the elimination of shape discontinuities in Fourier transform profilometry. Optics and Laser in Engineering 23, 245-257. Cen, H., Lu, R., 2009. Quantification of the optical properties of two-layer turbid materials using a hyperspectral imaging-based spatially-resolved technique. Applied Optics 48, 5612-5623. Cen, H., Lu, R., 2010. Optimization of the hyperspectral imaging-based spatially-resolved system for measuring the optical properties of biological materials. Optics Express 18, 17412-17432.   230 Cen, H., Lu, R., Nguyen-Do-Trong, N., Saeys, W., 2016. Spatially resolved spectroscopic technique for measuring optical properties of food, In: Lu, R. (Ed.), Light scattering technology for food property, quality and safety assessment. CRC Press, Boca Raton, FL, USA, pp. 159-185. Chayaprasert, W., Stroshine, R., 2005. Rapid sensing of internal browning in whole apples using a low-cost, low-field proton magnetic resonance sensor. Postharvest Biology and Technology 36, 291-301. Chen, F., Brown, G.M., Song, M., 2000. Overview of three-dimensional shape measurement using optical methods. Optical Engineering 39, 10-22. Chen, P., McCarthy, M.J., Kauten, R., 1989. NMR for internal quality evaluation of fruits and vegetables. Transactions of the ASAE 32, 1747-1753. Chen, P., Sun, Z., 1991. A review of non-destructive methods for quality evaluation and sorting of agricultural products. Journal of Agricultural Engineering Research 49, 85-98. Chen, Y.R., Chao, K., Kim, M.S., 2002. Machine vision technology for agricultural applications. Computers and Electronics in Agriculture 36, 173-191. Chene, Y., Rousseau, D., Lucidarme, P., Bertheloot, J., Caffier, V., Morel, P., Belin, E., Chapeau- Blondeau, F., 2012. On the use of depth camera for 3D phenotyping of entire plants. Computers and Electronics in Agriculture 82, 122-127. Chitode, J.S., 2008. Digital Signal Processing. Technical Publications, Pune, India. Chiu, Y.C., Chou, X.L., Grift, T.E., Chen, M.T., 2015. Automated detection of mechanically induced bruise areas in golden delicious apples using fluorescence imagery. Transactions of the ASABE 58, 215-225. Cho, B.K., Chayaprasert, W., Stroshine, R.L., 2008. Effects of internal browning and watercore on low field (5.4 MHz) proton magnetic resonance measurements of T-2 values of whole apples. Postharvest Biology and Technology 47, 81-89. Chollet, F., 2015. Keras. Clark, C.J., Burmeister, D.M., 1999. Magnetic resonance imaging of browning development in 'Braeburn' apple during controlled-atmosphere storage under high CO2. Hortscience 34, 915-919. Clark, C.J., Hockings, P.D., Joyce, D.C., Mazucco, R.A., 1997. Application of magnetic resonance imaging to pre- and post-harvest studies of fruits and vegetables. Postharvest Biology and Technology 11, 1-21. Clark, C.J., MacFall, J.S., Bieleski, R.L., 1998. Loss of watercore from 'Fuji' apple observed by magnetic resonance imaging. Scientia Horticulturae 73, 213-227. Clark, C.J., McGlone, V.A., Jordan, R.B., 2003. Detection of Brownheart in 'Braeburn' apple by transmission NIR spectroscopy. Postharvest Biology and Technology 28, 87-96.   231 Cortes, C., Vapnik, V., 1995. Support-vector networks. Machine Learning 20, 273-297. Crowe, T.G., Delwiche, M.J., 1996a. Real-time defect detection in fruit .1. Design concepts and development of prototype hardware. Transactions of the ASABE 39, 2299-2308. Crowe, T.G., Delwiche, M.J., 1996b. Real-time defect detection in fruit .2. An algorithm and performance of a prototype system. Transactions of the ASAE 39, 2309-2317. Cubero, S., Aleixos, N., Molto, E., Gomez-Sanchis, J., Blasco, J., 2011. Advances in machine vision applications for automatic inspection and quality evaluation of fruits and vegetables. Food Bioprocess Technology 4, 487-504. Cuccia, D.J., Bevilacqua, F., Durkin, A.J., Ayers, F.R., Tromberg, B.J., 2009. Quantitation and mapping of tissue optical properties using modulated imaging. Journal of Biomedical Optics 14, 024012. Cuccia, D.J., Bevilacqua, F., Durkin, A.J., Tromberg, B.J., 2005. Modulated imaging: quantitative analysis and tomography of turbid media in the spatial-frequency domain. Optics Letters 30, 1354- 1356. D'Andrea, C., Ducros, N., Bassi, A., Arridge, S., Valentini, G., 2010. Fast 3D optical reconstruction in turbid media using spatially modulated light. Biomedical Optics Express 1, 471- 481. Dam, J.S., Pedersen, C.B., Dalgaard, T., Fabricius, P.E., Aruna, P., Andersson-Engels, S., 2001. Fiber-optic probe for noninvasive real-time determination of tissue optical properties at multiple wavelengths. Applied Optics 40, 1155-1164. Danckaers, F., Huysmans, T., Dael, M.V., Verboven, P., Nicolai, B.M., Sijbers, J., 2015. Building a statistical shape model of the apple from corresponded surfaces. Chemical Engineering Transactions 44, 49-54. Davenel, A., Guizard, C.H., Labarre, T., Sevila, F., 1988. Automatic detection of surface defects on fruit by using a vision system. Journal of Agricultural Engineering Research 41, 1-9. DeEll, J.R., Prange, R.K., Murr, D.P., 1996. Chlorophyll fluorescence of Delicious apples at harvest as a potential predictor of superficial scald development during storage. Postharvest Biology and Technology 9, 1-6. DeEll, J.R., Toivonen, P.M.A., 2003. Practical applications of chlorophyll fluorescence in plant biology. Kluwer Academic Publishers, Norwell, MA, USA. Defraeye, T., Lehmann, V., Gross, D., Holat, C., Herremans, E., Verboven, P., Verlinden, B.E., Nicolai, B.M., 2013. Application of MRI for tissue characterisation of 'Braeburn' apple. Postharvest Biology and Technology 75, 96-105.   232 Dhakal, S., Li, Y., Peng, Y., Chao, K., Qin, J., Guo, L., 2014. Prototype instrument development for non-destructive detection of pesticide residue in apple surface using Raman technology. Journal of Food Engineering 123, 94-103. Dognitz, N., Wagnieres, G., 1998. Determination of tissue optical properties by steady-state spatial frequency-domain refelectometry. Lasers in medical science 13, 55-65. Doornbos, R.M.P., Lang, R., Aalders, M.C., Cross, F.W., Sterenborg, H.J.C.M., 1999. The determination of in vivo human tissue optical properties and absolute chromophore concentrations using spatially resolved steady-state diffuse reflectance spectroscopy. Physics in Medicine and Biology 44, 967-981. Doosti-Irani, O., Reza Golzarian, M., Aghkhani, M.H., Sadrnia, H., Doosti-Irani, M., 2016. Development of multiple regression model to estimate the apple's bruise depth using thermal maps. Postharvest Biology and Technology 116, 75-79. Elgar, H.J., Burmeister, D.M., Watkins, C.B., 1998. Storage and handling effects on a CO2-related internal browning disorder of 'Braeburn' apples. Hortscience 33, 719-722. ElMasry, G., Wang, N., Vigneault, C., 2009. Detecting chilling injury in Red Delicious apple using hyperspectral imaging and neural networks. Postharvest Biology and Technology 52, 1-8. ElMasry, G., Wang, N., Vigneault, C., Qiao, J., ElSayed, A., 2008. Early detection of apple bruises on different background colors using hyperspectral imaging. LWT - Food Science and Technology 41, 337-345. Farrell, T.J., Patterson, M.S., Wilson, B., 1992. A diffusion-theory model of spatially resolved steady-state diffuse reflectance for the noninvasive determination of tissue optical-properties in vivo. Medical Physics 19, 879-888. Faust, M., Shear, C.B., 1972. Russeting of apples, an interpretive review. HortScience 7, 233-235. Felicetti, D.A., Schrader, L.E., 2008. Photooxidative sunburn of apples: characterization of a third type of apple sunburn. International Journal of Fruit Science 8, 160-172. Ferguson, I.B., Watkins, C.B., 1989. Bitter pit in apple fruit. Horticultural Reviews 11, 289-355. Ferrari, C., Foca, G., Calvini, R., Ulrici, A., 2015. Fast exploration and classification of large hyperspectral image datasets for early bruise detection on apples. Chemometrics and Intelligent Laboratory Systems 146, 108-1109. Francis, F.J., Bramlage, W.J., Lord, W.J., 1965. Detection of watercore and internal breakdown in delicious apples by light transmittance. Proceedings of the American Society for Horticultural Science 87, 78-84. Gao, P., Pedrini, G., Osten, W., 2013. Structured illumination for resolution enhancement and autofocusing in digital holographic microscopy. Optics Letters 38, 1328-1330.   233 Garcia-Isais, C.A., Alcala Ochoa, N., 2014. One shot profilometry using a composite fringe pattern. Optics and Lasers in Engineering 53, 25-30. Garcia, J.L., Ruiz-Altisent, M., Barreiro, P., 1995. Factors influencing mechanical properties and bruise susceptibility of apples. Journal of Agricultural Engineering Research 61, 11-18. Gemperline, P., 2006. Practical guide to chemometrics. Second ed. CRC Press, Taylor & Francis Group, Boca Baton, FL, USA. Ghiglia, D.C., Pritt, M.D., 1998. Two-dimensional phase unwrapping: theory, algorthms, and software. John Wiley & Sons, Inc. Gigan, S., 2017. Optical microscopy aims deep. Nature Photonics 11, 14-16. Gioux, S., Mazhar, A., Cuccia, D.J., Durkin, A.J., Tromberg, B.J., Frangioni, J.V., 2009. Three- dimensional surface profile intensity correction for spatially modulated light. Journal of Biomedical Optics 14, 034045. Golding, J., Jobling, J., 2012. Apples, In: Rees, D., Farrell, G., Orchard, J. (Eds.), Crop Post- Harvest: Science and Technology: Perishables. Wiley-Blackwell, Chichester, UK, pp. 88-107. Gomez-Sanchis, J., Gomez-Chova, L., Aleixos, N., Camps-Valls, G., Montesinos-Herrero, C., Molto, E., Blasco, J., 2008a. Hyperspectral system for early detection of rottenness caused by Pencicillium digitatum in mandarins. Journal of Food Engineering 89, 80-86. Gomez-Sanchis, J., Molto, E., Camps-Valls, G., Gomez-Chova, L., Aleixos, N., Blasco, J., 2008b. Automatic correction of the effects of the light source on spherical objects. An application to the analysis of hyperspectral images of citrus fruits. Journal of Food Engineering 85, 191-200. Gonzalez, J.J., Valle, R.C., Bobroff, S., Viasi, W.V., Mitcham, E.J., McCarthy, M.J., 2001. Detection and monitoring of internal browning development in 'Fuji' apples using MRI. Postharvest Biology and Technology 22, 179-188. Gonzalez, K.M.M., 2009. Bruising profile of fresh 'Golden Delicious' apples, Horticulture and Landscape Architecture. Washington State University, Pullman, WA. Goodman, J.W., 2005. Introduction to Fourier Optics. Roberts & Company, Greenwood Village, CO. Gorbe, E., Calatayud, A., . Sci. Hortic. 138, -. 2012. Applications of chlorophyll fluorescence imaging technique in horticultural research: A review. Scientia Horticulturae 138, 24-35. Gowen, A.A., O'Donnell, C.P., Cullen, P.J., Downey, G., Frias, J.M., 2007. Hyperspectral imaging - an emerging process analytical tool for food quality and safety control. Trends in Food Science & Technology 18, 590-598. Guan, C., Hassebrook, L.G., Lau, D.L., 2003. Composite structured light pattern for three- dimensional video. Optics Express 11, 406-417.   234 Gunasekaran, S., 1996. Computer vision technology for food quality assurance. Trends in Food Science & Technology 7, 245-256. Gunasekaran, S., Paulsen, M.R., Shove, G.C., 1985. Optical methods for nondestructive quality evaluation of agricultural and biological materials. Journal of Agricultural Engineering Research 32, 209-241. Guo, H., He, H., Yu, Y., Chen, M., 2005. Least-squares calibration method for fringe projection profilometry Optical Engineering 44, 033603. Gustafsson, M., 2000. Surpassing the lateral resolution limit by a factor of two using structured illumination microscopy. Journal of microscopy 198, 82-87. Haff, R.P., Toyofuku, N., 2008. X-ray detection of defects and contaminants in the food industry. Sensing and Instrumentation for Food Quality and Safety 2, 262-273. Hahn, S.L., 1996. Hilbert transform in signal processing. Artech House, Inc., Norwood, MA, USA. Haralick, R.M., Shanmugam, K., Dinstein, I., 1979. Textural features for image classification. IEEE Transactions on Systems, Man and Cybernetics SMC-3, 610-621. Harker, F.R., Hallett, I.C., 1992. Physiological changes associated with development of mealiness of apple fruit during cooling storage. Hortscience 27, 1291-1294. Harker, F.R., Sutherland, P.W., 1993. Physiological changes associated with fruit ripening and the development of mealy texture during storage of nectarines. Postharvest Biology and Technology 2, 269-277. Harker, F.R., Watkins, C.B., Brookfield, P.L., Miller, M.J., Reid, S., Jackson, P.J., Bieleski, R.L., Bartley, T., 1999. Maturity and regional influences on watercore development and its postharvest disappearance in 'Fuji' apples. J. Am. Soc. Hortic. Sci. 124, 166-172. Hartley, R., Zisserman, A., 2003. Multiple view geometry in computer vision. Second ed. Cambridge University Press, New York, NY, USA. Heinemann, P.H., Varghese, Z.A., Morrow, C.T., Sommer III, H.J., Crassweller, R.M., 1995. Machine vision inspection of 'Golden Delicious' apples. Applied Engineering in Agriculture 11, 901-906. Heintzmann, R., 2006. Structured illumination methods, In: Pawley, J.B. (Ed.), Handbook Of Biological Confocal Microscopy. Springer, pp. 265-279. Heintzmann, R., Cremer, C.G., 1999. Laterally modulated excitation microscopy: improvement of resolution by using a diffraction grating, Optical Biopsies and Microscopic Techniques III. Proc. SPIE, Stockholm, Sweden.   235 Herraez, M.A., Burton, D.R., Lalor, M.J., Gdeisat, M.A., 2002. Fast two-dimensional phase- unwrapping algorithm based on sorting by reliability following a noncontinuous path. Applied Optics 41, 7437-7444. Herremans, E., Melado-Herreros, A., Defraeye, T., Verlinden, B., Hertog, M., Verboven, P., Val, J., Fernandez-Valle, M.E., Bongaers, E., Estrade, P., Wevers, M., Barreiro, P., Nicolai, B.M., 2014a. Comparison of X-ray CT and MRI of watercore disorder of different apple cultivars. 87. Herremans, E., Verboven, P., Defraeye, T., Rogge, S., Tri Ho, Q., Hertog, M., Verlinden, B.E., Bongaers, E., Wevers, M., Nicolai, B.M., 2014b. X-ray CT for quantitative food microstructure engineering. Nuclear Instruments and Methods in Physics Research Section B: Beam Interactions with Materials and Atoms 324, 88-94. Hu, D., Fu, X., He, X., Ying, Y., 2016. Characterization of the absorption and scattering properties of apple fruit using spatial-frequency domain imaging. Scientific Reports 6, 37920. Hu, M.K., 1962. Visual pattern recognition by moment invariants IRE transactions on information theory 8, 179-187. Huang, L., Chua, P.S.K., Asundi, A., 2010. Least-squares calibration method for fringe projection profilometry considering camera lens distorion. Applied Optics 49, 1539-1548. Huang, M., Lu, R.F., 2010. Apple mealiness detection using hyperspectral scattering technique. Postharvest Biology and Technology 58, 168-175. Huang, M., Zhu, Q.B., Wang, B.J., Lu, R.F., 2012. Analysis of hyperspectral scattering images using locally linear embedding algorithm for apple mealiness classification. Comput. Electron. Agric. 89, 175-181. Huang, N.E., Shen, S.S.P., 2014. Hilbert-Huang transform and its applications. World Scientific, Singapore. Huang, N.E., Shen, Z., Long, S.R., Wu, M.C., Shih, H.H., Zhang, Q., Yen, N.C., Tung, C.C., Liu, H.H., 1998. The empirical mode decomposition and the Hilbert spectrum for nonlinear and non- stationary time series analysis. Proc. of Royal Society of London A 454, 909-995. Huang, W., Li, J., Wang, Q., Chen, L., 2015. Development of a multispectral imaging system for online detection of bruises on apples. Journal of Food Engineering 146, 62-71. Hyun, J.S., Zhang, S., 2017. Superfast 3D absolute shape measurement using five binary patterns. Optics and Lasers in Engineering 90, 217-224. Ishimwe, R., Abutaleb, K., Ahmed, F., 2014. Applications of thermal imaging in agriculture - a review. Advances in Remote Sensing 3, 128-140. Itai, A., 2015. Watercore in Fruits, In: Kanayama, Y., Kochetov, A. (Eds.), Abiotic Stress Biology in Horticultural Plants. Springer, Tokyo, Jepan, pp. 127-145.   236 Itoh, K., 1982. Analysis of the phase unwrapping algorithm. Applied Optics 21, 2470. Jackson, J.E., 2003. Biology of apples and pears. Cambridge University Press, Cambridge, UK. Jacques, S.L., 2013. Optical properties of biological tissues: a review. Physics in Medicine & Biology 58, R37-R61. James, H.J., Jobling, J.J., 2009. Contrasting the structure and morphology of the radial and diffuse flesh browning disorders and CO2 injury of 'Cripps Pink' apples. Postharvest Biology and Technology 53, 36-42. Jarolmasjed, S., Espinoza, C.Z., Sankaran, S., 2017. Near infrared spectroscopy to predict bitter pit development in different varieties of apples. Journal of Food Measurement and Characterization 11, 987-993. Jarolmasjed, S., Espinoza, C.Z., Sankaran, S., Khot, L.R., 2016. Postharvest bitter pit detection and progression evaluation in 'Honeycrisp' apples using computed tomography images. Postharvest Biology and Technology 118, 35-42. Jia, P., Kofman, J., English, C., 2007. Comparision of linear and nonlinear calibration methods for phase-measuring profilometry. Optical Engineering 46, 043601. Jiang, L., Zhu, B., Cheng, X., Luo, Y., Tao, Y., 2009. 3D surface recontruction and analysis in automated apple stem-end/calyx indentification. Transactions of the ASABE 52, 1775-1784. Jiang, Y., LI, C., Paterson, A.H., 2016. High throughput phenotyping of cotton plant height using depth images under field conditions. Computers and Electronics in Agriculture 130, 57-68. Jijakli, M.H., Lepoivre, P., 2004. State of the art and challenges of post-harvest disease management in apples, In: Mukerji, K.G. (Ed.), Disease management of fruits and vegetables, Vol. 1. Kluwer Academic Publishers, Netherlands, pp. 59-94. Judge, T.R., Bryanston-Cross, P.J., 1994. A review of phase unwrapping techniques in fringe analysis. Optics and Laser in Engineering 21, 199-239. Kader, A.A., 2002. Postharvest technology of horticultural crops (Third Edition). University of California, Agriculture and Natural Resources. Kafle, G.K., Khot, L.R., Jarolmasjed, S., Si, Y., Lewis, K., 2016. Robustness of near infrared spectroscopy based spectral features for non-destructive bitter pit detection in honeycrisp apples. Postharvest Biology and Technology 120, 188-192. Kanick, S.C., McClatchy III, D.M., Krishnaswamy, V., Elliott, J.T., Paulsen, K.D., Pogue, B.W., 2014. Sub-diffusive scattering parameter maps recovered using wide-field high-frequency structured light imaging. Biomedical Optics Express 5, 3376-3390. Karoui, R., Blecker, C., 2011. Fluorescence Spectroscopy Measurement for Quality Assessment of Food Systems-a Review. Food Bioprocess Technol. 4, 364-386.   237 Kavdir, I., Guyer, D.E., 2002. Apple sorting using artificial neural networks and spectral imaging. Transctions of the ASAE 45, 1995-2005. Kavdir, I., Guyer, D.E., 2004. Comparison of artificial neural networks and statistical classifiers in apple sorting using textural features. Biosystems Engineering 89, 331-344. Kavdir, I., Guyer, D.E., 2008. Evaluation of different pattern recognition techniques for apple sorting. Biosystems Engineering 99, 211-219. Keresztes, J.C., Goodarzi, M., Saeys, W., 2016. Real-time pixel based early apple bruise detection using short wave infrared hyperspectral imaging in combination with calibration and glare correction techniques. Food Control 66, 215-226. Khanal, B.P., Shrestha, R., Huckstadt, L., Knoche, M., 2013. Russeting in apple seems unrelated to the mechanical properties of the cuticle at maturity. HortScience 48, 1135-1138. Khatiwada, B.P., Subedi, P.P., Hayer, C., Cunha Carlos Jnr, L.C., Walsh, K.B., 2016. Assessment of internal flesh browning in intact apple using visible-short wave near infrared spectroscopy. Postharvest Biology and Technology 120, 103-111. Kienle, A., Lilge, L.P., M.S., Hibst, R., Steiner, R.W., B.C., 1996. Spatially resolved absolute diffuse reflectance measurements for noninvasive determination of the optical scattering and absorption coefficients of biological tissue. Applied Optics 35, 2304-2314. Kim, M.S., Chao, K., Chan, D.E., Yang, C.C., Lefcourt, A.M., Delwiche, S.R., 2011. Hyperspectral and multispectral imaging technique for food quality and safety inspection, In: Cho, Y.J. (Ed.), Emerging technologies for food quality and food safety evaluation. CRC Press, Boca Raton, FL, USA, pp. 207-234. Kim, M.S., Lefcourt, A.M., Chen, Y.R., 2003. Multispectral laser-induced fluorescence imaging system for large biological samples. Applied Optics 42, 3927-3934. Kim, M.S., Lefcourt, A.M., Chen, Y.R., Kang, S.W., 2004. Uses of hyperspectral and multispectral laser-induced fluorescence imaging techniques for food safety inspection. Key Engineering Materials 270-273, 1055-1063. Kim, M.S., Lefcourt, A.M., Chen, Y.R., Kim, I., Chan, D.E., Chao, K., 2002. Multispectral detection of fecal contamination on apples based on hyperspectral imagery: Part II. Application of hyperspectral fluorescence imaging. Transactions of the ASAE 45, 2039-2047. Kim, S., Schatzki, T.F., 2000. Apple watercore sorting system using X-ray imagery: I. Algorithm development. Transactions of the ASAE 43, 1695-1702. Kleynen, O., Leemans, V., Destain, M.F., 2005. Development of a multi-spectral vision system for the detection of defects on apples. Journal of Food Engineering 69, 41-49. Knee, M., Miller, A.R., 2002. Mechanical injury, In: Knee, M. (Ed.), Fruit quality and its biological basis. Academic Press, Sheffield, UK, pp. 157-169.   238 Konecky, S.D., 2012. Diffuse optical tomography: imaging physics and image formation principles, In: Anastasio, M.A., La Riviere, P. (Eds.), Emgerging imaging technologies in medicine. CRC Press, Boca Raton, FL, p. 123138. Konecky, S.D., Mazhar, A., Cuccia, D.J., Durkin, A.J., Schotland, J.C., Tromberg, B.J., 2009. Quantitative optical tomography of sub-surface heterogeneities using spatially modulated structured light. Optics Express 17, 14780-14790. Krishnaswamy, V., Elliott, J.T., McClatchy, D.M., Barth, R.J., Wells, W.A., Pogue, B.W., Paulsen, K.D., 2014. Structured light scatteroscopy. Journal of Biomedical Optics 19, 070504. Kristensson, E., Berrocal, E., Alden, M., 2014. Two-pulse structured illumination imaging Optics Letters 39, 2584-2587. Krzewina, L.G., Kim, M.K., 2006. Single-exposure optical sectioning by color structured illumination microscopy. Optics Letters 31, 477-479. Kumar, A., Pang, G.K.H., 2002. Defect detection in textured materials using Gabor filters. IEEE Transactions on Industry Applications 38, 425-440. Kumari, S., Nirala, A.K., 2016. Biospeckle technique for the non-destructive differentiation of bruised and fresh regions of an Indian apple using intensity-based algorithms. Laser Physics 26, 115601. Kupferman, E., Waelti, H., 1993. Postharvest diseases and disorders of apples and pears. Tree Fruit Postharvest Journal 4, 3-4. Lal, A., Shan, C., Xi, P., 2016. Structured illumination microscopy image reconstruction algorithm. IEEE Journal of Selected Topics in Quantum Electronics 22. Larkin, K.G., 2014. The spiral phase transform, In: Rastogi, P., Hack, E. (Eds.), Phase estimation in optical interferometry. CRC Press, Boca Raton, FL, USA, pp. 121-139. Larkin, K.G., Bone, D.J., Oldfield, M.A., 2001. Natural demodulation of twodimensional fringe patterns. I. General background of the spiral phase quadrature transform. Journal of the Optical Society of America A 18, 1862–1870. Laughney, A.M., Krishnaswamy, V., Rizzo, E.J., Schwab, M.C., Barth Jr., R.J., Cuccia, D.J., Tromberg, B.J., Paulsen, K.D., Pogue, B.W., Wells, W.A., 2013. Spectral discrimination of breast pathologies in situ using spatial frequency domain imaging. Breast Cancer Research 15, R61. LeCun, Y., Boser, B., Denker, J.S., Henderson, D., Howard, R.E., Hubbard, W., Jackel, L.D., 1989. Backpropagation applied to handwritten zip code recognition. Neural Computation 1, 541-551. LeCun, Y., Bottou, L., Bengio, Y., Haffner, P., 1998. Gradient-based learning applied to document recognition. Proc. of The IEEE 86, 2278-2324.   239 Lee, K., Kang, S., Delwiche, S.R., Kim, M.S., Noh, S., 2008. Correlation analysis of hyperspectral imagery for multispectral wavelength selection for detection of defects on apples. Sensing and Instrumentation for Food Quality and Safety 2, 90-96. Leemans, V., Destain, M.F., 2004. A real-time grading method of apple based on features extracted from defects. Journal of Food Engineering 61, 83-89. Leemans, V., Magein, H., Destain, M.F., 1998. Defects segmentation on 'Golden Delicious' apples by using colour machine vision. Computers and Electronics in Agriculture 20, 117-130. Leemans, V., Magein, H., Destain, M.F., 1999. Defect segmentation on 'Jonagold' apples using colour vision and a Bayesian classification mothod. Computers and Electronics in Agriculture 23, 43-53. Leemans, V., Magein, H., Destain, M.F., 2002. On-line fruit grading according to their external quality using machine vision. Biosystems Engineering 83, 397-404. Lefcourt, A.M., Kim, M.S., 2006. Technique for normalizing intensity histograms of images when the approximate size of the target is known: detection of feces on apples using fluorescence imaging. Computers and Electronics in Agriculture 50, 135-147. Lefcourt, A.M., Kim, M.S., Chen, Y.R., 2003. Automated detection of fecal contamination of apples by multispectral laser-induced fluorescence imaging. Applied Optics 42, 3935-3943. Lefcourt, A.M., Kim, M.S., Chen, Y.R., 2005. Detection of fecal contamination in apple calyx by multispectral laser induced fluorescence. Transactions of the ASAE 48, 1587-1593. Lefcourt, A.M., Kim, M.S., Chen, Y.R., Kang, S., 2006. Systematic approach for using hyperspectral imaging data to develop multispectral imagining systems: detection of feces on apples. Computers and Electronics in Agriculture 54, 22-35. Li, J.B., Huang, W.Q., Zhao, C.J., 2015. Machine vision technology for detecting the external defects of fruits: a review. The Imaging Science Journal 63, 241-251. Li, R., 2016. Development of a structured illumination reflectance imaging system for enhanced detection subsurface and surface defects in apple fruit, Department of Biosystems and Agricultural Engineering. Michigan State University. Li, Y., Sun, Y., Peng, Y., Dhakal, S., Chao, K., Liu, Q., 2012. Rapid detection of pesticide residue in apple based on Raman spectroscopy, Sensing for Agriculture and Food Quality and Safety IV. Proc. SPIE, Baltimore, Maryland, USA Linderhed, A., 2009. Image empirical mode decomposition: a new tool for image processing. Advances in Adaptive Data Analysis 1, 265-294. Liu, B., Zhou, P., Liu, X., Sun, X., Li, H., Liu, M., 2013. Detection of pesticides in fruits by surface-enhanced Raman spectroscopy coupled with gold nanostructures. Food Bioprocess Technology 6, 710-718.   240 Liu, H., Su, W.H., Reichard, K., Yin, S., 2003. Calibration-based phase-shifting projected fringe profilometry for accurate absolute 3D surface profile measurement. Optics Communications 216, 65-80. Liu, Y., Gao, H., Gu, Q., Aoyama, T., Takaki, T., Ishii, I., 2014. High-frame-rate structured light 3-D vision for fast moving objects. Journal of Robotics and Mechatronics 26, 311-320. Lorente, D., Aleixos, N., Gomez-Sanchis, J., Cubero, S., Garcia-Navarrete, O.L., Blasco, J., Food Bioprocess Technol. 5, -. 2012. Recent advances and applications of hyperspectral imaging for fruit and vegetable quality assessment. Food Bioprocess Technology 5, 1121-1142. Lotze, E., Huybrechts, C., Sadie, A., Theron, K.I., Valcke, R.M., 2006. Fluorescence imaging as a non-destructive method for pre-harvest detection of bitter pit in apple fruit (Males domestica Borkh.). Postharvest Biology and Technology 40, 287-294. Lu, R., 2003. Detection of bruies on apples using near-infrared hyperspectral imaging. Transactions of the ASAE 46, 523-530. Lu, R., Cen, H., 2015. Measurement of food optical properties, In: Park, B., Lu, R. (Eds.), Hyperspectral imaging technology in food and agriculture. Springer, New York, pp. 203-226. Lu, R., Cen, H., Huang, M., Ariana, D.P., 2010. Spectral absorption and scattering properties of normal and bruised apple tissue. Transactions of the ASABE 53, 263-269. Lu, R.F., Chen, Y.R., 1999. Hyperspectral imaging for safety inspection of food and agricultural products, In: Chen, Y.R. (Ed.), Pathogen Detection and Remediation for Safe Eating. Spie-Int Soc Optical Engineering, Bellingham, pp. 121-133. Lu, Y., 2016. Food image recognition by using convolutional neural networks (CNNs). arXiv. Lu, Y., Huang, Y., Lu, R., 2017. Innovative hyperspectral imaging-based techniques for quality evaluation of fruits and vegetables: a review. Applied Science 7, 189. Lu, Y., Li, R., Lu, R., 2016. Structured-illumination reflectance imaging (SIRI) for enhanced detection of fresh bruises in apples. Postharvest Biology and Technology 117, 89-93. Lu, Y., Lu, R., 2017a. Histogram based automatic thresholding for bruise detection of apples by structured-illumination reflectance imaging. Biosystems Engineering 160, 30-41. Lu, Y., Lu, R., 2017b. Structured-illumination reflectance imaging coupled with spiral phase transform for bruise detection and three-dimensional geometry reconstruction of apples, ASABE Annual International Meeting. ASABE, Spokane, WA. Lu, Y., Lu, R., 2018. Fast bi-dimensional empirical mode decomposition as an image enhancement tecnique for fruit defect detection. Computers and Electronics in Agriculture 152, 314-323. Lukic, V., Markel, V.A., Schotland, J.C., 2009. Optical tomography with structured illumination. Optics Letters 34, 983-985.   241 Lurie, S., Watkins, C.B., 2012. Superficial scald, its etiology and control. Postharvest Biology and Technology 65, 44-60. Marigheto, N., Hills, B.P., 2005. MRI as a potential on-line sensor of apple quality, Information and Technology for Sustainable Fruit and Vegetable Production. FRUTIC, Montpellier, France, pp. 455-464. Marlow, G.C., Loescher, W.H., 1984. Watercore. Horticultural Reviews 6, 189-251. Marsland, S., 2015. Machine learning: an algorithmic perspective. . CRC Press, Boca Raton, FL, USA. Martinez-Romero, D., Serrano, M., Carbonell, A., Castillo, S., Riquelme, F., Valero, D., 2004. Mechanical damage during fruit post-harvest handling: technical and physiological implications, In: Dris, R., Jain, S.M. (Eds.), Production Practices and Quality Assessment of Food Crops Kluwer Academic Publishers, Netherlands, pp. 233-252. Martinsen, P., Oliver, R., Seelye, R., McGlone, V.A., Holmes, T., Davy, M., Johnston, J., Hallet, I., Moynihan, K., 2014. Quantifying the diffuse reflectance change caused by fresh bruises on apples. Transactions of the ASABE 57, 565-572. Martinsen, P., Schaare, P., 1998. Measuring soluble solids distribution in kiwifruit using near- infrared imaging spectroscopy. Postharvest Biology and Technology 14, 271-281. Mathanker, S.K., Weckler, P.R., Bowser, T.J., 2013. X-ray applications in food and agriculture: a review. Transactions of the ASABE 56, 1227-1239. Matousek, P., Morris, M.D., 2010. Emerging Raman applications and techniques in biomedical and pharmaceutical fields. Springer. Mazhar, A., Cuccia, D.J., Durkin, A.J., Tromberg, B.J., 2014. Spatial and temporal frequency domain tissue optical imaging, In: Wang, R., Tuchin, V.V. (Eds.), Advanced biophotonics: tissue optical sectioning. CRC Press, Boca Raton, FL, USA, pp. 109-136. Mazhar, A., Cuccia, D.J., Gioux, S., Durkin, A.J., Frangioni, J.V., Tromberg, B.J., 2010. Structured illumination enhances resolution and contrast in thick tissue fluorescence imaging. Journal of Biomedical Optics 15, 010506. McGlone, V.A., Martinsen, P.J., Clark, C.J., Jordan, R.B., 2005. On-line detection of Brownheart in Braeburn apples using near infrared transmission measurements. Postharvest Biology and Technology 37, 142-151. Meheriuk, M., Prange, R.K., Lidster, P.D., Porritt, S.W., 1994. Postharvest disorders of apples and pears. Agriculture Canada Publication 1737/E, Ottawa, Ont. Mehl, P.M., Chen, Y.R., Kim, M.S., Chan, D.E., 2004. Development of hyperspectral imaging technique for the detection of apple surface defects and contaminations. Journal of Food Engineering 61, 67-81.   242 Meilgaard, M.C., Civille, G.V., Carr, B.T., 2006. Sensory evaluation techniques. CRC Press, Boca Raton, FL, USA. Merner, L., Wang, Y., Zhang, S., 2013. Accurate calibration for 3D shape measurement system using a binary defocusing technique. Optics and Laser in Engineering 51, 514-519. Miller, B.K., Delwiche, M.J., 1991. Peach defect detection with machine vision. Transactions of the ASAE 34, 2588-2597. Mir, N., Wendorf, M., Perez, R., Beaudry, R.M., 1998. Chlorophyll fluorescence in relation to superficial scald development in apple. Journal of the American Society for Horticultural Science 123, 887-892. Mizushima, A., Lu, R., 2011. Cost benefits analysis of in-field presorting for the apple industry. Applied Engineering in Agriculture 27, 33-40. Moallem, P., Serajoddin, A., Pourghassem, H., 2017. Computer vision-based apple grading of golden delicious apples based on surface features. Information Processing in Agriculture 4, 33-40. Mohsenin, N.N., 1986. Physical properties of plant and animal materials: structure, physical characteristics and mechanical properties. Gordon and Breach Science Publishers, New York, USA. Moreda, G.P., Ortiz-Canavate, J., Garcia-Ramos, F.J., Ruit-Altisent, M., 2009. Non-destructive technologies for fruit and vegetable size determination - a review. Journal of Food Engineering 92, 119-136. Moshou, D., Wahlen, S., Strasser, R., Schenk, A., De Baerdemaeker, J., Ramon, H., 2005. Chlorophyll fluorescence as a tool for online quality sorting of apples. Biosystems Engineering 91, 163-172. Moshou, D., Wahlen, S., Strasser, R., Schenk, A., Ramon, H., 2003. Apple mealiness detection using fluorescence and self-organising maps. Computers and Electronics in Agriculture 40, 103- 114. Nadeau, K.P., Durkin, A.J., Tromberg, B.J., 2014. Advanced demodulation technique for the extraction of tissue optical properties and structured orientation contrast in the spatial frequency. Journal of Biomedical Optics 19, 056013. Nandy, S., Mostafa, A., Kumavor, P.D., Sanders, M., Brewer, M., Zhu, Q., 2016. Characterizing optical properties and spatial heterogeneity of human ovarian tissue using spatial frequency domain imaging. Journal of Biomedical Optics 21, 101402. Nara, K., Kato, Y., Motomura, Y., 2001. Involvement of terminal-arabinose and -galactose pectic compounds in mealiness of apple fruit during storage. Postharvest Biology and Technology 22, 141-150. NASS, U., 2016. Crop Production, Washington, D.C.   243 Neil, M.A.A., Juskaitis, R., Wilson, T., 1997. Method of obtaining optical sectioning by using structured light in a conventional microscope. Opt. Lett. 22, 1905-1907. Ng, H.F., 2006. Automatic thresholding for defect detection. Pattern Recog. Lett., 27(14), 1644- 1649. Pattern Recognition Letters 27, 1644-1649. Nicolai, B.M., Beullens, K., Bobelyn, E., Peirs, A., Saeys, W., Theron, K.I., Lammertyn, J., 2007. Nondestructive measurement of fruit and vegetable quality by means of NIR spectroscopy: a review. Postharvest Biology and Technology 46, 99-118. Nicolai, B.M., Defraeye, T., De Ketelaere, B., Herremans, E., Hertog, M., Saeys, W., Torricelli, A., Vandendriessche, T., Verboven, P., 5, -. 2014. Nondestructive measurement of fruit and vegetable quality. . Annual Review of Food Science and Technology 5, 285-312. Nicolai, B.M., Lotze, E., Peirs, A., Scheerlinck, N., Theron, K.I., 2006. Non-destructive measurement of bitter pit in apple fruit using NIR hyperspectral imaging. Postharvest Biology and Technology 40, 1-6. Niu, W., Zhong, L., Sun, P., Lu, X., 2015. Phase shifts extraction algorithm based on Gram– Schmidt orthonormalization of two vectors. Optical and Quantum Electronics 47, 2803-2810. Nixon, M.S., Aguado, A.S., 2012. Feature extraction & image processing for computer vision. Third edition ed. Elsevier Ltd., Singapore. Ntziachristos, V., 2010. Going deeper than microscopy: the optical imaging frontier in biology. Nature Methods 7, 603-614. Nunes, J.C., Bouaoune, Y., Delechelle, D., Niang, O., Bunel, P., 2003. Image analysis by bidimensional empirical mode decomposition. Image and Vision Computing 21, 1019-1026. O'Sullivan, T.D., Cerussi, A.E., Tromberg, B.J., Cuccia, D.J., 2012. Diffuse optical imaging using spatially and temporally modulated light Journal of Biomedical Optics 17, 071311. Ochoa, N.A., Garcia-Isais, C.A., 2015. Improvement on fringe projection profilometry. Optical Engineering 54, 104114. Ojala, T., Pietikainen, M.P., Maenpaa, T., 2002. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns. IEEE Transcations on Pattern Analysis and Machine Intelligence 24, 971-987. Opara, U.L., Pathare, P.B., 2014. Bruise damage measurement and analysis of fresh horticultural produce: a review. Postharvest Biology and Technology 91, 9-24. Otsu, N., 1979. A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics 9, 62-66. Pang, W., Studman, C.J., Ward, G.T., 1992. Bruising damage in apple-to-apple impact. Journal of Agricultural Engineering Research 52, 229-240.   244 Park, B., Lu, R., 2015. Hyperspectral imaging technology in food and agriculture. Springer, New York. Passoni, I., Dai Pra, A., Rabal, H., Trivi, M., Arizaga, R., 2005. Dynamic speckle processing using wavelets based entropy. Optics Communications 246, 219-228. Patterson, M.S., Moulton, J.D., Wilson, B.C., Berndt, K.W., Lakowicz, J.R., 1991. Frequency- domain reflectane for the determination of the scattering and absorption properties of tissue. Applied Optics 30, 4474-4476. Patterson, M.S., Pogue, B.W., 1994. Mathematical model for time-resolved and frequency-domain fluorescence spectroscopy in biological tissues. Applied Optics 33, 1963-1974. Paulus, S., Behmann, J., Mahlein, A.K., Plumer, L., Kuhlmann, H., 2014. Low-Cost 3D Systems: Suitable Tools for Plant Phenotyping. Sensors 14, 3001-3018. Penman, D.W., 2001. Determination of stem and calyx location on apples using automatic visual inspection. Computers and Electronics in Agriculture 33, 7-18. Pierson, C.F., Ceponis, M.J., McColloch, L.P., 1971. Market diseases of apples, pears and quinces. Agriculture Handbook No.376. USDA/ARS, Washington, D.C. Pietikainen, M., Hadid, A., Zhao, G., Ahonen, T., 2011. Computer vision using local binary patterns. Springer, London. Qin, J., Chao, K., Kim, M.S., 2010. Raman chemical imaging system for food safety and quality inspection. Transactions of the ASABE 53, 1873-1882. Qin, J., Chao, K., Kim, M.S., 2016. Raman scattering for food quality and safety assessment, In: Lu, R. (Ed.), Light scattering technology for food property, quality and safety assessment. CRC Press, Boca Raton, FL., USA, pp. 387-428. Qin, J., Lu, R., 2007. Measurement of the absorption and scattering properties of turbid liquid foods using hyperspectral imaging. Applied Spectroscopy 61, 388-396. Quiroga, J.A., Servin, M., 2003. Isotropic n-dimensional fringe pattern normalization. Optics Communications 224, 221-227. Racsko, J., Schrader, L.E., 2012. Sunburn of apple fruit: historical background, recent advances and future perspectives. Critical Reviews in Plant Sciences 31, 455-504. Rehkugler, G.E., Throop, J.A., 1986. Apple sorting with machine vision. Transactions of the ASAE 29, 1388-1397. Rehkugler, G.E., Throop, J.A., 1989. Image processing algorithm for apple defect detection. Transactions of the ASAE 32, 0267-0272.   245 Rizzolo, A., Vanoli, M., 2016. Time-resolved technique for measuring optical properties and quality of food, In: Lu, R. (Ed.), Light scattering technology for food property, quality and safety assessment. CRC Press, Boca Raton, FL, USA, pp. 187-224. Rizzolo, A., Vanoli, M., Spinelli, L., Torricelli, A., 2010. Sensory characteristics, quality and optical properties measured by time-resolved reflectance spectroscopy in stored apples. Postharvest Biology and Technology 25, 1-12. Rosell-Polo, J.R., Cheein, F.A., Gregorio, E., Andujar, D., Puigdomenech, L., Masip, J., Escola, A., 2015. Advances in structured light sensors applications in precision agriculture and livestock farming. Advances in Agronomy 133, 71-112. Rosin, P.L., 2001. Unimodal thresholding. Pattern Recognition 34, 2083-2096. Ruiz-Altisent, M., Ruiz-Garcia, L., Moreda, G.P., Lu, R., Hernandez-Sanchez, N., Correa, E.C., Diezma, B., Nicolai, B., Garcia-Ramos, J., 2010. Sensors for product characterization and quality of specialty crops-a review. Computers and Electronics in Agriculture 74, 176-194. Saager, R.B., Cuccia, D.J., Durkin, A.J., 2010. Determination of optical properties of turbid media spanning visible and near-infrared regimes via spatially modulated quantitative spectroscopy. Journal of Biomedical Optics 15, 017012. Saldner, H.O., Huntley, J.M., 1997. Profilometry using temporal phase unwrapping and a spatial light modulator-based fringe projector. Optical Engineering 36, 610-615. Saleh, B., 2011. Introduction to Subsurface Imaging. Cambridge Uiversity Press, New York. Salvi, J., Fernandez, S., Pribanic, T., Llado, X., 2010. A state of the art in structured light systems for surface profilometry. Pattern Recognition 43, 2666-2680. Salvi, J., Pages, J., Batlle, J., 2004. Pattern codification strategies in structured light systems. Pattern Recognition 37, 827-849. Schatzki, T.F., Haff, R.P., Young, R., Can, I., Le, L.C., Toyofuku, N., 1997. Defect detection in apples by means of x-ray imaging. Transactions of the ASAE 40, 1407-1415. Schrader, L., Sun, J., Felicetti, D., Seo, J.H., Jedlow, L., Zhang, J., 2003. Stree-induced disorders: effects on apple fruit quality Proceedings of the Washington Tree Fruit Post-harvest Conference, Wenatchee, Washington, pp. 1-7. Schrader, L., Zhang, J., Duplaga, W.K., 2001. Two types of sunburn in apple caused by high fruit surface (peel) temperature, Plant Health Progress. Seitz, S.M., Curless, B., Diebel, J., Scharstein, D., Szeliski, R., 2006. A comparison and evaluation of multi-view stereo reconstruction algorithms, Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06). IEEE.   246 Seo, J.H., Sun, J.S., Schrader, L., Tian, J., 2008. Use of Chlorophyll Fluorescence to Assess Heat Stress in Apple Fruit, In: Palmer, J.W. (Ed.), Proceedings of the International Symposium on Enhancing Economic and Environmental Sustainability of Fruit Production in a Global Economy. International Society Horticultural Science, Leuven 1, pp. 279-282. Sezgin, M., Sankur, B., 2004. Survey over image thresholding techniques and quantitative performance evaluation. Journal of Electronic Imaging 13, 146-168. Shahin, M.A., Tollner, E.W., Evans, M.D., Arabnia, H.R., 1999. Watercore features for sorting red delicious apples: a statistical approach. Transactions of the ASAE 42, 1889-1896 Shahin, M.A., Tollner, E.W., McClendon, R.W., 2001. Artificial intelligence classifiers for sorting apples based on watercore. Journal of Agricultural Engineering Research 79, 265-274. Shahin, M.A., Tollner, E.W., McClendon, R.W., Arabnia, H.R., 2002. Apple classification based on surface bruises using image processing and neural networks. Transactions of the ASAE 45, 1619-1627. Shapiro, L.G., Stockman, G.C., 2001. Computer vision. Prentice Hall, Upper Saddler River, NJ, USA. Shenderey, C., Shmulevich, I., Alchanatis, V., Egozi, H., Hoffman, A., Ostrovsky, V., Lurie, S., Arie, R.B., Schmilovitch, Z., 2010. NIRS detection of moldy core in apples. Food Bioprocess Technology 3, 79-86. Shewfelt, R.L., 1999. What is quality? Postharvest Biology and Technology 15, 197-200. Shewfelt, R.L., 2014. Measuring quality and maturity In: Florkowski, W.J., Shewfelt, R.L., Brueckner, B., Prussia, S.E. (Eds.), Postharvest handling: a system approach. Academic Press, San Diego, CA, USA, pp. 387-410. Shtienberg, D., 2012. Effects of host physiology on the development of core rot, caused by Alternaria alternata, in Red Delicious apples. The American Phytopathological Society 102, 769- 778. Si, Y., Sankaran, S., 2016. Computed tomography imaging-based bitter pit evaluation in apples. Biosystems Engineering 151, 9-16. Siedliska, A., Baranowski, P., Mazurek, W., 2014. Classification models of bruise and cultivar detection on the basis of hyperspectral imaging data. Computers and Electronics in Agriculture 106, 66-74. Simard, P.Y., Steinkraus, D., Platt, J.C., 2003. Best practices for convolutional neural networks applied to visual document analysis. ICDAR 3, 958-962. Sitnik, R., Kujawinska, M., Woznicki, J., 2002. Digital fringe projection system for large-volumn 360-deg shape measurement. Optical Engineering 41, 443-449.   247 Smith, E., Dent, G., 2005. Modern Raman spectroscopy - a practical approach. John Wiley & Sons, Ltd, Chichester, UK. Stewart, S., Priore, R.J., Nelson, M.P., Treado, P.J., 2012. Raman imaging. Annual Review of Analytical Chemistry 5, 337-360. Storch, M., Mishra, Y.N., Koegl, M., Kristensson, E., Will, S., Zigan, L., Berrocal, E., 2016. Two- phase SLIPI for instantaneous LIF and Mie imaging of transient fuel sprays Optics Letters 41, 5422-5425. Strang, G., 2009. Introduction to linear algebra. 4th ed. Wellesley-Cambridge. Studman, C.J., Brown, G.K., Timm, E.J., Schulte, N.L., Vreede, M.J., 1997. Bruising on blush and non-blush sides in apple-to-apple impacts. Transactions of the ASAE 40, 1655-1663. Su, W.H., 2007. Color-coded fringe projection for 3D shape measurements. Optics Express 15, 13167-13181. Su, X., Chen, W., 2004. Reliability-guided phase unwrapping algorithm: a review. Optics and Laser in Engineering 42, 245-261. Sun, D.W., 2010. Hyperspectral imaging for food quality analysis and control. Academic Press, London, UK. Tao, Y., 1996. Spherical transform of fruit images for on‐line defect extraction of mass objects. Optical Engineering 35, 344-350. Tao, Y., Wen, Z., 1999. An adaptive spherical image transform for high-speed fruit defect detection. Transactions of the ASAE 42, 241-246. Tartachnyk, I., Kuckenberg, J., Yuri, J.A., Noga, G., 2012. Identifying fruit characteristics for non- invasive detection of sunburn in apple. Scientia Horticulturae 134, 108-113. Thomas, B., Momany, M., Kner, P., 2013. Optical sectioning structured illumination microscopy with enhanced sensitivity. Journal of Optics 15, 094004. Throop, J.A., Aneshansley, D.J., Anger, W.C., Peterson, D.L., 2005. Quality evaluation of apples based on surface defects: development of an automated inspection systems. Postharvest Biology and Technology 36, 281-290. Throop, J.A., Aneshansley, D.J., Upchurch, B.L., 1995. An image processing algorithm to find new and old bruises. Applied Engineering in Agriculture 11, 751-757. Timm, E.J., Bollen, A.F., Dela Rue, B.T., Woodhead, I.M., 1998. Apple damage and compression forces in bulk bins druing orchard transport. Transactions of the ASABE 14, 165-172.   248 Torricelli, A., Spinelli, L., Contini, D., Vanoli, M., Rizzolo, A., Zerbini, P.E., 2008. Time-resolved reflectance spectroscopy for non-destructive assessment of food quality. Sensing and Instrumentation for Food Quality and Safety 2, 82-89. Tromberg, B.J., Coquoz, O., Fishkin, J.B., Pham, T., Anderson, E.R., Butler, J., Cahn, M., Gross, J.D., Venugopalan, V., Pham, D., 1997. Non-invasive measurements of breast tissue optical properties using frequency–domain photon migration. Philosophical Transactions of the Royal Society B: Biological Sciences 352, 661-668. Trusiak, M., Patorski, K., Wielgus, M., 2012. Adaptive enhancement of optical fringe patterns by selective reconstruction using FABEMD algorithm and Hilbert spiral transform. Optics Express 20, 23463-23479. Trusiak, M., Wielgus, M., Patorski, K., 2014. Advanced processing of optical fringe patterns by automated selective reconstruction and enhanced fast empirical mode decomposition. 230-240 52, 230-240. Tsai, R., 1987. A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses. Ieee Journal on Robotics and Automation 3, 323-344. Unay, D., Gosselin, B., Kleynen, O., Leemans, V., Destain, M.F., Debeir, O., 2011. Automatic grading of Bi-colored apples by multispectral machine vision. Computers and Electronics in Agriculture 75, 204-212. Upchurch, B.L., Throop, J.A., Aneshansley, D.J., 1994. Influence of time, bruise-type, and severity on near-infrared reflectance from apple surfaces for automatic bruise detection. Transactions of the ASAE 37, 1571-1575. Upchurch, B.L., Throop, J.A., Aneshansley, D.J., 1997. Detecting internal breakdown in apples using interactance measurements. Postharvest Biology and Technology 10, 15-19. USDA-NASS, 2017. Crop production (August 2017). Washington, DC: USDA National Agricultural Statistics Service. Vadivambal, R., Jayas, D.S., 2011. Applications of Thermal Imaging in Agriculture and Food Industry-A Review. Food Bioprocess Technol. 4, 186-199. Valero, C., Barreiro, P., Ruiz-Altisent, M., Cubeddu, R., Pifferi, A., Taroni, P., Torricelli, A., Valentini, G., Johnson, D., Dover, C., . J. Texture Stud. 36, -. 2005. Mealiness detection in apples using time resolved reflectance spectroscopy. Journal of Texture Studies 36, 439-458. van de Giessen, M., Angelo, J.P., Gioux, S., 2015. Real-time, profile-corrected single snapshot imaging of optical properties. Biomedical Optics Express 6, 4051-4062. Van Zeebroeck, M., Van Linden, V., Darius, P., De Ketelaere, B., Ramon, H., Tijskens, E., 2007a. The effect of fruit factors on the bruise susceptibility of apples. Postharvest Biology and Technology 46, 10-19.   249 Van Zeebroeck, M., Van Linden, V., Ramon, H., De Baerdemaeker, J., Nicolai, B.M., Tijskens, E., 2007b. Impact damage to apples during transport and handling. Postharvest Biology and Technology 45, 157-167. Vanoli, M., Rizzolo, A., Grassi, M., Spinelli, L., Verlinden, B.E., Torricelli, A., 2014. Studies on classification models to discriminate 'Braeburn' apples affected by internal browning using the optical properties measured by time-resolved reflectance spectroscopy. Postharvest Biology and Technology 91, 112-121. Vanoli, M., Rizzolo, A., Zerbini, P.E., Torricelli, A., 2010. Non-destructive detection of internal defects in apple fruit by time-resolved reflectance spectroscopy, In: Nunes, C. (Ed.), Environmentally Friendly and Safe Technologies for Quality of Fruits and Vegetables. Universidade do Algarve, Faro, Portugal, pp. 20-26. Vargas, J., Quiroga, J.A., Sorzano, C.O.S., Estrada, J.C., Carazo, J.M., 2012. Two-step demodulation based on the Gram–Schmidt orthonormalization method. Optics Letters 37, 443- 445. Varith, J., Hyde, G.M., Baritelle, A.L., Fellman, J.K., Sattabongkot, T., 2003. Non-contact bruise detection in apples by thermal imaging. Innovative Food Science & Emerging Technologies 4, 211-218. Vazquez-Arellano, M., Griepentrog, H.W., Reiser, D., Paraforos, D.S., 2016. 3-D imaging systems for agricultural applications - a review. Sensors 16, 618. Vervandier, J., Gioux, S., 2013. Single snapshot imaging of optical properties. Biomedical Optics Express 4, 2938–2944. Vo, M., Wang, Z., Pan, B., Pan, T., 2012. Hyper-accurate flexible calibration technique for fringe- projection-based three-dimensional imaging. Optics Express 20, 16926-16941. Volz, R.K., Biasi, W.V., Grant, J.A., Mitcham, E.J., 1998. Prediction of controlled atmosphere- induced flesh browning in 'Fuji' apple. Postharvest Biology and Technology 13, 97-107. Walsh, K., 2014. Postharvest regulation and quality standards on fresh produce, In: Florkowski, W.J., Shewfelt, R.L., Brueckner, B., Prussia, S.E. (Eds.), Postharvest handling: a system approach. Academic Press, San Diego, CA, USA, pp. 167-215. Wang, L., Ho, P.P., Liu, C., Zhang, G., Alfano, R.R., 1991. Ballistic 2-D imaging through scattering walls using an ultrafast optical Kerr gate. Science 253, 769-771. Wang, S.Y., Wang, P.C., Faust, M., 1988. Non-destructive detection of watercore in apple with nuclear magnetic resonance imaging. Sci. Hortic. 35, 227-234. Wang, Y., Laughner, J.I., Efimov, I.R., Zhang, S., 2013. 3D absolute shape measurement of live rabbit hearts with a superfast two-frequency phase-shifting technique Optics Express 21, 5822- 5832.   250 Wang, Z., Bovik, A.C., 2002. A universal image quality index. IEEE Signal Processing Letters 9, 81-84. Watkins, C.B., 2003. Postharvest physiological disorders of fresh crops, In: Thomas, B., Murphy, D., Murray, B. (Eds.), Encyclopedia of Applied Plant Sciences. Elsevier, Amsterdam, pp. 845-853. Watkins, C.B., Bramlage, W.J., Cregoe, B.A., 1995. Superficial scald of Granny Smith apples is expressed as a typical chilling injury. Journal of the American Society for Horticultural Science 120, 88-94. Weber, J.R., Cuccia, D.J., Durkin, A.J., Tromberg, B.J., 2009. Noncontact imaging of absorption and scattering in layered tissue using spatially modulated structured light. Journal of Applied Physics 105, 102028. Wicker, K., Heintzmann, R., 2010. Single-shot optical sectioning using polarization-coded structured illumination Journal of Optics 12, 084010. Williams, P., Norris, K., 1987. Near-infrared technology in the agricultural and food industries. American Association of Cereal Chemistries, Inc., St. Paul, MN, USA. Wills, R.B.H., Golding, J.B., 2016. Postharvest : an introduction to the physiology and handling of fruit and vegetables 6th Edition ed. UNSW Press, Sydney. Wilson, B.C., Jacques, S.L., 1990. Optical reflectance and transmittance of tissues: principles and applications. IEEE Journal of Quantum Electronics 26, 2186-2199. Wilson, T., 2011. Optical sectioning in fluorescence microscopy. Journal of Microscopy 242, 111- 116. Wu, Y.-C., Cao, Y.-P., Huang, Z.-F., Lu, M.-T., Chen, D.-L., 2012. Improved composite Fourier transform profilometry. Optics & Laser Technology 44, 2037-2042. Wunsche, J.N., Greer, D.H., Palmer, J.W., Lang, A., McGhie, T., 2001. Sunburn - The cost of a high light environment, In: Palmer, J.W., Wunsche, J.N. (Eds.), Proceedings of the Seventh International Symposium on Orchard and Plantation Systems. International Society Horticultural Science, Leuven 1, pp. 349-356. Xia, C., Wang, L., Chung, B.K., Lee, J.M., 2015. In Situ 3D segmentation of individual plant leaves using a RGB-D camera for agricultural automation. Sensors 15, 20463-20479. Xing, J., Bravo, C., Jancsok, P.T., Ramon, H., De Baerdemaeker, J., 2005 Detecting bruises on 'Golden Delicious' apples using hyperspectral imaging with multiple wavebands. Biosystems Engineering 90, 27-36. Xing, J., De Baerdemaeker, J., 2005. Bruise detection on 'Jonagold' apples using hyperspectral imaging. Postharvest Biology and Technology 37, 152-162.   251 Xu, X.Y., Xu, S.Z., Jin, L.H., Song, E.M., 2011a. Characteristic analysis of Otsu threshold and its applications. Pattern Recognition Letters 32, 956-961. Xu, Y., Ekstrand, L., Dai, J., Zhang, S., 2011b. Phase error compensation for three-dimensional shape measurement with projector defocusing. Applied Optics 50, 2572-2581. Yang, C.C., Kim, M.S., Kang, S., Tao, T., Chao, K., Lefcourt, A.M., Chan, D.E., 2011. The development of a simple multispectral algorithm for detection of fecal contamination on apples using a hyperspectral line-scan imaging system. Sensing and Instrumentation for Food Quality and Safety 5, 10-18. Yang, D., Ying, Y., 2011. Applications of Raman spectroscopy in agricultural products and food analysis: a review. Applied Spectroscopy Reviews 46, 539-560. Yang, Q., 1993. Finding stalk and calyx of apples using structured lighting. Computers and Electronics in Agriculture 8, 31-42. Yang, Q., 1994. An approach to apple surface feature detection by machine vision. Computers and Electronics in Agriculture 11, 249-264. Yang, Q., Marchant, J.A., 1996. Accurate blemish detection with active contour models. Computers and Electronics in Agriculture 14, 77-89. Yaseen, T., Sun, D.W., Cheng, J.H., 2017. Raman imaging for food quality and safety evaluation: Fundamentals and applications. Trends in Food Science & Technology 62, 177-189. Zappa, E., Busca, G., 2008. Comparison of eight unwrapping algorithms applied to Fourier- transform profilometry. Optics and Laser in Engineering 46, 106-116. Zdunek, A., Adamiak, A., Pieczywek, P.M., Kurenda, A., 2014. The biospeckle method for the investigation of agricultural crops: a review. Optics and Lasers in Engineering 52, 276-285. Zdunek, A., Pieczywek, P.M., Kurenda, A., 2016. Biospeckle technique for assessing quality of fruits and vegetables, In: Lu, R. (Ed.), Light scattering technology for food property, quality and safety assessment. CRC Press, Boca Raton, FL, USA, pp. 361-385. Zeile, M.Z., 2012. Adadelta: an adaptive learningrate method. arXiv preprint. Zhai, C., Peng, Y., Li, Y., Yang, Y., Chao, K., Qin, J., 2016. Nondestructive detection of carbendazim residue in apples by using surface-enhanced Raman spectroscopy, ASABE Annual International Meeting Orlando, FL, USA. Zhang, B., Huang, W., Gong, L., Li, J., Zhao, C., Liu, C., Huang, D., 2015. Computer vision detection of defective apples using automatic lightness correction and weighted RVM classifier. Journal of Food Engineering 146, 143-151.   252 Zhang, R., Ying, Y., Rao, X., Li, J., 2012. Quality and safety assessment of food and agricultural products by hyperspectral fluorescence imaging. Journal of the Science of Food and Agriculture 92, 2397-2408. Zhang, S., 2010. Recent progresses on real-time 3D shape measurement using digital fringe projection techniques. Optics and Laser in Engineering 48, 149-158. Zhang, S., 2016. High-speed 3D imaging with digital fringe projection techniques. CRC Press, Boca Raton, FL, USA. Zhang, S., 2018. High-speed 3D shape measurement with structured light methods: a review. Optics and Lasers in Engineering 106, 119-131. Zhang, S., Huang, P.S., 2006. High-resolution, real-time three-dimensional shape measurement Optical Engineering 45, 123601. Zhang, Z., 2000. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 22, 1330-1334. Zhang, Z., 2012. Microsoft Kinect sensor and its effect. . IEEE Multidedia 19, 4-10. Zhou, X., Lei, M., Dan, D., Yao, B., Qian, J., Yan, S., Yang, Y., Min, J., Peng, T., Ye, T., Chen, G., 2015. Double-exposure optical sectioning structured illumination microscopy based on Hilbert transform reconstruction. Plos One 10, e0120892. Zhou, X., Podoleanu, A.G., Yang, Z., Yang, T., Zhao, H., 2012. Morphological operation-based bi-dimensional empirical mode decomposition for automatic background removal of fringe patterns. Optics Express 20, 24247-24262. Zhu, B., Jiang, L., Tao, Y., 2007. Three-dimensional shape enhanced transform for automatic apple stem-end/calyx identification. Optical Engineering 46, 017201(017201-017209). Zhu, Q., Guan, J., Huang, M., Lu, R., Mendoza, F., 2016. Predicting bruise susceptibility of 'Golden Delicious' apples using hyperspectral scattering technique. Postharvest Biology and Technology 114, 86-94. Zou, X., Zhao, J., Li, Y., Mel, H., 2010. In-line detection of apple defects using three color cameras system. Computers and Electronics in Agriculture 70, 129-134.   253