‘ .' . A \ ‘ ~ 3. ~fififl MIC CHGANS TSAT IIIIIIII IIZIIII IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII 01572 5603 This is to certify that the thesis entitled DETECTING DEFECTS IN CHERRIES USING MACHINE VISION presented by Patchrawat Uthaisombut has been accepted towards fulfillment of the requirements for M.S. clegreeinComputer Science D5. George C. :tockman Major professor Date /é 2 V 9é Dr. Daniel E. Guyer 0-7639 MS U is an Affirmative Action/Equal Opportunity Institution LIBRAR MIChlgan st!“ Unlversuy PLACE ll RETURN BOX to romovo this checkout 1mm your rooord. TO AVOID FINES rotum on or bdoro dot. duo. DATE DUE DATE DUE DATE DUE MSU Is An Afilrmotivo Action/EM Opportunity lmtltulon Wan-on DETECTING DEFECTS IN CHERRIES USING MACHINE VISION By Patchrawat Uthaisombut A THESIS Submitted to Michigan State University in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE Department of Computer Science 1996 ABSTRACT DETECTING DEFECTS IN CHERRIES USING MACHINE VISION By Patchrawat Uthaisombut This thesis describes machine vision procedures which are able to classify defective cherries from non-defective cherries. Defects can be divided into bruises, dry cracks, and wet cracks. Bandpass filters that enhance the intensity contrast between bruised and unbruised cherries are determined. An optimum combination of two wavelengths is identified at 750 nm (infrared range) and 500 nm (green range). An optimum single wavelength is identified at 750 nm. The image acquisition using these filters is described. Four detection methods using single view infrared images are studied. Two methods perform well in classifying cherries with bruises and wet cracks from non- defective cherries. One detection method using single view green images is studied. It performs well in classifying cherries with dry cracks from non-defective cherries. One detection method using infrared images and another using green images are used in combination to perform the detection on the entire surface of cherries. Two images, infrared and green, are taken from each of 6 orthogonal directions from the cherries. The integrated classifier misclassified 13% of non—defective cherries, 16% of bruised cherries, 0% of cherries with wet cracks, and 10% of cherries with dry cracks. To my parents iii ACKNOWLEDGMENTS Many people contributed in many ways to make my M.S. thesis possible. I take this opportunity to express my gratitude to them. First, I wish to thank both of my thesis advisors Dr. Daniel E. Guyer and Dr. George C. Stockman for their time, effort, and guidance. I appreciate Dr. Guyer who gave me the research topic and supported me with a Research Assistantship. I also thank him for his encouragement. In many occasions, without his encouragement, I would not have had courage to make progress on the research. It was Dr. Stockman who introduced me to Computer Vision when I took his class. I wish to thank Dr. Stockman for recommending me to Dr. Guyer. Dr. Stockman always had an open door for discussion. He always gave me interesting and insightful ideas. Thanks are due to my committee member Dr. Anil K. Jain. I also appreciate Dr. Eric K. Torng, my Ph.D. dissertation advisor, for his patience and support. My sincere thanks go to all the members of my family. I thank my parents Adun and Walaitip for their support, patience, and encouragement to pursue higher edu- cation. I thank my sister Phorn—amarin for her understanding and patience. I also thank my girlfriend Rujida Leepipattanawit for her love, patience, and encourage- ment. Without her encouragement, I could not have finished this thesis. I wish to thank Dr. Robert H. Rasche and family. I appreciate their kindness for helping me settle down when I first came to the United States and taking care of me iv through out my stay at Michigan State University. Special thanks to Lora Mae Higbee who took care of all kinds of administrative work. I also thank Edward J. Timm for his assistance on the spectral radiometer. Finally, thanks to past and present members of the Pattern Recognition and Image Processing (PRIP) Laboratory. TABLE OF CONTENTS LIST OF FIGURES LIST OF TABLES 1 Introduction 1.1 Literature ................................... 1.2 Overview of Cherry Defects ......................... 1.3 Overview of the Thesis ............................ 2 Enhancing bruise contrast 2.1 Overview ................................... 2.2 Spectral Reflectance of Cherries ....................... 2.3 Gaussian Classifier .............................. 2.4 Error Estimation ............................... 2.5 Feature Selection ............................... 3 Image Acquisition 3.1 Imaging Configuration ............................ 3.2 Problems in Image Acquisition ....................... 3.2.1 Difficulties in Focusing ........................... 3.2.2 Image Misalignment ............................ 3.2.3 Structured Noise .............................. 4 Image Processing for Bruise Detection 4.1 Review Study of Bruise Detection using 2-band Images .......... 4.2 Image Preprocessing ............................. 4.3 Bruise Detection by the Global Histogram Method ............ 4.4 Bruise Detection by the Local Histogram Method ............. 4.5 Bruise Detection by Thresholding ...................... 4.6 Bruise Detection via Infrared Edge Detection ............... 4.6.1 Edge Detector ............................... 4.6.2 Finding Edge Detection Parameters .................... 4.7 Summary of Bruise Detection Methods ................... 5 Crack Detection 5.1 Dark Spot Mask Preparation ........................ 5.2 Classification Procedure ........................... vi vii 6 Integration 103 7 Discussion and Future Work 107 7.1 Discussion ................................... 107 7.2 Future Work ................................. 111 BIBLIOGRAPHY 1 14 1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.1 2.2 2.3 2.4 2.5 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 4.1 4.2 4.3 4.4 4.5 4.6 LIST or FIGURES Examples of non—defective cherries ..................... Examples of bruised cherries ......................... Examples of cherries with dry cracks .................... Examples of cherries with wet cracks .................... Examples of cherries with dry cracks and fungal organisms ........ Examples of cherries with wet cracks and fungal organisms ........ Examples of cherries with fresh stems .................... Examples of cherries with dry stems .................... Examples of cherry stems that are partially fresh and partially dry Components within the illumination chamber of the spectralradiometer . Spectral reflectance curves of cherries .................... Minimum resubstitution error versus number of features ......... Error estimation using a single feature ................... Error estimation using two features ..................... The imaging configuration .......................... Difficulty in focusing in different wavelengths ................ Focusing multiple objects at different distances .............. Variability in focus of cherry surface due to field of view ......... Demonstration of difficulty in focusing stems ................ Image misalignment ............................. Structured noise in image aquisition ..................... Model of the structured noise ........................ Details of the structured noise model .................... Noise inverse function construction ..................... Use of the noise inverse function ....................... Improvement of noise inverse function construction ............ Image before the structured noise is removed ................ Image after the structured noise is removed ................ Ideal 2D histogram .............................. Example of infrared and green images .................... Empirical 2D histogram ........................... Example of preprocessing .......................... Another example of preprocessing ...................... Diagram of a preprocessing procedure .................... viii 10 11 12 I3 14 15 15 16 16 22 23 28 29 30 34 36 45 47 48 49 52 53 54 56 57 58 4.7 Example of morphological opening ..................... 59 4.8 Unimodal and bimodal Gaussian distributions ............... 61 4.9 Diagram of the global histogram classification method .......... 63 4.10 Fading gray level near the edge of the cherry image ............ 65 4.11 Effect of amount of edge pixels removed and performance ......... 65 4.12 Visualization of new bimodality testing criteria .............. 66 4.13 Illustration of global histogram method failure ............... 67 4.14 Image segmentation in the local histogram method ............ 69 4.15 Diagram of the local histogram classification method ........... 70 4.16 Diagram of the last step of the classification ................ 71 4.17 Diagram of the thresholding classification method ............. 76 4.18 Diagram of infrared edge detection procedure ............... 80 4.19 Detailed diagram of edge detection in infrared edge detection procedure . 81 4.20 Intermediate results of infrared edge detection method .......... 83 4.21 Ramp model of gray level cross section on a bruise ............ 85 4.22 Relationship of window size and noise .................... 86 5.1 Edge and stem occlusion mask generation ................. 91 5.2 Examples of darks spots in green images .................. 92 5.3 Gray level histograms of green images .................... 93 5.4 Dark spot mask generation .......................... 94 5.5 Diagram of the green edge detection classification method ........ 95 5.6 Preprocessing step .............................. 96 5.7 Edge detection step .............................. 97 5.8 Masking step ................................. 98 5.9 Alternate classification procedure ...................... 99 5.10 Enhancing step ................................ 101 6.1 Diagram of the integrated classification procedure ............. 105 7.1 Structured light technique for crack detection ............... 113 1.1 2.1 2.2 4.1 4.2 4.3 4.4 4.5 4.6 4.7 5.1 6.1 LIST OF TABLES Appearance of common objects in cherry images .............. 14 Best wavelength combinations ........................ 27 Error estimate of 1- and 2—feature classifiers using different averaging band- widths ................................... 29 Error rate of the global histogram method ................. 64 List of experiment parameters of the local histogram method ....... 72 Error rate of the local histogram method .................. 73 Gray level statistics of unbruised and bruised surfaces ........... 76 Error rate of the thresholding method ................... 78 Error rate of the infrared edge detection method .............. 87 Summary of error rates of various bruise detection methods ....... 89 Error rate of the green edge detection method ............... 102 Error rate of the integrated classification procedure ............ 105 Chapter 1 Introduction Statement and Importance of Problem Current trends point to individual sorting of all processing and fresh market fruit and vegetables. Important factors include bruise damage resulting from harvesting and handling, naturally occurring or introduced surface defects, and variability in size and color. Sorting ranges from the individual producer manually sorting on his/ her location to consolidating fruit at a central location and sorting on a larger scale. The first approach works well for small scale fresh marketing but lacks quality control when the small sorting Operation is part of a larger marketing effort. Consolidating and sorting on a larger scale requires a larger labor force at the sorting location. Such a labor force is often difficult to train and manage and can be a major expense in the marketing process. Automated sorting technology, including machine vision, has the potential to rapidly and consistently sort fruit and vegetables. Electronic sorting technology is in 1 2 place for sorting in several commodity industries. Such technology has appeared to be unattractive to many members of the industry because currently available systems are too expensive to justify for smaller sorting lines as are often common in Michi- gan. Electronic sorting companies have focused on large scale applications which can afford such systems. Machine vision technology has increased in sophistication and ability while decreasing in cost. However, this technology remains unknown or un- affordable for many operations and, thus, the technology is not available to enhance final product quality and improve industry productivity. The most sophisticated optical / electronic sorting systems available today are able to color sort and size with ”good” accuracy, however, they are limited in their ability to sort surface and subsurface defects. The question remains whether technology is adequate and is affordable for measuring defects such as bruising and surface cracks. It is not unrealistic to think that a novel approach could be combined with inexpensive hardware making an automated sorting system feasible at the farm level. Keeping the sorting operation as close to the field as possible means that each system may have a lower capacity and thus the problem of sorting speed is not as important. Excessive sorting speed creates problems in the material handling aspects of the sorting operation with higher commodity velocities required, thus, increasing the potential for inaccurate sensing or damage from the sorting system itself. Specific Objectives The objective of this thesis is to study technology and techniques in the spectral reflectance, image processing, and pattern recognition areas for their ability to detect the common and grade reducing defects of brusing and cracking on dark sweet cherries. Brief Description of Experimental Approach The thesis will involve looking into lighting, optics, and image processing which can discern between defective and good quality tissue. Because of the physiology of the different tissues and the principles of reflectance and transmittance, there is a good possibility that a match of lighting and filtering combined with minimal image processing will result in the ability to discern good and defective tissue. Significance of Thesis The sweet cherry and other commodity industries can benefit from automated sorting because it has the potential of being more cost effective, less management intensive, more accurate, and more consistent than current approaches. The uniqueness and importance of this thesis is that it combines basic study in the area of defect detection that is not available on commercial systems with a practical application approach which will hopefully benefit smaller sorting operations. 1. 1 Literature There are many non-invasive techniques to inspect fruits and vegetables. This includes surface reflectance, transmittance, acoustic response, mechanical deformation, x-ray, computed tomography (CT), and magnetic resonance imaging (MRI). Chen and Sun [4] gave a review of non—destructive techniques for evaluating fruit quality. Imaging techniques discussed include acoustic properties, optical reflectance, X-ray, CT, and MRI. Marchant [17] presented a general knowledge on the use of computer vision systems for inspection of agricultural products. Several projects [7, 9, 12, 16, 18, 26, 27, 28, 30, 35, 36] used surface reflectance of fruits to perform automatic grading. Upchurch and Throop [34] used light transmission through apples to detect watercore damage. Some projects [2, 5, 25] studied the relation between acoustic response of fruits and their quality. Mizrach, Nahir, and Ronen [20] studied the relation of firmness of oranges and tomatoes and their deformation under an applied force. Tollner et. a1. [31] related the water content of an apple and the x—ray absorption. Important factors of fruit quality are surface defects, size, shape, color, firmness, and ripeness. Examples of surface defects are bruises, cracks, scars, cuts, and worm- holes. Forking in a carrot is an example of a shape defect. In many cases, color and firmness of fruits relate to their ripeness. Many studies [7, 18, 22, 27, 28, 29] discussed fruit grading based on their surface defects. Automatic grading by size and shape is discussed in [11, 12, 16, 36]. Delwiche et a1. [8] discussed the ambiguity in color stan- dards for cherries, apples, peaches, and tomatoes. Singh et a1. [26] discussed peach ripeness automatic grading based on a color set by the United States Department of 5 Agriculture (USDA). Armstrong et al. [2] and Mizrach et al. [20] discussed automatic grading by fruit firmness. Ferraz, Bilanski, and Alaee [9] discussed a procedure to grade potato tubers by their water content. Many agricultural products have been studied related to automatic grading. Parker [22] studied the efficiency of manual grading on cherries. Grading of ap— ples was studied in [2, 5, 28, 29, 30, 31, 32, 33, 34, 35]. Peach grading studies are [18, 19, 26, 27]. Delwiche, Tang, and Thompson [7] studied automatic grading of dried prunes. Potatoes are studied in [9, 11, 16]. Mirach, Nahir, and Ronen [20] studied firmness of tomatoes and oranges. Howarth and Searcy [12] studied carrot grading by shape. Wolfe and Swaminathan [36] studied bell pepper grading by shape. In many studies, optical filters were used to enhance the contrast among different kinds of surfaces. Several studies [7, 12, 18, 26, 27, 28, 30] used optical filters in automatic grading. Affeldt and Winner [1] and Parker [22] reported that using optical filters helps enhance the human perceptibility of defects on fruits. A number of studies [19, 22, 32, 33] analyzed fruit spectral reflectance to find optimal wavelengths that give the most contrast between different surface types. Next, we present some automatic grading systems in the literature. Miller and Delwiche [18] studied a peach defect detection algorithm. Color im- ages and an infrared image at 750nm were taken from each sample. The illumination chamber was a 584mm long 300 mm diameter cylindrical diffuser. The illumination source was four 50W tungsten halogen bulbs mounted on an end section of the cham- ber. A diffusion filter was placed inside the chamber such that the peaches were exposed only to indirect diffused light. The camera was mounted at the top of the 6 chamber. The peach was placed at the bottom of the chamber. Although the illu- mination chamber was designed for uniform illumination, the peach images did not have constant reflectance across the surface, but tended to be darker near the edges at surface concavities. The gray level of peach images was corrected by multiplying by a scale factor. The correction factor lookup table was obtained from images of spheres. The lookup table was indexed by the pixel angle with respect to the sphere centroid and normalized distance from the sphere edge. The peach image was then segmented into regions as follows. Edge detection was applied to the image. Local edge information was then analyzed to seperate defect edges from noise. Edge pixels were linked based on the edge magnitude. Except for bruise defects, defect regions were grown from previously identified edges. For bruise defects, the bruise region was detected by thresholding the infrared image. After the potential defect regions were identified, features of those regions were computed. This included area, perimeter, mean gray level, gray level variance, elongation, and vari- ance in gradient directions. The feature space was partitioned into regions associated with nondefective peach surface and various defect types. From the images of training samples, the means and covariance matrices of each defect type could be estimated. The potential defect regions from the test samples could be classified using the Bayes classifier. Overall, the algorithm misclassified 30% of the test samples, most of which were false dismisals, ie., samples were assigned a grade better than the actual grade. Singh et a1. [26] studied peach maturity automatic grading. The peach maturity is largely related to its surface color. In their experiment, red and green (R-G) values 7 from RGB color images were used. The blue values were not used because they have little contribution in classifying peach maturity. The peach maturity was divided into six classes based on the ”greenest” 10% of the peach surface. A peach was in a maturity class if no more than 10% of the greenest part of the surface fell in the color standard associated with that maturity or lower class. The illumination chamber was a 30.5 cm diameter translucent sphere. The top part of the sphere was cut and a camera was mounted at the top of the chamber. The bottom part of the sphere was cut to allow a peach sample to be placed. Four 50W tungsten halogen lamps were used for illumination. The mean and covariance matrix of each color of each maturity class was determined from the training samples. Three R—G images were acquired from each test sample. Each of the three images were taken from 120° rotation of the same peach. The Bayes classifier was used to classify each pixel in the image. The number of pixels in each color class were determined. The peach was then classified into a maturity class based on the number of pixels in each color class. 46% of automatic grading agreed with manual grading. 75% of automatic and manual grading were different within one maturity class. Throop and Aneshansley [30] studied an apple bruise detection procedure. Images of a delrin sphere and apples were captured by a line—scan camera. Each column of the delrin sphere image was normalized by dividing by the mean gray level of each column and multiplying by a constant 200. The apple image was cleaned with a 5 x 5 mean filter and normalized using a method similar to the delrin sphere. By normalizing the gray level of both the delrin image and apple image, their gray levels were approximately equal. The apple image was subtracted from the delrin image. A 8 bruise would appear brighter in the resulting image. A 5 x 5 mean filter was applied to the image and the image was thresholded. Small components were removed by morphologically closing the image with a 5 x 5 structuring image. The shape factor for each component (1.0 for a circle) was computed. If the shape factor was within a predetermined range, the component was counted as a bruise and the area was recorded. An average of 1 false bruise cluster could be expected on each apple with a 45 mm2 area. 9 1.2 Overview of Cherry Defects Cherries are divided into two classes, non-defective and defective. Defective cherries are further divided into those which are bruised and cracked. Some of the bruised cherries were bruised during the handling to the laboratory. The others were bruised manually by dropping from 50 cm height onto a steel surface. To allow bruises to develop, the bruised cherries were left at the room temperature for various lengths of time from 1 hour to 1 day. The shape and size of bruises vary and their color is hard to distinguish from non-defective surfaces by human eyes. Spectral reflectance features of cherries can be used to help detect bruises. It was decided that the reflectant wavelength to be used in taking bruised cherry images be optimized. Cherry images taken at the optimum wavelength would have the greatest brightness contrast between non-defective surfaces and bruises. In Section 2 (enhanc- ing bruise contrast), the results Show that the brightness of bruises and non-defective surfaces contrasted most in the near infrared range (750 nm, 40 nm bandwidth). Fig- ure 1.1 shows some images of non-defective cherries. Figure 1.2 shows some images of bruised cherries. These images were taken in the infrared range (750 nm, 40 nm bw) and green range (500 nm, 70 nm bw). The non-defective surfaces are bright while the bruises are darker. Background is made of a piece of paper printed black. It appears dark in both the infrared and the green ranges. All cracks occured naturally. In the experimental sample, it was not expected that the reflectant wavelength needed to be optimized. Cracks in images taken in the green range (500 nm, 70 nm bandwidth) were clearly visible. Figure 1.3 shows some (b) (d) (f) Figure 1.1: Examples of non-defective cherries. (a), (c), and (e) are images taken using the infrared filter (750 nm, 40 nm bw); (b), (d), and (f) are the images taken from the same cherries and same orientation as (a), (c), and (e) respectively using the green filter (500 nm, 70 nm bw) (b) (d) (0 Figure 1.2: Examples of bruised cherries. (infrared at top, green at bottom) images of cracks taken in the green range. It was discovered that the reflectance of a crack can vary depending on whether it is dry or wet. In the green range (500 nm), the average gray level of a small patch of area on a dry crack is usually brighter than a non—defective surface but sometimes cracks are darker than non—defective area. Even though the average gray level of dry cracks might be brighter or darker than the average gray level of non-defective surfaces, dry cracks have more texture than non-defective surfaces. There are more grey level variations in small patches of cracks than on non—defective areas. Figure 1.3 (b), (d), and (I) shows some pictures of dry cracks in the green range. In the infrared range (750 nm), dry cracks have about the same brightness as good surfaces. In fact, dry cracks are virtually invisible in the infrared range. Figure 12 (b) 4 (d) (0 Figure 1.3: Examples of cherries with dry cracks. (infrared at top, green at bottom) 1.3 (a), (c), and (6) shows some pictures of dry cracks in the infrared range. These pictures are taken in the same view as images in the green range. Unlike dry cracks, wet cracks are as dark as good tissue in the green range and appear quite dark in the infrared range. Figure 1.4 shows some wet cracks in the green and infrared range. Compare these pictures with those in Figure 1.3 of dry cracks. Sometimes, images of cracks taken in the infrared range have sharp dark lines. These dark lines are not the crack but they are fungal organisms. They appear very dark in the infrared range and as dark as good surfaces in the green range. Figure 1.5 shows some dry cracks with fungal organisms along the edges of the cracks. In old wet cracks, fungal organisms are often present. Figure 1.6 shows some wet cracks (b) (d) (0 Figure 1.4: Examples of cherries with wet cracks. (infrared at top, green at bottom) with fungal organisms. The other objects visible in images are cherry stems. In the infrared range, the average gray level of stems is not significantly different from that of non-defective surfaces. However, the gray level of stems has more variance. Sometimes stems appear brighter and sometimes darker than non-defective surfaces. In the green range, the gray level of stems also varies. Fresh stems which are visually green are bright in the green band (Figure 1.7). In contrast, dry stems which are visually brown are dark in the green range (Figure 1.8). On a stem that begins to dry, there can be fresh and dry portions. Figure 1.9 shows some pictures of cherry stems that are partially fresh and partially dry. Table 1.1 summarizes the appearance of various objects in cherry images. (d) ' ._ ' (0 Figure 1.5: Examples of cherries with dry cracks and fungal organisms. (infrared at top, green at bottom) green 40nm bw 500 70mm bw Table 1.1: Appearance of common objects in cherry images (b) (0 Figure 1.6: Examples of cherries with wet cracks and fungal organisms. (infrared at top, green at bottom) (d) (0 Figure 1.7: Examples of cherries with fresh stems. (infrared at top, green at bottom) 16 (C) (b) (d) (f) Figure 1.8: Examples of cherries with dry stems. (infrared at top, green at bottom) (d) Figure 1.9: Examples of cherry stems that are partially fresh and partially dry. (in— frared at top, green at bottom) 17 The reason why different surface types have different brightness in the infrared range can be partially described as follows. The outermost surface of a cherry is the cherry skin. The cherry skin is very thin. Underneath the skin is the cherry flesh. The cherry flesh is composed of cells. In a non—defective cherry, there are tiny gaps (viods) of air between flesh cells. When the cherry is illuminated, the infrared light energy penetrates the skin and slightly into the flesh. The infrared light is diffused by these tiny gaps of air. Thus, non-defective cherry surfaces appear bright in the infrared range. Bruises usually occur during harvesting and handling. On a bruised surface, the cherry skin is not broken but cells in the cherry flesh are damaged. The water in the cells is released and fills the spaces between cells. When the cherry is illuminated, the infrared light energy penetrates the skin into the flesh. In this case, there are no tiny gaps of air to diffuse the infrared and since water absorbs infrared light, bruises appear darker than non-defective surfaces. When cherries are still on trees, cracks usually occur after it rains. When it rains, the cherry skin and tree absorbs water. The rapid uptake and absorption of water results in cracking. On a crack, the cherry skin is broken and cells of the cherry flesh are separated. However, the cells themselves are not broken and tiny gaps of air between cells are not filled. Therefore, cracks diffuse and reflect infrared as much as non-defective surfaces. Thus, dry cracks remain as bright as non-defective surfaces in the infrared range. If cracks are old, they start to decay. Cells are destroyed and water fills the space between cells. Moreover, water is on the surface of the cracks. Hence, wet cracks (or 18 decay) absorb more infrared than non-defective surfaces. Therefore, wet cracks are darker than non-defective surfaces in the infrared range. 1.3 Overview of the Thesis In Chapter 2, a method to enhance contrast between bruised and non-defective sur- faces is described. The spectral reflectance curves of non-defective and bruised cherries were obtained using a spectral radiometer. A combination of 2 wavelengths at 750 nm (infrared band) with bandwidth 40 nm and at 500 nm (green band) with bandwidth 70 nm was identified to give the greatest brightness contrast between non-defective surface and bruised surface. The image acquisition is described in Chapter 3. An imaging configuration was constructed. A light diffuser was used to obtain uniform illumination. Two band- pass filters identified above were used together with a black/ white camera to acquire cherry images. In Chapter 4, four methods for classifying cherries as defective or non-defective using single view infrared images are presented. The main focus of clas- sification procedures in this chapter is to classify cherries as bruised or non-defective. These classification procedures are the global histogram method, the local histogram method, the thresholding method, and the edge detection method. They were tested with 25 non-defective cherries and 60 bruised cherries. The last two methods outper— formed the first two methods. The last two methods have comparable performance. Both of them misclassified 5% of good cherries and 15% to 25% of bruised cherries depending on a priori probability of bruised cherries in the test samples. Although 19 classification procedures in Chapter 4 mainly focused on classifying cherries as bruised or non-defective, the last two methods also performed well in classifying cherries as wet cracked or non-defective. In Chapter 5, a procedure for classifying cherries as defective or non-defective using single View images in the green range is presented. The main focus is to classify cherries as dry cracked or non-defective. The classifier was tested with 50 non—defective cherries and 14 dry cracked cherries. The classifi- cation procedure misclassified 2% of good cherries and 21% of dry cracked cherries. In Chapter 6, we describe an integration of classification procedures using multiple infrared and green images from different views of cherries. The procedure was tested with 23 non-defective cherries, and 73 defective cherries. Among defective cherries, 31 of them had bruises but no cracks, 32 of them had wet cracks, and 10 of them had dry cracks. The classification procedure misclassified 13% of non—defective cherries, 16% of bruised cherries, no wet cracked cherries, and 10% of dry cracked cherries. The conclusion and future work are discussed in Chapter 7. A bruise detection procedure using combined infrared and green images is discussed. A crack detection procedure using structured light is proposed. Chapter 2 Enhancing bruise contrast 2.1 Overview If we are to base cherry classification on the optical reflectance of cherries, it is necessary to be able to distinguish good area versus bruise on cherries based on their reflectances. To do this we can choose band(s) of reflectant light energy. To classify cherries well, we need to find the band(s) of wavelength where the reflectance of bruise area and good area have maximum contrast. The first step toward this goal is to find the spectral reflectance of cherries over some range of wavelength of interest. We studied the visible and near infrared ranges (400-1100 nm). Before selecting the best band(s) of wavelength, we have to first specify the criteria for classification. Knowing the classification criteria, we can then exhaustively search for the optimal wavelength(s). In this section we decribe how to determine the spectral reflectance of cherries, the Gaussian classifier, classifier evaluation, and wavelength selection procedure. 20 21 2.2 Spectral Reflectance of Cherries The percentage spectral reflectance of cherries is determined using a spectral ra- diometer. The spectral radiometer used was a L1-1800 Portable Spectralradiometer, Ll—COR Inc. To scan an object, the object is placed in an illumination chamber. The wall of the chamber is painted black to protect the object from the light from out- side and light reflecting inside the chamber itself. Figure 2.1 shows the components within the illumination chamber used together with the spectral radiometer. There is a coaxial optical fiber bundle conducting light to and from the illumination chamber. The coaxial optical fiber bundle is composed of two optical fiber bundles. The outer bundle conducts the light from the illumination source to the object. The diameter of the outer bundle is approximately 15 mm. The inner bundle conducts the light reflected from the object to the sensor of the spectral radiometer. The diameter of the inner bundle is approximately 5 mm. The sensitivity of the sensor of the spectral radiometer is in the range of 400 nm to 1100 nm with 1 nm increment. A standard white pad was scanned to establish a baseline. The standard white pad is a white object with a flat surface that reflects 98% of light over the range 400 nm to 1100 nm. 72 good cherries and 65 bruised cherries were scanned. Each cherry was scanned to get the raw spectral reflectance. This raw spectral reflectance was then normalized by the raw spectral reflectance of the standard white pad. The result was the percentage spectral reflectance of each cherry. Group spectral reflectance curves are shown in Figure 2.2 (b) and (c). Each cherry has its own reflectance curve as shown in Figure 2.2 (a). In other 22 .——— Inner fiber optic bundle --—— Outer fiber optic bundle ’— Cherry Top View a... I .1: 7’ :: Outer fiber optic bundle {_ _._ <—————— ————————— ————_—_—_‘ from light source Inner fiber optic bundle IO SCI'ISOI' I I "i’ Frontal View Figure 2.1: Components within the illumination chamber of the spectralradiometer 23 ............................................................ — good cherries i 0;..Igg 4005006007008009001000 —r'nean E ' ‘ E E _‘ -—r'nean . ' : ; 06_.7..-...:m6.antt$.d. mean-.s.d...., ........ 06..:..-..maan+s..d. meantsd .......... ........ .0 on .0 :- reflectance (traction) p to 0.2 0.1 0 g A g A 4 L 0 r I A i 3 g 400 500 600 700 800 900 1000 400 500 600 700 800 900 1000 wavelength (nanometer) wavelength (nanometer) (b) (c) Figure 2.2: Spectral reflectance curves of cherries. (a) some individual cherries. (b) statistics of good cherries. (c) statistics of bruised cherries. 24 words, each cherry has 701 features r,-, each feature is the percentage reflectance r,- at wavelength i=400 nm to 1100 nm with 1 nm increments. In the next two subsections, we describe the criteria for selecting which and how many of these features should be used for a classification decision. 2.3 Gaussian Classifier Each sample cherry is represented by its feature values. In our case, features of a cherry are percentage reflectance at some number of wavelengths. Gaussian classifiers assume that sample features are drawn from Gaussian (or normal) distributions. There is one Gaussian distribution for each class of samples. In our case the number of classes is two, good cherries and bruised cherries. The probability density function of a Gaussian distribution is shown in the Equation (2.1). 1 __ r- 2 ammo?) = e “+3 (2.1) v2rra A multivariate Gaussian distribution has, mean vector [2 and covariance matrix 6. Since we do not know the true value of these parameters, they are estimated from the available samples. We estimate the parameters using the maximum likelihood method as shown in Equation (2.2) and (2.3). 25 :1 — i, a.=192 - :mé . <22> In . =—;( ant—fl)! , 2:1,2 (2.3) n kl: The maximum likelihood decision function is shown in the Equation (2.4). A sample is classified as class 1 if the probability density of class 1 at the sample point is greater than that of class 2. good if Hail/11,01) Z Phi/12,02), 6(33) = (2-4) bruise otherwise In the next subsection, we describe evaluation of the classification procedures. 2.4 Error Estimation There are a number of ways to evaluate a classifier. The probability of misclas- sification is the most effective measure. Some of the methods are the resubstitu- tion method, the hold out method, the leave-one-out method, and the bootstrapping method. In this research, the resubstitution method was used. The resubstitution method uses all samples for training as well as testing. In our case, all 72 good cherry samples and 65 bruised cherry samples were used. Equation (2.5) shows the formula for the probability of misclassification. 26 number of samples misclassified P3 = (2.5) number of total samples The resulting error estimate from this method is biased optimistically. However we adopted this method in estimating error since it is easy to implement, requires the least computational time, and we primarily used it to compare the error among classfiers using different combinations of features. Note also that this error estimate considers false alarms and false dismisals to be equally costly, which may not be true in practice. 2.5 Feature Selection We have the spectral reflectance of cherries at discrete points in the range of 400 nm to 1100 nm with 1 nm increments. This means that for each cherry, 701 features exist. We can classify the cherries using these features. We have to limit the number of features to a small number. From the practical point of view, using reflectances at too many wavelengths is too costly in data collection and processing relative to the increase in classification accuracy. Theorectically, when we have finite samples, using too many features may cause over fitting to samples and it is possible that training samples will be classified with high accuracy but future data may not be classified accurately. Given a set of samples with d features and the number of desired features k, feature 27 selection tries to choose the best It features from the given d features. A combination of It: features is the best if the probability of misclassification using these It features is the minimum among all combinations of It: features from the given d features. A set of average spectral reflectance data were generated from the original data. In the data set, each 20 nm range of the original spectral reflectance data was averaged. We performed the feature selection on this data set a number of times with the number of desired features ranging from 1 to 5. The result from feature selection gives us two things. First, it helps us to choose the number of features to be used. Secondly, it tells us which combination of feature(s) is best. The plot of minimum estimated error versus number of features is shown in Figure 2.3. Table 2.1 shows the best combinations of 1 to 5 wavelengths. The more the features used the better we are able to classify cherries. However, at some point, adding more features does not give significantly better performance. The rule-of- thumb suggests we choose the number of features at the knee of the graph. Thus, we should choose 2 or 3 features; we choose 2 for sensing efficiency. wavelength (nm) # of features 1st feature 2nd feature 3rd feature 4th feature 5th feature 1 740 - —- - - 2 440 740 - - - 3 580 760 880 - - 4 580 760 820 840 - 5 440 580 620 660 920 Table 2.1: Best wavelength combinations Feature selection was performed again on the original data with 1 and 2 desired ................................................................ 0.25 0.2 0.15 resubstitution error 0.1 0.05 number of features Figure 2.3: Minimum resubstitution error versus number of features features and no bandwidth pooling. Recall that the original spectral reflectance has bandwidth 1 nm. The single best feature is the spectral reflectance at wavelength 732 nm with 1 nm bandwidth. The best combination of 2 features is the spectral reflectance at wavelength 427 nm and 747 nm with 1 nm bandwidth. This result suggests we use 2 bandpass filters centered at 427 nm and 747 nm with a bandwidth of 1 nm in an imaging system. Since filters with such a narrow bandwidth were not currently available at an affordable price, we decided to use filters with wider bands. Four sets of average spectral reflectance data were generated from the original data. In the first set, each 5 nm range of the original Spectral reflectance data was averaged. The other 3 sets of data were generated similarly with the bandwidth of 10, 20, and 40 nm. The feature selection procedure was performed on these data with the number of desired features 1 and 2. The result is shown in Table 2.2. From the table we claim 29 that we do not suffer too much from averaging the reflectance over ranges up to 40 nm. The reason is that neighboring bandwidths are strongly dependent. best error estimate bandwidth 1 feature 2 features 1 nm 24% 8.0% 5 nm 24% 8.8% 10 nm 24% 8.8% 20 nm 24% 8.8% 40 nm 24% 9.5% Table 2.2: Error estimate of 1- and 2—feature classifiers using different averaging bandwidths The plot of the error estimation versus each single feature is shown in Figure 2.4. The plot of the error estimation versus the two features is shown in Figure 2.5. The error estimation shown in these two figures are for the data set whose average bandwidth is 5 nm. resubstitution error 0 .0 .0 (A) # 01 .° to .0 —ul 0 , 3. A I , I 400 500 600 700 800 900 1000 wavelength (nm) Figure 2.4: Error estimation using a single feature 30 wavelength (nm) ‘1 m 8 8 c» 8 01 O O 400 400 500 600 700 800 900 10001100 wavelength (nm) Figure 2.5: Error estimation using two features. The whiter the less the error esti- mation. From Figure 2.5, the small region centered at 425 nm and 735 nm is where the error estimation is lowest. However, this region is too small and we may not get filters that fit this region. In the region bounded by 425-580 nm and 710—780 nm, the error estimation is uniform and not much higher than the lowest error. This region has size 155 nm and 70 nm respectively. It was decided to choose filters that fall in this region. The bandpass filters we actually obtained are centered at 500 nm with bandwidth 70 nm (465-535 nm), and centered at 750 nm with bandwidth 40 nm (730—770 nm). A set of average spectral reflectance data in these two ranges were generated from the original data. The error estimate of a classifier using these two features is 9.5%. Note that the reflectance at wavelength around 750 nm is also the best feature for l-feature classification. The error estimate of a classifier using the single feature averaged with bandwidth 40 nm around the wavelength 750 nm is 24%. 31 Empirical analysis using an image sensor and filters did not correlate well with feature selection using a spectral radiometer. As we shall see in Chapter 4, we could not realize the benefit of using two features from the real image sensor. Chapter 3 Image Acquisition 3.1 Imaging Configuration A method to discriminate between small surface elements of non-defective and bruised cherry flesh using reflectance was presented in Chapter 2. In this section, we describe how we image a cherry from a single viewpoint using special lighting, filtering, and a CCD camera. Figure 3.1 shows the imaging configuration. The background is a piece of black paper on the optical table. Cherry samples are placed at the center of the background paper which is considered as the center of the imaging configuration. The light sources are 100 watt incandescent Sylvania cool white light bulbs. The reflectors are paraboloids with a cross section of 20 cm diameters. The light bulbs together with the reflectors illuminate from three equally spaced directions around the center of the imaging configuration. They are positioned 15 cm. above the optical table and 30 cm. away from the center of the imaging configuration. Each reflector is directed toward a frosted glass diffuser placed around the cherry sample; the axis of 32 33 each reflector makes a 30 degree downward angle from the horizontal line. The top of the diffuser has a hole so that the cherry sample is visible to the camera. The camera is a WV—CD50 black and white Panasonic CCD camera. The lens is a Tamron SP. The camera is fixed vertically above the cherry sample. The lens is 50 cm above the optical table. Two bandpass filters from Corion are placed over the end of the lens. The first one has the passband centered at 750 nm with bandwidth 40 nm. The other has the passband centered at 500 nm with bandwidth 70 nm. Images are captured and stored as digital images on the computer. The capturing hardware is a Sun Multimedia board on a Sun Sparc 10. The image capturing software is the program SunVideo. The program is run using the Solaris 2.3 operating system and OpenLook window system. The images captured are in Sun’s raster format (.ras) of resolution 480 x 640 rows and columns respectively. The images are black and white with 256 gray levels. The size of the cherries that appear in the images are approximately 400 pixels in diameter. Finally, images are converted into hips format (.hips) so that they can be input to hips [6] software. 3.2 Problems in Image Acquisition There are three major problems in the imaging process. The first problem is focusing on the cherries. The second problem is the misalignment of different images taken from the same cherry orientation. The third problem is the structured noise in infrared images. The problems and the way they are handled are described in detail below. 34 light bulb light bulb and reflector and reflector _ _:E‘_ A 20 cm 30 cm lg; :1 l I I 1 l I ._ 'll'_ _ light bulb and reflector Top View camera _ _ _I[__ _ light bulb 50 cm. light bulb “d '°“°°'°' light bulb and reflector and reflector diffuser \ 15 cm. cherry If Figure 3.1: The imaging configuration 35 3.2.1 Difficulties in Focusing The first problem is focusing on the object. In each posture of a cherry, two images are taken. One image is taken using the infrared filter (750 nm, 40 nm bw). The other is taken using the green filter (500 nm, 70 nm bw). It was hoped that the same focus setting could be used for all imaging. Unfortunately, the focal lengths of the lens in the infrared and green ranges are different. Suppose we are using the infrared filter and adjust the focus of the lens so that the object in the scene is perfectly in focus. If we change the infrared filter to the green filter, the object would be out of focus. (Figure 3.2) If we try to adjust the lens so that the cherry is equally focused (blurred) either when we use the infrared filter or the green filter, the quality of the image is not acceptable. For certain operations using regions, the blurred data might be appropriate. However, use of edges for crack detection would require Sharp images. The other way is to take infrared images of all cherries first and then all the green images. However, that is not possible because if the cherry is moved it is not possible to put the cherry back in the exact same position and orientation. Thus, there is only one choice for our experiment, to adjust the focus setting every time the filter is changed. There is another complication due to focusing. Suppose there are five objects labeled A,B,C,D,E, and F in the scene which are 49.4 cm, 49.7 cm, 50.0 cm, 50.3 cm and 50.6 cm away from the camera respectively. (Figure 3.3) Suppose the lens is perfectly focused on the object C. Objects A and E would be out of focus. Objects B and D are imperfectly focused. A typical cherry is about 2.5 cm in diameter. It 36 «— camera ‘— camera infrared filter reen filter (750 nm) / 500 nm) I I I focal length in l [l the infrared range \ focal length in l l l the green range Figure 3.2: Difficulty in focusing in different wavelengths is not possible to focus on the entire area of one side of the cherry. (Figure 3.4) If the center area of the cherry is in focus, the edge area would be out of focus and vice versa. It is possible, though difficult, to adjust the lens to focus on the area between the center area and the edge. However, since it is needed to adjust the focusing everytime the filter is changed, focusing should be made easy. It was decided that the lens would be focused at the center area of the cherry. In doing so, the edge area would be out of focus. In subsequent chapters on image processing, some of the edge area would be removed and not be used during the image processing and cherry classification. Thus, it is acceptable that the edge areas of the cherries are out of focus. Cherry stems also pose a focusing problem. In some images, cherries are oriented so their stems point toward the camera. In such case, from the way we adjust the focus, we have to accept that the stem would be out of focus. (Figure 3.5) 37 ‘— camera / filter 1-. I I I I 50 cm. I I I Figure 3.3: Focusing multiple objects at different distances camera Figure 3.4: Variability in focus of cherry surface due to field of view 38 Top View Frontal View Figure 3.5: Demonstration of difficulty in focusing stems 3.2.2 Image Misalignment A second problem is the misalignment of images taken from the same cherry orien- tation. In each posture of a cherry, two images are taken using the infrared and the green filters. Each time the filters are swapped and the focusing is adjusted, the camera is unavoidably touched and there is a slight movement of the camera. (Fig- ure 3.6) Two images taken of a cherry in the same orientation are misaligned. The alignment is off by up to 15 pixels. It is difficult to rearrange the camera fixture to be more rigid. It was decided that the misalignment would be corrected by realigning the images manually. 39 _ _____I___- misalignment in the row direction two Images taken from the same cherry in the same orientation ,ll‘ _ misalignment in the In column direction Figure 3.6: Image misalignment 3.2.3 Structured Noise A third problem is the structured noise in images taken using the infrared filter. Figure 3.7 shows the structured noise. It is the picture of a blank background. The structured noise is approximately a ring shape. There are a number of candidates that could create such noise. To be thorough, all components of the imaging configuration are considered, namely the lighting, the diffuser, objects in the scene themselves, the infrared filter, the lens, the camera, the digitizer, cables or some combinations of them. A simple test can be done to identify the source of the structured noise. If the structured noise comes from the lighting or the diffuser or the cherry in the scene, by moving them, the noise should move. However, the noise stayed still. Thus, the noise did not come from those components. The noise stayed still when the infrared filter was moved. Thus, the noise did not come from the infrared filter. After the cable 40 Figure 3.7: Structured noise in image aquisition. The contrast of the image has been increased for visibility was changed, the noise still persisted. Thus, the noise did not come from the cable. Unfortunately, we cannot replace the camera because it was the only black and white camera available. So, we continued to test if the noise came from the digitizer by switching to different digitizer hardware and software. The noise persisted. Thus, it did not come from the digitizer. We concluded that the camera was the source of the noise. By inspecting inside the camera, there is a protective glass permanently mounted in front of the sensor matrix. We thought that this filter might be the source of the noise. Since we used the camera in a narrow band (40 nm), the spectral transmittance of the protective glass in that range might not be perfectly uniform thus resulting in the structured noise. However, there is no such visible structured noise in images taken using the green filter. Since this was the only available black and white camera, it was decided that images taken using the infrared filter be allowed to have such structured noise. The 41 noise would be removed later by software as described next. The noise can be thought of as a function that is applied to images. The idea to remove the noise is to construct the ”inverse” function of the noise. A conceptual model of how the noise is introduced in the image is shown in Figure 3.8 (a). The original image I passes through the noise function f. The noise function f imposes the noise on the image I and the image I ' is generated. We want to find the inverse function f "’1 of f. After f"1 is constructed, we can remove the structured noise by passing the noisy image I’ through f‘1 as shown in Figure 3.8 (b). The resulting image is I , the original image. I >[ f ’1’ original image image wrth . structural norse noise function (a) , I’ _1 I f f I”=I original image - 7 noise function inverse noise function (b) Figure 3.8: Model of the structured noise. (a) The noise function; (b) The inverse of the noise function is used to reverse the structured noise. Assume that f is additive. Its model is shown in Figure 3.9 (a). The structured noise 3 is ”added” to the input image. The model of f ‘1 is shown in Figure 3.9 (b). The structured noise S is ”subtracted” from the input image. Denote the inverse of an image 5' by 5". Another equilvalent model of f"1 is shown in Figure 3.9 (c). In 42 this case, the inverse of the structured noise is ”added” to the input image. Figure 3.9 ((1) illustrates how the inverse function is used to remove the structured noise. The structured noise S has to be determined. Let s be the mean gray level of S. Since we are interested in reconstructing only the AC component of S, we can assume that 3 (DC component of S) is 0. If the DC component of S is not removed from the cherry image, it only increases the gray level of the entire cherry image by s. The procedure to find S is shown in Figure 3.10 (a). An image of a blank white paper is taken (I ’ ) The mean gray level m of the image I ’ is determined. Since the mean gray level of S is 0 by assumption, then the mean gray level of I is also m. Since I is an image of a blank white paper, each pixel should have the same gray level and it should be m. Subtract m from the gray level value of each pixel in I’. The resulting image is S. S can be plugged into f ’1 and the construction of f ‘1 is done. f ‘1 can be used to remove the structured noise as shown in Figure 3.9 (d). The above method assumes that there is no white noise in the imaging process. Figure 3.10 (b) shows the model where white noise is included. Suppose the white noise on each pixel I (r, c) is a random variable IN(1°, c) independently and identically drawn from N (0,02), a normal distribution with mean 0 and variance 02. This model affects S found using the method just described. Figure 3.11 (a) shows that the reconstructed structured noise image is not composed of the structured noise alone but also composed of white noise. If this image is plugged into f "1 and we use f '1, such as it is, to remove the structured noise, the variance of the white noise is doubled in the resulting image, a variance of 02 from the input cherry image and a variance of 02 from the reconstructed structured noise image. See Equation (3.1). 43 -1 1____.g.>.__,1_5 1— —>I+S =I—S Figure 3.9: Details of the structured noise model. (a) inside the noise function; (b) inside the inverse of the noise function; (c) an equivalent form of (b); ((1) how the inverse of the noise function is used. 44 find average ’ gray level 0 )) s - (b) Figure 3.10: Noise inverse function construction. (a) estimating S of the. noise func- tion; (b) a more accurate model of the noise function; 45 f-l I+N’-N = I+2N '1 N’ I? . _ . N I f I+S+N f l I+N .. "n‘ %I+N (b) Figure 3.11: Use of the noise inverse function. (a) when the structured noise is removed, the white noise is doubled. (b) with the new estimation method, the white noise does not increase much. 46 I’=I+S+N1 S’=S+N2 I’—S’=(I+S+N)—(S+N) = I + N1 — N2 Var[I' — S’] = Var[N1] + Var[—N2] (3.1) = 02 + 02 = 202 The structured noise should be removed without increasing white noise too much. The following describes a method to estimate S. This method is based on the previous method. Instead of taking one image of a blank white paper to find one S, n images are taken and the image S is estimated independently n times. Construct the average image S from these 72 versions of S. The white noise in S is reduced 72 times. By keeping n sufficiently large, the white noise in S can be made arbitrary small. Figure 3.12 illustrates the procedure. See Equation (3.2) for computation of the white noise variance. S can be plugged into f“1 and used as before. This time, when f‘1 is applied, the output image has approximately the same amount of white noise as the input image. Figure 3.11 (b) illustrates this result. Note that we can reduce the white noise in the cherry image using the same technique. By taking multiple images of the cherry and finding the average image, the white noise variance in the resulting image would be smaller than the white noise variance in each individual image. 47 F=I+S+N St : %[(S+Nl)+(5+N2)+...+(S+Nn)] = 5+ _‘illv_‘ n I'—S'=(I+S+N)-(3+;—i%fl) = 1+ N — A) n Var[I' — S'] = Var[N] + VarI-Efinl'flll 2 = 02+ 0— n as 02 as it gets large 0 ~ .. ‘ ”II n.. .. S+Nl .. . f 0 S-l-N2 average __... 0)» A 8+5 11 0 ~ -. ' ”III A“ . . S-l-Nn Figure 3.12: Improvement of noise inverse function construction. The noise inverse function is estimated and the effect of white noise is reduced. 48 Nevertheless, the first method (Figure 3.10 (a) and Equation (3.1)) was used in estimating S and removing the structured noise throughout this research. Each side of each cherry was imaged once. No averaging was used. Figure 3.13] (a) shows an image with the structured noise. Figure 3.13 (b) is an enhanced version of Figure 3.13 (a). Figure 3.13 (c) is a gray level trace across the middle of Figure 3.13 (a). Figure 3.14 (a) shows the image after the structured noise is removed. Figure 3.14 (b) and (c) are an enhanced version and gray level trace of Figure 3.14 (a) respectively. Notice the difference of white noise in 3.13 (c) and 3.14 (0). Although the white noise variance increases two times, it is still in the acceptable level, at least visually (Figure 3.13 (a) and 3.14 (a)). (a) (b) (c) Figure 3.13: Image before the structured noise is removed. (a) an image before structured noise is removed; (b) an enhanced version of (a); (c) a gray level trace across the middle of (a). 49 (a) (b) (C) Figure 3.14: Image after the structured noise is removed. (a) an image after the structured noise is removed; (b) an enhanced version of (a); (c) a gray level trace across the middle of (a). Chapter 4 Image Processing for Bruise Detection The result of the feature selection in Chapter 2 suggested that the images taken at two wavelength bands, 750 nm and 500 nm, should be used for bruise detection. Be- cause sample histograms from images did not support this prediction, it was decided that only the infrared images were to be used in all bruise detection procedures in this research. A recent more careful review showed that using images from two wave- length bands as suggested by the result of feature selection may, in fact, be feasible. The review study is described in Section 4.1. In Sections 4.3 to 4.6, four different bruise detection methods are described. Only the infrared images were used in the bruise detection procedures in these sections. In Section 4.2, an image preprocessing procedure is described. The image preprocessing procedure described in Section 4.2 will be used in the first three bruise detection methods (Sections 4.3 to 4.5). The last bruise detection method (Section 4.6) has a separate preprocessing procedure. 50 51 4.1 Review Study of Bruise Detection using 2- band Images The result of the feature selection in Chapter 2 suggested that the images taken at two wavelength bands, 750 nm and 500 nm, should be used for bruise detec- tion/ classification. However, after creating empirical histograms such as in Figure 4.3 (a), the author concluded that the result of feature selection using spectral ra- diometer data could not be generalized to apply to data acquired from the image sensor and filters. It was decided that only the infrared images were to be used in all bruise detection procedures in this research. A more careful review done later showed that using images from two wavelength bands as suggested by the result of feature selection might, in fact, be feasible. Using two images, infrared and green, taken from the same view of the same cherry, the 2-dimensional gray level histogram was constructed. It was hoped that bruised and unbruised surfaces would Show in the 2D histogram as two well-separated clusters as shown in Figure 4.1. If the two clusters are well-separated, it would be easy to determine the decision line separating the two clusters, and the number of pixels on bruised and unbruised surfaces. The cherry could be classified as bruised or unbruised based on these values. The 2D histogram in Figure 4.3 (a) showed that in the empirical data, clusters corresponding to bruised and unbruised surface did not Show up as two well-separated clusters. Note that the cluster on the left corresponds to background pixels and the cluster on the right corresponds to foreground pixels. It is the cluster on the right that would hopefully show up as two separate clusters 52 corresponding to bruised and unbruised surfaces. It is thought that this artifact is mainly due to the fact that the light diffusion properties of the cherry surface in the infrared and the green ranges are different. In the infrared range, the cherry surface is like a Lambertian surface, reflecting light equally in all directions: In the green range, the cherry surface is specular or shiny, reflecting light in one direction more than the other. It was decided that instead of using a multi—spectral image, further study on bruise detection would utilize only the infrared-band image and some spatial analysis. bruised 0 unbruised Reflectance in the green band Reflectance in the infrared band Figure 4.1: Ideal 2D histogram. Two clusters representing bruised and unbruised surfaces are well separated. The line separating the two clusters is the decision line. Later in the research, some other problems were discovered. These problems are the structured noise in the infrared image and misalignment of the infrared and green images. These problems are discussed in Chapter 3. Due to image misalignment, a pixel on one part of the cherry was incorrectly matched with another pixel on another (nearby) part of the cherry. Especially on the edge of the cherry, a foreground pixel 53 (a) (b) Figure 4.2: Example of infrared and green images. (a) infrared image (Lambertian, reflecting light equally in all direction). (b) green image (specular, reflecting light in one direction more than the other). in one image could be incorrectly matched with a background pixel in the other image. Also, on the edge of the bruise, a pixel on a bruise in one image could be incorrectly matched with a pixel on an unbruised surface in the other image. Figure 4.3 (b) shows a 2D histogram that was constructed after the structured noise was removed, the infrared and green images were realigned, the background was removed, and some edge pixels were removed. The histogram still did not show two separate clusters corresponding to bruised and unbruised surfaces. However, a recent more careful review showed that using images from two wave- length bands as suggested by the result of feature selection may, in fact, be feasible. Figure 4.3 (c) shows a 2D histogram constructed from pixels in the upper boxes in Figures 4.2 (a) and (b). The boxes are overlaid on a bruised surface. Figure 4.3 (d) shows a 2D histogram constructed from pixels in the lower boxes in Figures 4.2 (a) and (b). The boxes are overlaid on an unbruised surface. The two clusters in Figures 4.3 (c) and ((1) do not appear to overlap each other much. Thus, bruise detection 54 (a) (b) '1 ’ ‘- «JHIT, ‘ ‘- Aer“; (C) (d) Figure 4.3: Empirical 2D histogram. (a) 2D histogram of images in Figures 4.2 (a) and (b) without preprocessing. The X-axis is the gray level from the infrared image. The Y-axis is the gray level from the green image. The cluster on the left corresponds to the background. The cluster on the right corresponds to the bruised and unbruised surface mixing together. (b) 2D histogram of images in Figures 4.2 (a) and (b) after structured noise cor- rection, image realignment, background removal, and edge pixel removal. (c) 2D histogram of the upper boxes in Figures 4.2 (a) and (b). The boxes are overlaid on a bruise. (d) 2D histogram of the lower boxes in Figures 4.2 (a) and (b). The boxes are overlaid on an unbruised surface. 55 using a multi-spectral image may, in fact, be feasible. Further study in this direction is needed. The remainder of this chapter will treat classification methods that use only a single infrared image. 4.2 Image Preprocessing The problems of misalignment and structured noise and the procedures to correct them were described in the previous chapter. In this section, it is assumed that the green band and the infrared band cherry images have been realigned and the structured noise has been removed. In Figure 4.4 (a) and 4.5 (a), some images prior to preprocessing are shown. There are three things to be considered, namely noise, stem, and edge area. The preprocessing procedure is illustrated in Figure 4.6. Some sample images and their intermediate images are shown in Figures 4.4 and 4.5. First, the noise in the image is reduced by applying a 5 x5 mean filter to the image. (Figure 4.4 (b) and 4.5 (b)). Note that the noise in this case is the white noise. The structured noise had been removed using the procedure in the previous chapter. Next, the background is removed by gray level thresholding. (Figure 4.4 (c) and 4.5 (c)). The grey level of the background is considerably lower than that of non-defective cherry surface. Therefore, there is no difficulty in choosing the threshold value. However, grey level on decay cracks can be lower than the background and thus lower than the threshold value. Mislabeling the defective area as background may lead to erroneous classification. This problem is solved by restoring holes in the image after thresholding. It is done by first finding all (I. ,III. .I, ll .1 . I 56 background connected components. Then for all background connected components except the one containing the image edge (the outermost one), the gray level of each pixel inside the component is restored from the image before thresholding. (Figure 4.4 (d) and 4.5 (d)) Next, the stem is removed from the image or otherwise portion of the stem would interfere with detection. Only the part of the stem that extends beyond the outline of the cherry is removed. This is done by the morphological opening of the image with a disk with the diameter bigger than the width of the stern (but smaller than the cherry). (Figure 4.4 (e) and 4.5 (6)) (d) Figure 4.4: Example of preprocessing. (a) is the image before any preprocessing; (b) is the image after the noise is reduced; (c) is the image after the background is removed; ((1) is the image after the holes are restored; (e) is the image after the stem is removed; (f) is the image after some edge pixels are removed. In Figures (c), (e), and (f), outlines are shown for comparison with previous steps. 57 (d) (0 Figure 4.5: Another example of preprocessing The morphological opening of an original image I I with a structuring element S can be described below. A structuring element is a binary image whose foreground area is usually considerably smaller than the foreground area of the original image. Opening provides the mean by which given subshapes of a complex shape can be selected. Opening an image with a disk structuring element smooths the contour, breaks narrow isthmuses, and eliminates small islands and sharp peaks or capes. Fig- ure 4.7 illustrates how morphological opening can be used to remove/ retain objects. The figure shows a shape that consists of a disk and a rod. By opening the image with a disk structuring element with a diameter larger than the width of the rod but smaller than the diameter of the disk, the rod is removed and the disk is retained. A more formal description of opening is described below. (Figure 4.7) A structuring element S is overlayed on the original image at some position P within the original 58 infrared image (750 nm, 40 nm bw) Rnggng Noise in the image is reduced using a 5x5 mean filter. R 1 Gray level of background pixels are significantly lower B 93”" "9d than that of foreground (cherry) pixels. Background ac 9'0"" pixels can be removed by thresholding. I Restorlng - . holes Holes in the cheny image are restored Removing The cherry stem is removed by stem morphological opening. Removln Edge pixels have lower gray level than pixels in the middle. Ed 6 ”(9'98 These pixels can cause false classification. They are 9 p removed by morphological erosion. preprocessed infrared image ® Preprocessing Figure 4.6: Diagram of a preprocessing procedure 59 image I. S is said to be properly overlayed on I with respect to P if S is positioned at P in I and for each foreground pixel of S, the corresponding pixel of I is also a foreground pixel. A pixel :c of I is coverable with respect to S and P if S is properly overlayed on I with respect to P and the corresponding pixel of x in S is also a. fore- ground pixel. A pixel a: of I is coverable with respect to S if there exists a positioning P of S within I such that S is proplerly overlayed on I with respect to P and :r: is coverable with respect to S and P. The morphological opening of an original image I with a structuring element S is the image that retains only foreground pixels of I which are coverable with respect to S ; The foreground pixels of I which are not coverable with respect to S are removed. (f) Figure 4.7: Example of morphological opening. (a) is the original image; (b) is the structuring element; (c) shows a proper overlaying of the structuring element; The light gray area in (c) are pixels that are coverable with respect to the structuring element and its position; ((1) shows an improper overlaying of the structuring element; The gray area in (e) are pixels that are coverable with respect to the structuring element. (f) is the morphological opening of (a) 60 In the next step, some edge pixels of the cherry will be removed. The gray level on the edge area in the cherry image is lower than the center area due to the curvature of the cherry. Most of our classification procedure is sensitive to the average gray level. These edge pixels have to be removed or otherwise they may lead to misclassification. The edge pixels are removed by morphological erosion. (Figure 4.4 (f) and 4.5 (f)) That is the last step in the image preprocessing. To summarize, beginning with a non-preprocessed image, the preprocessed image is obtained by reducing the noise, and removing the background, the stem, and edge pixels. 61 4.3 Bruise Detection by the Global Histogram Method In this section, we describe a single classification method that was tested for bruise detection. The method considers the gray level histogram of the entire image. It is assumed that the gray levels of good area and bruised area are significantly different. We assume further that the gray levels of good area and bruised area are nomally distributed. If the cherry has no bruise, the gray level histogram should appear as unimodal. Otherwise, the histogram should appear as bimodal. Figure 4.8 shows unimodal and bimodal histograms. (b) (d) Figure 4.8: Unimodal and bimodal Gaussian distributions. (a) two overlapping Gaus- sian distributions; (b) sum of distributions in (a), unimodal distribution; (c) two over- lapping Gaussian distributions; (d) sum of distributions in (a), bimodal distribution; of Q? 62 Figure 4.9 shows the classification procedure. First, the image is preprocessed using the procedure described in the previous section. The histogram of the gray levels of the eroded image is then constructed. The histogram is fitted by two Gaussian distributions. The fitness criteria is given in Equation (4.1). E2(P1a#13019P2a#2a02) = Z(H(Z) — I“{(iaplaula0'19p2a/4‘2a0’2))2 Where 3 H (z) = number of pixels having gray level 2', 0 g i S 255 A H(i3pla#19017p2t#2302) = P1N(#1,0’1)+P2N(#230’2) 1 _I I-u2 4.1 N(p,a) = 27rae 2 a ( ) PI fits H best if E2 is the minimum The bimodality is tested on PI. If it exhibits bimodality, the image is classified as arising from a bruised cherry. Otherwise, the image is classified as from a good cherry. Note that at this point we are not considering the non imaged surface of the cherry. The classification procedure was run twice with the size of the morphological erosion element changing from 16x16 to 50x50. Both classification variations were done on 25 good cherry images and 60 bruised cherry images. Note that these sample cherries are different samples from what we used in the spectral experiment. The sample cherries in the spectral experiment had rotted. The result of the experiment is shown in Table 4.1. errg is the error rate of classfying good cherries alone. errb is the error rate of classifying bruised cherries alone. err5 is the error rate of classifying infrared image (750 nm, 40 nm bw) Preprocessing preprocessed infrared image 63 cgfigttggéflg‘n Gray level histogram of the image is constructed. histogram The histogram is fitted by two Gaussian distributions. D'sfilififitglon Each Gaussian distribution has three parameters namely, its mean, variance, and size. ,I I I Gaussian distribution I I I parameters I ( I I, The two distributions combined is tested if it has Bim‘lgedsatllty one mode or two modes. If the distribution has two modes, the cherry is classified as defective. decision (good/bruised) Figure 4.9: Diagram of the global histogram classification method ff. .2 I}? 64 both good and bruised cherries assuming that the prior probability of bruised cherries is 5%. err5o is similar to err5 with prior probability 50% bruised cherries. The mathematical definition of errg,errb,err5, and err5o is shown in Equations (4.2) to (4.5). Table 4.1 shows the results of applying these definitions in the experiment. errg = P(Ii( is bimodal [II is from a good cherry) (4.2) errb = P(II is unimodal III is from a bruised cherry) (4.3) err5 = 0.05 errb + 0.95 errg (4.4) errso = 0.50 err;> + 0.50 errg (4.5) size of the misclassification rate erosion box errg err), err5 errso 16x16 68% 45% 67% 57% 50x50 8% 43% 9.8% 26% Table 4.1: Error rate of the global histogram method In variation 1, most good cherries are classified as bruised. The error can be explained as follows. The edge of the image is eroded by only 16 pixels. Figure 4.10 shows the trace of gray level across the middle of a cherry image. Since the gray level fades off near the edge due to the surface curvature of cherry, there is a low value component in the histogram. The histogram can be fitted with two Gaussian distributions and it appears bimodal. Figure 4.11 (b),(c) Shows the histogram of a good cherry that is misclassified. 65 (a) (b) Figure 4.10: Fading gray level near the edge (a) A bruised cherry; (b) The gray level trace across the middle of image (a). (a) (b) (C) (d) (C) Figure 4.11: Effect of amount of edge pixels removed and performance. This figure illustrates how a good cherry is misclassified. (a) a good cherry image; (b) the his- togram of the cherry image that is eroded by 16x16 box; (c) two fitting Gaussian distributions of the histogram in (b); (d) the histogram of the cherry image that is eroded by 50x50 box; (e) two fitting Gaussian distributions of the histogram in (d). The two distributions overlapped and appear as a single mode distribution. bi 66 Even though the good cherry that is eroded by a 16x16 box does not exhibit bimodality boldly, it is labelled as bimodal. We have used the unimodal—bimodality test of fitting Gaussian distributions in the rather strict sense. II(2') is bimodal if there are i1 and i2 such that 1. £1 < i; and 2. PI(2'1) and [17(2'2) are local maxima We can tighten the criteria for the bimodality test. So the criteria might be Given a constant C, II(i) is bimodal if there are 2'1, 2'2, and i3 such that 1. 2'1 < 2'2 < i3 2. II(i1) and PI(2'3) are local maxima 3. II(2'2) is a local minimum and 4. min( H(i1),H(i3)) - an.) 2 c Figure 4.12 visualizes parameters of the test. H(11) "(12) H(I3) Figure 4.12: Visualization of new bimodality testing criteria Inst \aII ( '10 C) O pix 67 The experiment was not performed with the bimodality in the relaxed sense. Instead, it was decided to remove more edge pixels from the image. This led to variation 2, where images are eroded by a 50x50 box. Error rates decrease for both good cherries and bruised cherries, especially good cherries. Figure 4.11 (d),(e) illustrate a good cherry correctly classified when the edge area pixels are removed by a 50x50 box. Good cherries are classified accurately when their fading edges are removed. However, many bruised cherries are still misclassfied. This is because if the bruise is small, its component in the histogram is also small and thus hidden within the histogram. Only when the bruise is large, will the histogram appear as bimodal. Figure 4.13 shows the gray level histogram of cherry images with large and small bruises. (C) (01) Figure 4.13: Illustration of global histogram method failure. (a) cherry image with large bruise; (b) gray level histogram of (a); (c) cherry image with small bruise; (d) gray level histogram of (c) 68 The results suggested use of smaller areas to construct the histogram so we may detect the bimodality in some smaller area even though the bruise is small. This led us to test the local histogram method. 69 4.4 Bruise Detection by the Local Histogram Method In this section, we describe the cherry classification method based on gray level his- tograms of subimages. The key idea of the local histogram method is to divide the image into subimages such that the size of the bruise area in the subimage is compa- rable to the size of the good area and thus the gray level histogram constructed from the subimage exhibits bimodality. (Figure 4.14). Some windows may totally lie over good area, some may totally lie over bruise area, and some may lie on the boundary between good and bruise areas. The gray level histogram of the last case is expected to exhibit bimodality. Figures 4.15 and 4.16 shows the classification procedure. Figure 4.14: Image segmentation in the local histogram method. (a) image segmented into windows of size 80x80 pixels; (b) image segmented into windows of size 40x40 pixels Four classification variations were tested based on the gray level histogram on subimages. They can be described as follows. The image is preprocessed using the procedure described in Section 4.1. The image is divided into square windows. The gray level histogram of each window is computed. Each histogram is fitted by two 70 infrared image (9 (750 nm, 40 nm bw) Preprocessing preprocessed infrared image W Ima The ima ' d’ 'ded ' ' ' ge is m mto a matrix of subimages. ' Dlvlsfgn Each subimage has size WxW. IIIIIQII .l'fifijfftii‘. . . IIRE?“3I matrix of subimages Illt C 4‘. Illa--1.- III-III. Histogram Gray level histogram of each subimage Construction is constructed. ] matrix of histograms D'sgllfigtg'on Each histogram is fitted by two Gaussian distributions matrix of Gaussian distribution parameters _________________ Each pair of Gaussian distributions is tested if Blmfdat'w it is bimodal. If the distribution is bimodal, the as cherry is classified as defective. O—l matrix of bimodality ' test result E Counting and Decision Making decision (good/bruised) Figure 4.15: Diagram of the local histogram classification method Ir- a-“ '-l 71 binary image or O—l matrix Count the number of 1’s in the matrix or Counting count the number of white pixels in the binary image. it The value "n" is compared against a predetermined Thresholding "threshold" value. If "n" is greater than the threshold, the cherry is classified as defective. decision (good/bruised) Counting and Decision Making Figure 4.16: Diagram of the last step of the classification 72 Gaussian distributions. Each window is labeled as bimodal or not according to its fitting distribution. The image is classified as bruise tissue or not based on the number of windows that are labeled bimodal. Different variations of the size of the erosion element and the size of the window were tried. The parameters of each experiment are given in Table 4.2. variation erosion element window size 1 16x16 80x80 2 50x50 80x80 3 16x16 40x40 4 50x50 40x40 Table 4.2: List of experiment parameters of the local histogram method The classification was performed based on the number of bimodal windows in the cherry images. The threshold was selected so that good cherries are misclassified at the 5% level. Definition of t5, errg, and err)> are shown in the Equations (4.7)-(4.9). The results of the classification of 85 cherries, 60 bruised cherries and 25 good cherries are shown in Table 4.3. w = number of windows in the image that exhibit bimodality (4.6) t5 = t such that p(w Z t | w is for a good cherry) = .05 (4.7) errg = P(w 2 t5 I w is of a good cherry) = .05 (4.8) errb = P(w < t5 | w is of a bruised cherry) (4.9) The image was segmented into windows with the hope that the gray level his- 73 misclassification rate variation errg err)J err5 err5o 1 5% 86% 9.1% 46% 2 5% 70% 8.2% 38% 3 5% 70% 8.2% 38% 4 5% 35% 6.5% 20% Table 4.3: Error rate of the local histogram method togram for a small bruise in that image will Show up more prominently in some windows. This was so. However, there was undesired bimodality showing up in windows that lie near the boundary of the cherry image. In cases 1 and 2, the window size was 80x80. Images were divided into 6x8 windows. Looking at the cherry alone, it fits in approximately a 4x4 window. In case 1, almost all windows that lie on the cherry image are bimodal. This was because near the cherry image boundary the gray level faded off. And it is these windows that dominantly contribute to the number of bimodal windows, not the bimodality that comes from windows lying on bruise and good surface. In case 2, more boundary pixels are removed, and hence there are fewer windows lying on the image boundary labelled as bimodal. Even though there are fewer such windows picked up, some still exist and moreover, they vary in total number. The error rate of case 2 is less than case 1 but is still very high. In case 3, where cherry images are eroded by a 16x16 box, almost all windows lying on the cherry boundary are labelled bimodal. In case 4, where a 50x50 erosion box was used, fewer windows are labelled bimodal. Thus, sufficient boundary pixels must be removed, otherwise bimodality will be exhibited. In cases 2 and 4, fading boundary pixels are removed, but there is still a number of 74 windows lying on the boundary that are labelled bimodal. Sometimes, on a window lying on the boundary, the number of cherry pixels is very small. The histogram in such cases is badly shaped and exhibits bimodality. In case 2, the size of the cherry can fit within about a 4x4 window. In case 4, it fits in about an 8x8 window. The ratio of internal windows to boundary windows in case 4 is much higher than that in case 2. Thus, in case 4 the effect of boundary windows is less than that in case 2. The accurary of the classification increases when fading boundary pixels are re- moved and window size is not too big. The experiment shows that when images are eroded by a 50x50 box and the window size of 40x40 is used, the classification error rate of bruised cherries is 35%. If either the erosion box is a 16x16 box or the window size is 80x80, the error rate of bruised cherries classification is 70% or more. 75 4.5 Bruise Detection by Thresholding In this section, we present a classification method based on thresholding the gray level of cherry image pixels. It can be described as follows. Each image is preprocessed using the procedure described in Section 4.1. The image is then thresholded by some specified value. This value is chosen so that it is of greater gray level than most bruise pixels and lower than the gray level of most good tissue pixels. After thresholding, we count the number of pixels whose gray level is lower than the threshold. We also count the number of pixels comprising the entire cherry. The ratio of of the number of low gray level value pixels to the number of total pixels is calculated. A high ratio means that there is a big fraction of low gray level pixels which are suspected to be of bruised surface. The ratio is compared against another specified threshold value. If the ratio is less than the threshold, we label the cherry as good, otherwise as bruised. Figure 4.17 shows the classification procedure. We are now in the position to find the values of two kinds of threshold, gray level threshold gt and threshold rt of the ratio of the number of bad tissue pixels to the total number of pixels. At first, manual segmentation was done on 20 bruised cherry images dividing good surface and bruise. Then the statistics of the gray level of each surface type was computed. The statistics are shown in Table 4.4. We assumed that the gray level of each surface type is normally distributed. From our data, on a bruised cherry image, the size of the bruise is, on the average, 85 = 35% of the size of the whole cherry image. Another value that we use is the percentage 76 infrared image (750 nm, 40 nm bw) Preprocessing preprocessed infrared image V The image is thresholded by a predetermined ThrGSIlO'dinQ threshold value. 11 l 12 binary gray level image COMPUIII‘IQ n1 = number of foreground pixels in 11. Bad Surface n2 = number of foreground pixels in 12. Ratio 1‘ = n2/n1 r, bruised surface ratio The value "r" is compared against a predetermined Thresholding "threshold': value, If "r" is greater than the threshold, the cherry IS classrfied as defective. decision (good/bruised) Figure 4.17: Diagram of the thresholding classification method surface type mean s.d. good area 163.57 16.11 bruise area 198.27 11.07 Table 4.4: Gray level statistics of unbruised and bruised surfaces 77 of bruised cherries, cb. Note the difference here, in the first case, it is the percentage of bruise surface on a bruised cherry, while in the second case, it is the number of bruised cherries per 100 cherries. Since the percentage of bruised cherries is likely to vary a great deal over different real world conditions, we decided to use two extreme values, c5 = 5% and 050 = 50%. Using these values we are able to find gt. gt = t’ such that (c, - s, + (1 — c.) . 0) -p(t | 55,0.) = (4-10) (Cb°gb+ (1 - Cb)’1)°P(t | “9309) 9,5 = 165 if e, = c5 = 5% (4.11) 9t,50 = 175 if Cb = C50 = 50% (4.12) Now images can be thresholded and the number of pixels whose gray level are lower than the threshold can be counted and the ratio can be computed. Next is to specify the ratio threshold. This threshold affects the error rate in such a way that if the error rate of classifying good cherries is low, the error of classifying bruised cherries is high, and vice versa. We specify another constraint: we want good cherries to be misclassified at only the 5% level. Now the threshold r, can be computed. 78 number of bad pixels 7' = number of cherry pixels r, = t' such that p(r > t' I good) = 5% 735 = 0.016 73,50 = 0.038 lbe=C5=5% if C), = C50 2 50% (4.13) (4.14) (4.15) (4.16) The results of the classification of 85 cherries, 60 bruised cherries and 25 good cherries are shown in Table 4.5. bruise prior misclassification rate probability errg err)> err5 errso 5% 5% 25% 6% - 50% 5% 15% - 10% Table 4.5: Error rate of the thresholding method 79 4.6 Bruise Detection via Infrared Edge Detection In histogram-based methods, detection of bruise tissue is based on histogram bi- modality of an individual window. This implies that ideally the window is placed on the boundary between good and bruised areas. This suggests the idea of implement- ing edge detection to detect the transitive area between good and bruised tissues. Detection of differences in adjacent areas should be less influenced by variations in brightness of individual cherries and less susceptible to being drowned out by the population variation recorded in a histogram. Bruised and good surface are assumed to have different gray levels. Thus the boundary area between them is expected to have a change or non-zero gray level gradient. This can be detected by gradient detection methods. Presented here is a gradient detection procedure and methods of finding the parameters required by the procedure. A gradient, or contrast, method has the advantage of being able to adjust to normal reflectance variations by looking for two contrasting regions regardless of gray level. 4.6.1 Edge Detector The image is divided into square windows of some size W x W. The mean gray level of all pairs of adjacent windows are compared. If the difference of the means are greater than a specific value, we say that there is a boundary between the two windows. Figure 4.18 and 4.19 shows the steps of the procedure. This method is essentially Robert’s edge detection. The above method is equiv- 80 infrared image (750 nm, 40 nm bw) © A procedure to detect edges Ed9° “teeth" is applied to the image. edge magnitude image Thresholding The image is thresholded by a predetermined value. binary edge image Countin and Decision aklng decision (good/bruised) Figure 4.18: Diagram of infrared edge detection procedure 81 [ infrared image (750 nm, 40 nm bw) Two images are generated. The first image is generated by Image dividing the images into square windows of size WxW. The Dupllcatlon second image is generated in the same way except that windows are shifted to the right by W/2 and downward by W/2. fl ] eee I | / ’1’, \\ \“ I” \ | I f—'———.I —i I I I - - | I _ _ _ _ I _ _ _ __ ROSOIUtlon Resolution The resolution of the image is reduced by replacing RGdUCIION Reduction each window by a single pixel. The gray level of the , , pixel is set to the mean gray level of all pixels in the [ 1 window. Edge Edge Robert’s edge detection is applied to both images. Detection Detection The gray level of each pixel is set to its edge magnitude. - . Each pixel in both images is enlarged into a square window of size WxW. The gray level in the window is uniform and equal to the Image gray level of the corresponding pixel. The gray level is generated: Combination - - - the gray level of each pixel is the maximum between the gray level of the corresponding pixels from the two enlarged images. (“—T—_—‘I fl .0. I F I ... I I’- I edge magitude Image I @ Edge detection Figure 4.19: Detailed diagram of edge detection in infrared edge detection procedure 82 alent to the following method. First, divide the image into square windows of some size. Next, reduce the image, ie., for each window, place in the reduced image a pixel with gray value equal to the mean gray value of pixels in that window. If the window size is 10x10 pixels, the reduced image is 10 times smaller than the original images. The next step is to apply Robert’s edge detector (2x2) to the reduced image. The resulting edge image is then thresholded with some specific value. Images have to be reduced because there is noise in the images, if the image is not reduced or reduction factor is too small, the response of the true edge will be small compared to a noise response. Also, we want a bruise to be at least a good fraction of W X W pixels. To get the highest response from the Robert’s edge detector, two adjacent windows must be placed on the images such that one window is entirely on the bruise and another is entirely on the good area. In the worst case, a window is placed on the center of the boundary of the bruise and good area where the response to the edge detector is lowest. To overcome this situation, we use overlapping windows. Two images are formed from the original image. The first image is divided into square windows of size W x W (Figure 4.20 (b)). The second image is also divided into square windows of size W x W but the windows are placed such that they are shifted to the right by W/Z and downward by W/2 (Figure 4.20 (c)). The two images are then processed separately, ie., for each image, it is reduced by a factor equal to the window size and then Robert’s edge detector is applied. Then, the results of the two images are combined. Each pixel is enlarged in each edge image back into a square window of size W x W (Figure 4.20 (h),(i)). The gray level in the window is uniform and equal to the gray (j) (k) Figure 4.20: Intermediate results of infrared edge detection method. (a) The original image; (b) The first image from overlapping window technique; (c) The second image from overlapping window technique; ((1) Reduced image of (b); (6) Reduced image of (c); (f) Robert’s edge detection output of (d); (g) Robert’s edge detection output of (e); (h) Enlarged image of (f); (i) Enlarged image of (g); (j) Combined image of (h) and (i); (k) Thresholded image of (j). 84 level of the corresponding pixel. Windows from each image still overlap each other. Each window is then divided into 4 sub—windows of size W/ 2 X W/ 2. The results of the edge detection is then combined by creating another image composed of windows of size W/ 2 x W/ 2. The gray value of each window is the maximum of the two gray levels of corresponding windows of the two edge images (Figure 4.20 (j)). The rest of the operation is the same as in the non-overlap method, i.e., thresholding the image with some specific value (Figure 4.20 (k)). This allows us to count the number of cells where high contrast was detected by the Robert’s operator. Perhaps hexagonal windows would be better, but for computational convenience, we made them square. To further classify the image as bruised or good, edge pixels are removed by erosion in order to remove small regions. The number of remaining edge pixels are counted and if the number of edge pixels is greater than some specific value, the image is considered that of a bruised cherry. There are three parameters values which need to be specified 1. window size 2. edge value threshold 3. number of edge pixel threshold Item 3 can be adjusted according to the desired misclassification rate of either good or bruised classes. The parameters in item 1 and 2 depend on the model of the cherry. In the next section a model of the cherry and the method of finding the parameters is described and the result of the classification is presented. 85 4.6.2 Finding Edge Detection Parameters The gray level of the cross section of the boundary area between bruise and good area. was modeled as a linear ramp edge. A gray level of the cross section across a bruised cherry is shown in Figure 4.21 (b). The gray level model of the boundary area is shown in Figure 4.21 (c). The length and depth of the ramp are assumed to vary over a wide range but the slope of the ramp is assumed to vary only slightly. (a) (b) (c) Figure 4.21: Ramp model of gray level cross section on a bruise. (a) a bruised cherry; (b) gray level across the middle of the image in (a); (c) model of (b). The appropriate size of the window must be determined. If the window size is too small, the response of the edge detector to noise is greater than to the ramp. This can be illustrated in Figure 4.22 where we use different window sizes. Using a large window size will give greater response to the ramp rather than noise. However, there is a drawback, if the window is greater than the length of the ramp, the ramp is smoothed out and it may produce too low of a response and we will not detect the edge. The upper bound width of the window should not be too much greater than the length of the ramp. We have selected the window width of 16 pixels. Note that the narrowest ramp we found in our samples is 10 pixels wide and the widest is 60 pixels wide. 86 (d) (e) Figure 4.22: Relationship of window size and noise. (a) a bruised cherry; (b)—(f) Robert’s edge detection on reduced images with window size 2x2, 4x4, 8x8, 16x16, 32x32 respectively. For display, the gray level in images (b) to (f) are increased by the factor of 25, 25, 20, 15, and 10 respectively. Note that noise is not removed from these images before Robert’s operator is applied. Next is to find the threshold of the response from the edge detector. The slopes of a number of ramps were measured. The slopes measured had values between 0.8 to 1.3 gray levels/pixel. Now the response threshold can be computed. It is necessary to detect the least steep ramp, so the slope value of 0.8 is used. The detector has two masks, each mask responds differently according to the direction of the boundary. The total response is the sum of absolute value of the response of each mask. The response varies from \/2 to 2. We use the least value («2) to compute the threshold. Thus, the response of the ramp to the detector is the product of the distance between 87 the centers of adjacent windows, the slope of the ramp, and the weight value \/2. least response = x/2 16 - 0.8 = 18.1 (4.17) threshold < least response threshold = 16 (4.18) The parameters to perform the detection have now been established. The detec- tion can be applied to cherry images and the number of boundary pixels is determined. To further do the classification we set the threshold of the number of boundary pixels so that only 5% of good cherries are misclassified. The result of the classification of 85 cherries, 60 bruised cherries and 25 good cherries is shown in Table 4.6. misclassification rate method errg errb errs err5o 16x16 5% 15% 5.5% 10% Table 4.6: Error rate of the infrared edge detection method 4.7 Summary of Bruise Detection Methods The error rates of each classification procedure described in this chapter are sum- marized in Table 4.7. Note that most entries in column errg have value 5% because the parameters of the classifiers are adjusted so that 5% of non-defective cherries are misclassified. The global histogram method has no parameters. Thus its misclassi- fication rate for non-defective cherries is not necessarily 5%. The global histogram 88 method and the local histogram method did not perform well. On the contrary, the thresholding method and the edge detection method performed well. The threshold- ing method misclassified 25% of bruised cherries given that the a priori probability of bruised cherries is 5%. This gives a weighted misclassification rate of 6%. The thresholding method misclassified 15% of bruised cherries if the a priori probability of a bruised cherry is 50%. This gives a weighted misclassification rate of 10%. The edge detection method misclassified 15% of bruised cherries independently of the a priori probability of bruised cherries. The weighted misclassification rates are 5.5% and 10% if the a priori probability of bruised cherries are 5% and 50% respectively. Although both the thresholding method and the edge detection method performed equally well on our test samples, detection of differences in adjacent areas should be less influenced by variations in brightness of individual cherries. Thus, the edge detection method should perform better in a larger class of samples. In terms of computation, the thresholding method needs least computation. The boundary detection method needs more computational power than the thresholding method. The global histogram method needs more computational power than the first two above. This is because it has to perform curve fitting optimization. This optimization is iterative and is a floating-point operation. The local histogram method also performs the same optimization and the optimization is done on each subimage. Thus, it needs more computational power than the global histogram method. 89 method misclassification rate errg I earn, errs I err5o Global l6xl6 68% 45% 67% 57% Global 50x50 8% 43% 9.8% 26% Local 16X16,80X80 5% 86% 9.1% 46% Local 50X50,80X80 5% 70% 8.2% 38% Local 16x16,40x40 5% 70% 8.2% 38% Local 50x50,40x40 5% 35% 6.5% 20% Thresholding 5% 5% 25% 6% - Thresholding 50% 5% 15% - 10% Edge Detection 5% 15% 5.5% 10% Table 4.7: Summary of error rates of various bruise detection methods Chapter 5 Crack Detection In this chapter, we describe a procedure for classifying cherries as defective or non- defective due to cracking using edge detection on single view green images. In partic- ular, this procedure tries to classify a cherry as to whether it has a dry crack or not. In this chapter, a defective surface means a dry crack surface and a non-defective surface means a non-dry-crack surface. The input to the procedure is a pair of infrared and green images of a cherry taken from the same viewpoint. The infrared image is used only for assisting the removal of background, stem, and edge pixels. The classification is based solely on the green image. In the first section, we describe how the infrared image is used to remove undesired regions in the green image. Also, we describe a procedure to remove another kind of undesired region, ”dark spots”. In the second section, the procedure for classifying dry crack cherries is described. 90 91 5.1 Dark Spot Mask Preparation The infrared image is used for assisting the removal of background, stem, and edge pixels. By applying the preprocessing procedure (Section 4.1) to the infrared image, we obtain an infrared image with the background, stem, and edge pixels removed. This image is converted into a binary image. Call it M1. Figure 5.1 illustrates the procedure. The white portion of the image is the portion of green image to be L retained. The black portion of the image is the portion of green image to be removed. infrared image (750 nm, 40 nm bw) @ The infrared image is preprocessed by Preprocessing the procedure discussed in Chapter 4. The gray image is converted into a binary image. Binarlzatlorl The pixel in the binary image is 1 if the corresponding pixel in the gray image is not a background pixel. binary image Figure 5.1: Edge and stem occlusion mask generation Since the imaging configuration is not perfect, some regions in the green image are very dark. (Figure 5.2) These dark spots lead to erroneous classification. Pre- liminary tests showed that removal of these dark spots is not difficult and can be done accurately. The procedure is based on gray level thresholding. The threshold is 92 determined for each individual image as follows. Reduce the green image resolution by a factor of 2. Remove the background, and stem using a procedure similar to a procedure described earlier except that the edge pixels are eroded by a. 10-pixel diameter disk. Construct the gray level histogram of the image with bin width 2. (Figure 5.3) A preliminary study showed that the gray level corresponding to the left shoulder of the main peak is a good threshold for separating dark spots from others. Start from leftmost bin, find the first bin which has a high value (800). From that bin, go back to the left until the value of adjacent bins is small (12), then stop. The gray level corresponding to the bin found will be used as the gray level threshold. (Figure 5.3) (a) (b) Figure 5.2: Examples of darks spots in green images To create the mask, beginning with the original green image, clean the green image with a mean filter of size 3x3. Then threshold the image using the threshold value found earlier. Fill small holes. Figure 5.4 illustrates this. Next, remove small components by mophological opening of the image. Then expand the hole slightly by eroding the image. Call the resulting image M2. 93 (a) I H (b) Figure 5.3: Gray level histograms of green images 5.2 Classification Procedure In this section, the precedure to classify the green image as defective or non-defective is described. Since the average gray level of the region of a dry crack may be higher or lower than a good surface, gray level thresholding may not give good separation between dry cracks and good surfaces. However, dry cracks have more texture than does good surface. In other words, there is more gray level variation on dry cracks than on good surface. Since edge detection is a simple method to detect gray level variation, using edge detection to detect a dry crack region is feasible. There are five major steps in the classification procedure, namely preprocessing, edge detection, masking, enhancing, and decision making. Figure 5.5 illustrates the procedure. In the preprocessing step, the green image is reduced by a factor of 2 and cleaned by a 3 x 3 mean filter. (Figure 5.6) Call the resulting image [1. In the edge detection step, Robert’s edge detection is applied to image 11. The resulting edge magnitude image is then thresholded. Figure 5.7 illustrates this. Call the resulting image [2. 94 green image (500 nm, 70 nm bw) Reduce The resolution of the image is reduced resolution by a factor of 2. Remove Noise is reduced by a 3x3 mean filter. Noise threshold value! Threshold The image is thresholded using the threshold previously determined. Fl“ Background connected components are removed small holes if the number of pixels in the component is small. 2333:3083?“ Small components are removed by opening 5:23;" Holes are expanded by dilation. dark spot mask Figure 5.4: Dark spot mask generation I 0 green image © (500 nm, 70 nm bw) Preprocessing I l Robert’s edge detection is applied to the image “9" 09mm“ and the image is thresholded. I 2 Mi Masking Undesired regions are removed. M 2 I 3 Foreground pixels in nondefective area are removed. Enhancing Foreground pixels in defective area are connected. I 4 ' Counting and Decision Making 95 decision (defective/nondefective) Figure 5.5: Diagram of the green edge detection classification method 96 I 0 green image - I The resolution of the image is reduced mags by a factor of 2. Reductlon The noise in the image is reduced using ”0‘39 a 3x3 mean filter. Cleaning 1] preprocessed green image - © Green Image Preprocessing Figure 5.6: Preprocessing step I1 Edge Detection 97 preprocessed green image Robert’s edge detection is applied to the image. edge magitude image Thresholding l 2 The edge magnitude image is thresholded by a prediterrnined value. binary edge image CF) Edge detection Figure 5.7: Edge detection step 98 In the masking step, undesired regions are removed from the image I; using the mask M1 and M2 prepared earlier. Recall that M1 removes the background, stem, and edge pixels from the image and M2 removes dark spots from the green image. (Figure 5.8) Call the resulting image 13. M] . "AND" M ____—’ "AND" ® Masking Figure 5.8: Masking step At this point, one possible way to classify the cherry as defective or not is to count the number of foreground pixels in image 13; call this number n. Then compare n against some threshold t and classify the cherry as defective if n > t, and non-defective otherwise. Figure 5.9 illustrates the procedure. Preprocessing ® Edge Detection I2 Masking Y Counting and Decision Making 99 green image (500 nm. 70 nm bw) Robert’s edge detection is applied to the image and the image is thresholded. Undesired regions are removed. decision (defective/nondefective) Figure 5.9: Alternate classification procedure 100 The detection is not perfect, however. Some pixels on non-defective surfaces are detected during the edge detection step and many pixels on defective surfaces are left undetected. If somehow pixels in the good area can be removed and pixels in the defective area can be connected, then it of a good cherry would be smaller and n of a defective cherry would be larger. The classification would be more accurate. The procedure in the enhancing step is a reliable procedure to do this. The operation in the enhancing step is based on the observation that non-defective regions give sparse responses to the edge detector while defective surfaces give denser response. We need a way to measure such density. Figure 5.10 illustrates the pro— cedure. Beginning with the image 13, contruct image I; as follows. For each white pixel in image [3, count the number of foreground pixels in the neighborhood; the value of the corresponding pixel in image If, is set to this number. If; is obtained from If, by thresholding. Then 15” is obtained by removing small connected components from Ié’. Effectively, pixels with a small number of neighbors (pixels on non-defective region) are removed. Finally, pixels remaining are connected by mophologically clos- ing the image [5”. Call the image I4. In the final step, decision making, the number of white pixels in image 1., are counted and compared against the threshold. If the number is greater than the threshold, the cherry is classified as defective. Otherwise, non-defective. The procedure was applied to 50 non-defective images and 14 dry crack images. One non-defective image and three dry crack images were misclassified. This is equiv- alent to 2% and 21% misclassification rate. The classification results are shown in Table 5.1. 101 13 binary edge image For each foreground pixel, count the number of foreground pixels Neighbor in the 31x31 square neighborhood. The value of each pixel in Counting the output image is set to this number. Note that for background pixels, this number is set to O. Thresholding Threshold the image with a predetermined value. ! Removing Small Connected components with size less than 8 pixels removed. Components Morphological Foreground pixels are connected using a morphological closing Closing operation. I 4 enhanced edge image ® Enhancing Figure 5.10: Enhancing step 102 (3er = misclassification rate of non-defective cherries errc = misclassification rate of cracked cherries errs = 0.05 errc + 0.95 errg errso = 0.50 errc + 0.50 errg misclassification rate errg errc err5 errso 2% 21% 3.0% 12% Table 5.1: Error rate of the green edge detection method fiT .— ' Chapter 6 Integration In the previous two chapters, classification procedures on single view single band images have been described. In this chapter, we describe a classification procedure that classifies cherries as defective (bruised or cracked) or non-defective using multiple images. 12 images were taken from each cherry. Images are taken from six different directions of the cherry. By thinking of the cherry as framed within a cube, the six directions of the camera correspond to the six directions perpendicular to six different faces of a cube. In each direction, two images are taken, one using the infrared filter and the other using the green filter. Only 10 images of each cherry are used for classification. Two images, infrared and green, where the stem overlaps the cherry are dropped manually. They are dropped because currently, overlaping stems cannot be isolated from the cherry. Keeping the images with such stems would lead to misclassification. Image dropping should not be confused with the preprocessing for removing stems in Section 4.1. Cherry images are taken in multiple directions. In top view images, stems overlap the cherries and 103 104 extend beyond the cherry edges. These images are the ones that are dropped here. In some other directions, such as frontal views and side views, stems are visible but no part of the stems overlap the cherries; all portions of the stems are beyond the cherry edge. These images are not dropped. Instead, the stems in these images are removed during the preprocessing in Section 4.1. Figure 3.4 shows the top and the frontal views. By dropping the stern images manually, it is assumed that mechanical sensors indicate the direction of the cherry stem, if the cherry has a stem; or that any stem present causes the cherry to be oriented in a specific direction relative to the stem. Future work is needed in order to engineer practical sensing environment - the research estimates the performance limits for such a system if it can be developed. Figure 6.1 shows the diagram of the classification procedure. Infrared images are fed into the infrared image edge detection classifier described in section 4.5. Green images are fed into the green image edge detection classifier described in Chapter 5. After that, the output from these classifiers are ”OR”ed. If one of the classifiers classifies the cherry as defective, the cherry is classified as defective. The cherry is classified as non-defective only if all the classifiers classify the cherry as non-defective. The procedure was applied to 23 non—defective cherries and 73 defective cherries. Among defective cherries, 31 of them had bruises but no cracks, 32 of them had wet cracks, and 10 of them had dry cracks. The classification result is shown in Table 6.1. Infrared images are good for classifying non-defective, bruised, and wet cracked cherries as defective or non-defective but they are not good for classifying dry cracked cherries. Green images are good for classifying non-defective and dry cracked cher- ries as defective or non-defective but they are not good for classifying bruised or wet Aqui ion Aftntimon —’ 105 a Infrared Class /--\ cherry lme “as... —> lute” e Aquie on Green 333‘“ I0“. I W 1 =defective Infrared Ime e Cleee er Amer Figure 6.1: Diagram of the integrated classification procedure misclassification rate infrared images green images both infrared and green images non-defective cherries 13% 0% 13% bruised cherries 16% 97% 16% wet cracked cherries 0% 97% 0% dry cracked cherries 50% 10% 10% Table 6.1: Error rate of the integrated classification procedure 106 cracked cherries. The integrated classifier classifies a cherry as defective if only one of its 10 images is defective. Compared to the single—view single-wavelength clas- sifier, the misclassification rate for non-defective cherries of the integrated classifier should be higher and the misclassification rate for defective cherries of the integrated classifier should be lower. The integrated classifier misclassified 13% of non-defective cherries. The integrated classifier can be improved by using underlying classifiers with higher classification accuracy and/ or finding a better scheme to integrate underlying classifiers. Chapter 7 Discussion and Future Work 7 .1 Discussion Bandpass filters that enhance the intensity contrast between bruised and unbruised cherries were determined. An optimum combination of two wavelengths were iden- tified at 750 nm in the infrared range and 500 nm in the green range. An optimum single wavelength band was identified to be at 750 nm. Bruise detection studied in this research used only the infrared images and some spatial analysis. Further study on bruise detection using both infrared and green images is needed. In all bruise detection methods described, removing edge pixels is considered a part of the classification procedure. Edge pixels were removed by morphological erosion with a disk of some size which is arbitrarily set to 16 or 50 pixel diameter. Although, the true labeling of images as good or bruised is difficult to determine accurately by naked eyes, it can be done accurately from the infrared images. True image labels are used for determining classification parameters in some methods and for calculating 107 108 the error estimate in all methods. The prior probability of bruised cherries was set to two values, 5% and 50%. In the global histogram method, there is no other parameter to be specified. Thus, this method needed no ground truth for determining parameters. However, the true class of the sample images is needed to calculate the error rate. In the local histogram method, there are two parameters, the window size and the threshold on the number of bimodal windows. The window size has been set arbitrarily to 80x80 and 40x40. To set the threshold on the number of bimodal windows, we need to know the true labels of training samples. In the thresholding method, there are two parameters, the gray level threshold and the threshold on the number of pixels having lower gray level than the former threshold. To compute these parameters, we need to label the cherry surface pixels as good or bruised. Since the boundary between bruised and good surface is diffused, precise manual labelling is difficult. However, the error is expected to be small. The other value needed for determining parameters is the prior probability of good and bruised cherries. In the infrared edge detection method, there are three parameters; the image reduction factor, edge magnitude threshold, and the threshold on the number of edge pixels. The value of the reduction factor is based on how much noise we have in the image and the length of the ramp edge. Edge magnitude threshold is determined according to the slope of the ramp edge. To compute the last parameter, the true labels of the sample images have to be known. In terms of computation, the thresholding method needs least computation. The .r 1.1“!" 109 edge detection method, the global histogram method and the local histogram method need more computational power in increasing order. In terms of accuracy, the thresh- olding method and the edge detection method do well. The local histogram method has less accurary and the global histogram method has least accuracy. In this research, a spectral radiometer was used to determine the spectral re- flectance of non-defective surfaces and bruised surfaces. This information was then used to determine the wavelength that gives the most contrast between non-defective and bruised surfaces. The ”contrast” in this case refers the difference of the ”average gray level” of non-defective and bruised surface. Suppose the image segmentation algorithm is based on texture analysis. Can the spectral reflectance information help in choosing the wavelength such that the texture classification can be done with highest accuracy? With current technology it is hard if not impossible to do so. This is because equipment used for obtaining spectral reflectance, such as a spectral radiometer, does not have enough spatial resolution. It gives an average spectral reflectance over a big region of the object surface. In fact, if we compare a spectral radiometer and a color camera, the color camera would give us 3 high resolution images in the red, green, and blue bands. A spectral radiometer with sensitivity from 400 to 1100 nm with 1nm increment would give us 701 (= W) images but each image is composed of only 1 pixel. So, a spectral radiometer can be thought of as a very coarse resolution camera. Therefore, it cannot be used in finding an optimum wavelength for texture segmentation. Cherries are quasi-spherical. Since cherries are not perfect Lambertian surfaces, ie., they do not diffuse light equally in all directions, the gray level of pixels in cherry 110 images near the edge and concavity are lower than in the center region. In this research, diffused light is used to illuminate the cherries to help make the gray level of cherry images uniform, but there are still some variation. This problem should be addressed by either designing a better illumination chamber or correcting image gray level by software. The image gray level might be corrected by multiplying with a scale factor from a lookup table. The lookup table can be obtained from training samples. The lookup table is indexed by the pixel angle with respect to the cherry image centroid and normalized distance from the cherry image edge. Another method to simplify the illumination is to construct a proprietary sensor or line-scan camera for cherry grading. It could be a ring shape with sensors looking toward the center of the ring. The cherry is passed through the ring for inspection. With current technology, such a sensor can be constructed. To conclude, cherries are divided into two classes, defective and non-defective. The defective class can be further divided into bruises, dry cracks and wet cracks. The spectral reflectances of bruised and unbruised cherries were analyzed. An optimal set of wavelengths that give the most contrast between bruised and unbruised cher- ries was identified. Five classification procedures for classifying cherries as defective or non-defective were proposed for testing. These methods use a single view single wavelength image of cherry to perform the classification. The first four methods, namely the global histogram method, the local histogram method, the thresholding method, and the infrared edge detection method use single view infrared images to perform the classification. The last two methods outperform the first two methods. The last two methods have comparable performance. When the parameters are ad- 111 justed so that they misclassify 5% of nondefective cherries, they misclassify 15% of bruised cherries. The last two methods also classify wet cracks perfectly but they classify dry cracks poorly. The fifth classification procedure, the green edge detection method, was developed to classify dry cracked cherries from others. It misclassifies 2% of non-defective cherries and 21% of dry cracked cherries. An integrated classifi- cation procedure based on 6 views and 2 waveband images, a total of 12 images, was developed. The procedure is based on a combination of the infrared edge detection method and the green edge detection method. It misclassified 13% of non-defective cherries, 16% of bruised cherries, 0% of wet cracked cherries, and 10% of dry cracked cherries. 7.2 Future Work In Chapter 2, we have determined that using reflectances from two wavelengths re- duces the error rate. In Chapter 4, cherries are classified as bruised or unbruised using the infrared image only. A procedure to classify cherries using two images from two wavelength bands is as follows. Each pixel has two values, the gray levels in the infrared and green images. The mean vector and the variance matrices of bruised and unbruised surfaces can be obtained from training samples. Each pixel in the test sample can be classified using a Bayes classifier. Then, the number of pixels in each class can be counted. Compare this number to the standard and classify the cherry accordingly. Another method of using images from two bands is to combine the two values from the infrared and green images into one value. This could possibly done 112 by computing the difference or the ratio of the gray levels of the pixel the infrared and green images. In this research, the attempt to find the optimum wavelength which gives the most contrast between non-defective surface and cracks has not been done. To find the optimum wavelength, one might want to analyze the spectral reflectance of non— defective cherry surface and the cherry flesh. However, as mentioned before, one might not be able to find the optimum wavelength if the image segmentation algorithm is based on texture analysis. One other possible technique to detect cracks is to use structured light. The idea is to project a known pattern of light such as stripes, dots, or a checker board onto a cherry. Together with the knowledge of relative direction between the light source and the camera, three dimensional information of the cherry can be determined from the image. Thus, a crack could be identified. Figure 7.1 shows some images from a structured light technique. Parallel stripes of light are projected from the left side of the camera. The light projector direction and the camera direction are 45° apart. On a cherry without cracks, stripes in the image are smooth curves. On a cherry with cracks, some stripes would be broken or have sharp turns. Stems might also be identified. In Chapter 6, a classification procedure that classifies cherries as defective (bruised or cracked) or non-defective using multiple images is described. 12 images are taken using the infrared and green filters from 6 different views of the cherry. Two images, infrared and green, where the stem overlaps the cherry are not used as input to the classification procedure. They are manually dropped. It is assumed that mechanical 113 (e) (f) Figure 7.1: Structured light technique for crack detection. (a), (b), and (c) are structured light, infrared, and green images of the same cherry in the same orientation. (d), (e), and (f) are similar. sensors indicate the direction of the cherry stem, if the cherry has a stem; or that any stem present causes the cherry to be oriented in a specific direction relative to the stem. Future work is needed in order to engineer a practical sensing environment. BIBLIOGRAPHY Bibliography [1] Affeldt, H. A. Jr., and Winner, P. W., Lighting Practice and Principles for manual citrus inspection, 1991 International Winter Meeting, ASAE, presentation paper no. 91-3549. [2] Armstrong, P., Zapp, H. R., and Brown, G. K., Impulsive excitation of acoustic vi- brations in apples for firmness determination, Trans. ASAE, 1990, Vol. 33(4), pp. 1353-1359. [3] Canny, J ., A computational approach to edge detection, IEEE trans. PAMI, 1986, Vol 8(6), pp. 679-698. .....a~ sorting of agricultural products, J. Agric. Engng. Res., 1991, Vol. 49(2), pp. 85-98. [5] Chen, P., Sun, Z., and Huarng, L., Factors afiecting acoustic responses of apples, Trans. ASAE, 1992, Vol. 35(6), pp. 1915-1920. [6] Cohen, Y., and Landy, M. S., The HIPS Image Processing Software, P.O.Box 373, Prince Street Station, New Work, NY 10012-0007, USA. [7] Delwiche, M. J ., Tang, 8., and Thompson, J. F., Prune defect detection by line-scan imaging, Trans. ASAE, 1990, Vol. 33(3), pp. 950-954. [8] Delwiche, M. J., Affeldt, H. A., Birth, G., et. al., Surface color measurement of fruits and vegetables. Nondestructive Technologies for Quality Evaluation of Fruits and Vegetables, Proceedings of the International Workshop funded by the United States- Israel Binational Agricultural Research and Development Fund (BARD), Spokane, Washington, 15-19 June, 1993, published by ASAE. [9] Ferraz, A. C. O., Bilanski, W. K., and Alaee, M., Spectral reflectance for potato tubers in the infrared, 1991 International Winter Meeting, ASAE, presentation paper no. 91-7559. [10] Haralick, R. M., and Shapiro, L. G., Computer and Robot Vision: Volumn 1, Addison- Wesley, 1992. [11] Howarth, M. S., and McClure, W. F., Agricultural product analysis by computer vision, 1987 International Summer Meeting, ASAE, presentation paper no. 87-3043. [12] Howarth, M. 8., and Searcy, S. W., Algorithms for grading carrots by machine vision, 1989 International Winter Meeting, ASAE, presentation paper no. 89-7502. 114 115 [13] Ikeuchi, K., and Horn, B. K. P., Numerical shape from shading and occluding bound- aries, Artificial Intelligence, 1981, Vol. 17, pp. 141-184. [14] Lee, C.-H., and Rosenfeld, A., Albedo estimation for scene segmentation, Pattern Recognition Letters, 1983, Vol. 1(3), pp. 155-160. [15] Lee, C.-H., and Rosenfeld, A., Improved methods of estimating shape from shading using the light source coordinate system, Artificial Intelligence, 1985, Vol. 26, pp. 125- 143. [16] Marchant, J. A., Onyango, C. M., and Street, M. J., Computer vision for potato inspection without singulation, Computer and Electronics in Agriculture, Vol. (4) 1990, pp. 235-244. [17] Marchant, J. A., Computer vision for produce inspection, CAB International, 1990, pp. 19-22. [18] Miller, B. K., and Delwiche, M. J ., Peach defect detection with machine vision, 1989 International Summer Meeting, ASAE/CSAE, presentation paper no. 89-6019. [19] Miller, B. K., and Delwiche, M. J ., Spectral analysis of peach surface defects Trans. ASAE, 1991, Vol. 34(6), pp. 2509-2515. [20] Mizrach, A., Nahir, D., and Ronen, 3., Mechanical thumb sensor for fruit and vegetable sorting, Trans. ASAE, 1992, Vol. 35(1), pp. 247-250. [21] Panda, D. P., and Rosenfeld, A., Image Segmentation by pixel classification in (gray level, edge value) space, IEEE trans. Computers, 1978, Vol. C-27(9), pp. 875-879. [22] Parker, B. F., Some effect of chromatic illumination, reflectance and product retation on sorting efi‘iciency of cherries and tomatoes, Ph.D. Thesis, Agricultural Engineering, Michigan State University, 1954. [23] Pentland, A. P., Local shading analysis, IEEE Trans. PAMI, 1984, Vol. PAMI-6(2), pp. 170-187. [24] Petrou, M., and Kittler, J. Optimal edge detectors for ramp edges, IEEE trans. PAMI, 1991, Vol. 13(5), pp. 483-491. [25] Rosenfeld, D., Shmulevich, I., and Rosenhouse, G., Three-dimensional simulation of the acoustic response of fruit for firmness sorting, Trans. ASAE, 1992, Vol. 35(4), pp. 1267-1274. [26] Singh, N., Delwiche, M. J ., Johnson, R. S., et. al., Peach maturity grading with color computer vision, 1992 International Summer Meeting, ASAE, presentation paper no. 92-3029. [27] Singh, N., and Delwiche, M. J ., Machine vision algorithms for defect sorting of stone- fruit Trans. ASAE, 1994, pp. 1989. [28] Throop, J. A., Aneshanesley, D. J., and Upchurch, B. L., Investigation of Texture Analysis Features to Apple Bruises, 1993 International Winter Meeting, ASAE, pre- sentation paper no. 93-3527. 116 [29] Throop, J. A., and Aneshansley, D. J ., Optimizing lighting and lens aperture for man:- imum contrast between bruised and unbruised apple tissue 1993 International Winter Meeting, ASAE, presentation paper no. 93-3595. g, [30] Throop, J. A., and Aneshansley, D. J ., Improvements in an image processing algorithm to find new and old bruises, 1993 International Winter Meeting, ASAE, presentation paper no. 93—6534. [31] Tollner, E. W., Hung, Y.-C., Upchurch, B. L., et. al., Relating X-ray absorption to density and water content in apples, Trans. ASAE, 1992, Vol. 35(6), pp. 1921-1928. [32] Upchurch, B. L., Affeldt, H. A., Hruschka, W. R., et. al., Spectrophotometric study of bruises on whole, red delicious apples, Trans. ASAE, 1990, Vol. 33(2), pp. 585-589. [33] Upchurch, B. L., Affeldt, H. A., Hruschka, W. R., et. al., Optical detection of bruises and early frost damage on apples, Trans. ASAE, 1991, Vol. 34(3), pp. 1004—1009. [34] Upchurch, B. L., and Throop, J. A., Considerations for implementing machine vision for detecting watercore in apples, SPIE, Vol. 1836 Optics in Agriculture and Forestry, 1992, pp. 291-297. [35] Upchurch, B. L., Throop, J. A., and Aneshansley, D. J ., Influence of time and bruise- type on near-infrared reflectance for automatic bruise detection. [36] Wolfe, R. R., and Swaminathan, M., Determining orientation and shape of bell peppers by machine vision, Trans. ASAE, 1987, Vol. 30(6), pp. 1853-1856. HICHIGRN STATE UNIV. LIBRARIES IIIIIIIIIIIIIIIIIIIIIIIIIII III IIII II IIIIIIIII IIIIIIIIII IIIIIII 31293 5603 01572