\ z ..I. . ‘ I ‘
M. .u. . 4 .
a : .H ,
m . . V
F . .
v ,
m. ,
.. . .
w I 4
31%. ”2., ,, H a . .mrrufiqvfia‘ Exé‘wfiwfiw‘gnhwm

 

 

(96%}-
52.250???»

This is to certify that the
thesis entitled

TOWARD AN AUTOMATED RIVET INSPECTION SYSTEM FOR
AGING AIRCRAFT

presented by

Unsang Park

has been accepted towards fulfillment
of the requirements for the

Master of degree in Department of Computer
Science Science and Engineering

 

Major Professor’s Signature
Mat/7 6 , W5}

Date

MSU is an Affinnative Action/Equal Opportunity Institution

 

 

 

LIBRARY
Michigan State
University

 

 

 

PLACE IN RETURN Box to remove this checkout from your record.
To AVOID FINES return on or before date due.
MAY BE RECALLED with earlier due date if requested.

 

DATE DUE DATE DUE DATE DUE

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

6/01 c/CIRC/DateDuepes-pJ 5

TOWARD AN AUTOMATED RIVET INSPECTION SYSTEM FOR AGING
AIRCRAFT

By

Unsang Park

A THESIS

Submitted to
Michigan State University
In partial fulfillment of the requirements
For the degree of

MASTER OF SCIENCE
Department of Computer Science and Engineering

' 2004

ABSTRACT
TOWARD AN AUTOMATED RIVET INSPECTION SYSTEM FOR AGING
AIRCRAFT
By
Unsang Park
This thesis describes work on an automated rivet inspection system for aging
aircraft using magneto-optic imaging (M01). M01 is a non-destructive evaluation
technique that is being used increasingly in aircraft inspection. Even though MOI offers
high efficiency in non-destructive inspection, the large area of material that needs
periodic inspection has created a need for more efficient data interpretation methods: an
automated inspection system. The proposed inspection algorithm focuses on rivets that
are one of the common places where cracks originate.
Motion-Based Filtering (MBF) is developed as an effective filtering method for
MOI images. MBF extracts only “moving objects” in a sequence of images and
suppresses stationary background by using a multiple frame subtraction method. The
filtered images are processed with rivet detection algorithms to properly locate rivets.
Two rivet detection algorithms are developed based on Hough transformation and
morphological operation. The detected rivets are classified by classification algorithms
implemented by Hough transformation or Bayes decision rule.
The off-line test of the prototype automated rivet inspection system on 245 M01
rivet images showed up to 98% accuracy, but more data is needed for testing. Work is
shown in speeding up the algorithms for possible real-time use. A proof-of-concept

inspection system showed the capability of processing 3 to 5 images per second.

Dedicated to my sweet heart, Jung-Bun Lee,

my parents Se-Bong Park and Kyung-Shin Lee.

iii

ACKNOWLEDGEMENTS

I would like to thank Dr. Lalita Udpa for giving me the research opportunity
about aircraft inspection system. She always inspired me with new challenges and
motivations during all my graduate school years.

I would like to thank Dr. George Stockman for his invaluable advice with many
challenging problems. He always gave me warm and kind advice regarding my research
and course work.

I would like to thank Dr. John Weng who taught me many useful techniques in
computer vision class. I also appreciate his comments for my thesis.

I would like to thank Dr. Pradeep Ramuhalli with his advice and help in weekly
meetings. I would like to thank Dr. Anil K. Jain for his advice about the sample
correlation problem.

I would like to thank all MOI project members, Fan Yuan, Zhiwei Zeng, and Xin

Liu, for their support on the project.

iv

TABLE OF CONTENTS

LIST OF TABLES ............................................................................................................ vii
LIST OF FIGURES ......................................................................................................... viii
Chapter 1: Introduction ................................................................................................. 1
1.1 Nondestructive Evaluation Techniques .............................................................. 2

l .1 . 1 Visual Inspection ............................................................................................... 2

l. 1 .2 Ultrasonic Inspection ......................................................................................... 3

1. l .3 Radiography ................................................................................................ 4

1.1.4 Eddy Current Inspection ............................................................................. 6

1.2 Magneto-optic Imaging (MOI) ........................................................................... 8
1.2.1 Eddy Current Excitation ............................................................................. 8
1.2.2 Magneto-Optic Sensing ............................................................................ 10
1.2.3 Imaging ..................................................................................................... 12

1.3 Problem Statement ............................................................................................ 14
1.4 Thesis Organization .......................................................................................... 14
Chapter 2: Image processing of Magneto-optic Images ............................................. 15
2.1 Introduction ....................................................................................................... 1 5
2.2 Analysis of M01 Images ................................................................................... 15
2.3 Motion segmentation ........................................................................................ 18
2.3.1 Background subtraction ............................................................................ 19
2.3.2 Frame subtraction ...................................................................................... 19
2.3.3 Optical flow .............................................................................................. 20

2.4 Motion-based Filtering (MBF) ......................................................................... 20
2.4.1 Frame subtraction with MOI images ........................................................ 21
2.4.2 Additive Frame Subtraction (AF S) ........................................................... 22
2.4.3 Choice of w in AF S ................................................................................... 24
2.4.4 Post—processing of MBF ........................................................................... 27
2.4.5 Overall Algorithm of MBF ....................................................................... 30

2.5 Results of MBF ................................................................................................. 32
2.6 Conclusion ........................................................................................................ 37
Chapter 3: Algorithms for An Automated Rivet Inspection System With MOI ........ 38
3. 1 Introduction ....................................................................................................... 3 8
3.2 M01 Images ...................................................................................................... 38
3.3 Overall Approach of Automated Rivet Inspection ........................................... 43
3.3.1 Automated Rivet Detection ....................................................................... 45
3.3.2 Hough Transformation Technique ............................................................ 46
3.3.3 Morphological Operation Technique ........................................................ 49

3.4 Automated Rivet Classification ........................................................................ 54
3.4.1 Two-pass Hough Transformation Classifier ............................................. 54
3.4.2 Bayesian Classifier .................................................................................... 57

3.4.3 Feature Selection ....................................................................................... 59

3.4.4 Distributions of the selected features ........................................................ 60
3.5 Results of Automated Rivet Inspection ............................................................ 63
3.6 Conclusion ........................................................................................................ 65
Chapter 4: REAL-TIME IMPLEMENTATION OF AUTOMATED RIVET
INSPECTION SYSTEM .................................................................................................. 66
4. 1 Introduction ....................................................................................................... 66
4.2 Optimization of Motion-Based Filtering .......................................................... 66
4.2.1 Frame Grabbing ........................................................................................ 68
4.2.2 RGB to Grayscale conversion ................................................................... 68
4.2.3 Frame subtraction and Combining ............................................................ 70
4.2.4 Thresholding ............................................................................................. 71
4.2.5 Median Filtering ........................................................................................ 71
4.2.6 Contrast Stretching .................................................................................... 75
4.3 Results of the optimized MBF .......................................................................... 76
4.4 Optimizing Rivet Detection and Classification ................................................ 77
4.5 Personal Computer-Based Proof-of Concept System ....................................... 78
4.6 Digital Signal Processor (DSP)-Based Prototype System ................................ 79
4.7 Results of Real-Time Automated Rivet Inspection System ............................. 80
4.8 Conclusion ........................................................................................................ 82
Chapter 5: CONCLUSION AND FUTURE WORK .. ............................................... 83
5.1 Conclusion ........................................................................................................ 83
5.2 Future Work ...................................................................................................... 85
BIBLIOGRAPHY ............................................................................................................. 87

vi

LIST OF TABLES

Table 2-1: Contrast of objects in M01 images before and after filtering. ........................ 37
Table 3-1: Accuracy of two rivet detection algorithms. ................................................... 63
Table 3-2: Inspection accuracies of three different inspection algorithms. ...................... 64

Table 4-1: Processing time for filtering one image with MBF algorithm using ten past

images. ...................................................................................................................... 67
Table 4-2: Time for converting color image to gray scale ............................................... 70
Table 4-3: Processing time of median filtering with various algorithms ......................... 74
Table 4-4: Measured time for each step of MBF before and afier optimization. ............. 76
Table 4—5: Execution times of Hough transformation-based and morphological operation-

based rivet detection. ................................................................................................ 77
Table 4-6: Execution time of the automated rivet inspection ........................................... 8O

vii

LIST OF FIGURES

Figure 1.1: Schematic of (a) ultrasonic inspection (b) oscilloscope display of the reflected

waves ........................................................................................................................... 3
Figure 1.2: Schematic of radiographic inspection .............................................................. 5
Figure 1.3: Eddy current induction in a specimen .............................................................. 7
Figure 1.4: Uniform sheet-type eddy current excitation ..................................................... 9

Figure 1.5: The effects of rotating sheet-type eddy current in M01 images of rivets. (a)
Linear eddy current excitation (b) Rotating linear eddy current excitation [10]. ..... 10

Figure 1.6: (a) Eddy current and induced B field on a flat specimen without any
abnormality (b) Eddy current and induced B field on a flat specimen with an

abnormality. .............................................................................................................. 11
Figure 1.7: Magnetization of magneto-optic sensor [8] .................................................... 11
Figure 1.8: Schematic of the M01 instrument. .; ............................................................... 12

Figure 1.9: M01 system including handheld sensor (left), excitation unit (center), and
wearable monitor (right). .......................................................................................... 13

Figure 2.1: Two consecutive MOI images in V; dark disks represent rivets, which are

moving to the left. ..................................................................................................... 16
Figure 2.2: Average pixel intensity changes in a selected area that encloses (a) only

background, and (b) both background and object ..................................................... 18
Figure 2.3: A binary difference image from two timely consecutive images ................... 23

Figure 2.4: The typical behavior of the white area of Di(x, y) in a sequence of difference

images. The area of Di" reaches its maximum at i=4 in this case. ........................... 26

Figure 2.5: The typical behavior of the white area of Di(x, y) in a sequence of difference
images. ...................................................................................................................... 27

Figure 2.6: (a) Combined difference image (b) after stretching (c) after thresholding. 30

Figure 2.7: Algorithm for motion-based filtering. w denotes number of difference images.
................................................................................................................................... 31

viii

Figure 2.8: (a),(b),(c),(d) MOI images in time sequence. Rivets are moving to the left
while sensor is moving to the right; (e) Motion-based filtered image of the original
image ((1). .................................................................................................................. 32

Figure 2.9: (a),(b),(c),(d) Synthetic MOI images in time sequence with uni-directional
scan. Rivets are moving to the left while sensor is moving to the right; (e) Motion-
based filtered image with uni-directional scan. ........................................................ 33

Figure 2.10: (a),(b),(c),(d) Synthetic MOI images in time sequence, multi-directional scan.

Rivets are moving to the left while sensor is moving to the right with small
sinusoidal up down motion; (e) Motion-based filtered image with multi-directional
scan. .......................................................................................................................... 34

Figure 2.11: Results of MBF. Rectangular box shows selected areas for measuring

contrast. (a)(b) Image 1, (c)(d) Image 2, (c)(f) Image 3. .......................................... 36
Figure 3.1: (a) Two normal rivets (b) Right rivet has a radial crack (c) A crack between

two rivets ................................................................................................................... 39
Figure 3.2: (a) Seam and two normal rivets (b) A crack along the seam ......................... 39
Figure 3.3: (a) (b) Normal seam and two rivets (c) ((1) Crack along the seam ................. 40

Figure 3.4: Sequence of images with corrosion dome. Corrosion is moving from up left to
down right, as the M01 sensor is moving. ................................................................ 41

Figure 3.5: Images of normal objects are shown differently because of the wobbling of
the M01 instrument. .................................................................................................. 42

Figure 3.6: Images of defective objects are shown differently because of the wobbling of
the MOI instrument. .................................................................................................. 43

Figure 3.7: (a) A schematic rivet with a radial crack (b) A schematic MOI image of the
rivet with a radial crack (0) A schematic rivet with a circumferential crack (b) A

schematic MOI image of the rivet with a circumferential crack. ............................. 44

Figure 3.8: Typical rivet images in M01 inspection after MBF filtering: (a) (b) normal
rivets (c) (d) defective rivets. .................................................................................... 44

Figure 3.9: Overall approach of automated rivet inspection algorithm ............................ 45

Figure 3.10: Schematic of voting in 3-dimensional accumulator in Hough transformation.
(a) Each edge votes for its center coordinate ........ (b) Accumulator collects the votes.

Figure 3.11: (a) Original MOI image (b) Rivet detection by morphological operation. .. 49

ix'

48

Figure 3.12: Schematic of connected components and bounding rectangles in an image.50
Figure 3.13: Morphology of structuring element. (a) Square shape (b) Circular shape... 51

Figure 3.14: Example results of intermediate steps of morphological operation for rivet
detection with intermediate variables: (a) initial rivet image (b) after iterative

erosion operation (c) after rivet detection. ................................................................ 52
Figure 3.15: Rivet detection algorithm using morphological operation. .......................... 53
Figure 3.16: (a) Original image. (b) Rivet detection by morphological operation. .......... 54

Figure 3.17: Second pass Hough transformation for rivet classification. E' represents the
image of rivet edge that is outside of the circle detected in the first Hough
transformation. .......................................................................................................... 55

Figure 3.18: Two-pass Hough transformation method: (a) raw image (b) after first Hough
transformation (0) after second Hough transformation. Right rivet is classified as
defective. ................................................................................................................... 56

Figure 3.19: Graphical representation of the feature f for Bayesian classifier (a) Hough
Transformation-based rivet detection (b) Morphological operation-based rivet

detection. ................................................................................................................... 60 .

Figure 3.20: The distributions of feature f from (a) Hough transformation-based rivet .
detection (b) Morphological operation-based rivet detection. .................................. 62

Figure 4.1: Effect of median filtering in MBF. (a) Before median filtering. (b) After
Median filtering with 5 by 5 kernel. Bright objects represent rivets and corrosion. 75

Figure 4.2: A proof-of-concept system for real-time automated rivet inspection. ........... 78
Figure 4.3: A prototype system for the real-time automated rivet inspection system. ..... 79
Figure 4.4: Results of the C++ proof-of-concept system. (a) Raw and processed MOI

image without defect. (b) Raw and processed MOI image with defective rivet, the
raw image is boxed to highlight the defective rivet. ................................................. 81

Chapter 1: Introduction

Nondestructive evaluation (NDE) is widely used in inspecting aircraft structures
to detect surface and subsurface defects. The Magneto-Optic Imager (M01) is a powerful
device for the Nondestructive Inspection (NDI) of aging aircraft. MOI produces analog
images of magnetic flux leakage associated with eddy current distribution around surface
and subsurface structures. The main advantages of using MOI are its fast inspection
speed and easy interpretation compared with conventional Eddy Current NDI instruments.
However, due to the magnetic domain wall structures of the sensor, the M01 images are
corrupted by serpentine pattern noise, which lowers the M01 inspection capabilities. The
noise also poses difficulties in the quantitative analysis of M01 images, which is one of
the crucial requirements for automatic inspection.

This thesis advances the state of rivet inspection from MOI along several lines. A
proposed automated rivet inspection algorithm is composed of three major steps, Motion-
based Filtering (MBF), rivet detection, and rivet classification. MBF is developed to
preprocess the M01 images to remove the background noise using a multiple frame
subtraction method. Objects, such as rivets and scams, persist from frame to frame and
appear to move as sown in Figure 1.1. From the processed image, rivets are detected
using Hough transformation or morphological operation. Finally, detected rivets are
classified as normal or defective by additional Hough transformation method or Bayesian
classifier. Algorithms shown to be successful are transformed in order to speed the

processing for real—time application.

   

(a) (b) (C)
Figure 1.1: A sequence of sample MOI images'. Rivets and seam appear to move as
the M01 scans over sample.
1.1 Nondestructive Evaluation Techniques

Nondestructive Evaluation (NDE) is the inspection of materials and components
for determination of their integrity or detection of any abnormality and, without
impairing their usefulness [1]. NDE is used for quality control of products in
manufacturing and monitoring the product during operation to assess the remaining life.
Based on the physical principles they rely on, NDE methods are categorized as Visual
inspection, Ultrasonic inspection, Radiographic inspection, Eddy Current inspection, etc.
A brief description of some of the widely used NDE techniques is provided in the

following sections.

1.1.1 Visual Inspection
Visual inspection is the oldest and most widely used among all the NDE
techniques. It is usually performed as a first step of the evaluation process because of its
simplicity and low cost. The specimen is illuminated with light and inspected by the

human eye. Using optical instruments such as microscope, telescope, and holography

 

' Text in the image came with the original data from Boeing and is not clearly legible. Throughout this
thesis some original images have been enhanced for acceptable printing.

may increase the visual inspection capability [1]. Visual inspection can detect and
analyze only surface abnormalities. More sophisticated techniques are needed for

subsurface inspection.

1.1.2 Ultrasonic Inspection

Ultrasonic inspection is a versatile NDE technique applicable for most materials,
metallic or non-metallic. It utilizes high frequency acoustic waves to detect surface and
subsurface defects [1]. The ultrasonic energy travels through a test sample and reflects
when it strikes a discontinuity. The reflected ultrasonic energy provides information
related to subsurface defects. From the propagation distance of ultrasonic energy, derived
from propagation velocity and time, the specimen thickness and location of defects are
calculated [2]. An estimate of the shape of the defect is also obtained by moving the
probe around the defect and measuring defect echo. Figure 1.1 shows the geometry of a

specimen with an internal defect and its corresponding pulse-echo response.

Crack echo Back wall echo

 

F
N
Fifi

 

Crack 2a1

 

 

 

 

 

 

 

(a) (b)

Figure 1.2: Schematic of (a) ultrasonic inspection (b) oscilloscope display of the

reflected waves

If L is the thickness of the specimen and v is the corresponding velocity, echoes from the

specimen back wall and defect appears at time 3E and 33‘. in the oscilloscope,

V V
respectively. Ultrasonic energy may propagate in any material which behaves in an
elastic manner and is therefore applicable to many types of materials including metal,
rubber, and plastics. Ultrasonic inspection has several advantages. First, it is portable and
needs only one accessible surface of the specimen. Second, it can be used for inspecting a
large (a few meters) thickness of sample. Third, it is relatively inexpensive and finally, it
allows rapid and automated inspection. However, ultrasonic inspection has a couple of
limitations. First, it needs a coupling medium to couple the ultrasonic energy into the
specimen. The test specimen is either immersed in water or a coupling gel is applied.
Second, with a coarse grained specimen, penetration depth may be reduced to as little as
50 or 100 mm at high frequencies and also reflections from grain boundaries result in
significant noise in the signal. Third, highly skilled operators are required for data

interpretation.

1.1.3 Radiography
Radiography is one of the most widely used NDE methods for the detection of
internal defects such as porosity and voids [1]. Planar defects can also be detected with
proper orientation. Radiography uses short wavelength electromagnetic radiation, such as
X-rays or gamma rays that can penetrate materials. As the radiation enters the material, it
is absorbed and the amount of absorption depends on the density and thickness of the

material [3]. Internal defects can be detected by observing the variation of the amount of

absorption of the radiation. The degree of absorption is usually recorded on a film which
is sensitive to the radiation source as in Figure 1.2. The location and shape of the defect is

directly reflected on the film after irradiation.

  

' / Specimen
H

Defect
Film

Defect image

 

Figure 1.3: Schematic of radiographic inspection

Radiography can be used for most types of solid materials, both ferrous and nonferrous
alloys, as well as nonmetallic materials and composites. However, radiographic
inspection has several limitations. First, it is not portable and it is often difficult or
impossible to position the film and source of radiation to obtain a radiograph of the
desired area. Second, the depth of inspection is limited by the degree of penetration of the
radiation in the specimen. The degree of penetration varies depending on the material.
Third, the defects must be parallel to the radiation beam and must be at least 2% of the
thickness of the specimen. Fourth, equipment including radiation source generator and

safety facility is rather expensive.

1.1.4 Eddy Current Inspection
Eddy current inspection has been used for over four decades as a leading in-
service NDE method. It can be applied to irregularly shaped conductive material for
detecting surface and subsurface abnormalities such as cracks and corrosion. Eddy
current testing uses alternating current flowing in a coil to produce an alternating

magnetic field in accordance with Ampere’s Law:

va=J (LU
where H is the magnetic field in Ampere/meter and j is the current density in
Amperes/meterz. F araday’s Law of Induction provides the basis for electromagnetic
induction and is given by the equation:

~ 63

v E=——— 14
X at ( )

where E is the time-varying electric field intensity in Volt/meter and B is the time-

varying magnetic flux density in Weber/meterz. This equation illustrates that a time-

varying magnetic field B will induce an emf in a conducting path. When the coil is
brought close to an electrically conducting material, and eddy current is induced in the
material due to the electromagnetic induction. The induced eddy current in the material
generates a secondary magnetic field opposite in direction to the primary field. Figure 1.3
shows the schematic of electromagnetic induction in the specimen by the eddy current

probe coil [2].

Primary
/ magnetic field
Secondary
f/ magnetic field

@ A— Secondary
/ eddy currents

Figure 1.4: Eddy current induction in a specimen

 

 

 

 

The secondary magnetic field may be detected either as a voltage change across a second
coil or by the perturbation of the impedance of the original coil. This impedance change
is correlated with the surface and subsurface structure of the material, hence used for
detecting defects in materials. The eddy currents in an infinitely large planar sample

decay exponentially from the surface of the sample according to the equation [4]:

j=joexp(—§-) 0-»

where y is the distance below the surface of the sample, jo is the current density at the

surface of the sample, and 6 is the depth of penetration. The depth of penetration is

defined as [5]:

 

(1-4)

where 0' is the conductivity, ,u is the relative permeability, and f is the inspection

frequency. As shown in equation (1-3), the frequency should be carefully chosen to keep
the depth of penetration large enough to reach the subsurface defects. Eddy current
inspection has several advantages. First, it is portable and needs access to only one side
of the specimen. Second, the equipment is cheap relative to Radiographic instruments.
Third, it allows rapid and automated inspection. The major drawback of eddy current
testing is that it can be used only for electrically conductive materials. Another drawback

is that the depth of inspection is limited to a few millimeters [l].

1.2 Magneto-optic Imaging (MOI)

Magneto-optic Imaging (M01) is a variant of the eddy current inspection method,
and it is becoming increasingly practical for defect assessment in aircraft structures. This
relatively new nondestructive testing method gives the inspector the ability to quickly
generate real-time images of defects in large surface areas [6][7][8][9]. M01 is based on
the use of a combination of principles of magneto-optic effects and eddy current
induction. The theory and principles of an MOI system are presented in the following

sections.

1.2.1 Eddy Current Excitation
MO Imaging differs in its operation from conventional eddy current testing in that
MOI generates uniform sheet eddy currents in the specimen over the area of the magneto-

optic sensor as shown in Figure 1.4 [8].

’\/ .
Primary
Eddy currents

 

Secondary Eddy
currents (linear)

 

 

 

 

 

 

 

 

 

Figure 1.5: Uniform sheet-type eddy current excitation

The rectangular gray area on the specimen in Figure 1.4 is the active area of inspection.
Alternatively, MOI uses rotating eddy current to eliminate the necessity of aligning
induced current perpendicular to the long axis of cracks for successfiil detection. Rotating
eddy current is provided using an alternate configuration with two separate primary coils
connected to a single foil by two single-tum secondary windings. The two-source coil
must produce currents perpendicular to each other and they must be out of phase by
ninety degrees [7]. The current density in the specimen is represented by the equation:

J =J0 sin(a)t+<l)) (1-5)
The current density in the two primary coils is represented as:

J = J0 sin(wt)f + J0 cos(cot)] (1-6)

where i and j are unit vectors along the x and y coordinate axes. The symbol J can be
interpreted as a linear current density rotating with an angular frequency a) [7]. Rotating
eddy current excitation eliminates the null zone observed in linear excitation in which a
rivet is shown as a slotted screw. Figure 1.5 shows the effect of rotating linear current on

the rivet images.

Null Crack

   

(a) (b)

Figure 1.6: The effects of rotating sheet-type eddy current in M01 images of rivets.

(a) Linear eddy current excitation (b) Rotating linear eddy current excitation [10].

1.2.2 Magneto-Optic Sensing
While the conventional eddy current method uses the impedance changes of a coil
to detect abnormalities, MOI produces analog images that reflect abnormalities in the
surface and subsurface of the specimen. The magneto-optic image is the cross-sectional
view of the secondary magnetic field associated with eddy currents induced in the
specimen. A flat specimen with no structure generates a secondary magnetic field
tangential to the specimen. On the other hand, abnormalities in the specimen generate a

magnetic field normal to the plane. This is shown in Figure 1.6 schematically.

 

 

 

 

 

 

(a) (b)

Figure 1.7: (a) Eddy current and induced B field on a flat specimen without any
abnormality (b) Eddy current and induced B field on a flat specimen with an

abnormality.

Magnetization of

/ I, / magneto-optic sensor
I Sheet currents in the
”My copper induction foil

/ {93' ‘ Specimen
r Rivet

Induced current

 

 

 

 

 

 

 

 

Figure 1.8: Magnetization of magneto-optic sensor [8].

The MOI sensor has an easy axis of magnetization, which is normal to the sample surface,
and hence only the normal component of the secondary magnetic field is detected by the

magneto-optic (MO) sensor. By placing the sensor parallel to the testing specimen, the

11

easy axis of the sensor is normal to the plane of the specimen. Thus, the secondary
magnetic field produced by rivets, cracks, and corrosion is detected by the sensor, and the
primary magnetic field parallel to the flat specimen is ignored. Figure 1.7 shows the

magnetized area in the MO sensor by the magnetic field produced by a rivet.

1.2.3 Imaging

A magneto—optic image is produced by applying the principles of Faraday rotation.

The Faraday magneto-optic rotation states that when polarized light travels through a
magneto-optic material in a magnetic field, its plane of polarization is rotated. The

amount of rotation is known as Faraday rotation and is calculated as [7][8][11]:

IIM

V
N

(1-7)

where I; is the wave vector of the light, I is the thickness of the material, and A7! is the

magnetic field. The MO sensor is characterized by a large specific Faraday rotation.

Light source

MO Sensor Polarizer Analyzer

Induction Sheet

Bias Coil *x I

I
Sample I I )I

I i ‘ I c It VT' ’*' ‘-
-"-- :-:-"I-- "EW‘ 1:?
,.. _,* 'fi“»".. ‘
n . . . . fifty _ . meam '.:»5‘3‘§5§‘**?€”
. -- _ J.\

.2 -é EEF‘ETI‘R'C‘TKWWSEEQ

  

 

Figure 1.9: Schematic of the M01 instrument.

12

In MOI, light is transmitted through a polarizer into the MO sensor, reaches the back-
reflecting surface under the sensor, and then returns to the analyzer to produce a
magneto-optic image. The schematic of the M01 system with the paths of polarized lights
is shown in Figure 1.8. A magneto-optic image is produced by a sequence of operations.
The MO sensor is cleared by an erase pulse, and then magnetized by the normal magnetic
field from abnormalities in the specimen. The magnetization of the MO sensor is retained
until the next erase pulse, allowing the imaging operation to be performed. Images are
generated and erased about 26 times a second [7]. A typical MOI system is shown in

Figure 1.9.

 

Figure 1.10: M01 system including handheld sensor (left), excitation unit (center),

and wearable monitor (right).

1.3 Problem Statement

MOI generates real-time analog images that are the basis for defect detection.
While the M01 image provides fast and easy inspection capability, it has several
drawbacks. One of the most serious drawbacks is that the M01 image contains serpentine
pattern noise from the domain structure of the MO sensor as seen in Figure 1.5. The noise
hinders the probability of detection (POD) of second and third layer cracks and corrosion.
Hence, the primary goal of this thesis is to develop a filtering algorithm to enhance the
quality of M01 images and develop an algorithm for automated inspection based on a
quantitative measure of M01 image features. In addition to accuracy of detection, the
algorithms for filtering and automated inspection need to meet real-time processing

conditions. Hence, problems of practical real-time implementation are also considered.

1.4 Thesis Organization
The thesis is organized into five chapters. Chapter 2 discusses the characteristics
of M01 images and introduces a motion-based filtering (MBF) algorithm, which is an
image processing algorithm for MOI images. Chapter 3 discusses the algorithm for
automated rivet inspection based on the MBF results. Chapter 4 discusses the issues
related to real-time implementation of the automated rivet inspection system. Chapter 5

gives a description of the results and discusses future work in this area.

14

Chapter 2: Image processing of Magneto-optic Images

2.1 Introduction

An MOI image contains serpentine pattern noise, which is due to the magnetic
domain wall structures of the magneto—optic sensor. The noise lowers the probability of
detection (POD) of defects and corrosion and makes data interpretation difficult [12]. In a
sequence of MOI images, noise associated with the domain structures in the sensor is
stationary as the sensor moves relative to test sample while images associated with
structures (rivet) or corrosion in the test sample move flom flame to flame due to relative
motion of sensor and sample. Conventional image processing methods process each
frame of M01 in a scan sequence [13][14]. In this chapter, we introduce a new image
filtering technique based on the characteristics of MOI revealed in a sequence of images
generated during scanning of the sample surface. The algorithm presented in this chapter
is based on separating the moving parts flom the stationary parts in the sequence of
images. Noise is reduced by retaining and boosting only moving parts and suppressing

the stationary part of images.

2.2 Analysis of M01 Images
The MOI inspection is generally conducted by a human operator by scanning the
surface of a sample with the sensor. The MOI image data is interpreted by the operator in
real-time or recorded as a video sequence for off-line interpretation. The method
proposed in this thesis uses the information in a sequence of M01 images for enhancing

the quality and hence the corresponding inspection capabilities of the MOI system. Let

15

V= {In(x, y), n=1,2,. . .} represent the video sequence, where 1,,(x, y) represents the tim
image in the video.

Figure 2.1 shows two consecutive flames in an MOI scanning video. The dark
disks represent rivets of l/8-inch diameter and the dark band running horizontally
through the center of the image shows a subsurface scam in the airflame structure. The
dark areas on the left and right sides are due to the leakage of uncompensated magnetic
field on the edge of the inspecting area. Figure 2.1 shows that the background
components are stationary flom flame to flame while objects in the sample are moving.
The rivets have moved to the left while the scan is directed to the right. The background
noise is associated with magnetic domain walls in the sensor and is stationary relative to

the sensor during the scan.

   

(a) (b)

Figure 2.1: Two consecutive MOI images in V; dark disks represent rivets, which are

moving to the left.

MOI image data can be divided into two components, dynamic foreground and
stationary background. Usually the foreground image is regarded as the signal and the

background regarded as noise to be suppressed. Each image can be expressed as below.

16

1n(x.y) = 0,106.}? + 30W) (2-1)
where 0,,(x, y) is the object component in the nth flame and B(x, y) is the serpentine noise

pattern that is fixed for all n. In Figure 2.1, with the notations in equation (2-1), 1,,(x, y) is

the entire image, 0,,(x, y) corresponds to the dark disks that represent the moving image of
rivet and the seam above the rivets, and B(x, y) is the background area.

Pixel intensity changes are observed in selected areas to show quantitatively the
presence of an object feature in the M01 images. Two areas are chosen in a MOI
inspection sequence as shown in Figure 2.1. Upper boxes A enclose only background
images, while lower boxes B enclose both background and object images as the M01
scans the sample. Both boxes have the same area, 3600 pixels. The average pixel
intensity changes are calculated in the selected boxes as pixel-by-pixel subtraction.

Equation (2-2) presents the average intensity change for two boxes respectively.

1 N M

n = W Z Z (In(p,q) -11(p,q)) (2-2)

p=l q=1

A}

where j represents the average pixel intensity, It represents the first image, and M, N are
the window dimensions. The values of In(p,q)-11(p,q) in equation (2-2) are clipped to zero
when they are negative.

Figure 2.2 shows the average pixel intensity changes, Ajn, Is It $100, with respect
to n in the selected boxed areas. The plot (a) corresponding to the region of box A is seen
to stay fairly constant whereas plot (b) containing the object varies widely with time n,

between background and object intensity levels.

17

f 50
‘_ 45

4O

35
-- 30
25 *
20

15-
10_ U

..‘l
v- C) 0) CO I\ (.0 to V C") N v— O .—
v— 1— N (O V LO (.0 I\ (I) O) O

  
 
  
 

7
6
5,,
4i
3
2
1
o

delta i
delta

t.

 

 

 

11.1.llllllllIllllUlLlllIllllllIIlli‘.-._.;i...‘llll._’-.l;.ll.l.1.:.i_llll.lllll

 

IO
19
28
37

46
55
64
73
82
91
100

‘—

n n

(a) (b)

Figure 2.2: Average pixel intensity changes in a selected area that encloses (a) only

background, and (b) both background and object.

This result is utilized for eliminating the stationary background flom the sequence
of flames and the issue of motion analysis is involved here. Various motion analysis

techniques are discussed in the next section for this purpose.

2.3 Motion segmentation
Detecting and estimating motion in a video sequence is increasingly gaining
interest in diverse areas such as video surveillance, object tracking, video compression,
and etc. [15][16] Frame subtraction techniques usually segment the motion first, and then
further analyzes the motion. On the contrary, the optical flow technique starts by
analyzing the motion which is then segmented based on the analysis if necessary. In MOI

images, motion analysis is mostly restricted to motion segmentation. Therefore, some

18

motion analysis techniques focusing on motion segmentation are discussed in following

sections.

2.3.1 Background subtraction

The simplest method of motion extraction is background subtraction [17]. Each
image is compared with the reference background image and the difference between the
two images is extracted. This method is used when the background is static over a
relatively long period of time, which is often the case in video surveillance. In video
surveillance, the reference background is periodically updated. The background
subtraction is very simple to implement and computational cost is very low, which makes
it optimal for real-time processing. However, keeping the reference image static is not
trivial in many situations. The reference is easily corrupted by a small camera oscillation.
In addition, even when the background is stationary with respect to noisy motion, it is
usually not static with respect to illumination. The illumination change is detected in the

background subtraction and estimated as a motion.

2.3.2 Frame subtraction
Frame subtraction against past images is used for motion detection when the
reference image is not obtainable. Most flame subtraction technique use two or three
consecutive flames [18] [19] [20]. The background image is assumed not to change much
across the flames. The detected change in two or three flames is used to retrieve the
outline of the moving object. The moving object is segmented flom the outline or further

processing is performed to refine the segmentation. The flame subtraction technique can

19

be used in more general cases than background subtraction because it does not need the
reference image. However, the way of segmenting the motion in flame subtraction
depends on the velocity of the motion in the image. If the motion is too slow, it cannot be

detected, while the segmentation is overestimated if it moves too fast.

2.3.3 Optical flow

Horn and Schunck used a motion constraint equation to analyze motion in a
sequence of images [21]. By solving the equation, each pixel in the image is evaluated for
its magnitude and direction of motion. A set of pixels that are correlated with the optical
flow is segmented as one region. The set of pixels with large magnitude of motion is
corresponding to the moving object in given image. Optical flow provides more
information about the motion in an image by estimating direction of motion, which
allows more complicated analysis than background subtraction or flame subtraction.
However, the motion between consecutive flames is assumed to be small in optical flow.
The greatest disadvantage of optical flow is its high computational cost. This lowers the
applicability of optical flow in real-time systems and for analyzing MOI images. Another
disadvantage of optical flow is the noisy result that causes difficulty in the motion

segmentation.

2.4 Motion-based Filtering (MBF)
In MOI images, a reference background image is hard to obtain because flames
may contain moving objects most of the times. The background can be obtained by

getting off the M01 instrument from a sample, but the background changes substantially

20

when the instrument is later put on the sample surface. However, the background is
stationary while MOI scans over a small region of a sample. In addition, the velocity of a
moving object is controllable because it is the scanning rate of the human operator or
robot system. Thus, flame subtraction across a certain number of flames is a suitable
motion analysis method for MCI images for the purpose of the motion segmentation. The

model for flame subtraction in M01 images is discussed in next section.

2.4.1 Frame subtraction with MOI images
Assume each image in the sequence has both object and background regions in it
and object pixels 0(x, y) have lower (darker) intensity than that of the bright background
B(x, y). Let D,- be the difference image obtained by subtracting current image 1,,(x, y) flom

the past image 1,,_,-(x, y).

Di(x»y) = In-i(x:}’) ' Ina,” (2’3)

Then, D,- can be divided into 4 segments Di] , D52, 03, and D,-4 as described below.

Di] = my» my) 60,.-. n 0”}
Diz = {(val (x,y) EBn-i an}
(2-4)

DR = my» my) 50,,- n 3,;

Di4 = {(xJ’N (LY) EBn-i n 0n}

21

where 0,, represents the moving objects, and 3,, represents the stationary background at

time n. It can further be noted that under the assumption 0,,(x,y) < B(x, y), difference

images in the 4 regions have zero, negative, or positive intensities. The expected

intensities in the different segments of D,- can be identified as below.

Di] = {(x,y)| D,-(x,y) = 0} .' zero intensity
Diz = {(x,y)| Di(x,y) = 0} .° zero intensity
(2-5)

D3 = {(x,y)| D,-(x,y) <0} : negative intensity

Di4 = {(x,y)| Di(x,y) > 0} : positive intensity

To simplify the problem, we replace the pixel intensities for D3 by zero. As a
result, the difference image pixels in Di], 0,2, and 0,3 have zero intensities and only

region Di" has positive intensity values. Therefore the resulting difference image can be
divided into two distinct regions with zero and non-zero intensities and the non-zero

intensity area, Di4, belongs to the object, 0,,, in 1,,(x, y) as illustrated in Figure 2.3.

2.4.2 Additive Frame Subtraction (AFS)

In Figure 2.3, part of the object is also lost while the background noise is

substantially removed. The object that is lost by subtraction belongs to Di] .

22

 

Figure 2.3: A binary difference image from two timely consecutive images.

The entire extent of moving objects, including Di], can be recovered using

multiple difference images. This is done by subtracting the current image flom several

past images, and combining the difference images using an OR operation to get the
binary image, Sn_0R, or a MAX operation to get the grayscale image, SnMAX, as described

below.

Sn,og(x,y) = 1gétwwibeyn (2-6)
Sn,MAX (x, y) = ggiDi (x, Y» (2-7)

where Smog or Sn,MAX is the integrated difference image with OR or MAX operation and
w represents the number of difference images. In the OR operation, any pixel value

Sn’0R(x, y) is the result of OR operation on {D 1(x, y), ..., Dw(x,y)}, which is zero if all

Di(x, y) are zeroes, and 1 otherwise. On the other hand, in MAX operation each pixel

23

value SnMA X(x, y) is the result of MAX operation on {D 1(x, y), ..., Dw(x,y)}, which is the
maximum value of D,-(x, y), 15 i SW. The method described above is called Additive

Frame Subtraction. In practice, Snag or Snmx can be computed recursively as described

below.

Sk,0R(an)=0R{S (xay)9Dk(xay)}9 S] :D] ZSkSW (2-8)

k—l,0R

SH”M (x, y) = MAXiSk—mu (x,y),Dk (x,y)}, S1 = D1 2 S k S w (2-9)

where SkOR IS Sn,0R and SkMAX IS SnMAX, when k=W.

Selecting the optimal value of w is an issue in Additive Frame Subtraction. Two

possible techniques for choosing w are presented below.

2.4.3 Choice of win AFS
The value w, which is the number of difference images, needs to be chosen with

care. In general, w is a function of the object size and scanning velocity. If w is too small,
the algorithm will fail to extract the entire object flom the M01 sequence and if w is too
large, it will result in a large computation time. The optimum value w can be obtained in
two ways.

(i) Static method: Let v be the scanning velocity (inches/sec), s the object size along the
scanning direction(inches), and f the flame rate(frames/sec). The relationship between w,

v, s, and f is expressed as

24

 

(2-10)

In equation (2-10), we can predefine w only if we set the scanning velocity to a

constant because 5 and f are constant.

(ii) Dynamic method: An alternative approach is to calculate the area of the region with

non-zero intensity, the number of pixels in Di" in the difference image Di(x, y). As
illustrated in Figure 2.4 and 2.5, typically the area of Di(x, y) increases with i and reaches

a maximum when the object in 1,, and 1,,_4 are completely separated. An optimal estimate

of w can be chosen as the corresponding i for which the area reaches its maximum.

25

  

(a) D3 (h) D2 (i) D]
Figure 2.4: The typical behavior of the white area of Di(x,y) in a sequence of

difference images. The area of DE reaches its maximum at i=4 in this case.

26

6000 —

5000 r

4000 r

3000 r

area

2000 r

1000 r

 

O L l L I I I I 4 I—J

1 2 3 4 5 6 7 8 9 10
i

Figure 2.5: The typical behavior of the white area of D,{x,y) in a sequence of

difference images.

2.4.4 Post-processing of MBF
Post processing is performed to enhance the image contrast and remove arbitrary
isolated noise pixels. As discussed in Section 2.3.3, the output of Additive Frame
Subtraction can be binary or grayscale. The grayscale filtered images are processed in a
few post processing steps, while the binary filtered images are directly used as initial data
for further analysis such as automatic rivet classification.
The first post processing performed on both binary and grayscale output images is

median filtering. The median filter is defined as:

J(x,y)=median{1(x,y)|x—w$x$x+w,y—wSySy+w}
(2-11)

w = [%J — l, W is an odd number

27

where J(x, y) is the filtered image, 1(x, y) is the original image, and W is the square kernel
window size. The next operation is contrast enhancement. The contrast enhancement
increases the contrast of the objects in the combined difference image, so that they
become more visible. A grayscale stretching is first applied to map the intensity
distribution of combined difference images to the range flom 0 to 255. The mapping

function is described as:

255 * Sn (x,y)

sn' , = .
(x Y) (maxtsntx.y»-rmn(sn(x,y»

 

(2-12)

A threshold operation is performed to remove small arbitrary noise pixels in the

stretched MAX image. The threshold operation is expressed as:

S", M X’(x, y) = { S", M X(x, y), if S", M X(x, y) 2 threshold (2-13)

0, otherwise
The threshold value can be chosen by Otsu’s method [22] which finds a value that
separates the pixel intensity distribution of two classes. The threshold value minimizes

within-class variance in equation (2-14) and at the same time maximizes between-class

variance.

0.50) = c110) 0,20) + qzm 07%) (2-14)

28

where own) is the within-class variance, 0'] (t) and 020) are variances of each class, q1(t)

is the probability that a pixel has an intensity value less than or equal to t, and q 2(t) is the

probability that a pixel has an intensity value greater than t. The value t can be found by a
sequential search through all possible values or through a recursive relationship described
in [20]. The Otsu’s threshold values for typical combined difference MOI images was
seen to lie between 50 and 80.

The resultant images after each post-processing step are shown in Figure 2.6. The
overall motion-based filtering algorithm summarized in the flowchart in Figure 2.7 is

applied to the sequence of images to generate the sequence of filtered images. These

results are presented next.

29

 

(C)

Figure 2.6: (a) Combined difference image (b) after stretching (c) after thresholding.

2.4.5 Overall Algorithm of MBF

The overall procedure for motion-based filtering is shown below and in Figure 2.7.

l. LetD1 =I,,_1-I,,, S1 = D1,i= 2.

2. Calculate D,- = 1,,_,--1,,

3. Calculate S,- with OR operation, if binary, or MAX operation, if grayscale, on SH and
Di.

4. For dynamic selection of w, calculate A i, which is the area of D,-,

30

5. Ifi<w and i<n-1, or A,- > AH, then i = i+1 and go to step 2.
6. Post-processing

Input Image, 1,,

I

i: 2, 51 = D], D] = ln-I-ln

I

N Di :lln-i-In +—_—

yes 1 no

Si = MAX(Si-I,Di) Si = 0R(Si-IDi)

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

l J

 

 

Compute A;

 

 

 

 

 

 

 

 

 

 

 

Stretch

v

Threshold

 

 

 

 

 

 

Output Image, 5,, = 5;

Figure 2.7: Algorithm for motion-based filtering. w denotes number of difference

images.

31

2.5 Results of MBF
Figure 2.8(e) shows the resulting gray scale image obtained by applying the
motion-based filtering algorithm to MOI images in Figure 2.8(a)-(d). Three difference
images are obtained by subtracting the image in Figure 2.8(d) flom those in Figure
2.8(a)-(c), and then MAX operation is performed on the difference images followed by
post-processing. As seen in Figure 2.8, background noise is reduced to nearly zero while

moving objects are retained in their original shape.

 

(d) (6)

Figure 2.8: (a),(b),(c),(d) MOI images in time sequence. Rivets are moving to the left
while sensor is moving to the right; (e) Motion-based filtered image of the original

image (d).

The seam, across the width of the image, is almost lost in the filtered image in

Figure 2.8(e). There are two reasons for this. First, the original image of the seam does

32

not have enough contrast to be separated flom the background. Second, if an object
extends across the image and parallel to the scan direction, it appears stationary in the

video sequence and hence is lost in the filtered image as illustrated in Figure 2.9.

 

(d) (6)

Figure 2.9: (a),(b),(c),(d) Synthetic MOI images in time sequence with uni-
directional scan. Rivets are moving to the left while sensor is moving to the right; (e)

Motion-based filtered image with uni-directional scan.

One possible way to overcome this problem is to perform a scan orthogonal to the
seam or a multidirectional scan so that the seam does not appear stationary. Figure 2.10
shows a synthetic video sequence, obtained using a scan path that is sinusoidal about the

seam direction.

33

 

(d) (e)

Figure 2.10: (a),(b),(c),(d) Synthetic MOI images in time sequence, multi-directional
scan. Rivets are moving to the left while sensor is moving to the right with small
sinusoidal up down motion; (e) Motion-based filtered image with multi—directional

scan.

The effectiveness of the performance of the motion-based filtering was measured
quantitatively with the raw and filtered gray scale images using image contrast as a
criterion. Contrast is one of measures of object visibility [23]. Image contrast is defined

as:

= fl (2-15)
Cb
where c0 and Cb are the average intensity of object and background, respectively

determined flom a sample window of each region. However equation (2-15) cannot be

34

used if the background intensity is equal to zero or the object intensity is lower than the

background intensity. Therefore a modified contrast function is defined as:

CO _Cbl

C' =l (2-16)

 

The contrast of an object will be 1 in an ideal case when the difference of intensities
between the object and background is 255. Three images are selected to measure the

effectiveness of the MBF in terms of contrast as in Figure 2.11.

35

 

(e) (D

Figure 2.11: Results of MBF. Rectangular box shows selected areas for measuring

contrast. (a)(b) Image 1, (c)(d) Image 2, (e)(f) Image 3.

A rectangular box of size 30x30 pixels is selected in background and object for each
image to measure the contrast. The measured contrasts of objects in M01 images in

Figure 2.11, before and after filtering, are shown in Table 2.1.

36

Table 2-1: Contrast of objects in M01 images before and after filtering.

 

 

 

 

 

 

 

 

 

Images Contrast

Original 0. 14
Image 1

Filtered 0.71

Original 0.17
Image2

Filtered 0.62

Original 0. 19
Image3

Filtered 0.49

Original 0.17
Average

Filtered 0.61

 

 

 

 

 

2.6 Conclusion
In this chapter, a new filtering method for enhanced MOI images was presented.
The MBF separates moving parts flom stationary parts in a sequence of images. The
proposed method is shown to remove effectively the serpentine background noise
associated with domain in M01 images. This can help the human operator interpret the
M01 data more accurately. The result of MBF also provides a binary image output for
automated rivet inspection. The following chapter studies automated aircraft inspection

methods based on the MB-filtered MOI images.

37

Chapter 3: Algorithms for An Automated Rivet Inspection System With

MOI

3.1 Introduction

This chapter introduces signal processing algorithms for automated rivet
inspection in airframe structures. Among the various types of problems in aircraft
inspection, the detection of cracks under rivets is one of the major challenges facing the
aviation industry. In the inspection of the aircraft skin, the large number of rivets makes
the manual inspection time consuming and laborious. The human operator needs to scan a
large area around every rivet and analyze the acquired images. In practice, the majority of
the rivets are good and only a few rivets are defective. This causes the human operator to
become accustomed to normal rivets and to tend to ignore defective rivets. An automated
rivet inspection system should increase speed, accuracy, consistency and hence reliability ‘
of the inspection. The rivet inspection algorithm can be used in manual scanning to assist
the human operator or in a fully automated robot inspection system. The rivet inspection
algorithm developed in this thesis comprises three major steps, namely, preprocessing,
rivet detection, and rivet classification. This chapter explains the implementation details

in each stage of the inspection algorithm and presents the inspection results on MOI data.

3.2 M01 Images
Some examples of M01 images of cracks and corrosion are shown in this section.
Distinctive features of the defect images will be used as the basis for classification.

Sample MOI images of cracks around rivets are shown in Figure 3.1.

38

 

(c)

Figure 3.1: (a) 'IVvo normal rivets (b) Right rivet has a radial crack (c) A crack
between two rivets.

Figure 3.2 and 3.3 show an example of an MOI image of a crack around a seam.

 

(a) (b)

Figure 3.2: (a) Seam and two normal rivets (b) A crack along the seam

39

 

(C) ((1)

Figure 3.3: (a) (b) Normal seam and two rivets (c) ((1) Crack along the seam.

In Figure 3.2, the crack is seen as a large object (indicated by an arrow), whereas in
Figure 3.3, the crack is seen as a breakage in the seam. This difference is due to the
geometrical configurations of the crack and seam. Figure 3.4 presents sample MOI

images of a corrosion dome.

4O

 

(c) (d)

Figure 3.4: Sequence of images with corrosion dome. Corrosion is moving from up

left to down right, as the MO] sensor is moving.

The serpentine pattern noise is particularly dominant in the corrosion image, and
severely degrades the inspection capability of M01. As observed in the previous example,
MOI images of defects do not represent the exact shape of defects, and the defect images
appear differently depending on the airflame structure around them. The defect images
are also validated by the finite element modeling technique [24]. Another challenge for
the MOI is that the images are affected by variations in the scanning procedure as shown

in Figure 3.5.

41

 

(C)

Figure 3.5: Images of normal objects are shown differently because of the wobbling

of the M01 instrument.

The sensor wobble causes a variation in the distance between test sample and M01,
which in turn results in the induced magnetic fields and hence the M01 images flom
same objects. Such variation in the data flom the same object presents a significant
challenge to the development of automated rivet recognition and classification. The

variation of M01 images with defective rivets is shown in Figure 3.6.

42

 

(a) (b)

Figure 3.6: Images of defective objects are shown differently because of the

wobbling of the MOI instrument.

Assuming some prior knowledge about the test geometry and defects images, algorithms

for automated rivet inspection will be discussed in the next section.

3.3 Overall Approach of Automated Rivet Inspection
With the wide variation in shapes of defects in M01 images, different inspection
algorithms need to be developed for each type of defect. Among the various problems,
rivet inspection is one of the most important because rivets are common sites where
cracks can develop. The cracks around a rivet are classified into two categories, radial
and circumferential, according to its configuration relative to the rivet as described in

Figure 3.7.

43

(a) (b)

(C) (d)

Figure 3.7 : (a) A schematic rivet with a radial crack (b) A schematic MOI image of
the rivet with a radial crack (c) A schematic rivet with a circumferential crack (b) A

schematic MOI image of the rivet with a circumferential crack.

A rivet inspection algorithm for detecting radial cracks around a rivet is discussed in the
remainder of this chapter. A typical rivet has roughly a circular or oval shape, while it has

an additional protruding blob when there is a radial crack as shown in Figure 3.8.

[‘3 - c.
I
(b) (d)

(a) (C)
Figure 3.8: Typical rivet images in M01 inspection after MBF filtering: (a)(b)

     

normal rivets (c)(d) defective rivets.

The overall approach of automated rivet inspection is a three-step procedure as depicted

in Figure 3.9.

44

 

 

 

Automated rivet inspection

i

Motion-based Filtering

4

Rivet detection

iv

Rivet classification

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 3.9: Overall approach of automated rivet inspection algorithm.

The raw image obtained by M01 is fed to the MBF module described in Chapter 2 to
generate a binary image. Ideally, the binary image is devoid of background noise and
contains all information about objects in the raw image such as rivets, defects, and
corrosion. The binary image result is used in the subsequent rivet detection and rivet
classification modules. MBF was discussed in Chapter 2, so that the following sections

discuss issues related to rivet detection and rivet classification.

3.3.1 Automated Rivet Detection
We introduce two different approaches for rivet detection. The first approach is
based on circular Hough transformation and the second approach is based on a
morphological image processing method. Both rivet detection methods identify all
circular objects in the images. Both algorithms detects a circle that encloses the rivet but

not the blob associated with the defect.

45

3.3.2 Hough Transformation Technique
The Hough transformation method was originally proposed to detect straight lines
in a given image. Later, the circular Hough transformation was proposed as an efficient
method for detecting circles in an image [25] [26]. The rivet detection based on Hough

transformation is composed of four steps as follows.

i) The MOI images processed with MBF has zero values for background and non-zero
values for object. Therefore, the edge detection is performed by detecting obj ect-

background discontinuities as:

I'(x,y)=l, ifI(x,y)¢0and{I(x—l,y)=00r1(x+l,y)=Oor
I(x,y-—l)=00rI(x,y+l)=0} (3.1)

0, otherwise
where, I (x, y) is the input image and I '(x, y) is the edge image.

ii) Gradient and magnitude of gradient for each edge pixel is computed using the Sobel

operator for the edge pixels detected in step i):

—1 o 1 —1—2 —1
Sobel, = -2 o 2, Sobel); o 0 o (3.2)
—1 01 1 2 1

where, Sobel”, and Sobel}, are the Sobel operators for x-axis and y-axis respectively.

46

gx= Sobel . .xI

x,1,j r+i—2,c+j—2

l=l j=l
i=3j=3
g)’ = . l . ISOberJJ XIr+i—2,c+j—2 (3'3)
1: J:
2 2
g = gx + g)

gx and g y are the gradients with respect to x-axis and y-axis respectively, g is the

gradient magnitude, and ‘x’ represents convolution operation.

iii) Transform the coordinates of the edge pixel to the three-dimensional (x, y, radius)

accumulator. The possible center coordinate is calculated for predefined range of radius r

as:

xc = x-rc056 (0056 = gx /g)
(3.4)

yc =y—rsin6 (sin6=gy/g)

where, xc and yc are the possible center coordinates and r is the given radius. Based on

equation 3.4, the accumulator collects the votes for candidate circles as:

A(xc,yc,r)=A(xc,yC,r)+l, for xczx—rcosfl (cost9=gx/g)
(3.5)

yc =y-rsinl9 (sinazgy/g)

47

 

where, A(xc, yc,r) is the 3-dimensional accumulator, x and y are the coordinates of edge

pixels in the input image. The schematic of the voting and accumulator is shown in

Figure 3.10.

 

  

 

 

 

 

 

A
height
V AIMS
>
width
(3) (b)

Figure 3.10: Schematic of voting in 3-dimensional accumulator in Hough
transformation. (a) Each edge votes for its center coordinate (b) Accumulator

collects the votes.

iv) Determine the Hough circle of the rivet by choosing peak values in the accumulator

as:

if A(xc,yc,rc)2T, then [(xc,yc) is center of a circle with radius rc

(3.6)
else, skip A(xc , yc a re)

where, T is the threshold value. If no accumulator element has a value larger than of

equal to the threshold T, then the given image is decided not to have a circular object.

48

For correct results of Hough transformation, some parameters need to be
predefined, such as range of r (ra S r S rb) and threshold value of the votes T. In our rivet
detection algorithm, the range of radius is between 15 and 45 pixels and the threshold

value is chosen as 150 experimentally. Typical results of the Hough transformation—based

rivet detection are shown in Figure 3.11.

 

(C) (d)

Figure 3.11: (a) and (c) Original MOI images (b) and (d) Rivet detection by Hough

transformation.

3.3.3 Morphological Operation Technique
In morphological operation-based rivet detection, the MBF filtered image is first

segmented using connected component analysis to determine the bounding rectangle of

49

each object. Object broken by a few pixels’ distance need to be considered as a connected
object, so that a closing operation is performed. A closing operation is a dilation
operation followed by erosion operation. The dilation of binary image B by structuring

element S, denoted by B 60 S, is defined by [20]:

BeS=bteJBsb (3.5)

where, Sb represents a translation of set of pixels S by a position vector b. The erosion,

denoted by B e S is defined by [20]:

BGS={b|b+seB VseS} (3.6)

Connected component analysis finds objects that are surrounded by the background

pixels. Two connected components and their bounding rectangles are shown in Figure

3.12.

 

 

 

 

 

 

. :e
.34..)
yak. .
«(25$
- ~ ,.' r . , H w
73;; ' Rafi. 1.x“: «1455::
$12.! “44$: 1‘. "93“!“
' 1.4," 1 L’."¢ :
2.; ,,.::\.'.‘{.‘ (3.1:;- .3
:-° a) “_ a .. . ,
”32$? Eli-‘99??-
~ "if-‘3" 9.933 .:
i‘a‘t; "’ ,{E"'J"I~‘:’
'3" ' 3.391 {La .
on ‘-
9,31,31-
. [cw--
'1,.~.§fl9
site:- .g:

 

 

 

 

 

 

 

 

 

 

Figure 3.12: Schematic of connected components and bounding rectangles in an

image.

50

An examination of the width, height, and aspect ratio of the bounding rectangle can
quickly determine potential segments of a rivet. The initial center and radius c and r are

calculated flom the bounding rectangle by:

c=I(cx,cy)
cx =(J1:1 +x2)/2
Cy =(yl +y2)/2

r=min(cx _x1,x2 _cx,y1-cyacy _y2)

(3.7)

Then, morphological erosion operation is performed. The erosion operation is
performed in each bounding rectangle with a structuring element of size W. The size Wis
chosen as the largest value for which the eroded image is not empty. The structuring

element is described in Figure 3.13.

   

(a) (b)

Figure 3.13: Morphology of structuring element. (a) Square shape (b) Circular
shape.

Either square or circular structuring elements can be used for rivet detection. The square-
shaped structuring element is easier to implement than circular structuring elements. The

unbiased center, c' and radius, r' are calculated after the erosion according to:

51

c'=I(cx',cy')
cx'=(xl '+x2')/2
c,..'=(y,'+y2)/2

r' = min(cx '-xl,x2 —cx ',yl —cy ',cy '— yz)

(3.6)

The intermediate results of this procedure are shown in Figure 3.14.

X] x2 xl’ xz'

     

.5 y2
(a) (b) (C)

Figure 3.14: Example results of intermediate steps of morphological operation for
rivet detection with intermediate variables: (a) initial rivet image (b) after iterative

erosion operation (c) after rivet detection.

The procedure for rivet detection using morphological operation is summarized in Figure

3.15.

52

 

Rivet detection via morphological
operation

v

Closing

+

Segment out connected component
until there is no more object ‘—

 

 

 

 

 

 

 

 

 

+

no

   
   
   
 

wl<width<w2 &
h1<height<h2 &
width/height<tl

 

 

Obtain center and radius c. r

v

Erode rivet with a circle of radius r

Area (rivet) = 0
V no

Obtain center and radius c’. r’

 

 

 

 

 

     

yes, r (— r-l

 

 

 

 

Figure 3.15: Rivet detection algorithm using morphological operation.

The variables w 1, W2, h 1, h 2, t, in the rivet detection algorithm are chosen flom values of

typical rivet dimensions. Typical results of the morphological operation-based rivet

detection are shown in Figure 3.16.

53

 

(C) ((1)

Figure 3.16: (a) and (c) Raw images. (b) and (d) Rivet detection by morphological

operation.

3.4 Automated Rivet Classification
The final step in this system is the automated classification of a rivet as ‘good’ or
‘bad’ (with radial crack). Two approaches for rivet classification are developed. The first
method uses a two-pass Hough transformation while the second approach is based on a

Bayesian classifier with appropriate feature.

3.4.1 Two-pass Hough Transformation Classifier
In this approach, a second Hough transformation is applied to the set of edges that

do not belong to the circles detected in the first Hough transformation. In the case when a

54

rivet has a crack, the edges associated with the crack will be detected as a small circle
whose center lies outside the rivet boundary. Consequently, the detection of a circle in the
second Hough transformation implies that the rivet is defective. For the second Hough
transformation, the range of radius is chosen between 3 and 30 and the threshold value is
chosen as 50. The flow chart of the Two-pass Hough transformation algorithm is shown

in Figure 3.17.

 

Second Hough transformation
on E ’

no A circle is detected

yes

 

 

 

 

 

Center coordinate

 

 

* is inside first circle yes

 

 

Good rivet Bad rivet

 

 

 

 

 

 

Figure 3.17: Second pass Hough transformation for rivet classification. E ’
represents the image of rivet edge that is outside of the circle detected in the first

Hough transformation.

A typical result of implementing the two-pass Hough transformation classifier is

presented in Figure 3.18 and 3.19.

55

   
 

y Maifiihhynfiw

    

are» ., Jazz-1;; étaMaltati'Sfifl-W‘Ii
.- 45:31?

(a) (b)

 

(C)

Figure 3.18: Two-pass Hough transformation method: (a) raw image (b) after first
Hough transformation (c) after second Hough transformation. Right rivet is

classified as defective.

56

 

 

(C)

Figure 3.19: Two-pass Hough transformation method: (a) raw image (b) after first
Hough transformation (c) after second Hough transformation. Right detected is

classified as defective.

The small circle close to the large circle indicates that the rivet is defective in Figure 3.18

(c) and Figure 3.19 (0).

3.4.2 Bayesian Classifier
Let (01 represent the class of ‘good’ rivets and (02 represents the class of

‘defective’ rivets. The Bayesian decision rule [27] makes a decision based on the

statistical model according to the following rule.

57

rivet e a)1 if [3(a)1 Ix) > [3(a)2 Ix); otherwise decide (02 (3.7)

where, PM), |x) and P(a)3|x) represents posterior probability, and x represents the M01
rivet data. The posterior probability [3(a)] Ix) is the probability that the data x belongs to

class a), given data x. From Bayes rule we have:

p(x l (01-)P(a)i)
p(x)

 

Pm, l x) = (3.8)

where, P(x| (0,) is the conditional probability density function of class (a), P( (OJ is the prior

probability, and p(x) is normalization factor to make the sum of posterior probability
equal to 1. Using Bayes rule, the decision rule with probability density functions and

prior probabilities is obtained as:

rivet e (01 if p(x|a)l)P(a)])> p(x|a)2)P(a)2); otherwise decide (02 (3.9)

where, p(-) represents a probability density function. Assuming equal prior probability, a

decision rule using the conditional probability density functions is obtained as:

rivet e (0' if p(x|w]) > p(x|o)2); otherwise decide (02 (3.10)

58

The probability density function p(x l (0,.) , assumed to be Gaussian, is expressed as in Eq.

2

(3.11) for the univariate case with the estimated mean pi and variance 01' .

2
pglwnzfizexp 1&1] (3.11)

The multivariate probability density function is obtained with estimated mean vector

17,- and covariance matrix it when the dimension of the feature vector is d can be

expressed as:

put I (0,.) = d},
((27)

 

exv[—l(i-fi,-)’E;‘(i-fl,)] (3-12)
:_ I1/2 2

I

3.4.3 Feature Selection

For the Bayesian rivet classifier, a variable d is defined as:

 

d:\/()c—c,c)2+(y——cy)2 —r (3.13)

where, (x, y) is the coordinate of a point on the edge outside of the circle obtained by the

rivet detection algorithm. A set of d values are obtained for each edge pixel (x, y) value as:

D={di |i=l,...,ne}, ne is the number of edge pixels (3.14)

The negative values of d are discarded from consideration because they do not give

information about the defect. A feature f is then computed as:

59

f = n’h largest element in D

(3.14)

n=Lt><ne_|

where, 0 < t < 1, ne is the total number of elements in D. Values larger than n are trimmed
out flom D to reduce the effect of noise. The value oft is chosen as 0.1 in our experiment.
Feature f represents a quantitative measure of the degree of circularity of the rivet. The

graphical representation of the feature f is shown in Figure 3.20.

 

(a) (b)

Figure 3.20: Graphical representation of the feature f for Bayesian classifier (a)
Hough Transformation-based rivet detection (b) Morphological operation-based

rivet detection.

3.4.4 Distributions of the selected feature
Our sample MOI images for the classification test contains 124 defective rivet
images and 169 normal rivet images. However defective rivet images are obtained from
only two defective rivets and normal images are obtained from 6 different rivets.

Therefore, inspecting the correlation of the feature f flom those images is a crucial part

60

for the validity of the rivet classification test. The variations of MOI images flom the
same object are shown graphically in Section 3.2. The variations of M01 images are also
shown by the distribution of feature f in Figure 3.21. The feature f flom normal and
defective rivets shows unimodal distributions in Figure 3.21. The distributions of feature f
allow the rivet classification test to be valid with the MOI rivet images, which are

captured multiple times flom the same object.

61

 

distribution of f in Hough transformation-based feature extraction

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

3O — q . . .
25 . . l D"°m‘a_' ——
H 20 . rm ,, ” 'l Tiff“:
g 15 ______ s1 . L * ewe:
10 ..n a _
l C i
s ,2. a
(a)
distribution of f in Morphological operation-based feature extraction 1
k 25 q i 1 Dno
Idefective
5
8

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

(b)

Figure 3.21: The distributions of feature f from (a) Hough transformation-based

rivet detection (b) Morphological operation-based rivet detection.

62

3.5 Results of Automated Rivet Inspection
A sample sequence of M01 images recorded during aircraft inspection is used to
test the rivet inspection algorithm. The MOI data contain a total of 293 rivet images with
124 defective rivet images and 169 normal rivet images. Both Hough transformation-
based method and Morphological operation-based method show excellent results for
automated rivet detection on the given data set and these results are summarized in Table

3.1.

Table 3-1: Accuracy of two rivet detection algorithms.

 

 

 

Detection algorithm Rivet detection
Hough Transformation 99% (291/293)
Morphological Operation 99% (291/293)

 

 

 

 

The automated rivet inspection is tested on a set of M01 images which contain 124
defective and 169 normal rivet images in total. Training data were selected with 30
defective and 30 normal rivet images. The remaining data containg 94 defective rivets
and 139 normal rivets were used for testing the performance. For the two-pass Hough
transformation classifier, entire rivet images are used for the heuristic search for optimal
parameters and test with same set of images. For Bayesian classifier, 100 training data
sets were selected randomly and the result presented in Table 3.2 is the average of 100

test results. The performance of the automated rivet inspection algorithm is shown in

Table3.2.

63

Table 3-2: Inspection accuracies of three different inspection algorithms.

 

 

 

 

Inspection algorithm Accuracy
Two-pass Hough 52 %
Hough-Bayes 95 %
Morph-Bayes 98 °/o

 

 

 

 

From the results shown in Table 3.2, the Bayesian classifier is seen to outperform the
Hough classifier. Between the two different rivet detection methods, morphological
operation-based rivet detection enables the following Bayesian classifier to perform
better. The superiority of the morphological operation-based rivet detection is in the
higher inter-class variance for the feature f between a defective rivet image and normal
rivet image with oval shape as illustrated in Figure 3.19. Even though the overall inter-
class variance is larger with Hough transformation-based rivet detection method as
shown in Figure 3.20, the classification between circular normal rivet image and
defective rivet image is a trivial problem in many cases.

The detected circles enclosing the rivet using the two rivet detection algorithms
differ when the rivet has an oval shape. The Hough transformation-based method finds a
circle that fits the edge of the rivet image as much as possible, while the morphological

operation-based method finds a circle that is aligned at the true center of the rivet image.

64

3.6 Conclusion

This chapter addressed the problem of automated processing of M01 data
obtained in aircraft skin inspection. Two rivet detection algorithms and two rivet
classification algorithms were introduced. The combinations of rivet detection and rivet
classification algorithms have been evaluated with the given MOI data. The off-line test
results show that the Bayesian classifier is superior to the two-pass Hough classifier. The
morphological operation-based rivet detection method is also shown to be beneficial for
the higher accuracy of the subsequent Bayesian classifier. These results have to be
regarded with caution: more defective rivets need to be imaged for more comprehensive
testing. The real-time implementation issues of the automated rivet inspection are

discussed in the following chapter.

65

Chapter 4: Real-Time Implementation Of Automated Rivet Inspection

System

4.1 Introduction

So far, we have discussed various algorithms for automated image enhancement
via Motion-based Filtering, rivet detection, and classification. In practice when the
aircraft skin is inspected, the data interpretation and rivet classification need to be
performed in real-time. The MOI instrument generates about 26 images per second [7],
so that the image processing system should be able to process a maximum of 26 images
per second for real—time inspection capability. The initial inspection algorithm was first
implemented and tested off-line on a general-purpose computer using MATLAB. Two
approaches were considered for enabling real-time operation of the system. The first
approach for making the software more efficient and faster was based on improvements
to the algorithms. The second approach relies on the use of a Digital Signal Processing
(DSP) board. The algorithm was translated into a C++ program. Further optimizations of
each component of the automated rivet inspection system for real-time processing

capability are discussed in the following sections.

4.2 Optimization of Motion-Based Filtering
The MBF algorithm, tested on flames of video clips recorded during the M01
inspection, was optimized earlier to provide the best quality of processed images.
However the efficiency of the algorithm in terms of time and memory was not considered

in its early development stage. The original algorithm for MBF was not fast enough for

66

real-time inspection. MOI instrument generates 26 images per second, which allows
about 38 milliseconds of processing time per image if all 26 images are to be processed.
However, in practice the data is over sampled, and hence the number of images per
second can be lower for real-time image processing. On reviewing the movie clips of
M01 images and subsampling, it was seen that three images per second was the minimum
processing speed required for real-time inspection. Therefore the lower limit of the 7
processing speed is based on three images per second, which translates to 333
milliseconds of processing time per image. Table 1 shows the initial processing time of

each step of MBF in Matlab and C++ programs.

 

Table 4-1: Processing time for filtering one image with MBF algorithm using ten

past images.

 

 

 

 

 

 

 

 

 

 

 

 

MATLAB C++
Capture - -
RGB to gray 20 ms (x11) 20 ms (x11)
subtraction 20 ms (x10) 1 ms (x10)
Max 1 ms (x10) 1 ms (x10)
Threshold 10 ms 1 ms
Median Filter (5 x 5) 50 ms 220 ms
Gray level stretch 15 ms 1 ms
Total 505 ms 462 ms

 

 

* Measured with 2GHz CPU, 512 RAM PC
* Image size: 240 x 320

67

The C++ version outperforms the Matlab version in flame subtraction, threshold, and
stretching operations. However Matlab is seen to outperform the C++ version in the
median filter operation, indicating the Matlab algorithm for median filter operation is
more efficient. This indicates the need for optimization in the C++ version. A systematic

optimization of each step of the algorithm is explained in the following sections.

4.2.1 Frame Grabbing
Frame grabbing is the first step of the image processing in M01 inspection. In our
configuration, a PCI slot based flame grabber, which gets input signal from a VCR, or a
USB port based web camera are used to capture frames. The time taken for flame capture
depends on the clock speed of the computer system and the overall architecture of the
hardware system. As tested with the 2 GHz CPU PC with 512 Mbytes RAM, the average
capture time varies flom 15 to 60 ms. This operation mostly depends on the performance

of the hardware, with no scope for software optimization.

4.2.2 RGB to Grayscale conversion
Almost all flame capture devices use a color image format of 16-bit or 24-bit. A
camera device which provide gray scale tends to be more expensive than the one with
color feature. The color image needs much more time for processing than a grayscale
image, because each of the red, green, and blue values need to be read and processed
separately. For the purpose of defect detection using MOI, a grayscale image with 0~255
intensity values is sufficient. Therefore converting the color image to gray scale before

performing any image processing is needed to reduce processing time. Our experiment

68

revealed that processing 8-bit gray scale image is about 7 times faster than processing 16-
bit color with respect to a simple 5 by 5 average filtering. However, converting a color
image to gray scale is also a time intensive operation, because the color to gray
conversion is performed on every pixel, which amounts to 76800 operations per image of

size 320 by 240 pixels. The color conversion formula is described as:

Gray = Redx0.299+Greenx0.587+Bluex0.114 (4-1)

An optimized color image conversion method was designed by building a color
conversion table so that color conversion is performed by a simple table lookup operation.
Given blue (B), green (G), and red (R) values for a pixel, the key for the table is

determined by:

R<<16||G<<8||B (M)

where, << represents a bit wise shifting operation and l] represents a bit wise OR
operation. For a 16-bit color image, there are 216 different key values and each entry has a
value of size 8 for the gray scale, which makes a total 2'9 bits (64 Kbytes). Using table
lookup for color conversion is very fast and the memory requirement of 64Kbytes is not
high. The measured time for the image conversion flom 16-bit to 8-bit is now only 2ms,

which is only 1/10 of the previous conversion method as described below.

69

Table 4-2: Time for converting color image to gray scale.

 

 

Normal Using Conversion Table bulld Space
. trme (off- .
Convers1on Table . requrred
lrne)
Converting l6-bit
color image to 8-bit 20~23 ms l~2 ms 5 ms 64 Kbytes
gray scale

 

 

 

 

 

 

 

* Measured with 2GHz CPU, 512 RAM PC
* Image size: 240 x 320

4.2.3 Frame subtraction and Combining
In the original algorithm of MBF [28], the combined difference image is obtained
by subtracting the current image flom w previous images and performing an OR

operation on difference images as:

Di(xay) : In_1'(x’y)—In(xay)
(4'3)
Sn (x9 y) : Ig§W{Di(x’ y)}

where, Di(x, y) is the ith difference image, 1,,(x, y) is the nth image, and Sn(x, y) is the nth
combined difference image. In this case there are w subtractions and w-l OR operations,
or 2x(w-l) operations total. However, the number of operations to get a combined
difference image can be reduced flom 2x(w-l) to w, if the OR operation is performed

first with w previous images and the current image is subtracted flom the combined

image as:

70

 

Sn(x9y) : 0R {In—[(x,y)I

lSiSw
(4-4)
5110(1)”) : 312(x9y) " 1,1083”)

This method is about twice as fast as the previous method.

4.2.4 Thresholding
The threshold operation can be integrated into the subtraction operation. When
subtraction is performed between two corresponding pixels, a zero value is assigned to

the destination pixel if the difference is less than the threshold as:

J(x, y) ={ [1(x, y) — I2 (x, y) if 11(x, y) — I2 (x, y) 2 threshold
(4-5)

0 otherwise

4.2.5 Median Filtering
Three median filtering algorithms, quick median search, moving median with
sorting, and moving median with histogram [29], were tested and moving median with
histogram was proven to be the fastest. The median filtering operation obtains the median
value flom the set of elements in each kernel window. Let the set S={i1,i2, ..., in} be the

elements in a wxw window. Then the median value is defined as:

Median = km largest element in S

(4—6)

71

 

k = 21, n is an odd number

In quick sort, a pivot element is arbitrarily chosen flom S and partitioning is

performed to satisfy the following conditions:

Vi such that i 6P], i < pivot
(4-7)

Vi such that i 6P2, i > pivot

where, P 1 and P2 are partitions. The partitioning operation is performed on P 1 and P2

recursively until the number of elements of all partitions are one. All partitions are
combined at the last step to obtain the sorted list of S. In modified quick sort, each.
partitioning is performed on only one of two partitions, which contains the k’h largest
element. The operation stops when the pivot is k’h largest, in which case the pivot is the
median value. The modified quick sort-based median search is faster because it
minimizes the sorting operation: an) whereas quicksort is @(n log n).

In moving median with sort, the median value is obtained at the first kernel
window by sorting S. After moving the window to the next position, w elements are
removed and w elements are added to S, thus the sorting operation becomes very fast by
removing the overlap of pixels.

In moving median with histogram, a histogram is built and operated during the
median filter. The histogram holds the counts of values in current window, so that the

median value is found by counting the histograms flom O to 255: the bin with the

72

accumulated counts larger than or equal to k becomes the median. This median bin is
remembered at each move of window, and as adding and removing elements flom the
histogram, the previous median bin is adjusted and the median is quickly searched. More
formal explanation of the moving median with histogram is described below. A

histogram of size 256 is defined as:

H(i) = 0, 0 s i s 255 (4-8)

The moving median with histogram initializes variables as:

H(I(x+i,y+i))=H(I(x+i,y+i))+1, —w<i<w, —w<j<w

j
M current = arg min H (i) Z k (4—9)
' J i=0
Mcurrent
Nless : HO)

1:

where, [(x, y) is the original image, Mame," is the median value in current window and k

is the variable defined in equation (4-5). As the window is moved, w elements are

removed flom H and the variables are updated as:

KS0.) < Mcurrent: the" Nless : Nless '1, H(i) = H(i) — 1
(4-10)
Else H(i) = H(l) — 1

where, S(i) represents an element in the given window. The variables are updated

similarly as w elements are added as:

73

 

”S(i) < Mcurrent, the” Nless = Nless +1, H(i) : [1(1) + 1
(4-11)
Else H(i) = H(i) + 1

After updating the variables, the median value is obtained quickly by the following

expression.
Mcurrent
Iflvless 2k, the" Mcurrent = arg max (Nless '- Z H(i)) 2- k 'j SMcurrent
J i= j

(4-12)

I
Else Mcurrent = arg min {(Nless + 2 H(i)) Z k} » Mcurrent —<j
J

i=Mcurrent

The measured time for the three median filtering algorithms are shown in Table 4.3 and

the moving median with histogram method is the fastest.

Table 4-3: Processing time of median filtering with various algorithms.

 

 

 

 

 

 

Algorithm Kernel size 5 by 5 7 by 7
Quick Median search 200 ~ 250 ms 400~450 ms
Moving Median with Sort 100~150 ms 150~200 ms
Moving Median with Histogram 20~25 ms 20~40 ms

 

 

 

* Measured with 2GHz CPU, 512 RAM PC
* Image size: 240 x 320

74

 

Figure 4.1 shows the effect of median filtering on a combined difference image with
severe noise. Median filtering is more important for improving the image quality as the

number of difference images increases.

   

(a) (b)

Figure 4.1: Effect of median filtering in MBF. (a) Before median filtering. (b) After

Median filtering with 5 by 5 kernel. Bright objects represent rivets and corrosion.

4.2.6 Contrast Stretching
Stretching is performed to enhance the contrast of objects in the filtered image.

The transformed image J(x, y) after stretching operation is denoted by:

_ x I(x,y) _
J(x,y)—255 —Max(1(x,y» (413)

For better contrast, the highest 1% of intensity values is trimmed in getting the maximal
intensity value. Rather than doing the full search, a histogram method can be used. The

modified stretching operation combined with histogram searching is expressed as:

75

1(x, y)

 

J(x, y) = 255 x
255
argmax[z H(i) 2 768]
N i=~

where H represents the histogram.

(4-14)

4.3 Results of the optimized MBF

The resulting execution time of the MBF algorithm after the optimization

described above is shown in Table 4.4.

Table 4-4: Measured time for each step of MBF before and after optimization.

 

 

 

 

 

 

 

 

 

 

 

Before optimization .After.
Intel 2GHz, C++ InfelltlznélI-zl:t12;+
Capture 20 ms Capture 20 ms
RGB to gray 20 ms (x11) RGB to gray 1 ms (x11)
Subtraction 1 ms (x10) Max 1 ms (x10)
Max 1 ms (7‘10) Subtraction 1 ms
Threshold 1 ms Threshold
Median Filter (5x5) 220 ms Median Filter (5x5) 25 ms
Gray level stretch 1 ms Gray level stretch 1 ms
Total 482 ms Total 68 ms

 

 

 

 

76

* Measured with 2GHz CPU, 512 RAM PC
* Image size: 240 x 320

The total execution of the optimized MBF algorithm is about 70 ms with subtraction
performed on 10 past images. Therefore, the optimized MBF algorithm is capable of

processing about fourteen images per second.

4.4 Optimizing Rivet Detection and Classification
The Hough transformation takes a few hundred milliseconds to execute, which
makes it unsuitable for real-time processing. On the contrary, the morphological
operation-based rivet detection takes only a few tens of milliseconds to execute. The

execution times of the two different rivet detection algorithms are shown below.

Table 4-5: Execution times of Hough transformation-based and morphological

operation-based rivet detection.

 

I L Hough transformation Morphological operation I

 

 

I Rivet detection I 400 ms 30 ms I

 

* Measured with 2GHz CPU, 512 RAM PC
* Image size: 240 x 320

Due to the fast execution time, morphological operation-based rivet detection is suited for
the real-time rivet inspection algorithm. Therefore, morphological rivet detection is
chosen as a suitable algorithm to be implemented in the real time rivet inspection system.
The performances of the two different rivet detection algorithm with respect to accuracy
were shown to be the same in Table 3.1 and 3.2. For rivet classification, the Bayesian
classifier is superior to the Hough transformation-based classifier with respect both to

execution time and accuracy. Thus the combination of morphological operation-based

77

 

rivet detection and Bayesian classifier is chosen in the real-time automated rivet

inspection system. Testing with more data is necessary to justify this analysis.

4.5 Personal Computer-Based Proof-of Concept System
The automated rivet inspection system was implemented on a PC in the C++
programming language to test its real-time capability. The experimental setup of the test

is shown in Figure 4.2.

 

 

'aw ...,. .
2:22:22

can:

     
 

Play on a I
R F b
ecord to VHS Video player rarne gra ber
r-based l
(a)

 

 

   

 

 

Collect MOI (b) Ire-r"
. Camera-based test
nnage Data

 

 

 

 

 

lr-r.
:23 $51
' , h m
r. 1... Secondary Web
Record to 3 E ,3 noise camera
movre file play movre

on a PC monitor

Figure 4.2: A proof-of-concept system for real-time automated rivet inspection.

78

 

As shown in Figure 4.2, two different proof-of concept systems were developed. The
major difference is the method of input of the image data: Figure 4.2 (a) uses a VCR and
Figure 4.2 (b) uses a camera. The VCR-based method is better than the camera-based

method because it does not involve a source of secondary noise.

4.6 Digital Signal Processor (DSP)-Based Prototype System
The automated rivet inspection algorithm is being implemented on a DSP board
platform. The program is developed in the flarnework of Code Composer Studio (CCS).
The CCS translates a C++-like program into an assembly program. The prototype system

is depicted in Figure 4.3.

 

”HI

      

— Iczis gggg
m

, , ,... coca

    

 

 

 

 

‘ Play on a
r" :1' Record to VHS Video player
Collects MOI
image Data
11—2.—
mi
: I DSP Board &
‘ . , Image Daughter card
5 . . ‘4
5‘ .‘

 

Programming with
Code Composer Studio
C++ 9 Assembly

 

Figure 4.3: A prototype system for the real-time automated rivet inspection system.

79

 

I'm-.1 an :5. gnu.

4.7 Results of Real-Time Automated Rivet Inspection System

The execution times of the automated rivet inspection processes are shown in Table 4.6.

Table 4-6: Execution time of the automated rivet inspection

 

 

 

 

 

 

Execution time 5..
MBF 68 ms
. Rivet detection 150 ms
(including preprocess) ..
Rivet classification 30 ms b
Total 249 ms I

 

 

 

 

* Measured with 2GHz CPU, 512 RAM PC
* Image size: 240 x 320

The results of the C++ program for automated rivet inspection is shown in Figure 4.4.
The program is capable of processing 3 ~ 5 images per second and shows a red box to

indicate that there is a defective rivet in current image.

80

Iramclleeond: 3

ram. "r—‘amflml

l: W. imam—V.

   

 

 

 

 

(a)

Alarming box .
IEZI E-El 1143;)

 

Flume Grub: 24.56
Rub to Gray: 2.31)
MBF: 99.07

 

   

 

(b)

Figure 4.4: Results of the C++ proof-of—concept system. (a) Raw and processed MOI
image without defect. (b) Raw and processed MOI image with defective rivet, the

raw image is boxed to highlight the defective rivet.

81

4.8 Conclusion
The firll rivet inspection algorithm, including motion-based filtering, rivet
detection and classification, was implemented and tested on a 2 GHz, 512 RAM PC. The
test showed capability of the system for processing 3 to 5 images per second. The rivet
inspection algorithm is being implemented on a DSP board, which is the target platform.
The implementation has to be adapted for the small memory of the DSP board. The
algorithm will also be optimized to fully utilize the fast image processing capabilities of

the DSP board.

82

Chapter 5: CONCLUSION AND FUTURE WORK

5.1 Conclusion
The Magneto-Optic Imager (MOI), while providing fast and easy nondestructive
inspection capabilities, provides two major challenges. First, the magneto-optic image is
severely contaminated with the serpentine pattern noise. Second, the large inspection area

of an aircraft body with thousands of rivets makes the manual inspection of M01 tedious

 

and prone to operator fatigue.

The serpentine pattern noise not only lowers the probability of detection (POD)

 

but also makes the inspection subjective. By analyzing the characteristics of magneto—
optic images, the Motion-Based Filtering (MBF) algorithm was developed, where only
moving objects in a sequence of images are extracted and stationary background is
suppressed by using a multiple flame subtraction method. The serpentine pattern noise is
removed by MBF due to its stationary nature in a sequence of images. The MBF greatly
improves the POD by removing background noise in the magneto-optic images. The
contrasts of M01 images are improved flom .17 to .61 on average by MBF operation.
MBF also affects the potential to build an automated inspection system based on MOI
because it is simpler and faster to process binary output images of MBF than the original
color images. With the binary output images of MBF processing, an automated inspection
algorithm was developed.

The automated inspection algorithm characterizes certain distinct features flom
defect images. The wide variance of features among various defects makes it necessary

for the inspection algorithm to be adaptive. The first automated inspection algorithm was

83

developed to inspect rivet images. Rivets are the key location from which a crack
originates. The inspection algorithm first detects a rivet in a magneto-optic image, obtains
features of the rivet, and then makes a decision based on the features. Rivet detection
algorithms, based on Hough transformation and morphological operations were
developed and evaluated. Both algorithms for rivet detection show promising results on
real magneto-optic image data, however, more images of defective rivets are needed for

convincing results.

.nm. hue-P. -v. ...u “-42.. ‘q

Two rivet classification algorithms were also developed for the purpose of ’1

identifying defective rivets, namely i) two-pass Hough transformation classifier and ii)

 

Bayesian classifier. The two-pass Hough transformation classifier applies Hough
transformation on the input image twice, first to detect the rivet and second to detect blob
like objects associated with a radial crack.The Bayesian classifier uses a feature value in
the image to differentiate defective rivets flom normal rivets. The feature represents the
length of a blob attached to a rivet image when there is a radial crack. These features
were collected from training data of defective and normal rivet images. The Bayesian
classifier obtains the feature value flom the test image and decides if it belongs to the
distribution of defective or normal rivet images, then classifies the rivet accordingly. The
Bayesian classifier outperforms the two-pass Hough classifier with respect to
classification accuracy. Between the two rivet detection algorithms, morphological
operation-based rivet detection is shown to provide higher accuracy for the following
Bayesian classifier. This result needs to be verified by testing with more images of

defective rivets.

84

A proof-of-concept system was designed to test the automated rivet inspection
algorithm with respect to its real-time capability. Each step of the automated rivet
inspection algorithm was optimized for real-time capability and higher inspection
accuracy. The final algorithm for automated rivet detection was composed of MBF,
morphological operation-based rivet detection, and Bayesian classifier. The proof-of-
concept system shows that the automated inspection algorithm is capable of processing 3
to 5 images per second, which can be considered near real time. This execution time is

expected to be much faster when the algorithm is implemented on a DSP board.

5.2 Future Work

The MBF algorithm is based on the assumption that the background noise of MOI
is static. While this is true in most MOI images, some MOI images show abrupt
background intensity changes, especially around a large object. This is believed to be due
to the large magnetic field around a large anomaly. Improving the MBF algorithm to
detect this static background change from a moving object in the image would make the
algorithm more robust. The background intensity change occurs in a fixed area and does
not move its centroid. Therefore, one of the possible approaches is to track the motion of
the an object in the image and differentiate it flom a static intensity change in the
background.

The automated inspection algorithm needs to be developed for types of defects
other than rivets, such as cracks around a seam and corrosion. After all algorithms are
developed for all types of defects, they should be combined for the automated aircraft

inspection system that is capable of inspecting various types of defects. In addition, the

85

 

inspection algorithm can use the temporal information of decisions. In MOI inspection,
an object is observed in more than one image, which generates a sequence of decisions
for an object. By observing the sequence of decisions, an erroneous decision due to a
perturbation of signal can be suppressed.

Finally, the automated rivet inspection needs to be implemented on the DSP
system. With the aid of Code Composure Studio (CCS) most of the C++ version program ’4'-
is expected to be reusable except for video capture and memory management routines.
The original algorithm needs to be optimized for the small memory of the DSP board.

The algorithm is also being optimized to fully utilize the specialized computing power of

 

the DSP board. The DSP version of the inspection system would be more convincing
evidence of the effectiveness of automated rivet inspection. The DSP board must be
customized to make the system smaller and integrated with the commercial MOI system
either as a portable inspection instrument or a component of a self— guiding robot which

represents the fully automated rivet inspection system.

86

BIBLIOGRAPHY

[l] B. Raj, T. J ayakumar, and M. Thavasimuthu, “Pratical Non-Destructive Testing,” 2nd
Ed. 2002

[2] K. G. Boving, Non-Destructive Evaluation Handbook. London: Butterworths, 1989.

[3] B. D. Cullity, “Elements of X-Ray Diflaction,” 2nd Ed., Addison-Wesley Publishing
Company, Inc.

[4] G. L. Fitzpatrick, D. K. Thome, R. L. Skaugest, E. Y. C. Shih, and W. C. L. Shih, f
“Novel Eddy Current Field Modulation of Magneto-Optic Garnet Films for Real-Time

Imaging of Fatigue Cracks and Hidden Corrosion, “ Physical Research Inc., Torrance,
California, 1993

[5] D. E. Bray and R. K. Stanley, Nondestructive Evaluation, New York, McGraw Hill I
1989.

 

[6] “Development of an Improved Magneto-Optic/Eddy-Current Imager,” Office of
Aviation Research, DOC/FAA/AR-97/37, Washington DC.

[7] Fitzpatrick G. L. et al. “Magneto-optic/Eddy Current Imaging of Aging Aircraft: a
New NDI Technique”, Material evaluation, pp. 1402-1407, Dec. 1993

[8] W. C. L. Shih, G. L. Fitzpatrick, D. K. Thome, R. L. Skaugset, and E. Y. C. Shih,
“Aircraft Inspection with the Magneto-optic/Eddy Current Imager,” CSNDT Journal,
Canada’s National NDT Magazine, March/April, 1994, Vol. 15, No. 2, pp. 13-30.

[9] G. L. F itzpatirck, D. K. Thome, R. L. Skaugset, E. Y. C. Shih, and W. C. L. Shih,
“Magneto-Optic/Eddy Current Imaging of Aging Aircraft: A New NDI Technique,”
Materials Evaluation, pp. 1402-1407, Dec. 1993

[10] Official Website of PRI Research & Development Corp,
http://www.primd.com/prod02.htm,

[11] G.L. Fitzpatrick, et al. “Magneto-Optic/Eddy Current Imaging of Subsurface
Corrosion and Fatigue Crack in Aging Aircraft”, Review of Progress in Quantative
Nondestructive Evaluation, V15, 1996

[12] V. J. Brechling and F. W. Spencer, “The Validation Process ads Applied to the
Magneto-Optic/Eddy Current Imager (MOI),” Materials Evaluation, pp. 815-818, July
1995.

[13] J aein L. and Udpa S.S., “Defect extraction in M01 image based on wavelet packet
transform and morphological technique”, in Review of Progress in Quantitative

87

Nondestructive Evaluation, eds. D.O. Thompson and DE cChimenti, Vol. 20, 2001, pp.
611-618

[14] Lemistre M. and Decitre J .M., “Corrosion and cracks detection in metallic
structures by magneto-optic image processing”, in Proceedings of International
Conference on Quality Control by Artificial Vision, editing by Cepadues, Le Creusot
(France), 2001, pp. 65-70.

[15] H. H. Nagel, “Overview On Image Sequence Analysis,” Image Sequence Processing
and Dynamic Scene Analysis, pp 2-39, Springer Verlag, 1983

[16] H. H. Nagel, “Image Sequence — Ten (Octal) Years — From Phenomenology
Towards a Theoretical Foundation,” Proc. International Conference On Pattern '1
Recognition,” pp. 1174-1185, Paris, France, 1986 l-

 

[17] A. Shio and J. Sklansky, “Segmentation of people in motion,” Proc. Of the IEEE '
Workshop on Visual Motion, pp. 325-332, Princeton, NJ, 1991 E

[l 8] C. Anderson, P. Burt, G. van der Wal, “Change detection and tracking using
pyramid transformation techniques,” SPIE Intelligent Robots and Computer Vision, V.
579, pp. 72-78, 1985

[19] Y. Z. Hsu, H. H. Nagel, and G. Rekers, “New likelihood test methods for change
detection in image sequences,” Computer Vision, Graphics and Image Processing, 24, pp.
73-106, 1984

[20] L. G. Shapiro, G. C. Stockman, “Computer Vision”. New Jersey: Prentice Hall,
2001

[21] B.K.P Horn, B.G. Schunck, “Determining Optical Flow”, Artificial Imtelligence, 17,
1981, pp. 185-203

[22] Otsu, N., “A threshold selection method flom gray-level histograms”, IEEE Trans.
Syst., SMC-9, 1979, 62-66

[23] D. S. Bright, D. E. Newbury, E. B. Steel, “Visibility of objects in computer
simulations of noisy micrographs”, J. of Microscopy, 189(8), Jan. 1998, 25-42

[24] P. Ramuhalli, J. Slade, U. Park, L. Xuan, L. Udpa, "Modeling and Signal Processing
of Magneto Optic Images for Aviation Applications," Presented at the International

Conference on Smart Materials Structures and Systems, Bangalore, India, Dec12-l4,
2002

[25] Duda, R., and Hart, P., “Use of the Hough Transform to Detect Lines and Curves in
Pictures”, Communications of the ACM 15, pp: 11-15, 1975.

88

[26] Tina Yu Tian, Mubarak Shah, “Recovering 3D Motion of Multiple Objects Using
Adaptive Hough Transform,” IEEE TRANSACTIONS ON PATTERN ANALYSIS
AND MACHINE INTELLIGENCE, VOL. 19, NO. 10, OCTOBER 1997

[27] R. O. Duda, P. E. Hart, D. G. Stork, “Pattern Classification,” John Wiley & Sons,
Inc. 2001, pp. 20-45

[28] Unsang Park, Lalita Udpa, George. C. Stockman, Bill Shih, Jerry Fitzpatrick, “Real-
time Implementation of Motion-based Filtering in Magneto-Optic Imager,” Presented at
30th Annual Review of Progress in Quantitative Nondestructive Evaluation, Green bay,
Wisconsin, Jul. 29-Aug. 1, 2003

[29]. T. S. Huang, G. J. Yang, G. Y. Tang, “A Fast Two-Dimensional Median Filtering

Algorithm,” IEEE Transactions on Acoustics, Speech, and Signal Processing, V. ASSP-
27, No. 1, Feb, 1979

89

 

IIIIIIIIIIIIIIIIIIIIIIIIIIIIIII

[Ill][IIIIIIIIIIIIIIIIIIIII