 

 

6

WWW» ‘
‘ h‘a-Ip IIHU.‘

.mW

. ,

at...

. __ ,9.
“—7.2.“
,,‘. A‘lu4
"gun .
' 3. 7" ‘
.0: u ..
".o...m <
. Lu -J' A»

u

z. i
.J,
A“. d“

 

r.

wu- '

{ LIBRARY
" Micmgan State
2.3 U " University
a 8/9 /o°55

This is to certify that the
thesis entitled

INTEGRATION OF MULTIPLE CUES IN BIOMETRIC
SYSTEMS

presented by

KARTHIK NANDAKUMAR

has been accepted towards fulfillment
of the requirements for the

M. S. degree in COMPUTER SCIENCE AND
ENGINEERING

 

 

MWI/

 

Major Professor’s Signature

Man 13, 100!

 

Date

MSU is an Afﬁrmative Action/Equal Opportunity Institution

 

PLACE IN RETURN Box to remove this checkout from your record.
To AVOID FINES return on or before date due.
MAY BE RECALLED with earlier due date if requested.

 

DATE DUE DATE DUE DATE DUE

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

INTEGRATION OF MULTIPLE CUES IN BIOMETRIC
SYSTEMS

By

Karthik Nandakumar

A THESIS

Submitted to
Michigan State University
in partial fulﬁllment of the requirements
for the degree of

MASTER OF SCIENCE

Department of Computer Science and Engineering

2005

 

ABSTRACT

INTEGRATION OF MULTIPLE CUES IN BIOMETRIC SYSTEMS
By

Karthik Nandakumar

Biometrics refers to the automatic recognition of individuals based on their physiolog-
ical and/or behavioral characteristics. Unimodal biometric systems perform person recog-
nition based on a single source of biometric information and are affected by problems like
noisy sensor data, non-universality and lack of individuality of the chosen biometric trait,
absence of an invariant representation for the biometric trait and susceptibility to circum-
vention. Some of these problems can be alleviated by using multimodal biometric systems
that consolidate evidence from multiple biometric sources. Integration of evidence ob-
tained from multiple cues is a challenging problem and integration at the matching score
level is the most common approach because it offers the best trade-off between the in-
formation content and the ease in fusion. In this thesis, we address two important issues
related to score level fusion. Since the matching scores output by the various modalities
are heterogeneous, score normalization is needed to transform these scores into a common
domain prior to fusion. We have studied the performance of different normalization tech-
niques and fusion rules using a multimodal biometric system based on face, ﬁngerprint and
hand-geometry modalities. The normalization schemes have been evaluated both in terms
of their efﬁciency and robustness to the presence of outliers in the training data. We have
also demonstrated how soft biometric attributes like gender, ethnicity, accent and height,
that by themselves do not have sufﬁcient discriminative ability to reliably recognize a per-
son, can be used to improve the recognition accuracy of the primary biometric identiﬁers
(e.g., ﬁngerprint and face). We have developed a mathematical model based on Bayesian

decision theory for integrating the primary and soft biometric cues at the score level.

 

© Cepyright by
Karthik Nandakumar

2005

 

To My Grandfather and Grandmother

iv

ACKNOWLEDGEMENTS

First and foremost, I would like to thank my grandfather Shri. RS. Venkatachari and
my grandmother Smt. P.S. Vanaja for all their prayers and blessings. Without their support
and encouragement at crucial periods of my life, it would not have been possible for me to
pursue graduate studies and aim for greater things in my career. Their hard work and posi-
tive attitude even at an age that is soon approaching 80, is my main source of inspiration. I
am proud to dedicate this thesis and all the good things in my life to them.

I am grateful to my advisor Dr. Anil Jain for providing me the opportunity to work in an
exciting and challenging ﬁeld of research. His constant motivation, support and infectious
enthusiasm has guided me towards the successful completion of this thesis. My interactions
with him have been of immense help in deﬁning my research goals and in identifying ways
to achieve them. His encouraging words have often pushed me to put in my best possible
efforts. I am hopeful that my Ph.D. experience with him would continue to be as memorable
and fruitful as my Masters work.

I also thank my guidance committee members Dr. Bill Punch and Dr. Arun Ross
for spending their time in carefully reviewing this thesis. Their valuable comments and
suggestions have been very useful in enhancing the presentation of this thesis. Special
thanks goes to Dr. Arun Ross for his guidance in the score normalization research. I am
also indebted to Dr. Sarat Dass for the enlightening research discussions that have shown
me the right path on many occasions, especially in the soft biometrics research.

The PRIP lab is an excellent place to work in and it is one place where you can always
ﬁnd good company, no matter what time of the day it is. My interactions with members of
this lab has certainly made me a better professional. Special thanks goes to Umut Uludag

for helping me acclimatize to the lab during the initial months. I would also like to thank

 

Xiaoguang Lu and Unsang Park for their assistance in the soft biometrics project and Mar-
tin Law who is always ready to help in case of any problems. Finally, I would like to Steve
Krawczyk for having taken the time to proof-read my thesis.

I would like to thank my friends Mahesh Arumugam and Arun Prabakaran for their
great company during my stay at MSU. I am also grateful to my friends Mahadevan Balakr-
ishnan, Jayaram Venkatesan, Hariharan Rajasekaran and Balasubramanian Rathakrishnan
for all their long conversations over phone that helped me feel at home.

Finally, I would like to thank my parents who have been the pillar of strength in all my
endeavors. I am always deeply indebted to them for all that they have given me. I also
thank the other members of my family including my brother and two sisters for their love,

affection and timely help.

vi

 

TABLE OF CONTENTS

Page

Abstract ......................................... ii

List of Figures ...................................... ix

List of Tables ...................................... xii
Chapters:

1. Introduction .................................... 1

1.1 Biometric Systems ............................. 1

1.2 Multimodal Biometric Systems ...................... 7

1.2.1 Why Multimodal Biometrics? ................... 7

1.2.2 How to Integrate Information? ................... 12

1.3 Thesis Contributions ............................ l3

2. Information Fusion in Multimodal Biometrics .................. 15

2.1 Architecture ................................ 15

2.2 Sources of Multiple Evidence ....................... 19

2.3 Levels of Fusion .............................. 20

2.3.1 Fusion Prior to Matching ..................... 20

2.3.2 Fusion After Matching ....................... 23

2.4 Fusion at the Matching Score Level .................... 24

2.4.1 Classiﬁcation Approach to Score Level Fusion .......... 24

2.4.2 Combination Approach to Score Level Fusion .......... 25

2.5 Evaluation of Multimodal Biometric Systems ............... 27

2.6 Summary .................................. 28

3. Score Normalization in Multimodal Biometric Systems ............. 31

3.1 Need for Score Normalization ....................... 33

3.2 Challenges in Score Normalization .................... 33

3.3 Normalization Techniques ......................... 36

3.4 Experimental Results ............................ 48

3.4.1 Generation of the Multimodal Database .............. 49

3.4.2 Impact of Normalization on Fusion Performance ......... 52

3.4.3 Robustness Analysis of Normalization Schemes ......... 58

3.5 Summary .................................. 62

vii

4. Soft Biometrics .................................. 63

4.1 Motivation and Challenges ......................... 63

4.2 Soft Biometric Feature Extraction ..................... 66

4.2.1 A Vision System for Soft Biometric Feature Extraction ..... 69

4.3 Fusion of Soft and Primary Biometric Information ............ 70

4.3.1 Identiﬁcation Mode ........................ 70

4.3.2 Veriﬁcation Mode ......................... 72

4.3.3 Computation of Soft Biometric Likelihoods ............ 73

4.4 Experimental Results ............................ 75

4.4.1 Identiﬁcation Performance ..................... 76

4.4.2 Veriﬁcation Performance ..................... 77

4.5 Summary .................................. 80

5. Conclusions and Future Work ........................... 84
5.1 Conclusions ................................. 84

5.2 Future Work ................................ 86
Bibliography ...................................... 88

viii

LIST OF FIGURES

Figure

1.1

1.2

1.3

1.4

1.5

1.6

2.1

2.2

2.3

2.4

Characteristics that are being used for biometric recognition; (a) Finger-
print; (b) Hand-geometry; (c) Iris; ((1) Retina; (e) Face; (f) Palmprint; (g)
Ear structure; (h) DNA; (i) Voice; (j) Gait; (k) Signature and (l) Keystroke

dynamics. ...................................

Information ﬂow in biometric systems .....................

Some illustrations of deployment of biometrics in civilian applications;
(a) A ﬁngerprint veriﬁcation system manufactured by Digital Persona Inc.
used for computer and network login; (b) An iris-based access control sys-
tem at the Umea airport in Sweden that veriﬁes the frequent travelers and
allows them access to ﬂights; (c) A cell phone manufactured by LG Elec-
tronics that recognizes authorized users using ﬁngerprints (sensors manu-
factured by Authentec Inc.) and allows them access to the phone’s spe-
cial functionalities such as mobile-banking; (d) The US-VISIT imrrrigra-
tion system based on ﬁngerprint and face recognition technologies and (e)
A hand geometry system at Disney World that veriﬁes seasonal and yearly

pass-holders to allow them fast entry ......................

Examples of noisy biometric data; (a) A noisy ﬁngerprint image due to

smearing, residual deposits, etc.; (b) A blurred iris image due to loss of focus.

Three impressions of a user's ﬁnger showing the poor quality of the ridges.

Four face images of the person in (a), exhibiting variations in (b) expres-

sion, (c) illumination and (d) pose. ......................

Architecture of multimodal biometric systems; (a) Serial and (b) Parallel.

Sources of multiple evidence in multimodal biometric systems. In the ﬁrst
four scenarios, multiple sources of information are derived from the same
biometric trait. In the ﬁfth scenario, information is derived from different

biometric traits. ................................
Levels of fusion in multibiometric systems. .................

Summary of approaches to information fusion in biometric systems. . . . .

ix

Page

2

4

6

7

10

16

18

29

3.1

3.2

3.3

3.4

3.5

3.6

3.7

3.8

3.9

3.10

3.11

3.12

3.13

3.14

3.15

4.1

Conditional distributions of genuine and impostor scores used in our ex-
periments; (a) Face (distance score), (b) Fingerprint (similarity score) and
(c) Hand-geometry (distance score). ..................... 34

Distributions of genuine and impostor scores after min-max normalization;

(a) Face, (b) Fingerprint and (c) Hand-geometry ................ 37
Distributions of genuine and impostor scores after z-score normalization;
(a) Face, (b) Fingerprint, and (c) Hand-geometry. .............. 40
Distributions of genuine and impostor scores after median-MAD normal-
ization; (a) Face, (b) Fingerprint and (c) Hand-geometry. .......... 41
Double sigmoid normalization (t = 200, n = 20, and r2 = 30). ...... 42

Distributions of genuine and impostor scores after double sigmoid normal-
ization; (a) Face, (b) Fingerprint and (c) Hand-geometry. .......... 44

Hampel inﬂuence function (a = 0.7, b = 0.85, and c = 0.95) ......... 46

Distributions of genuine and impostor scores after tanh normalization; (a)
Face, (b) Fingerprint and (c) Hand-geometry. ................ 47

ROC curves for individual modalities. .................... 51

ROC curves for sum of scores fusion method under different normalization
schemes ..................................... 53

ROC curves for max-score fusion method under different normalization
schemes ..................................... 56

ROC curves for min-score fusion method under different normalization

schemes ..................................... 57
Robustness analysis of min-max normalization. ............... 59
Robustness analysis of z-score normalization. ................ 60
Robustness analysis of tanh normalization ................... 61

An ATM kiosk equipped with a ﬁngerprint (primary biometric) sensor and
a camera to obtain soft attributes (gender, ethnicity and height) ........ 64

4.2

4.3

4.4

4.5

4.6

4.7

4.8

4.9

Examples of soft biometric traits ........................ 68

Framework for fusion of primary and soft biometric information. Here x is
the ﬁngerprint feature vector and y is the soft biometric feature vector. . . . 71

Improvement in identiﬁcation performance of a ﬁngerprint system after uti-
lization of soft biometric traits. a) Fingerprint with gender and ethnicity, b)
Fingerprint with height, and c) Fingerprint with gender, ethnicity and height. 78

Improvement in identiﬁcation performance of face recognition system after
utilization of the height of the user ....................... 79

Improvement in identiﬁcation performance of (face + ﬁngerprint) multi-
modal system after the addition of soft biometric traits. ........... 79

Improvement in veriﬁcation performance of a ﬁngerprint system after uti-
lization of soft biometric traits. a) Fingerprint with gender and ethnicity, b)

Fingerprint with height, and c) Fingerprint with gender, ethnicity and height. 81

Improvement in veriﬁcation performance of face recognition system after
utilization of the height of the user ....................... 82

Improvement in veriﬁcation performance of (face + ﬁngerprint) multimodal
system after the addition of soft biometric traits ................ 82

xi

LIST OF TABLES

Table Page

1.1 State-of—the-art error rates associated with ﬁngerprint, face and voice bio-
metric systems. Note that the accuracy estimates of biometric systems are
dependent on a number of test conditions. .................. 11

3.1 Summary of Normalization Techniques ................... 48
3.2 Genuine Acceptance Rate (GAR) (%) of different normalization and fusion
techniques at the 0.1% False Acceptance Rate (FAR). Note that the values

in the table represent average GAR, and the values indicated in parentheses
correspond to the standard deviation of GAR. ................ 52

xii

CHAPTER 1

Introduction

1.1 Biometric Systems

Identity management refers to the challenge of providing authorized users with secure
and easy access to information and services across a variety of networked systems. A reli-
able identity management system is a critical component in several applications that render
their services only to legitimate users. Examples of such applications include physical ac-
cess control to a secure facility, e-commerce, access to computer networks and welfare
distribution. The primary task in an identity management system is the determination of
an individual’s identity. Traditional methods of establishing a person’s identity include
knowledge-based (e.g., passwords) and token-based (e.g., II) cards) mechanisms. These
surrogate representations of the identity can easily be lost, shared or stolen. Therefore,
they are not sufﬁcient for identity veriﬁcation in the modern day world. Biometrics offers
a natural and reliable solution to the problem of identity determination by recognizing in-
dividuals based on their physiological and/or behavioral characteristics that are inherent to
the person [1].

Some of the physiological and behavioral characteristics that are being used for biomet-
ric recognition include ﬁngerprint, hand-geometry, iris, retina, face, palmprint, ear, DNA,
voice, gait, signature and keystroke dynamics (see Figure 1.1). A typical biometric sys-
tem consists of four main modules. The sensor module is responsible for acquiring the
biometric data from an individual. The feature extraction module processes the acquired

biometric data and extracts only the salient information to form a new representation of the

 

(e) (0 (g) (h)

   

(i) (j) (k) (I)

Figure 1.1: Characteristics that are being used for biometric recognition; (a) Fingerprint;
(b) Hand-geometry; (c) Iris; (d) Retina; (e) Face; (1) Palmprint; (g) Ear structure; (h) DNA;
(i) Voice; (i) Gait; (k) Signature and (l) Keystroke dynamics.

data. Ideally, this new representation should be unique for each person and also relatively
invariant with respect to changes in the different samples of the same biometric collected
from the same person. The matching module compares the extracted feature set with the
templates stored in the system database and determines the degree of similarity (dissimilar-
ity) between the two. The decision module either veriﬁes the identity claimed by the user
or determines the user’s identity based on the degree of similarity between the extracted
features and the stored template(s).

Biometric systems can provide three main functionalities, namely, (i) veriﬁcation, (ii)
identiﬁcation and (iii) negative identiﬁcation. Figure 1.2 shows the ﬂow of information in
veriﬁcation and identiﬁcation systems. In veriﬁcation or authentication, the user claims an
identity and the system veriﬁes whether the claim is genuine. For example, in an ATM
application the user may claim a speciﬁc identity, say John Doe, by entering his Personal
Identiﬁcation Number (PIN). The system acquires the biometric data from the user and
compares it only with the template of John Doe. Thus, the matching is 1:1 in a veriﬁcation
system. If the user’s input and the template of the claimed identity have a high degree of
similarity, then the claim is accepted as “genuine”. Otherwise, the claim is rejected and the
user is considered an “impostor”. In short, a biometric system operating in the veriﬁcation
mode answers the question “Are you who you say you are?”.

In a biometric system used for identiﬁcation, the user does not explicitly claim an iden-
tity. However, the implicit claim made by the user is that he is one among the persons
already enrolled in the system. In identiﬁcation, the user’s input is compared with the
templates of all the persons enrolled in the database and the identity of the person whose
template has the highest degree of similarity with the user’s input is output by the biometric
system. Typically, if the highest similarity between the input and all the templates is less

than a ﬁxed minimum threshold, the system outputs a reject decision which implies that

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Enrollment Process
ldentity,l
Feature Template,T
Extractor l
Biometric i
Sensor System
Database
Verification Process
Claimed Identity, | I
Input, X Matchin T
Feature 9 ‘ ' ,
Extractor ’ Module
Score, S . l
Biometric Decision
User Sensor Module
i
Accept/Reject
Identification Process
Feature Input, X ’ Matching {T 1,...,TN}
Extractor Module
S ,...,S
Biometric { ‘ N} l
Sensor Decision
Module
I
ldentity,l

Figure 1.2: Information ﬂow in biometric systems.

the user presenting the input is not one among the enrolled users. Therefore, the matching
is 1:N in an identiﬁcation system. An example of an identiﬁcation system could be access
control to a secure building. All users who are authorized to enter the building would be
enrolled in the system. Whenever a user tries to enter the building, he presents his biomet-
ric data to the system and upon determination of the user’s identity the system grants him
the preset access privileges. An identiﬁcation system answers the question “Are you really
someone who is known to the system?”.

Negative identiﬁcation systems are similar to identiﬁcation systems because the user
does not explicitly claim an identity. The main factor that distinguishes the negative identi-
ﬁcation functionality from identiﬁcation is the user’s implicit claim that he is not a person
who is already enrolled in the system. Negative identiﬁcation is also known as screening.
As in identiﬁcation, the matching is 1:N in screening. However in screening, the system
will output an identity of an enrolled person only if that person’s template has the highest
degree of similarity with the input among all the templates and if the corresponding similar-
ity value is greater than a ﬁxed threshold. Otherwise, the user’s claim that he is not already
known to the system is accepted. Screening is often used at airports to verify whether a
passenger’s identity matches with any person on a “watch-list”. Screening can also be used
to prevent the issue of multiple credential records (e.g., driver’s licence, passport) to the
same person. To summarize, negative identiﬁcation answers the question “Are you who
you say you are not?”.

Veriﬁcation functionality can be provided by traditional methods like passwords and ID
cards as well as by biometrics. The negative identiﬁcation functionality can be provided
only by biometrics. Further, biometric characteristics are inherent to the person whose
identity needs to be established. Hence, they cannot be lost, stolen, shared, or forgot-

ten. Therefore, biometric traits provide more security than traditional knowledge-based or

 

Figure 1.3: Some illustrations of deployment of biometrics in civilian applications; (a) A
ﬁngerprint veriﬁcation system manufactured by Digital Persona Inc. used for computer
and network login; (b) An iris-based access control system at the Umea airport in Sweden
that veriﬁes the frequent travelers and allows them access to ﬂights; (c) A cell phone man-
ufactured by LG Electronics that recognizes authorized users using ﬁngerprints (sensors
manufactured by Authentec Inc.) and allows them access to the phone’s special functional-
ities such as mobile-banking; (d) The US-VISIT immigration system based on ﬁngerprint
and face recognition technologies and (e) A hand geometry system at Disney World that
veriﬁes seasonal and yearly pass-holders to allow them fast entry.

token-based identiﬁcation methods. They also discourage fraud and eliminate the possibil-
ity of repudiation. Finally, they are more convenient to use because it eliminates the need
for remembering multiple complex passwords and carrying identiﬁcation cards. Although
biometric systems have some limitations [2], they offer a number of advantages over tra-
ditional security methods and this has led to their widespread deployment in a variety of
civilian applications. Figure 1.3 shows some examples of biometrics deployment in civilian

applications.
1.2 Multimodal Biometric Systems

1.2.1 Why Multimodal Biometrics?

Unimodal biometric systems perform person recognition based on a single source of

biometric information. Such systems are often affected by the following problems [3]:

   

(a) (b)

Figure 1.4: Examples of noisy biometric data; (a) A noisy ﬁngerprint image due to smear-
ing, residual deposits, etc.; (b) A blurred iris image due to loss of focus.

     

(a) (b) (C)

Figure 1.5: Three impressions of a user’s ﬁnger showing the poor quality of the ridges.

o Noisy sensor data : Noise can be present in the acquired biometric data mainly due
to defective or improperly maintained sensors. For example, accumulation of dirt or
the residual remains on a ﬁngerprint sensor can result in a noisy ﬁngerprint image as
shown in Figure 1.4(a). Failure to focus the camera appropriately can lead to blurring
in face and iris images (see Figure l.4(b)). The recognition accuracy of a biometric
system is highly sensitive to the quality of the biometric input and noisy data can

result in a signiﬁcant reduction in the accuracy of the biometric system [4].

o Non-universality: If every individual in the target population is able to present the
biometric trait for recognition, then the trait is said to be universal. Universality is
one of the basic requirements for a biometric identiﬁer. However, not all biomet-
ric traits are truly universal. The National Institute of Standards and Technology
(NIST) has reported that it is not possible to obtain a good quality ﬁngerprint from
approximately two percent of the population (people with hand-related disabilities,
manual workers with many cuts and bruises on their ﬁngertips, and people with very
oily or dry ﬁngers) [5] (see Figure 1.5). Hence, such people cannot be enrolled in a

ﬁngerprint veriﬁcation system. Similarly, persons having long eye-lashes and those

suffering from eye abnormalities or diseases like glaucoma, cataract, aniridia, and
nystagmus cannot provide good quality iris images for automatic recognition [6].
Non-universality leads to Failure to Enroll (FTE) and/or Failure to Capture (FTC)

errors in a biometric system.

Lack of individuality: Features extracted from biometric characteristics of different
individuals can be quite similar. For example, appearance-based facial features that
are commonly used in most of the current face recognition systems are found to
have limited discrimination capability [7]. A small proportion of the population can
have nearly identical facial appearance due to genetic factors (e.g., father and son,
identical twins, etc.). This lack of uniqueness increases the False Match Rate (FMR)

of a biometric system.

Lack of invariant representation: The biometric data acquired from a user during
veriﬁcation will not be identical to the data used for generating the user’s template
during enrollment. This is known as “intra-class variation”. The variations may be
due to improper interaction of the user with the sensor (e. g., changes due to rotation,
translation and applied pressure when the user places his ﬁnger on a ﬁngerprint sen-
sor, changes in pose and expression when the user stands in front of a camera, etc.),
use of different sensors during enrollment and veriﬁcation, changes in the ambient
environmental conditions (e.g., illumination changes in a face recognition system)
and inherent changes in the biometric trait (e.g., appearance of wrinkles due to ag-
ing or presence of facial hair in face images, presence of scars in a ﬁngerprint, etc.).
Figure 1.6 shows the intra-class variations in face images caused due to expression,
lighting and pose changes. Ideally, the features extracted from the biometric data

must be relatively invariant to these changes. However, in most practical biometric

systems the features are not invariant and therefore complex matching algorithms are
required to take these variations into account. Large intra-class variations usually

increase the False Non-Match Rate (FNMR) of a biometric system.

Susceptibility to circumvention: Although it is very difﬁcult to steal someone’s bio-
metric traits, it is still possible for an impostor to circumvent a biometric system using
spoofed traits. Studies [8,9] have shown that it is possible to construct gummy ﬁngers
using lifted ﬁngerprint impressions and utilize them to circumvent a biometric sys-
tem. Behavioral traits like signature and voice are more susceptible to such attacks
than physiological traits. Other kinds of attacks can also be launched to circumvent

a biometric system [10].

   

(a) (b) (c) (d)

Figure 1.6: Four face images of the person in (a), exhibiting variations in (b) expression,
(c) illumination and (d) pose.

Due to these practical problems, the error rates associated with unimodal biometric
systems are quite high which makes them unacceptable for deployment in security criti—

cal applications. The state-of-the-art error rates associated with ﬁngerprint, face and voice

Table 1.1: State-of-the-art error rates associated with ﬁngerprint, face and voice biometric
systems. Note that the accuracy estimates of biometric systems are dependent on a number
of test conditions.

 

 

 

 

 

 

 

 

Test Test Parameter False Reject False Accept
Rate Rate
Fingerprint FVC 2004 [11] Exaggerated skin 2% 2%
distortion, rotation
FpVTE 2003 [12] US. government 0.1% 1%
operational data
Face FRVT 2002 [13] Varied lighting, 10% 1%
outdoor/indoor
Voice NIST 2004 [14] Text independent, 5-10% 2—5%
multi-lingual

 

 

 

 

biometric systems are shown in Table 1.1. Some of the problems that affect unimodal bio-
metric systems can be alleviated by using multimodal biometric systems [15]. Systems
that consolidate cues obtained from two or more biometric sources for the purpose of per-
son recognition are called multimodal biometric systems. Multimodal biometric systems
have several advantages over unimodal systems. Combining the evidence obtained from
different modalities using an effective fusion scheme can signiﬁcantly improve the overall
accuracy of the biometric system. A multimodal biometric system can reduce the FIE/FTC
rates and provide more resistance against spooﬁng because it is difﬁcult to simultaneously
spoof multiple biometric sources. Multimodal systems can also provide the capability to
search a large database in an efﬁcient and fast manner. This can be achieved by using a rel-
atively simple but less accurate modality to prune the database before using the more com-
plex and accurate modality on the remaining data to perform the ﬁnal identiﬁcation task.
However, multimodal biometric systems also have some disadvantages. They are more ex-

pensive and require more resources for computation and storage than unimodal biometric

ll

systems. Multimodal systems generally require more time for enrollment and veriﬁcation
causing some inconvenience to the user. Finally, the system accuracy can actually degrade
compared to the unimodal system if a proper technique is not followed for combining the
evidence provided by the different modalities. However, the advantages of multimodal sys-
tems far outweigh the limitations and hence, such systems are being increasingly deployed

in security-critical applications.
1.2.2 How to Integrate Information?

The design of a multimodal biometric system is strongly dependent on the application
scenario. A number of multimodal biometric systems have been proposed in literature that
differ from one another in terms of their architecture, the number and choice of biometric
modalities, the level at which the evidence is accumulated, and the methods used for the
integration or fusion of information. Chapter 2 presents a detailed discussion on the design
of a multimodal biometric system.

Four levels of information fusion are possible in a multimodal biometric system. They
are fusion at the sensor level, feature extraction level, matching score level and decision
level. Sensor level fusion is quite rare because fusion at this level requires that the data ob-
tained from the different biometric sensors must be compatible, which is seldom the case
with biometric sensors. Fusion at the feature level is also not always possible because the
feature sets used by different biometric modalities may either be inaccessible or incompat-
ible. Fusion at the decision level is too rigid since only a limited amount of information
is available. Therefore, integration at the matching score level is generally preferred due
to the presence of sufﬁcient information content and the ease in accessing and combining

matching scores.

12

In the context of veriﬁcation, fusion at the matching score level can be approached in
two distinct ways. In the ﬁrst approach the fusion is viewed as a classiﬁcation problem,
while in the second approach it is viewed as a combination problem. In the classiﬁcation
approach, a feature vector is constructed using the matching scores output by the individual
matchers; this feature vector is then classiﬁed into one of two classes: “Accept” (genuine
user) or “Reject” (impostor). In the combination approach, the individual matching scores
are combined to generate a single scalar score which is then used to make the ﬁnal decision.
Both these approaches have been widely studied in the literature. Ross and Jain [16] have
shown that the combination approach performs better than some classiﬁcation methods
like decision tree and linear discriminant analysis. However, it must be noted that no single

classiﬁcation or combination scheme works well under all circumstances.

1.3 Thesis Contributions

A review of the proposed multimodal systems indicates that the major challenge in
multimodal biometrics is the problem of choosing the right methodology to integrate or fuse
the information obtained from multiple sources. In this thesis, we deal with two important

problems related to score level fusion.

0 In the ﬁrst part of the thesis, we follow the combination approach to score level
fusion and address some of the issues involved in computing a single matching score
given the scores of different modalities. Since the matching scores generated by the
different modalities are heterogeneous, normalization is required to transform these
scores into a common domain before combining them. While several normalization
techniques have been proposed, there has been no detailed study of these techniques.
In this thesis, we have systematically studied the effects of different normalization

schemes on the performance of a multimodal biometric system based on the face,

13

ﬁngerprint and hand-geometry modalities. Apart from studying the efﬁciency of the
normalization schemes, we have also analyzed their robustness to the presence of

outliers in the training data.

The second part of the thesis proposes a solution to the problem of integrating the
information obtained from the soft biometric identiﬁers like gender, ethnicity and
height with the primary biometric information like face and ﬁngerprint. A mathemat-
ical model based on the Bayesian decision theory has been developed to perform the
integration. Experiments based on this model demonstrate that soft biometric identi-
ﬁers can be used to signiﬁcantly improve the recognition performance of the primary
biometric system even when the soft biometric identiﬁers cannot be extracted with

100% accuracy.

14

CHAPTER 2

Information Fusion in Multimodal Biometrics

Multimodal biometric systems that have been proposed in literature can be classiﬁed
based on four parameters, namely, (i) architecture, (ii) sources that provide multiple ev-
idence, (iii) level of fusion and (iv) methodology used for integrating the multiple cues.
Generally, these design decisions depend on the application scenario and these choices
have a profound inﬂuence on the performance of a multimodal biometric system. In this
chapter, we compare the existing multibiometric systems based on the above four parame-

ters.

2.1 Architecture

Architecture of a multibiometric system refers to the sequence in which the multiple
cues are acquired and processed. Typically, the architecture of a multimodal biometric sys-
tem is either serial or parallel (see Figure 2.1). In the serial or cascade architecture, the pro-
cessing of the modalities takes place sequentially and the outcome of one modality affects
the processing of the subsequent modalities. In the parallel design, different modalities
operate independently and their results are combined using an appropriate fusion scheme.
Both these architectures have their own advantages and limitations.

The cascading scheme can improve the user convenience as well as allow fast and efﬁ-
cient searches in large scale identiﬁcation tasks. For example, when a cascaded multimodal
biometric system has sufﬁcient conﬁdence on the identity of the user after processing the
ﬁrst modality, the user may not be required to provide the other modalities. The system

can also allow the user to decide which modality he/she would present ﬁrst. Finally, if the

15

 

 

 

 

Fusion

 

 

 

   
 
  

 

 

 

 

 

 

 

 

 

 

   

     
 

 

 

+ ‘-—+ Decision
Matching
a
' I
' I
I I
' I
I I
' I
I I
I I
I
I
I
I
I
I
(a)
, Additional N0 - . .
——'\ Biometric? 'D°°'s'°n
Yes
Fusion
I
I
l
E . . No
' Add't'orial >Decision
l Biometric?
Fusion : Decision

 

 

 

 

 

(b)

Figure 2.1: Architecture of multimodal biometric systems; (a) Serial and (b) Parallel.

l6

system is faced with the task of identifying the user from a large database, it can utilize the
outcome of each modality to successively prune the database, thereby making the search
faster and more efﬁcient. Thus, a cascaded system can be more convenient to the user and
generally requires less recognition time when compared to its parallel counterpart. How-
ever, it requires robust algorithms to handle the different sequence of events. An example
of a cascaded multibiometric system is the one proposed by Hong and Jain in [17]. In this
system, face recognition is used to retrieve the top it matching identities and ﬁngerprint
recognition is used to verify these identities and make a ﬁnal identiﬁcation decision. A
multimodal system designed to operate in the parallel mode generally has a higher accu-
racy because it utilizes more evidence about the user for recognition. Most of the proposed
multibiometric systems have a parallel architecture because the primary goal of system de-
signers has been a reduction in the error rate of biometric systems (see [16], [18] and the
references therein).

The choice of the system architecture depends on the application requirements. User-
friendly and less security critical applications like bank ATMs can use a cascaded multi-
modal biometric system. On the other hand, parallel multimodal systems are more suited
for applications where security is of paramount importance (e.g., access to military instal—
lations). It is also possible to design a hierarchical (tree-like) architecture to combine the
advantages of both cascade and parallel architectures. This hierarchical architecture can be
made dynamic so that it is robust and can handle problems like missing and noisy biomet-
ric samples that arise in biometric systems. But the design of a hierarchical multibiometric

system has not received much attention from researchers.

17

Left Profile Frontal Right Profile

 

        
  
 

Multiple Instances Fingerprint
Optical
I _ Multiple F
Multiple Sensors ;:7 7 Biometric Traits ace

   

 

Multiple Multiple Units
Representations

   

   

Right Eye Left Eye

  

Minutiae

Figure 2.2: Sources of multiple evidence in multimodal biometric systems. In the ﬁrst four
scenarios, multiple sources of information are derived from the same biometric trait. In the
ﬁfth scenario, information is derived from different biometric traits.

2.2 Sources of Multiple Evidence

Multimodal biometric systems overcome some of the limitations of unimodal biomet-
ric systems by consolidating the evidence obtained from different sources (see Figure 2.2).
These sources may be (i) multiple sensors for the same biometric (e. g., optical and solid-
state ﬁngerprint sensors), (ii) multiple instances of the same biometric (e.g., multiple face
images of a person obtained under different pose/lighting conditions), (iii) multiple repre-
sentations and matching algorithms for the same biometric (e.g., multiple face matchers
like PCA and LDA), (iv) multiple units of the same biometric (e.g., left and right iris im-
ages), or (v) multiple biometric traits (e.g., face, ﬁngerprint and iris). In the ﬁrst four
scenarios, multiple sources of information are derived from the same biometric trait. In the
ﬁfth scenario, information is derived from different biometric traits.

The use of multiple sensors can address the problem of noisy sensor data, but all other
potential problems associated with unimodal biometric systems remain. A recognition sys-
tem that works on multiple units of the same biometric can ensure the presence of a live
user by asking the user to provide a random subset of biometric measurements (e.g., left
index ﬁnger followed by right middle ﬁnger). Multiple instances of the same biometric,
or multiple representations and matching algorithms for the same biometric may also be
used to improve the recognition performance of the system. However, all these methods
still suffer from some of the problems faced by unimodal systems. A multimodal biometric
system based on different traits is expected to be more robust to noise, address the prob-
lem of non-universality, improve the matching accuracy, and provide reasonable protection
against spoof attacks. Hence, the development of biometric systems based on multiple

biometric traits has received considerable attention from researchers.

l9

2.3 Levels of Fusion

Fusion in multimodal biometric systems can take place at four major levels, namely,
sensor level, feature level, score level and decision level. Figure 2.3 shows some examples
of fusion at the various levels. These four levels can be broadly categorized into fusion

prior to matching and fusion after matching [19].

2.3.1 Fusion Prior to Matching

Prior to matching, integration of information can take place either at the sensor level or
at the feature level. The raw data from the sensor(s) are combined in sensor level fusion
[20]. Sensor level fusion can be done only if the multiple cues are either instances of the
same biometric trait obtained from multiple compatible sensors or multiple instances of the
same biometric trait obtained using a single sensor. For example, the face images obtained
from several cameras can be combined to form a 3D model of the face. Another example
of sensor level fusion is the mosaicking of multiple ﬁngerprint impressions to form a more
complete ﬁngerprint image [21, 22]. In sensor level fusion, the multiple cues must be
compatible and the correspondences between points in the data must be known in advance.
Sensor level fusion may not be possible if the data instances are incompatible (e. g., it may
not be possible to integrate face images obtained from cameras with different resolutions).

Feature level fusion refers to combining different feature vectors that are obtained from
one of the following sources; multiple sensors for the same biometric trait, multiple in-
stances of the same biometric trait, multiple units of the same biometric trait or multiple
biometric traits. When the feature vectors are homogeneous (e.g., multiple ﬁngerprint im-
pressions of a user’s ﬁnger), a single resultant feature vector can be calculated as a weighted
average of the individual feature vectors. When the feature vectors are non-homogeneous

(e.g., feature vectors of different biometric modalities like face and hand geometry), we

20

 

   

..-
Leta-J

Slensqr Sources
eve
Fusion ®®

Decision

  
    

Feature Right 5,,
Level

    

BEE].

   
 

 

Sources

®®
63(5)

 

  
 
          
   
 

 
 

Sources

6%”

 

 

Score
Level
Fusion

Sources

6&9

 

Figure 2.3: Levels of fusion in multibiometric systems.

21

can concatenate them to form a single feature vector. Concatenation is not possible when
the feature sets are incompatible (e.g., ﬁngerprint minutiae and eigen-face coefﬁcients).
Attempts by Kumar et al. [23] in combining palmprint and hand-geometry features and by
Ross and Govindarajan [24] in combining face and hand-geometry features have met with
only limited success.

Biometric systems that integrate information at an early stage of processing are believed
to be more effective than those systems which perform integration at a later stage. Since
the features contain richer information about the input biometric data than the matching
score or the decision of a matcher, integration at the feature level should provide better
recognition results than other levels of integration. However, integration at the feature level
is difﬁcult to achieve in practice because of the following reasons: (i) The relationship
between the feature spaces of different biometric systems may not be known. In the case
where the relationship is known in advance, care needs to be taken to discard those features
that are highly correlated. This requires the application of feature selection algorithms prior
to classiﬁcation. (ii) Concatenating two feature vectors may result in a feature vector with
very large dimensionality leading to the ‘curse of dimensionality’ problem [25]. Although,
this is a general problem in most pattern recognition applications, it is more severe in
biometric applications because of the time, effort and cost involved in collecting large
amounts of biometric data. (iii) Most commercial biometric systems do not provide access
to the feature vectors which they use in their products. Hence, very few researchers have
studied integration at the feature level and most of them generally prefer fusion schemes

after matching.

22

2.3.2 Fusion After Matching

Schemes for integration of information after the classiﬁcation/matcher stage can be di-
vided into four categories: dynamic classiﬁer selection, fusion at the decision level, fusion
at the rank level and fusion at the matching score level. A dynamic classiﬁer selection
scheme chooses the results of that classiﬁer which is most likely to give the correct deci-
sion for the speciﬁc input pattern [26]. This is also known as the winner-take-all approach
and the device that performs this selection is known as an associative switch [27].

Integration of information at the abstract or decision level can take place when each
biometric matcher individually decides on the best match based on the input presented to
it. Methods like majority voting [28], behavior knowledge space [29], weighted voting
based on the Dempster-Shafer theory of evidence [30], AND rule and OR rule [31], etc.
can be used to arrive at the ﬁnal decision.

When the output of each biometric matcher is a subset of possible matches sorted in
decreasing order of conﬁdence, the fusion can be done at the rank level. Ho et al. [32] de-
scribe three methods to combine the ranks assigned by the different matchers. In the highest
rank method, each possible match is assigned the highest (minimum) rank as computed by
different matchers. Ties are broken randomly to arrive at a strict ranking order and the ﬁnal
decision is made based on the combined ranks. The Borda count method uses the sum of
the ranks assigned by the individual matchers to calculate the combined ranks. The logistic
regression method is a generalization of the Borda count method where the weighted sum
of the individual ranks is calculated and the weights are determined by logistic regression.

When the biometric matchers output a set of possible matches along with the quality
of each match (matching score), integration can be done at the matching score level. This
is also known as fusion at the measurement level or conﬁdence level. Next to the feature

vectors, the matching scores output by the matchers contain the richest information about

23

the input pattern. Also, it is relatively easy to access and combine the scores generated
by the different matchers. Consequently, integration of information at the matching score

level is the most common approach in multimodal biometric systems.

2.4 Fusion at the Matching Score Level

In the context of veriﬁcation, there are two approaches for consolidating the scores ob-
tained from different matchers. One approach is to formulate it as a classiﬁcation problem,
while the other approach is to treat it as a combination problem. In the classiﬁcation ap-
proach, a feature vector is constructed using the matching scores output by the individual
matchers; this feature vector is then classiﬁed into one of two classes: “Accept” (genuine
user) or “Reject” (impostor). Generally, the classiﬁer used for this purpose is capable of
learning the decision boundary irrespective of how the feature vector is generated. Hence,
the output scores of the different modalities can be non-homogeneous (distance or similar-
ity metric, different numerical ranges, etc.) and no processing is required prior to feeding
them into the classiﬁer. In the combination approach, the individual matching scores are
combined to generate a single scalar score which is then used to make the ﬁnal decision.
To ensure a meaningful combination of the scores from the different modalities, the scores

must be ﬁrst transformed to a common domain.
2.4.1 Classiﬁcation Approach to Score Level Fusion

Several classiﬁers have been used to consolidate the matching scores and arrive at a de-
cision. Wang et al. [33] consider the matching scores resulting from face and iris recogni-
tion modules as a two-dimensional feature vector. Fisher’s discriminant analysis and a neu-
ral network classiﬁer with radial basis function are then used for classiﬁcation. Verlinde and
Chollet [34] combine the scores from two face recognition experts and one speaker recog-

nition expert using three classiﬁers: k-NN classiﬁer using vector quantization, decision-tree

24

based classiﬁer and a classiﬁer based on a logistic regression model. Chatzis et a1. [35] use
fuzzy k-means and fuzzy vector quantization, along with a median radial basis function
neural network classiﬁer for the fusion of scores obtained from biometric systems based
on visual (facial) and acoustic (vocal) features. Sanderson et al. [19] use a support vector
machine classiﬁer to combine the scores of face and speech experts. They show that the
performance of such a classiﬁer deteriorates under noisy input conditions. To overcome
this problem, they implement structurally noise-resistant classiﬁers like a piece-wise linear
classiﬁer and a modiﬁed Bayesian classiﬁer. Ross and Jain [16] use decision tree and linear
discriminant classiﬁers for combining the scores of face, ﬁngerprint, and hand-geometry

modalities.

2.4.2 Combination Approach to Score Level Fusion

Kittler et al. [36] have developed a theoretical framework for consolidating the evidence
obtained from multiple classiﬁers using schemes like the sum rule, product rule, max rule,
min rule, median rule and majority voting. In order to employ these schemes, the matching
scores must be converted into posteriori probabilities conforming to a genuine user and
an impostor. They consider the problem of classifying an input pattern X into one of
m possible classes (in a veriﬁcation system, m = 2) based on the evidence provided by
R different classiﬁers or matchers. Let f,- be the feature vector (derived from the input
pattern X) presented to the 2"" matcher. Let the outputs of the individual matchers be
P (wj Iii), i.e., the posterior probability of the of class wj given the feature vector 5:}. Let
C 6 {011, L02, . ' . ,wm} be the class to which the input pattern X is ﬁnally assigned. The
following rules can be used to estimate oz
Product Rule: This rule is based on the assumption of statistical independence of the

representations f1, :32, - - - , 553. The input pattern is assigned to class c such that

25

R
c = argmaxj H P(wj|:f:'i).

i=1

In general, different biometric traits of an individual (e.g., face, ﬁngerprint and hand-
geometry) are mutually independent. This allows us to make use of the product rule in
a multimodal biometric system based on the independence assumption.

Sum Rule: The sum rule is more effective than the product rule when there is a high level
of noise leading to ambiguity in the classiﬁcation problem. The sum rule assigns the input

pattern to class e such that

R
c = argmaxj Z P(wj|f,-).

221

Max Rule: The max rule approximates the mean of the posteriori probabilities by the

maximum value. In this case, we assign the input pattern to class c such that

c = argmaxj max P(wj|5c’,-).
1

Min Rule: The min rule is derived by bounding the product of posteriori probabilities.

Here, the input pattern is assigned to class c such that

c = argmaxj min P(wj|;f3’,-).
I

Prabhakar and Jain [37] argue that the assumption of statistical independence of the

feature sets may not be true in a multimodal biometric system that uses different feature

26

representations and different matching algorithms on the same biometric trait. They pro-
pose a scheme based on non-parametric density estimation for combining the scores ob-
tained from four ﬁngerprint matching algorithms and use the likelihood ratio test to make
the ﬁnal decision. They show that their scheme is optimal in the Neyman-Pearson decision
sense, when sufﬁcient training data is available to estimate the joint densities.

The use of Bayesian statistics in combining the scores of different biometric matchers
was demonstrated by Bigun et al. [38]. They proposed a new algorithm for the fusion
module of a multimodal biometric system that takes into account the estimated accuracy
of the individual classiﬁers during the fusion process. They showed that their multimodal
system using image and speech data provided better recognition results than the individual
modalities.

The combined matching score can also be computed as a weighted sum of the matching
scores of the individual matchers [16,33]. Jain and Ross [39] have proposed the use of user-
speciﬁc weights for computing the weighted sum of scores from the different modalities.
The motivation behind this idea is that some biometric traits cannot be reliably obtained
from a small segment of the population. For example, we cannot obtain good quality
ﬁngerprints from users with dry ﬁngers. For such users, assigning a lower weight to the
ﬁngerprint score and a higher weight to the scores of the other modalities reduces their
probability of being falsely rejected. This method requires learning of user-speciﬁc weights
from the training scores available for each user. In [39], user-speciﬁc thresholds was also

suggested.

2.5 Evaluation of Multimodal Biometric Systems

The performance metrics of a biometric system such as accuracy, throughput, and scal-

ability can be estimated with a high degree of conﬁdence only when the system is tested on

27

a large representative database. For example, face [13] and ﬁngerprint [12] recognition sys-
tems have been evaluated on large databases (containing samples from more than 25,000 in-
dividuals) obtained from a diverse population under a variety of environmental conditions.
In contrast, current multimodal systems have been tested only on small databases contain-
ing fewer than 1, 000 individuals. Further, multimodal biometric databases can be either
true or virtual. In a true multimodal database (e.g., XM2VTS database [40]), different
biometric cues are collected from the same individual. Virtual multimodal databases con-
tain records which are created by consistently pairing a user from one unimodal database
with a user from another database. The creation of virtual users is based on the assump-
tion that different biometric traits of the same person are independent. This assumption of
independence of the various modalities has not been explicitly investigated till date. How-
ever, Indovina et al. [41] attempted to validate the use of virtual subjects. They randomly
created 1, 000 sets of virtual users and showed that the variation in performance among
these sets was not statistically signiﬁcant. Recently, NIST has released a true multimodal

database [42] containing the face and ﬁngerprint matching scores of 517 individuals.

2.6 Summary

We have presented a detailed discussion on the various approaches that have been pro-
posed for integrating evidence obtained from multiple cues in a biometric system. Figure
2.4 presents a high-level summary of these information fusion techniques. Most of research
work on fusion in multimodal biometric systems has focused on fusion at the matching
score level. In particular, the combination approach to score level fusion has received con-
siderable attention. However, there are still many open questions that have been left unan-
swered. There is no standard technique either for converting the scores into probabilities

or for normalizing the scores obtained from multiple matching algorithms. A systematic

28

 

I Information Fusion in Biometrics I

I l

I Prior to matching I I After matching I
l

l

I Sensor Level J I Feature Level I

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Dynamic Classifier
Raw Data Feature Sets massing, Fusion
Selection
i) Weighted summation
ii) Concatenation
I Confidence Level I I Rank Level I IAbstract Level I
Matching Scores Class Ranks Class Labels
I I i) Highest Flank i) Majority Voting
Classification Approach Combination Approach ii) Borda Count ii) Behavior Knowledge
iii) Logistic Space
Regression III) Dempster Shafer
i) Neural Networks i) Normalization + Theory of Evidence
ii) k. NN Weighted Sum + iv) AND Rule
iii) Decision Trees ) Lhreshloldlng v) OR Rule
. ii ormaization +
“0 SVM (Sum, Product.
Max, Min} Rules

+ Thresholding

Figure 2.4: Summary of approaches to information fusion in biometric systems.

29

evaluation of the different normalization techniques is not available. Further, most of the
score level fusion techniques can be applied only when the individual modalities can pro-
vide a reasonably good recognition performance. They cannot handle less reliable (soft)
biometric identiﬁers that can provide some amount of discriminatory information, but are
not sufﬁcient for recognition of individuals. For example, the height of a user gives some
indication on who the user could be. However, it is impossible to identify a user just based

on his height. Currently, there is no mechanism to deal with such soft information.

30

CHAPTER 3

Score Normalization in Multimodal Biometric Systems

Consider a multimodal biometric veriﬁcation system that follows the combination ap-
proach to fusion at the matching score level. The theoretical framework developed by
Kittler et al. in [36] can be applied to this system only if the output of each modality is of
the form P(genuine|X) i.e., the posteriori probability of user being “genuine” given the
input biometric sample X. In practice, most biometric systems output a matching score 3.
Verlinde et al. [43] have proposed that the matching score 3 is related to P(genuine|X) as

follows:

8 = f (P(genuine|X)) + 17(X), (3.1)

where f is a monotonic function and r] is the error made by the biometric system that
depends on the input biometric sample X. This error could be due to the noise intro-
duced by the sensor during the acquisition of the biometric signal and the errors made
by the feature extraction and matching processes. If we assume that n is zero, it is rea-
sonable to approximate P(genuine|X) by P(genuz’ne|s). In this case, the problem re-
duces to computing P(genuine|s) and this requires estimating the conditional densities
P(s| genuine) and P(s|impostor). Snelick et al. [44] assumed a normal distribution for
the conditional densities of the matching scores (p(8I genuine) ~ N (#9, 0'9) and
p(s|imp03t0r) ~ N01,, 00), and used the training data to estimate the parameters
#9, 0’9, [15, and 02-. The posteriori probability of the score being that of a genuine user was

then computed as,

31

p(s|genuine)

 

P ' = .
(genuznels) p(s|genuine) + p(s|z'mpostor)

The above approach has two main drawbacks. The assumption of a normal distribution
for the scores may not be true in many cases. For example, the scores of the ﬁngerprint
and hand-geometry matchers used in our experiments do not follow a normal distribution.
Secondly, the approach does not make use of the prior probabilities of the genuine and
impostor users that may be available to the system. Due to these reasons, we have proposed
the use of a non-parametric technique, viz., Parzen window density estimation method
[25], to estimate the actual conditional density of the genuine and impostor scores. After
estimating the conditional densities, the Bayes formula can be applied to calculate the

posteriori probability of the score being that of a genuine user. Thus,

p(s|genuine) * P(g)
p(8)

 

P(genuine|s) =

7

where p(s) = @(sIgenuine) * P(g) + p(s|impost0r) * P(i)) and P(g) and P(i) are the
prior probabilities of a genuine user and an impostor, respectively.

Although the Parzen window density estimation technique signiﬁcantly reduces the
error in the estimation of P(genuine|s) (especially when the conditional densities are non-
Gaussian), the density estimation still has inaccuracies non-zero due to the ﬁnite training
set and the problems in choosing the optimum window width during the density estimation
process. Further, the assumption that the value of n in equation (3.1) is zero is not valid in
most practical biometric systems. Since 1) depends on the input biometric sample X, it is
possible to estimate 77 only if the biometric system outputs a conﬁdence measure (that takes
into account the nature of the input X) on the matching score along with the matching score
itself. In the absence of this conﬁdence measure, the calculated value of P(genuz’ne|s) is

not a good estimate of P(genuine|X) and this can lead to poor recognition performance

32

of the multimodal system. Hence, when the outputs of individual modalities are matching
scores without any measures quantifying the conﬁdence on those scores, it would be better

to combine the matching scores directly without converting them into probabilities.

3.1 Need for Score Normalization

The following issues need to be considered prior to combining the scores of the match-
ers into a single score. The matching scores at the output of the individual matchers may
not be homogeneous. For example, one matcher may output a distance (dissimilarity) mea-
sure while another may output a proximity (similarity) measure. Further, the outputs of the
individual matchers need not be on the same numerical scale (range). Finally, the match-
ing scores at the output of the matchers may follow different statistical distributions. Due
to these reasons, score normalization is essential to transform the scores of the individual
matchers into a common domain prior to combining them. Score normalization is a critical
part in the design of a combination scheme for matching score level fusion.

Figure 3.1 shows the conditional distributions of the face, ﬁngerprint and hand-geometry
matching scores used in our experiments. The scores obtained from the face and hand-
geometry matchers are distance scores and those obtained from the ﬁngerprint matcher are
similarity scores. One can easily observe the non-homogeneity in these scores and the need

for normalization prior to any meaningful combination.

3.2 Challenges in Score Normalization

Score normalization refers to changing the location and scale parameters of the match-
ing score distributions at the outputs of the individual matchers, so that the matching scores
of different matchers are transformed into a common domain. When the parameters used

for normalization are determined using a ﬁxed training set, it is referred to as ﬁxed score

33

0.2 ~ -
—Genuine Scores
- - - lmpostor Scores

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

0.15» ‘
3‘.
‘c'i 0.1
0.05
I . - ‘ ‘
G A ‘ Q- .-
0 100 200 300
Raw Face Score (y)
(a)
1 V .
—Genuine Scores
- - - Impostor Scores
08> ’
l
I
0.6!
3 i
Q I
0.4:
I
I
I
0.22
I
I
0 :n- M A A-
0 200 400 600 800 1000
Raw Fingerprint Score (y)
(b)
0.2 * F
—Genuine Scores
- - - lmpostor Scores
0.15 I
g 0.1
0.05’
co 200 460 “860 800 1000
Raw Hand—geometry Score (y)
(C)

Figure 3.1: Conditional distributions of genuine and impostor scores used in our experi-
ments; (a) Face (distance score), (b) Fingerprint (similarity score) and (c) Hand-geometry
(distance score).

34

normalization [45]. In such a case, the matching score distribution of the training set is
examined and a suitable model is chosen to ﬁt the distribution. Based on the model, the
normalization parameters are determined. In adaptive score normalization, the normaliza-
tion parameters are estimated based on the current feature vector. This approach has the
ability to adapt to variations in the input data such as the change in the length of the speech
signal in speaker recognition systems.

The problem of score normalization in multimodal biometric systems is identical to the
problem of score normalization in metasearch. Metasearch is a technique for combining the
relevance scores of documents produced by different search engines, in order to improve
the performance of document retrieval systems [46]. Min-max normalization and z-score
normalization are some of the popular techniques used for relevance score normalization in
metasearch. In metasearch literature [47], the distribution of scores of relevant documents
is generally approximated as a Gaussian distribution with a large standard deviation while
that of non-relevant documents is approximated as an exponential distribution. In our ex—
periments, the distributions of the genuine and impostor ﬁngerprint scores closely follow
the distributions of relevant and non-relevant documents in metasearch. However, the face
and hand-geometry scores do not exhibit this behavior.

For a good normalization scheme, the estimates of the location and scale parameters
of the matching score distribution must be robust and efﬁcient. Robustness refers to in-
sensitivity to the presence of outliers. Efﬁciency refers to the proximity of the obtained
estimate to the optimal estimate when the distribution of the data is known. Huber [48] ex-
plains the concepts of robustness and efﬁciency of statistical procedures. He also explains
the need for statistical procedures that have both these desirable characteristics. Although
many techniques can be used for score normalization, the challenge lies in identifying a

technique that is both robust and efﬁcient.

35

3.3 Normalization Techniques

The simplest normalization technique is the Min-max normalization. Min-max normal-
ization is best suited for the case where the bounds (maximum and minimum values) of the
scores produced by a matcher are known. In this case, we can easily shift the minimum
and maximum scores to O and 1, respectively. However, even if the matching scores are
not bounded, we can estimate the minimum and maximum values for a set of matching
scores and then apply the min-max normalization. Let sij denote the 3"" matching score
output by the i“ modality, wherei = 1,2, - -- ,R andj = 1,2, - -- ,M (R is the number
of modalities and M is the number of matching scores available in the training set). The

min-max normalized score for the test score 35,, is given by

, sik — min ({3,})

3““ _ max ({Si.}) _ min ({si'D,

 

where {3,} = {3,1, 3,2, - - - , 3iM}- When the minimum and maximum values are estimated
from the given set of matching scores, this method is not robust (i.e., the method is highly
sensitive to outliers in the data used for estimation). Min-max normalization retains the
original distribution of scores except for a scaling factor and transforms all the scores into
a common range [0,1]. Distance scores can be transformed into similarity scores by sub-
tracting the normalized score from 1. Figure 3.2 shows the distributions of face, ﬁngerprint
and hand-geometry scores after min-max normalization.

Decimal scaling can be applied when the scores of different matchers are on a loga-
rithmic scale. For example, if one matcher has scores in the range [0,1] and the other has

scores in the range [0, 100], the following normalization could be applied.

I _ Sik
31* _ 10n’

36

0.35

 

 

—Genulne Scores
0.3 - - -lmpostor Scores

0.25 l

 

 

 

0.2’

PM

015*
0.1]
0.05» "

 

 

 

Go ‘02 ofa 0.6 ofe 1
Min-max Normalized Face Score (y)

 

 

 

 

 

 

(a)
1 . V
l — Genuine Scores
| - - ..
0.8] lmpostor Scores 4
l
I
0.6 1
S I
Q I
0.41 *
I
I
0.2 *i
I

 

 

 

0o 04.2 0.71 04.6 0.8 1
Min-max Normalized Fingerprint Score (y)

(b)

 

 

 

 

0.3 - - - lmpostor Scores

 

0.35 - . '
r— Genuine Scores [1

0.25‘ 4

0.2' ‘

PM

 

0.15
0.1*

005*

 

 

O
_. -_---_-‘

o 0.2 0.4 ole 0.8 1
Min-max Normalized Handgeometry Score (y)

(c)

 

 

Figure 3.2: Distributions of genuine and impostor scores after min-max normalization; (a)
Face, (b) Fingerprint and (c) Hand-geometry.

37

where n = log10 max ({5}; }). The problems with this approach are the lack of robustness
and the assumption that the scores of different matchers vary by a logarithmic factor. In our
experiments, the matching scores of the three modalities are not distributed on a logarithmic
scale and hence, this normalization technique cannot be applied.

The most commonly used score normalization technique is the z-score that uses the
arithmetic mean and standard deviation of the given data. This scheme can be expected
to perform well if prior knowledge about the average score and the score variations of
the matcher is available. If we do not have any prior knowledge about the nature of the
matching algorithm, then we need to estimate the mean and standard deviation of the scores

from a given set of matching scores. The normalized scores are given by
8:]: : Sik — H,
0

where p. is the arithmetic mean and o is the standard deviation of the given data. However,
both mean and standard deviation are sensitive to outliers and hence, this method is not
robust. Z-score normalization does not guarantee a common numerical range for the nor-
malized scores of the different matchers. If the distribution of the scores is not Gaussian,
z-score normalization does not retain the input distribution at the output. This is due to the
fact that mean and standard deviation are the optimal location and scale parameters only
for a Gaussian distribution. For an arbitrary distribution, mean and standard deviation are
reasonable estimates of location and scale, respectively, but are not optimal.

The distributions of the matching scores of the three modalities after z-score normal-
ization are shown in Figure 3.3. The face and hand-geometry scores are converted into

similarity scores by subtracting the scores from a large number (300 for face and 1000 for

38

hand-geometry in our experiments) before applying the z-score transformation. Figure 3.3
shows that z-score normalization fails to transform the scores of the different modalities
into a common numerical range and also does not retain the original distribution of scores
in the case of ﬁngerprint modality.

The median and median absolute deviation (MAD) are insensitive to outliers and the
points in the extreme tails of the distribution. Hence, a normalization scheme using median

and MAD would be robust and is given by

, 3,7, — median

Sik— MAD 7

 

where MAD = median ({lsi, — median ({s,.}) |}). However, the median and
the MAD estimators have a low efﬁciency compared to the mean and the standard deviation
estimators, i.e., when the score distribution is not Gaussian, median and MAD are poor
estimates of the location and scale parameters. Therefore, this normalization technique
does not retain the input distribution and does not transform the scores into a common
numerical range. This is illustrated by the distributions of the normalized face, ﬁngerprint,
and hand-geometry scores in Figure 3.4.

Cappelli et al. [49] have used a double sigmoid function for score normalization in a
multimodal biometric system that combines different ﬁngerprint matchers. The normalized

score is given by

1

s- —t
1+exp(—2(-1§l—

 

)) ifsk < t,

 

. _ otherwise
1+eXP(-2(-”S—sr2 ‘)) ’
where t is the reference operating point and TI and r2 denote the left and right edges of

the region in which the function is linear, i.e., the double sigmoid function exhibits linear

39

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

0.2 . v . ,
—Genuine Scores
- - -Impostor Scores
0.15» ‘
9.
K 0 1
0.05 ‘
G -471 ii» 0
Z—soore Normalized Face Score (y)
(a)
0.7 - a .
—Genuine Scores
06’ : - - - lmpostor Scores 4
I
0.5 i
I
o 4 5
S ' .1
m0.3» ;: l
.I
0.2 5
.I
. I
0.1 g: ‘
M
95 0 10 5 20
Z-soore Normalized Flngerprlnt goore (y)
(b)
0.25 . . -
—Genuine Scores
02, - - -lmpostor Scores
015* ‘
3‘.
K
0.1 »
0.05
9e -4 —2 o 2
Z—soore Normalized Hand-geometry Score (y)

(C)

Figure 3.3: Distributions of genuine and impostor scores after z-score normalization; (a)
Face, (b) Fingerprint, and (c) Hand-geometry.

40

0.14

 

 

 

 

 

 

—Genuine Scores
012* - - -lmpostor Scores
0.1 .
A 0.08»
b
a 0.06»
0.04)
'olo‘.
0.02L ,o’
c “‘O‘ M A ‘M

 

 

 

.4 —é 0 2 4
Median-MAD Normalized Face Score (y)

 

 

 

 

 

 

 

 

 

 

 

 

 

(a)
1 v - r
—Genuine Scores
i - - - lmpostor Scores
0.8) : ‘
I
I
I
0.6» :
3 I
a :
0.4» t
I
I
I
02 > : 1
I
I
I
L A A- .. A_
12:00 0 200 400 600 800 1000
Median-MAD Normalized Fingerprint Score (y)
(b)
0.2
015*
g 0.1 »
0.05’
.92 :9 is —3 0 3
Median-MAD Nomalized Hand-geometry Score (y)
(C)

Figure 3.4: Distributions of genuine and impostor scores after median-MAD normalization;
(a) Face, (b) Fingerprint and (c) Hand-geometry.

41

characteristics in the interval (t — r1, t — r2). Figure 3.5 shows an example of the double
sigmoid normalization, where the scores in the [0, 300] range are mapped to the [0, 1] range

usingt = 200, r, = 20 and r2 = 30.

 

    
  

.0
a:

p——-.——-——-——--———-—— —--———————_‘

.0
rs.

Normalized Score

1

 

 

 

. . 1E .
0 50 100 150 200 250 300
Original Score

Figure 3.5: Double sigmoid normalization (t = 200, r1 = 20, and r2 = 30).

This scheme transforms the scores into the [0, 1] interval. But, it requires careful tuning
of the parameters t, 1'1 and r2 to obtain good efﬁciency. Generally, t is chosen to be some
value falling in the region of overlap between the genuine and impostor score distribution,
and r1 and r2 are set so that they correspond to the extent of overlap between the two dis-
tributions toward the left and right of t, respectively. This normalization scheme provides
a linear transformation of the scores in the region of overlap, while the scores outside this
region are transformed non-linearly. The double sigmoid normalization is very similar to

the min-max normalization followed by the application of a two-quadrics (QQ) or a logistic

42

(LG) function as suggested by Snelick et al. [18]. When n and T2 are large, the double sig-
moid normalization closely resembles the QQ-min-max normalization. On the other hand,
we can make the double sigmoid normalization tend toward LG-min-max normalization by
assigning small values to 7'1 and r2.

Figure 3.6 shows the face, ﬁngerprint and hand-geometry score distributions after dou-
ble sigmoid normalization. The face and hand-geometry scores are converted into simi—
larity scores by subtracting the normalized scores from 1. The parameters of the double
sigmoid normalization were chosen as follows: t is chosen to be the center of the overlap-
ping regions between the genuine and impostor score distributions, and 7‘1 and r2 are set so
that they correspond to the minimum genuine similarity score and maximum impostor sim-
ilarity score, respectively. A matching score that is equally likely to be from a genuine user
and an impostor is chosen as the center (t) of the region of overlap. Then, TI is the differ-
ence between t and the minimum of the genuine scores, while r2 is the difference between
the maximum of the impostor scores and t. In order to make this normalization robust,
approximately 2% of the scores at the extreme tails of the genuine and impostor distribu-
tions were omitted when calculating r1 and r2. It must be noted that this scheme cannot
be applied as described here if there are multiple intervals of overlap between genuine and
impostor distributions. Although this normalization scheme transforms all the scores to a
common numerical range [0, 1], it does not retain the shape of the original distribution of
the ﬁngerprint scores.

The tanh-estimators introduced by Hampel et al. [50] are robust and highly efﬁcient.

The normalization is given by

1 ,- —
32k = — {tanh (0.01 (w)) + 1},
2 00H

43

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

0.4 . .
—Genuine Scores
- - - lmpostor Scores
0.31
’9.
E”. 0 2
0.1 i
- u .- - q - O; .
G0 0.2 0.4 0.6 0.8 1
Sigmoid Normalized Face Score (y)
(a)
1 v . v
—Genuine Scores
0.8 . - - - lmpostor Scores
:
I
0.6 ii
3 l
a I
0.4 >1
I
I
I
0.2. l
I
I
c L.
0 .2 0.4 0.6 .8 1
Sigmo?d Normalized Fingerprint gcore (y)
(b)
0.5 ~ . -
—Genuine Scores
04. - - - lmpostor Scores
0.3>
’>.‘
‘c'i
0.2’
0.1 i I
I
I
o f ' . ' .I
G0 0.2 0.4 0.8 0.8 1
Sigmoid Normalized Handgeometry Score (y)

(C)

Figure 3.6: Distributions of genuine and impostor scores after double sigmoid normaliza-
tion; (a) Face, (b) Fingerprint and (c) Hand-geometry.

where pay and 00” are the mean and standard deviation estimates, respectively, of the
genuine score distribution as given by Hampel estimators ‘. Hampel estimators are based

on the following inﬂuence (rm-function:

u 0 S In] < a,
a =1: sign(u) a S M < b,

Wu) = a :1: sign(u) =1: (Sell-11:1) b _<_ |u| < c,
0 In] _>_ c.

A plot of the Hampel inﬂuence function is shown in Figure 3.7. The Hampel inﬂuence
function reduces the inﬂuence of the points at the tails of the distribution (identiﬁed by a,
b, and c) during the estimation of the location and scale parameters. Hence, this method
is not sensitive to outliers. If many of the points that constitute the tail of the distributions
are discarded, the estimate is robust but not efﬁcient (optimal). On the other hand, if all the
points that constitute the tail of the distributions are considered, the estimate is not robust
but the efﬁciency increases. Therefore, the parameters a, b, and c must be carefully chosen
depending on the amount of robustness required which in turn depends on the estimate of
the amount of noise in the available training data.

In our experiments, the values of a, b and c were chosen such that 70% of the scores
were in the interval (m — a, m + a), 85% of the scores were in the interval (m — b, m + b),
and 95% of the scores were in the interval (m - c, m + c), where m is the median score. The
distributions of the scores of the three modalities after tanh normalization are shown in Fig-
ure 3.8. The distance to similarity transformation is achieved by subtracting the normalized
scores from 1. The nature of the tanh distribution is such that the genuine score distribution
in the transformed domain has a mean of 0.5 and a standard deviation of approximately

0.01. The constant 0.01 in the expression for tanh normalization determines the spread
lIn [44,45], the mean and standard deviation of all the training scores (both genuine and impostor) were

used for tanh normalization. However, we observed that considering the mean and standard deviation of only
the genuine scores results in a better recognition performance.

45

 

0.8 1 7 r

   
 

05'
04*

02

 

 

 

’3 ____________
9 0
-02~
~04-
-06*
-08 ’ ‘ *
—1 -05 0 05 1
U

Figure 3.7: Hampel inﬂuence function (a = 0.7, b = 0.85, and c = 0.95).

of the normalized genuine scores. In our experiments, the standard deviation of genuine
scores of face, ﬁngerprint and hand-geometry modalities are 16.7, 202.1, and 38.9, respec-
tively. We observe that the genuine ﬁngerprint scores have a standard deviation that is ap-
proximately 10 times the standard deviation of the genuine face and hand-geometry scores.
Hence, using the same constant, 0.01, for the ﬁngerprint modality is not inappropriate. To
avoid this problem, the constant factor in the tanh normalization for ﬁngerprint modality
was set to 0.1. Therefore, the standard deviation of the tanh normalized genuine ﬁngerprint
scores is roughly 0.1, which is about 10 times that of the face and hand-geometry modali-
ties. This modiﬁcation retains the inforrnation contained in the ﬁngerprint scores even after
the normalization, resulting in better performance.

Mosteller and Tukey [51] introduced the biweight location and scale estimators that are
robust and efﬁcient. But, the biweight estimators are iterative in nature (an initial estimate

of the biweight location and scale parameters is chosen, and this estimate is updated based

46

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

—Genuine Scores
02, ---lmpostor Scores
0.15»
3?
E”.
0.1? ‘
005» ,."~ i
83- 0.4 A. 0.3 0.55
Q'anh Normalizeg 19309 Score (y)
(a)
1 . . . . . f .
: —Genuine Scores
---l st 300
0.8» : mp0 or f6$4
:
0.6) :
a I
a: :
0.4» - 1
I
I
I
0.2» l 1
I
I
.WA . - .
(935 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75
Tanh Normalized Fingerprint Score (y)
(b)
0.4 . -
03*
’9.
E 0 2
0.1'

 

 

 

 

 

 

8.2 0.3 ‘04 0.5 06 0.7
Tanh Normalized Hand-geometry Score (y)

(C)

Figure 3.8: Distributions of genuine and impostor scores after tanh normalization; (a) Face,
(b) Fingerprint and (c) Hand-geometry.

47

on the training scores), and are applicable only for Gaussian data. The biweight location
and scale estimates of the data used in our experiments were very close to the mean and
standard deviation. Hence, the results of this scheme were quite similar to those produced
by the z-score normalization. Therefore, we have not considered biweight normalization
in our experiments. The characteristics of the different normalization techniques have been

tabulated in Table 3.1.

Table 3.1: Summary of Normalization Techniques

 

 

 

 

 

 

 

 

 

Normalization Technique Robustness Efﬁciency
Min-max No N/A
Decimal scaling No N/A

z-score No High (optimal for Gaussian data)

Median and MAD Yes Moderate
Double sigmoid Yes High
tanh-estimators Yes High
Biweight estimators Yes High

 

 

 

 

3.4 Experimental Results

Snelick et al. [18] have developed a general testing framework that allows system
designers to evaluate multimodal biometric systems by varying different factors like the
biometric traits, matching algorithms, normalization schemes, fusion methods and sam-
ple databases. To illustrate this testing methodology, they evaluated the performance of
a multimodal biometric system that used face and ﬁngerprint classiﬁers. Normalization
techniques like min-max, z-score, median and MAD, and tanh estimators were used to
transform the scores into a common domain. The transformed scores were then combined

using fusion methods like simple sum of scores, maximum score, minimum score, sum of

48

 

posteriori probabilities (sum rule), and product of posteriori probabilities (product rule).
Their experiments conducted on a database of more than 1, 000 users showed that the min-
max normalization followed by the sum of scores fusion method generally provided better
recognition performance than other schemes. However, the reasons for such a behavior
have not been presented by these authors. In this work, we have tried to analyze the rea-
sons for the differences in the performance of the different normalization schemes. We
have tried to systematically study the different normalization techniques to ascertain their
role in the performance of a multimodal biometric system consisting of face, ﬁngerprint
and hand-geometry modalities. In addition to the four normalization techniques employed

in [18], we have also analyzed the double sigmoid method of normalization.

3.4.1 Generation of the Multimodal Database

The multimodal database used in our experiments was constructed by merging two sep-
arate databases (of 50 users each) collected using different sensors and over different time
periods. The ﬁrst database (described in [16]) was constructed as follows: Five face images
and ﬁve ﬁngerprint impressions (of the same ﬁnger) were obtained from a set of 50 users.
Face images were acquired using a Panasonic CCD camera (640 x 480) and ﬁngerprint
impressions were obtained using a Digital Biometrics sensor (500 dpi, 640 x 480). Five
hand-geometry images were obtained from a set of 50 users (some users were present in
both the sets) and captured using a Pulnix TMC-7EX camera. The mutual independence
assumption of the biometric traits allows us to randomly pair the users from the two sets.
In this way, a multimodal database consisting of 50 virtual users was constructed, each user
having ﬁve biometric templates for each modality. The biometric data captured from every
user is compared with that of all the users in the database leading to one genuine score

vector and 49 impostor score vectors for each distinct input. Thus, 500 (50 x 10) genuine

49

 

score vectors and 24, 500 (50 x 10 x 49) impostor score vectors were obtained from this
database. The second database also consisted of 50 users whose face images were cap-
tured using a Sony video camera (256 x 384) and ﬁngerprint images were acquired using
an Identix sensor (500 dpi, 255 x 256). The Pulnix TMC-7EX camera was used to obtain
hand-geometry images. This database also gave rise to 500 genuine and 24,500 impostor
score vectors. Merging the scores from the two databases resulted in a database of 100
users with 1, 000 genuine score vectors and 49, 000 impostor score vectors. A score vector
8;, is a 3-tuple {31h 32k, 831:}, where 81k, 32k. and 83k correspond to the kth matching
scores obtained from the face, ﬁngerprint and hand-geometry matchers, respectively. Of
the 10 genuine and 10 x 49 impostor score vectors available for each user, 6 genuine and
6 impostor score vectors were randomly selected and used for training (for calculating the
parameters of each normalization technique or for density estimation by the Parzen win-
dow method). The remaining 4 genuine and 4 x 49 impostor score vectors of each user
were used for testing the performance of the system. Again assurrring the independence of
the three modalities, we create 43 “yirtual” users from each real user, by considering all
possible combinations of scores of the three modalities. Thus, we have 64, 000 (100 x 43)
genuine score vectors and 313, 600 (100 x 43 x 49) impostor score vectors to analyze the
system performance. This separation of the database into training and test sets was repeated
40 times and we have reported the average performance results. Fingerprint matching was
done using the minutiae features [52] and the output of the ﬁngerprint matcher was a simi-
larity score. Eigenface coefﬁcients were used to represent features of the face image [53].
The Euclidean distance between the eigenface coefﬁcients of the face template and that
of the input face was used as the matching score. The hand-geometry images were repre-
sented by a l4-dimensional feature vector [54] and the matching score was computed as

the Euclidean distance between the input feature vector and the template feature vector.

50

 

The recognition performance of the face, ﬁngerprint, and hand-geometry systems when
operated as unimodal systems is shown in Figure 3.9. From Figure 3.1(a), we observe that
there is a signiﬁcant overlap between the genuine and impostor distributions of the raw
face scores, and this explains the poor recognition performance of the face module. Fig-
ure 3.1(b) shows that most of the impostor ﬁngerprint scores are close to zero and that the
genuine ﬁngerprint scores are spread over a wide range of values. Moreover, the overlap
between the two conditional densities is small and, hence, the ﬁngerprint system performs
better than the face and hand-geometry modules. The overlap between the genuine and im-
postor distributions of the hand-geometry system is the highest among all the three modal-
ities as shown in Figure 3.1(c). Hence, the hand geometry based recognition performance

is low compared to the ﬁngerprint and face matchers.

100 .- ......, . ..f...” - . ....., . . ....-

 

 

Genuine Acceptance Rate - GAR (%)
8

 

 

 

50 ~
40 - - - Face ~
— Fingerprint
------ Handgeometry
30 ~ .
20 I“.: . ....rrl r A A ..rxrl r gmmmamml . . . it...
10’3 10'2 10“ 10° 10‘

False Acceptance Rate - FAR (%)

Figure 3.9: ROC curves for individual modalities.

51

3.4.2 Impact of Normalization on Fusion Performance

The performance of the multimodal biometric system has been studied under differ-
ent normalization and fusion techniques. The simple sum of scores, the max-score, and
the min-score fusion methods described in [18] were applied on the normalized scores.
The normalized scores were obtained by using one of the following techniques: simple
distance-to-similarity transformation with no change in scale (STrans), min-max normal-
ization (Minmax), z-score normalization (ZScore), median-MAD normalization (Median),
double sigmoid normalization (Sigmoid), tanh normalization (Tanh), and Parzen normal-
ization (Parzen)? Table 3.2 summarizes the average (over 40 trials) Genuine Acceptance
Rate (GAR) of the multimodal system along with the standard deviation of the GAR (shown
in parentheses) for different normalization and fusion schemes, at a False Acceptance Rate

(FAR) of0.1%.

Table 3.2: Genuine Acceptance Rate (GAR) (%) of different normalization and fusion
techniques at the 0.1% False Acceptance Rate (FAR). Note that the values in the table
represent average GAR, and the values indicated in parentheses correspond to the standard
deviation of GAR.

 

 

 

h.‘

 

 

 

 

 

 

 

 

 

 

 

Normalization Fusion Techniques
Techniques Sum of scores Max-score Min-score
STrans 98.3 (0.4) 46.7 (2.3) 83.9 (1.6)
Minmax 97.8 (0.6) 67.0 (2.5) 83.9 (1.6)
Zscore 98.6 (0.4) 92.1 (1.1) 84.8 (1.6)
Median 84.5 (1.3) 83.7 (1.6) 68.8 (2.2)
Sigmoid 96.5 (1.3) 83.7 (1.6) 83.1 (1.8)
Tanh 98.5 (0.4) 86.9 (1.8) 85.6 (1.5)
Parzen 95.7 (0.9) 93.6 (2.0) 83.9 (1.9)

 

 

2Conversion of matching scores into posteriori probabilities by the Parzen window method is really not a
nomralization technique. However, for the sake of convenience we refer to this method as Parzen normaliza-
tion. In the case of this method, the simple sum of scores, max score, and min score fusion schemes, reduce
to the sum rule, max rule, and rrrin rule described in [36], respectively.

52

I 1 .
...—av—

Figure 3.10 shows the recognition performance of the system when the scores are com-
bined using the sum of scores method. We observe that a multimodal system employing
the sum of scores method provides better performance than the best unimodal system (ﬁn-
gerprint in this case) for all normalization techniques except median-MAD normalization.
For example, at a FAR of 0.1%, the GAR of the ﬁngerprint module is about 83.6%, while
that of the multimodal system is high as 98.6% when z-score normalization is used. This

improvement in performance is signiﬁcant and it underscores the beneﬁt of multimodal

systems.

100

 

95

90

85

 

Genuine Acceptance Rate - GAR (%)

 

 

 

80L J
75: ................... ‘ .

70 A A A A LLLLI L A A A A AAA] A A A A A AAAI A 4 A A AAAA
10-3 10’2 10" 10° 10‘

False Acceptance Rate - FAR (%)

Figure 3.10: ROC curves for sum of scores fusion method under different normalization
schemes.

53

 

Among the various normalization techniques, we observe that the tanh and min-max
normalization techniques outperform other techniques at low FARs. At higher FARs,
z-score normalization provides slightly better performance than tanh and min-max nor-
malization. In a multimodal system using the sum of scores fusion method, the com-
bined score (81;) is just a linear transformation of the score vector {811“ 82k, 83k}, i.e.,
8],; = (alslk - 01) + (0232;, - 02) + (a383k - 03), where 81k, 321:, and 83k cor-
respond to the 10‘" matching scores obtained from the face, ﬁngerprint and hand-geometry
matchers, respectively. The effect of different normalization techniques is to deterrrrine the
weights a1, a2, and a3, and the biases bl, 02, and b3. Since the MAD of the ﬁngerprint
scores is very small compared to that of face and hand-geometry scores, the median-MAD
normalization assigns a much larger weight to the ﬁngerprint score (a2 > > a1, (13). This
is a direct consequence of the moderate efﬁciency of the median-MAD estimator. The dis-
tribution of the ﬁngerprint scores deviates drastically from the Gaussian assumption and,
hence, median and MAD are not the right measures of location and scale, respectively.
In this case, the combined score is approximately equal to the ﬁngerprint score and the
performance of the multimodal system is close to that of the ﬁngerprint module. On the
other hand, min-max normalization, z-score normalization, tanh and distance-to-similarity
transformation assign nearly optimal weights to the three scores. Therefore, the recognition
performance of the multimodal system when using one of these techniques along with the
sum of scores fusion method is signiﬁcantly better than that of the ﬁngerprint matcher. The
difference in performance between the min-max, z-score, tanh and distance-to-similarity
transformation is relatively small. However, it should be noted that the raw scores of the
three modalities used in our experiments are comparable and, hence, a simple distance-
to-sinrilarity conversion works reasonably well. If the scores of the three modalities are

signiﬁcantly different, then this method will not work.

54

The performance of the multimodal system using max-score fusion is shown in Figure
3.11. Here, z-score and Parzen normalization provide better recognition performance com-
pared to that of the ﬁngerprint matcher. In a max-score fusion method that uses only the
distance-to-similarity transformation, the hand-geometry scores begin to dominate. There-
fore, the performance is only slightly better than the hand-geometry module. For min-max
normalization, the face and hand-geometry scores are comparable and they dominate the
ﬁngerprint score. This explains why the performance of the multimodal system is close
to that of the face recognition system. When median-MAD, tanh, and double sigmoid
normalization are used, the ﬁngerprint scores are much higher compared to the face and
hand-geometry scores. This limits the performance of the system close to that of the ﬁn-
gerprint module. In z-score normalization, the scores of the three modules are comparable
and, hence, the combined score depends on all the three scores and not just the score of
one modality. This improves the relative performance of the max-score fusion method
compared to other normalization methods. Finally, Parzen normalization followed by max-
score fusion accepts the user even if one of the three modalities produces a high estimate of
the posteriori probability and rejects the user only if all the three modalities make errors in
the probability estimation process. Hence, this method has a high genuine acceptance rate.

Figure 3.12 shows the performance of the multimodal system when min-score fusion
method is employed. For median-MAD normalization, most of the face scores have smaller
values compared to the ﬁngerprint and hand-geometry scores. Therefore, for median-MAD
normalization the performance of the min-score method is close to the performance of the
face-recognition system. On the other hand, the ﬁngerprint scores have smaller values for
all other normalization schemes and, hence, their performance is very close to that of the

ﬁngerprint matcher.

55

 

 

100

90

80

70

60

50"

 

I

Genuine Acceptance Rate - GAR (%)

 

 

20 L A AAAAAAI A A AAAAAAl A A AAAAAA
10'2 10‘1 10° 10‘

False Acceptance Rate - FAR (%)

 

Figure 3.11: ROC curves for max-score fusion method under different normalization
schemes.

56

 

Genuine Acceptance Rate - GAR (%)

 

 

 

L A A A LLAAL A A A AAAAAL A A A A AAAI‘ A A A A AAAA
4o .2 -1 0 1
10 10 10 10

False Acceptance Rate - FAR (%)

 

Figure 3.12: ROC curves for min-score fusion method under different normalization
schemes.

57

3.4.3 Robustness Analysis of Normalization Schemes

For sum of scores fusion, we see that the performance of a robust normalization tech-
nique like tanh is almost the same as that of the non-robust techniques like min-max and
z-score normalization. However, the performance of such non-robust techniques is highly
dependent on the accuracy of the estimates of the location and scale parameters. The scores
produced by the matchers in our experiments are unbounded and, hence, can theoretically
produce any value. Also, the statistics of the scores (like average or deviation from the av-
erage) produced by these matchers is not known. Therefore, parameters like the minimum
and maximum scores in min-max normalization, and the average and standard deviation
of scores in z-score normalization have to be estimated from the available data. The data
used in our experiments does not contain any outliers and, hence, the performance of the
non-robust normalization techniques were not affected. In order to demonstrate the sensi-
tivity of the min-max and z-score normalization techniques in the presence of outliers, we
artiﬁcially introduced outliers in the ﬁngerprint scores.

For min-max normalization, a single large score whose value is 125%, 150%, 175% or
200% of the original maximum score is introduced into the ﬁngerprint data. Figure 3.13
shows the recognition performance of the multimodal system after the introduction of the
outlier. We can clearly see that the performance is highly sensitive to the maximum score.
A single large score that is twice the original maximum score can reduce the recognition
rate by 3-5% depending on the operating point of the system. The performance degradation
is more severe at lower values of FAR.

In the case of z-score normalization, a few large scores were introduced in the ﬁn-
gerprint data so that the standard deviation of the ﬁngerprint score is increased by 125%,
150%, 175% or 200% of the original standard deviation. In one trial, some large scores

were reduced to decrease the standard deviation to 75% of the original value. In the case

58

Genuine Acceptance Rate - GAR (%)

98

 

I
I
'I
.I
.....
‘I
‘I
.I

0000

.I
I‘.
‘I
I
.I

96

l
I“
‘I

II
I
I
I"
I"
I
.I
‘I

e
I“
\
I
I

I
.I
‘I
I
‘I
I
I
.I
4 I
9 “
I
‘I
I
‘I
I
‘I
......
I ‘I
I ‘I

a“
I
I

I
I
I
I
‘I
I

92 1 .50'TrueMax

 

I
‘I

   
   
 
 

 

 

” I!" ....... 1.75'TrueMax
90 - ~’ ..... ~
2.00'TrueMax
88 ‘
86 A A A _A A L A A A A A
10‘3 10‘2 10

False Acceptance Rate - FAR (%)

Figure 3.13: Robustness analysis of min-max normalization.

59

 

of an increase in standard deviation, the performance improves after the introduction of
outliers as indicated in Figure 3.14. Since the initial standard deviation was small, ﬁn-
gerprint scores were assigned a higher weight compared to the other modalities. As the
standard deviation is increased, the donrination 0f the ﬁngerprint scores was reduced and
this resulted in improved recognition rates. However, the goal of this experiment is to show
the sensitivity of the system to those estimated parameters that can be easily affected by
outliers. A similar experiment was done for tanh-normalization technique and, as shown
in Figure 3.15, there is no signiﬁcant variation in the performance after the introduction of

outliers. This result highlights the robustness of the tanh normalization method.

100 . . ..-f-., - . . ..e.f

 

 

Genuine Acceptance Rate - GAR (%)

 

 

 

82 A A A A A A A A l A A A A A A A A
10’3 10‘2 10"

False Acceptance Rate - FAR (%)

 

Figure 3.14: Robustness analysis of z-score normalization.

60

Genuine Acceptance Rate - GAR (%)

1 00
99
98

 

' 1.5'TrueStd

2'TrueStd
1 .75'TrueStd

 

 

10
False Acceptance Rate - FAR (%)

Figure 3.15: Robustness analysis of tanh normalization.

61

   
     

AAAAI A A AAAAAA

 

10

-1

r-raml 5.1-.1 1, 9W

 

3.5 Summary

Score normalization is an important issue to be dealt with in score level fusion and it
has signiﬁcant impact on the performance of a multibiometric system. We have examined
the effect of different score normalization techniques on the performance of a multimodal
biometric system. We have demonstrated that the normalization of scores prior to combin-
ing them improves the recognition performance of a multimodal biometric system that uses
the face, ﬁngerprint and hand-geometry traits for user authentication. Min-max, z-score,
and tanh normalization techniques followed by a simple sum of scores fusion method re-
sult in a superior GAR than all the other normalization and fusion techniques. We have
shown that both min-max and z-score methods are sensitive to outliers. On the other hand,
tanh normalization method is both robust and eﬂicient. If the location and scale parameters
of the matching scores (minimum and maximum values for min-max, or mean and stan-
dard deviation for z-score) of the individual modalities are known in advance, then simple
normalization techniques like min-max and z—score would sufﬁce. If these parameters are
to be estimated using some training scores, and if the training scores are noisy, then one
should choose a robust normalization technique like the tanh normalization. We have also
explored the use of non-parametric approaches like the Parzen window density estimation
method to convert the matching scores into posteriori probabilities and combining the prob-
abilities using fusion rules. The advantage of this method is that it avoids the need for any
knowledge about the distribution of matching scores. However, the performance of this

method is dependent on the type and width of the kernel used for the estimation of density.

62

F“!

 

CHAPTER 4

Soft Biometrics

Current biometric systems are not perfect (do not have zero error rates) and problems
like noise in the sensed biometric data, non-universality and lack of distinctiveness of the
chosen biometric trait lead to unacceptable error rates in recognizing a person. Using a
combination of biometric identiﬁers like face, ﬁngerprint, hand-geometry and iris makes
the resulting multibiometric system more robust to noise and can alleviate problems such as
non-universality and lack of distinctiveness, thereby reducing the error rates signiﬁcantly.
However, using multiple traits will increase the enrollment and veriﬁcation times, cause
more inconvenience to the users and increase the overall cost of the system. Therefore, we
propose another solution to reduce the error rates of the biometric system without caus-
ing any additional inconvenience to the user. Our solution is based on incorporating soft
identiﬁers of human identity like gender, ethnicity, height, eye color, etc. into a (primary)
biometric identiﬁcation system. Figure 4.1 shows a sample application (ATM kiosk) where
both primary (ﬁngerprint) and soft (gender, ethnicity, height estimated using a camera)

biometric information are utilized to verify the account holder’s identity.

4.1 Motivation and Challenges

The usefulness of soft biometric traits in improving the performance of the primary
biometric system can be illustrated by the following example. Consider three users A
(1.8m tall, male), B (1.7m tall, female), and C (1.6m tall, male) enrolled in a ﬁngerprint
system that works in the identiﬁcation mode. When user A presents his ﬁngerprint sample

X to the system, it is compared to the templates of all the three users stored in the database

63

' "1‘; o’-

'8“..,“A '-

 

- Ca men
1
1

Gender - Female
Ethnicity - Asian
Height - 160 cm

Fingerprint
—‘ Sensor

 

Figure 4.1: An ATM kiosk equipped with a ﬁngerprint (primary biometric) sensor and a
camera to obtain soft attributes (gender, ethnicity and height).

and the posteriori matching probabilities of all the three users given the template X is
calculated. Let us assume that the output of the ﬁngerprint matcher is P(A|X) = 0.42,
P(B|X) = 0.43, and P(C IX ) = 0.15. In this case, the test user will either be rejected due
to the proximity of the posteriori matching probabilities for users A and B, or be falsely
identiﬁed as user B. On the other hand, let us assume that there exists a secondary system
that automatically identiﬁes the gender of the user as male and measures the user’s height
as 1.78m. If we have this information in addition to the posteriori matching probabilities
given by the ﬁngerprint matcher, then a proper combination of these sources of information
will lead to a correct identiﬁcation of the test user as user A.

The ﬁrst biometric system developed by Alphonse M. Bertillon in 1883 used anthro-
pometric features such as the length and breadth of the head and the ear, length of the
middle ﬁnger and foot, height, etc. along with attributes like eye color, scars. and tatoo
marks for ascertaining a person’s identity [55]. Although each individual measurement in

the Bertillon system may exhibit some variability, a combination of all these measurements

 

was sufﬁcient to manually identify a person with reasonable accuracy. The Bertillon sys-
tem was dropped in favor of the Henry’s system of ﬁngerprint identiﬁcation due to three
main reasons: (i) lack of persistence - the anthropometric features can vary signiﬁcantly
over a period of time; (ii) lack of distinctiveness - features such as skin color or eye color
cannot be used for distinguishing between individuals coming from a sirrrilar background;
and (iii) the time, effort, training, and cooperation required to get reliable measurements.
Similar to the Bertillon system, Heckathom et al. [56] used attributes like gender, race, eye
color, height, and other visible marks like scars and tattoos to recognize individuals for the
purpose of welfare distribution. More recently, Ailisto et al. [57] showed that unobtrusive
user identiﬁcation can be performed in non-security applications such as health clubs using
a combination of “light” biometric identiﬁers like height, weight, and body fat percentage.
However, it is obvious that the features used in the above mentioned systems provide some
identity information, but are not sufﬁcient for accurate person recognition. Hence, these
attributes can be referred to as “soft biometric traits”. The soft biometric information com—
plements the identity information provided by traditional (primary) biometric identiﬁers
such as ﬁngerprint, iris, and voice. Therefore, utilizing soft biometric traits can improve
the recognition accuracy of primary biometric systems.

Wayman [58] proposed the use of soft biometric traits like gender and age, for ﬁltering
a large biometric database. Filtering refers to limiting the number of entries in a database
to be searched, based on characteristics of the interacting user. For example, if the user
can somehow be identiﬁed as a middle-aged male, the search can be restricted only to the
subjects with this proﬁle enrolled in the database. This greatly improves the speed or the
search efﬁciency of the biometric system. In addition to ﬁltering, the soft biometric traits
can also be used for tuning the parameters of the biometric system. Studies [59, 60] have

shown that factors such as age, gender, race, and occupation can affect the performance of

65

I1

 

a biometric system. For example, a young female Asian mine-worker is considered as one
of the most difﬁcult subjects for a ﬁngerprint system [60]. This provides the motivation for
tuning the system parameters like threshold on the matching score in a unimodal biometric
system, and thresholds and weights of the different modalities in a multimodal biometric
system to obtain the optimum performance for a particular user or a class of users. How-
ever, ﬁltering and system parameter tuning require soft biometric feature extractors that are
highly accurate.

Based on the above observations, the ﬁrst challenge in utilizing soft biometrics is the
automatic and reliable extraction of soft biometric information in a non-intrusive manner
without causing any inconvenience to the users. It must be noted that the unreliability and
inconvenience caused by the manual extraction of these features is one of the primary rea-
sons for the failure of Bertillon-like systems. In this thesis, we have analyzed the various
techniques that have been proposed for automatic extraction of characteristics like gender,
ethnicity, eye color, age, and height. We also brieﬂy describe the vision system that was
implemented to accomplish the task of identifying the gender, ethnicity, height, and eye
color of a person from real-time video sequences. Once the soft biometric information is
extracted, the challenge is to optimally combine this information with the primary biomet-
ric identiﬁer so that the overall recognition accuracy is enhanced. We have developed a

Bayesian framework for integrating the primary and soft biometric features.

4.2 Soft Biometric Feature Extraction

Any trait that provides some information about the identity of a person, but does not
provide sufﬁcient evidence to exactly deternrine the identity can be referred to as soft bio-

metric trait. Figure 4.2 shows some examples of soft biometric traits. Soft biometric traits

66

 

are available and can be extracted in a number of practical biometric applications. For ex-
ample, demographic attributes like gender, ethnicity, age, eye color, skin color, and other
distinguishing physical marks such as scars can be extracted from the face images used in
a face recognition system. The pattern class of ﬁngerprint images (right loop, left loop,
whorl, arch, etc.) is another example of soft trait. Gender, accent, and perceptual age of
the speaker can be inferred in a voice recognition system. Eye color can be easily found
from iris images. However, automatic and reliable extraction of soft biometric traits is a
difﬁcult task. In this section, we present a survey of the techniques that have been proposed
in the literature for extracting soft biometric information and brieﬂy describe our system
for determining height, gender, ethnicity, and eye color.

Several researchers have attempted to derive gender, ethnicity, and pose information
about the users from their face images. Gutta et al. [61] proposed a mixture of experts
consisting of ensembles of radial basis functions for the classiﬁcation of gender, ethnic
origin, and pose of human faces. Their gender classiﬁer (male vs female) had an accuracy
of 96%, while their ethnicity classiﬁer (Caucasian, South Asian, East Asian, and African)
had an accuracy of 92%. These results were reported on good quality face images from
the FERET database that had very little expression or pose changes. Based on the same
database, Moghaddam and Yang [62] showed that the error rate for gender classiﬁcation
can be reduced to 3.4% by using an appearance-based gender classiﬁer that uses non-linear
support vector machines. Shakhnarovich et al. [63] developed a demographic classiﬁcation
scheme that extracts faces from unconstrained video sequences and classiﬁes them based
on gender and ethnicity. The learning and feature selection modules used a variant of the
AdaBoost algorithm. Even under unconstrained environments, they showed that a classi-
ﬁcation accuracy of more than 75% can be achieved for both gender and ethnicity (Asian

vs non-Asian) classiﬁcation. For this data, the SVM classiﬁer of Moghaddam and Yang

67

 

 

'Gender Skin Color. Hair color

nA-Ilnrnar

0 Core! Carporation, Ottawa Canadamm

 

Weight
hhp1Mww.laurel -and—hardy.com/
goodies/homeehtml 0 CCA

 

Height
httn llwww altnnwnh " L
(D Alton Museum of History and Art

 

 

Eye color
http'J/ology.e amnh. org/genetics/longdetinition/ndexa. html
ncan Museum 01 Natural History200

Figure 4.2: Examples of soft biometric traits.

also had a similar performance and there was also a noticeable bias towards males in the
gender classiﬁcation (females had an error rate of 28%). Balci and Atalay [64] reported
a classiﬁcation accuracy of more than 86% for a gender classiﬁer that uses PCA for fea-
ture extraction and a multilayer perceptron for classiﬁcation. Jain and Lu [65] proposed
a Linear Discriminant Analysis (LDA) based scheme to address the problem of ethnicity
identiﬁcation from facial images. The users were identiﬁed as either Asian or non-Asian by
applying multiscale analysis to the input facial images. An ensemble framework based on
the product rule was used for integrating the LDA analysis at different scales. This scheme
had an accuracy of 96.3% on a database of 263 users (with approximately equal number of
males and females). Hayashi et al. [66] suggested the use of features like wrinkle texture
and color for estimating the age and gender of a person from the face image. However, they
did not report the accuracy of their technique.

Automatic age determination is a more difﬁcult problem than gender and ethnicity clas-

siﬁcation. Buchanan et al. [67] have studied the differences in the chemical composition

68

 

of ﬁngerprints that could be used to distinguish children from adults. Kwon and Lobo [68]
presented an algorithm for age classiﬁcation from facial images based on cranio-facial
changes in feature-position ratios and skin wrinkle analysis. They attempted to classify
users as “babies”, “young adults”, or “senior adults”. However, they did not provide any
classiﬁcation accuracy. More recently, Lanitis et al. [69] performed a quantitative evalua-
tion of the performance of three classiﬁers developed for the task of automatic age estima-
tion from face images. These classiﬁers used eigenfaces obtained using Principal Compo-
nent Analysis (PCA) as the input features. Quadratic models, shortest distance classiﬁer,
neural network classiﬁers, and hierarchical age estimators were used for estimating the
age. The best hierarchical age estimation algorithm had an average absolute error of 3.82
years which was comparable to the error made by humans (3.64 years) in performing the
same task. Minematsu et al. [70] showed that the perceptual age of a speaker can be auto-
matically estimated from voice samples. All these results indicate that the automatic age
estimation is possible (though the current technology is not very reliable).

The weight of a user can be measured by asking him to stand on a weight sensor while
providing the primary biometric. The height of a person can be estimated from a sequence
of real-time images. For example, Su-Kim et al. [71] used geometric features like vanishing
points and vanishing lines to compute the height of an object. With the rapid growth of
technology, especially in the ﬁeld of computer vision, we believe that the techniques for
soft biometric feature extraction would become more reliable and commonplace in the near

future.

4.2.1 A Vision System for Soft Biometric Feature Extraction

We have implemented a real-time vision system for automatic extraction of gender, eth-

nicity, height, and eye color. The system is designed to extract the soft biometric attributes

69

H!

 

as the person approaches the primary biometric system to present his primary biometric
identiﬁer (face and ﬁngerprint in our case). The soft biometric system is equipped with two
Sony EVI-D30 color pan/tilt/zoom cameras. Camera I monitors the scene for any human
presence based on the motion segmentation image. Once camera I detects an approach-
ing person, it measures the height of the person and then guides camera H to focus on the

person’s face. More details about this system can be found in [72].

4.3 Fusion of Soft and Primary Biometric Information
4.3.1 Identiﬁcation Mode

For a biometric system operating in the identiﬁcation mode, the framework for integrat-
ing primary and soft biometric information is shown in Figure 4.3. The primary biometric
system is based on m (m 2 1) traditional biometric identiﬁers like ﬁngerprint, face, iris and
hand-geometry. The soft biometric system is based on n (n 2 1) soft attributes like age,
gender, ethnicity, eye color and height. Let 021, (oz, - - - , w R represent the R users enrolled
in the database. Let x = [x1 , x2, - - - , xm] be the collection of primary biometric feature
vectors. For example, if the primary biometric system is a multimodal system with face and
ﬁngerprint modalities (m = 2), then X] represents the face feature vector and X2 represents
the ﬁngerprint feature vector. Let p(xj I04) be the likelihood of observing the primary bio—
metric feature vector xj given the user is 0),. If the output of each individual modality in

the primary biometric system is a set of matching scores (S = [81, 82, - ° ° , 8m]), one

 

can approximate p(xj [02,) by p(sj 01,-), provided the genuine matching score distribution
of each modality is known.
Let y = [311,312, . ° . , yn] be the soft biometric feature vector, where, for example,

yl could be gender, yz could be eye color, etc. We require an estimate of the posteriori

70

 

 

 

 

 

 

 

 

 

   

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Primary Biometric System Soft Biometric System
Templates
Feature x .
Extraction > Matching
Module Module
' Feature Extraction Module
P(W IX) y
Bayesian Integration Framework
1 P (w IX. y)
Decision

'—’ User Identity

 

 

Module

 

Figure 4.3: Framework for fusion of primary and soft biometric information. Here x is the
ﬁngerprint feature vector and y is the soft biometric feature vector.

71

probability of user w,- given both x and y. This posteriori probability can be calculated by

applying the Bayes rule as follows:

 

P(w.~lx,y) = p 0" $133“). (4.1)

If all the users are equally likely to access the system, then P (0),) = %, V i. Further,
if we assume that all the primary biometric feature vectors (x1, x2, - ' . ,xm) and all the
soft biometric variables (y1, y2, - - - , 3],) are independent of each other given the user’s

identity w,, the discriminant function g, (x, y) for user 0),- can be written as,

92(x, y) = Z 108 29(lewr) + Z 108p(yklwi)- (4.2)
j=1 k=1

4.3.2 Veriﬁcation Mode

A biometric system operating in the veriﬁcation mode classiﬁes each authentication
attempt as either a “genuine claim” or an “impostor attempt”. In the case of veriﬁcation,

the Bayes decision rule can be expressed as

P(genuine|x, y) _ p(x,y|genuine)P(genuine) r
P(impostor|x, y) _ p(x,y|impostor)P(impostor) — ’

 

 

(4.3)

where 7' is the threshold parameter. Increasing r reduces the false acceptance rate and
simultaneously increases the false reject rate and vice versa. If the prior probabilities of
the genuine and impostor classes are equal and if we assume that all the primary biometric
feature vectors and all the soft biometric attributes are independent of each other given the

class, the discriminant function can be written as,

72

 

 

 

Em p(x,~|genuine) Zn p(yk| genuine)
9(X, y) 0g (p(leimpostor)) + 0g (p(yk|impostor)

j=1 k=1
(4.4)

If the output of each individual modality in the primary biometric system is a set of
matching scores (S = [81, 82, - - - ,Sm]), one can approximate p(xj| genuine) and
p(x,- Iimpostor) by p(s ,- | genuine) and p(sj limpostor), respectively, provided the

genuine and impostor matching score distributions of each modality are known.

4.3.3 Computation of Soft Biometric Likelihoods

A simple method for computing the soft biometric likelihoods p(yk Iw,), k = 1, 2,- - .

is to estimate them based on the accuracy of the soft biometric feature extractors on a train-
ing database. For example, if the accuracy of the gender classiﬁer on a training database is

a, then

I. P(observed gender is male I true gender of the user is male) = a,
2. P(observed gender is female | true gender of the user is female) = a,
3. P(observed gender is male I true gender of the user is female) 2 1 — a,

4. P(observed gender is female | true gender of the user is male) = 1 — 0.

Similarly, if the average error made by the system in measuring the height of a person
is lie and the standard deviation of the error is 08, then it is reasonable to assume that
p(0bserved heightlwi) follows a Gaussian distribution with mean (h(w,-) + [18) and
standard deviation 0,, where h(w,-) is the true height of user 0),. However, there is a

potential problem when the likelihoods are estimated only based on the accuracy of the soft

73

 

biometric feature extractors. The discriminant function in equation (4.2) is dominated by
the soft biometric terms due to the large dynamic range of the soft biometric log-likelihood
values. For example, if the gender classiﬁer is 98% accurate (or = 0.98), the log-likelihood
for the gender term in equation (3) is —0.02 if the classiﬁcation is correct and —3.91 in
the case of a misclassiﬁcation. This large difference in the log-likelihood values is due to
the large variance of the soft biometric features. To offset this phenomenon, we introduce
a scaling factor [3, 0 g ,6 S 1, to ﬂatten the likelihood distribution of each soft biometric
trait. If qk, is an estimate of the likelihood p(yklw,-) based on the accuracy of the feature

extractor, the weighted likelihood 13(3)), Iw,) is computed as,

13(yilwi) = —— (4.5)

where Y], is the set of all possible values of the discrete variable y), and Bk is the weight
assigned to the kth soft biometric feature. If the feature 3}), is continuous with standard
deviation 0],, the likelihood can be scaled by replacing 0k with 95:1. This weighted like-
lihood approach is commonly used in the speech recognition community in the context of
estimating the word posterior probabilities using both acoustic and language models. In
this scenario, weights are generally used to scale down the probabilities obtained from the
acoustic model [73].

This method of likelihood computation also has other implicit advantages. An impostor
can easily circumvent the soft biometric feature extraction because it is relatively easy
to modify/hide one’s soft biometric attributes by applying cosmetics and wearing other
accessories (like a mask, shoes with high heels, etc.). In this scenario, the scaling factor
51: can act as the measure of the reliability of the soft biometric feature and its value can

be set depending on the environment in which the system operates. If the environment is

74

 

 

hostile (many users are likely to circumvent the system), the value of 0), must be closer
to 0. Finally, the discriminant functions given in equations (4.2) and (4.4) are optimal
only if the assumption of independence between all the biometric traits is true. If there
is any dependence between the features, the discriminant function is sub—optimal. In this
case, appropriate selection of the weights ,6], during training can result in better recognition

rates.

4.4 Experimental Results

Our experiments demonstrate the beneﬁts of utilizing the gender, ethnicity, and height
information of the user in addition to the face and ﬁngerprint biometric identiﬁers. A subset
of the “Joint Multibiometric Database” (JMD) collected at West Virginia University has
been used in our experiments. The selected subset contains 4 face images and 4 impressions
of the left index index ﬁnger obtained from 263 users over a period of six months. The
Identix FaceIt® SDK [74] is used for face matching. Fingerprint matching is based on the
algorithm in [52].

The ethnicity classiﬁer proposed in [65] was used in our experiments. This classiﬁer
identiﬁes the ethnicity of a test user as either Asian or non-Asian with an accuracy of
82.3%. If a “reject” option is introduced, the probability of making an incorrect classiﬁca-
tion is reduced to less than 2%, at the expense of rejecting 25% of the test images. A gender
classiﬁer was built following the same methodology used in [65] for ethnicity classiﬁcation
and the performance of the two classiﬁers were similar. The reject rate was ﬁxed at 25%
and in cases where the ethnicity or the gender classiﬁer made a reject decision, the corre-
sponding information is not utilized for updating the discriminant function, i.e., if the label
assigned to kth soft biometric trait is “reject”, then the log-likelihood term corresponding

to the kth feature in equations (4.2) and (4.4) is set to zero. During the collection of the

75

is

 

ltv ,. .

‘( _‘vzu—-—

ﬂ

 

WVU multimodal database, the approximate height of each user was also recorded. How-
ever, our real-time height measurement system was not applied to measure the height of the
user during each biometric data acquisition. Hence, we simulated values for the measured
height of user 0),- from a normal distribution with mean h(w,) + [16 and standard deviation

O’e, where h(0J,-) is the true height of user 02,-, [1,, = 2 cm and 08 = 5 cm.
4.4.1 Identiﬁcation Performance

To evaluate the performance of the biometric system in the identiﬁcation mode, one
face image and one ﬁngerprint impression of each user are randomly selected to be the
template (gallery) images and the remaining three samples are used as probe images. The
separation of the face and ﬁngerprint databases into gallery and probe sets, is repeated 50
times and the results reported are the averages for the 50 trials. The weights (,Bk) used for
the soft biometric likelihood computation are obtained by exhaustive search over an inde-
pendent training database. The multimodal database described in [72] is used for weight
estimation. For each soft biometric trait (gender, ethnicity and height), ﬂ], is increased in
steps of 0.05 and the value of ﬁle that maximizes the average rank-one recognition rate of
the biometric system (utilizing face, ﬁngerprint, and the corresponding soft information)
is chosen. Following this procedure, values of 0.1, 0.1 and 0.5 are obtained as the best
weights for gender, ethnicity, and height, respectively.

The effects of soft biometric identiﬁers on the performance of three primary biometric
systems, namely, ﬁngerprint, face, and a multimodal system using face and ﬁngerprint as
the modalities are studied. Figure 4.4 shows the Cumulative Match Characteristic (CMC)
of the ﬁngerprint biometric system and the improvement in performance achieved after
the utilization of soft biometric information. The use of ethnicity and gender information

along with the ﬁngerprint leads to an improvement of 1.3% in the rank one performance

76

 

 

 

as shown in Figure 4.4(a). From Figure 4.4(b), we can observe that the height information
also results in z 1% improvement in the rank one performance. The combined use of all
the three soft biometric traits results in an improvement of approximately 2.5% over the
primary biometric system, as shown in Figure 4.4(c).

Ethnicity and gender information do not provide any statistically signiﬁcant improve-
ment in the performance of a face recognition system. One of the reasons for this observed
effect could be that the FaceIt® algorithm already takes into account some of the features
containing the gender and ethnicity information. This hypothesis is supported by the ob-
servation that more than 90% of the faces that are incorrectly matched at the rank-one level
belong to either the same gender or ethnicity or both. On the other hand, we observe that
the height information which is independent of the facial features, leads to an improvement
of 0.5%-.1% in the face recognition performance (see Figure 4.5). The failure of the eth-
nicity and gender information in improving the face recognition performance demonstrates
that soft biometric traits would help in recognition only if the identity information provided
by them is complementary to that of the primary biometric identiﬁer.

Figure 4.6 depicts the performance gain obtained when the soft biometric identiﬁers are
used along with both face and ﬁngerprint modalities. Although the multimodal system con-
taining face and ﬁngerprint modalities is highly accurate with a rank-one recognition rate
of 97%, we still get a performance gain of more than 1% by the addition of soft biometric

information.
4.4.2 Veriﬁcation Performance

Genuine and impostor scores are obtained by computing the similarity between all dis-

tinct pairs of samples. The scores are then converted into log-likelihood ratios using the

77

out I... “1'2. L‘nl'

 

V... ;-élI'-—I’ '

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

£98 r if f Y _-.*.-..~‘-~-" I
8 * ..... (IA-Amer“: ...... 0 ....... . ....... 0
g 97. , -*J _.--.o ....... C """"
c 2'" . "I
396’ _i __.0"
I .' ‘-
3 .‘ r"
s 95"., ::
f 1f :-
i“
393-5 ...... FInQGFPfW (Fp) .
E (5 -+-Fp + Gender + Ethnicity
“9218838878910
T0p 11 matches
(a)
o\°
A98 T A . ....
8 *’._*.___*._,_*.-.+
'8 .t‘ " .0 ______ O ...... o ----- if
E 97’ *' ‘ . ....... o """
r: - ‘ ... """
i
396' ,~ ..~-°
g .i’o'".
g 95' 3
.22 i "
h J -(
33 941:
3 .-‘
593:5: --°~Fingerprint(Fp) .
E: 1’ ‘+'Fp 4» Height
92 A A A A A A L A
E 1 2 3 4 5 6 7 8 9 10
Top 11 matches
(13)
°\° an
A a:
8 *Ao--1*
5 98» ,4..- r--'*""*' .
a ‘ * -
*. IIIIII
g " o ...... o ------ 9 0
c 97' ' ..... o -----
o. if ...... o--
2 i ..... o
o 96' _’ ‘9‘"
5 ,: “--
E f
..,, 9g. 59
§ 94- '
g g" ---G-- Fingerprint (Fp)
1;, 93"; --*-~Fp + Soft Biometrics
.C
l- . . . . . A .
V92 A
a- 1 2 3 4 5 6 7 8 9 10

Top n matches

(C)

Figure 4.4: Improvement in identiﬁcation performance of a ﬁngerprint system after utiliza—
tion of soft biometric traits. 3) Fingerprint with gender and ethnicity, b) Fingerprint with
height, and c) Fingerprint with gender, ethnicity and height.

78

 

 

97 .— - ﬂ
96r r. r "1'
o‘*.“
*‘,.*'
95 or” ’

  
 
 
  

CO
A
W

 

-- Face
--*-- Face + Height

(0
(D

 

 

 

 

 

L l A l L

1 2 5 4 5 6 7 8 e 10
Topnmatches

P(T he true user is in the top 11 matches) %

 

Figure 4.5: Improvement in identiﬁcation performance of face recognition system after
utilization of the height of the user.

 

  
 

  

 

 

 

100 . v . .
33 4...- 4,. ...*.---'Av ----- t ----- * - - «I
A n' - --‘
E 99‘ ,-*"*. .A—--A—--"““" ‘- “t
1% 11’" 'Ar““
E 98" ‘e'
c " ......... o ------- 0
' ........ o ------- °
8' 97‘B ...... o """"" °
‘5 ..--"
5 C "
.E 96 .-
E 95 '3 ’
8
a ..
g 94* -°-Face
‘3 '-'0---Fingerprint(Fp)
E 934‘ -A- Face + Fp
(L 921 . 'd-‘lFace 51- Fp 1+ Soft Biometrics]

1 2 3 4 5 6 7 8 9 10
Topnmatches

Figure 4.6: Improvement in identiﬁcation performance of (face + ﬁngerprint) multimodal
system after the addition of soft biometric traits.

79

procedure outlined in [75]. This method involves the non-parametric estimation of gen-
eralized densities of the genuine and impostor matching scores of each modality. The
weights (6),) used for the soft biometric likelihood computation are estimated using a tech—
nique sinrilar to the identiﬁcation case, except that the weights are chosen to maximize the
Genuine Acceptance Rate (GAR) at 8 False Acceptance Rate (FAR) of 0.01%. The best
set of weights for the veriﬁcation scenario are 0.5, 0.5 and 0.75 for gender, ethnicity, and
height, respectively. This difference in the weights between the identiﬁcation and veriﬁca-
tion modes can be attributed to the different discriminant functions used for fusion as given
by equations (4.2) and (4.4), respectively.

Figure 4.7 shows the Receiver Operating Characteristic (ROC) curves when ﬁngerprint
is used as the primary biometric identiﬁer. At lower FAR values, the performance of the
ﬁngerprint modality is rather p00r. This is due to the large similarity scores for a few impos-
tor ﬁngerprint pairs that have very similar ridge structures. The addition of soft biometric
information helps to alleviate this problem, resulting in a substantial improvement (> 20%
increase in GAR at 0.001% FAR) in the performance at lower FAR values (see Figure 4.7).

In the case of face modality and the multimodal system using both face and ﬁngerprint,
the improvement in GAR is about 2% at 0.001% FAR (see Figures 4.8 and 4.9). This
improvement is still quite signiﬁcant given that the GAR of the primary biometric systems

at this operating point is already very high.
4.5 Summary

The objective of this chapter is to demonstrate that soft biometric identiﬁers such as
gender, height, and ethnicity can be useful in person recognition even when they cannot
be automatically extracted with 100% accuracy. To achieve this goal, we have developed a

Bayesian framework that can combine information from the primary biometric identiﬁers

80

 

Genuine Accept Rate (GAR) % Genuine Accept Rate (GAR) %

Genuine Accept Rate (GAR) %

Figure 4.7: Improvement in veriﬁcation performance of a ﬁngerprint system after utiliza—
tion of soft biometric traits. a) Fingerprint with gender and ethnicity, b) Fingerprint with

 

100

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

I tie-rt” "
tr".
m , x;
‘3.-
°°’ x?"
70’ a"?
.p" .5
60» f
...... Fingerprint (FD)
50‘ ...... Fp + Gender + Ethnicity
ii
40 a ‘ M
10" 10‘2 10'1 10° 10'
False Accept Rate (FAR) %
(a)
100 ' '
.t erect-1"“.‘Nh
90» $1.313“
’*.D"‘.
80. ’1’. ...
r' --’
70» f 3‘
I'v' .9-
--*“ Fp + H9lghl
so
Ii
10" 10'2 10“ 10° ‘0‘
False Accept Rate (FAR) %
(b)
100 T
90 t ‘1‘” rt
. .‘r .e‘
*.{*.’..d.
, o O‘
80 I! ..,.-
.. ...
70 h ....
....
80- 9' .
..,... Fingerprint (FD)
50*; .-*---Fp + Sott Biometrics
ii
40 ‘ ‘ ‘
10'3 10'2 10“ 10° 10'
False Accept Rate (FAR) %

(C)

height, and c) Fingerprint with gender, ethnicity and height.

81

 

IF?

100

 

95"

Genuine Accept Rate (GAR) %

 

  

f vvrvv—‘r *7 vv—vv

 

 
 

--0- Face

 

 

 

 

 

80w . 1
~*- Face 4» HeIght
75 A A A AAAAAA A A AAAAAAA . A AAAAAI A AAAA .
10‘3 10‘2 10“ 10° 101

False Accept Rate (FAR) %

Figure 4.8: Improvement in veriﬁcation performance of face recognition system after uti-

lization of the height of the user.

 

 

     
 

 

 

10° " WW
. . -t't-
,\° *.*—*11* o-~°-'*'°""“
E 95. *X",& c..III V .
< .i’ J " I
Q *r‘
o 1A" A’
ii 90» I
t: .A’
‘5
§ 85-
<
o .
.E "
g 30. 3.’ --o-- Fingerprint
8 e' -A-Face+ Fp
_; -+- Face + Fp + Soft Biometrics
75-3‘ ‘Aum-ZAAHUU-lAAHUUO A in“; 1
10 10 10 10 10
False Accept Rate (FAR) %

Figure 4.9: Improvement in veriﬁcation performance of (face + ﬁngerprint) multimodal

system after the addition of soft biometric traits.

82

 

(face, ﬁngerprint, etc.) and the soft biometric information such that it leads to higher accu-
racy in establishing the user’s identity. Our experiments indicate that soft biometric traits
can indeed substantially enhance the biometric system performance if they are complemen-
tary to the primary biometric traits. We have also presented a survey of the techniques that
have been developed for automatically extracting soft biometric features and described our
own implementation of a system that can identify soft biometric traits from real-time video

sequences.

83

 

' '.!~a"r_ _3.~.' ‘I A ' l4 ..
w

T;

B.

CHAPTER 5

Conclusions and Future Work

5.1 Conclusions

Although biometrics is becorrring an integral part of the identity management systems,
current biometric systems do not have 100% accuracy. Some of the factors that impact
the accuracy of biometric systems include noisy input, non-universality, lack of invariant
representation and non-distinctiveness. Further, biometric systems are also vulnerable to
security attacks. A biometric system that integrates multiple cues can overcome some of
these limitations and achieve better performance. Extensive research work has been done
to identify better methods to combine the information obtained from multiple sources. It is
difﬁcult to perform information fusion at the early stages of processing (sensor and feature
levels). In some cases, fusion at the sensor and feature levels may not even be possible. Fu-
sion at the decision level is too simplistic due to the limited information content available at
this level. Therefore, researchers have generally preferred integration at the matching score
level which offers the best compromise between information content and ease in fusion.
One of the problems in score level fusion is that the matching scores generated by different
biometric matchers are not always comparable. These scores can have different charac-
teristics and some normalization technique is required to make the combination of scores
meaningful. Another limitation of the existing fusion techniques is their inability to handle
soft biometric information, especially when the soft information is not very accurate. In

this thesis, we have addressed these two important problems in a systematic manner.

84

P

 

1 .

arm—- >

We have carried out a detailed evaluation of the various score normalization techniques
that have been proposed in literature in terms of their efﬁciency and robustness. First, we
studied the impact of the different normalization schemes on the performance of the multi-
modal biometric system consisting of face, ﬁngerprint, and hand-geometry modalities. Our
analysis shows that min-max, z-score and tanh techniques are efﬁcient and provide a good
recognition performance. However, we also observed that the min-max and z-score nor—
malization schemes are not robust if the training data used for estimating the normalization
parameters contains outliers. The tanh method is more robust to outliers due to the applica-
tion of inﬂuence functions to reduce the effect of noise. But careful selection of parameters
is required for the tanh normalization scheme to work efﬁciently. Although, non-parametric
density estimation is a theoretically sound method of normalizing the matching scores, it
does not work well in practice due to limited availability of training scores and problems
with choosing the appropriate kernel.

One of the major contributions of this thesis has been the development of a framework
for utilizing ancillary information about the user (also known as soft biometric traits) like
gender and height to improve the performance of traditional biometric systems. Although
the ancillary information by itself is not sufﬁcient for person recognition, it certainly pro-
vides some cues about the individual, which can be highly beneﬁcial when used appropri-
ately. We have developed a model based on Bayesian decision theory that can incorporate
the soft biometric information into the traditional biometric framework. Experiments on a
reasonably large multimodal database validate our claim about the utility of soft biometric
identiﬁers. We have also built a prototype system that can extract soft biometric traits from

individuals during the process of primary biometric acquisition.

85

 

5.2 Future Work

Some normalization schemes work well if the scores follow a speciﬁc distribution. For
example, z-score normalization is optimal if the scores of all the modalities follow a Gaus-
sian distribution. Therefore, we need to develop rules that would allow a practitioner to
choose a normalization scheme after analyzing the genuine and impostor score distribu-
tions of the individual matchers. The possibility of applying different normalization tech-
niques to the scores of different modalities must be explored. Guidelines for choosing the
design parameters of some normalization techniques (e.g., the values of the constants a, b,
and c in tanh normalization) need to be developed.

We believe that existing score level fusion techniques are quite ad-hoc and do not con-
sider the underlying mathematical/statistical rigor. A more principled approach would be
the computation of likelihood ratios based on the estimates of genuine and impostor score
distributions. Automatic bandwidth selection techniques can be utilized to ﬁnd the width
of the kernels to be employed in non-parametric density estimation. The use of general-
ized likelihoods that model the score distributions as a mixture of discrete and continuous
components must be explored. This method can also take into account possible correlation
between the multiple sources of information. Finally, such a method has the capability to
act as a black-box and can be used without any knowledge about the biometric matcher.
However, the major obstacle in following the likelihood ratio approach is the scarcity of
multimodal biometric data. If the method has to work effectively, then a large number of
matching scores would be required to estimate the densities accurately.

With regards to our prototype soft biometric system, performance improvement can be
achieved by incorporating more accurate mechanisms for soft biometric feature extraction.
The method of carrying out an exhaustive search for the weights of the soft biometric

identiﬁers is computationally inefﬁcient and requires a large training database. Since the

86

 

weights are used mainly for reducing the dynamic range of the log-likelihood values, it is
possible to develop simple heuristics for computing the weights efﬁciently. The Bayesian
framework in its current form cannot handle time-varying soft biometric identiﬁers such as
age and weight. We will investigate methods to incorporate such identiﬁers into the soft

biometric framework.

87

 

I? -—

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

BIBLIOGRAPHY

A. K. Jain, A. Ross, and S. Prabhakar. An Introduction to Biometric Recognition.
IEEE Transactions on Circuits and Systems for Video Technology, Special Issue on
Image- and Video-Based Biometrics, 14(1):4—20, January 2004.

S. Prabhakar, S. Pankanti, and A. K. Jain. Biometric Recognition: Security and Pri-
vacy Concerns. IEEE Security and Privacy Magazine, l(2):33—42, March-April 2003.

A. K. Jain and A. Ross. Multibiometric Systems. Communications of the ACM,
Special Issue on Multimodal Interfaces, 47(1):34—40, January 2004.

Y. Chen, S. C. Dass, and A. K. Jain. Fingerprint Quality Indices for Predicting Au-
thentication Performance. In Proceedings of Fifth International Conference on Audio-
and Video-Based Biometric Person Authentication (AVBPA) (To appear), New York,
U.S.A., July 2005.

NIST report to the United States Congress. Summary of NIST Stan-
dards for Biometric Accuracy, Tamper Resistance, and Interoperability. Avail-
able at ftp: / / sequoyah . nist . gov/pub/nist_internal_reports/
NI STAPP__Nov02 . pdf, November 2002.

BBC News. Long lashes thwart ID scan trial. Available at http: / /news .bbc .
co.uk/2/hi/uk_news/politics/3693375 . stm, May 2004.

M. Golfarelli, D. Maio, and D. Maltoni. On the Error-Reject Tradeoff in Biometric
Veriﬁcation Systems. IEEE Transactions on Pattern Analysis and Machine Intelli-
gence, l9(7):786—796, July 1997.

T. Matsumoto, H. Matsumoto, K. Yamada, and S. Hoshino. Impact of Artiﬁcial
“Gummy” Fingers on Fingerprint Systems. In Optical Security and Counterfeit De-
terrence Techniques IV, Proceedings of SPIE, volume 4677, pages 275—289, January
2002.

T. Putte and J. Kenning. Biometrical Fingerprint Recognition: Don’t Get Your Fin-
gers Burned. In Proceedings of IFIP T C8/WG8.8 Fourth Working Conference on
Smart Card Research and Advanced Applications, pages 289—303, 2000.

N. K. Ratha, J. H. Connell, and R. M. Bolle. An Analysis of Minutiae Matching
Strength. In Proceedings of Third International Conference on Audio- and Video-
Based Biometric Person Authentication (AVBPA), pages 223—228, Sweden, June
2001.

88

 

[1 l] D. Maio, D. Maltoni, R. Cappelli, J. L. Wayman, and A. K. Jain. FVC2004: Third
Fingerprint Veriﬁcation Competition. In Proceedings of International Conference on
Biometric Authentication, pages 1—7, Hong Kong, China, July 2004.

[12] C. Wilson, A. R. Hicklin, H. Korves, B. Ulery, M. Zoepﬂ, M. Bone, P. Grother, R. J.
Micheals, S. Otto, and C. Watson. Fingerprint Vendor Technology Evaluation 2003:
Summary of Results and Analysis Report. NIST Internal Report 7123; available at
http: / / fpvte . nist . gov/report/ ir_7 12 3_summary . pdf, June 2004.

[13] P. J. Philips, P. Grother, R. J. Micheals, D. M. Blackburn, E. Tabassi, and J. M.
Bone. FRVT2002: Overview and Summary. Available at ht tp: / /www. frvt .
org/FRVTZOOZ/documents.htm.

[14] D. A. Reynolds, W. Campbell, T. Gleason, C. Quillen, D. Sturim, P. Torres-
Carrasquillo, and A. Adami. The 2004 MIT Lincoln Laboratory Speaker Recognition
System. In Proceedings of IEEE International Conference on Acoustics, Speech, and
Signal Processing, Philadelphia, PA, March 2005.

[15] L. Hong, A. K. Jain, and S. Pankanti. Can Multibiometrics Improve Performance? In
Proceedings of IEEE Workshop on Automatic Identiﬁcation Advanced Technologies,
pages 59-64, New Jersey, U.S.A., October 1999.

[16] A. Ross and A. K. Jain. Information Fusion in Biometrics. Pattern Recognition
Letters, Special Issue on Multimodal Biometrics, 24(13):2115-2125, 2003.

[17] L. Hong and A. K. Jain. Integrating Faces and Fingerprints for Personal Identiﬁcation.
IEEE Transactions on Pattern Analysis and Machine Intelligence, 20(12): 1295—1307,
December 1998.

[18] R. Snelick, U. Uludag, A. Mink, M. Indovina, and A. K. Jain. Large Scale Evalua-
tion of Multimodal Biometric Authentication Using State-of-the-Art Systems. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 27(3):450—455, March
2005.

[19] C. Sanderson and K. K. Paliwal. Information Fusion and Person Veriﬁcation using
speech and face information. Research Paper IDIAP-RR 02-33, IDIAP, September
2002.

[20] S. S. Iyengar, L. Prasad, and H. Min. Advances in Distributed Sensor Technology.
Prentice Hall, 1995.

[21] A. Ross and A. K. Jain. Fingerprint Mosaicking. In Proceedings of International
Conference on Acoustic Speech and Signal Processing (ICASSP), pages 4064—4067,
Florida, U.S.A., May 2002.

[22] Y. S. Moon, H. W. Yeung, K. C. Chan, and S. 0. Chan. Template Synthesis and Image
Mosaicking for Fingerprint RegistrationzAn Experimental Study. In Proceedings of

89

 

International Conference on Acoustic Speech and Signal Processing (ICASSP), vol-
ume 5, pages 409—412, Quebec, Canada, May 2004.

[23] A. Kumar, D. C. M. Wong, H. C. Shen, and A. K. Jain. Personal Veriﬁcation Using
Palmth and Hand Geometry Biometric. In Proceedings of Fourth International
Conference on Audio- and Video-Based Biometric Person Authentication (AVBPA),
pages 668-678, Guildford, UK, June 2003.

[24] A. Ross and R. Govindarajan. Feature Level Fusion Using Hand and Face Biometrics.

In Proceedings of of SPIE Conference on Biometric Technology for Human Identiﬁ-
cation, volume 5779, pages 196—204, Florida, U.S.A., March 2005.

[25] R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Classiﬁcation. John Wiley & Sons,
2001.

[26] K. Woods, K. Bowyer, and W. P. Kegelmeyer. Combination of Multiple Classiﬁers
using Local Accuracy Estimates. IEEE Transactions on Pattern Analysis and Machine
Intelligence, l9(4):405—410, April 1997.

[27] K. Chen, L. Wang, and H. Chi. Methods of Combining Multiple Classiﬁers with
Different Features and Their Applications to Text-Independent Speaker Identiﬁcation.
International Journal of Pattern Recognition and Artiﬁcial Intelligence, ll(3):4l7—
445, 1997.

[28] L. Lam and C. Y. Suen. Application of Majority Voting to Pattern Recognition: An
Analysis of Its Behavior and Performance. IEEE Transactions on Systems, Man, and
Cybernetics, Part A: Systems and Humans, 27(5):553—568, 1997.

[29] L. Lam and C. Y. Suen. Optimal Combination of Pattern Classiﬁers. Pattem Recog-
nition Letters, 16:945-954. 1995.

[30] L. Xu, A. Krzyzak, and C. Y. Suen. Methods for Combining Multiple Classiﬁers and
their Applications to Handwriting Recognition. IEEE Transactions on Systems, Man,
and Cybernetics, 22(3):418-435, 1992.

[31] J. Daugman. Combining Multiple Biometrics. Available at http: / /www. cl .
cam.ac.uk/users/jgleOO/combine/combine.html.

[32] T. K. Ho, J. J. Hull, and S. N. Srihari. Decision Combination in Multiple Classiﬁer
Systems. IEEE Transactions on Pattern Analysis and Machine Intelligence, l6( 1 ):66—
75, January 1994.

[33] Y. Wang, T. Tan, and A. K. Jain. Combining Face and Iris Biometrics for Identity
Veriﬁcation. In Proceedings of Fourth International Conference on Audio- and Video-
Based Biometric Person Authentication (AVBPA), pages 805—813, Guildford, UK,
June 2003.

90

 

 

 

[34]

[35]

[36]

[37]

[38]

[39]

[40]

[41]

[42]

[43]

[44]

P. Verlinde and G. Cholet. Comparing Decision Fusion Paradigms using k-NN based
Classiﬁers, Decision Trees and Logistic Regression in a Multi-modal Identity Veriﬁ-
cation Application. In Proceedings of Second International Conference on Audio- and
Video-Based Biometric Person Authentication (AVBPA), pages 188—193, Washington
DC, U.S.A., March 1999.

V. Chatzis, A. G. Bors, and I. Pitas. Multimodal Decision-level Fusion for Person Au-
thentication. IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems
and Humans, 29(6):674—68l, November 1999.

J. Kittler, M. Hatef, R. P. Duin, and J. G. Matas. On Combining Classiﬁers. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 20(3):226—239, March
1998.

S. Prabhakar and A. K. Jain. Decision-level Fusion in Fingerprint Veriﬁcation. Pattern
Recognition, 35(4):861—874, 2002.

E. S. Bigun, J. Bigun, B. Duo, and S. Fischer. Expert Conciliation for Multimodal
Person Authentication Systems using Bayesian Statistics. In Proceedings of First In-
ternational Conference on Audio- and Video-Based Biometric Person Authentication

(AVBPA), pages 291—300, Crans-Montana, Switzerland, March 1997.

A. K. Jain and A. Ross. Learning User-speciﬁc Parameters in a Multibiometric Sys-
tem. In Proceedings of International Conference on Image Processing, pages 57—60,
New York, USA, September 2002.

K. Messer, J. Matas, J. Kittler, J. Luettin, and G. Maitre. XM2VTSDB: The Extended
M2VTS Database. In Proceedings of Second International Conference on Audio- and
Video-Based Biometric Person Authentication (AVBPA), pages 72—77, Washington
DC, U.S.A., March 1999.

M. Indovina, U. Uludag, R. Snelick, A. Mink, and A. K. Jain. Multimodal Biometric
Authentication Methods: A COTS Approach. In Proceedings of Workshop on Multi-
modal User Authentication, pages 99—106, Santa Barbara, USA, December 2003.

National Institute of Standards The Image Group of the Information Access Division
and Technology. Biometric Scores Set - Release 1. Available at http: / /www.
itl . nist . gov/ iad/ 894 . 03 /biometricscores, September 2004.

P. Verlinde, P. Druyts, G. Cholet, and M. Acheroy. Applying Bayes based Classiﬁers
for Decision Fusion in a Multi-modal Identity Veriﬁcation System. In Proceedings
of International Symposium on Pattern Recognition “In Memoriam Pierre Devijver”,
Brussels, Belgium, February 1999.

R. Snelick, M. Indovina, J. Yen, and A. Mink. Multimodal Biometrics: Issues in
Design and Testing. In Proceedings of Fifth International Conference on Multimodal
Interfaces, pages 68-72, Vancouver, Canada, November 2003.

91

 

[45] R. Brunelli and D. Falavigna. Person Identiﬁcation Using Multiple Cues. IEEE
Transactions on Pattern Analysis and Machine Intelligence, 12(10):955-966, October
1995.

[46] M. Montague and J. A. Aslam. Relevance Score Normalization for Metasearch. In
Proceedings of Tenth International Conference on Information and Knowledge Man-
agement, pages 427—433, Atlanta, USA, November 2001.

[47] R. Manmatha, T. Rath, and F. Feng. Modeling Score Distributions for Combining the
Outputs of Search Engines. In Proceedings of Twenty-Fourth International ACM SI-

GIR Conference on Research and Development in Information Retrieval, pages 267—
275, New Orleans, USA, 2001.

[48] P. J. Huber. Robust Statistics. John Wiley & Sons, 1981.

[49] R. Cappelli, D. Maio, and D. Maltoni. Combining Fingerprint Classiﬁers. In Proceed-
ings of First International Workshop on Multiple Classiﬁer Systems, pages 351—361,
June 2000.

[50] F. R. Hampel, P. J. Rousseeuw, E. M. Ronchetti, and W. A. Stahel. Robust Statistics:
The Approach Based on Inﬂuence Functions. John Wiley & Sons, 1986.

[51] F. Mosteller and J. W. Tukey. Data Analysis and Regression: A Second Course in
Statistics. Addison-Wesley, 1977.

[52] A. K. Jain, L. Hong, S. Pankanti, and R. Bolle. An Identity Authentication System
Using Fingerprints. Proceedings of the IEEE, 85(9): 1365-1388, 1997.

[53] M. Turk and A. Pentland. Eigenfaces for Recognition. Journal of Cognitive Neuro-
science, 3(1):71—86, 1991.

[54] A. K. Jain, A. Ross, and S. Pankanti. A Prototype Hand Geometry-based Veriﬁcation
System. In Proceedings of Second International Conference on Audio- and Video-
based Biometric Person Authentication (AVBPA), pages 166—171, Washington DC,
USA, March 1999.

[55] A. Bertillon. Signaletic Instructions including the theory and practice of Anthropo—
metrical Identiﬁcation, R.W. McClaughry Translation. The Werner Company, 1896.

[56] D. D. Heckathom, R. S. Broadhead, and B. Sergeyev. A Methodology for Reducing
Respondent Duplication and Impersonation in Samples of Hidden Populations. In
Annual Meeting of the American Sociological Association, Toronto, Canada, August
1997.

[57] H. Aillisto, M. Lindholm, S. M. Makela, and E. Vildjiounaite. Unobtrusive User
Identiﬁcation with Light Biometrics. In Proceedings of the Third Nordic Conference
on Human-Computer Interaction, pages 327—330, Tampere, Finland, October 2004.

92

 

 

l5;

 

[58] J. L. Wayman. Large-scale Civilian Biometric Systems - Issues and Feasibility. In
Proceedings of Card Tech / Secur Tech ID, 1997.

[59] G. Givens, J. R. Beveridge, B. A. Draper, and D. Bolme. A Statistical Assessment
of Subject Factors in the PCA Recognition of Human Subjects. In Proceedings of
C VPR Workshop: Statistical Analysis in Computer Vision, June 2003.

[60] E. Newham. The Biometrics Report. SJB Services, 1995.

[61] S. Gutta, J. R. J. Huang, P. Jonathon, and H. Wechsler. Mixture of Experts for Classi-
ﬁcation of Gender, Ethnic Origin, and Pose of Human Faces. IEEE Transactions on
Neural Networks, 11(4):948—960, July 2000.

[62] B. Moghaddam and M. H. Yang. Learning Gender with Support Faces. IEEE Trans-
actions on Pattern Analysis and Machine Intelligence, 24(5):707—7l 1, May 2002.

[63] G. Shakhnarovich, P. Viola, and B Moghaddam. A Uniﬁed Learning Framework
for Real Time Face Detection and Classiﬁcation. In Proceedings of International
Conference on Automatic Face and Gesture Recognition, Washington DC, USA,
May 2002.

[64] K. Balci and V. Atalay. PCA for Gender Estimation: Which Eigenvectors Contribute?
In Proceedings of Sixteenth International Conference on Pattern Recognition, vol-
ume 3, pages 363-366, Quebec City, Canada, August 2002.

[65] X. Lu and A. K. Jain. Ethnicity Identiﬁcation from Face Images. In Proceedings of

SPIE Conference on Biometric Technology for Human Identiﬁcation, volume 5404,
pages 114—123, April 2004.

[66] J. Hayashi, M. Yasumoto, H. Ito, and H. Koshimizu. Age and Gender Estima-
tion based on Wrinkle Texture and Color of Facial Images. In Proceedings of the
Sixteenth International Conference on Pattern Recognition, pages 405—408, Quebec
City, Canada, August 2002.

[67] M. V. Buchanan, K. Asano, and A. Bohanon. Chemical Characterization of Finger-
prints from Adults and Children. In Proceedings of SPIE Photonics East Conference,
volume 2941, pages 89—95, November 1996.

[68] Y. H. Kwon and N. V. Lobo. Age Classiﬁcation from Facial Images. In Proceedings
of IEEE Conference on Computer Vision and Pattern Recognition, pages 762—767,
April 1994.

[69] A. Lanitis, C. Draganova, and C. Christodoulou. Comparing Different Classiﬁers for
Automatic Age Estimation. IEEE Transactions on Systems, Man, and Cybernetics,
Part B: Cybernetics, 34(1):621-628, February 2004.

93

 

[70]

[71]

[72]

[73]

N. Minematsu, K. Yamauchi, and K. Hirose. Automatic Estimation of Perceptual
Age using Speaker Modeling Techniques. In Proceedings of the Eighth European
Conference on Speech Communication and Technology, pages 3005—3008, Geneva,
Switzerland, September 2003.

J. S. Kim et al. Object Extraction for Superimposition and Height Measurement. In
Proceedings of Eighth Korea-Japan Joint Workshop on Frontiers of Computer Vision,
January 2002.

A. K. Jain, K. Nandakumar, X. Lu, and U. Park. Integrating Faces, Fingerprints and
Soft Biometric Traits for User Recognition. In Proceedings of Biometric Authentica-
tion Workshop, LNCS 3087, pages 259—269, Prague, Czech Republic, May 2004.

F. Wessel, R. Schluter, K. Macherey, and H. Ney. Conﬁdence Measures for Large Vo-
cabulary Continuous Speech Recognition. IEEE Transactions on Speech and Audio
Processing, 9(3):288—298, March 2001.

[74] Identix Inc. FaceIt Identiﬁcation Software Developer Kit. Available at ht tp: / /

[75]

www.identix.com/products/pro_sdks_id.html.

S. C. Dass, K. Nandakumar, and A. K. Jain. A Principled Approach to Score Level
Fusion in Multimodal Biometric Systems. In Proceedings of Fifth International Con-
ference on Audio- and Video-based Biometric Person Authentication (AVBPA) ( To
appear), New York, U.S.A., July 2005.

94

 

 
  
   

\Ii]ili]]ij]il]]iij[lﬂniii]!

l.