FACESKETCHID:ASYSTEMFORFACIALSKETCHTOMUGSHOTMATCHING
by
ScottKlum
ATHESIS
Submitted
toMichiganStateUniversity
inpartialentoftherequirements
forthedegreeof
ComputerScience|MasterofScience
2014
ABSTRACT
FACESKETCHID:ASYSTEMFORFACIALSKETCHTOMUGSHOTMATCHING
by
ScottKlum
Facialcompositesarewidelyusedbylawenforcementagenciestoassistintheidencation
andapprehensionofsuspectsinvolvedincriminalactivities.Thesecomposites,generated
fromwitnessdescriptions,arepostedinpublicplacesandinthemediawiththehopethat
someviewerswillprovidetipsabouttheidentityofthesuspect.Thislegacymethodof
identifyingsuspectsisslow,tedious,andmaynotevenleadtothetimelyapprehensionof
thesuspect.Hence,thereisaneedforamethodthatcanautomaticallyandientlymatch
facialcompositestolargepolicemugshotdatabases.Asaresultofthisrequirement,facial
compositerecognitionisanimportanttopicforbiometricsresearchers.Whilesubstantial
progresshasbeenmadeinnon-forensicfacialcomposite(orviewedcomposite)recognition
overthepastdecade,verylittleworkhasbeendoneusingoperationalcompositesrelevantto
lawenforcementagencies.Furthermore,toourknowledge,nofacialcompositetomugshot
matchingsystemhasbeendocumentedthatisreadilydeployable.Thecontributionsofthis
thesisinclude:(i)anexplorationofcompositerecognitionusecasesinvolvingmultipleforms
offacialcomposites,(ii)theFaceSketchIDSystem,ascalableandoperationallydeployable
softwaresystemthatachievesstate-of-the-artmatchingaccuracyonfacialcompositesusing
twocomplementaryalgorithms(holisticandcomponent-based),and(iii)astudyofthe
oftrainingdataonalgorithmperformance.Experimentalresultsarepresentedusingalarge
mugshotgallerythatisrepresentativeofalawenforcementagency'smugshotdatabase.
Allresultsarecomparedagainstthreestate-of-the-art(COTS)face
recognitionsystems.
Copyrightby
SCOTTJEFFREYKLUM
2014
Formyfamily,whosesupportisthereasonthesepagesare
iv
ACKNOWLEDGEMENTS
Iwouldliketoacknowledgeandthankmyadvisor,Dr.AnilJain.I'vefeltanover-
whelmingsenseofpersonalgrowthwhilewritingthisthesis,nearlyallofwhichIowetoDr.
Jain'sexceptionalguidance.Iamforeverindebtedtohimforputtingupwithmeduringthe
timesofmygraduatedegree,andhelpingmethoroughlyenjoythegoodtimes.Ican
tlysaythatDr.Jainhaspositivelynotonlymyintellectualdevelopment,
butmylifeasawhole,asmuchasanyonethatIhaveknown.
Mythesiscommittee,whichincludesDr.XiaomingLiuandDr.ArunRossinadditionto
Dr.Jain,havebeentremendouslyhelpfulasI'veworkedtowardsmyMaster's.Thegreatest
impactbothDr.LiuandDr.Rosshavehadonmeissimplythroughexample;bothare
modelcomputerscientistsandresearchers.IfIcanholdmyselftothestandardsthatthey
havesetforthemselves,Iwillundoubtedlybesuccessful.
Next,Iwouldliketoacknowledgemyfriendandpersonalmentor,Dr.BrendanKlare.
Brendan'ssteadfastpositivityandenthusiasmhavebeenanindescribableboontowhatI've
achievedwhileworkingtowardsmyMaster's.EngineerslikeBrendan,whoarenotonly
incrediblyknowledgeableandcreativebutalsowell-spokenandpersonable,areoneofthe
biggestreasonsIlovethedisciplineI'vechosen.
I'dalsoliketothankmyfamilyfortheirunwaveringsupport.Iamluckytohavea
sister,Adrienne,whohasbeenaconstantsourceofencouragement(andplentyoflaughs)
throughoutmytimeincollege.Mymotherandfather,Cindyandarethegreatest
parentsayoungmancouldaskfor.Myfatherhasshownhowheisduringthe
courseofmyMaster's,anyrequestwithouttheslightesthesitation.Mymother,
asshealwayshasbeen,wasasourceofcalmduringmyMaster'swork;knowingIhaveher
unconditionallovehasbeenindescribablyreassuring.Ihaveawonderfulextendedfamily
whohasalsobeenasourceofjoythesepasttwoyears.MyGrandmaEllieandGrandpaBill
v
meritspecialrecognitionas,withouttheirsupport,itwouldhavebeenmuchmore
toattendcollege.
Ihaveworkedwithaspecialgroupofpeoplewhohavecontributedinsomewayto
myMaster's,andundoubtedly,I'llmissacknowledgingsomeoneinthefollowinglist:Josh
Klontz,CharlesOtto,SerhatBucak,SoweonYoon,SunpreetArora,HuHan,LaceyBest-
Rowden,RadhaChitta,ScottMcCallumandJakeRoberto.JoshandCharlesdeservea
specialmention,asbothhavebeenincrediblefriendsandcolleaguesandhavehelpedme
becomeabettercomputerscientist.Sunpreethasalsobeenasoundingboardforme,not
tomentionagoodfriend.Myroommates,AkosNagyandNickSkhavebecomelifelong
friendsandIamthankfultohavemetthemduringmyMaster's.
Finally,IwanttothankmybestfriendKristinHouck.TheEnglishlanguagelacksthe
descriptivepowertodetailwhatKristinhasmeantandmeanstome;tryingtodescribeit
wouldonlycheapenthings.Iwilljustsay:
IloveyouKristin.Thankyouforeverything.
vi
TABLEOFCONTENTS
LISTOFTABLES..................................viii
LISTOFFIGURES.................................ix
CHAPTER1INTRODUCTION.........................1
1.1Background....................................1
1.2Contributions...................................5
CHAPTER2RELATEDWORK.........................6
CHAPTER3CONSTRUCTINGAFACIALCOMPOSITE.........9
3.1Hand-DrawnComposites.............................9
3.2Software-GeneratedComposites.........................10
3.3SurveillanceComposites.............................11
3.4ViewedComposites................................12
CHAPTER4THEFACESKETCHIDSYSTEM................14
4.1SystemSp...............................14
4.2FacialCompositetoMugshotMatchingAlgorithms..............15
4.2.1HolisticAlgorithm............................16
4.2.2Component-BasedAlgorithm......................18
CHAPTER5EXPERIMENTALPROTOCOL.................21
5.1Hand-DrawnComposites.............................22
5.2Software-GeneratedComposites.........................24
5.3SurveillanceComposites.............................24
5.4ViewedSoftware-GeneratedComposites.....................25
CHAPTER6EXPERIMENTALRESULTS..................26
6.1Hand-DrawnComposites.............................26
6.2Software-GeneratedComposites.........................32
6.3SurveillanceComposites.............................37
6.4ViewedSoftware-GeneratedComposites.....................39
CHAPTER7SUMMARYANDFUTUREWORK..............41
BIBLIOGRAPHY..................................43
vii
LISTOFTABLES
Table2.1Priorworkonfacialcompositetophotographmatching...........7
Table5.1Asummaryofdatabasesusedinthisstudy..................21
Table5.2Demographicdistributionofthe100,000mugshotsusedtoextendthe
gallery......................................22
Table6.1Trueacceptrates(TAR)atfalseacceptrates(FAR)of0.1%and1%
fortheFaceSketchIDSystemandthreetCOTSmatchersafter
demographicring.ScoreslistedfortheFaceSketchIDSystem
usingthetrainingsetswhichprovidethehighestretrievalratesforboth
algorithmsandfusingthematchscores.Standarddeviationsofthe5-fold
cross-validationarereportedwhentestingonthePRIP-HDCdatabase..35
Table6.2Retrievalranksfor(surveillancecomposite,mugshot)obtainedfromthe
FaceSketchIDSystemand(surveillanceimage,mugshot)obtainedfrom
thethreeCOTSmatchers.RanksmarkedasFTEindicatetheCOTS
matcherfailedtoenrollthequerysurveillancemugshot.Allretrieval
ranksagalleryof100,000subjectsafterdemographic...38
viii
LISTOFFIGURES
Figure1.1Examplesoffacialcompositesthatwereusedincasesinwhichthe
suspectwassuccessfullyapprehended.Examplesofhand-drawncom-
positesandtheirmugshotmatesareshownforDavidBerkowitz(Son
ofSam)(a)[1],TimothyMcVeigh(theOklahomaCitybomber)(b)
[2],andTedKaczynski(theUnabomber)(c)[3].Software-generated
composites(d,e,f)thatwerecreatedusingthesoftwareFACES[4]are
shownwithmatedmugshots.........................2
Figure2.1Examplesofsurveillanceimagesthatareofntlypoorquality
suchthatCOTSmatchersareexpectedtofailtothetruematein
amugshotdatabase.Surveillancecompositescanbedrawnbasedon
theseimages,whichcanbeusedtoidentifyasuspectmoreaccurately.
SurveillancecompositesshownaredrawnbySandraEnslow[5]......8
Figure3.1Examplesofmugshots(a)andmatedhand-drawncomposites(b)and
software-generatedcomposites(c)createdusingFACES[4].Thehand-
drawncompositesshownweredrawnbyeitherLoisGibson[6]orforensic
artistsattheMichiganStatePolice(MSP).................10
Figure3.2Anexamplesurveillanceimage(a)andthecorrespondingsurveillance
composite(b)andmugshot(c)usedinthisstudy.Allsurveillancedata
wasprovidedbythePCSO..........................12
Figure3.3Foragivenphotograph(a)fromtheARdatabase[7],theviewedfacial
compositesusedinourstudyconsistofahand-drawncomposite(b),
acompositecreatedusingIdenti-Kit(c),andtwocompositescreated
usingFACES(d,e)bytwotoperators................13
Figure4.1TheFaceSketchIDSystemgraphicaluserinterface(a).Optionsexist
formanuallymodifyingqueryeyelocations(b),one-to-onecomparison
ofqueryandtargetimages(c),andviewingimagesafteralgorithmic
normalizationandpreprocessing(d).Thecompositeshownwasdrawn
byLoisGibson[6]...............................15
ix
Figure4.2TheholisticalgorithmpipelineusedbytheFaceSketchIDSystem.Fol-
lowingnormalizationandpreprocessing,SIFT[8]andMLBP[9]fea-
turesareextracted.Optimalsubspacesarelearnedforeachpatchand
theprojectedfeaturesareconcatenated.AfteraPCAsteptoreduce
templatesize,thefeaturevectorisnormalizedusingthe
L
2
norm.
ThesimilarityscoresbasedontheSIFTandMLBPfeaturevectors
whencomparingcompositeandmugshotarefusedusingthesumrule
afterz-scorenormalization..........................17
Figure4.3Thecomponent-basedalgorithmpipelineusedbytheFaceSketchIDSys-
tem.Followingnormalization,STASM[10]isusedtodetect77facial
landmarks.Eachofthe3componentsused(nose,mouth,andeyes)are
extractedandnormalizedtoaspwidthandheight.MLBPfeatures
areextractedfrompatchesinsidethefacialcomponent.APCAstepis
usedtoreducethenoiseintheMLBPdescriptorforagivenpatch.Op-
timalsubspacesarelearnedforeachpatchandtheprojectedfeatures
areconcatenatedandnormalized.Intheinterestofbrevity,weomit
thePCAstepusedtoreducethetemplatesizeinthisdiagram.
L
2
similarityscoresaregeneratedforeachoftheselectedcomponents.
Theoverall(facialcomposite,mugshot)similarityscoreisthesumof
theindividualcomponentscoresafterz-scorenormalization........19
Figure5.1Exampleimagesfromthe100,000PCSOmugshotsusedtoextendthe
experimentalgallery..............................22
Figure6.1Resultsfortheholistic(a,d)andcomponent-based(b,e)algorithms
whenmatchinghand-drawncompositestomugshots.Thebestperfor-
manceisachievedthroughafusionoftheholisticandcomponent-based
algorithmmatchscores(c,f).ThreeCOTSmatchersareincludedin
(c,f)asabaseline...............................27
Figure6.2ExamplesofsuccessfulRank-1matchesof(hand-drawncomposite,mugshot)
pairs(a,b).Afailurecase(c)showsarelativelyaccuratecomposite
whichwasreturnedatarankhigherthan200................28
Figure6.3Examplesofretrievalrankimprovementwhenmatchinghand-drawn
compositestomugshotsaftermatchscorefusionoftheholisticand
component-basedalgorithms.........................29
Figure6.4Examplesofpoorqualitycompositeswithlittleinformationotherthan
outlinesoffacialcomponents(a)orunrealisticappearance(b).Both
pairsshownareretrievedathigherthanRank-5000afterdemographic
....................................30
x
Figure6.5COTS-2facescoresforthePRIP-HDCdatabasecompos-
ites(a)(

=4
:
89,
˙
=1
:
48),thePRIP-HDCdatabasemugshots
(

=5
:
89,
˙
=1
:
37),andthePCSOdatabasemugshots(c)(

=6
:
51,
˙
=1
:
37).Theinqualityisrelativelysmallerbetweenthe
compositesandthePRIP-HDCmugshotsversusthatofthePCSO
mugshots.Therefore,thereportedperformanceforallfacematchers
maybeboostedduetothefactthatitiseasiertomatchimageswith
similarfacialquality..............................31
Figure6.6Examplesofsuccessfulmatchesof(hand-drawncomposite,mugshot)
pairs(a,b)fromthesequestereddataset.Afailurecase(c)showsa
relativelyaccuratecompositewhichwasreturnedatarankhigherthan200.31
Figure6.7AcompositeandphotographsdepictingDzhokharTsarnaev.Retrieval
ranksfortheFaceSketchIDSystemarelistedbelowthecorresponding
photograph.Allranksareafterdemographic(15-25yearold,
white,male)..................................32
Figure6.8Resultsfortheholisticandcomponent-basedalgorithmswhenmatch-
ingsoftware-generatedcompositestomugshotsbefore(a)andafter(b)
demographicCOTSmatchersareincludedasabaseline.....33
Figure6.9ExamplesofsuccessfulRank-1matchesof(software-generatedcompos-
ite,mugshot)pairs(a,b).Afailurecase(c)showsarelativelyaccurate
compositewhichwasreturnedatarankhigherthan200..........34
Figure6.10Examplesofretrievalrankimprovementwhenmatchingsoftware-generated
compositestomugshotsaftermatchscorefusionoftheholisticand
component-basedalgorithms.........................35
Figure6.11Examplesofahand-drawncomposite(a)andasoftware-generatedcom-
posite(d)forwhichtheRank-1matchisanimpostor(b,e)thatismore
similarinappearencetothecompositethanthegenuinemugshotmatch
(c,f).......................................36
Figure6.12ResultsfortheFaceSketchIDSystemwhenmatching75hand-drawn
compositesand75software-generatedcompositestomugshotsbefore
andafterdemographicCOTSmatchersareincludedasabaseline.37
Figure6.13Asurveillanceframe(a)ofTamerlanTsarnaevwasusedtocreateahigh-
qualitysurveillancecomposite(drawnbyJaneWankmiller[11])(b)to
improveperformancewhenmatchingagainstaphotograph(c).After
demographic(20-30yearold,white,male),theFaceSketchID
SystemisabletoretrievethephotographbyRank-20...........38
xi
Figure6.14Resultsfortheholistic,component-based,andfusedalgorithmsonviewed
software-generatedcompositescreatedusingFACESandIdentiKit.For
thecompositescreatedusingFACES,twooperators(anAmericanand
anAsian)createdcomposites[12]......................39
xii
CHAPTER1
INTRODUCTION
Facialcompositesarecommonlyusedinlawenforcementtoassistinidentifyingsuspects
involvedinacrimewhennofacialimageofthesuspectisavailableatthecrimescene
(e.g.,fromasurveillancecameraoramobilephone).Afteracompositeofasuspect's
faceiscreated,authoritiesdisseminatethecompositetolawenforcementandmediaoutlets
withthehopethatsomeonewillrecognizetheindividualandprovidepertinentinformation
leadingtoanarrest.Facialcompositesareparticularlyvaluablewheneyewitness'orvictim's
descriptionsaretheonlyformofevidenceavailable[13].Unfortunately,thisprocessis
tanddoesnotleverageallavailableresources,inparticular,theextensivemugshot
databasesmaintainedbylawenforcementagencies.Successfultechniquesforautomatically
matchingfacialcompositestomugshotswillimprovetheenessoffacialcomposites
andallowforfasterapprehensionofsuspects.
1.1Background
Facialcompositesusedinlawenforcementcanbedividedintothreecategories:
(i)
Hand-drawncomposites
:Facialcompositesdrawnbyforensicartistsbasedon
thedescriptionprovidedbyawitness.Hand-drawncompositeshavebeenused
incriminalinvestigationsdatingasfarbackasthe19thcentury[14].Examples
ofhighcasesinwhichahand-drawncompositewasusedareshownin
Figs.1.1(a,b,c).
(ii)
Software-generatedcomposites
:Facialcompositescreatedusingsoftwarekits
whichallowanoperatortoselectvariousfacialcomponents(Figs.1.1(d,e,f)).
Software-generatedcompositeshavebecomeapopularandmorerdableal-
1
(a)(b)(c)
(d)(e)(f)
Figure1.1:Examplesoffacialcompositesthatwereusedincasesinwhichthesuspectwas
successfullyapprehended.Examplesofhand-drawncompositesandtheirmugshotmates
areshownforDavidBerkowitz(SonofSam)(a)[1],TimothyMcVeigh(theOklahoma
Citybomber)(b)[2],andTedKaczynski(theUnabomber)(c)[3].Software-generated
composites(d,e,f)thatwerecreatedusingthesoftwareFACES[4]areshownwithmated
mugshots.
2
ternativetohand-drawncomposites.Accordingto[14],80%oflawenforcement
agenciesreportusingsomeformofsoftwaretocreatefacialcompositesofsus-
pects.Wenotethat,basedonconversationswithlawenforcementagencies,the
actualadoptionanduseofcomposite-generationsoftwaremaybelowerthan
reportedin[14].
(iii)
Surveillancecomposites
:Facialcompositesdrawnbyforensicartistsbasedon
poorqualitysurveillanceimages.Surveillancecompositesareusedinscenarios
whene-shelf(COTS)systemsareexpectedtofailonquery
(probe)faceimages(duetopoorlighting,osefaces,occlusion,etc.).
Whereasforensicartiststypicallyrequireafewyearsoftrainingtobecometin
drawingcomposites,onlyafewhoursoftrainingaretypicallyrequiredbeforeapolice
canstartusingcomposite-generationsoftware.Irrespectiveofthequalityandcapabilityof
thesoftware,mostcompositesoftwarepackagesrelyonchoosingasetoffacialcomponents
(e.g.,eyes,nose,mouth)basedontheinformationcontainedinthewitness'description.It
isimportanttoemphasizethatirrespectiveofthemethodusedtogeneratethecomposite,
thequalityoftheresultingcomposite(namely,itsresemblancetothesuspect'srealface)
mainlydependsontheaccuracyofthedescriptionprovidedbythewitnessandtheskillof
theartist/operator.Wenotethatin[15],wereferredtohand-drawncompositesas\forensic
sketches"andsoftware-generatedcompositesas\compositesketches".Thenamingconven-
tionshavebeencorrectedinthisthesistotheprevailinglawenforcementterminology.
Whileseveralmethodsthatmatchviewed
1
andhand-drawncompositestomugshotshave
beenreportedintheliterature[16,17,18,19,20,21,22,23],onlyafewmethodshavebeen
publishedforautomaticmatchingofsoftware-generatedcompositestomugshots[19,12].
Inallthepreviousstudiesreportedonsoftware-generatedcomposites,withtheexception
of[15],compositeswerecreatedwhiletheoperatorwasviewingthehighqualitymugshot.
1
Manystudiesonfacialcompositetophotographmatchinghavereliedonviewedcom-
positesinwhichthecompositeisdrawnbyhandwhileviewingthephotograph.
3
Thistypeofviewedcompositedoesnotaccuratelythecreationofcompositesused
incriminalinvestigationsbecausethemugshotofthesuspectisunknownorunavailable.
Indeed,therewouldbenoneedtocreatethecompositeifweknewthesuspectandhadhis
mugshot.Whilesurveillancecompositesarealsocreatedwhenviewingimagesofthesuspect,
theseimagesareofpoorqualitycomparedwithmugshots.Tothebestofourknowledge,no
studieshavereportedperformancewhenmatchingsurveillancecompositestomugshots.An
extendedreviewoffacialcompositetomugshotmatchingliteratureispresentedinChapter
2.Chapter3describestheconstructionprocessforeachtypeoffacialcompositeusedin
thisthesis.
Tothebestofourknowledge,nomatcherisavailablethatisdesignedforfacialcomposite-
to-mugshotmatchingandisdeployedatlawenforcementagencies,thoughinvestigators
oftenattempttouseCOTSmatchersinthismannerwithlimitedsuccess.Toaddressthis
need,thisthesispresentstheFaceSketchIDSystemasastandalonesoftwaresystemthat
canmatchfacialcompositestotheirmugshotmateswithstate-of-the-artaccuracy.System
spcanbefoundinChapter4.TheFaceSketchIDSystemusestwocomplementary
algorithmswhenmatchingfacialcompositestomugshots:(i)aholisticalgorithmand(ii)a
component-basedalgorithm,bothofwhicharedescribedinChapter4.Itisimportantto
pointoutthatthematchingperformanceoftheFaceSketchIDSystemcriticallydependson
theaccuracyofthecomposite(intermsofitsresemblancetothesuspect'sface)aswellas
thebetweenthetimethemugshotinthedatabasewascapturedandthetimethe
compositewascreated.Nevertheless,whiletheaccuracyofcomposite-to-mugshotmatching
istlylowerthanmugshot-to-mugshotmatching,compositetomugshotmatching
systemsareneededtomaximizetheopportunityofapprehendingsuspectsinheinousand
egregiouscrimeswheretheevidenceintheformofasuspect'sphotographislacking.
Chapter5describestheexperimentsusedtoevaluatetheFaceSketchIDSystem.Experi-
mentalresultswhenmatchinghand-drawn,software-generated,andsurveillancecomposites
totheirmugshotmatesarereportedinChapter6.ThreeCOTSfacematchersareused
4
toestablishbaselinerecognitionaccuracywhenmatchingfacialcompositestomugshots.
AllCOTSfacematchersusedinourexperimentshavebeenstudiedintheFaceVendor
RecognitionTest(FRVT)
2
.Wealsoinvestigatetheofthetypeofdatausedto
trainthealgorithmsleveragedbytheFaceSketchIDSystem.Tofacilitatecomparisonswith
previouslypublishedresults,wealsodetailtheFaceSketchIDSystem'sperformancewhen
matchingviewedsoftware-generatedcompositestophotographs.
1.2Contributions
Theprimarycontributionsofthisthesisare:
(i)Anexplorationofcompositerecognitionusecasesinvolvingmultipleformsof
facialcomposites.
(ii)TheFaceSketchIDSystem,ascalableandoperationallydeployablesoftware
systemthatachievesstate-of-the-artmatchingaccuracyonfacialcompositesus-
ingtwocomplementaryalgorithms(holisticandcomponent-based).
(iii)Astudyoftheoftrainingdataonalgorithmperformance.
2
http://www.nist.gov/itl/iad/ig/frvt-home.cfm
5
CHAPTER2
RELATEDWORK
Automatedfacematchingbetweentwofacialphotographsisawellstudiedproblemincom-
putervisionandbiometrics[24].However,matchingfacialcompositestophotographsis
amorechallengingproblemwithonlyalimitedamountofpublishedwork,someofwhich
include:[16,17,18,19,20,22,21,23,12,15].Ofthese,moststudieshaveusedcomposites
drawnwhileviewingthemugshotorphotograph(viewedhand-drawncomposites).Further,
thestudiesthatconsideredoperationalhand-drawncompositesdidnotaddresstheuseof
software-generatedcompositeswhicharereportedtobewidelyusedbylawenforcement
agencies[14].
Toourknowledge,onlytwopreviousstudiesfocusedonautomaticfacerecognitionsys-
temsusingsoftware-generatedcomposites.Theusedacombinationoflocalandglobal
featurestorepresentcomposites[19],butitrequireduserinputintheformofrelevance
feedbackinthematchingorrecognitionphase.Further,theauthorsin[19]usedasmall
galleryintheirexperiments(300facialphotographs).ThemethodproposedbyHanetal.
[12],usedacomponent-basedapproachtomatchfacialcompositestomugshots.WhileHan
etal.usedalargergallerywith10,000mugshotsandcreatedamatchingmethodthatisfully
automatic,thesoftware-generatedcompositesusedwerecreatedwhileviewingthemugshot
photograph(viewedsoftware-generatedcomposites)andthereforedonotoperational
scenarios.
Ourworkuseshand-drawncompositesfromcriminalinvestigationsandsoftware-generated
compositescreatedusingdescriptionsfromvolunteersgiventwodaysafterviewingamugshot,
mimickingawitnessofanactualcrimescene.Furthermore,wecomparetherecognitionac-
curacyofhand-drawncompositeswhenalgorithmsaretrainedusingttrainingdata
sets.Weshowtheimprovedperformanceofmatchingfacialcompositestomugshotswhen
6
Table2.1:Priorworkonfacialcompositetophotographmatching.
PublicationApproachLimitations
ViewedHand-Drawn
Tangand
Wang[16]
Photograph-to-compositecon-
versionusingeigentransform
Viewedcompositesarenotof
anyvalueinlawenforcement
andforensicsapplications.
Methodsthatconvert
compositetophotographor
viceversaareoftensolvinga
moreproblemthanthe
facialcompositetophotograph
matchingtask.
Liuetal.[17]Photograph-to-compositecon-
versionusinglocallylinear
embedding
Gaoetal.[25]Photograph-to-compositecon-
versionusingembeddedhidden
Markovmodel
Wangand
Tang[20]
Photograph-to-compositecon-
versionusingmultiscaleMarkov
randommodel
LinandTang
[18]
Commondiscriminantfeature
extraction
Zhangetal.
[26]
PrincipleComponentAnalysis
(PCA)basedalgorithm
Hand-Drawn
UhlandLobo
[27]
Photometricstandardization
Software-generatedcomposites,
whicharewidelyusedinlaw
enforcement,werenot
considered.
Klareand
Jain[22]
SIFTandMBLPfeaturede-
scriptorswithlocal-feature
baseddiscriminantanalysis
Bhattetal.
[21]
Multi-scalecircularWeber'slo-
caldescriptor
Software-
Generated
Yuenand
Man[19]
Pointdistributionmodelandge-
ometricalrelationship
Compositeswerecreatedwhile
viewingthephotographofthe
subject(viewed
software-generatedcomposites).
Hand-drawncompositeswere
notconsidered.
Hanetal.[12]Component-basedrepresenta-
tionusingMLBPdescriptors
Contributions
Proposed
Method
Facialcompositetomugshot
matchingalgorithmsare
deployedintheFaceSketchID
System.Wefusethematch
scoresoftwont(holistic
andcomponent-based)
algorithmstoboostthe
matchingperformance.
Hand-drawncomposites,
software-generatedcomposites,
andsurveillancecompositesare
considered.Weinvestigatethe
oftrainingthe
algorithmsonerenttypesof
(composite,photograph)data.
7
Figure2.1:Examplesofsurveillanceimagesthatareoftlypoorqualitysuchthat
COTSmatchersareexpectedtofailtothetruemateinamugshotdatabase.
Surveillancecompositescanbedrawnbasedontheseimages,whichcanbeusedtoidentify
asuspectmoreaccurately.SurveillancecompositesshownaredrawnbySandraEnslow[5].
thematchscoresoftwoentalgorithms(holisticandcomponent-based)arefused.We
alsodetailtheuseofsurveillancecomposites,whichhavenotpreviouslybeenreportedinthe
literature(Fig.2.1).AdditionalexamplesofsurveillancecompositescanbefoundinFigs.
3.2and6.13.Allexperimentalresultsarebasedoncomparisonsagainstmugshotmatesfor
thespfacialcompositesandanextendedgalleryof100,000mugshots.Thesizeofour
galleryisrepresentativeofalawenforcementagency'smugshotdatabase.Asummaryof
relatedworkcanbefoundinTable2.1.
Chapter3describestheprocessofcreatingthetypesoffacialcompositesusedinthis
thesis.WealsodetailthedatabasesusedtoevaluatetheFaceSketchIDSystem.
8
CHAPTER3
CONSTRUCTINGAFACIALCOMPOSITE
Aspreviouslymentioned,lawenforcementagenciesrelyonthreemodalitiesoffacialcom-
posites:(i)hand-drawncomposites,(ii)software-generatedcompositesand(iii)surveillance
composites.Hand-drawncomposites(Section3.1)aredrawnbasedonaverbaldescription.
Typically,hand-drawncompositesaredrawnbyaforensicartistwithspecialtraining.Sim-
ilarly,software-generatedcomposites(Section3.2)aredrawnbasedonaverbaldescription,
butarecreatedusingmenu-drivensoftware.Inmostcompositesoftwarepackages,theop-
eratorselectsfromasetoffacialcomponentstosynthesizeaface.Facialcompositescan
alsobecreatedusingpoorqualityorosefacialimages.Thesesurveillancecomposites,
whichareusedwhenCOTSfacematchersareexpectedtofailontheoriginalfaceimages,
aredescribedinSection3.3.Fig.3.1showsexamplehand-drawnandsoftware-generated
compositesalongwithmatedmugshotsthatareusedinourexperiments.Fortheremainder
ofthispaperwewillusethe(querymodality,targetmodality)orderedpairconventionto
denotematchingscenarios.
3.1Hand-DrawnComposites
Allhand-drawncompositesusedinourstudywerecreatedbyforensicartistsforreal-world
criminalinvestigations.Tocreateahand-drawncomposite,anartistdrawsafacebasedon
descriptionsprovidedbyeitheroneormultipleeyewitnesses.Forthistypeofcomposite,the
timebetweenobservationandrecallbyawitnessvariesdependingonthecircumstances.
Atotalof265hand-drawncompositesalongwiththeirmatedmugshotsareusedinour
experiments,whichwewillrefertoasthePatternRecognitionandImageProcessing(PRIP)
Hand-DrawnComposite(PRIP-HDC)database.Ofthe265totalhand-drawncomposites,
73weredrawnbyLoisGibson[6],43weredrawnbyKarenTaylor[2],56wereprovidedby
9
(a)
(b)
(c)
Figure3.1:Examplesofmugshots(a)andmatedhand-drawncomposites(b)and
software-generatedcomposites(c)createdusingFACES[4].Thehand-drawncomposites
shownweredrawnbyeitherLoisGibson[6]orforensicartistsattheMichiganStatePolice
(MSP).
thePinellasCounty's(PCSO),46weredrawnbyforensicartistsemployedby
theMichiganStatePolice(MSP),and47weredownloadedfromtheInternet.
3.2Software-GeneratedComposites
Anumberofsoftwaresystemsareavailabletocreatecomposites:E-FIT[28],EvoFit[29],
FACES[4],Identi-Kit[30],Mac-a-Mug[31],andPhoto-Fit[31].Ofthese,Identi-Kitand
FACESaremostwidelyusedbylawenforcementagenciesintheUnitedStates[14].Both
Identi-KitandFACESallowuserstochoosefromasetofcandidatecomponentsorfeatures
(e.g.eyes,mouth,nose).FACESprovidesalargernumberoffeaturesandoptions,andithas
beenobservedtobemoreaccurateincapturingfacialcharacteristicsthanIdenti-Kit[12].
Forthesetworeasons,weusedFACEStocreatecompositesforourmatchingexperiments.
Tocreatethesoftware-generatedcomposites,weusedaproceduredesignedtomimic
real-worldcompositesynthesisdetailedin[32].Volunteers(adultsrangingfrom20-40years
10
ofage)wereaskedtoviewamugshotofasuspectforoneminute.Twodayslaterthey
wereaskedtodescribethemugshottotheFACESsoftwareoperator(theauthorofthis
thesis)whohadnotseenthemugshot.Volunteersalsoprovideddemographicinformation
aboutthesuspecttothebestoftheirability(gender,race/ethnicity,agerange).During
thedescriptionprocess,theFACESoperatorusedacognitiveinterviewtechnique[33]to
enhancethevolunteer'smemoryofthesuspect'sfacialfeaturesinthemugshot.Toreduce
theproblemofoperatorcontamination[32],inwhichpreviouslycreatedcomposites
thecreationofthecurrentcomposite,arandomfacewasgeneratedinitiallywhichwasthen
mobasedonthevolunteer'sdescription.
Wenotethattherearecertainlimitationsincreatingsoftware-generatedcomposites.For
example,itistoachievecertaintypesofshadingandskintextureinthecomposite.
Theoptionsforlocalizingacomponentonthefacearelimited,andthereforeachieving
thedesiredalignmentofcomponentsisalsochallenging.Intotal,75software-generated
compositesweresynthesized,eachtaking30minutestocreate,onaverage.Thisdatabase
willbereferredtoasthePRIPSoftware-GeneratedComposite(PRIP-SGC)database.
3.3SurveillanceComposites
Giventheubiquityofsurveillancetechnology,lawenforcementagenciesattempttomakeuse
ofallofthefacialimagedataattheirdisposalregardlessofitsquality.Atthelowerendof
thequalityspectrumareimagescapturedbyolder-generationcellphones,retailsurveillance
cameras,andATMcameraswhichoftenareblurred,havetshadowsorocclusion,or
containanoseface.Insomecases,thesefacialimagesareoftqualitytobeused
inaCOTSfacematcher.However,inmostcasesthesurveillanceimageryisofextremely
poorqualityandtheCOTSfacematchersfailtothecorrespondingindividualwithina
mugshotgallery.Tomakeuseofthesepoorqualityfaceimages,lawenforcementagencies
oftenemployaforensicartisttocreateahighqualityfacialcompositefromthesurveillance
faceimage.Weinvestigatethepossibilityofusingthisformoffacialcompositetoimprove
11
(a)(b)(c)
Figure3.2:Anexamplesurveillanceimage(a)andthecorrespondingsurveillance
composite(b)andmugshot(c)usedinthisstudy.Allsurveillancedatawasprovidedby
thePCSO.
uponthematchingperformanceofCOTSmatchersusingasetof(surveillancecomposite,
mugshot)pairsprovidedtousbythePCSO(Fig.3.2).Wenotethatwhilethesurveillance
imagesusedinthisthesisareofrelativelyhighquality,weintroducetheuseofsurveillance
compositesasaproof-of-concepttoshowthattheycanbeusedsuccessfully.Amorerealistic
usecaseofsurveillancecompositeswouldinvolvethecompositesdepictedinFig.2.1,but
wedonothavethematedmugshotsforthesecomposites.
3.4ViewedComposites
Whileviewedcompositesarenotapplicableinforensicscenarios,wehavefoundthemto
beusefulduringalgorithmtraining.Eighteen-hundred(viewedhand-drawncomposite,pho-
tograph)pairsusedinourstudyareavailablefromtheChineseUniversityofHongKong
(CUHK)
3
.TheCUHKFaceSketchdatabase[20]contains188pairsfromtheCUHKstudent
database,123pairsfromtheARdatabase[7],295pairsfromtheXM2VTSdatabase[34],
and1,194pairs[35]fromtheFERETdatabase[36].Wewillrefertothissetof1,800(viewed
hand-drawncomposite,photograph)pairsastheCUHK-VHDCdatabase.
Additionally,asetofviewedhand-drawncompositesweredrawnbyforensicartistsat
theMSPfor93ofthe265mugshotsinthePRIP-HDCdatabase.Weinvestigatethe
oftrainingourmatchingalgorithmsonthesecomposites,whichwewillrefertoasthePRIP
3
http://mmlab.ie.cuhk.edu.hk/facesketch.html
12
(a)
(b)(c)
(d)(e)
Figure3.3:Foragivenphotograph(a)fromtheARdatabase[7],theviewedfacial
compositesusedinourstudyconsistofahand-drawncomposite(b),acompositecreated
usingIdenti-Kit(c),andtwocompositescreatedusingFACES(d,e)bytwot
operators.
ViewedHand-DrawnComposite(PRIP-VHDC)database.EighteofthePRIP-VHDC
subjectshaveasinglefacialcomposite,andtheremaining8havetwocompositeseach.
TodemonstratethestrengthoftheFaceSketchIDSystemcomparedtopreviouslyre-
portedresults,weincludeasetofviewedsoftware-generatedcompositesinourmatching
experimentsfrom[12].ThisdatawillbereferredtoasthePRIPViewedSoftware-Generated
Composite(PRIP-VSGC)database.Foreachofthe123photographsfromtheARdatabase
usedinthePRIP-VSGCdatabase,threecompositeswerecreated.Twocompositeswere
createdusingFACESandthethirdwascreatedusingIdenti-Kit.Examplesofviewedcom-
positesusedinourexperimentscanbefoundinFig.3.3.
Chapter4describestheFaceSketchIDSystemindetail,includingsystemsp
aswellasthealgorithmsthatareusedincompositerecognition.
13
CHAPTER4
THEFACESKETCHIDSYSTEM
TheFaceSketchIDSystemwasdevelopedtoaddressthelackofafullyautomaticande
meanstomatchfacialcompositestomugshots.SystemspfortheFaceSketchID
SystemaredescribedinSection4.1,whilethematchingalgorithmsusedbytheFaceSketchID
SystemaredescribedinSection4.2.
4.1SystemSp
TheFaceSketchIDSystemsupportsadrag-and-dropenrollmentinterfacewithoptionsfor
manuallymodifyingdetectedeyelocations,viewingbothprobeandtargetimagesafteralgo-
rithmprocessing,andsearchingforknownindividualsbynamewithinthemugshotmatches.
TheFaceSketchIDSystemalsosupportsthemugshotgalleryviademographicinfor-
mationintheformofagerange,race,andgender.Tosimplifydeployment,galleryimages
canbeenrolledtoandaccessedfromremotelocations(e.g.anserver).
TheFaceSketchIDSystemiscompatiblewithWindows,OSX,andUbuntuLinuxenvi-
ronments.SourcecodefortheFaceSketchIDSystemiswritteninC++.TheFaceSketchID
SystemusesOpenCV[37]asamatrixlibrary,Eigen[38]forstatisticallearning,andQt[39]
fortheGUI.SomemodulesoftheFaceSketchIDSystemareavailableinOpenBR[40].Ona
2.9GHzIntelCorei7laptopwith8GBofRAM,enrollment(includingeyedetection)and
matchingspeedsare1.07templatespersecondperthreadandabout22,000comparisonsper
secondperthread,respectively.Templatesareapproximately5.73KBinsize.Sp,
templatescontainfeaturevectorsforthetwomatchingalgorithmsinadditiontodemographic
information(age,race,andgender)anddetectedeyelocations.Wenotethatthecombination
ofareasonablysmalltemplatesizeinconjunctionwithrapidrecognitioncapabilitiesshould
facilitatealargeusabilityimprovementoverthetraditionalmethodofusingcompositedata
14
(a)(b)
(c)(d)
Figure4.1:TheFaceSketchIDSystemgraphicaluserinterface(a).Optionsexistfor
manuallymodifyingqueryeyelocations(b),one-to-onecomparisonofqueryandtarget
images(c),andviewingimagesafteralgorithmicnormalizationandpreprocessing(d).The
compositeshownwasdrawnbyLoisGibson[6].
(e.g.disseminationtomediaoutlets).Avideodemonstratingthematchingprocesscanbe
foundat
http://biometrics.cse.msu.edu/images/ImgProjects/help_match.mp4
.
4.2FacialCompositetoMugshotMatchingAlgorithms
TheFaceSketchIDSystemleveragestwocomplementaryalgorithmswhenmatchingfacial
compositestotheirmugshotmates.ThedevelopedbyKlareandJain[23],isdesigned
foruseinheterogenousfacerecognition.Thatis,itisenotonlywhenmatching(facial
composite,mugshot)pairs,butalsoinnear-infrared,thermal,andcross-distancematching
scenarios.Thesecond,developedbyHanetal.[12],wasoriginallydesignedforsoftware-
15
generatedcompositetomugshotmatching.Momadetotheoriginalalgorithms
in[23]and[12]toimprovematchingaccuracy,templatesize,andalgorithmspeedsare
alsodiscussed.Inthecaseofthecomponent-basedalgorithm,moareheavily
bythealgorithmin[41].Eachalgorithmisdescribedindetailinthefollowing
sections.Wenotethatbothalgorithmshavebeenextensivelytunedtoperformwellin
(composite,mugshot)recognition.
4.2.1HolisticAlgorithm
TheholisticalgorithmusedbytheFaceSketchIDSystemhasbeenfoundtobeane
techniqueformatchingafacialcompositeprobeagainstagalleryofmugshots[23].One
strengthoftheholisticalgorithmisthatitrepresentsbothfacialcompositesandmugshots
withlocaldescriptor-basedfeatures,eliminatingtheneedtosynthesizeapseudo-composite
fromthemugshotasisdonein[16]and[25].Thus,thealgorithmforrepresentingamugshot
isanalogoustoafacialcompositeandwillbeomittedfromthefollowingdescription.
Afterdetectingeyelocations,thefacialcompositeisnormalizedtoaheightand
widthandtransformedsuchthatrightandlefteyesareatthesamepositionforevery
composite.Thecenter-surrounddivisivenormalization(CSDN)[42]isthenappliedto
thecompositetocompensatefortheerencesrelatedtothechangeinmodalitybetween
compositeandmugshot.Subsequently,SIFT[8]featuresareextractedfromadensegrid
acrosstheface.InparalleltotheCSDNpreprocessingandSIFTfeatureextractionpipeline,
thenormalizedcompositeisprocessedwiththeTan&Triggspipeline[43](consistingof
aGammaaceofGaussianandcontrastequalization)andmulti-scale
localbinarypattern(MLBP)[9]featuresareextracted.Notethatthedensegridusedas
keypointsforSIFTdescriptorextractionandthepatchesusedtocomputeMLBPfeatures
correspondtothesamelocationswithinthecomposite.Although[23]usesadditional
toimprovegeneralizationacrossmultiplemodalities,ourempiricalstudyshowsthatusing
onlytheCSDNpriortoextractingSIFTfeaturesandtheTan&Triggspipelineprior
16
Figure4.2:TheholisticalgorithmpipelineusedbytheFaceSketchIDSystem.Following
normalizationandpreprocessing,SIFT[8]andMLBP[9]featuresareextracted.Optimal
subspacesarelearnedforeachpatchandtheprojectedfeaturesareconcatenated.Aftera
PCAsteptoreducetemplatesize,thefeaturevectorisnormalizedusingthe
L
2
norm.
ThesimilarityscoresbasedontheSIFTandMLBPfeaturevectorswhencomparing
compositeandmugshotarefusedusingthesumruleafterz-scorenormalization.
toextractingMLBPfeatures,resultsinthebestmatchingperformanceforcomposites.
TheTan&Triggspreprocessingpipeline,inparticular,resultedintlybetter
qualitativeperformance.Thatis,afterincorporatingthispipelineintoboththeholisticand
component-basedalgorithms,impostormatchesreturnedatlowrankswereobservedtobe
moresimilarinappearancetothequerycomposite(whichcouldbearguedtobeameasure
robustnessforthisrecognitiontask).Wenotethatalthoughin[43]thepreprocessingpipeline
isusedtohandleinlighting,ourqualitativeanalysisindicatesthatithelpshandle
inimagequality(resolution,compressionartifacts,etc.)aswell.
ForbothSIFTandMLBPfeatures,optimalsubspacesarelearnedforeachpatchusing
lineardiscriminantanalysis(LDA)afterapplyingPrincipalComponentAnalysis(PCA)to
reduceredundancyintheextractedfeatures.KlareandJainusedarandom-sampleLDA
(RS-LDA)techniqueintroducedin[44]tohandlethesmallsamplesizeproblem.Sincewe
haverelativelymoretrainingdata,RS-LDAhasbeenreplacedbyLDAintheFaceSketchID
Systemtoimprovealgorithmspeed.
Afterlearninganoptimalsubspaceforeachpatchandprojectingthepatch-wisefeatures
intotheirrespectivesubspaces,theprojectedfeaturesareconcatenatedtoformasingle
17
featurevectorforbothfeaturerepresentations.PCAisappliedtothefeaturevectorto
reducetemplatesize,andtheresultingfeaturevectorisnormalizedusingthe
L
2
norm.To
measurethesimilaritybetweenfeaturevectors,theholisticalgorithmusesthe
L
2
similarity
measure.Afterz-scorenormalization,scoresfromtheSIFTandMLBPrepresentationsare
fusedviaasum-of-scorefusionrulewithequalweightappliedtobothrepresentations.A
diagramoftheholisticalgorithmpipelineisshowninFig.4.2.
4.2.2Component-BasedAlgorithm
Asmentionedearlier,thecomponent-basedmethodusedbytheFaceSketchIDSystemwas
proposedin[12]tomatchsoftware-generatedcomposites(createdusingFACESandIdenti-
Kit)tophotographs.Similartotheholisticalgorithm,theprocessforrepresentinganimage
isnotdependentonitsmodality.Thus,theprocessforrepresentingacompositeusingthe
component-basedmethodwillbedescribed.
Inthecomponent-basedalgorithm,facialcomponentsareautomaticallylocalizedby
detectinglandmarkswithanactiveshapemodel(ASM)viatheSTASMlibrary[10].As
in[41],theASMisinitializedusingeyelocationsprovidedbyaCOTSeyedetectorthat
isbundledwiththeFaceSketchIDSystem.Wenotethatwhilealignmentiscriticalforthe
holisticalgorithm,itisarguablymoreimportantwhenextractinglandmarksusedbythe
component-basedrepresentationbecausetherelativelysmallsizeofthefacialcomponents
limitsthedescriptivetolerancetonoise(viamisalignment).
Afterfacialcomponentsareextractedandnormalizedtoaspwidthandheightand
processedusingtheTan&Triggspreprocessingpipeline,MLBP[9]descriptorsareused
tocapturethetextureandstructureofpatchesineachfacialcomponent.APCAstepis
usedtoreducethenoisepresentinthepatch-wiseMLBPrepresentationforthegivenfacial
component.Similartoholisticalgorithm,thecomponent-basedmethodusesLDAtolearn
theoptimalsubspaceandimproverecognitionaccuracy.APCAstepisusedtoreduce
templatesize.
18
Figure4.3:Thecomponent-basedalgorithmpipelineusedbytheFaceSketchIDSystem.
Followingnormalization,STASM[10]isusedtodetect77faciallandmarks.Eachofthe3
componentsused(nose,mouth,andeyes)areextractedandnormalizedtoaspwidth
andheight.MLBPfeaturesareextractedfrompatchesinsidethefacialcomponent.A
PCAstepisusedtoreducethenoiseintheMLBPdescriptorforagivenpatch.Optimal
subspacesarelearnedforeachpatchandtheprojectedfeaturesareconcatenatedand
normalized.Intheinterestofbrevity,weomitthePCAstepusedtoreducethe
templatesizeinthisdiagram.
L
2
similarityscoresaregeneratedforeachoftheselected
components.Theoverall(facialcomposite,mugshot)similarityscoreisthesumofthe
individualcomponentscoresafterz-scorenormalization.
In[12],cosinesimilaritiesbetweencorrespondingpatchesoffacialcomponentsarecom-
putedandtheoverallcomponentsimilarityistheaverageofthepatch-wisesimilarities.
Concatenatingthepatch-wisefeaturevectorspriortocomputinganoverallcomponentsim-
ilarityreducestemplatesizeandincreasescomparisonspeedwithintheframeworkofthe
FaceSketchIDSystem.Further,thetwocomparisontechniquesresultincomparableper-
formance.Aswith[12],themostaccuratecomponentstobeusedduringscorefusionare
determinedempirically.Scoresarenormalizedpriortofusionusingz-scorenormalization
andequalweightsareappliedtoallcomponents.Forboththehand-drawnandsoftware-
generatedcomposites,thecomponent-basedalgorithmusesthemouth,nose,andeyecom-
ponents.Adiagramofthecomponent-basedalgorithmpipelineisshowninFig.4.3.
Asimplesum-fusionruleisusedwhenfusingthematchscoresfrombothalgorithms
afterz-scorenormalization.Whenmatchingbothhand-drawnandsoftware-generatedcom-
positestomugshots,weightsof0.6and0.4areassignedtothematchscoresoftheholistic
andcomponent-basedrepresentations,respectively(i.e.theholisticSIFTandMLBPmatch
19
scoreshaveweightsof0.2,whileeachofthecomponent-basedmatchscoreshaveweightsof
approximately0.13).
Chapter5describestheexperimentalprotocolsusedtoevaluatetheFaceSketchIDSystem
indetail.
20
CHAPTER5
EXPERIMENTALPROTOCOL
Thegallerysetforallexperiments(withtheexceptionofthoselistedinSection5.4)consists
ofmatedmugshotsforthelistedcompositesplusasetof100,000mugshotsfromthePCSO
database.Experimentalresultsarereportedwithandwithoutlteringthemugshotgallery
usingdemographicinformation(intheformofagerange,race,andgender).Ground-truth
demographicinformationwasprovidedforthe100,000mugshotsusedtoextendthegallery
fromthePCSO.Weestimatedemographicinformationforthemugshotsforwhichwedo
nothaveground-truthinformation(thematedcompositesaretohaveanagerange

5yearsrelativetotheestimatedageofthemugshot).Threecommercialfacematchers
areusedasbaselines,whichwillbereferredtoasCOTS-1,COTS-2,andCOTS-3.Note
thatweareunabletotrainanyoftheCOTSsystems.Table5.1summarizesthedatabases
usedinourstudy.Fig.5.1showsexamplemugshotsusedtoextendthegalleryinmatching
experiments.Table5.2showsthedistributionofdemographicsintheextendedgallery.
Table5.1:Asummaryofdatabasesusedinthisstudy.
DatabaseDetailsNo.ofPairs
PRIP-HDC
H
and-
d
rawn
c
omposites(HDC)
265
withmugshotmates
PRIP-SGC
S
oftware-
g
enerated
c
omposites(SGC)
75
withmugshotmates
PRIP-VHDC
V
iewed
h
and-
d
rawn
c
omposites(VHDC)
93
withmugshotmates
PRIP-VSGC
V
iewed
s
oftware-
g
enerated
123
c
omposites(VSGC)withphotographmates
CUHK-VHDC
V
iewed
h
and-
d
rawn
c
omposites(VHDC)
1800
withphotographmates
21
Figure5.1:Exampleimagesfromthe100,000PCSOmugshotsusedtoextendthe
experimentalgallery.
Table5.2:Demographicdistributionofthe100,000mugshotsusedtoextendthegallery.
Males
Ethnicity
AgeRange
<
20
20-30
30-40
40-50
50-60
>
60
Asian
1
182
155
89
46
17
Black
101
7447
6511
4171
2953
917
Hispanic
10
2480
2945
1599
619
184
White
46
11441
11424
10013
9133
3285
Other
1
83
78
91
40
30
Females
Ethnicity
AgeRange
<
20
20-30
30-40
40-50
50-60
>
60
Asian
0
60
53
37
26
16
Black
2
2011
1660
1111
634
151
Hispanic
0
312
308
135
105
16
White
0
5002
4930
3944
2783
667
Other
0
19
28
9
6
3
5.1Hand-DrawnComposites
Experimentalresultsformatching(hand-drawncomposite,mugshot)pairsarereported
basedona5-foldcross-validationscheme.Trainingandtestingsetsaredisjoint;thatis,
nosubjectthatwasusedtotrainanalgorithmwasusedwhentestingitsperformance.The
265subjectsinthePRIP-HDCdatasetareassignedtoacross-validationfoldviaanMD5
hashingfunctionbasedonthesubject'sidenThus,thenumberofsubjectsinthetest-
ingsubsetofagivenfoldvariesbutis,onaverage,53.Thettrainingsetsusedto
22
trainthetwoalgorithmsinthehand-drawncompositetomugshotmatchingexperimentsare
asfollows:
1.
PRIP-HDC
Facialcompositetomugshotmatchingalgorithmsaretrainedonthe
setofapproximately212(hand-drawncomposite,mugshot)pairsavailablepercross-
validationfoldinthePRIP-HDCdatabase.EachsubjectinthePRIP-HDCdatabase
isonlyassociatedwithtwoimages:ahand-drawncompositeandamugshot.
2.
CUHK-VHDC
Algorithmsaretrainedonthesetof1,800(viewedhand-drawncom-
posite,photograph)pairsfromtheCUHK-VHDCdatabase.SimilartothePRIP-HDC
database,theCUHK-VHDCdatabasecontainsonlytwoimagespersubject.Note
thatalthoughwenolongerneedtousecross-validationbecausewearetrainingon
onedatasetandtestingonanother,wemaintainthetestingsplitsusedin(1)toallow
forcomparisonbetweenexperiments.Thisprocedureisusedinallofthefollowing
experimentsinwhichcross-validationisnotnecessary.
3.
PRIP-HDC+CUHK-VHDC
Algorithmsaretrainedonthesetofapproximately
212(hand-drawncomposite,mugshot)pairsavailableinthePRIP-HDCdatabaseas
inprotocol(1).ForthetrainingsubsetofeachfoldofthePRIP-HDCdatabase,the
entireCUHK-VHDCdatabaseisadded.
4.
PRIP-HDC+CUHK-VHDC+PRIP-VHDC
Algorithmsaretrainedwiththe
sameprotocolasinprotocol(3).Whenthetestingsetdoesnotcontainsubjectsfrom
thePRIP-VHDCdataset,thosepairsareaddedtothetrainingset.
Wealsoreportperformanceonasetof32(hand-drawncomposite,mugshot)pairswhich
havebeensequesteredfromthetrainingexperimentsreportedabove.Forthesepairs,the
trainingsetthatresultsinthebestrecognitionperformanceisused.Notethatcross-
validationisnotnecessaryintheseexperimentssincewearetrainingandtestingont
data.
23
Finally,wereporttheperformanceofacompositerelatedtotheBostonMarathonbomb-
ingsuspect(see[45])usingthesameexperimentalsetupasthesequesteredpairs.Thecom-
positedepictstheyoungerbrother,DzhokharTsarnaev,drawnbyanartistinthecourt
duringapreliminarytrial
4
.Whileitcanbeconsideredasaviewedhand-drawncomposite,
weincludetheperformanceasanadditionalexampleofthecapabilitiesoftheFaceSketchID
System.
5.2Software-GeneratedComposites
DuetotherelativelysmallsizeofthePRIP-SGCdatabase,wecouldonlytrainthetwo
matchingalgorithmsusingtheCUHK-VHDCdatabase.Thus,thereisnoneedforcross-
validationwhentestingonsoftware-generatedcomposites.
Weincludeacomparisonbetweenthe75compositesfromthePRIP-SGCdatabaseand
thecorresponding75compositesfromthePRIP-HDCdatabase[15].Inthisexperiment,
onlytheperformanceafterfusingthematchscoresofbothalgorithmsisreported.
5.3SurveillanceComposites
ToevaluatetheperformanceoftheFaceSketchIDSystemon(surveillancecomposite,mugshot)
pairs,wetraintheholisticandcomponent-basedalgorithmsusingtheoptimaltrainingsets
asdeterminedbytheexperimentsinSection5.1.Wecompareretrievalrankswhenusing
(surveillancecomposite,mugshot)pairsintheFaceSketchIDSystemversusretrievalranks
whenusing(surveillanceimage,mugshot)pairsinCOTSmatchers.Foragivensubjectin
thisdataset,thereare,onaverage,6compositescreatedbytartists.Wereportthe
retrievalrankachievedusingthemostaccuratecompositeandthemostaccurateframefrom
thesurveillancevideoforeachsubject.Wealsoincludeasurveillancecompositedepicting
TamerlanTsarnaev(theolderbrotherallegedtobeinvolvedintheBostonMarathonbomb-
4
http://www.businessinsider.com/dzhokhar-tsarnaev-court-room-sketch-2013-7
24
ing)[11]andreporttheFaceSketchIDSystem'sperformancewhenmatchingthecomposite
tophotographsfrom[45].ExamplesurveillancedatausedinthisstudycanbefoundinFigs.
3.2(a,b,c).
5.4ViewedSoftware-GeneratedComposites
Weincluderesultswhenmatching(viewedsoftware-generatedcomposite,photograph)pairs
fromthePRIP-VSGCdatabasetocompareagainstpreviouslypublishedresults.Asnoted
earlier,thePRIP-VSGCdatabasecontains123pairsfromtheARdatabase,withthree
compositesavailableperpair.CompositesgeneratedusingFACESbyboththeAmerican
andAsianoperatorsaswellasthecompositesgeneratedusingIdenti-Kitareusedtoevaluate
theperformanceofbothalgorithms.Boththeholisticandcomponent-basedalgorithmsare
trainedontheCUHK-VHDCdatabaseforthisexperiment.Wenotethatasubsetofthe
CUHK-VHDCdatabasecontainspairsfromtheARdatabasewhicharenotincludedsince
theywouldbiasthematchingperformanceifusedduringalgorithmtraining.Wedonot
includetheperformanceofCOTSsystemsinthisexperimentastheycanbefoundin[12].A
galleryof10,000mugshotsfromthePCSOisusedtoextendthegalleryintheseexperiments
tobeconsistentwiththatin[12].Whilewehaveagerange,genderandraceinformation
tothegalleryset,onlygender-baseddemographicwasreportedin[12].Thus,
toprovideafaircomparisonwith[12],allretrievalratesforthe(viewedsoftware-generated
composite,photograph)experimentsdonotinvolveanydemographic
Chapter6discussestheresultsoftheexperimentslistedabove.
25
CHAPTER6
EXPERIMENTALRESULTS
ThefollowingsectionsdescribetherecognitionperformanceoftheFaceSketchIDSystem.
Section6.1andSection6.2describetheperformanceofhand-drawnandsoftware-generated
composites,respectively.InSection6.3,wediscusstheperformanceoftheFaceSketchIDSys-
temonsurveillancecomposites.Finally,wereporttheFaceSketchIDSystem'sperformance
onviewedsoftware-generatedcompositesinSection6.4toallowforcomparisonsagainst
previouslypublishedresults.
6.1Hand-DrawnComposites
Fig.6.1showsrankretrievalresultsoftheexperimentsinvolvinghand-drawncomposites.
Fortheholisticalgorithm(Fig.6.1(a)),thechoiceofdatausedtotrainthealgorithmhas
littlectontherecognitionperformance,withtheexceptionoftrainingononlythepairs
fromtheCUHK-VHDCdatabase.ComparingRank-100andRank-200retrievalrates,the
besttrainingset(PRIP-HDC+CUHK-VHDC+PRIP-VHDC)fromtheworsttrain-
ingset(CUHK-VHDC)byapproximately2.4%and2.6%,respectively.Itisinterestingto
notethatwhiletheviewedhand-drawncompositecombinationsoftrainingdataexhibit
similarperformance,trainingtheholisticalgorithmontheCUHK-VHDCdatabasealone
ismarkedlyworsethananyothercompositedatabase.Oneexplanationforthisisthat
trainingononly(viewedhand-drawncomposite,mugshot)pairslikelycausesthesubspaces
tobebiasedtowards(viewedhand-drawncomposite,photograph)matching(inwhichtex-
tureandstructureareextremelyaccurate).Intheidealcase,inwhichthe(hand-drawn
composite,mugshot)pairsareextremelysimilar,usingtheCUHK-VHDCdatabasetotrain
theholisticalgorithmmaybeadvantageous.Amorecommoncaseincompositerecognition
isthatcertainaspectsofthecompositeareinaccurateorcontainatamountof
26
(a)(b)(c)
(d)(e)(f)
Figure6.1:Resultsfortheholistic(a,d)andcomponent-based(b,e)algorithmswhen
matchinghand-drawncompositestomugshots.Thebestperformanceisachievedthrough
afusionoftheholisticandcomponent-basedalgorithmmatchscores(c,f).ThreeCOTS
matchersareincludedin(c,f)asabaseline.
noise(introducedbytheartistintheblanks"ofthewitness'orvictim'sdescription).
Thus,itisunderstandablethatatrainingsetconsistingofboth(hand-drawncomposite,
mugshot)and(viewedhand-drawncomposite,mugshot)pairsresultsinthebestrecognition
performancefortheholisticalgorithm.
Bycontrast,thecomponent-basedalgorithm(Fig.6.1(b))hastheworstrecognition
performancewhentrainedonthePRIP-HDCdatabasealone.Thereasonforthisisthat,at
thecomponentlevel(e.g.onlyconsideringthemouth),therecanbeatamountof
noiseifthecomponentsaremisaligned.Incaseswherethecomponentalignmentisd
subspaceslearnedarelikelybynoisemorethanthestructureandtextureofthe
27
(a)(b)
(c)
Figure6.2:ExamplesofsuccessfulRank-1matchesof(hand-drawncomposite,mugshot)
pairs(a,b).Afailurecase(c)showsarelativelyaccuratecompositewhichwasreturnedat
arankhigherthan200.
componentitself.Sincefaciallandmarkdetectionismorestableinthecaseofthe(viewed
hand-drawncomposite,mugshot)pairs,usingthisdatatotrainlikelyreducesnoisepresent
inthelearnedsubspacesasaconsequenceofhavingmoredatawithaccuratealignment.This
explanationthefactthattrainingusingviewedhand-drawncompositeandhand-
drawncompositedataresultsinthebestperformanceofthecomponent-basedalgorithm
whenmatching(hand-drawncomposite,mugshot)pairs.Thatis,withthestabilityprovided
tothelearnedsubspacebytheviewedhand-drawncompositesincombinationwiththehand-
drawncomposites(whichhavealessaccurate,butmoreforensicallyrelevant,appearance),
thecomponent-basedmethodisabletoperformreasonablywell.
ThebestperformancethattheFaceSketchIDSystemisabletoachieveon(hand-drawn
composite,mugshot)pairsresultsfromfusingtheholisticandcomponent-basedmatchscores
(Fig.6.1(c)).TheRank-200performanceaftermatchscorefusionisapproximately5.7%
betterthanthatoftheholisticalgorithmandapproximately8.3%betterthanthatofthe
component-basedalgorithm.WeincludeCOTSretrievalratesinFig.6.1(c)asabaseline
againstourbestachievedmatchingperformance.Examplesofsuccessfulandunsuccess-
28
MethodRank
Holistic30
Component-
4961
Based
Fused8
MethodRank
Holistic556
Component-
206
Based
Fused34
Figure6.3:Examplesofretrievalrankimprovementwhenmatchinghand-drawncomposites
tomugshotsaftermatchscorefusionoftheholisticandcomponent-basedalgorithms.
ful(hand-drawncomposite,mugshot)matchcasescanbefoundinFig.6.2.Examplesof
(hand-drawncomposite,mugshot)matchcasesinwhichmatchscorefusionimprovedthe
performancecanbefoundinFig.6.3.Filteringthegallerybasedondemographicinforma-
tion(age,gender,andrace)tlyimprovestheretrievalratesforbothholisticand
component-basedalgorithmsaswellasthethreeCOTSmatchers(Figs.6.1(d,e,f)).Again,
thehighestretrievalrateperformanceisachievedbythematchscorefusionoftheholistic
andcomponent-basedalgorithms,withaRank-200retrievalrateofapproximately40%.
ThedatainthePRIP-HDCdatabasevariestlyintermsofartistskilland
imagequality(resolution,compressionartifacts,etc.)asitconsistsofimagescollected
frommultiplesources.Somecompositeshavelittleinformationotherthantheoutlines
ofmajorfacialfeatures(Fig.6.4(a)),whereasothershavelessanatomicallycorrectfacial
proportionsresultingina\cartoon-like"appearance(Fig.6.4(b)).Itistoovercome
thechallengesintrinsictothesepoorqualitycompositesand,asexpected,theperformance
oftheFaceSketchIDSystemrsinthesecases.Wealsonotethatthefaceof
thecompositesinthePRIP-HDCdatabase(asdeterminedbyCOTS-2)hasbeenobserved
tobemoreconsistentwiththatofthePRIP-HDCmugshots(Figs.6.5(a,b))ascompared
29
(a)(b)
Figure6.4:Examplesofpoorqualitycompositeswithlittleinformationotherthanoutlines
offacialcomponents(a)orunrealisticappearance(b).Bothpairsshownareretrievedat
higherthanRank-5000afterdemographic
tothemugshotsfromthePCSO(usedtoextendtheexperimentalgallery)(Fig.6.5(c)).
Thislimitationofourexperimentaldatabasesmayimplythatusingtdatathathas
consistentfacialqualitymayresultintperformanceforalloftheevaluatedface
matchers.However,weexpectthattherelativeperformanceoftheFaceSketchIDSystem
andtheCOTSfacematcherswillremainconsistentwiththeresultsreportedabove.
Figs.6.6(a),(b),and(c)showexampleretrievalsfromthesequesteredsetafterde-
mographic(usingthePRIP-HDC+CUHK-VHDC+PRIP-VHDCdatabasetotrain
bothalgorithms).Forthese32(hand-drawncomposite,mugshot)pairs,theFaceSketchID
SystemisabletomatchtwosubjectsatRank-1,ninebyRank-100and13byRank-200
30
(a)(b)(c)
Figure6.5:COTS-2facescoresforthePRIP-HDCdatabasecomposites(a)
(

=4
:
89,
˙
=1
:
48),thePRIP-HDCdatabasemugshots(

=5
:
89,
˙
=1
:
37),andthe
PCSOdatabasemugshots(c)(

=6
:
51,
˙
=1
:
37).Theinqualityisrelatively
smallerbetweenthecompositesandthePRIP-HDCmugshotsversusthatofthePCSO
mugshots.Therefore,thereportedperformanceforallfacematchersmaybeboosteddue
tothefactthatitiseasiertomatchimageswithsimilarfacialquality.
(a)(b)
(c)
Figure6.6:Examplesofsuccessfulmatchesof(hand-drawncomposite,mugshot)pairs
(a,b)fromthesequestereddataset.Afailurecase(c)showsarelativelyaccuratecomposite
whichwasreturnedatarankhigherthan200.
31
3682417
5442
Figure6.7:AcompositeandphotographsdepictingDzhokharTsarnaev.Retrievalranks
fortheFaceSketchIDSystemarelistedbelowthecorrespondingphotograph.Allranksare
afterdemographic(15-25yearold,white,male).
afterdemographic.Notethatmatching13outof32pairsbyRank-200equates
toaretrievalrateof40.63%,whichisroughlyequaltothebestresultsreportedinthe
cross-validationexperimentsabove.
Fig.6.7showsthecompositeandthephotographsdepictingDjokarTsarnaevalong
withtherankatwhichtheywereretrieved(afterdemographicAsnotedin[45],
thesephotographswouldlikelynotbeinalawenforcementagency'sdatabase.Nevertheless,
theFaceSketchIDSystemisabletoretrievemultiplephotographsatlowranks(correctly
recognizingeofsixphotographsbyRank-50).
6.2Software-GeneratedComposites
Fig.6.8(a)showstheperformanceoftheholisticandcomponent-basedalgorithmsinaddi-
tiontotheCOTSmatcherswhenmatching(software-generatedcomposite,mugshot)pairs.
Theholisticalgorithmperformswell,achievinga8%retrievalratebyRank-25,14.6%by
Rank-100,and20%byRank-200.Incontrastwiththeholisticalgorithm,thecomponent-
basedalgorithmperformsworse,achievingRank-25,-100,and-200retrievalsratesof8%,
32
(a)(b)
Figure6.8:Resultsfortheholisticandcomponent-basedalgorithmswhenmatching
software-generatedcompositestomugshotsbefore(a)andafter(b)demographic
COTSmatchersareincludedasabaseline.
10.6%,and12%,respectively.Thismaybeduetothefactthatitistoprecisely
recognize(i.e.matchatlowranks)anindividualbasedpurelyonasetofcomponentsfrom
thesoftware-generatedcomposite,asitishardtoachievethesamelevelofspcityasis
possiblewhendrawingfacialcompositesbyhand.Twoscenarioscontributetothisculty:
(i)aparticularcomponentmatchesveryhighlywithanimpostorand/or,(ii)overall,the
componentsinthecompositearenottlysimilartothoseinthemugshot.Filter-
ingwithdemographicinformationimprovestheRank-200performanceoftheholisticand
composite-basedalgorithmswhenmatching(software-generatedcomposite,mugshot)pairs
(Fig.6.8(b))to38.6%and32%,respectively.
AsmentionedinSection6.1,bothalgorithmsbfromtheuseoftrainingdata
thatmirrorsthedatausedduringtesting.Becausewedonothaveenough(software-
generatedcomposite,mugshot)pairs,theonlytrainingdatausedwerefromtheCUHK-
VHDCdatabases.Thus,whileweachieverespectableRank-200retrievalratesofapprox-
imately20%and12%fortheholisticandcomponent-basedalgorithms,respectively,we
33
(a)(b)
(c)
Figure6.9:ExamplesofsuccessfulRank-1matchesof(software-generatedcomposite,
mugshot)pairs(a,b).Afailurecase(c)showsarelativelyaccuratecompositewhichwas
returnedatarankhigherthan200.
believethattheperformanceofbothalgorithmscouldbeimprovedifmoreoperational
software-generatedcompositedatawereavailable.
Aswiththehand-drawncomposites,scorefusion(usingthesumrule)ofthetwoalgo-
rithmsresultsinthebestretrievalratesfortheFaceSketchIDSystemon(software-generated
composite,mugshot)pairs.FusionimprovestheRank-200performancetoapproximately
32%beforedemographicand44%afterdemographicExamplesofsuccess-
fulandunsuccessful(software-generatedcomposite,mugshot)matchcasescanbefoundin
Fig.6.9.Examplesof(software-generatedcomposite,mugshot)matchcasesinwhichmatch
scorefusionimprovedtheperformancecanbefoundinFig.6.10.Trueacceptrates(TAR)at
falseacceptrates(FAR)of0.1%and1.0%arereportedinTable6.1forboththePRIP-HDC
andthePRIP-SGCdatabasestofacilitatecomparisonswithotherpublishedresults.
Itisworthnotingthatinmanycaseswherefacialcompositeswerenotsuccessfully
matchedtotheircorrectmugshotmate,theRank-1retrievalismoresimilarinappearance
tothecompositethanthetruemate.WeviewthisnotasafailureoftheFaceSketchID
System,butaresultoftheinherentinsynthesizinganaccuratecompositewhich
34
MethodRank
Holistic5
Component-
467
Based
Fused1
MethodRank
Holistic492
Component-
146
Based
Fused6
Figure6.10:Examplesofretrievalrankimprovementwhenmatchingsoftware-generated
compositestomugshotsaftermatchscorefusionoftheholisticandcomponent-based
algorithms.
Table6.1:Trueacceptrates(TAR)atfalseacceptrates(FAR)of0.1%and1%forthe
FaceSketchIDSystemandthreetCOTSmatchersafterdemographic
ScoreslistedfortheFaceSketchIDSystemusingthetrainingsetswhichprovidethe
highestretrievalratesforbothalgorithmsandfusingthematchscores.Standarddeviations
ofthe5-foldcross-validationarereportedwhentestingonthePRIP-HDCdatabase.
Matcher
TestingTAR@FAR=
Database0.1%1.0%
FaceSketchID
PRIP-HDC17.9%

3.9%54.6%

3.1%
PRIP-SGC27.1%65.4%
COTS-1
PRIP-HDC12.2%

4.9%38.1%

4.7%
PRIP-SGC12.9%47.3%
COTS-2
PRIP-HDC11.5%

2.8%49.1%

25.5%
PRIP-SGC17%100%
COTS-3
PRIP-HDC6.4%

1.8%24.9%

5%
PRIP-SGC6.9%23.2%
canbedueto(i)inaccurateorrathervaguedescriptionofthesuspectprovidedbythe
witness,(ii)theagebetweenthetimethesuspect'smugshotwascapturedand
whenhewasseenbyawitness/victim,and(iii)inexperienceoftheforensicartistsorthe
limitationsofthecompositesoftware.Examplesofsuch(hand-drawncomposite,mugshot)
and(software-generatedcomposite,mugshot)pairsareshowninFig.6.11.
Fig.6.12showstheperformanceoftheFaceSketchIDSystemon75hand-drawncompos-
35
(a)(b)(c)
(d)(e)(f)
Figure6.11:Examplesofahand-drawncomposite(a)andasoftware-generatedcomposite
(d)forwhichtheRank-1matchisanimpostor(b,e)thatismoresimilarinappearenceto
thecompositethanthegenuinemugshotmatch(c,f).
itesand75software-generatedcompositesdepictingthesameindividualsbothbeforeand
afterdemographic[15].Forthese75subjects,ourexperimentalresultsindicatethat
thesoftware-generatedcompositesaremoreaccuratethanthehand-drawncomposites(32%
vs.12%beforeatRank-200,respectively).Inanattempttoprovideanunbiased
comparison,thealgorithmsaretrainedonthe(viewedhand-drawncomposite,photograph)
pairsfromtheCUHK-VHDCdatabase.Becauseofthis,itissurprisingthatthesoftware-
generatedcompositesperformtlybetterthanthehand-drawncomposites.
Themostprobableexplanationforthisisthatthetexturesinthesoftware-generated
composites,ascomparedwiththehand-drawncomposites,morecloselymatchthemugshots.
Sp,FACESusescomponentsfromvisible-lightimagestocreateacomposite(giving
thecompositeaphoto-realisticappearance).Thus,whilethehand-drawncompositemay
beasaccurateasitssoftware-generatedcounterpart,thenatureoftheproblem(recognition
bycomparisonagainstamugshotgallery)mayfavorthesoftware-generatedcomposites.
However,weneedtocollectmoresoftware-generatedcompositedata(usingothersoftware
kits)tomakeaclaimregardingthesuperiorityofagivencompositemodality.
36
Figure6.12:ResultsfortheFaceSketchIDSystemwhenmatching75hand-drawn
compositesand75software-generatedcompositestomugshotsbeforeandafter
demographicCOTSmatchersareincludedasabaseline.
6.3SurveillanceComposites
Table6.2showstheretrievalranksof(surveillancecomposite,mugshot)pairsfortheFace-
SketchIDSystemand(surveillanceimage,mugshot)pairsfortheCOTSmatchersafterde-
mographicWhenusingsurveillancecomposites,theFaceSketchIDSystemisable
toachieveresultscomparabletoCOTSsystemswhenusingsurveillanceframes.
Wenotethatitistomakemeaningfulinferencesfromthesmallnumberof
(surveillancecomposite,mugshot)pairsusedinthisstudy.Further,thesurveillanceimages
usedinourstudyareofreasonablyhighqualitysuchthat,ingeneral,theuseofsurveil-
lancecompositesisunnecessary(astheCOTSmatchersdowellonthesurveillanceframes).
However,caseswithhighlyose,occluded,orblurredfacespresentchallengestostate-
of-the-artunconstrainedfacerecognitionsystems(suchasthoseinFig.2.1).Wehopeto
37
Table6.2:Retrievalranksfor(surveillancecomposite,mugshot)obtainedfromthe
FaceSketchIDSystemand(surveillanceimage,mugshot)obtainedfromthethreeCOTS
matchers.RanksmarkedasFTEindicatetheCOTSmatcherfailedtoenrollthequery
surveillancemugshot.Allretrievalranksagalleryof100,000subjectsafter
demographic
SubjectFaceSketchIDCOTS-1COTS-2COTS-3
111460
20
95,962
2
11
15,131
332
2
FTE367
422
11
733
(a)(b)(c)
Figure6.13:Asurveillanceframe(a)ofTamerlanTsarnaevwasusedtocreatea
high-qualitysurveillancecomposite(drawnbyJaneWankmiller[11])(b)toimprove
performancewhenmatchingagainstaphotograph(c).Afterdemographic(20-30
yearold,white,male),theFaceSketchIDSystemisabletoretrievethephotographby
Rank-20.
acquireadditionalsurveillancedatatoevaluatetheperformanceoftheFaceSketchIDSystem
inmorechallengingsurveillancecases.
Fig.6.13showsasurveillancecompositeofTamerlanTsarnaevwhichwascreatedusing
apoorqualitysurveillanceframe.AswithDjokarTsarnaev,thephotographsshownof
Tamerlanwouldnotbeinthemugshotdatabase.However,theFaceSketchIDSystemis
abletomatchTamerlan'scompositetohisphotographatRank-2113andRank-20before
andafterdemographicrespectively.Thisresultiscomparabletotheperformance
achievedusingthemostaccurateCOTSsystemwhenmatchingthebest(surveillanceframe,
38
Figure6.14:Resultsfortheholistic,component-based,andfusedalgorithmsonviewed
software-generatedcompositescreatedusingFACESandIdentiKit.Forthecomposites
createdusingFACES,twooperators(anAmericanandanAsian)createdcomposites[12].
photograph)pairreportedin[45].
6.4ViewedSoftware-GeneratedComposites
Hanetal.[12]reportRank-1,Rank-100,andRank-200retrievalratesof10.6%,65%,and
73.2%whenmatchingviewedsoftware-generatedcomposites(createdbyanAmericanoper-
ator)tophotographs,respectively.TheFaceSketchIDSystemisabletoachievecomparable
Rank-1,Rank-100,andRank-200retrievalratesofapproximately14.6%,67.4%,75.6%on
thesamedatasetafterfusingthematchscoresoftheholisticandcomponent-basedalgo-
rithms(Fig.6.14).
Hanetal.alsoreportthematchingperformancewhenFACEScompositesarecreatedby
anAsianuser.SimilartotheAmerican-createdcomposites,theFaceSketchIDSystemisable
toachievehigherretrievalrates,withanimprovementatRank-200ofapproximately4.9%
overthatin[12].Whenmatchingsoftware-generatedcompositescreatedusingIdenti-Kit
tomugshotsusingtheFaceSketchIDSystem,weobserveasimilarperformancedegradation
comparedwithcompositescreatedusingFACESaswasreportedin[12].Forthecom-
positescreatedwithIdenti-Kit,matchscorefusionviathesumruledoesnotimprovethe
39
performanceoftheFaceSketchIDSystembecauseofthepoorperformanceoftheholistic
algorithm(althoughanotherfusionmethodologymaybehelpful).Thiscanbeexplainedby
thefactthattheIdenti-Kitcompositeshavelittleinformationotherthanoutlinesoffacial
components.Forthesepairs,thecomponent-basedalgorithmdetailedinthisthesisachieves
comparableresultswiththehighestRank-200retrievalratereportedin[12].
40
CHAPTER7
SUMMARYANDFUTUREWORK
Facialcompositesdrawnbyforensicartists(hand-drawncomposites)orcreatedusingsoft-
ware(software-generatedcomposites)areroutinelyusedbylawenforcementagenciestoassist
inidenandapprehensionofsuspectsinvolvedincriminalactivities,especiallywhen
nophotographofthesuspectatthecrimesceneisavailable.Thisthesisimprovesupon
theusabilityandrecognitionperformanceofthesecompositesthrough:(i)Anexploration
ofcompositeusecases,(ii)developmentoftheFaceSketchIDSystem,whichprovideslaw
enforcementagenciesatoolwithwhichtomoreelymakeuseoffacialcompositedata,
and(iii)aninvestigationoftheoftrainingthetwoalgorithmsont(composite,
photograph)databasestoachievethebestmatchingperformance.
TheFaceSketchIDSystemcombinesthestrengthsoftwotrepresentationand
matchingalgorithms(holisticandcomponent-based)toachievestate-of-the-artaccuracies
forboth(hand-drawncomposite,mugshot)and(software-generatedcomposite,mugshot)
pairs.Wealsoshowtheperformanceofsurveillancecomposites,whichareusedforpoor
qualitysurveillanceimageswhereCOTSsystemsareexpectedtofail.Thisscenarioislikely
tobecomeanimportantapplicationoffacialcompositestomugshotmatchinggiventhe
growingnumberofsurveillancecamerasaroundtheglobe.Threestate-of-the-artcommercial
matcherswereusedasbaselinesforourexperiments.Filteringofthelargegallery(100,000
mugshots)basedondemographicinformationshowedatimprovementinretrieval
accuracyinallmatchingexperiments.
Tofurtherimprovethesystem'sperformanceandrobustnesswouldrequireacquiring
additional(composite,photograph)pairs.Aspecialemphasisshouldbegivento
surveillancecomposites,spthosewithsevereoseoroccludedfaces.Wearealso
currentlydevelopingmatchingalgorithmsthatcandirectlyusetheverbaldescription(from
41
awitness)inadditiontothefacialcompositetoimprovethematchingperformanceofthe
FaceSketchIDSystem.
42
BIBLIOGRAPHY
43
BIBLIOGRAPHY
[1]TurnerEntertainmentNetworks,\SonofSam,"2014.[Online].Available:http:
//www.trutv.com/library/crime/photogallery/serial-killer-sketches.html?curPhoto=4
[2]K.Taylor,
ForensicArtandIllustration
.CRCPress,2001.
[3]TurnerEntertainmentNetworks,\TedKaczynski:TheUnabomber,"2014.[Online].
Available:http://www.crimelibrary.com/terrorists
spies/terrorists/kaczynski/5.html
[4]FACES4.0,IQBiometrics,http://www.iqbiometrix.com.
[5]I.Lovett,\Losangelesidentifyvideoassaultsuspects,"
NewYorkTimes
,Jan-
uary2011.
[6]L.Gibson,
ForensicArtEssentials
.Elsevier,2008.
[7]A.M.MartinezandR.Benavente,\TheARfacedatabase,"
CVCTechnicalReport
,
vol.24,June1998.
[8]D.Lowe,\Distinctiveimagefeaturesfromscale-invariantkeypoints,"
Int.J.Comput.
Vision
,vol.60,no.2,pp.91{110,January2004.
[9]T.Ojala,M.ainen,andT.a,\Multiresolutiongrayscaleandrotation
invarianttexturewithlocalbinarypatterns,"
IEEETrans.PatternAnal.
Mach.Intell.
,vol.24,no.7,pp.971{987,August2002.
[10]S.MilborrowandF.Nicolls,\Locatingfacialfeatureswithanextendedactiveshape
model,"in
Proc.ECCV
,2008.
[11]L.Best-Rowden,H.Han,C.Otto,B.Klare,andA.Jain,\Unconstrainedfacerecog-
nition:Identifyingapersonofinterestfromamediacollection,"
MSUTech.Report
,
2014.
[12]H.Han,B.Klare,K.Bonnen,andA.Jain,\Matchingcompositesketchestofacephotos:
Acomponentbasedapproach,"
IEEETrans.Inf.ForensicsSecurity
,vol.8,no.1,pp.
191{204,January2013.
[13]A.Jain,B.Klare,andU.Park,\Facematchingandretrievalinforensicsapplications,"
IEEEMultimedia
,vol.19,no.1,pp.20{28,2012.
[14]D.McQuiston-Surrett,L.Topp,andR.Malpass,\Useoffacialcompositesystemsin
USlawenforcementagencies,"
Psychology,CrimeandLaw
,vol.12,no.5,pp.505{517,
2006.
[15]S.Klum,B.Klare,H.Han,andA.Jain,\Sketchbasedfacerecognition:Forensicvs.
compositesketches,"in
Proc.IEEEICB
,2013.
44
[16]X.TangandX.Wang,\Facesketchrecognition,"
IEEETrans.CircuitsSyst.Video
Technol.
,vol.14,no.1,pp.50{57,January2004.
[17]Q.Liu,X.Tang,H.Jin,H.Lu,andS.Ma,\Anonlinearapproachforfacesketch
synthesisandrecognition,"in
Proc.IEEECVPR
,June2005,pp.1005{1010.
[18]D.LinandX.Tang,\Inter-modalityfacerecognition,"in
Proc.ECCV
,2006,pp.13{26.
[19]P.YuenandC.Man,\Humanfaceimagesearchingsystemusingsketches,"
IEEETrans.
Syst.,Man,Cybern.A,Syst.,Humans
,vol.37,no.4,pp.493{504,July2007.
[20]X.WangandX.Tang,\Facephoto-sketchsynthesisandrecognition,"
IEEETrans.
PatternAnal.Mach.Intell.
,vol.31,no.11,pp.1955{1967,November2009.
[21]H.Bhatt,S.Bharadwaj,R.Singh,andM.Vatsa,\MemeticallyoptimizedMCWLD
formatchingsketcheswithdigitalfaceimages,"
IEEETrans.Inf.ForensicsSecurity
,
vol.7,no.5,pp.1522{1535,October2012.
[22]B.Klare,Z.Li,andA.Jain,\Matchingforensicsketchestomugshotphotos,"
IEEE
Trans.PatternAnal.Mach.Intell.
,vol.33,no.3,pp.639{646,March2011.
[23]B.KlareandA.Jain,\Heterogeneousfacerecognitionusingkernelprototypesimilar-
ities,"
IEEETrans.PatternAnal.Mach.Intell.
,vol.35,no.6,pp.1410{1422,June
2013.
[24]S.LiandA.Jain(eds.),
HandbookofFaceRecognition
,2nded.Springer,2011.
[25]X.Gao,J.Zhong,andC.Tian,\Facesketchsynthesisalgorithmbasedone-hmm
andselectiveensemble,"
IEEETrans.CircuitsSyst.VideoTechnol.
,vol.18,no.4,pp.
487{496,April2008.
[26]Y.Zhang,C.McCullough,J.Sullins,andC.Ross,\Hand-drawnfacesketchrecognition
byhumansandapca-basedalgorithmforforensicapplications,"
IEEETrans.Syst.,
Man,Cybern.A,Syst.,Humans
,vol.40,no.3,pp.475{485,May2010.
[27]R.UhlandN.Lobo,\Aframeworkforrecognizingafacialimagefromapolicesketch,"
in
Proc.IEEECVPR
,June1996,pp.586{593.
[28]E-FIT,VisionMetricLtd.,http://www.vision-metric.com.
[29]C.Frowd,P.Hancock,andD.Carson,\EvAholistic,evolutionaryfacialimaging
techniqueforcreatingcomposites,"
ACMTrans.Appl.Percept.
,vol.1,no.1,pp.19{39,
July2004.
[30]Identi-Kit,Identi-KitSolutions,http://www.identikit.net/.
[31]G.WellsandL.Hasel,\Facialcompositeproductionbyeyewitnesses,"
CurrentDirec-
tionsinPsychologicalScience
,vol.16,no.1,pp.6{10,February2007.
45
[32]C.Frowd,D.Carson,H.Ness,D.McQuiston-Surrett,J.Richardson,H.Baldwin,and
P.Hancock,\Contemporarycompositetechniques:Theimpactofaforensically-relevant
targetdelay,"
LegalandCriminologicalPsychology
,vol.10,no.1,pp.63{81,February
2005.
[33]R.Geiselman,R.Fisher,D.MacKinnon,andH.Holland,\Eyewitnessmemoryen-
hancementwiththecognitiveinterview,"
AmericanJournalofPsychology
,vol.99,pp.
385{401,1986.
[34]K.Messer,J.Matas,J.Kittler,J.Luettin,andG.Maitre,\Xm2vtsdb:theextended
m2vtsdatabase,"in
Proc.ofInternationalConferenceonAudio-andVideo-BasedPer-
sonAuthentication
,vol.24,June1999,pp.72{77.
[35]W.Zhang,X.Wang,andX.Tang,\Coupledinformation-theoreticencodingforface
photo-sketchrecognition,"in
Proc.IEEECVPR
,2011.
[36]P.J.Phillips,H.Moon,P.J.Rauss,andS.Rizvi,\Theferetevaluationmethodologyfor
facerecognitionalgorithms,"
IEEETrans.PatternAnal.Mach.Intell.
,vol.22,no.10,
pp.1090{1104,2000.
[37]Itseez,\TheOpenCVLibrary,"www.opencv.org,2000.
[38]G.Guennebaud,B.Jacob
etal.
,\Eigenv3,"http://eigen.tuxfamily.org,2010.
[39]Digia,\Qt5,"http://qt-project.org/,2013.
[40]J.Klontz,B.Klare,S.Klum,A.Jain,andM.Burge,\Opensourcebiometricrecogni-
tion,"in
Proc.IEEEBTAS
,2013.
[41]K.Bonnen,B.Klare,andA.K.Jain,\Component-basedrepresentationinautomated
facerecognition,"
IEEETrans.Inf.ForensicsSecurity
,vol.8,no.1,pp.239{253,Jan-
uary2012.
[42]E.MeyersandL.Wolf,\Usingbiologicallyinspiredfeaturesforfaceprocessing,"
Int.
J.Comput.Vision
,vol.76,no.1,pp.93{104,2008.
[43]X.TanandB.Triggs,\Enhancedlocaltexturefeaturesetsforfacerecognitionunder
lightingconditions,"
IEEETrans.ImageProc.
,vol.19,no.6,pp.1635{1650,
June2010.
[44]X.WangandX.Tang,\Randomsamplingforsubspacefacerecognition,"
Int.J.Com-
put.Vision
,vol.70,no.1,pp.91{104,2006.
[45]J.KlontzandA.Jain,\Acasestudyonunconstrainedfacialrecognitionusingthe
bostonmarathonbombingssuspects,"
MSUTech.Report
,2013.
46