A technology once seen only in television dramas, automatic face recognition systems are now deployed in many important applications. Recognition of individuals from facial images is used for de-duplication of identification cards (e.g., driver's licenses and passports), verification of prisoner identities, and tag suggestions for personal photo collections. Face images acquired in such applications are conducive to the current capabilities of face recognition algorithms; state-of-the-art systems are able to recognize constrained face images with close to 99% accuracy. However, the performance of automatic face recognition degrades when processing unconstrained face images (i.e., image acquisition is uncontrolled and subjects may be uncooperative). In such scenarios, a face image may simultaneously contain multiple confounding factors, or covariates, such as variations in facial pose, illumination, expression, occlusion, resolution, and facial aging.The first contribution of this dissertation is a framework for matching a collection of unconstrained face media (images, videos, 3D model, demographics, facial sketch) when multiple instances of a subject's face are available. This is particularly relevant to forensicinvestigations where the goal is to identify a \person of interest" based on low quality face images and videos (e.g., captured by surveillance cameras or mobile phones of bystanders) and other information compiled during the investigation (e.g., gender, race, age, facial sketch). While traditional face matching methods generally take a single media (i.e., a still face image, video track, or face sketch) as input, this work considers using the entire gamut of media as a probe to generate a single candidate list for the person of interest. We show that the proposed approach boosts the likelihood of correctly identifying the person of interest through the use of different fusion schemes, 3D face models, and incorporation of quality measures for fusion and video frame selection.Secondly, this dissertation proposes an automatic measure of the quality of an unconstrained face image, where quality is defined as a measure of the utility of a face image to automatic face recognition. A large database of unconstrained face images is first annotated with target quality labels using two methods: (i) human assessments of face image quality, and (ii) quality values computed from similarity scores. A support vector regression model trained on image features automatically extracted using a deep convolutional neural network is then used to predict the quality of an unseen face image. Results demonstrate that target quality values from human assessments and similarity scores are not highly correlated with each other, but both are useful for applications of face image quality, such as to reject low-quality face images prior to matching and to rank a collection of face images based on quality.Finally, this dissertation addresses the important problem of facial aging, which is a challenge for both constrained and unconstrained applications. The two underlying premises of automatic face recognition are uniqueness and permanence. We investigate the permanence property by addressing the following: Does face recognition ability of state-of-the-art systems degrade with elapsed time between enrolled and query face images? If so, what is the rate of decline with respect to the elapsed time? While previous studies have reported degradation in accuracy, no formal statistical analysis of large-scale longitudinal data has been conducted. We conduct such an analysis on two mugshot databases, which are the largest facial aging databases studied to date in terms of number of subjects, images per subject, and elapsed times. Longitudinal analysis shows that despite decreasing genuine scores, 99% of subjects can still be recognized at 0.01% FAR up to approximately 6 years elapsed time, and that age, sex, and race only marginally influence these trends. The methodology presented in this dissertation should be periodically repeated to determine age-invariant properties of face recognition as state-of-the-art evolves to better address facial aging.
Read
