3D modeling of human faces has emerged as a widely studied field within computer vision, with applications in virtual reality, animation, medical imaging, and more, and is going to be a very promising area of research in the coming years. Specifically, 3D modeling of single view face images has been known to be a particularly challenging task because of its ill-posed nature, but it comes with a wide range of applications. Two of the most promising approaches in this regard are template-based approaches, such as the 3D morphable model (3DMM) of faces, and implicit 3D modeling approaches, such as implicit 3D-GANs. Over the years, 3DMM based approaches have improved their capability to synthesize highly controllable 3D faces and generate accurate 3D face reconstructions of faces images, while implicit 3D-GANs have been shown to generate high-fidelity 3D faces. However, even after significant advancements in these approaches, face generative tasks, such as face inpainting and controllable face generation, are still primarily performed in the 2D image space. Faces are structured 3D objects with inherent attributes such as shape, pose and albedo, and their projection in 2D images is affected by external factors such as illumination and camera parameters. Without an explicit consideration of these factors, existing generative approaches have to implicitly model facial geometry and appearance. We contend that generative models that explicitly take these factors into account can leverage 3D priors, and more controllably and accurately generate new faces, or fill in the missing regions in face images.Further, the ill-posed nature of reconstructing 3D models from monocular face images makes it a challenging task. This becomes even more challenging when facial occlusions such as face masks, glasses, microphones, etc. are involved. This highlights the need for the development of occlusion-aware 3D face reconstruction algorithms. We argue that such an algorithm should be (i) robust to occlusions of varying types, sizes, and locations; and (ii) capable of generating diverse, yet realistic solutions for the occluded parts to account for a lack of unique solution. This thesis addresses the aforementioned challenges, by presenting the following: (i) a 3D-aware face inpainting approach that considerably improves upon 2D-based baselines, especially under challenging conditions; (ii) a controllable 3D face generation approach that combines the capabilities of 3DMMs and implicit 3D-GANs by learning correspondence between them; and (iii) an occlusion-aware 3D face reconstruction approach that generates a diverse, yet realistic set of 3D reconstructions from a single occluded face image, with lower error on the visible face regions than the baselines.
Read
