The recent developments in 3D technology have profoundly impacted the digital world. Thus, the geometric modeling of 3D shapes through surfaces and curves plays an ever more critical role in society than before. Similarly, regularly sampled data on volumes (3D images) have found a wide range of applications in various areas, such as medicine, biology, engineering, military, entertainment, etc. Compared to 2D images, 3D images have one more dimension to store information. Therefore, they can directly approximate the physical world without first slicing up the object under investigation, which, however, leads to much more data complexity and technical difficulty. To address the problem of increased dimensionality, one may often leverage the lower dimensional geometric features abundant in various applications to facilitate computational tasks. Thus, how geometry could assist in 3D image processing is an intriguing topic, which we explore from three aspects in this dissertation.With three real-world problems, we present novel geometry-aided 3D image processing algorithms with contributions in several areas, including biology, computer vision, and computer graphics. In these applications, we demonstrate how geometric structures are vital in guiding image processing, improving accuracy, and facilitating downstream analyses.First, we analyze the 3D location-dependent fluorescence data of plant cells. We showed that with the volumetric diffusion using the 3D Laplacian matrix, we could produce an accurate adaptive local threshold to segment cytoskeleton in 3D microscope images of plant cells. Moreover, we propose several indices describing geometric and topological characteristics of cytoskeletons to help biologists understand actin filament dynamics in plant cells.Second, we employ 3D-direction-dependent lighting conditions and introduce shadow masks to our face relighting pipeline generated from rough geometry to remove or add more accurate shadows for human face images. In addition, with the assistance of geometric information, we may generate relatively accurate spherical harmonics coefficients (a representation for low-frequency 3D-direction fields) to model the illumination for the relighting task.Last, we explore how geometry shapes could help with a deep learning algorithm for a particular field of both locations and directions, the dynamical neural radiance field for human heads. Given a set of input images or a video of a talking human head, we first embed a geometric model of the human head shape into the 3D implicit field. Then, one set of latent codes anchored on a morphable 3D surface mesh automatically turns the human face with any specific pose and expression into a radiance field. With the radiance field, it is straightforward to assemble RGB values for rays from arbitrary camera positions along chosen directions to form photorealistic rendering of novel views of the same person with changing expressions. Thus, we may reanimate the human head with better realism than a crude textured surface mesh. Furthermore, with the help of the radiance field, we may refine the geometric surfaces to align better with the input videos if surface meshes are needed for downstream applications.Last, we explore how geometry shapes could help with a deep learning algorithm for a particular field of both locations and directions, the dynamical neural radiance field for human heads. Given a set of input images or a video of a talking human head, we first embed a geometric model of the human head shape into the 3D implicit field. Then, one set of latent codes anchored on a morphable 3D surface mesh automatically turns the human face with any specific pose and expression into a radiance field. With the radiance field, it is straightforward to assemble RGB values for rays from arbitrary camera positions along chosen directions to form photorealistic rendering of novel views of the same person with changing expressions. Thus, we may reanimate the human head with better realism than a crude textured surface mesh. Furthermore, with the help of the radiance field, we may refine the geometric surfaces to align better with the input videos if surface meshes are needed for downstream applications.
Read
