New perspectives in neural architecture search : architecture embeddings, efficient performance estimations, and their applications

Neural architecture search (NAS) has made significant strides in recent years, but challenges remain in terms of the stability of search performance and the high computational requirements of sampling-based NAS. Studying architecture representations offers a promising solution to these challenges, as it encourages neural architectures with similar structures or computations to cluster together. This helps to map neural architectures with similar performance to the same regions in the latent space and leads to smoother transitions in the latent space, benefiting downstream search. Additionally, learning curve extrapolation can accelerate the search process by estimating the final validation accuracy of a neural network from the learning curve of a partially trained network. Overall, understanding the neural architecture representations and their associated learning curves through theoretical analysis and empirical evaluations is crucial for achieving stable and scalable NAS.This dissertation presents our contributions to the field of neural architecture search (NAS), which push the limits of NAS and achieve state-of-the-art performance. Our contributions include efficient one-shot NAS via hierarchical masking, addressing the joint optimization problem of architecture representations and search through unsupervised pre-training, improving the generalization ability of architecture representations with computation-aware self-supervised training, and developing a method for facilitating multi-fidelity NAS research.