Many applications of representation learning, such as privacy preservation and algorithmic fairness, desire explicit control over some unwanted information being discarded. This goal is formulated as satisfying two objectives: maximizing utility for predicting a target attribute while simultaneously being invariant (independent) to a known sensitive attribute (like gender or race). Solutions to invariant representation learning (IRepL) problems lead to a trade-off between utility and invariance when they are competing. Most existing works are empirical and implicitly look for single or multiple points on the utility-invariance trade-off. They do not explicitly seek to characterize the entire trade-off front optimally and do not provide invariance and convergence guarantees. In this thesis, we address the shortcoming mentioned above by considering simple linear modeling and building upon them. As a first step, we derive a closed-form solution for the global optima of the underlying linear IRepL optimization problem. In further development, we consider neural network-based encoders, where we model the utility of the target task and the invariance to the sensitive attribute via kernelized ridge regressors. This setting leads to a stable iterative optimization scheme toward global/local optima(s). However, such a setting cannot guarantee universal invariance.This drawback motivated us to further study the case where the invariance measure is modeled universally via functions in some reproducing kernel Hilbert spaces (RKHS)s. By modeling the encoder and target networks via functions in some RKHS, too, we derive a closed formula for a near-optimal trade-off, corresponding optimal representation dimensionality, and the associated encoder(s). Our findings have an immediate application to fairness in terms of demographic parity.
Read
