Adaptive on-device deep learning systems

"Mobile systems such as smartphones, drones, and augmented-reality headsets are revolutionizing our lives. On-device deep learning is regarded as the key enabling technology for realizing their full potential. This is because communication with cloud adds additional latency or cost, or the applications must operate even with intermittent internet connectivity.The key to achieving the full promise of these mobile vision systems is effectively analyzing the streaming video frames. However, processing streaming video frames taken in mobile settings is challenging in two folds. First, the processing usually involves multiple computer vision tasks. This multi-tenant characteristic requires mobile vision systems to concurrently run multiple applications that target different vision tasks. Second, the context in mobile settings can be frequently changed. This requires mobile vision systems to be able to switch applications to execute new vision tasks encountered in the new context.In this article, we fill this critical gap by proposing NestDNN, a framework that enables resource-aware multi-tenant on-device deep learning for continuous mobile vision. NestDNN enables each deep learning model to offer flexible resource-accuracy trade-offs. At runtime,it dynamically selects the optimal resource-accuracy trade-off for each deep learning model to fit the model's resource demand to the system's available runtime resources. In doing so, NestDNN efficiently utilizes the limited resources in mobile vision systems to jointly maximize the performance of all the concurrently running applications.Although NestDNN is able to efficiently utilize the resource by being resource-aware, it essentially treats the content of each input image equally and hence does not realize the full potential of such pipelines. To realize its full potential, we further propose FlexDNN, a novel content-adaptive framework that enables computation-efficient DNN-based on-device video stream analytics based on early exit mechanism. Compared to state-of-the-art earlyexit-based solutions, FlexDNN addresses their key limitations and pushes the state-of-the-artforward through its innovative fine-grained design and automatic approach for generating the optimal network architecture."--Pages ii-iii.