Video is the dominant modality that people use to consume content and share experiences in time series. The significant expansion of video data available both on the internet and in everyday life has spurred the creation of intelligent systems that can automatically analyze video content and comprehend human actions. Compared to static images, videos describe how the world changes as time elapses. The uniqueness of videos, beyond what can be understood from a single image, is the context of action understanding. Over the last ten years, we have seen huge success in recognizing human actions in a video, by deep neural networks. However, this action recognition has several limitations for real applications. It primarily focuses on recognizing action patterns within only a few seconds. This is still far from progressing to a human-level intelligence of video understanding. People can directly perceive uncurated long videos in the real world. We want the model directly applied to long-form videos, which are untrimmed and contain multiple actions/events. In approaching this challenge, in this thesis, we first study the representation learning of actions/events in long-form videos. We develop models to learn the fine-grained motion representations across multiple actions/events in a video.My research seeks to enable machine visions to represent motions over a long-horizon range, by exploiting the potential of multi-modal video-language contexts. We also address learning the actions jointly performed by a group of people, by modeling their interactions. After that, we investigate leveraging the long-range dependencies of the events in boosting temporal reasoning downstream tasks, including online action detection and spatiotemporal object grounding. Finally, considering the wide applications of video models, we focus on cultivating trustworthiness in the models for long-form videos from static bias mitigation and interpretable reasoning perspectives.
Read
