Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals

Stefan Stojanov, David Wendt, Seungwoo (Simon) Kim, Rahul Venkatesh, Kevin Feigelis, Jiajun Wu, Dan Yamins

Self-Supervised Learning of Motion Concepts by Optimizing Counterfactuals: 3 upvotes on Hugging Face Daily Papers, #21 of 27 papers on 2025-03-27. Day-by-day upvote history.

Estimating motion in videos is an essential computer vision problem with many downstream applications, including controllable video generation and robotics. Current solutions are primarily trained using synthetic data or require tuning of situation-specific heuristics, which inherently limits these models' capabilities in real-world contexts. Despite recent developments in large-scale self-supervised learning from videos, leveraging such representations for motion estimation remains relatively underexplored. In this work, we develop Opt-CWM, a self-supervised technique for flow and occlusion estimation from a pre-trained next-frame prediction model. Opt-CWM works by learning to optimize counterfactual probes that extract motion information from a base video model, avoiding the need for fixed heuristics while training on unrestricted video inputs. We achieve state-of-the-art performance for motion estimation on real-world videos while requiring no labeled data.

Paper page on Hugging Face · arXiv

Data: hysts-bot-data/daily-papers-stats and the Daily Papers API. Open data: tardellirs/paper-pulse-data. Sister project: Model Pulse, the download history of every model on the Hub.