Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation

Published in The 38th AAAI Conference on Artificial Intelligence, 2024

Recommended citation: A. Davtyan and P. Favaro. (2024). "Learn the Force We Can: Enabling Sparse Motion Control in Multi-Object Video Generation." Proceedings of the 38th AAAI Conference on Artificial Intelligence.

[PDF] [Project Page]

Abstract

We propose a novel unsupervised method to autoregressively generate videos from a single frame and a sparse motion input. Our trained model can generate unseen realistic object-to-object interactions. Although our model has never been given the explicit segmentation and motion of each object in the scene during training, it is able to implicitly separate their dynamics and extents. Key components in our method are the randomized conditioning scheme, the encoding of the input motion control, and the randomized and sparse sampling to enable generalization to out of distribution but realistic correlations. Our model, which we call YODA, has therefore the ability to move objects without physically touching them. Through extensive qualitative and quantitative evaluations on several datasets, we show that YODA is on par with or better than state of the art video generation prior work in terms of both controllability and video quality. For videos, visit our project website https://araachie.github.io/yoda.