Generating Videos of Zero-Shot Compositions of Actions and Objects
Megha Nawhal, Mengyao Zhai, Andreas Lehrmann, Leonid Sigal, Greg Mori

TL;DR
This paper introduces a novel framework for zero-shot human-object interaction video generation, enabling the creation of unseen action-object combinations in complex scenes, advancing the field of video synthesis.
Contribution
We propose HOI-GAN, a new adversarial model with multiple discriminators for zero-shot human-object interaction video generation, addressing generalization in complex scene synthesis.
Findings
HOI-GAN outperforms existing methods on EPIC-Kitchens and 20BN-Something-Something v2 datasets.
The framework effectively generates unseen action-object combinations.
Qualitative results show realistic and diverse interaction videos.
Abstract
Human activity videos involve rich, varied interactions between people and objects. In this paper we develop methods for generating such videos -- making progress toward addressing the important, open problem of video generation in complex scenes. In particular, we introduce the task of generating human-object interaction videos in a zero-shot compositional setting, i.e., generating videos for action-object compositions that are unseen during training, having seen the target action and target object separately. This setting is particularly important for generalization in human activity video generation, obviating the need to observe every possible action-object combination in training and thus avoiding the combinatorial explosion involved in modeling complex scenes. To generate human-object interaction videos, we propose a novel adversarial framework HOI-GAN which includes multiple…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHuman Pose and Action Recognition · Multimodal Machine Learning Applications · Generative Adversarial Networks and Image Synthesis
