Controlling the World by Sleight of Hand

TL;DR

This paper introduces CosHand, a generative model trained on unlabeled videos that predicts the effects of hand-object interactions, enabling realistic image synthesis of future states conditioned on specific actions.

Contribution

It presents a novel action-conditional generative model that learns from unlabeled videos to predict and synthesize future images after hand-object interactions.

Findings

Strong generalization to unseen objects and environments

Ability to generate multiple possible future outcomes

Effective modeling of interaction uncertainty

Abstract

Humans naturally build mental models of object interactions and dynamics, allowing them to imagine how their surroundings will change if they take a certain action. While generative models today have shown impressive results on generating/editing images unconditionally or conditioned on text, current methods do not provide the ability to perform object manipulation conditioned on actions, an important tool for world modeling and action planning. Therefore, we propose to learn an action-conditional generative models by learning from unlabeled videos of human hands interacting with objects. The vast quantity of such data on the internet allows for efficient scaling which can enable high-performing action-conditional models. Given an image, and the shape/location of a desired hand interaction, CosHand, synthesizes an image of a future after the interaction has occurred. Experiments show that the resulting model can predict the effects of hand-object interactions well, with strong generalization particularly to translation, stretching, and squeezing interactions of unseen objects in unseen environments. Further, CosHand can be sampled many times to predict multiple possible effects, modeling the uncertainty of forces in the interaction/environment. Finally, method generalizes to different embodiments, including non-human hands, i.e. robot hands, suggesting that generative video models can be powerful models for robotics.