Visual Reaction: Learning to Play Catch with Your Drone

TL;DR

This paper introduces a new dataset and model for teaching drones to play catch by predicting object movements and planning actions in complex visual environments, outperforming existing baselines.

Contribution

The paper presents a novel dataset and an integrated forecasting and planning model for visual reaction in dynamic environments involving drones.

Findings

Model outperforms tracking-based baselines

Integrated forecaster and planner improve accuracy

Dataset includes 30K throws with diverse object types

Abstract

In this paper we address the problem of visual reaction: the task of interacting with dynamic environments where the changes in the environment are not necessarily caused by the agent itself. Visual reaction entails predicting the future changes in a visual environment and planning accordingly. We study the problem of visual reaction in the context of playing catch with a drone in visually rich synthetic environments. This is a challenging problem since the agent is required to learn (1) how objects with different physical properties and shapes move, (2) what sequence of actions should be taken according to the prediction, (3) how to adjust the actions based on the visual feedback from the dynamic environment (e.g., when objects bouncing off a wall), and (4) how to reason and act with an unexpected state change in a timely manner. We propose a new dataset for this task, which includes 30K throws of 20 types of objects in different directions with different forces. Our results show that our model that integrates a forecaster with a planner outperforms a set of strong baselines that are based on tracking as well as pure model-based and model-free RL baselines. The code and dataset are available at github.com/KuoHaoZeng/Visual_Reaction.