Visual Reaction: Learning to Play Catch with Your Drone
Kuo-Hao Zeng, Roozbeh Mottaghi, Luca Weihs, Ali Farhadi

TL;DR
This paper introduces a new dataset and model for teaching drones to play catch by predicting object movements and planning actions in complex visual environments, outperforming existing baselines.
Contribution
The paper presents a novel dataset and an integrated forecasting and planning model for visual reaction in dynamic environments involving drones.
Findings
Model outperforms tracking-based baselines
Integrated forecaster and planner improve accuracy
Dataset includes 30K throws with diverse object types
Abstract
In this paper we address the problem of visual reaction: the task of interacting with dynamic environments where the changes in the environment are not necessarily caused by the agent itself. Visual reaction entails predicting the future changes in a visual environment and planning accordingly. We study the problem of visual reaction in the context of playing catch with a drone in visually rich synthetic environments. This is a challenging problem since the agent is required to learn (1) how objects with different physical properties and shapes move, (2) what sequence of actions should be taken according to the prediction, (3) how to adjust the actions based on the visual feedback from the dynamic environment (e.g., when objects bouncing off a wall), and (4) how to reason and act with an unexpected state change in a timely manner. We propose a new dataset for this task, which includes…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
Visual Reaction: Learning to Play Catch With Your Drone· youtube
Taxonomy
TopicsHuman Pose and Action Recognition · Anomaly Detection Techniques and Applications · Multimodal Machine Learning Applications
