Learning Visual Predictive Models of Physics for Playing Billiards

TL;DR

This paper presents a visual predictive model of physics that enables an agent to plan goal-directed actions in novel environments by simulating physical interactions, demonstrated through a billiards game.

Contribution

The authors introduce a novel object-centric visual prediction model that learns physical laws from raw visual input for planning in unseen environments.

Findings

The model accurately predicts physical interactions in simulated billiards.

It enables goal-directed planning for pushing balls into targets or collisions.

The approach generalizes to new environments without prior exposure.

Abstract

The ability to plan and execute goal specific actions in varied, unexpected settings is a central requirement of intelligent agents. In this paper, we explore how an agent can be equipped with an internal model of the dynamics of the external world, and how it can use this model to plan novel actions by running multiple internal simulations ("visual imagination"). Our models directly process raw visual input, and use a novel object-centric prediction formulation based on visual glimpses centered on objects (fixations) to enforce translational invariance of the learned physical laws. The agent gathers training data through random interaction with a collection of different environments, and the resulting model can then be used to plan goal-directed actions in novel environments that the agent has not seen before. We demonstrate that our agent can accurately plan actions for playing a simulated billiards game, which requires pushing a ball into a target position or into collision with another ball.