# Learning Physics-Based Manipulation in Clutter: Combining Image-Based Generalization and Look-Ahead Planning

## TL;DR

This paper presents a method for physics-based manipulation in cluttered environments that combines image-based generalization with look-ahead planning, enabling robots to adapt to various objects and complex interactions in real-world scenarios.

## Contribution

It introduces an approach that uses abstract image representations and physics simulation for multi-step manipulation planning, improving generalization and adaptability.

## Key findings

- Successful transfer from simulation to real-world environments.
- Effective generalization across different object shapes and numbers.
- Enhanced multi-step manipulation capabilities in cluttered scenes.

## Abstract

Physics-based manipulation in clutter involves complex interaction between multiple objects. In this paper, we consider the problem of learning, from interaction in a physics simulator, manipulation skills to solve this multi-step sequential decision making problem in the real world. Our approach has two key properties: (i) the ability to generalize and transfer manipulation skills (over the type, shape, and number of objects in the scene) using an abstract image-based representation that enables a neural network to learn useful features; and (ii) the ability to perform look-ahead planning in the image space using a physics simulator, which is essential for such multi-step problems. We show, in sets of simulated and real-world experiments (video available on https://youtu.be/EmkUQfyvwkY), that by learning to evaluate actions in an abstract image-based representation of the real world, the robot can generalize and adapt to the object shapes in challenging real-world environments.