Generalizing Object-Centric Task-Axes Controllers using Keypoints

TL;DR

This paper introduces a modular framework for robotic manipulation that generalizes across diverse object properties by learning object-centric controllers from visual data, enabling flexible and scalable task execution.

Contribution

The paper presents a novel approach to learn modular, object-centric task-axes controllers from visual input, improving generalization across varied object shapes and sizes.

Findings

Successfully generalizes to objects with different sizes and shapes

Outperforms monolithic policies in diverse manipulation tasks

Demonstrates effective multi-view dense correspondence learning

Abstract

To perform manipulation tasks in the real world, robots need to operate on objects with various shapes, sizes and without access to geometric models. It is often unfeasible to train monolithic neural network policies across such large variance in object properties. Towards this generalization challenge, we propose to learn modular task policies which compose object-centric task-axes controllers. These task-axes controllers are parameterized by properties associated with underlying objects in the scene. We infer these controller parameters directly from visual input using multi-view dense correspondence learning. Our overall approach provides a simple, modular and yet powerful framework for learning manipulation tasks. We empirically evaluate our approach on multiple different manipulation tasks and show its ability to generalize to large variance in object size, shape and geometry.