ATK: Automatic Task-driven Keypoint Selection for Robust Policy Learning

TL;DR

This paper introduces ATK, a method for automatically selecting task-relevant 2D keypoints to improve the robustness and transferability of visuomotor policies in robotic tasks, especially under visual disturbances.

Contribution

ATK is a novel automatic keypoint selection method that optimizes for minimal, task-relevant keypoints to enhance policy robustness and transferability.

Findings

Keypoints improve robustness to visual disturbances.

Automatic selection outperforms manual or random keypoints.

Effective transfer from simulation to real-world tasks.

Abstract

Visuomotor policies often suffer from perceptual challenges, where visual differences between training and evaluation environments degrade policy performance. Policies relying on state estimations, like 6D pose, require task-specific tracking and are difficult to scale, while raw sensor-based policies may lack robustness to small visual disturbances. In this work, we leverage 2D keypoints--spatially consistent features in the image frame--as a flexible state representation for robust policy learning and apply it to both sim-to-real transfer and real-world imitation learning. However, the choice of which keypoints to use can vary across objects and tasks. We propose a novel method, ATK, to automatically select keypoints in a task-driven manner so that the chosen keypoints are predictive of optimal behavior for the given task. Our proposal optimizes for a minimal set of keypoints that focus on task-relevant parts while preserving policy performance and robustness. We distill expert data (either from an expert policy in simulation or a human expert) into a policy that operates on RGB images while tracking the selected keypoints. By leveraging pre-trained visual modules, our system effectively encodes states and transfers policies to the real-world evaluation scenario despite wide scene variations and perceptual challenges such as transparent objects, fine-grained tasks, and deformable objects manipulation. We validate ATK on various robotic tasks, demonstrating that these minimal keypoint representations significantly improve robustness to visual disturbances and environmental variations. See all experiments and more details at https://yunchuzhang.github.io/ATK/.