KOROL: Learning Visualizable Object Feature with Koopman Operator Rollout for Manipulation

TL;DR

KOROL introduces a vision-based manipulation framework using Koopman operators to learn interpretable object features, enabling accurate system state propagation and improved task success in robotic manipulation.

Contribution

The paper presents a novel method that uses Koopman operators with learned visual features for manipulation, eliminating the need for ground-truth object states in real-world applications.

Findings

Outperforms existing model-based imitation learning and diffusion policies.

Maintains high task success rates with learned visual features.

Effective in both simulated and real-world robot tasks.

Abstract

Learning dexterous manipulation skills presents significant challenges due to complex nonlinear dynamics that underlie the interactions between objects and multi-fingered hands. Koopman operators have emerged as a robust method for modeling such nonlinear dynamics within a linear framework. However, current methods rely on runtime access to ground-truth (GT) object states, making them unsuitable for vision-based practical applications. Unlike image-to-action policies that implicitly learn visual features for control, we use a dynamics model, specifically the Koopman operator, to learn visually interpretable object features critical for robotic manipulation within a scene. We construct a Koopman operator using object features predicted by a feature extractor and utilize it to auto-regressively advance system states. We train the feature extractor to embed scene information into object features, thereby enabling the accurate propagation of robot trajectories. We evaluate our approach on simulated and real-world robot tasks, with results showing that it outperformed the model-based imitation learning NDP by 1.08$\times$ and the image-to-action Diffusion Policy by 1.16$\times$. The results suggest that our method maintains task success rates with learned features and extends applicability to real-world manipulation without GT object states. Project video and code are available at: \url{https://github.com/hychen-naza/KOROL}.