Preference Aligned Visuomotor Diffusion Policies for Deformable Object Manipulation

TL;DR

This paper introduces RKO, a novel method for adapting pretrained visuomotor diffusion policies to reflect user preferences in deformable object manipulation, demonstrating improved performance and efficiency.

Contribution

The paper presents RKO, a new preference-alignment framework combining RPO and KTO, tailored for deformable object manipulation with limited demonstrations.

Findings

Preference-aligned policies outperform vanilla diffusion policies.

RKO achieves higher sample efficiency in real-world cloth-folding tasks.

Structured preference learning enables personalized robot behavior.

Abstract

Humans naturally develop preferences for how manipulation tasks should be performed, which are often subtle, personal, and difficult to articulate. Although it is important for robots to account for these preferences to increase personalization and user satisfaction, they remain largely underexplored in robotic manipulation, particularly in the context of deformable objects like garments and fabrics. In this work, we study how to adapt pretrained visuomotor diffusion policies to reflect preferred behaviors using limited demonstrations. We introduce RKO, a novel preference-alignment method that combines the benefits of two recent frameworks: RPO and KTO. We evaluate RKO against common preference learning frameworks, including these two, as well as a baseline vanilla diffusion policy, on real-world cloth-folding tasks spanning multiple garments and preference settings. We show that preference-aligned policies (particularly RKO) achieve superior performance and sample efficiency compared to standard diffusion policy fine-tuning. These results highlight the importance and feasibility of structured preference learning for scaling personalized robot behavior in complex deformable object manipulation tasks.