What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models

TL;DR

This paper introduces PAIR-VLA, a reinforcement learning fine-tuning framework that enhances visual robustness of VLA models in robotic manipulation by using paired visual variants to guide policy responses.

Contribution

The authors propose a novel RL fine-tuning method with auxiliary objectives that improve policy robustness to visual shifts by leveraging paired visual variants during training.

Findings

Consistently improves over standard PPO in diverse visual shift scenarios.

Achieves 16.62% and 9.10% average improvements on two VLA architectures.

Demonstrates transferability of invariance and sensitivity guidance across different visual shifts.

Abstract

Reinforcement learning (RL) fine-tuning has shown promise for Vision-Language-Action (VLA) models in robotic manipulation, but deployment-time visual shifts pose practical challenges. A key difficulty is that standard task rewards supervise task success, but offer limited guidance on whether a visual change is task-irrelevant or changes the behavior required for manipulation. We propose PAIR-VLA (Paired Action Invariance & Sensitivity for Visually Robust VLA), an RL fine-tuning framework to address this difficulty by adding two auxiliary objectives over paired visual variants during PPO optimization: an invariance term that reduces the discrepancy between action distributions for a task-preserving pair (e.g., different distractors), and a sensitivity objective that encourages separable action distributions for a task-altering pair (e.g., target object in a different pose). Together, these objectives turn visual variants from mere observation diversity into behavior-level guidance on policy responses during RL fine-tuning. We evaluate on ManiSkill3 across two representative VLA architectures, OpenVLA and $\pi_{0.5}$, under diverse out-of-distribution visual shifts including unseen distractors, texture changes, target object pose variation, viewpoint shifts, and lighting changes. Our method consistently improves over standard PPO, achieving average improvements of 16.62% on $\pi_{0.5}$ and 9.10% on OpenVLA. Notably, ablations further show generalization across visual shifts: invariance guidance learned from distractor and texture variants transfers to target-pose and lighting shifts, while adding sensitivity guidance on target-pose variants further improves robustness to nuisance shifts, highlighting the broader transferability of behavior-level RL guidance.