Gaze-Regularized Vision-Language-Action Models for Robotic Manipulation

TL;DR

This paper introduces a gaze-regularized training method for vision-language-action models in robotics, aligning model attention with human gaze to improve manipulation performance and interpretability without extra inference costs.

Contribution

The authors propose a novel gaze-regularization framework that enhances VLA models by aligning their attention with human gaze patterns, improving efficiency, robustness, and interpretability.

Findings

Achieved 4-12% performance improvements on manipulation benchmarks.

Models trained with gaze regularization require fewer training steps.

Attention patterns learned are interpretable and mirror human strategies.

Abstract

Despite advances in Vision-Language-Action (VLA) models, robotic manipulation struggles with fine-grained tasks because current models lack mechanisms for active visual attention allocation. Human gaze naturally encodes intent, planning, and execution patterns -- offering a powerful supervisory signal for guiding robot perception. We introduce a gaze-regularized training framework that aligns VLA models' internal attention with human visual patterns without architectural modifications or inference-time overhead. Our method transforms temporally aggregated gaze heatmaps into patch-level distributions and regularizes the transformer's attention through KL divergence, creating an inductive bias toward task-relevant features while preserving deployment efficiency. When integrated into existing VLA architectures, our approach yields 4-12% improvements across manipulation benchmarks. The gaze-regularized models reach equivalent performance with fewer training steps and maintain robustness under lighting variations and sensor noise. Beyond performance metrics, the learned attention patterns produce interpretable visualizations that mirror human strategies, enhancing trust in robotic systems. Moreover, our framework requires no eye-tracking equipment and applies directly to existing datasets. These results demonstrate that human perceptual priors can significantly accelerate robot learning while improving both task performance and system interpretability.