Mechanistic Finetuning of Vision-Language-Action Models via Few-Shot Demonstrations

TL;DR

This paper introduces Robotic Steering, a mechanistic finetuning method for vision-language-action models that uses few-shot demonstrations to selectively adapt task-specific model components, improving robustness and interpretability in robotic applications.

Contribution

The paper proposes a novel mechanistic finetuning approach that leverages few-shot demonstrations to identify and adapt task-specific attention heads in VLAs for robotics.

Findings

Outperforms LoRA in robotic tasks

Achieves better robustness under task variation

Reduces computational cost and enhances interpretability

Abstract

Vision-Language Action (VLAs) models promise to extend the remarkable success of vision-language models (VLMs) to robotics. Yet, unlike VLMs in the vision-language domain, VLAs for robotics require finetuning to contend with varying physical factors like robot embodiment, environment characteristics, and spatial relationships of each task. Existing fine-tuning methods lack specificity, adapting the same set of parameters regardless of a task's visual, linguistic, and physical characteristics. Inspired by functional specificity in neuroscience, we hypothesize that it is more effective to finetune sparse model representations specific to a given task. In this work, we introduce Robotic Steering, a finetuning approach grounded in mechanistic interpretability that leverages few-shot demonstrations to identify and selectively finetune task-specific attention heads aligned with the physical, visual, and linguistic requirements of robotic tasks. Through comprehensive on-robot evaluations with a Franka Emika robot arm, we demonstrate that Robotic Steering outperforms LoRA while achieving superior robustness under task variation, reduced computational cost, and enhanced interpretability for adapting VLAs to diverse robotic tasks.