Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy

TL;DR

This paper introduces BayesVLA, a Bayesian factorization approach that decomposes vision-language-action policies to improve out-of-distribution generalization and instruction following in VLA models.

Contribution

It proposes a novel Bayesian factorization method that addresses modality imbalance and preserves generalization in VLA models, reducing reliance on external data and tuning.

Findings

Superior generalization to unseen instructions, objects, and environments.

Effectively mitigates shortcut learning and language forgetting.

Validated through information-theoretic analysis and extensive experiments.

Abstract

The pursuit of out-of-distribution generalization in Vision-Language-Action (VLA) models is often hindered by catastrophic forgetting of the Vision-Language Model (VLM) backbone during fine-tuning. While co-training with external reasoning data helps, it requires experienced tuning and data-related overhead. Beyond such external dependencies, we identify an intrinsic cause within VLA datasets: modality imbalance, where language diversity is much lower than visual and action diversity. This imbalance biases the model toward visual shortcuts and language forgetting. To address this, we introduce BayesVLA, a Bayesian factorization that decomposes the policy into a visual-action prior, supporting seeing-to-act, and a language-conditioned likelihood, enabling prompt-to-specify. This inherently preserves generalization and promotes instruction following. We further incorporate pre- and post-contact phases to better leverage pre-trained foundation models. Information-theoretic analysis formally validates our effectiveness in mitigating shortcut learning. Extensive experiments show superior generalization to unseen instructions, objects, and environments compared to existing methods. Project page is available at: https://xukechun.github.io/papers/BayesVLA.