Latent Chain-of-Thought for Visual Reasoning

TL;DR

This paper introduces a novel latent chain-of-thought approach for visual reasoning in large vision-language models, using scalable Bayesian inference and diversity-seeking reinforcement learning to improve reasoning accuracy and interpretability.

Contribution

It reformulates reasoning as posterior inference, proposing a scalable training algorithm with diversity-seeking RL and Bayesian inference to enhance generalization and interpretability of LVLMs.

Findings

Improves performance on seven reasoning benchmarks

Enhances model generalization across unseen tasks

Increases interpretability of reasoning processes

Abstract

Chain-of-thought (CoT) reasoning is critical for improving the interpretability and reliability of Large Vision-Language Models (LVLMs). However, existing training algorithms such as SFT, PPO, and GRPO may not generalize well across unseen reasoning tasks and heavily rely on a biased reward model. To address this challenge, we reformulate reasoning in LVLMs as posterior inference and propose a scalable training algorithm based on amortized variational inference. By leveraging diversity-seeking reinforcement learning algorithms, we introduce a novel sparse reward function for token-level learning signals that encourage diverse, high-likelihood latent CoT, overcoming deterministic sampling limitations and avoiding reward hacking. Additionally, we implement a Bayesian inference-scaling strategy that replaces costly Best-of-N and Beam Search with a marginal likelihood to efficiently rank optimal rationales and answers. We empirically demonstrate that the proposed method enhances the state-of-the-art LVLMs on seven reasoning benchmarks, in terms of effectiveness, generalization, and interpretability.