Reinforcing Multimodal Reasoning Against Visual Degradation

TL;DR

This paper introduces ROMA, a reinforcement learning framework that enhances multimodal reasoning models' robustness against visual degradations like blur and low resolution, without sacrificing performance on clean data.

Contribution

ROMA modifies RL fine-tuning dynamics with novel strategies to improve robustness against visual corruptions while maintaining accuracy on clean inputs.

Findings

ROMA improves robustness by +2.4% on seen corruptions.

ROMA improves robustness by +2.3% on unseen corruptions.

ROMA matches clean accuracy on multimodal reasoning benchmarks.

Abstract

Reinforcement Learning has significantly advanced the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet the resulting policies remain brittle against real-world visual degradations such as blur, compression artifacts, and low-resolution scans. Prior robustness techniques from vision and deep RL rely on static data augmentation or value-based regularization, neither of which transfers cleanly to critic-free RL fine-tuning of autoregressive MLLMs. Reinforcing reasoning against such corruptions is non-trivial: naively injecting degraded views during rollout induces reward poisoning, where perceptual occlusions trigger hallucinated trajectories and destabilize optimization. We propose ROMA, an RL fine-tuning framework that modifies the optimization dynamics to reinforce reasoning against visual degradation while preserving clean-input performance. A dual-forward-pass strategy uses teacher forcing to evaluate corrupted views against clean-image trajectories, avoiding new rollouts on degraded inputs. For distributional consistency, we apply a token-level surrogate KL penalty against the worst-case augmentation; to prevent policy collapse under regularization, an auxiliary policy gradient loss anchored to clean-image advantages preserves a reliable reward signal; and to avoid systematically incorrect invariance, correctness-conditioned regularization restricts enforcement to successful trajectories. On Qwen3-VL 4B/8B across seven multimodal reasoning benchmarks, our method improves robustness by +2.4% on seen and +2.3% on unseen corruptions over GRPO while matching clean accuracy.