VACoT: Rethinking Visual Data Augmentation with VLMs

TL;DR

VACoT introduces a dynamic visual augmentation framework that enhances robustness of vision models during inference through structured, post-hoc image transformations, significantly improving performance on challenging perception tasks.

Contribution

The paper proposes VACoT, a novel inference-time augmentation method using reinforcement learning to improve vision model robustness with minimal training overhead.

Findings

VACoT improves robustness on out-of-distribution inputs.

VACoT enhances OCR performance in adversarial scenarios.

Extensive experiments validate VACoT's effectiveness across 13 benchmarks.

Abstract

While visual data augmentation remains a cornerstone for training robust vision models, it has received limited attention in visual language models (VLMs), which predominantly rely on large-scale real data acquisition or synthetic diversity. Consequently, they may struggle with basic perception tasks that conventional models handle reliably. Given the substantial cost of pre-training and fine-tuning VLMs, continue training on augmented data yields limited and diminishing returns. In this paper, we present Visual Augmentation Chain-of-Thought (VACoT), a framework that dynamically invokes image augmentations during model inference. By incorporating post-hoc transformations such as denoising, VACoT substantially improves robustness on challenging and out-of-distribution inputs, especially in OCR-related adversarial scenarios. Distinct from prior approaches limited to local cropping, VACoT integrates a structured collection of general visual augmentations, broadening the query image views while reducing training complexity and computational overhead with efficient agentic reinforcement learning. We propose a conditional reward scheme that encourages necessary augmentation while penalizing verbose responses, ensuring concise and effective reasoning in perception tasks. We demonstrate the superiority of VACoT with extensive experiments on 13 perception benchmarks and further introduce AdvOCR to highlight the generalization benefits of post-hoc visual augmentations in adversarial scenarios.