Whispering to a Blackbox: Bootstrapping Frozen OCR with Visual Prompts

TL;DR

This paper presents Whisperer, a visual prompting framework that enhances frozen OCR models by learning diffusion-based preprocessors, achieving significant error rate reductions through a sample-efficient, behavioral cloning approach that amplifies random improvements.

Contribution

Introduces a novel diffusion-based visual prompting method that improves frozen OCR performance via a four-stage behavioral cloning curriculum, avoiding reinforcement learning pitfalls.

Findings

Achieves 8% absolute reduction in CER on synthetic degraded text images.

Surpasses hand-engineered preprocessing baselines like CLAHE.

Uses a sample-efficient four-stage training process over 60 GPU-hours.

Abstract

In the landscape of modern machine learning, frozen pre-trained models provide stability and efficiency but often underperform on specific tasks due to mismatched data distributions. This paper introduces the Whisperer, a novel visual prompting framework that learns diffusion-based preprocessors to adapt inputs in pixel space, effectively "whispering" enhancements to frozen downstream models like EasyOCR. By framing the process as behavioral cloning of stochastically discovered improvement policies, our method achieves an 8% absolute (10.6% relative) reduction in Character Error Rate (CER) on a challenging dataset of 300k degraded synthetic text images, surpassing hand-engineered baselines such as CLAHE. The key innovation is a four-stage training curriculum that uses behavioral cloning to amplify "lucky" improvements discovered through the stochastic exploration of a partially trained diffusion model. This approach is highly sample-efficient and avoids the pitfalls of traditional reinforcement learning. Crucially, we frame this not as naive reinforcement learning, but as behavioral cloning of an exploration policy: we stochastically sample intermediate diffusion outputs, select those that improve CER by chance, and then train the model to reproduce them. This bootstrapping curriculum (4 stages over 60 GPU-hours) amplifies random successes into a systematic strategy. In summary, by whispering to the frozen OCR through its inputs, we improve an imperfect classifier without touching its weights.