OCR-Agent: Agentic OCR with Capability and Memory Reflection

TL;DR

This paper introduces OCR-Agent, an iterative self-correction framework for Vision-Language Models that enhances their reasoning and accuracy through Capability and Memory Reflection, outperforming existing models on OCR benchmarks.

Contribution

The paper presents a novel self-correction framework with Capability and Memory Reflection, enabling models to diagnose errors, review past attempts, and improve answers without extra training.

Findings

Outperforms state-of-the-art models on OCRBench v2

Achieves top results in Visual Understanding and Reasoning

Enhances reasoning robustness through structured self-reflection

Abstract

Large Vision-Language Models (VLMs) have demonstrated significant potential on complex visual understanding tasks through iterative optimization methods.However, these models generally lack effective self-correction mechanisms, making it difficult for them to independently rectify cognitive biases. Consequently, during multi-turn revisions, they often fall into repetitive and ineffective attempts, failing to achieve stable improvements in answer quality.To address this issue, we propose a novel iterative self-correction framework that endows models with two key capabilities: Capability Reflection and Memory Reflection. This framework guides the model to first diagnose errors and generate a correction plan via Capability Reflection, then leverage Memory Reflection to review past attempts to avoid repetition and explore new solutions, and finally, optimize the answer through rigorous re-reasoning. Experiments on the challenging OCRBench v2 benchmark show that OCR-Agent outperforms the current open-source SOTA model InternVL3-8B by +2.0 on English and +1.2 on Chinese subsets, while achieving state-of-the-art results in Visual Understanding (79.9) and Reasoning (66.5) - surpassing even larger fine-tuned models. Our method demonstrates that structured, self-aware reflection can significantly enhance VLMs' reasoning robustness without additional training. Code: https://github.com/AIGeeksGroup/OCR-Agent.