MemOCR: Layout-Aware Visual Memory for Efficient Long-Horizon Reasoning

TL;DR

MemOCR is a multimodal memory system that uses visual layout to adaptively compress and prioritize information, enabling efficient long-horizon reasoning within limited context windows.

Contribution

It introduces a structured visual memory that dynamically allocates information density based on layout, improving reasoning efficiency under tight context constraints.

Findings

Outperforms text-based baselines on long-context QA benchmarks.

Achieves more effective context utilization under extreme memory budgets.

Uses reinforcement learning for robust memory compression across varying budgets.

Abstract

Long-horizon agentic reasoning necessitates effectively compressing growing interaction histories into a limited context window. Most existing memory systems serialize history as text, where token-level cost is uniform and scales linearly with length, often spending scarce budget on low-value details. To this end, we introduce MemOCR, a multimodal memory agent that improves long-horizon reasoning under tight context budgets by allocating memory space with adaptive information density through visual layout. Concretely, MemOCR maintains a structured rich-text memory (e.g., headings, highlights) and renders it into an image that the agent consults for memory access, visually prioritizing crucial evidence while aggressively compressing auxiliary details. To ensure robustness across varying memory budgets, we train MemOCR with reinforcement learning under budget-aware objectives that expose the agent to diverse compression levels. Across long-context multi-hop and single-hop question-answering benchmarks, MemOCR outperforms strong text-based baselines and achieves more effective context utilization under extreme budgets.