CoMemo: LVLMs Need Image Context with Image Memory

TL;DR

CoMemo introduces a dual-path architecture with a novel positional encoding to improve multimodal processing in LVLMs, effectively addressing information neglect and spatial awareness issues, leading to better performance across various benchmarks.

Contribution

The paper presents CoMemo, a new architecture combining image memory and a novel positional encoding to enhance visual context understanding in LVLMs.

Findings

Outperforms conventional LVLMs on seven benchmarks

Effectively preserves 2D spatial relationships in high-resolution images

Reduces neglect of middle visual content in extended contexts

Abstract

Recent advancements in Large Vision-Language Models built upon Large Language Models have established aligning visual features with LLM representations as the dominant paradigm. However, inherited LLM architectural designs introduce suboptimal characteristics for multimodal processing. First, LVLMs exhibit a bimodal distribution in attention allocation, leading to the progressive neglect of middle visual content as context expands. Second, conventional positional encoding schemes fail to preserve vital 2D structural relationships when processing dynamic high-resolution images. To address these limitations, we propose CoMemo - a dual-path architecture that combines a Context image path with an image Memory path for visual processing, effectively alleviating visual information neglect. Additionally, we introduce RoPE-DHR, a novel positional encoding mechanism that employs thumbnail-based positional aggregation to maintain 2D spatial awareness while mitigating remote decay in extended sequences. Evaluations across seven benchmarks,including long-context comprehension, multi-image reasoning, and visual question answering, demonstrate CoMemo's superior performance compared to conventional LVLM architectures. Project page is available at https://lalbj.github.io/projects/CoMemo/.