VDE Bench: Evaluating The Capability of Image Editing Models to Modify Visual Documents

TL;DR

VDE Bench is a new benchmark dataset and evaluation framework designed to assess image editing models' ability to modify dense, bilingual Chinese-English visual documents while preserving text style and background.

Contribution

It introduces a high-quality dataset of 942 instruction-based samples and a novel OCR-based evaluation framework for complex visual document editing tasks.

Findings

Human verification aligns well with automated metrics.

Existing models struggle with dense, bilingual documents.

VDE Bench is the first systematic benchmark for this task.

Abstract

In recent years, image editing models have made significant progress, enabling users to manipulate visual content in a flexible and interactive manner through natural language instructions. However, an important yet underexplored research direction remains dense visual document image editing, which involves modifying textual content within images while faithfully preserving the original text style and background context. Existing methods primarily focus on English scenarios and images with relatively sparse text, and thus cannot adequately address dense, structurally complex documents or non-Latin scripts such as Chinese. To bridge this gap, we propose VDE Bench (Visual Doc Edit Bench), a rigorously human annotated and evaluated benchmark specifically designed to assess the performance of image editing models on bilingual Chinese-English and complex visual document editing tasks. The benchmark comprises a high quality dataset of 942 instruction based image editing samples, whose seed images encompass dense Chinese and English text documents including academic papers, posters, presentation slides, examination materials, and newspapers. Furthermore, we introduce a novel evaluation framework that systematically quantifies editing performance at the OCR parsing level, thereby enabling fine grained assessment of text modification accuracy. Based on this benchmark, we conduct a comprehensive evaluation of representative image editing models. Human verification demonstrates a high degree of consistency between human judgments and automated evaluation metrics. VDE Bench constitutes the first systematic benchmark for evaluating the performance of image editing models on bilingual dense text visual documents.