DocEdit-v2: Document Structure Editing Via Multimodal LLM Grounding

TL;DR

DocEdit-v2 introduces an end-to-end multimodal framework leveraging large models to accurately localize, interpret, and execute document editing requests, significantly improving performance over previous methods.

Contribution

It presents a novel multimodal document editing system with three key components, enabling precise localization, command reformulation, and execution using large multimodal models.

Findings

Outperforms baselines in edit command generation (2-33%)

Improves RoI bounding box detection (12-31%)

Enhances overall document editing accuracy (1-12%)

Abstract

Document structure editing involves manipulating localized textual, visual, and layout components in document images based on the user's requests. Past works have shown that multimodal grounding of user requests in the document image and identifying the accurate structural components and their associated attributes remain key challenges for this task. To address these, we introduce the DocEdit-v2, a novel framework that performs end-to-end document editing by leveraging Large Multimodal Models (LMMs). It consists of three novel components: (1) Doc2Command, which simultaneously localizes edit regions of interest (RoI) and disambiguates user edit requests into edit commands; (2) LLM-based Command Reformulation prompting to tailor edit commands originally intended for specialized software into edit instructions suitable for generalist LMMs. (3) Moreover, DocEdit-v2 processes these outputs via Large Multimodal Models like GPT-4V and Gemini, to parse the document layout, execute edits on grounded Region of Interest (RoI), and generate the edited document image. Extensive experiments on the DocEdit dataset show that DocEdit-v2 significantly outperforms strong baselines on edit command generation (2-33%), RoI bounding box detection (12-31%), and overall document editing (1-12\%) tasks.