Document Image Rectification Bases on Self-Adaptive Multitask Fusion

TL;DR

This paper introduces SalmRec, a self-adaptive multi-task fusion network for document image rectification that effectively leverages task interactions to improve dewarping accuracy across multiple benchmarks.

Contribution

The paper presents a novel self-adaptive multi-task fusion approach with an inter-task feature aggregation module and gating mechanism to enhance geometric distortion perception.

Findings

Significant improvement on DIR300, DocUNet, and DocReal benchmarks.

Effective feature aggregation reduces negative task interference.

Ablation studies confirm the benefits of the proposed modules.

Abstract

Deformed document image rectification is essential for real-world document understanding tasks, such as layout analysis and text recognition. However, current multi-task methods -- such as background removal, 3D coordinate prediction, and text line segmentation -- often overlook the complementary features between tasks and their interactions. To address this gap, we propose a self-adaptive learnable multi-task fusion rectification network named SalmRec. This network incorporates an inter-task feature aggregation module that adaptively improves the perception of geometric distortions, enhances feature complementarity, and reduces negative interference. We also introduce a gating mechanism to balance features both within global tasks and between local tasks effectively. Experimental results on two English benchmarks (DIR300 and DocUNet) and one Chinese benchmark (DocReal) demonstrate that our method significantly improves rectification performance. Ablation studies further highlight the positive impact of different tasks on dewarping and the effectiveness of our proposed module.