UPOCR: Towards Unified Pixel-Level OCR Interface

TL;DR

UPOCR introduces a unified vision Transformer-based model for pixel-level OCR tasks, enabling simultaneous high performance across diverse applications like text removal, segmentation, and tampered text detection.

Contribution

The paper presents UPOCR, a novel unified OCR model that uses image-to-image transformation and task prompts to handle multiple pixel-level OCR tasks with a single architecture.

Findings

Achieves state-of-the-art results on three OCR tasks

Demonstrates effective task-specific feature learning with prompts

Simplifies OCR model deployment with a unified approach

Abstract

In recent years, the optical character recognition (OCR) field has been proliferating with plentiful cutting-edge approaches for a wide spectrum of tasks. However, these approaches are task-specifically designed with divergent paradigms, architectures, and training strategies, which significantly increases the complexity of research and maintenance and hinders the fast deployment in applications. To this end, we propose UPOCR, a simple-yet-effective generalist model for Unified Pixel-level OCR interface. Specifically, the UPOCR unifies the paradigm of diverse OCR tasks as image-to-image transformation and the architecture as a vision Transformer (ViT)-based encoder-decoder. Learnable task prompts are introduced to push the general feature representations extracted by the encoder toward task-specific spaces, endowing the decoder with task awareness. Moreover, the model training is uniformly aimed at minimizing the discrepancy between the generated and ground-truth images regardless of the inhomogeneity among tasks. Experiments are conducted on three pixel-level OCR tasks including text removal, text segmentation, and tampered text detection. Without bells and whistles, the experimental results showcase that the proposed method can simultaneously achieve state-of-the-art performance on three tasks with a unified single model, which provides valuable strategies and insights for future research on generalist OCR models. Code will be publicly available.