OCR-VQGAN: Taming Text-within-Image Generation

TL;DR

OCR-VQGAN is a novel image generation model that incorporates OCR features to produce high-fidelity figures with readable text, addressing a previously underexplored area in diagram and figure synthesis.

Contribution

The paper introduces OCR-VQGAN, a new model leveraging OCR pre-trained features and a novel dataset for improved figure and diagram generation with readable text.

Findings

Effective preservation of text in generated figures.

Improved figure reconstruction quality.

Impact of perceptual loss weighting on results.

Abstract

Synthetic image generation has recently experienced significant improvements in domains such as natural image or art generation. However, the problem of figure and diagram generation remains unexplored. A challenging aspect of generating figures and diagrams is effectively rendering readable texts within the images. To alleviate this problem, we present OCR-VQGAN, an image encoder, and decoder that leverages OCR pre-trained features to optimize a text perceptual loss, encouraging the architecture to preserve high-fidelity text and diagram structure. To explore our approach, we introduce the Paper2Fig100k dataset, with over 100k images of figures and texts from research papers. The figures show architecture diagrams and methodologies of articles available at arXiv.org from fields like artificial intelligence and computer vision. Figures usually include text and discrete objects, e.g., boxes in a diagram, with lines and arrows that connect them. We demonstrate the effectiveness of OCR-VQGAN by conducting several experiments on the task of figure reconstruction. Additionally, we explore the qualitative and quantitative impact of weighting different perceptual metrics in the overall loss function. We release code, models, and dataset at https://github.com/joanrod/ocr-vqgan.