DEVICE: Depth and Visual Concepts Aware Transformer for OCR-based Image Captioning

TL;DR

This paper introduces DEVICE, a transformer model that incorporates depth and semantic visual concepts to improve OCR-based image captioning, resulting in more accurate and comprehensive scene descriptions.

Contribution

The paper proposes a novel transformer architecture that integrates depth information and semantic visual concepts for enhanced scene understanding in OCR-based captioning.

Findings

DEVICE outperforms state-of-the-art models on TextCaps dataset

Incorporating depth improves scene text relational reasoning

Semantic-guided alignment enhances visual concept utilization

Abstract

OCR-based image captioning is an important but under-explored task, aiming to generate descriptions containing visual objects and scene text. Recent studies have made encouraging progress, but they are still suffering from a lack of overall understanding of scenes and generating inaccurate captions. One possible reason is that current studies mainly focus on constructing the plane-level geometric relationship of scene text without depth information. This leads to insufficient scene text relational reasoning so that models may describe scene text inaccurately. The other possible reason is that existing methods fail to generate fine-grained descriptions of some visual objects. In addition, they may ignore essential visual objects, leading to the scene text belonging to these ignored objects not being utilized. To address the above issues, we propose a Depth and Visual Concepts Aware Transformer (DEVICE) for OCR-based image captinong. Concretely, to construct three-dimensional geometric relations, we introduce depth information and propose a depth-enhanced feature updating module to ameliorate OCR token features. To generate more precise and comprehensive captions, we introduce semantic features of detected visual concepts as auxiliary information, and propose a semantic-guided alignment module to improve the model's ability to utilize visual concepts. Our DEVICE is capable of comprehending scenes more comprehensively and boosting the accuracy of described visual entities. Sufficient experiments demonstrate the effectiveness of our proposed DEVICE, which outperforms state-of-the-art models on the TextCaps test set.