Multi-Modal Reasoning Graph for Scene-Text Based Fine-Grained Image Classification and Retrieval

TL;DR

This paper introduces a multi-modal reasoning graph that combines visual and textual cues using graph convolutional networks to improve fine-grained image classification and retrieval.

Contribution

It proposes a novel multi-modal reasoning framework that integrates textual and visual features via GCNs for enhanced scene-text based image analysis.

Findings

Outperforms previous state-of-the-art in fine-grained classification.

Achieves superior results in image retrieval tasks.

Effectively leverages scene text for improved visual understanding.

Abstract

Scene text instances found in natural images carry explicit semantic information that can provide important cues to solve a wide array of computer vision problems. In this paper, we focus on leveraging multi-modal content in the form of visual and textual cues to tackle the task of fine-grained image classification and retrieval. First, we obtain the text instances from images by employing a text reading system. Then, we combine textual features with salient image regions to exploit the complementary information carried by the two sources. Specifically, we employ a Graph Convolutional Network to perform multi-modal reasoning and obtain relationship-enhanced features by learning a common semantic space between salient objects and text found in an image. By obtaining an enhanced set of visual and textual features, the proposed model greatly outperforms the previous state-of-the-art in two different tasks, fine-grained classification and image retrieval in the Con-Text and Drink Bottle datasets.