# Integration of Text-maps in Convolutional Neural Networks for Region Detection among Different Textual Categories

## TL;DR

This paper introduces a novel method that combines visual appearance and semantic text information in CNNs using text-maps, significantly improving region detection of textual categories in images, especially for supermarket product coding.

## Contribution

The work presents a new visual text representation called text-map, integrated into CNNs, enhancing textual region detection over appearance-only methods.

## Key findings

- Outperforms state-of-the-art appearance-based algorithms
- Improves precision by 42 points and recall by 33 points
- Effective in supermarket product item coding

## Abstract

In this work, we propose a new technique that combines appearance and text in a Convolutional Neural Network (CNN), with the aim of detecting regions of different textual categories. We define a novel visual representation of the semantic meaning of text that allows a seamless integration in a standard CNN architecture. This representation, referred to as text-map, is integrated with the actual image to provide a much richer input to the network. Text-maps are colored with different intensities depending on the relevance of the words recognized over the image. Concretely, these words are previously extracted using Optical Character Recognition (OCR) and they are colored according to the probability of belonging to a textual category of interest. In this sense, this solution is especially relevant in the context of item coding for supermarket products, where different types of textual categories must be identified, such as ingredients or nutritional facts. We evaluated our solution in the proprietary item coding dataset of Nielsen Brandbank, which contains more than 10,000 images for train and 2,000 images for test. The reported results demonstrate that our approach focused on visual and textual data outperforms state-of-the-art algorithms only based on appearance, such as standard Faster R-CNN. These enhancements are reflected in precision and recall, which are improved in 42 and 33 points respectively.