A deep learning pipeline for product recognition on store shelves

TL;DR

This paper presents a deep learning pipeline for recognizing thousands of grocery products on store shelves, combining object detection and similarity search with learned global descriptors for accurate and rapid identification.

Contribution

It introduces a novel approach that integrates deep learning-based object detection with a learned global descriptor for product recognition, handling large product catalogs and variable shelf images.

Findings

High recognition accuracy on shelf images

Effective handling of new and changing products

Rapid inference suitable for real-time applications

Abstract

Recognition of grocery products in store shelves poses peculiar challenges. Firstly, the task mandates the recognition of an extremely high number of different items, in the order of several thousands for medium-small shops, with many of them featuring small inter and intra class variability. Then, available product databases usually include just one or a few studio-quality images per product (referred to herein as reference images), whilst at test time recognition is performed on pictures displaying a portion of a shelf containing several products and taken in the store by cheap cameras (referred to as query images). Moreover, as the items on sale in a store as well as their appearance change frequently over time, a practical recognition system should handle seamlessly new products/packages. Inspired by recent advances in object detection and image retrieval, we propose to leverage on state of the art object detectors based on deep learning to obtain an initial productagnostic item detection. Then, we pursue product recognition through a similarity search between global descriptors computed on reference and cropped query images. To maximize performance, we learn an ad-hoc global descriptor by a CNN trained on reference images based on an image embedding loss. Our system is computationally expensive at training time but can perform recognition rapidly and accurately at test time.