Fine-Grained Product Class Recognition for Assisted Shopping

TL;DR

This paper introduces a comprehensive system for visually recognizing fine-grained product classes in grocery stores to assist shoppers, especially the visually impaired, by combining text recognition, visual patch analysis, and active learning.

Contribution

It presents a novel multi-component system that integrates text recognition, discriminative visual patches, and active learning for scalable, robust product class recognition in real-world shopping scenarios.

Findings

Robust recognition across different store environments.

Scalable to large product databases with minimal re-training.

Improved accuracy through active learning.

Abstract

Assistive solutions for a better shopping experience can improve the quality of life of people, in particular also of visually impaired shoppers. We present a system that visually recognizes the fine-grained product classes of items on a shopping list, in shelves images taken with a smartphone in a grocery store. Our system consists of three components: (a) We automatically recognize useful text on product packaging, e.g., product name and brand, and build a mapping of words to product classes based on the large-scale GroceryProducts dataset. When the user populates the shopping list, we automatically infer the product class of each entered word. (b) We perform fine-grained product class recognition when the user is facing a shelf. We discover discriminative patches on product packaging to differentiate between visually similar product classes and to increase the robustness against continuous changes in product design. (c) We continuously improve the recognition accuracy through active learning. Our experiments show the robustness of the proposed method against cross-domain challenges, and the scalability to an increasing number of products with minimal re-training.