Simultaneous Food Localization and Recognition

TL;DR

This paper introduces the first method for simultaneous food localization and recognition, enabling automatic nutrition diaries by detecting and identifying food items in images with high precision.

Contribution

It presents a novel two-step approach combining food activation maps and recognition, applicable to both conventional and egocentric images.

Findings

High precision in food localization and recognition

Effective in both conventional and egocentric images

Achieves reasonable recall with few bounding boxes

Abstract

The development of automatic nutrition diaries, which would allow to keep track objectively of everything we eat, could enable a whole new world of possibilities for people concerned about their nutrition patterns. With this purpose, in this paper we propose the first method for simultaneous food localization and recognition. Our method is based on two main steps, which consist in, first, produce a food activation map on the input image (i.e. heat map of probabilities) for generating bounding boxes proposals and, second, recognize each of the food types or food-related objects present in each bounding box. We demonstrate that our proposal, compared to the most similar problem nowadays - object localization, is able to obtain high precision and reasonable recall levels with only a few bounding boxes. Furthermore, we show that it is applicable to both conventional and egocentric images.