Food Ingredients Recognition through Multi-label Learning

TL;DR

This paper presents a multi-label CNN approach for recognizing food ingredients from images, enabling automatic food diary construction and ingredient prediction even for unseen recipes, supported by new datasets.

Contribution

Introduces a novel multi-label CNN method for food ingredient recognition and provides new datasets to facilitate research in this area.

Findings

Model accurately predicts ingredients from images.

Training with diverse recipes improves generalization.

Neuron visualization shows specialization for ingredients.

Abstract

Automatically constructing a food diary that tracks the ingredients consumed can help people follow a healthy diet. We tackle the problem of food ingredients recognition as a multi-label learning problem. We propose a method for adapting a highly performing state of the art CNN in order to act as a multi-label predictor for learning recipes in terms of their list of ingredients. We prove that our model is able to, given a picture, predict its list of ingredients, even if the recipe corresponding to the picture has never been seen by the model. We make public two new datasets suitable for this purpose. Furthermore, we prove that a model trained with a high variability of recipes and ingredients is able to generalize better on new data, and visualize how it specializes each of its neurons to different ingredients.