An End-to-End Food Image Analysis System

TL;DR

This paper presents an end-to-end deep learning framework for food image analysis that simultaneously localizes, classifies, and estimates portion sizes of multiple food items in real-world images, improving accuracy and applicability.

Contribution

The paper introduces a novel integrated system combining localization, classification, and portion estimation in a single end-to-end model for food images.

Findings

Effective localization and classification of multiple food items.

Improved portion size estimation using a GAN-based food energy distribution map.

Validated on real-life food image dataset from a nutrition study.

Abstract

Modern deep learning techniques have enabled advances in image-based dietary assessment such as food recognition and food portion size estimation. Valuable information on the types of foods and the amount consumed are crucial for prevention of many chronic diseases. However, existing methods for automated image-based food analysis are neither end-to-end nor are capable of processing multiple tasks (e.g., recognition and portion estimation) together, making it difficult to apply to real life applications. In this paper, we propose an image-based food analysis framework that integrates food localization, classification and portion size estimation. Our proposed framework is end-to-end, i.e., the input can be an arbitrary food image containing multiple food items and our system can localize each single food item with its corresponding predicted food type and portion size. We also improve the single food portion estimation by consolidating localization results with a food energy distribution map obtained by conditional GAN to generate a four-channel RGB-Distribution image. Our end-to-end framework is evaluated on a real life food image dataset collected from a nutrition feeding study.