Improving Dietary Assessment Via Integrated Hierarchy Food Classification

TL;DR

This paper presents a hierarchical multi-task network that integrates visual and nutritional data to improve the quality of food classification predictions, reducing errors in energy and nutrient estimation.

Contribution

The authors introduce a novel multi-domain, hierarchical classification framework that enhances prediction quality by combining visual and nutritional information.

Findings

Achieved comparable visual classification accuracy to existing methods.

Reduced energy and nutrient estimation errors in misclassified foods.

Validated on a modified VIPER-FoodNet dataset.

Abstract

Image-based dietary assessment refers to the process of determining what someone eats and how much energy and nutrients are consumed from visual data. Food classification is the first and most crucial step. Existing methods focus on improving accuracy measured by the rate of correct classification based on visual information alone, which is very challenging due to the high complexity and inter-class similarity of foods. Further, accuracy in food classification is conceptual as description of a food can always be improved. In this work, we introduce a new food classification framework to improve the quality of predictions by integrating the information from multiple domains while maintaining the classification accuracy. We apply a multi-task network based on a hierarchical structure that uses both visual and nutrition domain specific information to cluster similar foods. Our method is validated on the modified VIPER-FoodNet (VFN) food image dataset by including associated energy and nutrient information. We achieve comparable classification accuracy with existing methods that use visual information only, but with less error in terms of energy and nutrient values for the wrong predictions.