NutritionVerse-Direct: Exploring Deep Neural Networks for Multitask Nutrition Prediction from Food Images

TL;DR

NutritionVerse-Direct is a neural network model that accurately predicts multiple nutritional components from food images, improving dietary assessment methods for aging individuals by automating and enhancing precision.

Contribution

The paper introduces NutritionVerse-Direct, a vision transformer-based neural network architecture that directly predicts comprehensive nutritional information from food images, outperforming previous models.

Findings

Achieved a 25.5% reduction in mean average error over previous models.

Effectively predicts calories, mass, protein, fat, and carbohydrates from food images.

Demonstrates potential for improved dietary intake estimation accuracy.

Abstract

Many aging individuals encounter challenges in effectively tracking their dietary intake, exacerbating their susceptibility to nutrition-related health complications. Self-reporting methods are often inaccurate and suffer from substantial bias; however, leveraging intelligent prediction methods can automate and enhance precision in this process. Recent work has explored using computer vision prediction systems to predict nutritional information from food images. Still, these methods are often tailored to specific situations, require other inputs in addition to a food image, or do not provide comprehensive nutritional information. This paper aims to enhance the efficacy of dietary intake estimation by leveraging various neural network architectures to directly predict a meal's nutritional content from its image. Through comprehensive experimentation and evaluation, we present NutritionVerse-Direct, a model utilizing a vision transformer base architecture with three fully connected layers that lead to five regression heads predicting calories (kcal), mass (g), protein (g), fat (g), and carbohydrates (g) present in a meal. NutritionVerse-Direct yields a combined mean average error score on the NutritionVerse-Real dataset of 412.6, an improvement of 25.5% over the Inception-ResNet model, demonstrating its potential for improving dietary intake estimation accuracy.