Multi-Task Image-Based Dietary Assessment for Food Recognition and Portion Size Estimation

TL;DR

This paper presents an end-to-end multi-task deep learning framework for simultaneous food recognition and portion size estimation from images, utilizing a new dataset and advanced feature adaptation techniques.

Contribution

It introduces a novel multi-task learning approach with soft parameter sharing and cross-domain feature adaptation for improved dietary assessment tasks.

Findings

Outperforms baseline methods in food classification accuracy.

Achieves lower mean absolute error in portion size estimation.

Demonstrates the effectiveness of multi-task learning in dietary assessment.

Abstract

Deep learning based methods have achieved impressive results in many applications for image-based diet assessment such as food classification and food portion size estimation. However, existing methods only focus on one task at a time, making it difficult to apply in real life when multiple tasks need to be processed together. In this work, we propose an end-to-end multi-task framework that can achieve both food classification and food portion size estimation. We introduce a food image dataset collected from a nutrition study where the groundtruth food portion is provided by registered dietitians. The multi-task learning uses L2-norm based soft parameter sharing to train the classification and regression tasks simultaneously. We also propose the use of cross-domain feature adaptation together with normalization to further improve the performance of food portion size estimation. Our results outperforms the baseline methods for both classification accuracy and mean absolute error for portion estimation, which shows great potential for advancing the field of image-based dietary assessment.