V-Nutri: Dish-Level Nutrition Estimation from Egocentric Cooking Videos

TL;DR

V-Nutri leverages egocentric cooking videos and process cues to improve dish-level nutrition estimation, addressing limitations of static image-based methods by incorporating cooking process information.

Contribution

The paper introduces V-Nutri, a novel staged framework combining visual backbones and process keyframes for enhanced nutrition estimation from egocentric videos.

Findings

Process cues improve nutrition estimation accuracy.

Backbone capacity and event detection quality influence benefits.

Annotated HD-EPIC dataset and benchmark established.

Abstract

Nutrition estimation of meals from visual data is an important problem for dietary monitoring and computational health, but existing approaches largely rely on single images of the finally completed dish. This setting is fundamentally limited because many nutritionally relevant ingredients and transformations, such as oils, sauces, and mixed components, become visually ambiguous after cooking, making accurate calorie and macronutrient estimation difficult. In this paper, we investigate whether the cooking process information from egocentric cooking videos can contribute to dish-level nutrition estimation. First, we further manually annotated the HD-EPIC dataset and established the first benchmark for video-based nutrition estimation. Most importantly, we propose V-Nutri, a staged framework that combines Nutrition5K-pretrained visual backbones with a lightweight fusion module that aggregates features from the final dish frame and cooking process keyframes extracted from the egocentric videos. V-Nutri also includes a cooking keyframes selection module, a VideoMamba-based event-detection model that targets ingredient-addition moments. Experiments on the HD-EPIC dataset show that process cues can provide complementary nutritional evidence, improving nutrition estimation under controlled conditions. Our results further indicate that the benefit of process keyframes depends strongly on backbone representation capacity and event detection quality. Our code and annotated dataset is available at https://github.com/K624-YCK/V-Nutri.