Should I take a walk? Estimating Energy Expenditure from Video Data

TL;DR

This paper introduces Vid2Burn, a benchmark dataset for estimating human caloric expenditure from video data, highlighting the challenges of generalizing across different activity types and proposing a new research direction.

Contribution

The paper presents Vid2Burn, a comprehensive benchmark dataset for energy expenditure estimation from videos, and evaluates state-of-the-art methods, emphasizing the need for models that generalize beyond activity categories.

Findings

State-of-the-art models struggle with unseen activity types

Energy expenditure estimation from video is a challenging task

Cross-category generalization remains a key challenge

Abstract

We explore the problem of automatically inferring the amount of kilocalories used by human during physical activity from his/her video observation. To study this underresearched task, we introduce Vid2Burn -- an omni-source benchmark for estimating caloric expenditure from video data featuring both, high- and low-intensity activities for which we derive energy expenditure annotations based on models established in medical literature. In practice, a training set would only cover a certain amount of activity types, and it is important to validate, if the model indeed captures the essence of energy expenditure, (e.g., how many and which muscles are involved and how intense they work) instead of memorizing fixed values of specific activity categories seen during training. Ideally, the models should look beyond such category-specific biases and regress the caloric cost in videos depicting activity categories not explicitly present during training. With this property in mind, Vid2Burn is accompanied with a cross-category benchmark, where the task is to regress caloric expenditure for types of physical activities not present during training. An extensive evaluation of state-of-the-art approaches for video recognition modified for the energy expenditure estimation task demonstrates the difficulty of this problem, especially for new activity types at test-time, marking a new research direction. Dataset and code are available at https://github.com/KPeng9510/Vid2Burn.