Multimodal Fusion of Glucose Monitoring and Food Imagery for Caloric Content Prediction

TL;DR

This paper presents a multimodal deep learning framework that combines glucose monitoring, demographic, microbiome, and food imagery data to improve the accuracy of caloric content prediction for dietary management in diabetes.

Contribution

The study introduces a novel multimodal fusion approach that integrates physiological, demographic, microbiome, and visual data for enhanced caloric estimation accuracy.

Findings

Achieved RMSRE of 0.2544, over 50% better than baselines.

Demonstrated the effectiveness of multimodal data fusion in dietary assessment.

Improved caloric prediction accuracy for personalized diabetes management.

Abstract

Effective dietary monitoring is critical for managing Type 2 diabetes, yet accurately estimating caloric intake remains a major challenge. While continuous glucose monitors (CGMs) offer valuable physiological data, they often fall short in capturing the full nutritional profile of meals due to inter-individual and meal-specific variability. In this work, we introduce a multimodal deep learning framework that jointly leverages CGM time-series data, Demographic/Microbiome, and pre-meal food images to enhance caloric estimation. Our model utilizes attention based encoding and a convolutional feature extraction for meal imagery, multi-layer perceptrons for CGM and Microbiome data followed by a late fusion strategy for joint reasoning. We evaluate our approach on a curated dataset of over 40 participants, incorporating synchronized CGM, Demographic and Microbiome data and meal photographs with standardized caloric labels. Our model achieves a Root Mean Squared Relative Error (RMSRE) of 0.2544, outperforming the baselines models by over 50%. These findings demonstrate the potential of multimodal sensing to improve automated dietary assessment tools for chronic disease management.