SnappyMeal: Design and Longitudinal Evaluation of a Multimodal AI Food Logging Application

TL;DR

SnappyMeal is a multimodal AI food logging system designed to improve flexibility and accuracy in dietary tracking through goal-dependent questions and information retrieval, evaluated over real-world use.

Contribution

The paper introduces SnappyMeal, a novel multimodal AI system for food logging that enhances flexibility and accuracy compared to traditional methods.

Findings

Users praised multiple input methods and perceived high accuracy.

SnappyMeal effectively captures diverse food instances in real-world settings.

The system demonstrates potential for improved dietary tracking through multimodal AI.

Abstract

Food logging, both self-directed and prescribed, plays a critical role in uncovering correlations between diet, medical, fitness, and health outcomes. Through conversations with nutritional experts and individuals who practice dietary tracking, we find current logging methods, such as handwritten and app-based journaling, are inflexible and result in low adherence and potentially inaccurate nutritional summaries. These findings, corroborated by prior literature, emphasize the urgent need for improved food logging methods. In response, we propose SnappyMeal, an AI-powered dietary tracking system that leverages multimodal inputs to enable users to more flexibly log their food intake. SnappyMeal introduces goal-dependent follow-up questions to intelligently seek missing context from the user and information retrieval from user grocery receipts and nutritional databases to improve accuracy. We evaluate SnappyMeal through publicly available nutrition benchmarks and a multi-user, 3-week, in-the-wild deployment capturing over 500 logged food instances. Users strongly praised the multiple available input methods and reported a strong perceived accuracy. These insights suggest that multimodal AI systems can be leveraged to significantly improve dietary tracking flexibility and context-awareness, laying the groundwork for a new class of intelligent self-tracking applications.