A SAM based Tool for Semi-Automatic Food Annotation

TL;DR

This paper introduces a semi-automatic food annotation tool based on SAM, designed to facilitate food image segmentation and categorization for non-experts, thereby addressing data scarcity in AI-driven nutrition research.

Contribution

It presents a user-friendly, prompt-based food annotation tool and a fine-tuned SAM model, MealSAM, tailored specifically for food image segmentation, enhancing accessibility and data collection.

Findings

The tool enables efficient, user-guided food segmentation.

MealSAM improves segmentation accuracy for food images.

The approach promotes broader participation in food data annotation.

Abstract

The advancement of artificial intelligence (AI) in food and nutrition research is hindered by a critical bottleneck: the lack of annotated food data. Despite the rise of highly efficient AI models designed for tasks such as food segmentation and classification, their practical application might necessitate proficiency in AI and machine learning principles, which can act as a challenge for non-AI experts in the field of nutritional sciences. Alternatively, it highlights the need to translate AI models into user-friendly tools that are accessible to all. To address this, we present a demo of a semi-automatic food image annotation tool leveraging the Segment Anything Model (SAM). The tool enables prompt-based food segmentation via user interactions, promoting user engagement and allowing them to further categorise food items within meal images and specify weight/volume if necessary. Additionally, we release a fine-tuned version of SAM's mask decoder, dubbed MealSAM, with the ViT-B backbone tailored specifically for food image segmentation. Our objective is not only to contribute to the field by encouraging participation, collaboration, and the gathering of more annotated food data but also to make AI technology available for a broader audience by translating AI into practical tools.