Food Image Classification and Segmentation with Attention-based Multiple Instance Learning

TL;DR

This paper introduces a weakly supervised approach using attention-based multiple instance learning for food image classification and segmentation, reducing the need for costly pixel-level annotations while maintaining effective performance.

Contribution

It proposes a novel weakly supervised methodology combining multiple instance learning and attention mechanisms for food image analysis without pixel-level labels.

Findings

Effective classification and segmentation achieved without pixel annotations

Attention mechanism generates meaningful semantic heat maps

Validated on FoodSeg103 dataset with promising results

Abstract

The demand for accurate food quantification has increased in the recent years, driven by the needs of applications in dietary monitoring. At the same time, computer vision approaches have exhibited great potential in automating tasks within the food domain. Traditionally, the development of machine learning models for these problems relies on training data sets with pixel-level class annotations. However, this approach introduces challenges arising from data collection and ground truth generation that quickly become costly and error-prone since they must be performed in multiple settings and for thousands of classes. To overcome these challenges, the paper presents a weakly supervised methodology for training food image classification and semantic segmentation models without relying on pixel-level annotations. The proposed methodology is based on a multiple instance learning approach in combination with an attention-based mechanism. At test time, the models are used for classification and, concurrently, the attention mechanism generates semantic heat maps which are used for food class segmentation. In the paper, we conduct experiments on two meta-classes within the FoodSeg103 data set to verify the feasibility of the proposed approach and we explore the functioning properties of the attention mechanism.