Exploring PCA-based feature representations of image pixels via CNN to enhance food image segmentation

TL;DR

This paper introduces an unsupervised CNN-based PCA feature extraction method for food ingredient segmentation, achieving stable, interpretable results with minimal labeled data on FoodSeg103.

Contribution

It proposes a novel PCA-based feature representation approach using CNNs for improved, unsupervised food image segmentation, emphasizing optimal feature selection and clustering without dataset fine-tuning.

Findings

Principal component maps enhance clustering quality.

Eigenvalue count correlates with optimal cluster number.

Achieved 0.5423 mIoU on FoodSeg103 dataset.

Abstract

For open vocabulary recognition of ingredients in food images, segmenting the ingredients is a crucial step. This paper proposes a novel approach that explores PCA-based feature representations of image pixels using a convolutional neural network (CNN) to enhance segmentation. An internal clustering metric based on the silhouette score is defined to evaluate the clustering quality of various pixel-level feature representations generated by different feature maps derived from various CNN backbones. Using this metric, the paper explores optimal feature representation selection and suitable clustering methods for ingredient segmentation. Additionally, it is found that principal component (PC) maps derived from concatenations of backbone feature maps improve the clustering quality of pixel-level feature representations, resulting in stable segmentation outcomes. Notably, the number of selected eigenvalues can be used as the number of clusters to achieve good segmentation results. The proposed method performs well on the ingredient-labeled dataset FoodSeg103, achieving a mean Intersection over Union (mIoU) score of 0.5423. Importantly, the proposed method is unsupervised, and pixel-level feature representations from backbones are not fine-tuned on specific datasets. This demonstrates the flexibility, generalizability, and interpretability of the proposed method, while reducing the need for extensive labeled datasets.