Weakly Supervised Panoptic Segmentation for Defect-Based Grading of Fresh Produce

TL;DR

This paper explores using the Segment Anything Model to generate dense segmentation masks from sparse annotations for defect grading in bananas, reducing annotation effort and enabling defect quantification in low-data agricultural settings.

Contribution

It demonstrates a novel application of SAM for weakly supervised panoptic segmentation in agriculture, specifically for banana defect detection, highlighting its potential and limitations.

Findings

SAM-generated masks align with human annotations but have failure cases.

The approach reduces annotation effort significantly.

It provides practical estimates of defect count and size.

Abstract

Visual inspection for defect grading in agricultural supply chains is crucial but traditionally labor-intensive and error-prone. Automated computer vision methods typically require extensively annotated datasets, which are often unavailable in decentralized supply chains. We address this challenge by evaluating the Segment Anything Model (SAM) to generate dense panoptic segmentation masks from sparse annotations. These dense predictions are then used to train a supervised panoptic segmentation model. Focusing on banana surface defects (bruises and scars), we validate our approach using 476 field images annotated with 1440 defects. While SAM-generated masks generally align with human annotations, substantially reducing annotation effort, we explicitly identify failure cases associated with specific defect sizes and shapes. Despite these limitations, our approach offers practical estimates of defect number and relative size from panoptic masks, underscoring the potential and current boundaries of foundation models for defect quantification in low-data agricultural scenarios. GitHub: https://github.com/manuelknott/banana-defect-segmentation