PASTA: Vision Transformer Patch Aggregation for Weakly Supervised Target and Anomaly Segmentation

TL;DR

PASTA introduces a weakly supervised vision transformer-based method for real-time, pixel-precise target and anomaly segmentation in unstructured environments, reducing training time significantly.

Contribution

It presents a novel weakly supervised pipeline using ViT features and semantic prompts for zero-shot segmentation, outperforming domain-specific baselines.

Findings

Achieves up to 88.3% IoU for target segmentation.

Reduces training time by 75.8% compared to baselines.

Performs well on industrial and agricultural datasets.

Abstract

Detecting unseen anomalies in unstructured environments presents a critical challenge for industrial and agricultural applications such as material recycling and weeding. Existing perception systems frequently fail to satisfy the strict operational requirements of these domains, specifically real-time processing, pixel-level segmentation precision, and robust accuracy, due to their reliance on exhaustively annotated datasets. To address these limitations, we propose a weakly supervised pipeline for object segmentation and classification using weak image-level supervision called 'Patch Aggregation for Segmentation of Targets and Anomalies' (PASTA). By comparing an observed scene with a nominal reference, PASTA identifies Target and Anomaly objects through distribution analysis in self-supervised Vision Transformer (ViT) feature spaces. Our pipeline utilizes semantic text-prompts via the Segment Anything Model 3 to guide zero-shot object segmentation. Evaluations on a custom steel scrap recycling dataset and a plant dataset demonstrate a 75.8% training time reduction of our approach to domain-specific baselines. While being domain-agnostic, our method achieves superior Target (up to 88.3% IoU) and Anomaly (up to 63.5% IoU) segmentation performance in the industrial and agricultural domain.