Static Segmentation by Tracking: A Label-Efficient Approach for Fine-Grained Specimen Image Segmentation

TL;DR

This paper introduces Static Segmentation by Tracking (SST), a label-efficient method that leverages specimen image sequences to perform fine-grained trait segmentation with minimal annotations, significantly reducing manual labeling effort.

Contribution

The paper presents SST, a novel approach that reframes trait segmentation as a tracking problem using pseudo-video sequences, enabling high-quality segmentation with only one labeled image per species.

Findings

Achieves high-quality trait segmentation with only one labeled image.

Introduces a cycle-consistent loss for improved fine-tuning.

Demonstrates applicability to one-shot instance segmentation and image retrieval.

Abstract

We study image segmentation in the biological domain, particularly trait segmentation from specimen images (e.g., butterfly wing stripes, beetle elytra). This fine-grained task is crucial for understanding the biology of organisms, but it traditionally requires manually annotating segmentation masks for hundreds of images per species, making it highly labor-intensive. To address this challenge, we propose a label-efficient approach, Static Segmentation by Tracking (SST), based on a key insight: while specimens of the same species exhibit natural variation, the traits of interest show up consistently. This motivates us to concatenate specimen images into a ``pseudo-video'' and reframe trait segmentation as a tracking problem. Specifically, SST generates masks for unlabeled images by propagating annotated or predicted masks from the ``pseudo-preceding'' images. Built upon recent video segmentation models, such as Segment Anything Model 2, SST achieves high-quality trait segmentation with only one labeled image per species, marking a breakthrough in specimen image analysis. To further enhance segmentation quality, we introduce a cycle-consistent loss for fine-tuning, again requiring only one labeled image. Additionally, we demonstrate the broader potential of SST, including one-shot instance segmentation in natural images and trait-based image retrieval.