BYOCL: Build Your Own Consistent Latent with Hierarchical Representative Latent Clustering

TL;DR

BYOCL introduces a hierarchical clustering-based segmentation method that enhances consistency across image sequences, significantly reducing computational costs and outperforming existing models without additional training.

Contribution

It presents a novel, training-free, plug-and-play hierarchical clustering approach for consistent segmentation across image sequences using foundation models.

Findings

Outperforms SAM in extensive experiments

Reduces time and space complexity exponentially

Achieves state-of-the-art segmentation results

Abstract

To address the semantic inconsistency issue with SAM or other single-image segmentation models handling image sequences, we introduce BYOCL. This novel model outperforms SAM in extensive experiments, showcasing its Hierarchical prototype capabilities across CLIP and other representations. BYOCL significantly reduces time and space consumption by dividing inputs into smaller batches, achieving exponential time reduction compared to previous methods. Our approach leverages the SAM image encoder for feature extraction, followed by Intra-Batch and Inter-Batch clustering algorithms. Extensive experiments demonstrate that BYOCL far exceeds the previous state-of-the-art single image segmentation model. Our work is the first to apply consistent segmentation using foundation models without requiring training, utilizing plug-and-play modules for any latent space, making our method highly efficientModels are available at \href{https://github.com/cyt1202/BYOCL.git