Bootstrap Segmentation Foundation Model under Distribution Shift via Object-Centric Learning

TL;DR

This paper introduces SlotSAM, a self-supervised, object-centric learning method that enhances foundation models' ability to generalize across out-of-distribution data, especially in challenging environments like medical and camouflaged images.

Contribution

SlotSAM reconstructs encoder features into object-centric representations, improving foundation models' robustness and generalization with minimal fine-tuning and a simple, adaptable approach.

Findings

Significantly improves out-of-distribution generalization

Enhances object-level perceptual capabilities of foundation models

Requires limited parameter fine-tuning

Abstract

Foundation models have made incredible strides in achieving zero-shot or few-shot generalization, leveraging prompt engineering to mimic the problem-solving approach of human intelligence. However, when it comes to some foundation models like Segment Anything, there is still a challenge in performing well on out-of-distribution data, including camouflaged and medical images. Inconsistent prompting strategies during fine-tuning and testing further compound the issue, leading to decreased performance. Drawing inspiration from how human cognition processes new environments, we introduce SlotSAM, a method that reconstructs features from the encoder in a self-supervised manner to create object-centric representations. These representations are then integrated into the foundation model, bolstering its object-level perceptual capabilities while reducing the impact of distribution-related variables. The beauty of SlotSAM lies in its simplicity and adaptability to various tasks, making it a versatile solution that significantly enhances the generalization abilities of foundation models. Through limited parameter fine-tuning in a bootstrap manner, our approach paves the way for improved generalization in novel environments. The code is available at github.com/lytang63/SlotSAM.