DINO Eats CLIP: Adapting Beyond Knowns for Open-set 3D Object Retrieval

TL;DR

This paper introduces DEC, a novel framework combining DINO and CLIP models with dynamic multi-view integration and virtual feature synthesis to improve open-set 3D object retrieval.

Contribution

It proposes a new method that leverages DINO for better fine-grained features and synthesizes virtual features to enhance open-set discrimination in 3D retrieval.

Findings

DEC outperforms previous methods on standard benchmarks.

Chunking and Adapting Module improves robustness of multi-view features.

Virtual Feature Synthesis significantly enhances open-set recognition.

Abstract

Vision foundation models have shown great promise for open-set 3D object retrieval (3DOR) through efficient adaptation to multi-view images. Leveraging semantically aligned latent space, previous work typically adapts the CLIP encoder to build view-based 3D descriptors. Despite CLIP's strong generalization ability, its lack of fine-grainedness prompted us to explore the potential of a more recent self-supervised encoder-DINO. To address this, we propose DINO Eats CLIP (DEC), a novel framework for dynamic multi-view integration that is regularized by synthesizing data for unseen classes. We first find that simply mean-pooling over view features from a frozen DINO backbone gives decent performance. Yet, further adaptation causes severe overfitting on average view patterns of known classes. To combat it, we then design a module named Chunking and Adapting Module (CAM). It segments multi-view images into chunks and dynamically integrates local view relations, yielding more robust features than the standard pooling strategy. Finally, we propose Virtual Feature Synthesis (VFS) module to mitigate bias towards known categories explicitly. Under the hood, VFS leverages CLIP's broad, pre-aligned vision-language space to synthesize virtual features for unseen classes. By exposing DEC to these virtual features, we greatly enhance its open-set discrimination capacity. Extensive experiments on standard open-set 3DOR benchmarks demonstrate its superior efficacy.