ADAM: Autonomous Discovery and Annotation Model using LLMs for Context-Aware Annotations

TL;DR

ADAM is a self-refining, training-free framework that uses large language models and visual embeddings to enable open-world object detection and annotation without prior category labels.

Contribution

It introduces a novel self-refining, category-agnostic approach combining LLMs and visual embeddings for open-world object annotation without retraining.

Findings

Effective annotation of novel categories demonstrated on COCO and PASCAL datasets.

No fine-tuning or retraining required for new object categories.

Improves open-world object detection through self-refinement and contextual reasoning.

Abstract

Object detection models typically rely on predefined categories, limiting their ability to identify novel objects in open-world scenarios. To overcome this constraint, we introduce ADAM: Autonomous Discovery and Annotation Model, a training-free, self-refining framework for open-world object labeling. ADAM leverages large language models (LLMs) to generate candidate labels for unknown objects based on contextual information from known entities within a scene. These labels are paired with visual embeddings from CLIP to construct an Embedding-Label Repository (ELR) that enables inference without category supervision. For a newly encountered unknown object, ADAM retrieves visually similar instances from the ELR and applies frequency-based voting and cross-modal re-ranking to assign a robust label. To further enhance consistency, we introduce a self-refinement loop that re-evaluates repository labels using visual cohesion analysis and k-nearest-neighbor-based majority re-labeling. Experimental results on the COCO and PASCAL datasets demonstrate that ADAM effectively annotates novel categories using only visual and contextual signals, without requiring any fine-tuning or retraining.