Semantic search for 100M+ galaxy images using AI-generated captions

TL;DR

This paper presents AION-Search, a scalable semantic search engine for over 140 million galaxy images using AI-generated captions and contrastive alignment, enabling discovery of rare phenomena without manual labeling.

Contribution

It introduces a novel pipeline combining vision-language models and contrastive learning to enable large-scale, zero-shot semantic search in unlabeled scientific image datasets.

Findings

Outperforms direct image similarity search in identifying rare phenomena.

Achieves state-of-the-art zero-shot performance on galaxy image search.

Nearly doubles recall for challenging targets with VLM-based re-ranking.

Abstract

Finding scientifically interesting phenomena through slow, manual labeling campaigns severely limits our ability to explore the billions of galaxy images produced by telescopes. In this work, we develop a pipeline to create a semantic search engine from completely unlabeled image data. Our method leverages Vision-Language Models (VLMs) to generate descriptions for galaxy images, then contrastively aligns a pre-trained multimodal astronomy foundation model with these embedded descriptions to produce searchable embeddings at scale. We find that current VLMs provide descriptions that are sufficiently informative to train a semantic search model that outperforms direct image similarity search. Our model, AION-Search, achieves state-of-the-art zero-shot performance on finding rare phenomena despite training on randomly selected images with no deliberate curation for rare cases. Furthermore, we introduce a VLM-based re-ranking method that nearly doubles the recall for our most challenging targets in the top-100 results. For the first time, AION-Search enables flexible semantic search scalable to 140 million galaxy images, enabling discovery from previously infeasible searches. More broadly, our work provides an approach for making large, unlabeled scientific image archives semantically searchable, expanding data exploration capabilities in fields from Earth observation to microscopy. The code, data, and app are publicly available at https://github.com/NolanKoblischke/AION-Search