Bridging the Gap: Examining Vision Foundation Models for Optical and Radio Astronomy Applications

TL;DR

This paper evaluates how vision foundation models can be adapted for optical and radio astronomy, demonstrating their potential to improve classification and detection tasks despite domain-specific challenges.

Contribution

It provides a comprehensive framework for selecting, fine-tuning, and optimizing vision foundation models specifically for astrophysical datasets.

Findings

Features from foundation models improve optical galaxy classification.

Models achieve comparable or better radio object detection performance.

Radio galaxy classification remains challenging with foundation models.

Abstract

Vision foundation models, which have demonstrated significant potential in many multimedia applications, are often underutilized in the natural sciences. This is primarily due to mismatches between the nature of domain-specific scientific data and the typical training data used for foundation models, leading to distribution shifts. Scientific data often differ substantially in structure and characteristics, and researchers frequently face the challenge of optimizing model performance with limited labeled data of only a few hundred or thousand images. This work evaluates the performance of vision foundation models in astrophysics, with a focus on identifying the best practices for adapting them to domain-specific datasets. We aim to establish a framework for selecting, fine-tuning, and optimizing these models for common tasks in optical and radio astronomy. We compared multiple foundation models, including self-supervised, weakly supervised, and distillation-based architectures, across two representative optical and radio datasets. Experiments involved different fine-tuning strategies, projector heads, and data preprocessing techniques, with performance evaluated on classification and detection metrics. Features extracted by specific foundation models improved classification accuracy for optical galaxy images compared to conventional supervised training. Similarly, these models achieved equivalent or superior performance in object detection tasks with radio images. However, classification performance for radio galaxy images was generally poor, often falling short of supervised approaches. These findings demonstrate that vision foundation models can be effectively adapted to astrophysical applications, provided practitioners iterate on model selection, training strategies, and data handling.