Toward a Spectral Foundation Model: An Attention-Based Approach with Domain-Inspired Fine-Tuning and Wavelength Parameterization

TL;DR

This paper introduces an attention-based spectral emulator with domain-specific fine-tuning and wavelength parameterization, significantly improving spectral fitting accuracy for astrophysical surveys.

Contribution

It presents a novel spectral emulator leveraging attention mechanisms, domain-inspired fine-tuning, and wavelength as a model parameter, outperforming existing methods.

Findings

Achieves 5-10x better performance than current methods.

Effectively captures long-range wavelength correlations.

Enables spectrum generation on arbitrary wavelength grids.

Abstract

Astrophysical explorations are underpinned by large-scale stellar spectroscopy surveys, necessitating a paradigm shift in spectral fitting techniques. Our study proposes three enhancements to transcend the limitations of the current spectral emulation models. We implement an attention-based emulator, adept at unveiling long-range information between wavelength pixels. We leverage a domain-specific fine-tuning strategy where the model is pre-trained on spectra with fixed stellar parameters and variable elemental abundances, followed by fine-tuning on the entire domain. Moreover, by treating wavelength as an autonomous model parameter, akin to neural radiance fields, the model can generate spectra on any wavelength grid. In the case with a training set of O(1000), our approach exceeds current leading methods by a factor of 5-10 across all metrics.