The Scaling Law in Stellar Light Curves

TL;DR

This paper explores how self-supervised Transformer models, specifically GPT-2, can effectively analyze stellar light curves, showing improved representation with scale and significantly higher sample efficiency for stellar property inference.

Contribution

It demonstrates the effectiveness of large-scale self-supervised Transformers in analyzing stellar light curves, surpassing supervised methods in sample efficiency and revealing scaling law properties.

Findings

Representation improves with model size up to 10^9 parameters.

Self-supervised Transformer achieves 3-10x better sample efficiency.

Scaling laws emerge in learned representations.

Abstract

Analyzing time series of fluxes from stars, known as stellar light curves, can reveal valuable information about stellar properties. However, most current methods rely on extracting summary statistics, and studies using deep learning have been limited to supervised approaches. In this research, we investigate the scaling law properties that emerge when learning from astronomical time series data using self-supervised techniques. By employing the GPT-2 architecture, we show the learned representation improves as the number of parameters increases from $10^4$ to $10^9$, with no signs of performance plateauing. We demonstrate that a self-supervised Transformer model achieves 3-10 times the sample efficiency compared to the state-of-the-art supervised learning model when inferring the surface gravity of stars as a downstream task. Our research lays the groundwork for analyzing stellar light curves by examining them through large-scale auto-regressive generative models.