Transformers for scientific data: a pedagogical review for astronomers

TL;DR

This review introduces transformers, a deep learning architecture originally for NLP, explaining their mathematics, architecture, and applications in astronomy, aiming to help scientists adopt this technology for data analysis.

Contribution

It provides a pedagogical overview of transformers tailored for astronomers, including mathematical foundations, architecture details, and practical applications in time series and imaging data.

Findings

Transformers are effective for astronomical data analysis.

The review clarifies the mathematics behind self-attention.

Applications include time series and imaging in astronomy.

Abstract

The deep learning architecture associated with ChatGPT and related generative AI products is known as transformers. Initially applied to Natural Language Processing, transformers and the self-attention mechanism they exploit have gained widespread interest across the natural sciences. The goal of this pedagogical and informal review is to introduce transformers to scientists. The review includes the mathematics underlying the attention mechanism, a description of the original transformer architecture, and a section on applications to time series and imaging data in astronomy. We include a Frequently Asked Questions section for readers who are curious about generative AI or interested in getting started with transformers for their research problem.