Galactica: A Large Language Model for Science

TL;DR

Galactica is a large language model trained on scientific data that excels in understanding, reasoning, and organizing scientific knowledge, outperforming existing models on various scientific tasks and setting new benchmarks.

Contribution

The paper introduces Galactica, a specialized scientific language model trained on extensive scientific data, demonstrating superior performance on scientific reasoning and knowledge tasks.

Findings

Outperforms GPT-3 on LaTeX equations by 68.2%

Achieves state-of-the-art on PubMedQA and MedMCQA

Outperforms BLOOM and OPT-175B on BIG-bench

Abstract

Information overload is a major obstacle to scientific progress. The explosive growth in scientific literature and data has made it ever harder to discover useful insights in a large mass of information. Today scientific knowledge is accessed through search engines, but they are unable to organize scientific knowledge alone. In this paper we introduce Galactica: a large language model that can store, combine and reason about scientific knowledge. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. We outperform existing models on a range of scientific tasks. On technical knowledge probes such as LaTeX equations, Galactica outperforms the latest GPT-3 by 68.2% versus 49.0%. Galactica also performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%. It also sets a new state-of-the-art on downstream tasks such as PubMedQA and MedMCQA dev of 77.6% and 52.9%. And despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. We believe these results demonstrate the potential for language models as a new interface for science. We open source the model for the benefit of the scientific community.