NLP4Neuro: Sequence-to-sequence learning for neural population decoding

TL;DR

This paper evaluates the use of large language models for decoding behavior from brain-wide neural activity, demonstrating improved accuracy and interpretability in neural circuit analysis.

Contribution

It systematically tests off-the-shelf LLMs on neural decoding tasks, highlighting the benefits of pre-training and mixture-of-experts models for brain-wide activity analysis.

Findings

Pre-trained LLMs improve neural decoding accuracy.

Mixture-of-experts LLMs enhance behavioral prediction.

Interpretable neuron salience maps align with neuroanatomy.

Abstract

Delineating how animal behavior arises from neural activity is a foundational goal of neuroscience. However, as the computations underlying behavior unfold in networks of thousands of individual neurons across the entire brain, this presents challenges for investigating neural roles and computational mechanisms in large, densely wired mammalian brains during behavior. Transformers, the backbones of modern large language models (LLMs), have become powerful tools for neural decoding from smaller neural populations. These modern LLMs have benefited from extensive pre-training, and their sequence-to-sequence learning has been shown to generalize to novel tasks and data modalities, which may also confer advantages for neural decoding from larger, brain-wide activity recordings. Here, we present a systematic evaluation of off-the-shelf LLMs to decode behavior from brain-wide populations, termed NLP4Neuro, which we used to test LLMs on simultaneous calcium imaging and behavior recordings in larval zebrafish exposed to visual motion stimuli. Through NLP4Neuro, we found that LLMs become better at neural decoding when they use pre-trained weights learned from textual natural language data. Moreover, we found that a recent mixture-of-experts LLM, DeepSeek Coder-7b, significantly improved behavioral decoding accuracy, predicted tail movements over long timescales, and provided anatomically consistent highly interpretable readouts of neuron salience. NLP4Neuro demonstrates that LLMs are highly capable of informing brain-wide neural circuit dissection.