SpikeProphecy: A Large-Scale Benchmark for Autoregressive Neural Population Forecasting

TL;DR

SpikeProphecy introduces a comprehensive benchmark for evaluating autoregressive neural models predicting neural spike activity, emphasizing detailed performance decomposition over traditional scalar metrics.

Contribution

It presents a novel population metric decomposition and applies it to large-scale electrophysiology data, revealing insights into model performance and brain-region predictability.

Findings

Decomposition reveals distinct aspects of predictive performance.

Predictability ranking consistent across multiple models and regions.

Identifies a sub-Poisson evaluation floor and challenges in ANN-to-SNN transfer.

Abstract

Neural population models, which predict the joint firing of many simultaneously recorded neurons forward in time, are typically evaluated by a single aggregate Pearson correlation $r$ between predicted and actual spike counts, a number that masks critical structure. We argue that how we evaluate spike forecasting matters as much as what we build, and introduce SpikeProphecy, the first large-scale benchmark for causal, autoregressive spike-count forecasting on real electrophysiology recordings. Our core contribution is a population metric decomposition that separates aggregate performance into temporal fidelity, spatial pattern accuracy, and magnitude-invariant alignment. The decomposition surfaces aspects of the underlying data that an aggregate scalar collapses together. We apply the protocol to 105 Neuropixels sessions (Steinmetz 2019 + IBL Repeated Site; ~89,800 neurons) with seven architecture baselines spanning four structural families: four SSMs (three diagonal and one non-diagonal), a Transformer, an LSTM, and a spiking network. The decomposition surfaces a brain-region predictability ranking that reproduces across all seven baselines and survives ANCOVA correction for firing-statistics constraints (region $\Delta R^2 = 0.018$ above the firing-statistics covariates). It also exposes a sub-Poisson evaluation floor where rigorous metrics combine with genuine biophysical constraints on regular spike trains, and yields a negative result on KL-on-output-rates distillation for ANN-to-SNN transfer in this Poisson count domain.