Identifying interactions across brain areas while accounting for individual-neuron dynamics with a Transformer-based variational autoencoder

TL;DR

This paper introduces a Transformer-based variational autoencoder integrated with a GLM to identify neural interactions across brain areas while accounting for individual neuron dynamics and trial-to-trial variability.

Contribution

It presents a novel hybrid model, GLM-Transformer, combining deep latent variable modeling with interpretable GLM components for neural data analysis.

Findings

Accurately recovers known neural coupling structures in synthetic data

Remains robust to shared background fluctuations

Identifies visual processing pathways consistent with established hierarchies

Abstract

Advances in large-scale recording technologies now enable simultaneous measurements from multiple brain areas, offering new opportunities to study signal transmission across interacting components of neural circuits. However, neural responses exhibit substantial trial-to-trial variability, often driven by unobserved factors such as subtle changes in animal behavior or internal states. To prevent evolving background dynamics from contaminating identification of functional coupling, we developed a hybrid neural spike train model, GLM-Transformer, that incorporates flexible, deep latent variable models into a point process generalized linear model (GLM) having an interpretable component for cross-population interactions. A Transformer-based variational autoencoder captures nonstationary individual-neuron dynamics that vary across trials, while standard nonparametric regression GLM coupling terms provide estimates of directed interactions between neural populations. We incorporate a low-rank structure on population-to-population coupling effects to improve scalability. Across synthetic datasets and mechanistic simulations, GLM-Transformer recovers known coupling structure and remains robust to shared background fluctuations. When applied to the Allen Institute Visual Coding dataset, it identifies feedforward pathways consistent with established visual hierarchies. This work offers a step toward improved identification of neural population interactions, and contributes to ongoing efforts aimed at achieving interpretable results while harvesting the benefits of deep learning.