Statistical Learning for Latent Embedding Alignment with Application to Brain Encoding and Decoding

TL;DR

This paper introduces a statistical learning framework for aligning latent embeddings in brain encoding and decoding, improving sample efficiency and handling subject variability, with theoretical guarantees and strong empirical results.

Contribution

It proposes a lightweight, modular alignment framework with inverse semi-supervised learning and meta transfer learning, enhancing brain decoding with limited data and heterogeneity.

Findings

Achieves competitive performance on large-scale fMRI-image data

Provides finite-sample generalization bounds and safety guarantees

Operates efficiently by keeping encoders and decoders frozen

Abstract

Brain encoding and decoding aims to understand the relationship between external stimuli and brain activities, and is a fundamental problem in neuroscience. In this article, we study latent embedding alignment for brain encoding and decoding, with a focus on improving sample efficiency under limited fMRI-stimulus paired data and substantial subject heterogeneity. We propose a lightweight alignment framework equipped with two statistical learning components: inverse semi-supervised learning that leverages abundant unpaired stimulus embeddings through inverse mapping and residual debiasing, and meta transfer learning that borrows strength from pretrained models across subjects via sparse aggregation and residual correction. Both methods operate exclusively at the alignment stage while keeping encoders and decoders frozen, allowing for efficient computation, modular deployment, and rigorous theoretical analysis. We establish finite-sample generalization bounds and safety guarantees, and demonstrate competitive empirical performance on the large-scale fMRI-image reconstruction benchmark data.