NeuroSketch: An Effective Framework for Neural Decoding via Systematic Architectural Optimization

TL;DR

NeuroSketch is a systematic architecture optimization framework that significantly improves neural decoding performance across multiple modalities and tasks, setting new state-of-the-art results.

Contribution

This study introduces NeuroSketch, a novel framework for neural decoding that systematically optimizes model architectures, demonstrating its effectiveness across diverse datasets and modalities.

Findings

CNN-2D outperforms other architectures in neural decoding

Performance improves with macro- to micro-level architecture optimization

NeuroSketch achieves state-of-the-art results across all evaluated datasets

Abstract

Neural decoding, a critical component of Brain-Computer Interface (BCI), has recently attracted increasing research interest. Previous research has focused on leveraging signal processing and deep learning methods to enhance neural decoding performance. However, the in-depth exploration of model architectures remains underexplored, despite its proven effectiveness in other tasks such as energy forecasting and image classification. In this study, we propose NeuroSketch, an effective framework for neural decoding via systematic architecture optimization. Starting with the basic architecture study, we find that CNN-2D outperforms other architectures in neural decoding tasks and explore its effectiveness from temporal and spatial perspectives. Building on this, we optimize the architecture from macro- to micro-level, achieving improvements in performance at each step. The exploration process and model validations take over 5,000 experiments spanning three distinct modalities (visual, auditory, and speech), three types of brain signals (EEG, SEEG, and ECoG), and eight diverse decoding tasks. Experimental results indicate that NeuroSketch achieves state-of-the-art (SOTA) performance across all evaluated datasets, positioning it as a powerful tool for neural decoding. Our code and scripts are available at https://github.com/Galaxy-Dawn/NeuroSketch.