Overcoming the Domain Gap in Neural Action Representations

TL;DR

This paper introduces a method to reduce the domain gap in neural action representations across different subjects by using 3D pose data and data swapping techniques, validated on diverse multimodal datasets.

Contribution

It proposes a novel approach combining 3D pose guidance and data swapping to improve generalization of neural action models across individuals.

Findings

Effective domain gap reduction demonstrated across fly, human ECoG, and RGB video datasets.

Improved model generalization to unseen subjects.

Versatile approach applicable to multiple neural and behavioral modalities.

Abstract

Relating animal behaviors to brain activity is a fundamental goal in neuroscience, with practical applications in building robust brain-machine interfaces. However, the domain gap between individuals is a major issue that prevents the training of general models that work on unlabeled subjects. Since 3D pose data can now be reliably extracted from multi-view video sequences without manual intervention, we propose to use it to guide the encoding of neural action representations together with a set of neural and behavioral augmentations exploiting the properties of microscopy imaging. To reduce the domain gap, during training, we swap neural and behavioral data across animals that seem to be performing similar actions. To demonstrate this, we test our methods on three very different multimodal datasets; one that features flies and their neural activity, one that contains human neural Electrocorticography (ECoG) data, and lastly the RGB video data of human activities from different viewpoints.