Representational drift changes the encoding of fast and slow-varying natural scene features differently

TL;DR

This study investigates how neural representations of natural scene features drift over time, revealing that fast-varying features like local motion are more affected than slow-varying scenery features, using a novel contrastive learning approach.

Contribution

Introduces a weakly supervised contrastive learning method to extract shared neural encoding of natural scene features and quantify their drift over sessions.

Findings

Fast-varying local motion features exhibit 5-6 times higher drift rate than scenery features.

The learned embedding captures multiple features at high temporal resolution.

Decoding performance decreases over sessions, indicating representational drift.

Abstract

Representational drift refers to an unstable mapping between neural activity and input sensory or output behavioral variables. While much work has focused on the effect of representational drift on single, simple external variables, we investigate the differences in representational drift across spatiotemporal features in a moving visual stimulus. The neural responses across animals to the same movie reflect both common, encoded stimulus features and idiosyncratic individual variation. To extract the shared neural encoding of stimulus features only, we learn a latent space embedding using weakly supervised contrastive learning. This approach pulls neural activity together in the embedding space if they are responses to the same stimulus segment and push them apart if not. This approach enables us to probe how stimulus features fluctuating as fast as 33 ms (the movie frame rate) are encoded by variable neural codes across animals. It also allows us to investigate how representational drift changes the encoding in individuals across sessions. We observe that our learned embedding is near-optimal for decoding natural features (background scenery, local motion, complex spatio-temporal features, and time) and neural activity from novel animals. This suggests that our embedding retains the encoding of multiple features at higher temporal granularity compared to previous methods. To quantify representational drift, we apply the trained decoder (which achieves near-optimal performance in one session) to a subsequent session recorded 90 minutes later. We then use the decrease in decoding performance as a proxy for the magnitude of drift. We show that the drift changes the encoding of fast-varying local motion features at a rate 5-6 times higher than slower-varying scenery features. Drift also perturbs the local geometry in the embedding.