Geometry of naturalistic object representations in recurrent neural network models of working memory

TL;DR

This study investigates how recurrent neural networks encode naturalistic object information in working memory, revealing shared and task-specific representational structures across multiple cognitive tasks.

Contribution

It introduces sensory-cognitive models trained on naturalistic stimuli across nine N-back tasks, highlighting how different RNN architectures represent object features and maintain information.

Findings

Multi-task RNNs encode both relevant and irrelevant info simultaneously.

Latent spaces are largely shared in vanilla RNNs but task-specific in gated RNNs.

Object features are embedded in less orthogonalized spaces than perceptual space.

Abstract

Working memory is a central cognitive ability crucial for intelligent decision-making. Recent experimental and computational work studying working memory has primarily used categorical (i.e., one-hot) inputs, rather than ecologically relevant, multidimensional naturalistic ones. Moreover, studies have primarily investigated working memory during single or few cognitive tasks. As a result, an understanding of how naturalistic object information is maintained in working memory in neural networks is still lacking. To bridge this gap, we developed sensory-cognitive models, comprising a convolutional neural network (CNN) coupled with a recurrent neural network (RNN), and trained them on nine distinct N-back tasks using naturalistic stimuli. By examining the RNN's latent space, we found that: (1) Multi-task RNNs represent both task-relevant and irrelevant information simultaneously while performing tasks; (2) The latent subspaces used to maintain specific object properties in vanilla RNNs are largely shared across tasks, but highly task-specific in gated RNNs such as GRU and LSTM; (3) Surprisingly, RNNs embed objects in new representational spaces in which individual object features are less orthogonalized relative to the perceptual space; (4) The transformation of working memory encodings (i.e., embedding of visual inputs in the RNN latent space) into memory was shared across stimuli, yet the transformations governing the retention of a memory in the face of incoming distractor stimuli were distinct across time. Our findings indicate that goal-driven RNNs employ chronological memory subspaces to track information over short time spans, enabling testable predictions with neural data.