From seeing to remembering: Images with harder-to-reconstruct representations leave stronger memory traces

TL;DR

This paper demonstrates that images with more difficult-to-reconstruct representations from a sparse coding model tend to leave stronger memory traces, linking perceptual processing to memory durability.

Contribution

It introduces a sparse coding model that predicts memory strength from reconstruction residuals, connecting perception and memory in a novel way.

Findings

Reconstruction error predicts memory accuracy.

Reconstruction error explains response latencies during retrieval.

Model-driven psychophysics confirms the theoretical prediction.

Abstract

Much of what we remember is not due to intentional selection, but simply a by-product of perceiving. This raises a foundational question about the architecture of the mind: How does perception interface with and influence memory? Here, inspired by a classic proposal relating perceptual processing to memory durability, the level-of-processing theory, we present a sparse coding model for compressing feature embeddings of images, and show that the reconstruction residuals from this model predict how well images are encoded into memory. In an open memorability dataset of scene images, we show that reconstruction error not only explains memory accuracy but also response latencies during retrieval, subsuming, in the latter case, all of the variance explained by powerful vision-only models. We also confirm a prediction of this account with 'model-driven psychophysics'. This work establishes reconstruction error as a novel signal interfacing perception and memory, possibly through adaptive modulation of perceptual processing.