Can deep learning match the efficiency of human visual long-term memory in storing object details?

TL;DR

This study compares human visual long-term memory with deep learning models, finding that humans recognize images after one exposure while models require about ten exposures, highlighting a significant gap in memory efficiency.

Contribution

The paper provides a rigorous, quantitative comparison showing that current deep learning models need multiple exposures to match human recognition performance after a single exposure.

Findings

Humans recognize images after a single exposure.

Deep learning models need approximately ten exposures.

Scaling up data or model size offers limited improvements.

Abstract

Humans have a remarkably large capacity to store detailed visual information in long-term memory even after a single exposure, as demonstrated by classic experiments in psychology. For example, Standing (1973) showed that humans could recognize with high accuracy thousands of pictures that they had seen only once a few days prior to a recognition test. In deep learning, the primary mode of incorporating new information into a model is through gradient descent in the model's parameter space. This paper asks whether deep learning via gradient descent can match the efficiency of human visual long-term memory to incorporate new information in a rigorous, head-to-head, quantitative comparison. We answer this in the negative: even in the best case, models learning via gradient descent require approximately 10 exposures to the same visual materials in order to reach a recognition memory performance humans achieve after only a single exposure. Prior knowledge induced via pretraining and bigger model sizes improve performance, but these improvements are not very visible after a single exposure (it takes a few exposures for the improvements to become apparent), suggesting that simply scaling up the pretraining data size or model size might not be a feasible strategy to reach human-level memory efficiency.