Building, Reusing, and Generalizing Abstract Representations from Concrete Sequences

TL;DR

This paper introduces a hierarchical variable learning model (HVM) that efficiently learns and abstracts patterns from sequences, outperforming standard algorithms and LLMs in transfer and compression tasks, aligning with human cognition.

Contribution

The paper presents a novel non-parametric hierarchical model that learns abstract sequence representations and demonstrates its effectiveness in language tasks and human-like transfer capabilities.

Findings

HVM learns more efficient dictionaries than Lempel-Ziv.

HVM's sequence likelihood correlates with human recall times.

HVM effectively balances compression and generalization.

Abstract

Humans excel at learning abstract patterns across different sequences, filtering out irrelevant details, and transferring these generalized concepts to new sequences. In contrast, many sequence learning models lack the ability to abstract, which leads to memory inefficiency and poor transfer. We introduce a non-parametric hierarchical variable learning model (HVM) that learns chunks from sequences and abstracts contextually similar chunks as variables. HVM efficiently organizes memory while uncovering abstractions, leading to compact sequence representations. When learning on language datasets such as babyLM, HVM learns a more efficient dictionary than standard compression algorithms such as Lempel-Ziv. In a sequence recall task requiring the acquisition and transfer of variables embedded in sequences, we demonstrate HVM's sequence likelihood correlates with human recall times. In contrast, large language models (LLMs) struggle to transfer abstract variables as effectively as humans. From HVM's adjustable layer of abstraction, we demonstrate that the model realizes a precise trade-off between compression and generalization. Our work offers a cognitive model that captures the learning and transfer of abstract representations in human cognition and differentiates itself from LLMs.