Active Use of Latent Constituency Representation in both Humans and Large Language Models

TL;DR

This study shows that both humans and large language models construct similar hierarchical constituency representations of sentences, as evidenced by their behavior in a novel word deletion task, revealing shared latent tree structures.

Contribution

The paper demonstrates that latent tree-structured constituency representations emerge in both humans and LLMs, bridging cognitive science and AI understanding of sentence structure.

Findings

Humans and LLMs tend to delete entire constituents, not just words.

Latent constituency trees can be reconstructed from deletion behaviors.

Naive sequence models do not exhibit this constituency behavior.

Abstract

Understanding how sentences are internally represented in the human brain, as well as in large language models (LLMs) such as ChatGPT, is a major challenge for cognitive science. Classic linguistic theories propose that the brain represents a sentence by parsing it into hierarchically organized constituents. In contrast, LLMs do not explicitly parse linguistic constituents and their latent representations remains poorly explained. Here, we demonstrate that humans and LLMs construct similar latent representations of hierarchical linguistic constituents by analyzing their behaviors during a novel one-shot learning task, in which they infer which words should be deleted from a sentence. Both humans and LLMs tend to delete a constituent, instead of a nonconstituent word string. In contrast, a naive sequence processing model that has access to word properties and ordinal positions does not show this property. Based on the word deletion behaviors, we can reconstruct the latent constituency tree representation of a sentence for both humans and LLMs. These results demonstrate that a latent tree-structured constituency representation can emerge in both the human brain and LLMs.