Analyzing Encoded Concepts in Transformer Language Models

TL;DR

This paper introduces ConceptX, a framework for analyzing how transformer language models encode various linguistic concepts, revealing layer-specific representations and the complexity of encoded concepts.

Contribution

It presents a novel clustering-based method to identify and explain latent concepts in transformer models, highlighting their layered encoding of linguistic features.

Findings

Lower layers encode lexical concepts like affixation

Middle and higher layers capture core linguistic relations

Some encoded concepts are multi-faceted and not fully explained by existing concepts

Abstract

We propose a novel framework ConceptX, to analyze how latent concepts are encoded in representations learned within pre-trained language models. It uses clustering to discover the encoded concepts and explains them by aligning with a large set of human-defined concepts. Our analysis on seven transformer language models reveal interesting insights: i) the latent space within the learned representations overlap with different linguistic concepts to a varying degree, ii) the lower layers in the model are dominated by lexical concepts (e.g., affixation), whereas the core-linguistic concepts (e.g., morphological or syntactic relations) are better represented in the middle and higher layers, iii) some encoded concepts are multi-faceted and cannot be adequately explained using the existing human-defined concepts.