Discovery of Natural Language Concepts in Individual Units of CNNs

TL;DR

This paper investigates how individual units in CNNs trained on language tasks respond to specific linguistic concepts, revealing that units are selectively responsive to morphemes, words, and phrases, thus shedding light on the internal representations of deep language models.

Contribution

The paper introduces a concept alignment method to analyze unit responses and demonstrates that CNN units are selectively responsive to linguistic concepts across various architectures and tasks.

Findings

Units respond to specific morphemes, words, and phrases.

Analysis across multiple models and datasets confirms selective responsiveness.

Provides new insights into how CNNs understand natural language.

Abstract

Although deep convolutional networks have achieved improved performance in many natural language tasks, they have been treated as black boxes because they are difficult to interpret. Especially, little is known about how they represent language in their intermediate layers. In an attempt to understand the representations of deep convolutional networks trained on language tasks, we show that individual units are selectively responsive to specific morphemes, words, and phrases, rather than responding to arbitrary and uninterpretable patterns. In order to quantitatively analyze such an intriguing phenomenon, we propose a concept alignment method based on how units respond to the replicated text. We conduct analyses with different architectures on multiple datasets for classification and translation tasks and provide new insights into how deep models understand natural language.