Inducing Meaningful Units from Character Sequences with Dynamic Capacity Slot Attention

TL;DR

This paper introduces a novel unsupervised model that learns meaningful units from character sequences by discovering continuous representations, extending object discovery architectures to language processing.

Contribution

It presents a Dynamic Capacity Slot Attention model that uncovers abstract units in character sequences without segmentation, applicable across multiple languages.

Findings

Model successfully discovers units similar to known linguistic units

Representations capture meaningful information at higher abstraction levels

Effective across different languages

Abstract

Characters do not convey meaning, but sequences of characters do. We propose an unsupervised distributional method to learn the abstract meaningful units in a sequence of characters. Rather than segmenting the sequence, our Dynamic Capacity Slot Attention model discovers continuous representations of the objects in the sequence, extending an architecture for object discovery in images. We train our model on different languages and evaluate the quality of the obtained representations with forward and reverse probing classifiers. These experiments show that our model succeeds in discovering units which are similar to those proposed previously in form, content and level of abstraction, and which show promise for capturing meaningful information at a higher level of abstraction.