Betti numbers of attention graphs is all you really need

TL;DR

This paper demonstrates that topological features, specifically Betti numbers, of attention graphs in BERT can be used for text classification, achieving results comparable to traditional methods.

Contribution

It introduces the novel application of persistent topological features to analyze attention graphs in neural networks for NLP tasks.

Findings

Betti numbers effectively classify text data.

Topological features match conventional classification accuracy.

First topological analysis of attention-based neural networks.

Abstract

We apply methods of topological analysis to the attention graphs, calculated on the attention heads of the BERT model ( arXiv:1810.04805v2 ). Our research shows that the classifier built upon basic persistent topological features (namely, Betti numbers) of the trained neural network can achieve classification results on par with the conventional classification method. We show the relevance of such topological text representation on three text classification benchmarks. For the best of our knowledge, it is the first attempt to analyze the topology of an attention-based neural network, widely used for Natural Language Processing.