Text analysis and deep learning: A network approach

TL;DR

This paper introduces a novel unsupervised method combining transformer models with network analysis to extract semantic networks from text, enabling insights into language use and model behavior over time.

Contribution

It presents the first unsupervised approach to derive semantic networks directly from deep language models, reducing subjective choices in representation.

Findings

Semantic ties reflect discourse semantics over time

Clusters of semantic and syntactic relations are identified

Method can inform analysis of deep learning model behavior

Abstract

Much information available to applied researchers is contained within written language or spoken text. Deep language models such as BERT have achieved unprecedented success in many applications of computational linguistics. However, much less is known about how these models can be used to analyze existing text. We propose a novel method that combines transformer models with network analysis to form a self-referential representation of language use within a corpus of interest. Our approach produces linguistic relations strongly consistent with the underlying model as well as mathematically well-defined operations on them, while reducing the amount of discretionary choices of representation and distance measures. It represents, to the best of our knowledge, the first unsupervised method to extract semantic networks directly from deep language models. We illustrate our approach in a semantic analysis of the term "founder". Using the entire corpus of Harvard Business Review from 1980 to 2020, we find that ties in our network track the semantics of discourse over time, and across contexts, identifying and relating clusters of semantic and syntactic relations. Finally, we discuss how this method can also complement and inform analyses of the behavior of deep learning models.