Discovering influential text using convolutional neural networks

TL;DR

This paper introduces a convolutional neural network-based method for identifying influential text phrases that affect human evaluations, enhancing interpretability and discovery of causal text features in social science experiments.

Contribution

The paper presents a novel NLP interpretability technique that discovers clusters of predictive text phrases, improving over existing methods in identifying causal text effects.

Findings

The method accurately detects known causal phrases.

It discovers diverse text treatments beyond benchmark methods.

It predicts outcomes as well or better than existing approaches.

Abstract

Experimental methods for estimating the impacts of text on human evaluation have been widely used in the social sciences. However, researchers in experimental settings are usually limited to testing a small number of pre-specified text treatments. While efforts to mine unstructured texts for features that causally affect outcomes have been ongoing in recent years, these models have primarily focused on the topics or specific words of text, which may not always be the mechanism of the effect. We connect these efforts with NLP interpretability techniques and present a method for flexibly discovering clusters of similar text phrases that are predictive of human reactions to texts using convolutional neural networks. When used in an experimental setting, this method can identify text treatments and their effects under certain assumptions. We apply the method to two datasets. The first enables direct validation of the model's ability to detect phrases known to cause the outcome. The second demonstrates its ability to flexibly discover text treatments with varying textual structures. In both cases, the model learns a greater variety of text treatments compared to benchmark methods, and these text features quantitatively meet or exceed the ability of benchmark methods to predict the outcome.