Classifying Scientific Publications with BERT -- Is Self-Attention a Feature Selection Method?

TL;DR

This paper explores how BERT's self-attention mechanism highlights domain-specific words in scientific articles and compares it to traditional feature selection methods, revealing that attention focuses on relevant terms but classification relies on contextualized outputs.

Contribution

The study characterizes self-attention as a feature selection method in scientific text classification and compares its effectiveness with conventional feature selection techniques.

Findings

Self-attention focuses on domain-related words in scientific articles.

Traditional feature selection outperforms self-attention in classification accuracy.

Attended words are more related to research fields according to ConceptNet.

Abstract

We investigate the self-attention mechanism of BERT in a fine-tuning scenario for the classification of scientific articles over a taxonomy of research disciplines. We observe how self-attention focuses on words that are highly related to the domain of the article. Particularly, a small subset of vocabulary words tends to receive most of the attention. We compare and evaluate the subset of the most attended words with feature selection methods normally used for text classification in order to characterize self-attention as a possible feature selection approach. Using ConceptNet as ground truth, we also find that attended words are more related to the research fields of the articles. However, conventional feature selection methods are still a better option to learn classifiers from scratch. This result suggests that, while self-attention identifies domain-relevant terms, the discriminatory information in BERT is encoded in the contextualized outputs and the classification layer. It also raises the question whether injecting feature selection methods in the self-attention mechanism could further optimize single sequence classification using transformers.