IDS at SemEval-2020 Task 10: Does Pre-trained Language Model Know What to Emphasize?

TL;DR

This paper introduces a zero-shot method using pre-trained language models' self-attention to identify words for emphasis in text, outperforming TF-IDF baselines and revealing specialized attention heads.

Contribution

The study demonstrates that PLMs can be used to select important words for emphasis without training, and identifies attention heads that specialize in this task.

Findings

Zero-shot emphasis detection outperforms TF-IDF baseline.

Certain attention heads are specialized for emphasis recognition.

PLMs can recognize important words in sentences.

Abstract

We propose a novel method that enables us to determine words that deserve to be emphasized from written text in visual media, relying only on the information from the self-attention distributions of pre-trained language models (PLMs). With extensive experiments and analyses, we show that 1) our zero-shot approach is superior to a reasonable baseline that adopts TF-IDF and that 2) there exist several attention heads in PLMs specialized for emphasis selection, confirming that PLMs are capable of recognizing important words in sentences.