A Cross-Task Analysis of Text Span Representations

TL;DR

This paper systematically evaluates various span representation methods across multiple NLP tasks, revealing that optimal representations vary by task and are influenced by whether the encoder is fixed or fine-tuned.

Contribution

It provides a comprehensive empirical comparison of six span representation methods across six tasks, including two newly introduced tasks, highlighting task-specific preferences.

Findings

Simple span representations are generally reliable.

Optimal span representation varies by task and facet.

Choice of span representation impacts fixed encoders more.

Abstract

Many natural language processing (NLP) tasks involve reasoning with textual spans, including question answering, entity recognition, and coreference resolution. While extensive research has focused on functional architectures for representing words and sentences, there is less work on representing arbitrary spans of text within sentences. In this paper, we conduct a comprehensive empirical evaluation of six span representation methods using eight pretrained language representation models across six tasks, including two tasks that we introduce. We find that, although some simple span representations are fairly reliable across tasks, in general the optimal span representation varies by task, and can also vary within different facets of individual tasks. We also find that the choice of span representation has a bigger impact with a fixed pretrained encoder than with a fine-tuned encoder.