Pareto Probing: Trading Off Accuracy for Complexity

TL;DR

This paper introduces Pareto hypervolume as a metric to evaluate the trade-off between probe complexity and performance in linguistic probing of contextual word representations, revealing limitations of simple tasks and advocating for more challenging probing methods like dependency parsing.

Contribution

It proposes Pareto hypervolume as a new evaluation metric for probing complexity and performance, and advocates for using full dependency parsing as a more effective probing task.

Findings

Probes often do not align with expectations regarding information encoding.

Simple probing tasks are inadequate for evaluating linguistic structure.

Dependency parsing reveals significant syntactic knowledge gaps.

Abstract

The question of how to probe contextual word representations for linguistic structure in a way that is both principled and useful has seen significant attention recently in the NLP literature. In our contribution to this discussion, we argue for a probe metric that reflects the fundamental trade-off between probe complexity and performance: the Pareto hypervolume. To measure complexity, we present a number of parametric and non-parametric metrics. Our experiments using Pareto hypervolume as an evaluation metric show that probes often do not conform to our expectations -- e.g., why should the non-contextual fastText representations encode more morpho-syntactic information than the contextual BERT representations? These results suggest that common, simplistic probing tasks, such as part-of-speech labeling and dependency arc labeling, are inadequate to evaluate the linguistic structure encoded in contextual word representations. This leads us to propose full dependency parsing as a probing task. In support of our suggestion that harder probing tasks are necessary, our experiments with dependency parsing reveal a wide gap in syntactic knowledge between contextual and non-contextual representations.