Does the Objective Matter? Comparing Training Objectives for Pronoun Resolution

TL;DR

This paper compares four training objectives for pronoun resolution using pre-trained language models, revealing their strengths and instabilities across in-domain and out-of-domain settings.

Contribution

It provides a fair comparison of different training objectives for pronoun resolution, highlighting their performance differences and stability issues.

Findings

Sequence ranking performs best in-domain.

Semantic similarity performs best out-of-domain.

Sequence ranking shows seed-wise instability.

Abstract

Hard cases of pronoun resolution have been used as a long-standing benchmark for commonsense reasoning. In the recent literature, pre-trained language models have been used to obtain state-of-the-art results on pronoun resolution. Overall, four categories of training and evaluation objectives have been introduced. The variety of training datasets and pre-trained language models used in these works makes it unclear whether the choice of training objective is critical. In this work, we make a fair comparison of the performance and seed-wise stability of four models that represent the four categories of objectives. Our experiments show that the objective of sequence ranking performs the best in-domain, while the objective of semantic similarity between candidates and pronoun performs the best out-of-domain. We also observe a seed-wise instability of the model using sequence ranking, which is not the case when the other objectives are used.