Few-Shot Anaphora Resolution in Scientific Protocols via Mixtures of In-Context Experts

TL;DR

This paper introduces MICE, a novel method that combines multiple in-context experts to improve few-shot anaphora resolution in scientific protocols, achieving significant performance gains without large annotated datasets.

Contribution

MICE is the first approach to demonstrate effective in-context learning for few-shot anaphora resolution in scientific protocols, leveraging mixtures of experts for improved accuracy.

Findings

30% increase in F1 score over baseline

Effective training of compact student models

First experimental validation of in-context learning for this task

Abstract

Anaphora resolution is an important task for information extraction across a range of languages, text genres, and domains, motivating the need for methods that do not require large annotated datasets. In-context learning has emerged as a promising approach, yet there are a number of challenges in applying in-context learning to resolve anaphora. For example, encoding a single in-context demonstration that consists of: an anaphor, a paragraph-length context, and a list of corresponding antecedents, requires conditioning a language model on a long sequence of tokens, limiting the number of demonstrations per prompt. In this paper, we present MICE (Mixtures of In-Context Experts), which we demonstrate is effective for few-shot anaphora resolution in scientific protocols (Tamari et al., 2021). Given only a handful of training examples, MICE combines the predictions of hundreds of in-context experts, yielding a 30% increase in F1 score over a competitive prompt retrieval baseline. Furthermore, we show MICE can be used to train compact student models without sacrificing performance. As far as we are aware, this is the first work to present experimental results demonstrating the effectiveness of in-context learning on the task of few-shot anaphora resolution in scientific protocols.