FewRel 2.0: Towards More Challenging Few-Shot Relation Classification

TL;DR

FewRel 2.0 introduces a more challenging dataset for few-shot relation classification, emphasizing domain adaptation and NOTA detection, revealing current models' limitations and the need for further research.

Contribution

The paper presents FewRel 2.0, a new dataset with domain shift and NOTA relations, highlighting the challenges in current few-shot relation classification models.

Findings

Models struggle with domain adaptation and NOTA detection.

Existing techniques are insufficient for these challenges.

The dataset and baselines are publicly available.

Abstract

We present FewRel 2.0, a more challenging task to investigate two aspects of few-shot relation classification models: (1) Can they adapt to a new domain with only a handful of instances? (2) Can they detect none-of-the-above (NOTA) relations? To construct FewRel 2.0, we build upon the FewRel dataset (Han et al., 2018) by adding a new test set in a quite different domain, and a NOTA relation choice. With the new dataset and extensive experimental analysis, we found (1) that the state-of-the-art few-shot relation classification models struggle on these two aspects, and (2) that the commonly-used techniques for domain adaptation and NOTA detection still cannot handle the two challenges well. Our research calls for more attention and further efforts to these two real-world issues. All details and resources about the dataset and baselines are released at https: //github.com/thunlp/fewrel.