Revisiting Few-shot Relation Classification: Evaluation Data and Classification Schemes

TL;DR

This paper critically examines the data and evaluation methods in Few-Shot Relation Classification, introduces a more realistic benchmark, and proposes a novel classification scheme that improves handling of the none-of-the-above category.

Contribution

It presents a new realistic benchmark for FSL RC derived from existing datasets and introduces a novel classification scheme with learned vectors for the NOTA category.

Findings

State-of-the-art models perform poorly on the new benchmark.

The proposed classification scheme with learned NOTA vectors improves FSL RC performance.

Analysis of embedding constraints informs better classification strategies.

Abstract

We explore Few-Shot Learning (FSL) for Relation Classification (RC). Focusing on the realistic scenario of FSL, in which a test instance might not belong to any of the target categories (none-of-the-above, aka NOTA), we first revisit the recent popular dataset structure for FSL, pointing out its unrealistic data distribution. To remedy this, we propose a novel methodology for deriving more realistic few-shot test data from available datasets for supervised RC, and apply it to the TACRED dataset. This yields a new challenging benchmark for FSL RC, on which state of the art models show poor performance. Next, we analyze classification schemes within the popular embedding-based nearest-neighbor approach for FSL, with respect to constraints they impose on the embedding space. Triggered by this analysis we propose a novel classification scheme, in which the NOTA category is represented as learned vectors, shown empirically to be an appealing option for FSL.