Do Models Learn the Directionality of Relations? A New Evaluation: Relation Direction Recognition

TL;DR

This paper introduces the Relation Direction Recognition (RDR) task to evaluate whether deep models like BERT understand relation directionality, revealing gaps despite similar traditional performance metrics.

Contribution

It proposes the RDR evaluation task and metrics, and assesses state-of-the-art models' ability to recognize relation directionality, highlighting areas for improvement.

Findings

Models show significant gaps in recognizing relation directionality.

Traditional metrics do not reflect directionality understanding.

Suggestions for improving directionality recognition are discussed.

Abstract

Deep neural networks such as BERT have made great progress in relation classification. Although they can achieve good performance, it is still a question of concern whether these models recognize the directionality of relations, especially when they may lack interpretability. To explore the question, a novel evaluation task, called Relation Direction Recognition (RDR), is proposed to explore whether models learn the directionality of relations. Three metrics for RDR are introduced to measure the degree to which models recognize the directionality of relations. Several state-of-the-art models are evaluated on RDR. Experimental results on a real-world dataset indicate that there are clear gaps among them in recognizing the directionality of relations, even though these models obtain similar performance in the traditional metric (e.g. Macro-F1). Finally, some suggestions are discussed to enhance models to recognize the directionality of relations from the perspective of model design or training.