Orthogonal Attention: A Cloze-Style Approach to Negation Scope Resolution

TL;DR

This paper introduces Orthogonal Attention, a novel Cloze-Style attention mechanism, applied to Negation Scope Resolution, outperforming existing transformer-based models across multiple datasets.

Contribution

The paper proposes a new Orthogonal Attention mechanism and demonstrates its effectiveness in improving Negation Scope Resolution over standard transformer finetuning.

Findings

Outperforms state-of-the-art XLNet models on all datasets

Introduces four variants of Orthogonal Attention

Achieves best results on BioScope, SFU, and Sherlock datasets

Abstract

Negation Scope Resolution is an extensively researched problem, which is used to locate the words affected by a negation cue in a sentence. Recent works have shown that simply finetuning transformer-based architectures yield state-of-the-art results on this task. In this work, we look at Negation Scope Resolution as a Cloze-Style task, with the sentence as the Context and the cue words as the Query. We also introduce a novel Cloze-Style Attention mechanism called Orthogonal Attention, which is inspired by Self Attention. First, we propose a framework for developing Orthogonal Attention variants, and then propose 4 Orthogonal Attention variants: OA-C, OA-CA, OA-EM, and OA-EMB. Using these Orthogonal Attention layers on top of an XLNet backbone, we outperform the finetuned XLNet state-of-the-art for Negation Scope Resolution, achieving the best results to date on all 4 datasets we experiment with: BioScope Abstracts, BioScope Full Papers, SFU Review Corpus and the *sem 2012 Dataset (Sherlock).