Tackling Domain-Specific Winograd Schemas with Knowledge-Based Reasoning and Machine Learning

TL;DR

This paper addresses the Winograd Schema Challenge by defining a domain-specific approach using keywords, developing a knowledge-based reasoning method, and combining it with BERT through ensemble techniques to improve reasoning accuracy.

Contribution

It introduces a keyword-based domain restriction, a semantic role reasoning method, and an ensemble approach integrating knowledge reasoning with BERT for better WSC performance.

Findings

Ensemble method outperforms individual approaches.

Keyword domain restriction improves reasoning focus.

Modified robust accuracy provides more reliable evaluation.

Abstract

The Winograd Schema Challenge (WSC) is a common-sense reasoning task that requires background knowledge. In this paper, we contribute to tackling WSC in four ways. Firstly, we suggest a keyword method to define a restricted domain where distinctive high-level semantic patterns can be found. A thanking domain was defined by key-words, and the data set in this domain is used in our experiments. Secondly, we develop a high-level knowledge-based reasoning method using semantic roles which is based on the method of Sharma [2019]. Thirdly, we propose an ensemble method to combine knowledge-based reasoning and machine learning which shows the best performance in our experiments. As a machine learning method, we used Bidirectional Encoder Representations from Transformers (BERT) [Kocijan et al., 2019]. Lastly, in terms of evaluation, we suggest a "robust" accuracy measurement by modifying that of Trichelair et al. [2018]. As with their switching method, we evaluate a model by considering its performance on trivial variants of each sentence in the test set.