Syntax-informed Question Answering with Heterogeneous Graph Transformer

TL;DR

This paper introduces a method to enhance pre-trained transformer models for question answering by incorporating explicit linguistic knowledge through a heterogeneous graph transformer, improving performance without retraining from scratch.

Contribution

It presents a novel approach to integrate symbolic linguistic information into pre-trained models using a heterogeneous graph transformer, extending their capabilities for question answering.

Findings

The approach is competitive with baseline models like BERT on SQuAD.

Incorporating syntactic structures improves question answering performance.

The method is extensible to other linguistic features like semantics and pragmatics.

Abstract

Large neural language models are steadily contributing state-of-the-art performance to question answering and other natural language and information processing tasks. These models are expensive to train. We propose to evaluate whether such pre-trained models can benefit from the addition of explicit linguistics information without requiring retraining from scratch. We present a linguistics-informed question answering approach that extends and fine-tunes a pre-trained transformer-based neural language model with symbolic knowledge encoded with a heterogeneous graph transformer. We illustrate the approach by the addition of syntactic information in the form of dependency and constituency graphic structures connecting tokens and virtual vertices. A comparative empirical performance evaluation with BERT as its baseline and with Stanford Question Answering Dataset demonstrates the competitiveness of the proposed approach. We argue, in conclusion and in the light of further results of preliminary experiments, that the approach is extensible to further linguistics information including semantics and pragmatics.