Parameter-Efficient Neural Question Answering Models via Graph-Enriched Document Representations

TL;DR

This paper introduces a graph convolutional network-based question answering model that achieves competitive performance with significantly fewer parameters, challenging the necessity of large language models for multi-hop QA tasks.

Contribution

The paper demonstrates that graph-enriched document representations via GCNs can match or surpass state-of-the-art QA performance with minimal resources, reducing reliance on large pretrained models.

Findings

GCN-enriched representations improve QA results on HotPotQA

The model performs comparably to SOTA with less than 5% of the parameters

Simple topology GCNs can be effective for multi-hop QA

Abstract

As the computational footprint of modern NLP systems grows, it becomes increasingly important to arrive at more efficient models. We show that by employing graph convolutional document representation, we can arrive at a question answering system that performs comparably to, and in some cases exceeds the SOTA solutions, while using less than 5\% of their resources in terms of trainable parameters. As it currently stands, a major issue in applying GCNs to NLP is document representation. In this paper, we show that a GCN enriched document representation greatly improves the results seen in HotPotQA, even when using a trivial topology. Our model (gQA), performs admirably when compared to the current SOTA, and requires little to no preprocessing. In Shao et al. 2020, the authors suggest that graph networks are not necessary for good performance in multi-hop QA. In this paper, we suggest that large language models are not necessary for good performance by showing a na\"{i}ve implementation of a GCN performs comparably to SoTA models based on pretrained language models.