How much should you ask? On the question structure in QA systems

TL;DR

This paper investigates which parts of questions are essential for QA systems to produce correct answers, revealing that natural language and grammar are often disregarded, and demonstrates the use of LIME for explainability in QA models.

Contribution

It introduces the application of LIME to explain QA model predictions and shows that minimal question parts can suffice for accurate answers, challenging assumptions about natural language importance.

Findings

QA models can answer correctly with minimal question words

LIME effectively explains QA model predictions

Natural language structure is less critical than expected

Abstract

Datasets that boosted state-of-the-art solutions for Question Answering (QA) systems prove that it is possible to ask questions in natural language manner. However, users are still used to query-like systems where they type in keywords to search for answer. In this study we validate which parts of questions are essential for obtaining valid answer. In order to conclude that, we take advantage of LIME - a framework that explains prediction by local approximation. We find that grammar and natural language is disregarded by QA. State-of-the-art model can answer properly even if 'asked' only with a few words with high coefficients calculated with LIME. According to our knowledge, it is the first time that QA model is being explained by LIME.