Improving Neural Model Performance through Natural Language Feedback on Their Explanations

TL;DR

This paper presents MERCURIE, an interactive system that improves explainable NLP models by incorporating natural language feedback, reducing inconsistencies and boosting reasoning accuracy across multiple domains.

Contribution

Introduces MERCURIE, a novel system enabling natural language correction of explanations, significantly reducing errors and improving reasoning accuracy in explainable NLP models.

Findings

40% fewer inconsistencies in generated graphs

1.2-point accuracy improvement on defeasible reasoning

Released a dataset of 450k reasoning graphs

Abstract

A class of explainable NLP models for reasoning tasks support their decisions by generating free-form or structured explanations, but what happens when these supporting structures contain errors? Our goal is to allow users to interactively correct explanation structures through natural language feedback. We introduce MERCURIE - an interactive system that refines its explanations for a given reasoning task by getting human feedback in natural language. Our approach generates graphs that have 40% fewer inconsistencies as compared with the off-the-shelf system. Further, simply appending the corrected explanation structures to the output leads to a gain of 1.2 points on accuracy on defeasible reasoning across all three domains. We release a dataset of over 450k graphs for defeasible reasoning generated by our system at https://tinyurl.com/mercurie .