Interactively Diagnosing Errors in a Semantic Parser

TL;DR

This paper presents an interactive debugging system for a semantic parser that uses model-based diagnosis to identify and localize errors, aiming to reduce the effort needed for maintaining hand-curated language systems.

Contribution

It introduces a novel approach to semantic parser debugging by framing error diagnosis as a model-based diagnosis problem, with initial system implementation and evaluation.

Findings

Successfully diagnosed semantic errors on synthetic examples

Demonstrated the feasibility of model-based diagnosis for error localization

Identified design challenges and future directions for interactive debugging

Abstract

Hand-curated natural language systems provide an inspectable, correctable alternative to language systems based on machine learning, but maintaining them requires considerable effort and expertise. Interactive Natural Language Debugging (INLD) aims to lessen this burden by casting debugging as a reasoning problem, asking the user a series of questions to diagnose and correct errors in the system's knowledge. In this paper, we present work in progress on an interactive error diagnosis system for the CNLU semantic parser. We show how the first two stages of the INLD pipeline (symptom identification and error localization) can be cast as a model-based diagnosis problem, demonstrate our system's ability to diagnose semantic errors on synthetic examples, and discuss design challenges and frontiers for future work.