WatChat: Explaining perplexing programs by debugging mental models

TL;DR

WatChat is a framework that explains unexpected program behaviors by identifying and debugging misconceptions in the user's mental model of the language or API, using formal semantics and program synthesis.

Contribution

The paper introduces WatChat, a novel approach that infers and corrects misconceptions in mental models to generate explanations for program behavior.

Findings

WatChat effectively explains surprising behaviors in JavaScript and Git.

It produces explanations similar to human reasoning, outperforming language models.

The framework demonstrates practical utility in real-world programming domains.

Abstract

Often, a good explanation for a program's unexpected behavior is a bug in the programmer's code. But sometimes, an even better explanation is a bug in the programmer's mental model of the language or API they are using. Instead of merely debugging our current code ("giving the programmer a fish"), what if our tools could directly debug our mental models ("teaching the programmer to fish")? In this paper, we apply recent ideas from computational cognitive science to offer a principled framework for doing exactly that. Given a "why?" question about a program, we automatically infer potential misconceptions about the language/API that might cause the user to be surprised by the program's behavior -- and then analyze those misconceptions to provide explanations of the program's behavior. Our key idea is to formally represent misconceptions as counterfactual (erroneous) semantics for the language/API, which can be inferred and debugged using program synthesis techniques. We demonstrate our framework, WatChat, by building systems for explanation in two domains: JavaScript type coercion, and the Git version control system. We evaluate WatChatJS and WatChatGit by comparing their outputs to experimentally-collected human-written explanations in these two domains: we show that WatChat's explanations exhibit key features of human-written explanation, unlike those of a state-of-the-art language model.