Fast Few-shot Debugging for NLU Test Suites
Christopher Malon, Kai Li, Erik Kruus

TL;DR
This paper introduces a fast few-shot debugging method for transformer-based NLU models that improves accuracy on specific phenomena with minimal impact on overall performance, outperforming existing quick-fix approaches.
Contribution
The paper proposes a novel fast debugging technique that samples in-danger examples from training data, achieving better accuracy trade-offs than existing methods.
Findings
Superior original accuracy at comparable debugging accuracy
Faster debugging process than full retraining
Effective correction of specific phenomena in NLU models
Abstract
We study few-shot debugging of transformer based natural language understanding models, using recently popularized test suites to not just diagnose but correct a problem. Given a few debugging examples of a certain phenomenon, and a held-out test set of the same phenomenon, we aim to maximize accuracy on the phenomenon at a minimal cost of accuracy on the original test set. We examine several methods that are faster than full epoch retraining. We introduce a new fast method, which samples a few in-danger examples from the original training set. Compared to fast methods using parameter distance constraints or Kullback-Leibler divergence, we achieve superior original accuracy for comparable debugging accuracy.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Speech Recognition and Synthesis
