Real-Time Visual Feedback to Guide Benchmark Creation: A Human-and-Metric-in-the-Loop Workflow

TL;DR

This paper introduces VAIDA, a real-time visual feedback system for guiding crowdworkers in creating more robust NLP benchmarks, reducing artifacts and improving sample quality through human-and-metric-in-the-loop workflows.

Contribution

VAIDA is a novel, domain-agnostic paradigm that enhances benchmark creation by providing real-time feedback, decreasing artifacts, and improving sample robustness against models.

Findings

VAIDA reduces artifacts in benchmark samples by 45.8%.

It decreases crowdworker effort and frustration.

Created samples are more adversarial, lowering model performance.

Abstract

Recent research has shown that language models exploit `artifacts' in benchmarks to solve tasks, rather than truly learning them, leading to inflated model performance. In pursuit of creating better benchmarks, we propose VAIDA, a novel benchmark creation paradigm for NLP, that focuses on guiding crowdworkers, an under-explored facet of addressing benchmark idiosyncrasies. VAIDA facilitates sample correction by providing realtime visual feedback and recommendations to improve sample quality. Our approach is domain, model, task, and metric agnostic, and constitutes a paradigm shift for robust, validated, and dynamic benchmark creation via human-and-metric-in-the-loop workflows. We evaluate via expert review and a user study with NASA TLX. We find that VAIDA decreases effort, frustration, mental, and temporal demands of crowdworkers and analysts, simultaneously increasing the performance of both user groups with a 45.8% decrease in the level of artifacts in created samples. As a by product of our user study, we observe that created samples are adversarial across models, leading to decreases of 31.3% (BERT), 22.5% (RoBERTa), 14.98% (GPT-3 fewshot) in performance.