Adversarial Question Answering Robustness: A Multi-Level Error Analysis and Mitigation Study

TL;DR

This study systematically analyzes adversarial vulnerabilities in transformer-based QA systems, identifies key failure modes, and demonstrates that targeted mitigation strategies, especially entity-aware contrastive learning, significantly improve robustness without sacrificing accuracy.

Contribution

It introduces a comprehensive multi-level error analysis framework and proposes NER-guided contrastive learning as an effective mitigation strategy for adversarial QA robustness.

Findings

Scaling models improves robustness and accuracy.

Optimal adversarial fine-tuning ratio is 80% clean, 20% adversarial.

Entity-aware contrastive learning reduces the adversarial gap to near parity.

Abstract

Question answering (QA) systems achieve impressive performance on standard benchmarks like SQuAD, but remain vulnerable to adversarial examples. This project investigates the adversarial robustness of transformer models on the AddSent adversarial dataset through systematic experimentation across model scales and targeted mitigation strategies. We perform comprehensive multi-level error analysis using five complementary categorization schemes, identifying negation confusion and entity substitution as the primary failure modes. Through systematic evaluation of adversarial fine-tuning ratios, we identify 80% clean + 20% adversarial data as optimal. Data augmentation experiments reveal a capacity bottleneck in small models. Scaling from ELECTRA-small (14M parameters) to ELECTRA-base (110M parameters) eliminates the robustness-accuracy trade-off, achieving substantial improvements on both clean and adversarial data. We implement three targeted mitigation strategies, with Entity-Aware contrastive learning achieving best performance: 89.89% AddSent Exact Match (EM) and 90.73% SQuAD EM, representing 94.9% closure of the adversarial gap. To our knowledge, this is the first work integrating comprehensive linguistic error analysis with Named Entity Recognition (NER)-guided contrastive learning for adversarial QA, demonstrating that targeted mitigation can achieve near-parity between clean and adversarial performance.