READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises

TL;DR

READIN is a comprehensive Chinese benchmark with realistic input noises from speech and typing errors, designed to evaluate and improve the robustness of NLP models in real-world scenarios.

Contribution

It introduces the first large-scale Chinese benchmark with diverse, realistic input noises from user-generated data, covering multiple tasks and input methods.

Findings

Models show significant performance drops on noisy data.

Robust training methods have limited effectiveness against realistic noises.

The benchmark highlights the need for improved robustness techniques.

Abstract

For many real-world applications, the user-generated inputs usually contain various noises due to speech recognition errors caused by linguistic variations1 or typographical errors (typos). Thus, it is crucial to test model performance on data with realistic input noises to ensure robustness and fairness. However, little study has been done to construct such benchmarks for Chinese, where various language-specific input noises happen in the real world. In order to fill this important gap, we construct READIN: a Chinese multi-task benchmark with REalistic And Diverse Input Noises. READIN contains four diverse tasks and requests annotators to re-enter the original test data with two commonly used Chinese input methods: Pinyin input and speech input. We designed our annotation pipeline to maximize diversity, for example by instructing the annotators to use diverse input method editors (IMEs) for keyboard noises and recruiting speakers from diverse dialectical groups for speech noises. We experiment with a series of strong pretrained language models as well as robust training methods, we find that these models often suffer significant performance drops on READIN even with robustness methods like data augmentation. As the first large-scale attempt in creating a benchmark with noises geared towards user-generated inputs, we believe that READIN serves as an important complement to existing Chinese NLP benchmarks. The source code and dataset can be obtained from https://github.com/thunlp/READIN.