Understanding and Tackling Label Errors in Individual-Level Nature Language Understanding

TL;DR

This paper highlights the importance of considering individual-level factors in natural language understanding tasks, introduces a new annotation guideline, and demonstrates improved accuracy with large language models on re-annotated datasets.

Contribution

It proposes a novel annotation guideline incorporating individual factors for more accurate dataset creation in individual-level NLU tasks.

Findings

Error rates in datasets were as high as 31.7% and 23.3%.

Large language models achieved over 87% accuracy on re-annotated datasets.

Adding individual factors improves model performance and annotation quality.

Abstract

Natural language understanding (NLU) is a task that enables machines to understand human language. Some tasks, such as stance detection and sentiment analysis, are closely related to individual subjective perspectives, thus termed individual-level NLU. Previously, these tasks are often simplified to text-level NLU tasks, ignoring individual factors. This not only makes inference difficult and unexplainable but often results in a large number of label errors when creating datasets. To address the above limitations, we propose a new NLU annotation guideline based on individual-level factors. Specifically, we incorporate other posts by the same individual and then annotate individual subjective perspectives after considering all individual posts. We use this guideline to expand and re-annotate the stance detection and topic-based sentiment analysis datasets. We find that error rates in the samples were as high as 31.7\% and 23.3\%. We further use large language models to conduct experiments on the re-annotation datasets and find that the large language models perform well on both datasets after adding individual factors. Both GPT-4o and Llama3-70B can achieve an accuracy greater than 87\% on the re-annotation datasets. We also verify the effectiveness of individual factors through ablation studies. We call on future researchers to add individual factors when creating such datasets. Our re-annotation dataset can be found at https://github.com/24yearsoldstudent/Individual-NLU