Mashee at SemEval-2024 Task 8: The Impact of Samples Quality on the Performance of In-Context Learning for Machine Text Classification

TL;DR

This paper investigates how the quality of samples affects in-context learning performance in text classification, showing that selecting high-quality samples improves evaluation metrics in few-shot scenarios.

Contribution

It introduces a method using the chi-square test to select high-quality samples, enhancing in-context learning effectiveness in few-shot text classification.

Findings

High-quality samples improve all evaluated metrics.

Sample quality significantly impacts ICL performance.

Chi-square test effectively identifies valuable samples.

Abstract

Within few-shot learning, in-context learning (ICL) has become a potential method for leveraging contextual information to improve model performance on small amounts of data or in resource-constrained environments where training models on large datasets is prohibitive. However, the quality of the selected sample in a few shots severely limits the usefulness of ICL. The primary goal of this paper is to enhance the performance of evaluation metrics for in-context learning by selecting high-quality samples in few-shot learning scenarios. We employ the chi-square test to identify high-quality samples and compare the results with those obtained using low-quality samples. Our findings demonstrate that utilizing high-quality samples leads to improved performance with respect to all evaluated metrics.