$\textit{GeoHard}$: Towards Measuring Class-wise Hardness through Modelling Class Semantics

TL;DR

This paper introduces GeoHard, a novel method for measuring class-wise hardness in NLP tasks by modeling class geometry in semantic space, revealing insights into data difficulty and improving model training.

Contribution

It pioneers the concept of class-wise hardness, proposes GeoHard for its measurement, and demonstrates its effectiveness and generality across multiple datasets and models.

Findings

GeoHard outperforms previous instance-level metrics by over 59% in correlation.

Class-wise hardness is consistent across datasets, models, and human judgment.

Understanding class-wise hardness aids in improving task learning.

Abstract

Recent advances in measuring hardness-wise properties of data guide language models in sample selection within low-resource scenarios. However, class-specific properties are overlooked for task setup and learning. How will these properties influence model learning and is it generalizable across datasets? To answer this question, this work formally initiates the concept of $\textit{class-wise hardness}$. Experiments across eight natural language understanding (NLU) datasets demonstrate a consistent hardness distribution across learning paradigms, models, and human judgment. Subsequent experiments unveil a notable challenge in measuring such class-wise hardness with instance-level metrics in previous works. To address this, we propose $\textit{GeoHard}$ for class-wise hardness measurement by modeling class geometry in the semantic embedding space. $\textit{GeoHard}$ surpasses instance-level metrics by over 59 percent on $\textit{Pearson}$'s correlation on measuring class-wise hardness. Our analysis theoretically and empirically underscores the generality of $\textit{GeoHard}$ as a fresh perspective on data diagnosis. Additionally, we showcase how understanding class-wise hardness can practically aid in improving task learning.