Striking a Balance: Alleviating Inconsistency in Pre-trained Models for Symmetric Classification Tasks

TL;DR

This paper addresses the inconsistency issue in pre-trained models for symmetric classification tasks in NLP by applying a consistency loss, improving prediction stability without sacrificing accuracy across multiple datasets.

Contribution

It introduces a consistency loss function specifically designed to reduce prediction inconsistency in symmetric classification tasks during fine-tuning.

Findings

Significant improvement in prediction consistency on paraphrase detection datasets.

No notable decrease in classification accuracy across tested datasets.

Effective approach for symmetric tasks without compromising performance.

Abstract

While fine-tuning pre-trained models for downstream classification is the conventional paradigm in NLP, often task-specific nuances may not get captured in the resultant models. Specifically, for tasks that take two inputs and require the output to be invariant of the order of the inputs, inconsistency is often observed in the predicted labels or confidence scores. We highlight this model shortcoming and apply a consistency loss function to alleviate inconsistency in symmetric classification. Our results show an improved consistency in predictions for three paraphrase detection datasets without a significant drop in the accuracy scores. We examine the classification performance of six datasets (both symmetric and non-symmetric) to showcase the strengths and limitations of our approach.