Unsupervised Calibration through Prior Adaptation for Text Classification using Large Language Models

TL;DR

This paper introduces an unsupervised method to calibrate large language models for text classification by adapting prior class distributions using minimal in-domain data, improving performance without labeled samples.

Contribution

The work presents a novel prior adaptation technique that calibrates LLMs for classification tasks without requiring labeled data, outperforming previous calibration methods.

Findings

Outperforms unadapted models across various training shot scenarios

Effective calibration achieved with few in-domain samples

Surpasses previous calibration approaches without adaptation data

Abstract

A wide variety of natural language tasks are currently being addressed with large-scale language models (LLMs). These models are usually trained with a very large amount of unsupervised text data and adapted to perform a downstream natural language task using methods like fine-tuning, calibration or in-context learning. In this work, we propose an approach to adapt the prior class distribution to perform text classification tasks without the need for labelled samples and only few in-domain sample queries. The proposed approach treats the LLM as a black box, adding a stage where the model posteriors are calibrated to the task. Results show that these methods outperform the un-adapted model for different number of training shots in the prompt and a previous approach were calibration is performed without using any adaptation data.