Fill In The Gaps: Model Calibration and Generalization with Synthetic Data

TL;DR

This paper introduces a novel calibration method using synthetic data, particularly leveraging large language models, to improve model accuracy and calibration without sacrificing generalizability across NLP tasks.

Contribution

The paper proposes a synthetic data-based calibration approach utilizing LLMs, deriving ECE bounds under PAC learning, and demonstrating significant accuracy and calibration improvements.

Findings

Up to 34% increase in accuracy

33% decrease in expected calibration error

Effective across four NLP tasks

Abstract

As machine learning models continue to swiftly advance, calibrating their performance has become a major concern prior to practical and widespread implementation. Most existing calibration methods often negatively impact model accuracy due to the lack of diversity of validation data, resulting in reduced generalizability. To address this, we propose a calibration method that incorporates synthetic data without compromising accuracy. We derive the expected calibration error (ECE) bound using the Probably Approximately Correct (PAC) learning framework. Large language models (LLMs), known for their ability to mimic real data and generate text with mixed class labels, are utilized as a synthetic data generation strategy to lower the ECE bound and improve model accuracy on real test data. Additionally, we propose data generation mechanisms for efficient calibration. Testing our method on four different natural language processing tasks, we observed an average up to 34\% increase in accuracy and 33\% decrease in ECE.