Efficient Text Classification with Conformal In-Context Learning

TL;DR

This paper evaluates Conformal In-Context Learning (CICLe), a resource-efficient method combining traditional classifiers with LLM prompting, showing consistent improvements and efficiency gains across diverse NLP classification tasks, especially with imbalanced data.

Contribution

The paper provides a comprehensive evaluation of CICLe's effectiveness and efficiency across multiple NLP benchmarks, demonstrating its advantages over existing prompt-based methods.

Findings

CICLe improves base classifier performance and outperforms few-shot prompting with sufficient data.

It reduces prompt length and number of shots by up to 34.45% and 25.16%.

CICLe is especially effective for high class imbalance tasks.

Abstract

Large Language Models (LLMs) demonstrate strong in-context learning abilities, yet their effectiveness in text classification depends heavily on prompt design and incurs substantial computational cost. Conformal In-Context Learning (CICLe) has been proposed as a resource-efficient framework that integrates a lightweight base classifier with Conformal Prediction to guide LLM prompting by adaptively reducing the set of candidate classes. However, its broader applicability and efficiency benefits beyond a single domain have not yet been systematically explored. In this paper, we present a comprehensive evaluation of CICLe across diverse NLP classification benchmarks. The results show that CICLe consistently improves over its base classifier and outperforms few-shot prompting baselines when the sample size is sufficient for training the base classifier, and performs comparably in low-data regimes. In terms of efficiency, CICLe reduces the number of shots and prompt length by up to 34.45% and 25.16%, respectively, and enables the use of smaller models with competitive performance. CICLe is furthermore particularly advantageous for text classification tasks with high class imbalance. These findings highlight CICLe as a practical and scalable approach for efficient text classification, combining the robustness of traditional classifiers with the adaptability of LLMs, and achieving substantial gains in data and computational efficiency.