Logistic Regression makes small LLMs strong and explainable "tens-of-shot" classifiers

TL;DR

This paper demonstrates that penalized logistic regression on small LLM embeddings can match or outperform large LLMs in simple classification tasks, offering benefits in privacy, cost, and explainability.

Contribution

It introduces a method where small LLM embeddings combined with logistic regression achieve competitive performance with large models, enhancing accessibility and interpretability.

Findings

Logistic regression on small LLM embeddings matches large LLM performance.

Requires no additional labeled data beyond validation set.

Provides stable and interpretable explanations for decisions.

Abstract

For simple classification tasks, we show that users can benefit from the advantages of using small, local, generative language models instead of large commercial models without a trade-off in performance or introducing extra labelling costs. These advantages, including those around privacy, availability, cost, and explainability, are important both in commercial applications and in the broader democratisation of AI. Through experiments on 17 sentence classification tasks (2-4 classes), we show that penalised logistic regression on the embeddings from a small LLM equals (and usually betters) the performance of a large LLM in the "tens-of-shot" regime. This requires no more labelled instances than are needed to validate the performance of the large LLM. Finally, we extract stable and sensible explanations for classification decisions.