A Little Confidence Goes a Long Way

TL;DR

This paper presents resource-efficient methods for binary classification in large language models by leveraging hidden state probes, enabling high performance without labeled data or extensive computation.

Contribution

It introduces novel unsupervised probing techniques and confidence scoring methods that match large LLM performance with significantly less resources.

Findings

Comparable accuracy to state-of-the-art LLMs

Requires no labeled data for training

Uses significantly less computational resources

Abstract

We introduce a group of related methods for binary classification tasks using probes of the hidden state activations in large language models (LLMs). Performance is on par with the largest and most advanced LLMs currently available, but requiring orders of magnitude fewer computational resources and not requiring labeled data. This approach involves translating class labels into a semantically rich description, spontaneous symmetry breaking of multilayer perceptron probes for unsupervised learning and inference, training probes to generate confidence scores (prior probabilities) from hidden state activations subject to known constraints via entropy maximization, and selecting the most confident probe model from an ensemble for prediction. These techniques are evaluated on four datasets using five base LLMs.