RAPTOR: Ridge-Adaptive Logistic Probes

TL;DR

RAPTOR introduces a simple, regularized logistic probe for analyzing large language models, achieving high accuracy, stability, and low training costs, with theoretical insights into its behavior.

Contribution

The paper presents RAPTOR, a ridge-regularized logistic probe that improves concept vector estimation in LLMs and provides a theoretical framework explaining its effectiveness.

Findings

RAPTOR matches or exceeds baseline accuracy in LLM probing.

It offers stable concept vectors under ablation and low training costs.

Theoretical analysis aligns with empirical observations on real embeddings.

Abstract

Probing studies what information is encoded in a frozen LLM's layer representations by training a lightweight predictor on top of them. Beyond analysis, probes are often used operationally in probe-then-steer pipelines: a learned concept vector is extracted from a probe and injected via additive activation steering by adding it to a layer representation during the forward pass. The effectiveness of this pipeline hinges on estimating concept vectors that are accurate, directionally stable under ablation, and inexpensive to obtain. Motivated by these desiderata, we propose RAPTOR (Ridge-Adaptive Logistic Probe), a simple L2-regularized logistic probe whose validation-tuned ridge strength yields concept vectors from normalized weights. Across extensive experiments on instruction-tuned LLMs and human-written concept datasets, RAPTOR matches or exceeds strong baselines in accuracy while achieving competitive directional stability and substantially lower training cost; these quantitative results are supported by qualitative downstream steering demonstrations. Finally, using the Convex Gaussian Min-max Theorem (CGMT), we provide a mechanistic characterization of ridge logistic regression in an idealized Gaussian teacher-student model in the high-dimensional few-shot regime, explaining how penalty strength mediates probe accuracy and concept-vector stability and yielding structural predictions that qualitatively align with trends observed on real LLM embeddings.