White-Box Sensitivity Auditing with Steering Vectors

TL;DR

This paper introduces a white-box sensitivity auditing framework for large language models that uses activation steering to assess biases and properties more rigorously than traditional black-box methods.

Contribution

It presents a novel internal sensitivity testing approach leveraging model internals, improving bias detection in high-stakes LLM decision tasks.

Findings

White-box auditing reveals significant bias dependence not detected by black-box tests.

The method effectively identifies biases related to protected attributes in simulated decision tasks.

Code implementation is publicly available for reproducibility and further research.

Abstract

Algorithmic audits are essential tools for examining systems for properties required by regulators or desired by operators. Current audits of large language models (LLMs) primarily rely on black-box evaluations that assess model behavior only through input-output testing. These methods are limited to tests constructed in the input space, often generated by heuristics. In addition, many socially relevant model properties (e.g., gender bias) are abstract and difficult to measure through text-based inputs alone. To address these limitations, we propose a white-box sensitivity auditing framework for LLMs that leverages activation steering to conduct more rigorous assessments through model internals. Our auditing method conducts internal sensitivity tests by manipulating key concepts relevant to the model's intended function for the task. We demonstrate its application to bias audits in four simulated high-stakes LLM decision tasks. Our method consistently indicates substantial dependence on protected attributes in model predictions, even in settings where standard black-box evaluations suggest little or no bias. Our code is openly available at https://github.com/hannahxchen/llm-steering-audit