Offscript: Automated Auditing of Instruction Adherence in LLMs

TL;DR

Offscript is an automated tool designed to evaluate whether large language models adhere to specified custom instructions, revealing frequent deviations and providing a scalable auditing method.

Contribution

This paper introduces Offscript, a novel automated auditing system for assessing instruction adherence in LLMs, addressing the lack of existing evaluation mechanisms.

Findings

Offscript detected potential instruction deviations in 86.4% of analyzed conversations.

22.2% of detected deviations were confirmed as material violations by human review.

Automated auditing proves effective for evaluating compliance with behavioral instructions.

Abstract

Large Language Models (LLMs) and generative search systems are increasingly used for information seeking by diverse populations with varying preferences for knowledge sourcing and presentation. While users can customize LLM behavior through custom instructions and behavioral prompts, no mechanism exists to evaluate whether these instructions are being followed effectively. We present Offscript, an automated auditing tool that efficiently identifies potential instruction following failures in LLMs. In a pilot study analyzing custom instructions sourced from Reddit, Offscript detected potential deviations from instructed behavior in 86.4% of conversations, 22.2% of which were confirmed as material violations through human review. Our findings suggest that automated auditing serves as a viable approach for evaluating compliance to behavioral instructions related to information seeking.