Sparse Activation Editing for Reliable Instruction Following in Narratives

TL;DR

This paper introduces Concise-SAE, a training-free neuron editing method that enhances language models' ability to follow instructions in narratives, validated on a new diverse benchmark, FreeInstruct.

Contribution

It presents a novel, training-free neuron editing framework for improving instruction following in narratives, without needing labeled data.

Findings

Achieves state-of-the-art instruction adherence

Maintains high generation quality

Effective across diverse narrative tasks

Abstract

Complex narrative contexts often challenge language models' ability to follow instructions, and existing benchmarks fail to capture these difficulties. To address this, we propose Concise-SAE, a training-free framework that improves instruction following by identifying and editing instruction-relevant neurons using only natural language instructions, without requiring labelled data. To thoroughly evaluate our method, we introduce FreeInstruct, a diverse and realistic benchmark of 1,212 examples that highlights the challenges of instruction following in narrative-rich settings. While initially motivated by complex narratives, Concise-SAE demonstrates state-of-the-art instruction adherence across varied tasks without compromising generation quality.