Style-Specific Neurons for Steering LLMs in Text Style Transfer

TL;DR

This paper introduces sNeuron-TST, a method that steers large language models to improve text style transfer by deactivating style-specific neurons, enhancing stylistic diversity while addressing fluency issues with contrastive decoding.

Contribution

The paper proposes a novel neuron-based steering technique for LLMs in style transfer, including neuron deactivation and an improved decoding method, demonstrating effectiveness across multiple benchmarks.

Findings

Enhanced stylistic diversity in generated text.

Deactivation of style-specific neurons improves style transfer.

Contrastive decoding mitigates fluency loss.

Abstract

Text style transfer (TST) aims to modify the style of a text without altering its original meaning. Large language models (LLMs) demonstrate superior performance across multiple tasks, including TST. However, in zero-shot setups, they tend to directly copy a significant portion of the input text to the output without effectively changing its style. To enhance the stylistic variety and fluency of the text, we present sNeuron-TST, a novel approach for steering LLMs using style-specific neurons in TST. Specifically, we identify neurons associated with the source and target styles and deactivate source-style-only neurons to give target-style words a higher probability, aiming to enhance the stylistic diversity of the generated text. However, we find that this deactivation negatively impacts the fluency of the generated text, which we address by proposing an improved contrastive decoding method that accounts for rapid token probability shifts across layers caused by deactivated source-style neurons. Empirical experiments demonstrate the effectiveness of the proposed method on six benchmarks, encompassing formality, toxicity, politics, politeness, authorship, and sentiment.