Nearest Neighbor Language Models for Stylistic Controllable Generation

TL;DR

This paper introduces a memory-augmented language model architecture that can control stylistic attributes like politeness, formality, and toxicity during text generation, demonstrating the potential for style-specific controllable language models.

Contribution

The paper proposes a novel memory-augmented language model architecture that enables explicit control over stylistic attributes in generated text, validated through extensive experiments.

Findings

Style-specific datastores enhance generation quality.

Results vary significantly across different styles.

Pretraining data influences style control effectiveness.

Abstract

Recent language modeling performance has been greatly improved by the use of external memory. This memory encodes the context so that similar contexts can be recalled during decoding. This similarity depends on how the model learns to encode context, which can be altered to include other attributes, such as style. We construct and evaluate an architecture for this purpose, using corpora annotated for politeness, formality, and toxicity. Through extensive experiments and human evaluation we demonstrate the potential of our method to generate text while controlling style. We find that style-specific datastores improve generation performance, though results vary greatly across styles, and the effect of pretraining data and specific styles should be explored in future work.