Weird Generalization is Weirdly Brittle

TL;DR

This study investigates the phenomenon of weird generalization in models, confirming its occurrence under certain conditions but demonstrating its brittleness and the effectiveness of simple interventions to mitigate it.

Contribution

The paper provides an extended replication of weird generalization results, showing its fragility and proposing straightforward prompt-based interventions as effective solutions.

Findings

Weird generalization only occurs in specific models and datasets.

Simple prompt-based interventions can mitigate weird generalization.

Generic interventions without specific trait anticipation are still effective.

Abstract

Weird generalization is a phenomenon in which models fine-tuned on data from a narrow domain (e.g. insecure code) develop surprising traits that manifest even outside that domain (e.g. broad misalignment)-a phenomenon that prior work has highlighted as a critical safety concern. Here, we present an extended replication study of key weird generalization results across an expanded suite of models and datasets. We confirm that surprising (and dangerous) traits can emerge under certain circumstances, but we find that weird generalization is exceptionally brittle: it emerges only for specific models on specific datasets, and it vanishes under simple training-time, prompt-based interventions. We find that the most effective interventions provide prompt context that makes the generalized behavior the expected behavior. However, we show that even very generic interventions that do not anticipate specific generalized traits can still be effective in mitigating weird generalization's effects. Our findings thus help clarify the nature of the safety threat that weird generalization poses and point toward an easily implemented set of solutions.