BIRD: Behavior Induction via Representation-structure Distillation

TL;DR

BIRD is a novel framework that transfers human-aligned behaviors to models by matching internal representation structures, significantly improving out-of-distribution robustness and offering practical insights for model alignment.

Contribution

Introduces BIRD, a flexible representation-structure distillation method for transferring aligned behaviors, outperforming existing techniques in robustness and providing insights for teacher model selection.

Findings

BIRD improves robust accuracy by up to 16% over baselines.

Effective even when the teacher is simpler and smaller than the student.

Representation properties explain up to 85% of transfer success variance.

Abstract

Human-aligned deep learning models exhibit behaviors consistent with human values, such as robustness, fairness, and honesty. Transferring these behavioral properties to models trained on different tasks or data distributions remains challenging: aligned behavior is easily forgotten during fine-tuning, and collecting task-specific data that preserves this behavior can be prohibitively costly. We introduce BIRD (Behavior Induction via Representation-structure Distillation), a flexible framework for transferring aligned behavior by matching the internal representation structure of a student model to that of a teacher. Applied to out-of-distribution robustness in image classification, BIRD outperforms fine-tuning, transfer learning, and continual learning methods, improving robust accuracy by up to 16% over the next strongest baseline. It remains effective even when the teacher is trained on a much simpler dataset and is $25 \times$ smaller than the student. In a large-scale study of over 400 teacher-student pairs, we show that three interpretable and computable properties of the teacher's representations (i.e., task relevance, behavioral relevance, and complementary knowledge) explain up to 85% of the variance in transfer success. These insights offer practical guidance for teacher selection and design. BIRD turns small, well-aligned models into scalable alignment seeds, removing a key bottleneck in deploying safe AI systems in the wild.