Beyond Mimicry to Contextual Guidance: Knowledge Distillation for Interactive AI

TL;DR

This paper introduces a novel knowledge distillation method for interactive AI, shifting from output mimicry to contextual guidance, enabling adaptable, scalable, and policy-aligned responses in multi-turn conversations.

Contribution

It proposes a framework where a teacher model creates a library of strategic guidance for scenarios, improving interactive AI performance without retraining.

Findings

Enhanced service quality and customer satisfaction.

Maintains alignment with firm policies.

Outperforms standard fine-tuning methods.

Abstract

As large language models increasingly mediate firm - customer interactions, firms face a tradeoff: the most capable models perform well but are costly and difficult to control at scale. Existing knowledge distillation methods address this challenge by training weaker, deployable models to imitate frontier outputs; however, such open-loop approaches are poorly suited to interactive, multi-turn settings where responses must be sequenced coherently across conversational states. We propose a shift in what knowledge is distilled - from output imitation to contextual guidance. We develop a framework in which a superior teacher model constructs a reusable library of strategic textual guidance for particular scenarios likely to be encountered by the student. When deployed, the student retrieves the context-specific guidance at inference time, enabling adaptive behavior without retraining. Using customer-service interactions, we show that this approach improves service quality and customer satisfaction relative to standard fine-tuning while maintaining alignment with firm policies. The results position inference-time textual guidance as a scalable and controllable approach to distillation for interactive AI agents in marketing settings.