Online Domain-aware LLM Decoding for Continual Domain Evolution

TL;DR

This paper introduces ODD, a real-time decoding framework that adaptively fuses a base LLM with a prefix-tree prior to handle continuous domain evolution and concept drift without retraining.

Contribution

The paper proposes a novel online decoding method that dynamically adjusts to domain changes, improving LLM performance in evolving environments without costly retraining.

Findings

ODD outperforms baseline methods across all metrics.

Achieves 0.065 ROUGE-L gain and 13.6% cosine similarity improvement.

Demonstrates robustness to lexical and contextual shifts.

Abstract

LLMs are typically fine-tuned offline on domain-specific data, assuming a static domain. In practice, domain knowledge evolves continuously through new regulations, products, services, and interaction patterns. Retraining or fine-tuning LLMs for every new instance is computationally infeasible. Additionally, real-world environments also exhibit temporal dynamics with shifting data distributions. Disregarding this phenomenon, commonly referred to as concept drift, can significantly diminish a model's predictive accuracy. This mismatch between evolving domains and static adaptation pipelines highlights the need for efficient, real-time adaptation without costly retraining. In response, we introduce Online Domain-aware Decoding framework (ODD). ODD performs probability-level fusion between a base LLM and a prefix-tree prior, guided by adaptive confidence modulation using disagreement and continuity signals. Empirical evaluation under diverse drift scenarios demonstrates that ODD consistently surpasses LLM-Greedy and LLM-Temp Scaled across all syntactic and semantic NLG metrics. It yields an absolute ROUGE-L gain of 0.065 and a 13.6% relative improvement in Cosine Similarity over the best baseline. These results demonstrate ODD 's robustness to evolving lexical and contextual patterns, making it suitable for dynamic LLM applications.