Routing Distilled Knowledge via Mixture of LoRA Experts for Large Language Model based Bundle Generation

TL;DR

RouteDK introduces a dynamic routing framework with mixture of LoRA experts to distill and integrate diverse knowledge types from large language models, achieving high accuracy and efficiency in bundle generation.

Contribution

The paper proposes RouteDK, a novel knowledge routing framework with input-aware dynamic fusion of LoRA experts for efficient large language model distillation.

Findings

Achieves accuracy comparable or superior to teacher LLMs.

Outperforms state-of-the-art bundle generation methods.

Maintains strong computational efficiency.

Abstract

Large Language Models (LLMs) have shown potential in automatic bundle generation but suffer from prohibitive computational costs. Although knowledge distillation offers a pathway to more efficient student models, our preliminary study reveals that naively integrating diverse types of distilled knowledge from teacher LLMs into student LLMs leads to knowledge conflict, negatively impacting the performance of bundle generation. To address this, we propose RouteDK, a framework for routing distilled knowledge through a mixture of LoRA expert architecture. Specifically, we first distill knowledge from the teacher LLM for bundle generation in two complementary types: high-level knowledge (generalizable rules) and fine-grained knowledge (session-specific reasoning). We then train knowledge-specific LoRA experts for each type of knowledge together with a base LoRA expert. For effective integration, we propose a dynamic fusion module, featuring an input-aware router, where the router balances expert contributions by dynamically determining optimal weights based on input, thereby effectively mitigating knowledge conflicts. To further improve inference reliability, we design an inference-time enhancement module to reduce variance and mitigate suboptimal reasoning. Experiments on three public datasets show that our RouteDK achieves accuracy comparable to or even better than the teacher LLM, while maintaining strong computational efficiency. In addition, it outperforms state-of-the-art approaches for bundle generation.