BRIDGE: Budget-aware Reasoning via Intermediate Distillation with Guided Examples
Xuan-An Le, Minh-Nam Tran, Son Nguyen

TL;DR
BRIDGE is a two-phase, budget-aware distillation framework that effectively transfers knowledge from large models to tiny models by using intermediate teachers and synthetic rationales, improving performance with fewer resources.
Contribution
Introduces BRIDGE, a novel two-phase distillation method that leverages a mid-sized teacher assistant and synthetic data to efficiently transfer reasoning capabilities under budget constraints.
Findings
Achieves 28-41% performance gains on benchmarks.
Reduces teacher query costs by 10x while maintaining high accuracy.
Surpasses direct distillation baselines with fewer resources.
Abstract
Distilling knowledge from large proprietary models (e.g., GPT-4) to tiny deployable models (less than 1B parameters) faces a critical capacity-budget trap: the 1000x capacity gap between teachers and students prevents effective direct transfer, while API costs prohibit extensive data collection. We introduce BRIDGE (Budget-Aware Reasoning via Intermediate Distillation), a two-phase framework that resolves these constraints through strategic intermediation and budget asymmetry. In Phase 1, a mid-sized Teacher Assistant (TA; e.g., about 7B) learns from the black-box teacher on a strictly limited subset of data (e.g., 3-5%), selected via a zero-API-cost pipeline that balances entropic difficulty and semantic diversity using only local TA inference. In Phase 2, we exploit this asymmetry-teacher queries are expensive, whereas TA inference is free to amplify supervision: the refined TA…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsExplainable Artificial Intelligence (XAI) · Topic Modeling · Machine Learning and Algorithms
