Knowledge boosting during low-latency inference

TL;DR

This paper introduces knowledge boosting, a technique enabling small models to benefit from large models' knowledge during low-latency streaming inference despite communication delays, improving performance in speech tasks.

Contribution

The paper proposes a novel method called knowledge boosting that allows large models to enhance small model inference in real-time streaming applications despite delays.

Findings

Larger performance gains when the gap between small and large models is wide.

Effective for speech separation and enhancement tasks with delays up to 48 ms.

Demonstrates feasibility of large-small model collaboration in low-latency scenarios.

Abstract

Models for low-latency, streaming applications could benefit from the knowledge capacity of larger models, but edge devices cannot run these models due to resource constraints. A possible solution is to transfer hints during inference from a large model running remotely to a small model running on-device. However, this incurs a communication delay that breaks real-time requirements and does not guarantee that both models will operate on the same data at the same time. We propose knowledge boosting, a novel technique that allows a large model to operate on time-delayed input during inference, while still boosting small model performance. Using a streaming neural network that processes 8 ms chunks, we evaluate different speech separation and enhancement tasks with communication delays of up to six chunks or 48 ms. Our results show larger gains where the performance gap between the small and large models is wide, demonstrating a promising method for large-small model collaboration for low-latency applications. Code, dataset, and audio samples available at https://knowledgeboosting.cs.washington.edu/.