Input Conditioned Layer Dropping in Speech Foundation Models

TL;DR

This paper introduces an input-driven layer dropping method for speech models that dynamically adapts the network's depth based on input features, improving efficiency without sacrificing performance.

Contribution

It proposes a novel input-conditioned layer dropping technique that uses a lightweight network to select layers, enhancing dynamic adaptability in speech foundation models.

Findings

Outperforms random layer dropping in experiments

Achieves comparable or better results than early exit strategies

Effective across multiple speech and audio benchmarks

Abstract

Curating foundation speech models for edge and IoT settings, where computational resources vary over time, requires dynamic architectures featuring adaptable reduction strategies. One emerging approach is layer dropping ($\mathcal{LD}$) which skips fraction of the layers of a backbone network during inference to reduce the computational load. This allows transforming static models into dynamic ones. However, existing approaches exhibit limitations either in the mode of selecting layers or by significantly modifying the neural architecture. To this end, we propose input-driven $\mathcal{LD}$ that employs the network's input features and a lightweight layer selecting network to determine the optimum combination of processing layers. Extensive experimentation on 4 speech and audio public benchmarks, using two different pre-trained foundation models, demonstrates the effectiveness of our approach, thoroughly outperforming random dropping and producing on-par (or better) results to early exit.