Revisiting Architecture-aware Knowledge Distillation: Smaller Models and Faster Search

TL;DR

This paper introduces TRADE, a trust region Bayesian optimization-based method for efficient architecture search in knowledge distillation, outperforming traditional NAS and fixed architectures in neural network compression.

Contribution

The paper proposes TRADE, a novel, efficient architecture search algorithm for knowledge distillation that considers a broader search space and reduces computational costs.

Findings

TRADE outperforms conventional NAS in KD tasks.

TRADE finds effective architectures faster than existing methods.

Experimental results demonstrate superior performance of TRADE-based models.

Abstract

Knowledge Distillation (KD) has recently emerged as a popular method for compressing neural networks. In recent studies, generalized distillation methods that find parameters and architectures of student models at the same time have been proposed. Still, this search method requires a lot of computation to search for architectures and has the disadvantage of considering only convolutional blocks in their search space. This paper introduces a new algorithm, coined as Trust Region Aware architecture search to Distill knowledge Effectively (TRADE), that rapidly finds effective student architectures from several state-of-the-art architectures using trust region Bayesian optimization approach. Experimental results show our proposed TRADE algorithm consistently outperforms both the conventional NAS approach and pre-defined architecture under KD training.