NetBooster: Empowering Tiny Deep Learning By Standing on the Shoulders of Deep Giants

TL;DR

NetBooster is a framework that enhances tiny neural networks by expanding and contracting their architectures, significantly improving their performance on large datasets and tasks, thus advancing tiny deep learning deployment.

Contribution

The paper introduces NetBooster, a novel architecture augmentation strategy for tiny neural networks that addresses under-fitting and boosts their effectiveness.

Findings

NetBooster outperforms existing tiny deep learning methods.

It effectively mitigates under-fitting in TNNs.

Demonstrates consistent improvements across datasets and tasks.

Abstract

Tiny deep learning has attracted increasing attention driven by the substantial demand for deploying deep learning on numerous intelligent Internet-of-Things devices. However, it is still challenging to unleash tiny deep learning's full potential on both large-scale datasets and downstream tasks due to the under-fitting issues caused by the limited model capacity of tiny neural networks (TNNs). To this end, we propose a framework called NetBooster to empower tiny deep learning by augmenting the architectures of TNNs via an expansion-then-contraction strategy. Extensive experiments show that NetBooster consistently outperforms state-of-the-art tiny deep learning solutions.