Enhancing Once-For-All: A Study on Parallel Blocks, Skip Connections and Early Exits

TL;DR

This paper introduces OFAv2, an improved version of Once-For-All NAS, incorporating architectural enhancements and training strategies to boost accuracy on Tiny ImageNet while preserving eco-friendliness.

Contribution

The paper presents OFAv2 with architectural modifications, new training phases, and a novel knowledge distillation method to enhance NAS performance without sacrificing ecological benefits.

Findings

Up to 12.07% accuracy improvement on Tiny ImageNet

Maintains ecological advantages of original OFA

Introduces new training phases and knowledge distillation techniques

Abstract

The use of Neural Architecture Search (NAS) techniques to automate the design of neural networks has become increasingly popular in recent years. The proliferation of devices with different hardware characteristics using such neural networks, as well as the need to reduce the power consumption for their search, has led to the realisation of Once-For-All (OFA), an eco-friendly algorithm characterised by the ability to generate easily adaptable models through a single learning process. In order to improve this paradigm and develop high-performance yet eco-friendly NAS techniques, this paper presents OFAv2, the extension of OFA aimed at improving its performance while maintaining the same ecological advantage. The algorithm is improved from an architectural point of view by including early exits, parallel blocks and dense skip connections. The training process is extended by two new phases called Elastic Level and Elastic Height. A new Knowledge Distillation technique is presented to handle multi-output networks, and finally a new strategy for dynamic teacher network selection is proposed. These modifications allow OFAv2 to improve its accuracy performance on the Tiny ImageNet dataset by up to 12.07% compared to the original version of OFA, while maintaining the algorithm flexibility and advantages.