CE-NAS: An End-to-End Carbon-Efficient Neural Architecture Search Framework

TL;DR

CE-NAS introduces a carbon-efficient neural architecture search framework that dynamically manages resources based on carbon intensity, significantly reducing emissions while maintaining state-of-the-art performance.

Contribution

It proposes a reinforcement learning-based resource adjustment and multi-objective optimization to enhance carbon efficiency in NAS processes.

Findings

Reduces carbon emissions by up to 7.22X on HW-NasBench.

Achieves SOTA accuracy on CIFAR-10 with low carbon footprint.

Maintains competitive performance on ImageNet with minimal carbon consumption.

Abstract

This work presents a novel approach to neural architecture search (NAS) that aims to increase carbon efficiency for the model design process. The proposed framework CE-NAS addresses the key challenge of high carbon cost associated with NAS by exploring the carbon emission variations of energy and energy differences of different NAS algorithms. At the high level, CE-NAS leverages a reinforcement-learning agent to dynamically adjust GPU resources based on carbon intensity, predicted by a time-series transformer, to balance energy-efficient sampling and energy-intensive evaluation tasks. Furthermore, CE-NAS leverages a recently proposed multi-objective optimizer to effectively reduce the NAS search space. We demonstrate the efficacy of CE-NAS in lowering carbon emissions while achieving SOTA results for both NAS datasets and open-domain NAS tasks. For example, on the HW-NasBench dataset, CE-NAS reduces carbon emissions by up to 7.22X while maintaining a search efficiency comparable to vanilla NAS. For open-domain NAS tasks, CE-NAS achieves SOTA results with 97.35% top-1 accuracy on CIFAR-10 with only 1.68M parameters and a carbon consumption of 38.53 lbs of CO2. On ImageNet, our searched model achieves 80.6% top-1 accuracy with a 0.78 ms TensorRT latency using FP16 on NVIDIA V100, consuming only 909.86 lbs of CO2, making it comparable to other one-shot-based NAS baselines.