How green is continual learning, really? Analyzing the energy consumption in continual training of vision foundation models

TL;DR

This paper systematically evaluates the energy consumption of various continual learning algorithms for vision models, highlighting the importance of inference energy and introducing a new efficiency metric.

Contribution

It provides a comprehensive empirical analysis of energy use in continual learning and proposes the Energy NetScore to assess energy-accuracy trade-offs.

Findings

Different continual learning algorithms have varying energy impacts.

Inference energy consumption is critical for sustainability assessments.

The Energy NetScore effectively measures energy efficiency in continual learning.

Abstract

With the ever-growing adoption of AI, its impact on the environment is no longer negligible. Despite the potential that continual learning could have towards Green AI, its environmental sustainability remains relatively uncharted. In this work we aim to gain a systematic understanding of the energy efficiency of continual learning algorithms. To that end, we conducted an extensive set of empirical experiments comparing the energy consumption of recent representation-, prompt-, and exemplar-based continual learning algorithms and two standard baseline (fine tuning and joint training) when used to continually adapt a pre-trained ViT-B/16 foundation model. We performed our experiments on three standard datasets: CIFAR-100, ImageNet-R, and DomainNet. Additionally, we propose a novel metric, the Energy NetScore, which we use measure the algorithm efficiency in terms of energy-accuracy trade-off. Through numerous evaluations varying the number and size of the incremental learning steps, our experiments demonstrate that different types of continual learning algorithms have very different impacts on energy consumption during both training and inference. Although often overlooked in the continual learning literature, we found that the energy consumed during the inference phase is crucial for evaluating the environmental sustainability of continual learning models.