Step Out and Seek Around: On Warm-Start Training with Incremental Data

TL;DR

This paper introduces CKCA, a novel continuous training method that improves warm-starting with incremental data by using feature regularization and adaptive knowledge distillation, leading to significant accuracy gains.

Contribution

The paper proposes CKCA, a new algorithm combining feature regularization and adaptive knowledge distillation to enhance warm-start training with incremental data.

Findings

Achieves up to 8.39% higher top-1 accuracy on ImageNet.

Outperforms existing warm-starting methods consistently.

Effectively mitigates forgetting while transferring knowledge.

Abstract

Data often arrives in sequence over time in real-world deep learning applications such as autonomous driving. When new training data is available, training the model from scratch undermines the benefit of leveraging the learned knowledge, leading to significant training costs. Warm-starting from a previously trained checkpoint is the most intuitive way to retain knowledge and advance learning. However, existing literature suggests that this warm-starting degrades generalization. In this paper, we advocate for warm-starting but stepping out of the previous converging point, thus allowing a better adaptation to new data without compromising previous knowledge. We propose Knowledge Consolidation and Acquisition (CKCA), a continuous model improvement algorithm with two novel components. First, a novel feature regularization (FeatReg) to retain and refine knowledge from existing checkpoints; Second, we propose adaptive knowledge distillation (AdaKD), a novel approach to forget mitigation and knowledge transfer. We tested our method on ImageNet using multiple splits of the training data. Our approach achieves up to $8.39\%$ higher top1 accuracy than the vanilla warm-starting and consistently outperforms the prior art with a large margin.