Solving ImageNet: a Unified Scheme for Training any Backbone to Top Results

TL;DR

This paper introduces USI, a unified, hyper-parameter-free training scheme for ImageNet that consistently improves performance across diverse architectures, transforming model training into an automatic, efficient process.

Contribution

The paper presents USI, a universal training scheme that eliminates the need for architecture-specific tuning, enabling consistent top results on ImageNet for various models.

Findings

USI outperforms previous state-of-the-art methods on all tested architectures.

USI reduces training complexity by removing hyper-parameter tuning.

USI enables fair comparison of different backbones on the speed-accuracy trade-off.

Abstract

ImageNet serves as the primary dataset for evaluating the quality of computer-vision models. The common practice today is training each architecture with a tailor-made scheme, designed and tuned by an expert. In this paper, we present a unified scheme for training any backbone on ImageNet. The scheme, named USI (Unified Scheme for ImageNet), is based on knowledge distillation and modern tricks. It requires no adjustments or hyper-parameters tuning between different models, and is efficient in terms of training times. We test USI on a wide variety of architectures, including CNNs, Transformers, Mobile-oriented and MLP-only. On all models tested, USI outperforms previous state-of-the-art results. Hence, we are able to transform training on ImageNet from an expert-oriented task to an automatic seamless routine. Since USI accepts any backbone and trains it to top results, it also enables to perform methodical comparisons, and identify the most efficient backbones along the speed-accuracy Pareto curve. Implementation is available at:https://github.com/Alibaba-MIIL/Solving_ImageNet