Mixtraining: A Better Trade-Off Between Compute and Performance

TL;DR

MixTraining is a novel framework that interleaves self-supervised and supervised learning epochs to improve model accuracy while reducing computational costs, offering a better trade-off between compute and performance.

Contribution

It introduces a unified training approach that combines SSL and SL with smooth transitions, enhancing synergy and efficiency in model training.

Findings

Achieves 8.81% absolute accuracy gain on TinyImageNet.

Accelerates training by up to 1.29x with ViT-Tiny.

Demonstrates superior compute-performance trade-off compared to traditional methods.

Abstract

Incorporating self-supervised learning (SSL) before standard supervised learning (SL) has become a widely used strategy to enhance model performance, particularly in data-limited scenarios. However, this approach introduces a trade-off between computation and performance: while SSL helps with representation learning, it requires a separate, often time-consuming training phase, increasing computational overhead and limiting efficiency in resource-constrained settings. To address these challenges, we propose MixTraining, a novel framework that interleaves several SSL and SL epochs within a unified mixtraining training phase, featuring a smooth transition between two learning objectives. MixTraining enhances synergy between SSL and SL for improved accuracy and consolidates shared computation steps to reduce computation overhead. MixTraining is versatile and applicable to both single-task and multi-task learning scenarios. Extensive experiments demonstrate that MixTraining offers a superior compute-performance trade-off compared to conventional pipelines, achieving an 8.81% absolute accuracy gain (18.89% relative accuracy gain) on the TinyImageNet dataset while accelerating training by up to 1.29x with the ViT-Tiny model.