Spatial Ensemble: a Novel Model Smoothing Mechanism for Student-Teacher Framework

TL;DR

This paper introduces Spatial Ensemble, a new model smoothing technique for student-teacher frameworks that combines fragments of historical student models to improve learning performance and complements existing temporal smoothing methods.

Contribution

The paper proposes Spatial Ensemble, a novel model smoothing mechanism that stitches fragments of historical student models, enhancing student-teacher training effectiveness.

Findings

Spatial Ensemble achieves comparable performance to TMA.

Combining Spatial Ensemble with TMA improves results significantly.

On ImageNet, it improves BYOL accuracy by 0.9%. on CIFAR-10 with FixMatch, accuracy increases by 6%.

Abstract

Model smoothing is of central importance for obtaining a reliable teacher model in the student-teacher framework, where the teacher generates surrogate supervision signals to train the student. A popular model smoothing method is the Temporal Moving Average (TMA), which continuously averages the teacher parameters with the up-to-date student parameters. In this paper, we propose "Spatial Ensemble", a novel model smoothing mechanism in parallel with TMA. Spatial Ensemble randomly picks up a small fragment of the student model to directly replace the corresponding fragment of the teacher model. Consequentially, it stitches different fragments of historical student models into a unity, yielding the "Spatial Ensemble" effect. Spatial Ensemble obtains comparable student-teacher learning performance by itself and demonstrates valuable complementarity with temporal moving average. Their integration, named Spatial-Temporal Smoothing, brings general (sometimes significant) improvement to the student-teacher learning framework on a variety of state-of-the-art methods. For example, based on the self-supervised method BYOL, it yields +0.9% top-1 accuracy improvement on ImageNet, while based on the semi-supervised approach FixMatch, it increases the top-1 accuracy by around +6% on CIFAR-10 when only few training labels are available. Codes and models are available at: https://github.com/tengteng95/Spatial_Ensemble.