Siamese Labels Auxiliary Learning

TL;DR

This paper introduces Siamese Labels Auxiliary Learning (SiLa), a novel auxiliary training method that enhances model performance and generalization without increasing test parameters, compatible with various network structures.

Contribution

The paper proposes SiLa learning, demonstrating its effectiveness in improving model performance and generalization, and its compatibility with dynamic neural networks.

Findings

SiLa improves model performance without increasing test parameters.

SiLa enhances the generalization ability of models.

SiLa is applicable to various network structures, including dynamic neural networks.

Abstract

In deep learning, auxiliary training has been widely used to assist the training of models. During the training phase, using auxiliary modules to assist training can improve the performance of the model. During the testing phase, auxiliary modules can be removed, so the test parameters are not increased. In this paper, we propose a novel auxiliary training method, Siamese Labels Auxiliary Learning (SiLa). Unlike Deep Mutual Learning (DML), SiLa emphasizes auxiliary learning and can be easily combined with DML. In general, the main work of this paper include: (1) propose SiLa Learning, which improves the performance of common models without increasing test parameters; (2) compares SiLa with DML and proves that SiLa can improve the generalization of the model; (3) SiLa is applied to Dynamic Neural Networks, and proved that SiLa can be used for various types of network structures.