Towards efficient feature sharing in MIMO architectures

TL;DR

This paper introduces a novel unmixing step in MIMO architectures to enable effective feature sharing among subnetworks, improving performance especially for small models on mobile devices.

Contribution

The paper proposes a new unmixing technique that addresses the lack of feature sharing in MIMO architectures, enhancing their efficiency and applicability.

Findings

Unmixing step enables feature sharing in MIMO architectures.

Improved performance on CIFAR-100 for small models.

Potential for better deployment on mobile and AR/VR devices.

Abstract

Multi-input multi-output architectures propose to train multiple subnetworks within one base network and then average the subnetwork predictions to benefit from ensembling for free. Despite some relative success, these architectures are wasteful in their use of parameters. Indeed, we highlight in this paper that the learned subnetwork fail to share even generic features which limits their applicability on smaller mobile and AR/VR devices. We posit this behavior stems from an ill-posed part of the multi-input multi-output framework. To solve this issue, we propose a novel unmixing step in MIMO architectures that allows subnetworks to properly share features. Preliminary experiments on CIFAR-100 show our adjustments allow feature sharing and improve model performance for small architectures.