Switchable Representation Learning Framework with Self-compatibility

TL;DR

The paper introduces SFSC, a framework that trains multiple compatible sub-models with different capacities simultaneously, improving resource-adaptive visual search systems by addressing gradient conflicts and dynamic prioritization.

Contribution

SFSC is the first to enable training of multiple compatible models in one process, enhancing adaptability across resource-constrained platforms.

Findings

Achieves state-of-the-art performance on multiple datasets.

Effectively mitigates gradient conflicts among sub-models.

Dynamically adjusts sub-model priorities based on uncertainty.

Abstract

Real-world visual search systems involve deployments on multiple platforms with different computing and storage resources. Deploying a unified model that suits the minimal-constrain platforms leads to limited accuracy. It is expected to deploy models with different capacities adapting to the resource constraints, which requires features extracted by these models to be aligned in the metric space. The method to achieve feature alignments is called ``compatible learning''. Existing research mainly focuses on the one-to-one compatible paradigm, which is limited in learning compatibility among multiple models. We propose a Switchable representation learning Framework with Self-Compatibility (SFSC). SFSC generates a series of compatible sub-models with different capacities through one training process. The optimization of sub-models faces gradients conflict, and we mitigate this problem from the perspective of the magnitude and direction. We adjust the priorities of sub-models dynamically through uncertainty estimation to co-optimize sub-models properly. Besides, the gradients with conflicting directions are projected to avoid mutual interference. SFSC achieves state-of-the-art performance on the evaluated datasets.