ScaleNet: Searching for the Model to Scale

TL;DR

ScaleNet introduces a joint search method for base models and scaling strategies, enabling the development of larger, high-performing models with reduced search costs by combining super-supernet training and evolutionary algorithms.

Contribution

It proposes a novel joint search framework for base models and scaling strategies, improving scalability and performance over existing methods.

Findings

Achieves significant performance improvements across various FLOPs.

Reduces search cost by at least 2.53 times.

Demonstrates effective scaling with a hierarchical sampling strategy.

Abstract

Recently, community has paid increasing attention on model scaling and contributed to developing a model family with a wide spectrum of scales. Current methods either simply resort to a one-shot NAS manner to construct a non-structural and non-scalable model family or rely on a manual yet fixed scaling strategy to scale an unnecessarily best base model. In this paper, we bridge both two components and propose ScaleNet to jointly search base model and scaling strategy so that the scaled large model can have more promising performance. Concretely, we design a super-supernet to embody models with different spectrum of sizes (e.g., FLOPs). Then, the scaling strategy can be learned interactively with the base model via a Markov chain-based evolution algorithm and generalized to develop even larger models. To obtain a decent super-supernet, we design a hierarchical sampling strategy to enhance its training sufficiency and alleviate the disturbance. Experimental results show our scaled networks enjoy significant performance superiority on various FLOPs, but with at least 2.53x reduction on search cost. Codes are available at https://github.com/luminolx/ScaleNet.