Fast and Accurate Model Scaling

TL;DR

This paper analyzes different strategies for scaling convolutional neural networks, revealing that a simple compound approach focusing on width scaling offers better efficiency and similar accuracy compared to traditional methods.

Contribution

The paper introduces a fast compound scaling method that primarily scales width, resulting in more efficient models with lower activation growth and comparable accuracy.

Findings

Scaling strategies impact model parameters and runtime differently.

Many scaling methods achieve similar accuracy with different resource costs.

The proposed method achieves near square-root growth in activations, improving efficiency.

Abstract

In this work we analyze strategies for convolutional neural network scaling; that is, the process of scaling a base convolutional network to endow it with greater computational complexity and consequently representational power. Example scaling strategies may include increasing model width, depth, resolution, etc. While various scaling strategies exist, their tradeoffs are not fully understood. Existing analysis typically focuses on the interplay of accuracy and flops (floating point operations). Yet, as we demonstrate, various scaling strategies affect model parameters, activations, and consequently actual runtime quite differently. In our experiments we show the surprising result that numerous scaling strategies yield networks with similar accuracy but with widely varying properties. This leads us to propose a simple fast compound scaling strategy that encourages primarily scaling model width, while scaling depth and resolution to a lesser extent. Unlike currently popular scaling strategies, which result in about $O(s)$ increase in model activation w.r.t. scaling flops by a factor of $s$, the proposed fast compound scaling results in close to $O(\sqrt{s})$ increase in activations, while achieving excellent accuracy. This leads to comparable speedups on modern memory-limited hardware (e.g., GPU, TPU). More generally, we hope this work provides a framework for analyzing and selecting scaling strategies under various computational constraints.