FLOPs as a Direct Optimization Objective for Learning Sparse Neural Networks

TL;DR

This paper introduces a method to directly optimize neural networks for a specified number of FLOPs, enabling resource-efficient models tailored to different hardware constraints during training.

Contribution

It extends existing sparsity techniques by incorporating FLOPs as a direct optimization objective, allowing targeted model compression based on FLOPs constraints.

Findings

Successfully trains models to meet specific FLOPs targets

Demonstrates resource-efficient neural networks for image classification

Adapts to different system constraints like GPU and mobile devices

Abstract

There exists a plethora of techniques for inducing structured sparsity in parametric models during the optimization process, with the final goal of resource-efficient inference. However, few methods target a specific number of floating-point operations (FLOPs) as part of the optimization objective, despite many reporting FLOPs as part of the results. Furthermore, a one-size-fits-all approach ignores realistic system constraints, which differ significantly between, say, a GPU and a mobile phone -- FLOPs on the former incur less latency than on the latter; thus, it is important for practitioners to be able to specify a target number of FLOPs during model compression. In this work, we extend a state-of-the-art technique to directly incorporate FLOPs as part of the optimization objective and show that, given a desired FLOPs requirement, different neural networks can be successfully trained for image classification.