Reduced-Order Modeling of Deep Neural Networks

TL;DR

This paper presents a novel reduced-order modeling approach inspired by dynamical systems to accelerate deep neural network inference by replacing convolutional layers with smaller fully-connected layers, maintaining accuracy.

Contribution

It introduces a maximum volume algorithm-based method for simplifying neural networks, demonstrating significant speedups with minimal accuracy loss.

Findings

Smaller fully-connected layers can replace convolutional layers effectively.

The method achieves speedups on various datasets.

Minimal accuracy drop observed in practical cases.

Abstract

We introduce a new method for speeding up the inference of deep neural networks. It is somewhat inspired by the reduced-order modeling techniques for dynamical systems.The cornerstone of the proposed method is the maximum volume algorithm. We demonstrate efficiency on neural networks pre-trained on different datasets. We show that in many practical cases it is possible to replace convolutional layers with much smaller fully-connected layers with a relatively small drop in accuracy.