State-driven Implicit Modeling for Sparsity and Robustness in Neural Networks

TL;DR

This paper introduces State-driven Implicit Modeling (SIM), a novel training approach for implicit neural models that enhances sparsity and robustness while reducing computational costs by avoiding implicit differentiation.

Contribution

The paper proposes a convex, parallelizable training method for implicit models that constrains internal states to match baseline models, improving efficiency and model properties.

Findings

Enhanced sparsity in neural networks

Improved robustness of models against perturbations

Reduced training computational costs

Abstract

Implicit models are a general class of learning models that forgo the hierarchical layer structure typical in neural networks and instead define the internal states based on an ``equilibrium'' equation, offering competitive performance and reduced memory consumption. However, training such models usually relies on expensive implicit differentiation for backward propagation. In this work, we present a new approach to training implicit models, called State-driven Implicit Modeling (SIM), where we constrain the internal states and outputs to match that of a baseline model, circumventing costly backward computations. The training problem becomes convex by construction and can be solved in a parallel fashion, thanks to its decomposable structure. We demonstrate how the SIM approach can be applied to significantly improve sparsity (parameter reduction) and robustness of baseline models trained on FashionMNIST and CIFAR-100 datasets.