Layer-wise Derivative Controlled Networks

TL;DR

ChainzRule introduces a derivative-aware neural architecture that balances accuracy, efficiency, and stability by regularizing intermediate derivatives, outperforming standard models with fewer parameters.

Contribution

The paper proposes ChainzRule, a novel neural architecture using Differential Regularization (DREG) to control derivatives, enhancing stability without sacrificing representational power.

Findings

Outperformed standard models in benchmarks with 15.5x fewer parameters.

Reduced peak gradient volatility by 23.1% on MNIST.

Achieved 70.17% accuracy on Yelp Full ordinal regression.

Abstract

As machine learning models grow in complexity, they increasingly struggle with three conflicting demands: the need for high accuracy, the requirement for hardware efficiency, and the necessity of functional stability. Traditional architectures often achieve performance at the expense of spiky or unpredictable behavior, where small changes in input lead to massive swings in output -- a critical flaw for real-world deployment in sensitive environments. This paper introduces ChainzRule (CR), a novel neural architecture designed to harmonize these competing goals. ChainzRule replaces standard piecewise-linear activations with a Polynomial Engine governed by Differential Regularization (DREG). Unlike traditional methods that impose global, coarse-grained constraints on a model's Lipschitz constant, DREG acts as a targeted regularization on intermediate derivatives. This approach suppresses extreme sensitivity without attenuating the representational power inherent in the Polynomial Engine. In head-to-head "Fair Fight" benchmarks, ChainzRule outperformed standard models while using 15.5x fewer parameters. On the MNIST dataset, it reduced peak gradient volatility by an average of 23.1%, ensuring a smoother and more predictable manifold. On Yelp Full ordinal regression under explicit DREG regularization, ChainzRule achieves 70.17% accuracy, validating that derivative-aware regularization is compatible with competitive performance on realistic tasks. By embedding gradient awareness into the architecture via DREG, ChainzRule demonstrates that stability and accuracy need not be competing objectives.