Gradient Residual Connections

TL;DR

This paper introduces gradient residual connections that leverage gradient information to enhance neural networks' ability to approximate high-frequency functions, improving performance on synthetic and real-world tasks.

Contribution

It proposes a novel gradient-based residual connection, providing theoretical intuition and demonstrating improved approximation of high-frequency functions over standard residuals.

Findings

Gradient residuals improve high-frequency function approximation.

The method enhances super-resolution performance.

Comparable results to standard residuals on classification and segmentation.

Abstract

Existing work has linked properties of a function's gradient to the difficulty of function approximation. Motivated by these insights, we study how gradient information can be leveraged to improve neural network's ability to approximate high-frequency functions, and we propose a gradient-based residual connection as a complement to the standard identity skip connection used in residual networks. We provide simple theoretical intuition for why gradient information can help distinguish inputs and improve the approximation of functions with rapidly varying behaviour. On a synthetic regression task with a high-frequency sinusoidal ground truth, we show that conventional residual connections struggle to capture high-frequency patterns. In contrast, our gradient residual substantially improves approximation quality. We then introduce a convex combination of the standard and gradient residuals, allowing the network to flexibly control how strongly it relies on gradient information. After validating the design choices of our proposed method through an ablation study, we further validate our approach's utility on the single-image super-resolution task, where the underlying function may be high-frequency. Finally, on standard tasks such as image classification and segmentation, our method achieves performance comparable to standard residual networks, suggesting its broad utility.