Beyond Convolution: A Taxonomy of Structured Operators for Learning-Based Image Processing

TL;DR

This paper systematically categorizes and analyzes various structured operators that extend beyond standard convolution in neural networks, aiming to enhance learning-based image processing capabilities.

Contribution

It introduces a comprehensive taxonomy of five families of operators that generalize or replace convolution, with formal definitions and analysis of their properties and applications.

Findings

Decomposition-based operators effectively separate structural and noise components.

Adaptive weighted operators modulate kernel contributions based on spatial or signal content.

Attention-based operators relax the locality assumption, enabling more flexible modeling.

Abstract

The convolution operator is the fundamental building block of modern convolutional neural networks (CNNs), owing to its simplicity, translational equivariance, and efficient implementation. However, its structure as a fixed, linear, locally-averaging operator limits its ability to capture structured signal properties such as low-rank decompositions, adaptive basis representations, and non-uniform spatial dependencies. This paper presents a systematic taxonomy of operators that extend or replace the standard convolution in learning-based image processing pipelines. We organise the landscape of alternative operators into five families: (i) decomposition-based operators, which separate structural and noise components through singular value or tensor decompositions; (ii) adaptive weighted operators, which modulate kernel contributions as a function of spatial position or signal content; (iii) basis-adaptive operators, which optimise the analysis bases together with the network weights; (iv) integral and kernel operators, which generalise the convolution to position-dependent and non-linear kernels; and (v) attention-based operators, which relax the locality assumption entirely. For each family, we provide a formal definition, a discussion of its structural properties with respect to the convolution, and a critical analysis of the tasks for which the operator is most appropriate. We further provide a comparative analysis of all families across relevant dimensions -- linearity, locality, equivariance, computational cost, and suitability for image-to-image and image-to-label tasks -- and outline the open challenges and future directions of this research area.