FC2T2: The Fast Continuous Convolutional Taylor Transform with Applications in Vision and Graphics

TL;DR

This paper introduces FC2T2, a fast convolutional Taylor transform inspired by the FMM, enabling efficient approximation of continuous convolutional operators with significant computational savings in vision and graphics applications.

Contribution

The paper presents FC2T2, a novel series expansion method that accelerates continuous convolution computations and integrates as implicit layers in neural networks for vision and graphics.

Findings

Achieves up to 200x reduction in FLOPs compared to state-of-the-art methods.

Enables efficient forward and backward passes for neural network layers involving continuous convolutions.

Demonstrates applicability in vision and graphics tasks with minimal accuracy loss.

Abstract

Series expansions have been a cornerstone of applied mathematics and engineering for centuries. In this paper, we revisit the Taylor series expansion from a modern Machine Learning perspective. Specifically, we introduce the Fast Continuous Convolutional Taylor Transform (FC2T2), a variant of the Fast Multipole Method (FMM), that allows for the efficient approximation of low dimensional convolutional operators in continuous space. We build upon the FMM which is an approximate algorithm that reduces the computational complexity of N-body problems from O(NM) to O(N+M) and finds application in e.g. particle simulations. As an intermediary step, the FMM produces a series expansion for every cell on a grid and we introduce algorithms that act directly upon this representation. These algorithms analytically but approximately compute the quantities required for the forward and backward pass of the backpropagation algorithm and can therefore be employed as (implicit) layers in Neural Networks. Specifically, we introduce a root-implicit layer that outputs surface normals and object distances as well as an integral-implicit layer that outputs a rendering of a radiance field given a 3D pose. In the context of Machine Learning, $N$ and $M$ can be understood as the number of model parameters and model evaluations respectively which entails that, for applications that require repeated function evaluations which are prevalent in Computer Vision and Graphics, unlike regular Neural Networks, the techniques introduce in this paper scale gracefully with parameters. For some applications, this results in a 200x reduction in FLOPs compared to state-of-the-art approaches at a reasonable or non-existent loss in accuracy.