Predictions Based on Pixel Data: Insights from PDEs and Finite Differences

TL;DR

This paper demonstrates that small neural networks can exactly represent numerical discretizations of PDEs for time-sequence data, using connections between convolution and finite difference operators, supported by numerical experiments.

Contribution

It provides a theoretical framework linking convolutional neural networks to finite difference methods for PDEs, showing exact representation capabilities for discretized PDEs.

Findings

Small networks can exactly represent PDE discretizations

Connections between convolution and finite differences are exploited

Numerical experiments validate the theoretical results

Abstract

As supported by abundant experimental evidence, neural networks are state-of-the-art for many approximation tasks in high-dimensional spaces. Still, there is a lack of a rigorous theoretical understanding of what they can approximate, at which cost, and at which accuracy. One network architecture of practical use, especially for approximation tasks involving images, is (residual) convolutional networks. However, due to the locality of the linear operators involved in these networks, their analysis is more complicated than that of fully connected neural networks. This paper deals with approximation of time sequences where each observation is a matrix. We show that with relatively small networks, we can represent exactly a class of numerical discretizations of PDEs based on the method of lines. We constructively derive these results by exploiting the connections between discrete convolution and finite difference operators. Our network architecture is inspired by those typically adopted in the approximation of time sequences. We support our theoretical results with numerical experiments simulating the linear advection, heat, and Fisher equations.