Neural Networks Trained to Solve Differential Equations Learn General Representations

TL;DR

This paper presents a new method using SVCCA to measure how neural network layers generalize across parametrized tasks, demonstrating that initial layers are more general while deeper layers become task-specific.

Contribution

The authors introduce a fast SVCCA-based technique for assessing layer generality in neural networks trained on parametrized PDE problems, validated against transfer learning methods.

Findings

First hidden layer is highly general across tasks

Deeper layers become increasingly task-specific

SVCCA method is faster than existing techniques

Abstract

We introduce a technique based on the singular vector canonical correlation analysis (SVCCA) for measuring the generality of neural network layers across a continuously-parametrized set of tasks. We illustrate this method by studying generality in neural networks trained to solve parametrized boundary value problems based on the Poisson partial differential equation. We find that the first hidden layer is general, and that deeper layers are successively more specific. Next, we validate our method against an existing technique that measures layer generality using transfer learning experiments. We find excellent agreement between the two methods, and note that our method is much faster, particularly for continuously-parametrized problems. Finally, we visualize the general representations of the first layers, and interpret them as generalized coordinates over the input domain.