On the Versatile Uses of Partial Distance Correlation in Deep Learning

TL;DR

This paper explores the use of partial distance correlation, a statistical measure, to compare neural network models' functions, offering a versatile tool for analysis, regularization, and improving model robustness.

Contribution

It introduces a practical method to deploy partial distance correlation for large-scale neural networks, enabling diverse applications like model comparison, disentangled representation learning, and robustness enhancement.

Findings

Partial distance correlation effectively compares neural network features.

The method serves as a versatile regularizer and constraint.

Applications include model conditioning, disentanglement, and adversarial robustness.

Abstract

Comparing the functional behavior of neural network models, whether it is a single network over time or two (or more networks) during or post-training, is an essential step in understanding what they are learning (and what they are not), and for identifying strategies for regularization or efficiency improvements. Despite recent progress, e.g., comparing vision transformers to CNNs, systematic comparison of function, especially across different networks, remains difficult and is often carried out layer by layer. Approaches such as canonical correlation analysis (CCA) are applicable in principle, but have been sparingly used so far. In this paper, we revisit a (less widely known) from statistics, called distance correlation (and its partial variant), designed to evaluate correlation between feature spaces of different dimensions. We describe the steps necessary to carry out its deployment for large scale models -- this opens the door to a surprising array of applications ranging from conditioning one deep model w.r.t. another, learning disentangled representations as well as optimizing diverse models that would directly be more robust to adversarial attacks. Our experiments suggest a versatile regularizer (or constraint) with many advantages, which avoids some of the common difficulties one faces in such analyses. Code is at https://github.com/zhenxingjian/Partial_Distance_Correlation.