Revisiting Generalization Power of a DNN in Terms of Symbolic Interactions

TL;DR

This paper investigates the generalization capabilities of deep neural networks by analyzing the role of interactions, revealing distinct distribution patterns for generalizable versus non-generalizable interactions and providing a method to disentangle them.

Contribution

It introduces a novel interaction-based perspective on DNN generalization, with a theory that distinguishes between types of interactions and matches real network behaviors.

Findings

Generalizable interactions follow a decay-shaped distribution.

Non-generalizable interactions follow a spindle-shaped distribution.

The theory effectively disentangles interaction types in DNNs.

Abstract

This paper aims to analyze the generalization power of deep neural networks (DNNs) from the perspective of interactions. Unlike previous analysis of a DNN's generalization power in a highdimensional feature space, we find that the generalization power of a DNN can be explained as the generalization power of the interactions. We found that the generalizable interactions follow a decay-shaped distribution, while non-generalizable interactions follow a spindle-shaped distribution. Furthermore, our theory can effectively disentangle these two types of interactions from a DNN. We have verified that our theory can well match real interactions in a DNN in experiments.