Internal Representations of Vision Models Through the Lens of Frames on Data Manifolds

TL;DR

This paper introduces a novel 'neural frame' approach to analyze how deep learning models process local data variations, revealing insights into invariance, generalization, and the effects of training choices.

Contribution

It proposes a new geometric framework for understanding internal representations in vision models, linking perturbations to model behavior across layers.

Findings

Training with augmentation induces model invariance.

Trade-off observed between adversarial robustness and generalization.

Layer-wise processing of local variations offers new insights.

Abstract

While the last five years have seen considerable progress in understanding the internal representations of deep learning models, many questions remain. This is especially true when trying to understand the impact of model design choices, such as model architecture or training algorithm, on hidden representation geometry and dynamics. In this work we present a new approach to studying such representations inspired by the idea of a frame on the tangent bundle of a manifold. Our construction, which we call a neural frame, is formed by assembling a set of vectors representing specific types of perturbations of a data point, for example infinitesimal augmentations, noise perturbations, or perturbations produced by a generative model, and studying how these change as they pass through a network. Using neural frames, we make observations about the way that models process, layer-by-layer, specific modes of variation within a small neighborhood of a datapoint. Our results provide new perspectives on a number of phenomena, such as the manner in which training with augmentation produces model invariance or the proposed trade-off between adversarial training and model generalization.