Two SVDs produce more focal deep learning representations

TL;DR

This paper introduces a novel, efficient method using two consecutive SVDs to produce focal representations in deep learning, potentially enhancing generalization and robustness.

Contribution

It proposes a new SVD-based approach that is more efficient and yields focal representations, a property hypothesized to improve neural network performance.

Findings

The method is more efficient than prior approaches.

Produced representations exhibit focality, supporting better generalization.

Focality may be key for neural network robustness.

Abstract

A key characteristic of work on deep learning and neural networks in general is that it relies on representations of the input that support generalization, robust inference, domain adaptation and other desirable functionalities. Much recent progress in the field has focused on efficient and effective methods for computing representations. In this paper, we propose an alternative method that is more efficient than prior work and produces representations that have a property we call focality -- a property we hypothesize to be important for neural network representations. The method consists of a simple application of two consecutive SVDs and is inspired by Anandkumar (2012).