Dissipative residual layers for unsupervised implicit parameterization of data manifolds

TL;DR

This paper introduces a novel unsupervised method for implicitly modeling data manifolds by estimating an attractive dynamical system using residual neural networks, with spectral localization and regularization to ensure local attraction.

Contribution

It presents a new approach combining residual neural networks with spectral localization to model data manifolds as attractive dynamical systems, including initialization and regularization techniques.

Findings

Effective in denoising tasks

Improves generative modeling performance

Ensures local attraction of the data manifold

Abstract

We propose an unsupervised technique for implicit parameterization of data manifolds. In our approach, the data is assumed to belong to a lower dimensional manifold in a higher dimensional space, and the data points are viewed as the endpoints of the trajectories originating outside the manifold. Under this assumption, the data manifold is an attractive manifold of a dynamical system to be estimated. We parameterize such a dynamical system with a residual neural network and propose a spectral localization technique to ensure it is locally attractive in the vicinity of data. We also present initialization and additional regularization of the proposed residual layers. % that we call dissipative bottlenecks. We mention the importance of the considered problem for the tasks of reinforcement learning and support our discussion with examples demonstrating the performance of the proposed layers in denoising and generative tasks.