Regularizers versus Losses for Nonlinear Dimensionality Reduction: A Factored View with New Convex Relaxations

TL;DR

This paper presents a unified view of nonlinear dimensionality reduction methods as regularized loss minimization with singular value truncation, introducing new convex regularizers for improved manifold unfolding.

Contribution

It offers a factored perspective distinguishing loss and regularizer roles, and derives novel convex regularizers combining distance maximization with rank reduction.

Findings

New convex regularizers for manifold unfolding

A unified framework for nonlinear dimensionality reduction methods

Identification of a useful new loss for manifold unfolding

Abstract

We demonstrate that almost all non-parametric dimensionality reduction methods can be expressed by a simple procedure: regularized loss minimization plus singular value truncation. By distinguishing the role of the loss and regularizer in such a process, we recover a factored perspective that reveals some gaps in the current literature. Beyond identifying a useful new loss for manifold unfolding, a key contribution is to derive new convex regularizers that combine distance maximization with rank reduction. These regularizers can be applied to any loss.