Linearly-constrained nonsmooth optimization for training autoencoders

TL;DR

This paper introduces a new linearly constrained nonsmooth optimization approach with an $l_1$-norm penalty for training autoencoders, demonstrating theoretical equivalence and practical efficiency through convergence analysis and numerical experiments.

Contribution

It develops a novel linearly constrained regularized model with $l_1$-norm penalty for autoencoder training, along with a smoothing proximal gradient algorithm with proven convergence.

Findings

Algorithm converges to a generalized d-stationary point.

Numerical experiments show high efficiency and robustness.

The model shares minimizers with the original regularized model.

Abstract

A regularized minimization model with $l_1$-norm penalty (RP) is introduced for training the autoencoders that belong to a class of two-layer neural networks. We show that the RP can act as an exact penalty model which shares the same global minimizers, local minimizers, and d(irectional)-stationary points with the original regularized model under mild conditions. We construct a bounded box region that contains at least one global minimizer of the RP, and propose a linearly constrained regularized minimization model with $l_1$-norm penalty (LRP) for training autoencoders. A smoothing proximal gradient algorithm is designed to solve the LRP. Convergence of the algorithm to a generalized d-stationary point of the RP and LRP is delivered. Comprehensive numerical experiments convincingly illustrate the efficiency as well as the robustness of the proposed algorithm.