Multiscale Residual Mixture of PCA: Dynamic Dictionaries for Optimal Basis Learning

TL;DR

This paper introduces the Deep Residual Oja Network (DRON), a hierarchical deep learning framework for efficient over-complete dictionary learning and reconstruction, demonstrating exponential error reduction with depth.

Contribution

It proposes a novel deep residual approach for dictionary learning, linking it to existing methods and enabling efficient, non-optimization-based reconstruction.

Findings

Deep residual approach achieves exponential error decrease with depth.

DRON provides an efficient alternative to traditional dictionary learning methods.

Theoretical analysis shows deep frameworks facilitate over-complete dictionary representation.

Abstract

In this paper we are interested in the problem of learning an over-complete basis and a methodology such that the reconstruction or inverse problem does not need optimization. We analyze the optimality of the presented approaches, their link to popular already known techniques s.a. Artificial Neural Networks,k-means or Oja's learning rule. Finally, we will see that one approach to reach the optimal dictionary is a factorial and hierarchical approach. The derived approach lead to a formulation of a Deep Oja Network. We present results on different tasks and present the resulting very efficient learning algorithm which brings a new vision on the training of deep nets. Finally, the theoretical work shows that deep frameworks are one way to efficiently have over-complete (combinatorially large) dictionary yet allowing easy reconstruction. We thus present the Deep Residual Oja Network (DRON). We demonstrate that a recursive deep approach working on the residuals allow exponential decrease of the error w.r.t. the depth.