Cascaded Compressed Sensing Networks: A Reversible Architecture for Layerwise Learning

TL;DR

This paper introduces a reversible, layerwise learning architecture using cascaded compressed sensing, eliminating the need for auxiliary networks and improving stability and performance in neural network training.

Contribution

The paper proposes a novel layerwise learning method using compressed sensing to achieve target propagation without auxiliary networks, enhancing stability and efficiency.

Findings

Achieves better performance than auxiliary network-based methods

Provides a stable, reversible architecture for layerwise learning

Demonstrates effective target propagation via compressed sensing

Abstract

Recently, the method that learns networks layer by layer has attracted increasing interest for its ease of analysis. For the method, the main challenge lies in deriving an optimization target for each layer by inversely propagating the global target of the network. The propagation problem is ill posed, due to involving the inversion of nonlinear activations from lowdimensional to high-dimensional spaces. To address the problem, the existing solution is to learn an auxiliary network to specially propagate the target. However, the network lacks stability, and moreover, it results in higher complexity for network learning. In the letter, we show that target propagation could be achieved by modeling the network s each layer with compressed sensing, without the need of auxiliary networks. Experiments show that the proposed method could achieve better performance than the auxiliary network-based method.