Dictionary Learning by Dynamical Neural Networks

TL;DR

This paper introduces a novel dynamical neural network approach for dictionary learning, leveraging top-down feedback and contrastive learning to enable local gradient computation and efficient parallel implementation.

Contribution

It presents a new dynamical neural network model that solves the l1-minimizing dictionary learning problem with provable gradient computation using local information.

Findings

Successful construction of a dynamical network for dictionary learning

Mathematical proof of true gradient computability in the network

Numerical results demonstrating effectiveness on dictionary learning tasks

Abstract

A dynamical neural network consists of a set of interconnected neurons that interact over time continuously. It can exhibit computational properties in the sense that the dynamical system's evolution and/or limit points in the associated state space can correspond to numerical solutions to certain mathematical optimization or learning problems. Such a computational system is particularly attractive in that it can be mapped to a massively parallel computer architecture for power and throughput efficiency, especially if each neuron can rely solely on local information (i.e., local memory). Deriving gradients from the dynamical network's various states while conforming to this last constraint, however, is challenging. We show that by combining ideas of top-down feedback and contrastive learning, a dynamical network for solving the l1-minimizing dictionary learning problem can be constructed, and the true gradients for learning are provably computable by individual neurons. Using spiking neurons to construct our dynamical network, we present a learning process, its rigorous mathematical analysis, and numerical results on several dictionary learning problems.