Characterizing the Weight Space for Different Learning Models

TL;DR

This paper explores the structure of the weight space in deep neural networks and associative memory models, revealing differences in vulnerability to adversarial attacks and providing insights into their solution landscapes.

Contribution

It characterizes the solution space of deep neural networks and associative memory models, comparing their robustness and learning mechanisms against adversarial examples.

Findings

Associative Memory Models are more resistant to adversarial attacks than Deep Neural Networks.

The solution space includes exact trained patterns, generalized patterns, and adversarial patterns.

Different learning paradigms exhibit distinct characteristics in their weight space.

Abstract

Deep Learning has become one of the primary research areas in developing intelligent machines. Most of the well-known applications (such as Speech Recognition, Image Processing and NLP) of AI are driven by Deep Learning. Deep Learning algorithms mimic human brain using artificial neural networks and progressively learn to accurately solve a given problem. But there are significant challenges in Deep Learning systems. There have been many attempts to make deep learning models imitate the biological neural network. However, many deep learning models have performed poorly in the presence of adversarial examples. Poor performance in adversarial examples leads to adversarial attacks and in turn leads to safety and security in most of the applications. In this paper we make an attempt to characterize the solution space of a deep neural network in terms of three different subsets viz. weights belonging to exact trained patterns, weights belonging to generalized pattern set and weights belonging to adversarial pattern sets. We attempt to characterize the solution space with two seemingly different learning paradigms viz. the Deep Neural Networks and the Dense Associative Memory Model, which try to achieve learning via quite different mechanisms. We also show that adversarial attacks are generally less successful against Associative Memory Models than Deep Neural Networks.