# Random active path model of deep neural networks with diluted binary synapses

## TL;DR

This paper introduces a random active path model for deep neural networks with binary synapses, analyzing how connection removal affects collective properties and phase transitions, providing insights into generalization performance.

## Contribution

It proposes a novel random active path model to study deep neural networks with binary synapses under perturbation, revealing phase transition behavior.

## Key findings

- Identifies a critical perturbation value separating spin glass and paramagnetic regimes.
- Shows the paramagnetic phase likely has poor generalization.
- Demonstrates a first-order phase transition in the model.

## Abstract

Deep learning has become a powerful and popular tool for a variety of machine learning tasks. However, it is challenging to understand the mechanism of deep learning from a theoretical perspective. In this work, we propose a random active path model to study collective properties of deep neural networks with binary synapses, under the removal perturbation of connections between layers. In the model, the path from input to output is randomly activated, and the corresponding input unit constrains the weights along the path into the form of a $p$-weight interaction glass model. A critical value of the perturbation is observed to separate a spin glass regime from a paramagnetic regime, with the transition being of the first order. The paramagnetic phase is conjectured to have a poor generalization performance.