Advancing the Understanding of Fixed Point Iterations in Deep Neural Networks: A Detailed Analytical Study

TL;DR

This paper provides a comprehensive analytical study of fixed point iterations in deep neural networks, establishing conditions for multiple fixed points and exploring robustness, thereby deepening understanding of neural network stabilization phenomena.

Contribution

It introduces new theoretical conditions for the existence of multiple fixed points in looped neural networks and extends analysis to robust fixed point iterations, supported by empirical case studies.

Findings

Multiple fixed points can exist in looped neural networks under certain conditions.

Robust fixed points can number up to 2^d, where d is the feature dimension.

Preliminary empirical results support the theoretical analysis.

Abstract

Recent empirical studies have identified fixed point iteration phenomena in deep neural networks, where the hidden state tends to stabilize after several layers, showing minimal change in subsequent layers. This observation has spurred the development of practical methodologies, such as accelerating inference by bypassing certain layers once the hidden state stabilizes, selectively fine-tuning layers to modify the iteration process, and implementing loops of specific layers to maintain fixed point iterations. Despite these advancements, the understanding of fixed point iterations remains superficial, particularly in high-dimensional spaces, due to the inadequacy of current analytical tools. In this study, we conduct a detailed analysis of fixed point iterations in a vector-valued function modeled by neural networks. We establish a sufficient condition for the existence of multiple fixed points of looped neural networks based on varying input regions. Additionally, we expand our examination to include a robust version of fixed point iterations. To demonstrate the effectiveness and insights provided by our approach, we provide case studies that looped neural networks may exist $2^d$ number of robust fixed points under exponentiation or polynomial activation functions, where $d$ is the feature dimension. Furthermore, our preliminary empirical results support our theoretical findings. Our methodology enriches the toolkit available for analyzing fixed point iterations of deep neural networks and may enhance our comprehension of neural network mechanisms.