A simple algorithm for output range analysis for deep neural networks

TL;DR

This paper introduces a Simulated Annealing-based algorithm for estimating the output ranges of deep neural networks, effectively handling non-linearity and complex constraints with proven convergence and robust empirical performance.

Contribution

It presents a novel, architecture-agnostic algorithm for output range analysis in DNNs, especially ResNets, with theoretical guarantees and practical efficiency.

Findings

Effective in estimating output ranges of complex DNNs

Converges reliably to global optima in non-convex landscapes

Demonstrates robustness across various architectures and constraints

Abstract

This paper presents a novel approach for the output range estimation problem in Deep Neural Networks (DNNs) by integrating a Simulated Annealing (SA) algorithm tailored to operate within constrained domains and ensure convergence towards global optima. The method effectively addresses the challenges posed by the lack of local geometric information and the high non-linearity inherent to DNNs, making it applicable to a wide variety of architectures, with a special focus on Residual Networks (ResNets) due to their practical importance. Unlike existing methods, our algorithm imposes minimal assumptions on the internal architecture of neural networks, thereby extending its usability to complex models. Theoretical analysis guarantees convergence, while extensive empirical evaluations-including optimization tests involving functions with multiple local minima-demonstrate the robustness of our algorithm in navigating non-convex response surfaces. The experimental results highlight the algorithm's efficiency in accurately estimating DNN output ranges, even in scenarios characterized by high non-linearity and complex constraints. For reproducibility, Python codes and datasets used in the experiments are publicly available through our GitHub repository.