Reconstruction of training samples from loss functions

TL;DR

This paper introduces a mathematical framework using algebraic geometry to analyze deep neural network loss functions, demonstrating that training samples can be reconstructed from loss surface data, revealing potential security risks.

Contribution

It develops a novel algebraic geometric approach with virtual polynomials to analyze loss surfaces, enabling reconstruction of training inputs and network architecture from loss data.

Findings

Training samples can be reconstructed up to scalar multiples from loss surface points.

Loss surface structures depend on network shape, not on training data.

The framework exposes risks of information leakage in neural network training.

Abstract

This paper presents a new mathematical framework to analyze the loss functions of deep neural networks with ReLU functions. Furthermore, as as application of this theory, we prove that the loss functions can reconstruct the inputs of the training samples up to scalar multiplication (as vectors) and can provide the number of layers and nodes of the deep neural network. Namely, if we have all input and output of a loss function (or equivalently all possible learning process), for all input of each training sample $x_i \in \mathbb{R}^n$, we can obtain vectors $x'_i\in \mathbb{R}^n$ satisfying $x_i=c_ix'_i$ for some $c_i \neq 0$. To prove theorem, we introduce the notion of virtual polynomials, which are polynomials written as the output of a node in a deep neural network. Using virtual polynomials, we find an algebraic structure for the loss surfaces, called semi-algebraic sets. We analyze these loss surfaces from the algebro-geometric point of view. Factorization of polynomials is one of the most standard ideas in algebra. Hence, we express the factorization of the virtual polynomials in terms of their active paths. This framework can be applied to the leakage problem in the training of deep neural networks. The main theorem in this paper indicates that there are many risks associated with the training of deep neural networks. For example, if we have N (the dimension of weight space) + 1 nonsmooth points on the loss surface, which are sufficiently close to each other, we can obtain the input of training sample up to scalar multiplication. We also point out that the structures of the loss surfaces depend on the shape of the deep neural network and not on the training samples.