On Surjectivity of Neural Networks: Can you elicit any behavior from your model?

TL;DR

This paper investigates whether modern neural networks can generate any possible output, revealing that many architectures are almost always surjective, which has implications for model safety and vulnerability to adversarial attacks.

Contribution

It proves that key components of modern neural networks are almost always surjective, enabling inverse mappings and exposing inherent vulnerabilities.

Findings

Networks with pre-layer normalization are surjective.

Transformers and diffusion models admit inverse mappings.

Surjectivity implies potential for adversarial vulnerabilities.

Abstract

Given a trained neural network, can any specified output be generated by some input? Equivalently, does the network correspond to a function that is surjective? In generative models, surjectivity implies that any output, including harmful or undesirable content, can in principle be generated by the networks, raising concerns about model safety and jailbreak vulnerabilities. In this paper, we prove that many fundamental building blocks of modern neural architectures, such as networks with pre-layer normalization and linear-attention modules, are almost always surjective. As corollaries, widely used generative frameworks, including GPT-style transformers and diffusion models with deterministic ODE solvers, admit inverse mappings for arbitrary outputs. By studying surjectivity of these modern and commonly used neural architectures, we contribute a formalism that sheds light on their unavoidable vulnerability to a broad class of adversarial attacks.