A Generalization Bound for a Family of Implicit Networks

TL;DR

This paper derives a theoretical generalization bound for a broad class of implicit neural networks defined by contractive fixed point operators, addressing a gap in understanding their learning guarantees.

Contribution

It introduces a generalization bound for implicit networks using covering number and Rademacher complexity, expanding theoretical insights into these architectures.

Findings

Provides a bound based on covering numbers and Rademacher complexity.

Applies to a large family of implicit networks with contractive operators.

Bridges empirical success with theoretical understanding.

Abstract

Implicit networks are a class of neural networks whose outputs are defined by the fixed point of a parameterized operator. They have enjoyed success in many applications including natural language processing, image processing, and numerous other applications. While they have found abundant empirical success, theoretical work on its generalization is still under-explored. In this work, we consider a large family of implicit networks defined parameterized contractive fixed point operators. We show a generalization bound for this class based on a covering number argument for the Rademacher complexity of these architectures.