Approximate Description Length, Covering Numbers, and VC Dimension
Amit Daniely, Gal Katzhendler

TL;DR
This paper investigates the relationship between Approximate Description Length (ADL) and classical complexity measures like Covering Numbers and VC Dimension, revealing their equivalence for real-valued functions but not for high-dimensional ranges.
Contribution
It establishes the conditions under which ADL aligns with classical complexity measures, enhancing understanding of neural network generalization bounds.
Findings
ADL is equivalent to Covering Numbers and VC Dimension for real-valued functions.
The equivalence between ADL and classical measures breaks down for functions with high-dimensional ranges.
Provides insights into the applicability of ADL in neural network complexity analysis.
Abstract
Recently, Daniely and Granot [arXiv:1910.05697] introduced a new notion of complexity called Approximate Description Length (ADL). They used it to derive novel generalization bounds for neural networks, that despite substantial work, were out of reach for more classical techniques such as discretization, Covering Numbers and Rademacher Complexity. In this paper we explore how ADL relates to classical notions of function complexity such as Covering Numbers and VC Dimension. We find that for functions whose range is the reals, ADL is essentially equivalent to these classical complexity measures. However, this equivalence breaks for functions with high dimensional range.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Applications · Machine Learning and Algorithms · Machine Learning and Data Classification
