On the VC-dimension of neural networks with binary weights

TL;DR

This paper analyzes the VC-dimension of neural networks with binary weights, revealing that it is influenced by atypical pattern sets and providing bounds through enumeration and number theory.

Contribution

It offers the first detailed analysis of VC-dimension for binary-weight neural networks, combining enumeration and number theory methods.

Findings

VC-dimension determined by atypical pattern sets

Exact results for small systems via enumeration

Lower bounds for large systems using number theory

Abstract

We investigate the VC-dimension of the perceptron and simple two-layer networks like the committee- and the parity-machine with weights restricted to values $\pm1$. For binary inputs, the VC-dimension is determined by atypical pattern sets, i.e. it cannot be found by replica analysis or numerical Monte Carlo sampling. For small systems, exhaustive enumerations yield exact results. For systems that are too large for enumerations, number theoretic arguments give lower bounds for the VC-dimension. For the Ising perceptron, the VC-dimension is probably larger than $N/2$.