On Learning with LAD

TL;DR

This paper provides a theoretical explanation for LAD's resistance to overfitting by estimating its VC dimension, supported by empirical evidence, focusing on classifiers based on DNF Boolean functions with limited complexity.

Contribution

It offers a novel theoretical analysis of LAD's overfitting resistance through VC dimension estimation for specific DNF models.

Findings

LAD classifiers do not overfit despite optimization.

VC dimension estimates support LAD's generalization ability.

Empirical results confirm theoretical predictions.

Abstract

The logical analysis of data, LAD, is a technique that yields two-class classifiers based on Boolean functions having disjunctive normal form (DNF) representation. Although LAD algorithms employ optimization techniques, the resulting binary classifiers or binary rules do not lead to overfitting. We propose a theoretical justification for the absence of overfitting by estimating the Vapnik-Chervonenkis dimension (VC dimension) for LAD models where hypothesis sets consist of DNFs with a small number of cubic monomials. We illustrate and confirm our observations empirically.