On Error and Compression Rates for Prototype Rules

TL;DR

This paper analyzes the error and compression rates of prototype learning rules, especially OptiNet, in Euclidean spaces, demonstrating near optimal error rates and improved compression under certain conditions.

Contribution

It provides the first derivation of error and compression rates for OptiNet and introduces a new adaptive compression scheme that enhances prototype rule efficiency.

Findings

OptiNet achieves near minimax-optimal error rates.

OptiNet attains non-trivial compression rates.

Adaptive compression scheme improves efficiency under geometric margin conditions.

Abstract

We study the close interplay between error and compression in the non-parametric multiclass classification setting in terms of prototype learning rules. We focus in particular on a recently proposed compression-based learning rule termed OptiNet (Kontorovich, Sabato, and Urner 2016; Kontorovich, Sabato, and Weiss 2017; Hanneke et al. 2021). Beyond its computational merits, this rule has been recently shown to be universally consistent in any metric instance space that admits a universally consistent rule--the first learning algorithm known to enjoy this property. However, its error and compression rates have been left open. Here we derive such rates in the case where instances reside in Euclidean space under commonly posed smoothness and tail conditions on the data distribution. We first show that OptiNet achieves non-trivial compression rates while enjoying near minimax-optimal error rates. We then proceed to study a novel general compression scheme for further compressing prototype rules that locally adapts to the noise level without sacrificing accuracy. Applying it to OptiNet, we show that under a geometric margin condition, further gain in the compression rate is achieved. Experimental results comparing the performance of the various methods are presented.