On-Line AdaTron Learning of Unlearnable Rules

TL;DR

This paper investigates the performance of on-line AdaTron learning for unlearnable, non-separable rules, showing that optimizing the learning rate enhances generalization error and approaches optimal performance.

Contribution

It demonstrates that while AdaTron is effective for learnable rules, optimizing the learning rate significantly improves its performance on unlearnable problems.

Findings

Optimized learning rate improves generalization error.

AdaTron does not achieve optimal performance on unlearnable rules without tuning.

Transfer function shape influences learning performance.

Abstract

We study the on-line AdaTron learning of linearly non-separable rules by a simple perceptron. Training examples are provided by a perceptron with a non-monotonic transfer function which reduces to the usual monotonic relation in a certain limit. We find that, although the on-line AdaTron learning is a powerful algorithm for the learnable rule, it does not give the best possible generalization error for unlearnable problems. Optimization of the learning rate is shown to greatly improve the performance of the AdaTron algorithm, leading to the best possible generalization error for a wide range of the parameter which controls the shape of the transfer function.)