Which Learning Algorithms Can Generalize Identity-Based Rules to Novel Inputs?

TL;DR

This paper introduces a framework to analyze when learning algorithms can generalize identity-based rules to new inputs, revealing many algorithms fail unless trained on nearly all possible stimuli, contrasting with human capabilities.

Contribution

It provides a new theoretical framework for understanding the limitations of algorithms in generalizing identity-based rules, supported by computational demonstrations.

Findings

Algorithms require extensive training data to generalize identity rules.

Multilayer neural networks fail to generalize identity rules with limited training.

Framework distinguishes algorithms capable of generalization from those that are not.

Abstract

We propose a novel framework for the analysis of learning algorithms that allows us to say when such algorithms can and cannot generalize certain patterns from training data to test data. In particular we focus on situations where the rule that must be learned concerns two components of a stimulus being identical. We call such a basis for discrimination an identity-based rule. Identity-based rules have proven to be difficult or impossible for certain types of learning algorithms to acquire from limited datasets. This is in contrast to human behaviour on similar tasks. Here we provide a framework for rigorously establishing which learning algorithms will fail at generalizing identity-based rules to novel stimuli. We use this framework to show that such algorithms are unable to generalize identity-based rules to novel inputs unless trained on virtually all possible inputs. We demonstrate these results computationally with a multilayer feedforward neural network.