Distant generalization by feedforward neural networks

TL;DR

This paper explores the ability of feedforward neural networks to generalize over long input distances, analyzing their structure and comparing their performance with support vector machines, highlighting potential issues with spurious generalizations.

Contribution

It provides an analysis of the neural network structure related to distant generalization and compares it with support vector machines, revealing potential limitations and problems.

Findings

Neural networks may struggle with distant generalization due to their structure.

Spurious or random generalizations can occur in neural networks.

Support vector machines show different generalization behaviors.

Abstract

This paper discusses the notion of generalization of training samples over long distances in the input space of a feedforward neural network. Such a generalization might occur in various ways, that differ in how great the contribution of different training features should be. The structure of a neuron in a feedforward neural network is analyzed and it is concluded, that the actual performance of the discussed generalization in such neural networks may be problematic -- while such neural networks might be capable for such a distant generalization, a random and spurious generalization may occur as well. To illustrate the differences in generalizing of the same function by different learning machines, results given by the support vector machines are also presented.