Some observations concerning Off Training Set (OTS) error

TL;DR

This paper discusses the limitations of a theorem related to Off Training Set (OTS) error, emphasizing its restricted applicability to non-overlapping data distributions, which are less relevant in typical machine learning scenarios.

Contribution

It clarifies the conditions under which the OTS error theorem applies, highlighting its limited relevance to real-world machine learning where data distributions often overlap.

Findings

OTS error can be large even with small training error

The theorem's applicability is limited to non-overlapping distributions

Real-world data typically involves overlapping distributions

Abstract

A form of generalisation error known as Off Training Set (OTS) error was recently introduced in [Wolpert, 1996b], along with a theorem showing that small training set error does not guarantee small OTS error, unless assumptions are made about the target function. Here it is shown that the applicability of this theorem is limited to models in which the distribution generating training data has no overlap with the distribution generating test data. It is argued that such a scenario is of limited relevance to machine learning.