Inductive Inference in Supervised Classification

TL;DR

This paper explores Bayesian inductive inference methods for supervised classification, comparing classifiers based on de Finetti and Kingman exchangeability, and analyzing their asymptotic behaviors with large training and test datasets.

Contribution

It provides a comparative analysis of classifiers derived from different exchangeability assumptions and investigates their asymptotic properties in supervised learning.

Findings

Classifiers based on de Finetti exchangeability handle test items independently with infinite training data.

Classifiers based on partition exchangeability benefit from joint labeling of test items.

Inductive learning saturates as test data size approaches infinity.

Abstract

Inductive inference in supervised classification context constitutes to methods and approaches to assign some objects or items into different predefined classes using a formal rule that is derived from training data and possibly some additional auxiliary information. The optimality of such an assignment varies under different conditions due to intrinsic attributes of the objects being considered for such a task. One of these cases is when all the objects' features are discrete variables with a priori known categories. As another example, one can consider a modification of this case with a priori unknown categories. These two cases are the main focus of this thesis and based on Bayesian inductive theories, de Finetti type exchangeability is a suitable assumption that facilitates the derivation of classifiers in the former scenario. On the contrary, this type of exchangeability is not applicable in the latter case, instead, it is possible to utilise the partition exchangeability due to John Kingman. These two types of exchangeabilities are discussed and furthermore here I investigate inductive supervised classifiers based on both types of exchangeabilities. I further demonstrate that the classifiers based on de Finetti type exchangeability can optimally handle test items independently of each other in the presence of infinite amounts of training data while on the other hand, classifiers based on partition exchangeability still continue to benefit from joint labelling of all the test items. Additionally, it is shown that the inductive learning process for the simultaneous classifier saturates when the amount of test data tends to infinity.