Information Forests

TL;DR

Information Forests extend Random Forests by using an information divergence criterion for node splitting, aiming to partition data into highly informative subsets to improve classification confidence.

Contribution

The paper introduces a novel classification method that replaces entropy-based splits with divergence-based splits, enhancing the informativeness of data partitions.

Findings

Outperforms traditional Random Forests in classification confidence

Effectively partitions data into highly informative subsets

Relates to active and semi-supervised learning paradigms

Abstract

We describe Information Forests, an approach to classification that generalizes Random Forests by replacing the splitting criterion of non-leaf nodes from a discriminative one -- based on the entropy of the label distribution -- to a generative one -- based on maximizing the information divergence between the class-conditional distributions in the resulting partitions. The basic idea consists of deferring classification until a measure of "classification confidence" is sufficiently high, and instead breaking down the data so as to maximize this measure. In an alternative interpretation, Information Forests attempt to partition the data into subsets that are "as informative as possible" for the purpose of the task, which is to classify the data. Classification confidence, or informative content of the subsets, is quantified by the Information Divergence. Our approach relates to active learning, semi-supervised learning, mixed generative/discriminative learning.