A Review of Feature Selection Methods Based on Mutual Information

TL;DR

This paper reviews information theoretic feature selection methods, clarifying key concepts and providing a unifying framework to understand various heuristic criteria, while highlighting open challenges in the field.

Contribution

It offers a comprehensive overview of feature relevance, redundancy, and complementarity, and introduces a unifying theoretical framework for feature selection methods.

Findings

Clarification of key information theoretic concepts

Unifying framework for heuristic feature selection criteria

Identification of open problems in feature selection research

Abstract

In this work we present a review of the state of the art of information theoretic feature selection methods. The concepts of feature relevance, redundance and complementarity (synergy) are clearly defined, as well as Markov blanket. The problem of optimal feature selection is defined. A unifying theoretical framework is described, which can retrofit successful heuristic criteria, indicating the approximations made by each method. A number of open problems in the field are presented.