Information Theory and its Relation to Machine Learning

TL;DR

This paper presents a new perspective on machine learning by analyzing four fundamental problems, emphasizing the importance of learning target selection, and explores the connection between information theory and ML.

Contribution

It introduces a novel framework for understanding ML problems, reviews existing links between information theory and ML, and proposes a conjecture for a unified mathematical interpretation.

Findings

A theorem relating similarity measures to information measures.

A review of information theoretical approaches in ML.

A conjecture for a unified theory of learning target selection.

Abstract

In this position paper, I first describe a new perspective on machine learning (ML) by four basic problems (or levels), namely, "What to learn?", "How to learn?", "What to evaluate?", and "What to adjust?". The paper stresses more on the first level of "What to learn?", or "Learning Target Selection". Towards this primary problem within the four levels, I briefly review the existing studies about the connection between information theoretical learning (ITL [1]) and machine learning. A theorem is given on the relation between the empirically-defined similarity measure and information measures. Finally, a conjecture is proposed for pursuing a unified mathematical interpretation to learning target selection.