Notes on Kullback-Leibler Divergence and Likelihood

TL;DR

This paper explores the relationship between Kullback-Leibler divergence and likelihood theory, providing intuition and discussing its applications in neural coding, with a focus on understanding its fundamental role in information theory.

Contribution

It offers an intuitive explanation of KL divergence through likelihood theory and discusses its recent applications in neural coding literature.

Findings

KL divergence arises naturally from likelihood theory

Provides intuitive understanding of KL divergence

Highlights applications in neural coding

Abstract

The Kullback-Leibler (KL) divergence is a fundamental equation of information theory that quantifies the proximity of two probability distributions. Although difficult to understand by examining the equation, an intuition and understanding of the KL divergence arises from its intimate relationship with likelihood theory. We discuss how KL divergence arises from likelihood theory in an attempt to provide some intuition and reserve a rigorous (but rather simple) derivation for the appendix. Finally, we comment on recent applications of KL divergence in the neural coding literature and highlight its natural application.