Sum decomposition of divergence into three divergences

TL;DR

This paper presents a novel decomposition of divergence functions, expressing symmetric Bregman divergence as a sum of Jensen divergences and Bregman divergence, and further includes f-divergences explicitly.

Contribution

It introduces new sum decompositions of divergence functions, linking Bregman, Jensen, and f-divergences in a unified framework.

Findings

Symmetric Bregman divergence decomposes into Jensen divergences and Bregman divergence.

A new sum decomposition involving f-divergences is established.

The results unify different divergence measures in a common framework.

Abstract

Divergence functions play a key role as to measure the discrepancy between two points in the field of machine learning, statistics and signal processing. Well-known divergences are the Bregman divergences, the Jensen divergences and the f-divergences. In this paper, we show that the symmetric Bregman divergence can be decomposed into the sum of two types of Jensen divergences and the Bregman divergence. Furthermore, applying this result, we show another sum decomposition of divergence is possible which includes f-divergences explicitly.