Generalized Twin Gaussian Processes using Sharma-Mittal Divergence

TL;DR

This paper introduces a generalized regression framework using Sharma-Mittal divergence within Twin Gaussian Processes, enhancing predictive performance by leveraging a broader class of divergence measures.

Contribution

It presents the first application of Sharma-Mittal divergence to Twin Gaussian Processes, generalizing the traditional KL-based approach with theoretical analysis and empirical validation.

Findings

SMTGP outperforms KL-based TGP in predictions

The framework offers a larger model class through learnable parameters

Theoretical insights into properties of Sharma-Mittal divergence in TGP

Abstract

There has been a growing interest in mutual information measures due to their wide range of applications in Machine Learning and Computer Vision. In this paper, we present a generalized structured regression framework based on Shama-Mittal divergence, a relative entropy measure, which is introduced to the Machine Learning community in this work. Sharma-Mittal (SM) divergence is a generalized mutual information measure for the widely used R\'enyi, Tsallis, Bhattacharyya, and Kullback-Leibler (KL) relative entropies. Specifically, we study Sharma-Mittal divergence as a cost function in the context of the Twin Gaussian Processes (TGP)~\citep{Bo:2010}, which generalizes over the KL-divergence without computational penalty. We show interesting properties of Sharma-Mittal TGP (SMTGP) through a theoretical analysis, which covers missing insights in the traditional TGP formulation. However, we generalize this theory based on SM-divergence instead of KL-divergence which is a special case. Experimentally, we evaluated the proposed SMTGP framework on several datasets. The results show that SMTGP reaches better predictions than KL-based TGP, since it offers a bigger class of models through its parameters that we learn from the data.