Robust Estimation for Kernel Exponential Families with Smoothed Total Variation Distances

TL;DR

This paper introduces a robust estimation method for kernel exponential families using smoothed total variation distances, demonstrating improved robustness against distribution contamination and analyzing Monte Carlo approximation accuracy.

Contribution

It proposes the STV-based estimator for kernel exponential families and provides theoretical analysis of its robustness and Monte Carlo approximation performance.

Findings

STV-based estimator is robust against distribution contamination.

Theoretical guarantees for robustness in kernel exponential families.

Analysis of Monte Carlo approximation accuracy.

Abstract

In statistical inference, we commonly assume that samples are independent and identically distributed from a probability distribution included in a pre-specified statistical model. However, such an assumption is often violated in practice. Even an unexpected extreme sample called an {\it outlier} can significantly impact classical estimators. Robust statistics studies how to construct reliable statistical methods that efficiently work even when the ideal assumption is violated. Recently, some works revealed that robust estimators such as Tukey's median are well approximated by the generative adversarial net (GAN), a popular learning method for complex generative models using neural networks. GAN is regarded as a learning method using integral probability metrics (IPM), which is a discrepancy measure for probability distributions. In most theoretical analyses of Tukey's median and its GAN-based approximation, however, the Gaussian or elliptical distribution is assumed as the statistical model. In this paper, we explore the application of GAN-like estimators to a general class of statistical models. As the statistical model, we consider the kernel exponential family that includes both finite and infinite-dimensional models. To construct a robust estimator, we propose the smoothed total variation (STV) distance as a class of IPMs. Then, we theoretically investigate the robustness properties of the STV-based estimators. Our analysis reveals that the STV-based estimator is robust against the distribution contamination for the kernel exponential family. Furthermore, we analyze the prediction accuracy of a Monte Carlo approximation method, which circumvents the computational difficulty of the normalization constant.