Robust modal regression with direct log-density derivative estimation

TL;DR

This paper introduces a novel approach for robust modal regression by directly estimating the gradient of the modal regression risk using kernel and neural network methods, enabling effective optimization and improved robustness against noisy data.

Contribution

It proposes a new method for directly approximating the gradient of the modal regression risk, facilitating more accurate and robust modal regression.

Findings

The proposed methods outperform existing approaches on artificial datasets.

The neural network-based approach is compatible with stochastic gradient methods like Adam.

Theoretical proof of the hill-climbing property ensures convergence towards the mode.

Abstract

Modal regression is aimed at estimating the global mode (i.e., global maximum) of the conditional density function of the output variable given input variables, and has led to regression methods robust against heavy-tailed or skewed noises. The conditional mode is often estimated through maximization of the modal regression risk (MRR). In order to apply a gradient method for the maximization, the fundamental challenge is accurate approximation of the gradient of MRR, not MRR itself. To overcome this challenge, in this paper, we take a novel approach of directly approximating the gradient of MRR. To approximate the gradient, we develop kernelized and neural-network-based versions of the least-squares log-density derivative estimator, which directly approximates the derivative of the log-density without density estimation. With direct approximation of the MRR gradient, we first propose a modal regression method with kernels, and derive a new parameter update rule based on a fixed-point method. Then, the derived update rule is theoretically proved to have a monotonic hill-climbing property towards the conditional mode. Furthermore, we indicate that our approach of directly approximating the gradient is compatible with recent sophisticated stochastic gradient methods (e.g., Adam), and then propose another modal regression method based on neural networks. Finally, the superior performance of the proposed methods is demonstrated on various artificial and benchmark datasets.