Hyperparameter Learning for Conditional Kernel Mean Embeddings with Rademacher Complexity Bounds

TL;DR

This paper introduces a scalable hyperparameter learning framework for conditional kernel mean embeddings using Rademacher complexity bounds, improving over existing methods and enabling integration with deep neural networks.

Contribution

It proposes a novel hyperparameter learning method based on Rademacher complexity bounds that avoids costly cross validation and supports scalable, non-approximate kernel tuning.

Findings

Outperforms existing hyperparameter tuning methods in experiments

Enables scalable kernel hyperparameter optimization without kernel approximations

Can be extended to learn deep neural network weights for better generalization

Abstract

Conditional kernel mean embeddings are nonparametric models that encode conditional expectations in a reproducing kernel Hilbert space. While they provide a flexible and powerful framework for probabilistic inference, their performance is highly dependent on the choice of kernel and regularization hyperparameters. Nevertheless, current hyperparameter tuning methods predominantly rely on expensive cross validation or heuristics that is not optimized for the inference task. For conditional kernel mean embeddings with categorical targets and arbitrary inputs, we propose a hyperparameter learning framework based on Rademacher complexity bounds to prevent overfitting by balancing data fit against model complexity. Our approach only requires batch updates, allowing scalable kernel hyperparameter tuning without invoking kernel approximations. Experiments demonstrate that our learning framework outperforms competing methods, and can be further extended to incorporate and learn deep neural network weights to improve generalization.