Sparse solutions of the kernel herding algorithm by improved gradient approximation

TL;DR

This paper introduces a modified kernel herding algorithm that improves sparsity and convergence speed in kernel quadrature, supported by theoretical analysis and numerical experiments demonstrating enhanced efficiency and stability.

Contribution

The study proposes new gradient approximation algorithms for kernel herding, achieving sparser solutions and faster convergence while maintaining the method's stability and efficiency.

Findings

Enhanced sparsity of solutions in kernel quadrature.

Faster convergence speeds with the proposed algorithms.

Numerical results confirm improved computational efficiency.

Abstract

The kernel herding algorithm is used to construct quadrature rules in a reproducing kernel Hilbert space (RKHS). While the computational efficiency of the algorithm and stability of the output quadrature formulas are advantages of this method, the convergence speed of the integration error for a given number of nodes is slow compared to that of other quadrature methods. In this paper, we propose a modified kernel herding algorithm whose framework was introduced in a previous study and aim to obtain sparser solutions while preserving the advantages of standard kernel herding. In the proposed algorithm, the negative gradient is approximated by several vertex directions, and the current solution is updated by moving in the approximate descent direction in each iteration. We show that the convergence speed of the integration error is directly determined by the cosine of the angle between the negative gradient and approximate gradient. Based on this, we propose new gradient approximation algorithms and analyze them theoretically, including through convergence analysis. In numerical experiments, we confirm the effectiveness of the proposed algorithms in terms of sparsity of nodes and computational efficiency. Moreover, we provide a new theoretical analysis of the kernel quadrature rules with fully-corrective weights, which realizes faster convergence speeds than those of previous studies.