Self-Attention through Kernel-Eigen Pair Sparse Variational Gaussian Processes

TL;DR

This paper introduces KEP-SVGP, a novel approach combining Gaussian processes with self-attention mechanisms that explicitly models asymmetry and reduces computational complexity, improving uncertainty estimation in large-scale data.

Contribution

The work proposes Kernel-Eigen Pair Sparse Variational Gaussian Processes (KEP-SVGP) to effectively model asymmetric attention kernels and lower computational costs in uncertainty-aware self-attention.

Findings

Achieves superior performance on various benchmarks.

Reduces time complexity through eigenfunction-based inference.

Effectively models asymmetric attention kernels.

Abstract

While the great capability of Transformers significantly boosts prediction accuracy, it could also yield overconfident predictions and require calibrated uncertainty estimation, which can be commonly tackled by Gaussian processes (GPs). Existing works apply GPs with symmetric kernels under variational inference to the attention kernel; however, omitting the fact that attention kernels are in essence asymmetric. Moreover, the complexity of deriving the GP posteriors remains high for large-scale data. In this work, we propose Kernel-Eigen Pair Sparse Variational Gaussian Processes (KEP-SVGP) for building uncertainty-aware self-attention where the asymmetry of attention kernels is tackled by Kernel SVD (KSVD) and a reduced complexity is acquired. Through KEP-SVGP, i) the SVGP pair induced by the two sets of singular vectors from KSVD w.r.t. the attention kernel fully characterizes the asymmetry; ii) using only a small set of adjoint eigenfunctions from KSVD, the derivation of SVGP posteriors can be based on the inversion of a diagonal matrix containing singular values, contributing to a reduction in time complexity; iii) an evidence lower bound is derived so that variational parameters and network weights can be optimized with it. Experiments verify our excellent performances and efficiency on in-distribution, distribution-shift and out-of-distribution benchmarks.