$\lambda$-Regularized A-Optimal Design and its Approximation by $\lambda$-Regularized Proportional Volume Sampling

TL;DR

This paper introduces a $oldsymbol{ ext{lambda}}$-regularized proportional volume sampling algorithm for the $oldsymbol{ ext{A}}$-optimal design problem, extending previous work to include regularization and providing approximation guarantees with applications in ridge regression.

Contribution

It generalizes the proportional volume sampling algorithm to $oldsymbol{ ext{lambda}}$-regularized cases, offering improved approximation bounds for the $A$-optimal design problem with regularization.

Findings

Achieves a $(1+rac{oldsymbol{ extepsilon}}{oldsymbol{ extsqrt{1+oldsymbol{ extlambda'}}}})$-approximation.

Extends previous bounds to $oldsymbol{ extlambda}>0$, approaching optimality as $oldsymbol{ extlambda} ightarrowoldsymbol{ extinfty}$.

Provides polynomial-time implementation of the sampling algorithm.

Abstract

In this work, we study the $\lambda$-regularized $A$-optimal design problem and introduce the $\lambda$-regularized proportional volume sampling algorithm, generalized from [Nikolov, Singh, and Tantipongpipat, 2019], for this problem with the approximation guarantee that extends upon the previous work. In this problem, we are given vectors $v_1,\ldots,v_n\in\mathbb{R}^d$ in $d$ dimensions, a budget $k\leq n$, and the regularizer parameter $\lambda\geq0$, and the goal is to find a subset $S\subseteq [n]$ of size $k$ that minimizes the trace of $\left(\sum_{i\in S}v_iv_i^\top + \lambda I_d\right)^{-1}$ where $I_d$ is the $d\times d$ identity matrix. The problem is motivated from optimal design in ridge regression, where one tries to minimize the expected squared error of the ridge regression predictor from the true coefficient in the underlying linear model. We introduce $\lambda$-regularized proportional volume sampling and give its polynomial-time implementation to solve this problem. We show its $(1+\frac{\epsilon}{\sqrt{1+\lambda'}})$-approximation for $k=\Omega\left(\frac d\epsilon+\frac{\log 1/\epsilon}{\epsilon^2}\right)$ where $\lambda'$ is proportional to $\lambda$, extending the previous bound in [Nikolov, Singh, and Tantipongpipat, 2019] to the case $\lambda>0$ and obtaining asymptotic optimality as $\lambda\rightarrow \infty$.