Optimal Approximation -- Smoothness Tradeoffs for Soft-Max Functions

TL;DR

This paper explores the optimal balance between approximation accuracy and smoothness in soft-max functions, introducing novel mechanisms tailored for specific applications like machine learning, mechanism design, and differential privacy.

Contribution

It identifies and develops new soft-max functions with optimal tradeoffs for various approximation and smoothness measures, expanding beyond the traditional exponential mechanism.

Findings

The exponential mechanism has optimal tradeoff for expected additive approximation and Rényi Divergence smoothness.

The piecewise linear soft-max achieves optimal worst-case additive approximation and promotes sparsity.

The power mechanism offers improved multiplicative approximation and smoothness for private submodular optimization.

Abstract

A soft-max function has two main efficiency measures: (1) approximation - which corresponds to how well it approximates the maximum function, (2) smoothness - which shows how sensitive it is to changes of its input. Our goal is to identify the optimal approximation-smoothness tradeoffs for different measures of approximation and smoothness. This leads to novel soft-max functions, each of which is optimal for a different application. The most commonly used soft-max function, called exponential mechanism, has optimal tradeoff between approximation measured in terms of expected additive approximation and smoothness measured with respect to R\'enyi Divergence. We introduce a soft-max function, called "piecewise linear soft-max", with optimal tradeoff between approximation, measured in terms of worst-case additive approximation and smoothness, measured with respect to $\ell_q$-norm. The worst-case approximation guarantee of the piecewise linear mechanism enforces sparsity in the output of our soft-max function, a property that is known to be important in Machine Learning applications [Martins et al. '16, Laha et al. '18] and is not satisfied by the exponential mechanism. Moreover, the $\ell_q$-smoothness is suitable for applications in Mechanism Design and Game Theory where the piecewise linear mechanism outperforms the exponential mechanism. Finally, we investigate another soft-max function, called power mechanism, with optimal tradeoff between expected \textit{multiplicative} approximation and smoothness with respect to the R\'enyi Divergence, which provides improved theoretical and practical results in differentially private submodular optimization.