On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning

TL;DR

This paper explores the mathematical properties of the softmax function using convex analysis and monotone operator theory, revealing its connection to the log-sum-exp function and implications for reinforcement learning.

Contribution

It establishes that softmax is the monotone gradient map of the log-sum-exp function and analyzes how the inverse temperature affects its Lipschitz and co-coercivity properties.

Findings

Softmax is the monotone gradient map of the log-sum-exp function.

Inverse temperature influences Lipschitz and co-coercivity properties.

Application demonstrated in game-theoretic reinforcement learning.

Abstract

In this paper, we utilize results from convex analysis and monotone operator theory to derive additional properties of the softmax function that have not yet been covered in the existing literature. In particular, we show that the softmax function is the monotone gradient map of the log-sum-exp function. By exploiting this connection, we show that the inverse temperature parameter determines the Lipschitz and co-coercivity properties of the softmax function. We then demonstrate the usefulness of these properties through an application in game-theoretic reinforcement learning.