On the differentiability of the value function of switched linear systems under arbitrary and controlled switching

TL;DR

This paper investigates the differentiability of value functions in switched linear systems, revealing they can be non-differentiable on dense sets even with smooth costs, impacting control and learning methods.

Contribution

It demonstrates that value functions in switched linear systems can be non-differentiable on dense sets, challenging assumptions in control and reinforcement learning.

Findings

Value functions are Lipschitz continuous with Lipschitz costs.

Value functions can be non-differentiable on dense subsets.

Implications for control and reinforcement learning algorithms.

Abstract

This paper studies the differentiability of the value function of switched linear systems under arbitrary switching and controlled switching, referred to as worst-case and optimal value functions respectively. First, we show that the value functions are Lipschitz continuous, when the cost function is Lipschitz continuous. Then, as the central contribution of this work, we show with examples that each of these functions can be non-differentiable on dense subsets of the state space, even if the cost function is smooth and Lipschitz continuous. This has implications for optimal control and reinforcement learning since it implies that the exact computation of these value functions requires templates involving functions that are non-differentiable on dense subsets.