On Differentiating Parameterized Argmin and Argmax Problems with Application to Bi-level Optimization

TL;DR

This paper investigates the differentiation of parameterized argmin and argmax problems within bi-level optimization, providing theoretical insights and motivating examples relevant to machine learning and computer vision.

Contribution

It offers a detailed analysis of differentiating argmin and argmax problems, highlighting distinctions and challenges in bi-level optimization contexts.

Findings

Differentiation methods vary between argmin and argmax problems.

Constraints significantly impact differentiation techniques.

The paper provides motivating examples illustrating key concepts.

Abstract

Some recent works in machine learning and computer vision involve the solution of a bi-level optimization problem. Here the solution of a parameterized lower-level problem binds variables that appear in the objective of an upper-level problem. The lower-level problem typically appears as an argmin or argmax optimization problem. Many techniques have been proposed to solve bi-level optimization problems, including gradient descent, which is popular with current end-to-end learning approaches. In this technical report we collect some results on differentiating argmin and argmax optimization problems with and without constraints and provide some insightful motivating examples.