Learning Variational Models with Unrolling and Bilevel Optimization

TL;DR

This paper compares bilevel optimization and algorithm unrolling for learning variational models, revealing that unrolling can outperform bilevel methods and highlighting the importance of stepsize tuning.

Contribution

It provides a theoretical analysis of both approaches using a simple toy model, showing conditions under which unrolling is more effective.

Findings

Unrolling can outperform bilevel optimization in certain settings.

Stepsize tuning significantly impacts unrolling performance.

Number of unrolled iterations has minor effect on results.

Abstract

In this paper we consider the problem of learning variational models in the context of supervised learning via risk minimization. Our goal is to provide a deeper understanding of the two approaches of learning of variational models via bilevel optimization and via algorithm unrolling. The former considers the variational model as a lower level optimization problem below the risk minimization problem, while the latter replaces the lower level optimization problem by an algorithm that solves said problem approximately. Both approaches are used in practice, but unrolling is much simpler from a computational point of view. To analyze and compare the two approaches, we consider a simple toy model, and compute all risks and the respective estimators explicitly. We show that unrolling can be better than the bilevel optimization approach, but also that the performance of unrolling can depend significantly on further parameters, sometimes in unexpected ways: While the stepsize of the unrolled algorithm matters a lot (and learning the stepsize gives a significant improvement), the number of unrolled iterations plays a minor role.