DFWLayer: Differentiable Frank-Wolfe Optimization Layer

TL;DR

This paper introduces DFWLayer, a differentiable layer based on the Frank-Wolfe algorithm, enabling efficient large-scale constrained optimization within neural networks with competitive accuracy and constraint adherence.

Contribution

It presents a novel differentiable layer leveraging the Frank-Wolfe method, avoiding projections and Hessian computations for efficient constrained optimization in neural networks.

Findings

DFWLayer achieves competitive solution accuracy.

It maintains strict adherence to constraints.

The layer is efficient for large-scale problems.

Abstract

Differentiable optimization has received a significant amount of attention due to its foundational role in the domain of machine learning based on neural networks. This paper proposes a differentiable layer, named Differentiable Frank-Wolfe Layer (DFWLayer), by rolling out the Frank-Wolfe method, a well-known optimization algorithm which can solve constrained optimization problems without projections and Hessian matrix computations, thus leading to an efficient way of dealing with large-scale convex optimization problems with norm constraints. Experimental results demonstrate that the DFWLayer not only attains competitive accuracy in solutions and gradients but also consistently adheres to constraints.