Physarum Powered Differentiable Linear Programming Layers and Applications

TL;DR

This paper introduces a novel, efficient, and differentiable linear programming solver inspired by physarum dynamics, enabling its integration as a layer in deep neural networks for tasks like image segmentation and meta-learning.

Contribution

The paper presents a new physarum-inspired differentiable LP solver that is easy to implement, fast, and suitable for integration into neural networks, outperforming existing methods in certain tasks.

Findings

Performs comparably with projected gradient descent on segmentation

Outperforms differentiable CVXPY-SCS on meta-learning

Converges quickly without a feasible initial point

Abstract

Consider a learning algorithm, which involves an internal call to an optimization routine such as a generalized eigenvalue problem, a cone programming problem or even sorting. Integrating such a method as a layer(s) within a trainable deep neural network (DNN) in an efficient and numerically stable way is not straightforward -- for instance, only recently, strategies have emerged for eigendecomposition and differentiable sorting. We propose an efficient and differentiable solver for general linear programming problems which can be used in a plug and play manner within DNNs as a layer. Our development is inspired by a fascinating but not widely used link between dynamics of slime mold (physarum) and optimization schemes such as steepest descent. We describe our development and show the use of our solver in a video segmentation task and meta-learning for few-shot learning. We review the existing results and provide a technical analysis describing its applicability for our use cases. Our solver performs comparably with a customized projected gradient descent method on the first task and outperforms the differentiable CVXPY-SCS solver on the second task. Experiments show that our solver converges quickly without the need for a feasible initial point. Our proposal is easy to implement and can easily serve as layers whenever a learning procedure needs a fast approximate solution to a LP, within a larger network.