Theseus: A Library for Differentiable Nonlinear Optimization

TL;DR

Theseus is an open-source library built on PyTorch that enables efficient, application-agnostic differentiable nonlinear least squares optimization for robotics and vision tasks, supporting advanced features like sparse solvers and GPU acceleration.

Contribution

Theseus introduces a versatile, efficient framework for differentiable nonlinear optimization that integrates key features like sparse solvers and implicit differentiation, applicable across various domains.

Findings

Significant efficiency improvements demonstrated in multiple applications.

Better scalability achieved through support for sparse solvers and GPU acceleration.

Versatile framework applicable to diverse robotics and vision tasks.

Abstract

We present Theseus, an efficient application-agnostic open source library for differentiable nonlinear least squares (DNLS) optimization built on PyTorch, providing a common framework for end-to-end structured learning in robotics and vision. Existing DNLS implementations are application specific and do not always incorporate many ingredients important for efficiency. Theseus is application-agnostic, as we illustrate with several example applications that are built using the same underlying differentiable components, such as second-order optimizers, standard costs functions, and Lie groups. For efficiency, Theseus incorporates support for sparse solvers, automatic vectorization, batching, GPU acceleration, and gradient computation with implicit differentiation and direct loss minimization. We do extensive performance evaluation in a set of applications, demonstrating significant efficiency gains and better scalability when these features are incorporated. Project page: https://sites.google.com/view/theseus-ai