LEO: Learning Energy-based Models in Factor Graph Optimization

TL;DR

This paper introduces LEO, a novel energy-based learning method that enables end-to-end training of observation models within graph-based estimation frameworks, even when the optimizers are non-differentiable.

Contribution

LEO is the first approach to directly optimize observation models in graph optimization by energy-based learning, accommodating non-differentiable optimizers and improving estimation accuracy.

Findings

LEO achieves lower errors than baselines on navigation and pushing tasks.

LEO requires fewer samples to learn effective observation models.

LEO can handle non-differentiable graph optimizers effectively.

Abstract

We address the problem of learning observation models end-to-end for estimation. Robots operating in partially observable environments must infer latent states from multiple sensory inputs using observation models that capture the joint distribution between latent states and observations. This inference problem can be formulated as an objective over a graph that optimizes for the most likely sequence of states using all previous measurements. Prior work uses observation models that are either known a-priori or trained on surrogate losses independent of the graph optimizer. In this paper, we propose a method to directly optimize end-to-end tracking performance by learning observation models with the graph optimizer in the loop. This direct approach may appear, however, to require the inference algorithm to be fully differentiable, which many state-of-the-art graph optimizers are not. Our key insight is to instead formulate the problem as that of energy-based learning. We propose a novel approach, LEO, for learning observation models end-to-end with graph optimizers that may be non-differentiable. LEO alternates between sampling trajectories from the graph posterior and updating the model to match these samples to ground truth trajectories. We propose a way to generate such samples efficiently using incremental Gauss-Newton solvers. We compare LEO against baselines on datasets drawn from two distinct tasks: navigation and real-world planar pushing. We show that LEO is able to learn complex observation models with lower errors and fewer samples. Supplementary video: https://youtu.be/YqzlUPudfkA