Can we learn gradients by Hamiltonian Neural Networks?

TL;DR

This paper introduces a meta-learning approach using Hamiltonian Neural Networks to learn gradients, demonstrating improved performance over traditional methods on artificial tasks and MNIST.

Contribution

It presents a novel meta-learner based on ODE neural networks that learns gradients, offering increased flexibility and automatic inductive bias for optimization tasks.

Findings

Outperforms LSTM-based meta-learner on artificial tasks and MNIST

Surpasses classic optimization methods on artificial tasks

Achieves comparable results to traditional methods on MNIST

Abstract

In this work, we propose a meta-learner based on ODE neural networks that learns gradients. This approach makes the optimizer is more flexible inducing an automatic inductive bias to the given task. Using the simplest Hamiltonian Neural Network we demonstrate that our method outperforms a meta-learner based on LSTM for an artificial task and the MNIST dataset with ReLU activations in the optimizee. Furthermore, it also surpasses the classic optimization methods for the artificial task and achieves comparable results for MNIST.