Highway Value Iteration Networks

TL;DR

This paper introduces highway value iteration networks (VINs) that incorporate highway mechanisms to enable effective long-term planning with hundreds of layers, outperforming traditional VINs and deep neural networks.

Contribution

The paper embeds highway value iteration into VINs, allowing deep, trainable planning modules with enhanced information flow for long-term tasks.

Findings

Deep highway VINs outperform traditional VINs in long-term planning tasks.

Highway VINs can be trained effectively with hundreds of layers.

The approach improves information flow and gradient propagation in deep planning networks.

Abstract

Value iteration networks (VINs) enable end-to-end learning for planning tasks by employing a differentiable "planning module" that approximates the value iteration algorithm. However, long-term planning remains a challenge because training very deep VINs is difficult. To address this problem, we embed highway value iteration -- a recent algorithm designed to facilitate long-term credit assignment -- into the structure of VINs. This improvement augments the "planning module" of the VIN with three additional components: 1) an "aggregate gate," which constructs skip connections to improve information flow across many layers; 2) an "exploration module," crafted to increase the diversity of information and gradient flow in spatial dimensions; 3) a "filter gate" designed to ensure safe exploration. The resulting novel highway VIN can be trained effectively with hundreds of layers using standard backpropagation. In long-term planning tasks requiring hundreds of planning steps, deep highway VINs outperform both traditional VINs and several advanced, very deep NNs.