Mutual Information Decay Curves and Hyper-Parameter Grid Search Design for Recurrent Neural Architectures

TL;DR

This paper introduces a mutual information-based method for designing hyper-parameter grid searches in recurrent neural networks, leading to state-of-the-art results on benchmark datasets.

Contribution

It proposes a novel approach using mutual information to analyze long-distance dependencies for hyper-parameter tuning in recurrent neural architectures.

Findings

Achieved state-of-the-art results on multiple benchmarks.

Demonstrated the effectiveness of mutual information in hyper-parameter design.

Provided insights into long-distance dependency analysis.

Abstract

We present an approach to design the grid searches for hyper-parameter optimization for recurrent neural architectures. The basis for this approach is the use of mutual information to analyze long distance dependencies (LDDs) within a dataset. We also report a set of experiments that demonstrate how using this approach, we obtain state-of-the-art results for DilatedRNNs across a range of benchmark datasets.