# Population-based Global Optimisation Methods for Learning Long-term Dependencies with RNNs

## TL;DR

This paper investigates the use of population-based global optimisation methods, specifically evolution strategies and particle swarm optimisation, to improve training of RNNs for long-term dependency learning in time-series data, showing promising results.

## Contribution

It introduces the application of gradient-free PBO techniques to RNN training for long-term dependencies, demonstrating their effectiveness over traditional methods.

## Key findings

- ES shows consistent performance improvements
- PBO methods outperform traditional gradient-based optimisation
- Effective in volatility forecasting tasks

## Abstract

Despite recent innovations in network architectures and loss functions, training RNNs to learn long-term dependencies remains difficult due to challenges with gradient-based optimisation methods. Inspired by the success of Deep Neuroevolution in reinforcement learning (Such et al. 2017), we explore the use of gradient-free population-based global optimisation (PBO) techniques -- training RNNs to capture long-term dependencies in time-series data. Testing evolution strategies (ES) and particle swarm optimisation (PSO) on an application in volatility forecasting, we demonstrate that PBO methods lead to performance improvements in general, with ES exhibiting the most consistent results across a variety of architectures.