Evolino for recurrent support vector machines

TL;DR

This paper introduces Evoke, a novel recurrent SVM-like model with internal states that learns to classify context-sensitive languages and outperforms existing RNNs in time series prediction tasks.

Contribution

It presents Evoke, a new method combining SVMs and recurrent neural networks with internal adaptive states, capable of handling long-term dependencies in sequences.

Findings

Evoke can classify context-sensitive languages.

Evoke outperforms recent RNNs on time series prediction.

Evoke integrates kernel-based outputs with recurrent structures.

Abstract

Traditional Support Vector Machines (SVMs) need pre-wired finite time windows to predict and classify time series. They do not have an internal state necessary to deal with sequences involving arbitrary long-term dependencies. Here we introduce a new class of recurrent, truly sequential SVM-like devices with internal adaptive states, trained by a novel method called EVOlution of systems with KErnel-based outputs (Evoke), an instance of the recent Evolino class of methods. Evoke evolves recurrent neural networks to detect and represent temporal dependencies while using quadratic programming/support vector regression to produce precise outputs. Evoke is the first SVM-based mechanism learning to classify a context-sensitive language. It also outperforms recent state-of-the-art gradient-based recurrent neural networks (RNNs) on various time series prediction tasks.