Lempel-Ziv Networks

TL;DR

This paper explores the adaptation of the Lempel-Ziv Jaccard Distance into a deep learning framework, the Lempel-Ziv Network, aiming to improve sequence processing for long sequences, but finds it does not outperform standard LSTMs.

Contribution

The paper introduces the Lempel-Ziv Network as a deep learning analog of LZJD for continuous domains, providing a proof of concept and discussing baseline tuning issues.

Findings

Lempel-Ziv Network does not outperform LSTMs on tested tasks.

Successful proof of concept achieved for deep-learning adaptation.

Highlights issues with baseline tuning in new research areas.

Abstract

Sequence processing has long been a central area of machine learning research. Recurrent neural nets have been successful in processing sequences for a number of tasks; however, they are known to be both ineffective and computationally expensive when applied to very long sequences. Compression-based methods have demonstrated more robustness when processing such sequences -- in particular, an approach pairing the Lempel-Ziv Jaccard Distance (LZJD) with the k-Nearest Neighbor algorithm has shown promise on long sequence problems (up to $T=200,000,000$ steps) involving malware classification. Unfortunately, use of LZJD is limited to discrete domains. To extend the benefits of LZJD to a continuous domain, we investigate the effectiveness of a deep-learning analog of the algorithm, the Lempel-Ziv Network. While we achieve successful proof of concept, we are unable to improve meaningfully on the performance of a standard LSTM across a variety of datasets and sequence processing tasks. In addition to presenting this negative result, our work highlights the problem of sub-par baseline tuning in newer research areas.