Accelerating Deep Learning Classification with Error-controlled Approximate-key Caching

TL;DR

This paper introduces an approximate-key caching method with error correction to speed up deep learning classification tasks in networking, balancing reduced inference costs with controlled accuracy loss.

Contribution

It proposes a novel caching paradigm called approximate-key caching combined with an auto-refresh error correction algorithm for deep learning inference acceleration.

Findings

Reduces deep learning inference workload and increases throughput.

Analytical modeling and trace-driven evaluation demonstrate effectiveness.

Outperforms state-of-the-art similarity caching methods.

Abstract

While Deep Learning (DL) technologies are a promising tool to solve networking problems that map to classification tasks, their computational complexity is still too high with respect to real-time traffic measurements requirements. To reduce the DL inference cost, we propose a novel caching paradigm, that we named approximate-key caching, which returns approximate results for lookups of selected input based on cached DL inference results. While approximate cache hits alleviate DL inference workload and increase the system throughput, they however introduce an approximation error. As such, we couple approximate-key caching with an error-correction principled algorithm, that we named auto-refresh. We analytically model our caching system performance for classic LRU and ideal caches, we perform a trace-driven evaluation of the expected performance, and we compare the benefits of our proposed approach with the state-of-the-art similarity caching -- testifying the practical interest of our proposal.