DeepCache: Principled Cache for Mobile Deep Vision

TL;DR

DeepCache is a cache system designed for mobile deep vision that exploits temporal locality in video streams to improve inference efficiency, reduce energy consumption, and operate seamlessly with existing models.

Contribution

It introduces a principled cache design that leverages video and model structure without requiring model modifications or manual tuning.

Findings

Average inference time reduced by 18%

Maximum inference time reduction of 47%

System energy consumption decreased by 20%

Abstract

We present DeepCache, a principled cache design for deep learning inference in continuous mobile vision. DeepCache benefits model execution efficiency by exploiting temporal locality in input video streams. It addresses a key challenge raised by mobile vision: the cache must operate under video scene variation, while trading off among cacheability, overhead, and loss in model accuracy. At the input of a model, DeepCache discovers video temporal locality by exploiting the video's internal structure, for which it borrows proven heuristics from video compression; into the model, DeepCache propagates regions of reusable results by exploiting the model's internal structure. Notably, DeepCache eschews applying video heuristics to model internals which are not pixels but high-dimensional, difficult-to-interpret data. Our implementation of DeepCache works with unmodified deep learning models, requires zero developer's manual effort, and is therefore immediately deployable on off-the-shelf mobile devices. Our experiments show that DeepCache saves inference execution time by 18% on average and up to 47%. DeepCache reduces system energy consumption by 20% on average.