Query Time Optimized Deep Learning Based Video Inference System

TL;DR

This paper presents a method to reduce query time in a deep learning video inference system by saving intermediate neural network outputs, achieving a 20% speedup through a space-time trade-off.

Contribution

It introduces a novel approach of saving middle layer outputs to optimize query time in video inference systems, demonstrated with prototype implementations.

Findings

Achieved 20% reduction in query time

Validated the approach with prototype systems

Demonstrated the effectiveness of space-time trade-off

Abstract

This is a project report about how we tune Focus[1], a video inference system that provides low cost and low latency, through two phases. In this report, we will decrease the query time by saving the middle layer output of the neural network. This is a trade-off strategy that involves using more space to save time. We show how this scheme works using prototype systems, and it saves 20% of the time. The code repository URL is here, https://github.com/iphyer/CS744 FocousIngestOpt.