Sequential Dynamic Decision Making with Deep Neural Nets on a Test-Time Budget

TL;DR

This paper introduces an adaptive method that reduces test-time latency in deep neural network-based decision systems by dynamically switching between shallow and deep models based on state recognition, achieving up to 5X speed-up.

Contribution

It proposes a novel gating mechanism that selectively employs shallow networks in sequential decision making, balancing computational efficiency and performance.

Findings

Achieves up to 5X speed-up in decision-making tasks.

Maintains performance with minimal loss using the gating approach.

Demonstrates effectiveness across multiple applications.

Abstract

Deep neural network (DNN) based approaches hold significant potential for reinforcement learning (RL) and have already shown remarkable gains over state-of-art methods in a number of applications. The effectiveness of DNN methods can be attributed to leveraging the abundance of supervised data to learn value functions, Q-functions, and policy function approximations without the need for feature engineering. Nevertheless, the deployment of DNN-based predictors with very deep architectures can pose an issue due to computational and other resource constraints at test-time in a number of applications. We propose a novel approach for reducing the average latency by learning a computationally efficient gating function that is capable of recognizing states in a sequential decision process for which policy prescriptions of a shallow network suffices and deeper layers of the DNN have little marginal utility. The overall system is adaptive in that it dynamically switches control actions based on state-estimates in order to reduce average latency without sacrificing terminal performance. We experiment with a number of alternative loss-functions to train gating functions and shallow policies and show that in a number of applications a speed-up of up to almost 5X can be obtained with little loss in performance.