SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services

TL;DR

This paper introduces a dynamic batching scheme based on semi-Markov decision processes to optimize response time and energy efficiency in batch service servers with parallel computing resources.

Contribution

It formulates the batching problem as an SMDP and proposes an approximate solution that significantly reduces computational complexity while balancing latency and power consumption.

Findings

SMDP-based policies outperform traditional batching methods.

The proposed scheme reduces space complexity by 63.5%.

The scheme effectively balances power and latency in various scenarios.

Abstract

For servers incorporating parallel computing resources, batching is a pivotal technique for providing efficient and economical services at scale. Parallel computing resources exhibit heightened computational and energy efficiency when operating with larger batch sizes. However, in the realm of online services, the adoption of a larger batch size may lead to longer response times. This paper aims to provide a dynamic batching scheme that delicately balances latency and efficiency. The system is modeled as a batch service queue with size-dependent service times. Then, the design of dynamic batching is formulated as a semi-Markov decision process (SMDP) problem, with the objective of minimizing the weighted sum of average response time and average power consumption. A method is proposed to derive an approximate optimal SMDP solution, representing the chosen dynamic batching policy. By introducing an abstract cost to reflect the impact of "tail" states, the space complexity and the time complexity of the procedure can decrease by 63.5% and 98%, respectively. Numerical results showcase the superiority of SMDP-based batching policies across various parameter setups. Additionally, the proposed scheme exhibits noteworthy flexibility in balancing power consumption and latency.