SMDP-Based Dynamic Batching for Improving Responsiveness and Energy Efficiency of Batch Services
Yaodan Xu, Sheng Zhou, Zhisheng Niu

TL;DR
This paper introduces a dynamic batching scheme based on semi-Markov decision processes to optimize response time and energy efficiency in batch service servers with parallel computing resources.
Contribution
It formulates the batching problem as an SMDP and proposes an approximate solution that significantly reduces computational complexity while balancing latency and power consumption.
Findings
SMDP-based policies outperform traditional batching methods.
The proposed scheme reduces space complexity by 63.5%.
The scheme effectively balances power and latency in various scenarios.
Abstract
For servers incorporating parallel computing resources, batching is a pivotal technique for providing efficient and economical services at scale. Parallel computing resources exhibit heightened computational and energy efficiency when operating with larger batch sizes. However, in the realm of online services, the adoption of a larger batch size may lead to longer response times. This paper aims to provide a dynamic batching scheme that delicately balances latency and efficiency. The system is modeled as a batch service queue with size-dependent service times. Then, the design of dynamic batching is formulated as a semi-Markov decision process (SMDP) problem, with the objective of minimizing the weighted sum of average response time and average power consumption. A method is proposed to derive an approximate optimal SMDP solution, representing the chosen dynamic batching policy. By…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Manufacturing and Logistics Optimization · Industrial Automation and Control Systems · IoT-based Smart Home Systems
Methodstravel james
