Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing

TL;DR

This paper introduces a fine-grained, multi-objective resource optimization system for big data processing, significantly reducing latency and cost through hierarchical modeling and optimization techniques.

Contribution

It presents a novel architecture with instance-level modeling and optimization methods that improve resource management efficiency in big data systems.

Findings

Reduced latency by 37-72% in production workloads.

Lowered costs by 43-78% compared to existing systems.

Achieved fast optimization in 0.02-0.23 seconds.

Abstract

Big data processing at the production scale presents a highly complex environment for resource optimization (RO), a problem crucial for meeting performance goals and budgetary constraints of analytical users. The RO problem is challenging because it involves a set of decisions (the partition count, placement of parallel instances on machines, and resource allocation to each instance), requires multi-objective optimization (MOO), and is compounded by the scale and complexity of big data systems while having to meet stringent time constraints for scheduling. This paper presents a MaxCompute-based integrated system to support multi-objective resource optimization via fine-grained instance-level modeling and optimization. We propose a new architecture that breaks RO into a series of simpler problems, new fine-grained predictive models, and novel optimization methods that exploit these models to make effective instance-level recommendations in a hierarchical MOO framework. Evaluation using production workloads shows that our new RO system could reduce 37-72% latency and 43-78% cost at the same time, compared to the current optimizer and scheduler, while running in 0.02-0.23s.