Container Profiler: Profiling Resource Utilization of Containerized Big Data Pipelines

TL;DR

The paper introduces the Container Profiler, a tool for detailed resource usage profiling of containerized tasks, enabling continuous monitoring and bottleneck identification in complex data pipelines.

Contribution

It presents a novel, comprehensive profiling tool that measures multiple resource metrics at various system levels with minimal overhead.

Findings

Effective profiling of bioinformatics pipeline stages

Negligible overhead of the profiling tool

Insights into resource utilization patterns

Abstract

This paper presents the Container Profiler, a software tool that measures and records the resource usage of any containerized task. Our tool profiles the CPU, memory, disk, and network utilization of containerized tasks collecting over fifty Linux operating system metrics at the virtual machine, container, and process levels. The Container Profiler supports performing time series profiling at a configurable sampling interval to enable continuous monitoring of the resources consumed by containerized tasks and pipelines. To investigate the utility of the Container Profiler, we profile the resource utilization requirements of a multi-stage bioinformatics analytical pipeline (RNA sequencing using unique molecular identifiers). We examine profiling metrics to assess patterns of CPU, disk, and network resource utilization across the different stages of the pipeline. We also quantify the profiling overhead of our Container Profiler tool to assess the impact of profiling a running pipeline with different levels of profiling granularity verifying that impacts are negligible. The Container Profiler provides a useful tool that can be used to continuously monitor the resource consumption of long and complex containerized applications that run locally or on the cloud. This can help identify bottlenecks where more resources are needed to improve performance.