Toward Efficient Interactions between Python and Native Libraries

TL;DR

This paper identifies common inefficiency patterns in Python-native library interactions, introduces PieProf to diagnose these issues, and demonstrates significant speedups in real-world applications through targeted optimizations.

Contribution

The paper presents PieProf, a novel lightweight profiler that pinpoints interaction inefficiencies between Python and native libraries, enabling effective performance improvements.

Findings

Identified two common interaction inefficiency patterns.

PieProf effectively associates inefficiencies with high-level Python code.

Optimizations guided by PieProf achieved up to 6.3× speedup.

Abstract

Python has become a popular programming language because of its excellent programmability. Many modern software packages utilize Python for high-level algorithm design and depend on native libraries written in C/C++/Fortran for efficient computation kernels. Interaction between Python code and native libraries introduces performance losses because of the abstraction lying on the boundary of Python and native libraries. On the one side, Python code, typically run with interpretation, is disjoint from its execution behavior. On the other side, native libraries do not include program semantics to understand algorithm defects. To understand the interaction inefficiencies, we extensively study a large collection of Python software packages and categorize them according to the root causes of inefficiencies. We extract two inefficiency patterns that are common in interaction inefficiencies. Based on these patterns, we develop PieProf, a lightweight profiler, to pinpoint interaction inefficiencies in Python applications. The principle of PieProf is to measure the inefficiencies in the native execution and associate inefficiencies with high-level Python code to provide a holistic view. Guided by PieProf, we optimize 17 real-world applications, yielding speedups up to 6.3$\times$ on application level.