Studying the Impact of TensorFlow and PyTorch Bindings on Machine Learning Software Quality

TL;DR

This study investigates how using TensorFlow and PyTorch bindings in various programming languages affects machine learning software quality, focusing on correctness and efficiency, revealing that non-default bindings can improve training and inference times without sacrificing accuracy.

Contribution

First comprehensive analysis of multi-language bindings impact on ML software quality, demonstrating potential efficiency gains with maintained correctness.

Findings

Non-default bindings can match default Python accuracy.

Using non-default bindings can reduce training and inference time.

Bindings in C#, Rust, JavaScript can improve efficiency without accuracy loss.

Abstract

Bindings for machine learning frameworks (such as TensorFlow and PyTorch) allow developers to integrate a framework's functionality using a programming language different from the framework's default language (usually Python). In this paper, we study the impact of using TensorFlow and PyTorch bindings in C#, Rust, Python and JavaScript on the software quality in terms of correctness (training and test accuracy) and time cost (training and inference time) when training and performing inference on five widely used deep learning models. Our experiments show that a model can be trained in one binding and used for inference in another binding for the same framework without losing accuracy. Our study is the first to show that using a non-default binding can help improve machine learning software quality from the time cost perspective compared to the default Python binding while still achieving the same level of correctness.