Binary-level Software Compatibility Tool Agreement

TL;DR

This paper evaluates the agreement and effectiveness of various binary-level ABI compatibility tools through extensive experiments, providing insights and guidance for developers on assessing ABI compatibility.

Contribution

It systematically tests and compares well-known ABI compatibility tools, highlighting their strengths, weaknesses, and providing empirical guidance for their use.

Findings

High variability in tool predictions

Some tools provide detailed underlying issue insights

Guidance for selecting and improving ABI testing tools

Abstract

Application Binary Interface (ABI) compatibility is essential for system or software updates to ensure that libraries continue to function. Tools that can assess a binary or library ABI can thus be used to make predictions about compatibility, and predict downstream bugs by informing developers and users about issues. In this work, we are interested in describing a set of well-known tools for assessing ABI, and testing them in a controlled set experiments to assess tool agreement. We run 7660 smaller experiments across tools (N=30,640 total results) to evaluate not only predictions, but also each tool's ability to provide detail about underlying issues. In this work, along with highlighting the problem of assessing ABI compatibility and critiquing the pros and cons of currently available tools, we provide guidance to developers interested to test ABI based on our empirical results and suggestions for future work.