XAMT: Cross-Framework API Matching for Testing Deep Learning Libraries

TL;DR

XAMT introduces a cross-framework fuzzing approach that matches and compares APIs across different deep learning libraries to detect bugs that are missed by intra-framework testing, improving library reliability.

Contribution

It proposes a novel cross-framework API matching and differential testing method for deep learning libraries, uncovering bugs undetectable by existing intra-framework fuzzing techniques.

Findings

Matched 839 APIs across five frameworks

Detected 17 bugs, 12 confirmed

Uncovered bugs that manifest across backends

Abstract

Deep learning powers critical applications such as autonomous driving, healthcare, and finance, where the correctness of underlying libraries is essential. Bugs in widely used deep learning APIs can propagate to downstream systems, causing serious consequences. While existing fuzzing techniques detect bugs through intra-framework testing across hardware backends (CPU vs. GPU), they may miss bugs that manifest identically across backends and thus escape detection under these strategies. To address this problem, we propose XAMT, a cross-framework fuzzing method that tests deep learning libraries by matching and comparing functionally equivalent APIs across different frameworks. XAMT matches APIs using similarity-based rules based on names, descriptions, and parameter structures. It then aligns inputs and applies variance-guided differential testing to detect bugs. We evaluated XAMT on five popular frameworks, including PyTorch, TensorFlow, Keras, Chainer, and JAX. XAMT matched 839 APIs and identified 238 matched API groups, and detected 17 bugs, 12 of which have been confirmed. Our results show that XAMT uncovers bugs undetectable by intra-framework testing, especially those that manifest consistently across backends. XAMT offers a complementary approach to existing methods and offers a new perspective on the testing of deep learning libraries.