DiTOX: Fault Detection and Localization in the ONNX Optimizer

TL;DR

DiTOX is an automated framework that systematically tests the correctness of the ONNX Optimizer, revealing numerous previously unknown issues and enhancing trust in model optimization processes.

Contribution

We introduce DiTOX, a novel differential testing framework for the ONNX Optimizer that detects, isolates, and reports faults, improving validation of AI model optimizers.

Findings

9.2% of models crashed or produced invalid results

30% of classification models showed output discrepancies

15 issues were uncovered, 14 of which were previously unknown

Abstract

The ONNX Optimizer, part of the official ONNX repository and widely adopted for graph-level model optimizations, is used by default to optimize ONNX models. Despite its popularity, its ability to preserve model correctness has not been systematically evaluated. We present DiTOX, an automated framework for comprehensively assessing the correctness of the ONNX Optimizer using differential testing, fault localization, and evaluation techniques that generalize to other compiler optimizers. DiTOX applies optimization passes to a corpus of ONNX models, executes both original and optimized versions on user-defined inputs, and detects discrepancies in behavior or optimizer failures. When divergences are observed, DiTOX isolates the responsible optimization pass through iterative, fine-grained analysis. We evaluated DiTOX on 130 models from the ONNX Model Hub spanning vision and language tasks. We found that 9.2% of model instances crashed the optimizer or produced invalid models under default settings. Moreover, output discrepancies occurred in 30% of classification models and 16.6% of object detection and segmentation models, while text-based models were largely robust. Overall, DiTOX uncovered 15 issues -- 14 previously unknown -- affecting 9 of the 47 optimization passes as well as the optimizer infrastructure. All issues were reported to the ONNX Optimizer developers. Our results demonstrate that DiTOX provides a simple and effective approach for validating AI model optimizers and is readily extensible beyond ONNX.