Knowledge-Based Version Incompatibility Detection for Deep Learning

TL;DR

This paper introduces a novel approach that leverages online discussions and a question-answering model to detect deep learning version incompatibilities beyond traditional dependency specifications, improving accuracy and coverage.

Contribution

It reformulates version incompatibility detection as a QA problem and constructs a knowledge graph from Stack Overflow discussions, enhancing detection of undocumented and hardware-related issues.

Findings

Achieves 84% accuracy in extracting version knowledge.

Detects 65% of known issues with 92% precision.

Outperforms state-of-the-art methods significantly.

Abstract

Version incompatibility issues are rampant when reusing or reproducing deep learning models and applications. Existing techniques are limited to library dependency specifications declared in PyPI. Therefore, these techniques cannot detect version issues due to undocumented version constraints or issues involving hardware drivers or OS. To address this challenge, we propose to leverage the abundant discussions of DL version issues from Stack Overflow to facilitate version incompatibility detection. We reformulate the problem of knowledge extraction as a Question-Answering (QA) problem and use a pre-trained QA model to extract version compatibility knowledge from online discussions. The extracted knowledge is further consolidated into a weighted knowledge graph to detect potential version incompatibilities when reusing a DL project. Our evaluation results show that (1) our approach can accurately extract version knowledge with 84% accuracy, and (2) our approach can accurately identify 65% of known version issues in 10 popular DL projects with a high precision (92%), while two state-of-the-art approaches can only detect 29% and 6% of these issues with 33% and 17% precision respectively.