Nalin: Learning from Runtime Behavior to Find Name-Value Inconsistencies in Jupyter Notebooks

TL;DR

Nalin is a novel technique that uses dynamic analysis and neural machine learning to automatically detect name-value inconsistencies in Python code, improving code understandability and correctness.

Contribution

It introduces the first formulation of name-value inconsistency detection as a classification problem using runtime data and machine learning.

Findings

High accuracy in detecting inconsistencies (precision 80%, recall 76%)

Effective on large real-world Python programs

Complements existing code analysis techniques

Abstract

Variable names are important to understand and maintain code. If a variable name and the value stored in the variable do not match, then the program suffers from a name-value inconsistency, which is due to one of two situations that developers may want to fix: Either a correct value is referred to through a misleading name, which negatively affects code understandability and maintainability, or the correct name is bound to a wrong value, which may cause unexpected runtime behavior. Finding name-value inconsistencies is hard because it requires an understanding of the meaning of names and knowledge about the values assigned to a variable at runtime. This paper presents Nalin, a technique to automatically detect name-value inconsistencies. The approach combines a dynamic analysis that tracks assignments of values to names with a neural machine learning model that predicts whether a name and a value fit together. To the best of our knowledge, this is the first work to formulate the problem of finding coding issues as a classification problem over names and runtime values. We apply Nalin to 106,652 real-world Python programs, where meaningful names are particularly important due to the absence of statically declared types. Our results show that the classifier detects name-value inconsistencies with high accuracy, that the warnings reported by Nalin have a precision of 80% and a recall of 76% w.r.t. a ground truth created in a user study, and that our approach complements existing techniques for finding coding issues.