Cross-Domain Evaluation of a Deep Learning-Based Type Inference System

TL;DR

This paper evaluates the cross-domain generalization of a deep learning-based type inference system, Type4Py, revealing challenges due to dataset shifts and rare data types, and explores adaptation techniques to improve performance.

Contribution

It introduces a new dataset, CrossDomainTypes4Py, for evaluating type inference across different software domains and assesses the effectiveness of adaptation methods.

Findings

Dataset shifts significantly reduce inference accuracy.

Long-tailed data distributions hinder model performance.

Unsupervised adaptation improves cross-domain inference.

Abstract

Optional type annotations allow for enriching dynamic programming languages with static typing features like better Integrated Development Environment (IDE) support, more precise program analysis, and early detection and prevention of type-related runtime errors. Machine learning-based type inference promises interesting results for automating this task. However, the practical usage of such systems depends on their ability to generalize across different domains, as they are often applied outside their training domain. In this work, we investigate Type4Py as a representative of state-of-the-art deep learning-based type inference systems, by conducting extensive cross-domain experiments. Thereby, we address the following problems: class imbalances, out-of-vocabulary words, dataset shifts, and unknown classes. To perform such experiments, we use the datasets ManyTypes4Py and CrossDomainTypes4Py. The latter we introduce in this paper. Our dataset enables the evaluation of type inference systems in different domains of software projects and has over 1,000,000 type annotations mined on the platforms GitHub and Libraries. It consists of data from the two domains web development and scientific calculation. Through our experiments, we detect that the shifts in the dataset and the long-tailed distribution with many rare and unknown data types decrease the performance of the deep learning-based type inference system drastically. In this context, we test unsupervised domain adaptation methods and fine-tuning to overcome these issues. Moreover, we investigate the impact of out-of-vocabulary words.