TypyBench: Evaluating LLM Type Inference for Untyped Python Repositories

TL;DR

TypyBench is a new benchmark for evaluating large language models' ability to perform type inference in untyped Python codebases, highlighting current limitations and guiding future improvements.

Contribution

Introduces TypyBench, a benchmark with novel metrics TypeSim and TypeCheck, for comprehensive evaluation of LLMs' type inference in Python repositories.

Findings

LLMs achieve decent TypeSim scores but struggle with complex nested types.

Significant type consistency errors are observed across repositories.

Future research should focus on repository-level type consistency rather than just type similarity.

Abstract

Type inference for dynamic languages like Python is a persistent challenge in software engineering. While large language models (LLMs) have shown promise in code understanding, their type inference capabilities remain underexplored. We introduce TypyBench, a benchmark designed to evaluate LLMs' type inference across entire Python repositories. TypyBench features two novel metrics: TypeSim, which captures nuanced semantic relationships between predicted and ground truth types, and TypeCheck, which assesses type consistency across codebases. Our evaluation of various LLMs on a curated dataset of 50 high-quality Python repositories reveals that, although LLMs achieve decent TypeSim scores, they struggle with complex nested types and exhibit significant type consistency errors. These findings suggest that future research should shift focus from improving type similarity to addressing repository-level consistency. TypyBench provides a foundation for this new direction, offering insights into model performance across different type complexities and usage contexts. Our code and data are available at https://github.com/typybench/typybench.