Pop Quiz! Do Pre-trained Code Models Possess Knowledge of Correct API Names?
Terry Yue Zhuo, Xiaoning Du, Zhenchang Xing, Jiamou Sun, Haowei Quan,, Li Li, Liming Zhu

TL;DR
This study investigates whether pre-trained code models understand correct API names, revealing their limitations and the impact of pre-training strategies, and suggests that incorporating API structure could enhance their API knowledge and code generation capabilities.
Contribution
The paper introduces a knowledge probing approach to evaluate API name understanding in code models and analyzes how pre-training strategies influence this knowledge acquisition.
Findings
Current code models struggle with API name understanding.
Pre-training strategies significantly affect API knowledge learning.
Natural language context helps locate Python API names.
Abstract
Recent breakthroughs in pre-trained code models, such as CodeBERT and Codex, have shown their superior performance in various downstream tasks. The correctness and unambiguity of API usage among these code models are crucial for achieving desirable program functionalities, requiring them to learn various API fully qualified names structurally and semantically. Recent studies reveal that even state-of-the-art pre-trained code models struggle with suggesting the correct APIs during code generation. However, the reasons for such poor API usage performance are barely investigated. To address this challenge, we propose using knowledge probing as a means of interpreting code models, which uses cloze-style tests to measure the knowledge stored in models. Our comprehensive study examines a code model's capability of understanding API fully qualified names from two different perspectives: API…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Engineering Research · Advanced Malware Detection Techniques
MethodsCodeBERT
