Long-Range Modeling of Source Code Files with eWASH: Extended Window Access by Syntax Hierarchy
Colin B. Clement, Shuai Lu, Xiaoyu Liu, Michele Tufano, Dawn Drain,, Nan Duan, Neel Sundaresan, Alexey Svyatkovskiy

TL;DR
This paper introduces eWASH, a syntax hierarchy-based method to incorporate entire source code file context into fixed-length transformer models, improving code understanding and generation tasks.
Contribution
It presents a syntax hierarchy integration approach that extends context window capabilities without changing model architecture, achieving state-of-the-art results in Python code tasks.
Findings
Achieved new state-of-the-art in Python code completion and summarization.
Introduced new benchmarks for code completion with normalized literals and method body summarization.
Demonstrated the effectiveness of syntax hierarchy integration in leveraging full file context.
Abstract
Statistical language modeling and translation with transformers have found many successful applications in program understanding and generation tasks, setting high benchmarks for tools in modern software development environments. The finite context window of these neural models means, however, that they will be unable to leverage the entire relevant context of large files and packages for any given task. While there are many efforts to extend the context window, we introduce an architecture-independent approach for leveraging the syntactic hierarchies of source code for incorporating entire file-level context into a fixed-length window. Using concrete syntax trees of each source file we extract syntactic hierarchies and integrate them into context window by selectively removing from view more specific, less relevant scopes for a given task. We evaluate this approach on code generation…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Engineering Research · Topic Modeling · Natural Language Processing Techniques
