On-the-Fly Syntax Highlighting using Neural Networks

TL;DR

This paper introduces a neural network-based method for real-time syntax highlighting that improves accuracy and efficiency over traditional regex-based tools, especially for complex language features.

Contribution

It presents a novel deep learning approach for on-the-fly syntax highlighting that handles both correct and incorrect code derivations, surpassing existing regex-based methods.

Findings

Achieves near-perfect accuracy in syntax highlighting

Outperforms regex-based strategies in speed and correctness

Effective across multiple programming languages

Abstract

With the presence of online collaborative tools for software developers, source code is shared and consulted frequently, from code viewers to merge requests and code snippets. Typically, code highlighting quality in such scenarios is sacrificed in favor of system responsiveness. In these on-the-fly settings, performing a formal grammatical analysis of the source code is not only expensive, but also intractable for the many times the input is an invalid derivation of the language. Indeed, current popular highlighters heavily rely on a system of regular expressions, typically far from the specification of the language's lexer. Due to their complexity, regular expressions need to be periodically updated as more feedback is collected from the users and their design unwelcome the detection of more complex language formations. This paper delivers a deep learning-based approach suitable for on-the-fly grammatical code highlighting of correct and incorrect language derivations, such as code viewers and snippets. It focuses on alleviating the burden on the developers, who can reuse the language's parsing strategy to produce the desired highlighting specification. Moreover, this approach is compared to nowadays online syntax highlighting tools and formal methods in terms of accuracy and execution time, across different levels of grammatical coverage, for three mainstream programming languages. The results obtained show how the proposed approach can consistently achieve near-perfect accuracy in its predictions, thereby outperforming regular expression-based strategies.