Building Code with Dynamic Staging

TL;DR

This paper introduces a dynamic staging approach for embedded DSLs that constructs executable code incrementally during parsing, eliminating the need for deep AST embeddings and enabling efficient, reusable, and flexible language semantics.

Contribution

It proposes a novel method for defining DSL semantics through action functions during parsing, allowing incremental code assembly without object representations or external traversals.

Findings

Produces code as efficient as handwritten implementations

Enables reuse of code between EDSL and host language

Supports complex semantics like variable scope and control flow

Abstract

When creating a new domain-specific language (DSL) it is common to embed it as a part of a flexible host language, rather than creating it entirely from scratch. The semantics of an embedded DSL (EDSL) is either given directly as a set of functions (shallow embedding), or an AST is constructed that is later processed (deep embedding). Typically, the deep embedding is used when the EDSL specifies domain-specific optimizations (DSO) in a form of AST transformations. In this paper we show that deep embedding is not necessary to specify most optimizations. We define language semantics as action functions that are executed during parsing. These actions build incrementally a new, arbitrary complex program function. The EDSL designer is able to specify many aspects of the semantics as a runnable code, such as variable scoping rules, custom type checking, arbitrary control flow structures, or DSO. A sufficiently powerful staging mechanism helps assembling the code from different actions, as well as evaluate the semantics in arbitrarily many stages. In the end, we obtain code that is as efficient as one written by hand. We never create any object representation of the code. No external traversing algorithm is used to process the code. All program fragments are functions with their entire semantics embedded within the function bodies. This approach allows reusing the code between EDSL and the host language, as well as combining actions of many different EDSLs.