Decoding Lua: Formal Semantics for the Developer and the Semanticist

TL;DR

This paper presents a formal, mechanized semantics for Lua 5.2, validated against the reference interpreter, aiming to bridge understanding for both language semanticists and Lua developers.

Contribution

It introduces a formal, mechanized small-step semantics for Lua 5.2, accessible to both semanticists and developers, and validated through testing against the Lua reference implementation.

Findings

Model accurately represents Lua 5.2 semantics

Mechanization in PLT Redex confirms correctness

Provides a foundation for testing and extending Lua implementations

Abstract

We provide formal semantics for a large subset of the Lua programming language, in its version 5.2. We validate our model by mechanizing it and testing it against the test suite of the reference interpreter of Lua, confirming that our model accurately represents the language. In addition, we set us an ambitious goal: to target both a PL semanticist ---not necessarily versed in Lua---, and a Lua developer ---not necessarily versed in semantic frameworks. To the former, we present the peculiarities of the language, and how we model them in a traditional small-step operational semantics, embedded within Felleisen-Hieb's reduction semantics with evaluation contexts. The mechanization is, naturally, performed in PLT Redex, the de facto tool for mechanizing reduction semantics. To the reader unfamiliar with such concepts, we provide, to our best possible within the space limitations, a gentle introduction of the model. It is our hope that developers of the different Lua implementations and dialects understand the model and consider it both for testing their work and for experimenting with new language features.

Equations

$$\text{setnamespace}2em\text{inference}\text{v}\notin \{\text{nil},\text{false}\}\text{if}\text{v}\text{then}{\text{s}}_1\text{else}{\text{s}}_2\text{end}{\to }^{\text{s}}{\text{s}}_1[]$$

$$\text{setnamespace}2em\text{inference}\text{v}=\text{nil}\vee \text{v}=\text{false}\text{if}\text{v}\text{then}{\text{s}}_1\text{else}{\text{s}}_2\text{end}{\to }^{\text{s}}{\text{s}}_2[]$$

$$\text{setnamespace}0.5em\text{inference}\ast \sigma \text{’}=(\text{r},\text{v}),\sigma \begin{array}{rl}\sigma \text{:}\text{local}\text{x}& \text{=}\text{v}\text{in}\text{s}\text{end}{\to }^{s\_\sigma }\sigma \text{’}\text{:}\text{s}[\text{x}\backslash \text{r}]\end{array}[]$$

$$\text{s}::=...|\text{local}\text{x}\text{=}\text{e}\text{in}\text{s}\text{end}|\text{x}\text{=}\text{e}$$

$$\text{e}::=...|\text{r}$$

$${\to }^{\text{s}}\text{llparenthesis}\text{\$iter}\text{e}\text{do}\text{s}\text{end}{\text{rrparenthesis}}_{\text{textsc}Break}$$

$${\to }^{\text{s}}\text{if}\text{e}\text{then}\text{s}\text{\$iter}\text{e}\text{do}\text{s}\text{end}$$

$${\to }^{\text{s}}\text{s}$$

$$\text{llparenthesis}{\text{E}}_{\text{lf}}[[\text{break}]]{\text{rrparenthesis}}_{\text{textsc}Break}$$

$$\text{llparenthesis}\text{;}{\text{rrparenthesis}}_{\text{textsc}Break}$$

$$\text{setnamespace}0.5em\text{inference}\text{op}\in \{\text{+},\text{-},\text{*},\text{/},\hat{},\text{\%},<,\le \}{\text{v}}_1,{\text{v}}_2\in \text{number}{\text{v}}_1\text{op}{\text{v}}_2{\to }^e\mathbf{\delta }(\text{op},{\text{v}}_1,{\text{v}}_2)[]$$

$$\begin{gathered} \text{setnamespace}0.5em\text{inference}\text{op}\in \{\text{+},\text{-},\text{*},\text{/},\hat{},\text{\%}\} \\ {\text{v}}_1\notin \text{number}\vee {\text{v}}_2\notin \text{number} \\ {\nu }_1=\mathbf{\delta }(\text{tonumber},{\text{v}}_1,10)\in \text{number} \\ {\nu }_2=\mathbf{\delta }(\text{tonumber},{\text{v}}_2,10)\in \text{number}{\text{v}}_1\text{op}{\text{v}}_2{\to }^e\mathbf{\delta }(\text{op},{\nu }_1,{\nu }_2)[] \end{gathered}$$

$$\begin{gathered} \text{setnamespace}0.5em\text{inference}\text{op}\in \{\text{+},\text{-},\text{*},\text{/},\hat{},\text{\%}\} \\ {\text{v}}_1\notin \text{number}\vee {\text{v}}_2\notin \text{number} \\ (\mathbf{\delta }(\text{tonumber},{\text{v}}_1,10)\notin \text{number}\vee \\ \mathbf{\delta }(\text{tonumber},{\text{v}}_2,10)\notin \text{number}){\text{v}}_1\text{op}{\text{v}}_2{\to }^e\text{llparenthesis}{\text{v}}_1\text{op}{\text{v}}_2{\text{rrparenthesis}}_{\text{textsc}ArithWO}[] \end{gathered}$$

$$\begin{gathered} \text{setnamespace}0.5em\text{inference}\mathbf{\delta }(\mathsf{rawget},\text{objr},{\text{v}}_1,{\theta }_1)\ne \text{nil} \\ {\theta }_2=\mathbf{\delta }(\mathsf{rawset},\text{objr},{\text{v}}_1,{\text{v}}_2,{\theta }_1){\theta }_1:\text{objr}\text{[}{\text{v}}_1\text{]}\text{=}{\text{v}}_2{\to }^{s\_\theta }{\theta }_2:\text{;}[] \end{gathered}$$

$$\text{setnamespace}0.5em\text{inference}\mathbf{\delta }(\mathsf{rawget},\text{objr},{\text{v}}_1,\theta )=\text{nil}\theta :\text{objr}\text{[}{\text{v}}_1\text{]}\text{=}{\text{v}}_2{\to }^{s\_\theta }\theta :\text{llparenthesis}\text{objr}\text{[}{\text{v}}_1\text{]}\text{=}{\text{v}}_2{\text{rrparenthesis}}_{\text{textsc}\mathsf{NewIndex}}[]$$

$$\text{setnamespace}0.5em\text{inference}\mathbf{\delta }(\mathsf{type},{\text{v}}_1)\ne "table"\theta :{\text{v}}_1\text{[}{\text{v}}_2\text{]}\text{=}{\text{v}}_3{\to }^{s\_\theta }\theta :\text{llparenthesis}{\text{v}}_1\text{[}{\text{v}}_2\text{]}\text{=}{\text{v}}_3{\text{rrparenthesis}}_{\text{textsc}\mathsf{NewIndex}}[]$$

$$\text{setnamespace}0.5em\text{inference}{\text{v}}_2=\mathbf{\delta }(\mathsf{rawget},\text{objr},{\text{v}}_1,\theta ){\text{v}}_2\ne \text{nil}\theta \text{:}\text{objr}\text{[}{\text{v}}_1\text{]}{\to }^{e\_\theta }\theta \text{:}{\text{v}}_2[]$$

$$\text{setnamespace}0.5em\text{inference}\mathbf{\delta }(\mathsf{rawget},\text{objr},\text{v},\theta )=\text{nil}\theta \text{:}\text{objr}\text{[}\text{v}\text{]}{\to }^{e\_\theta }\theta \text{:}\text{llparenthesis}\text{objr}\text{[}\text{v}\text{]}{\text{rrparenthesis}}_{\text{textsc}\mathsf{Index}}[]$$

$$\text{setnamespace}0.5em\text{inference}\mathbf{\delta }(\mathsf{type},{\text{v}}_1)\ne ‘‘table"\theta \text{:}{\text{v}}_1\text{[}{\text{v}}_2\text{]}{\to }^{e\_\theta }\theta \text{:}\text{llparenthesis}{\text{v}}_1\text{[}{\text{v}}_2\text{]}{\text{rrparenthesis}}_{\text{textsc}\mathsf{Index}}[]$$

$$\begin{gathered} \text{setnamespace}0.5em\text{inference}\forall 1\le \mathsf{i},{\text{field}}_{\mathsf{i}}=\text{v}\vee {\text{field}}_{\mathsf{i}}=\text{[}\text{v}\text{]}\text{=}{\text{v}}^{\prime } \\ {\theta }_2=\text{(}\text{objr},<\mathsf{addkeys}(\text{{}{\text{field}}_1\text{,}...\text{}})\text{,}\text{nil}>\text{)},{\theta }_1{\theta }_1\text{:}\text{{}{\text{field}}_1\text{,}...\text{}}{\to }^{e\_\theta }{\theta }_2\text{:}\text{objr}[] \end{gathered}$$

$$\begin{gathered} \text{setnamespace}0.5em\text{inference}{\sigma }^{\prime }=({\text{r}}_1,{\text{v’}}_1),...,({\text{r}}_{\text{n}},{\text{v’}}_{\mathsf{n}}),\sigma \\ \mathsf{i}\le \mathsf{m}\Rightarrow {\text{v’}}_{\mathsf{i}}={\text{v}}_{\mathsf{i}}\mathsf{i}>\mathsf{m}\Rightarrow {\text{v’}}_{\mathsf{i}}=\text{nil}\begin{array}{rl}\sigma \text{:}(\text{function}& \text{l}({\text{x}}_1,...,{\text{x}}_{\mathsf{n}})\text{s}\text{end})({\text{v}}_1,...,{\text{v}}_{\text{m}}){\to }^{\mathsf{funcall}}\\ & {\sigma }^{\prime }\text{:}\text{llparenthesis}\text{s}[{\text{x}}_1\backslash {\text{r}}_1,...,{\text{x}}_{\text{n}}\backslash {\text{r}}_{\text{n}}]{\text{rrparenthesis}}_{\text{textsc}Return}\end{array} \end{gathered}$$

$$\begin{gathered} \text{setnamespace}0.5em\text{inference}{\sigma }^{\prime }=({\text{r}}_1,{\text{v}}_1),...,({\text{r}}_{\text{n}},{\text{v}}_{\mathsf{n}}),\sigma \\ \mathsf{i}\le \mathsf{m}\Rightarrow {\text{v’}}_{\mathsf{i}}={\text{v}}_{\mathsf{i}}\mathsf{i}>\mathsf{m}\Rightarrow {\text{v’}}_{\mathsf{i}}=\text{nil} \\ \text{tuple}=<{\text{v}}_{\mathsf{n}+1},...,{\text{v}}_{\mathsf{m}}>\begin{array}{rl}\sigma \text{:}& (\text{function}\text{l}({\text{x}}_1,...,{\text{x}}_{\mathsf{n}},\text{...})\text{s}\text{end})({\text{v}}_1,...,{\text{v}}_{\text{m}})\\ & {\to }^{\mathsf{funcall}}{\sigma }^{\prime }\text{:}\text{llparenthesis}\text{s}[{\text{x}}_1\backslash {\text{r}}_1,...,{\text{x}}_{\text{n}}\backslash {\text{r}}_{\text{n}},\text{...}\backslash \text{tuple}]{\text{rrparenthesis}}_{\text{textsc}Return}\end{array} \end{gathered}$$

$$\text{setnamespace}0.5em\text{inference}\mathbf{\delta }(\mathsf{type},\text{v})\ne ‘‘function"\sigma \text{:}\text{v}({\text{v}}_1,...,{\text{v}}_{\text{n}}){\to }^{\mathsf{funcall}}\sigma \text{:}\text{llparenthesis}\text{v}({\text{v}}_1,...,{\text{v}}_{\text{n}}){\text{rrparenthesis}}_{\text{textsc}\mathsf{WFunCall}}$$

$$\text{setnamespace}0.5em\begin{array}{rl}\sigma \text{:}\text{v}\text{:}& \text{name}\text{(}{\text{e}}_1\text{,}...\text{,}{\text{e}}_{\mathsf{n}}\text{)}{\to }^{\mathsf{funcall}}\sigma \text{:}\text{v}\text{[“}\text{name}\text{"}\text{] (}\text{v}\text{,}{\text{e}}_1\text{,}...\text{,}{\text{e}}_{\mathsf{n}}\text{)}\end{array}$$

$$\text{setnamespace}0.5em\text{inference}\mathsf{l}\in \{\mathsf{type},\mathsf{assert},\mathsf{error},\mathsf{pcall},\mathsf{select},...\}\theta \text{:}\text{\$builtIn}\mathsf{l}\text{(}{\text{v}}_1,...,{\text{v}}_{\mathsf{n}}\text{)}{\to }^{\mathsf{builtIn}}\theta \text{:}\mathbf{\delta }(\mathsf{l},{\text{v}}_1,...,{\text{v}}_{\mathsf{n}})[]$$

$$\text{setnamespace}0.5em\text{inference}\mathsf{l}\in \{\mathsf{ipairs},\mathsf{next},\mathsf{pairs},\mathsf{getmetatable},...\}\theta \text{:}\text{\$builtIn}\mathsf{l}\text{(}{\text{v}}_1,...,{\text{v}}_{\mathsf{n}}\text{)}{\to }^{\mathsf{builtIn}}\theta \text{:}\mathbf{\delta }(\mathsf{l},{\text{v}}_1,...,{\text{v}}_{\mathsf{n}},\theta )[]$$

$$\begin{gathered} \text{setnamespace}0.5em\text{inference}\mathsf{l}\in \{\mathsf{rawset},\mathsf{setmetatable}\} \\ {\theta }_2=\mathbf{\delta }(\mathsf{l},{\text{v}}_1,...,{\text{v}}_{\mathsf{n}},{\theta }_1){\theta }_1\text{:}\text{\$builtIn}\mathsf{l}\text{(}{\text{v}}_1,...,{\text{v}}_{\mathsf{n}}\text{)}{\to }^{\mathsf{builtIn}}{\theta }_2\text{:}{\text{v}}_1[] \end{gathered}$$

Full text

In addition, we set us an ambitious goal: to target both a PL semanticist —not necessarily versed in Lua—, and a Lua developer —not necessarily versed in semantic frameworks. To the former, we present the peculiarities of the language, and how we model them in a traditional small-step operational semantics, embedded within Felleisen-Hieb’s reduction semantics with evaluation contexts. The mechanization is, naturally, performed in PLT Redex, the de facto tool for mechanizing reduction semantics.

To the reader unfamiliar with such concepts, we provide, to our best possible within the space limitations, a gentle introduction of the model. It is our hope that developers of the different Lua implementations and dialects understand the model and consider it both for testing their work and for experimenting with new language features.

1. Introduction

Lua is a lightweight imperative scripting language, featuring dynamic typing, automatic memory management, data description facilities, and metaprogramming mechanisms to adapt the language to specific domains (spe, ). The typical use case of a Lua application is as an extension library embedded in a host application, commonly written in C or C++. In that setting, Lua offers the possibility to add scripting facilities to the host application, combining the flexibility and rapid prototyping of a dynamic language within the static guarantees and optimizations of stricter programming languages.

Lua is extensively used in many diverse applications, ranging from game development, most notably by “AAA” games (ieru01, ) but also in mobile games and game frameworks, plugin development (for example, in the photo editing software Adobe Photoshop Lightroom111http://www.adobe.com/devnet/photoshoplightroom.html, and the type-setting system LuaTex222http://www.luatex.org/languages.html), web application firewalls333https://blog.cloudflare.com/cloudflares-new-waf-compiling-to-lua, and embedded systems444https://www.lua.org/uses.html.

Lua is informally specified by both its reference manual and its reference interpreter, developed and maintained by the core Lua authors. Thanks to Lua’s success, several alternative implementations555http://lua-users.org/wiki/LuaImplementations, as well as code linters and static analyzers666http://lua-users.org/wiki/ProgramAnalysis can be found in the wild. However, the informal nature of the specification means that developers of those tools must resort to their intuition, formed by study of the reference manual, inspection of the source code of the interpreter, and experimentation.

In this work, we present a comprehensive formalization of (most of) Lua 5.2, which we argue will facilitate the development and testing of these alternative implementations and analysis tools, as well as the prototyping of new features and extensions to the Lua language.

The formalism we use to express the semantics of Lua is mainly Felleisen-Hieb’s reduction semantics with evaluation contexts (syntheo, ), better known as just reduction semantics. This formalism, together with concepts from standard small-step operational semantics, has already been proven useful for tackling the formalization of real programming languages (guh, ; python, ; get, ; semcontext, ).

While providing the semantics on paper is an important contribution, it does not suffice to ensure that our characterization of the language is correct. For this reason we mechanize the semantics in PLT Redex (plt, ), following the success of previously mechanized reduction semantics for other scripting languages such as JavaScript (guh, ) and Python (python, ).

We tested the mechanization of our formal semantics of Lua against the test suite of the reference interpreter, successfully passing every test within the scope of the formalization. We take this as strong evidence to support the claim that our semantics is a sound representation of the selected subset of the language’s features, including:

• •

Every type of Lua value, except coroutines and userdata (see below);

• •

Metatables;

• •

Identity of closures;

• •

Dynamic execution of source code;

• •

Error handling (throwing error objects and catching them with protected mode).

• •

A large collection of the services of the standard library.

We purposely left out the following features for future work:

• •

Coroutines, in essence single-shot delimited continuations;

• •

Userdata, opaque handles to data from the host application and native libraries;

• •

Garbage collection;

• •

The goto statement;

• •

Services from the standard library that interface with the operating system, such as file manipulation, or have large complex C implementations, such as string pattern matching.

Our work is heavily motivated by previous work formalizing and mechanizing other scripting languages (guh, ; python, ). However, we take a different path, motivated by the specificities of Lua. The most important difference is that we do not attempt to reduce the language to a minimal core, and then specify a desugaring or elaboration step from source programs to this minimal core. We keep most of the original surface syntax of Lua, with minimal changes. Our goal is to make the semantics and its traces understandable by programmers that are already familiar with the Lua language but are not used to the concepts of reduction semantics that form the core of our formalization.

Another important difference is that we keep a strict separation between the semantic model (what forms the core of what is presented in this work) and its mechanization. While this is a rather technical point, it has an impact in the model, which we will timely discuss.

The rest of the paper is organized as follows: Section 2 presents a very brief description of Lua, with emphasis on some of the features that we formalize in later sections; Section 3 presents the basic concepts that our formalization uses, via a formalization of a very small subset of Lua; Section 4 expands the formalization of the previous section to show the most important parts of our complete semantics; Section 5 briefly discusses the mechanization and its tests; Section 6 discusses related work; finally, Section 7 summarizes our contributions and discusses future avenues of research.

2. Lua, an extensible scripting language

We organize our presentation of Lua around the examples of memoization and object-oriented programming, shown in figures 1 and 2, respectively. They serve to introduce several characteristics of Lua: its syntax, the versatility of its single data structure (tables), its metaprogramming mechanisms and some aspects of its scoping rules.

2.1. Memoization

The code777Taken from http://lua-users.org/wiki/FuncTables . shown in Figure 1 implements a memoization function, memoize, which takes a function fn as argument and returns its memoized version. The values of fn already computed will be stored in a table (t in line 2). At their core, tables are associative arrays that can be indexed with any Lua value except nil. We will show later in this section that tables also come with syntax sugar and metaprogramming facilities that can greatly extend their functionality beyond simple associative arrays.

Line 3 is where the memoized version of fn is returned through an anonymous function. This function takes x as argument and, before computing fn(x), performs a look-up in the table for value x (line 4). If the result of the look-up is nil it means no result was found, so it proceeds to compute fn(x) and store it in the table (line 5). The resulting value, either computed or retrieved from the table, is returned in line 6. The function memoize is used in lines 10–14 to improve the performance of a function that performs a sum from 1 to x.

All procedures and functions in Lua, anonymous or named, are first-class values, and form lexically-scoped closures. The anonymous function that memoize returns will effectively capture into its scope the table t, as expected.

Note that the definitions of memoize, t, and memsum are prefixed by the keyword local. Without it, all of these declarations are simple assignments, and do not introduce new names in the current scope. In an assignment, if there is no variable in scope with that name, then the variable is global: the assignment will actually store its rvalue in a table called the environment, with a string containing the variable’s name as the key. Using a variable that is not in scope also looks up the variable in the environment.

The environment is available to the programmer through a variable _ENV, which is always in scope. This means that any occurrence of a variable $x$ that is not in scope is just syntax sugar for _ENV[“$x$”]. Since it is a variable, the programmer can change the environment at will by simply assigning another table to _ENV.

2.2. Simple OOP in Lua

Another interesting example888Taken from http://lua-users.org/wiki/ObjectOrientationTutorial . is listed in Figure 2. It presents the implementation of some basics concepts of object-oriented programming, namely classes and objects, by combining tables, first-class functions, and the metatable mechanism. It also presents some syntax sugar provided by Lua to better support OOP.

In Lua, a class is essentially implemented as a dictionary (e.g., table), in which the method names form the keys of the dictionary, and the method implementations are the associated values. Objects are also model with tables, containing the fields and their values.

In the example, we have a class MyClass with its corresponding constructor (line 4) and only one field value with its setter (line 10) and getter (line 14). The function declarations in these lines are actually syntax sugar for assignments, where the left-hand sides are, respectively, MyClass[“new”], MyClass[“set_value”], and MyClass[“get_value”]. For the two methods on line 10 and line 14 the use of : instead of . also includes an extra first parameter for these functions, named self.

In the last lines of Figure 2 we show how to create an instance of MyClass (line 18), and how to invoke the methods. In line 20 we can observe the invocation of set_value with yet another syntax sugar: mc:set_value(6) is equivalent to mc[“set_value”](mc, 6).

If classes contain methods, and objects contain fields, how is mc[“set_value”] looking up the set_value method? The answer is the metatable mechanism, used in lines 2 and 5. In line 5, the call to setmetatable assigns MyClass as the metatable of the empty table {} passed as argument, and then returns this empty table.

A metatable can modify the behavior of a table with regards to most of Lua’s operations. For this example, the behavior we are modifying is look-up of non-existing keys. Each behavior that can be modified has an associated handler. For look-up of non-existing keys the handler is called __index (line 2). A handler is usually a function, but in the case of __index it can be another table, in this case MyClass. A non-existing key then will be looked up in this table, and this is how mc[“set_value”] results in the method set_value from MyClass.

Lua also specifies handlers for setting a non-existing key, for calling a value as if it were a function, for most of the binary and unary operators, for setting finalizers, and even for some functions in the standard library. Lua programmers typically use metatables for object-oriented programming (including more elaborated object models than class-based single inheritance), for operator overloading, and for proxies.

3. Basics of the formalization

In this section, we gently introduce the semantic framework used throughout the paper by providing semantics to a small subset of Lua. Essentially, we mix classical ideas from operational semantics based on abstract machines —the notion of “operational”, how a program performs its computations, and the representation of a store or memory— together with reduction semantics with evaluation contexts (plt, ), formalism from which we take the tools for modeling the concepts of environment and continuations, and to obtain a modular description of the semantics from simple computations to the execution of complete programs.

An interesting aspect of Felleisen-Hieb’s reduction semantics is the possibility of defining the semantics of a language by decomposing it into fragments, describing the fragment’s semantics in isolation with a separate relation. For our small subset of the Lua language, we describe three fragments: pure statements, pure expressions (following Lua’s distinction of statements and expressions), and stateful (i.e., memory changing) statements. Then, we compose the three using a fourth relation, therefore finally providing the semantics for entire programs.

We show the grammar for stateless programs in Figure 3. The statements are conditional branching and skip (denoted with ; ). The expressions are nil  (the absence of a useful value), boolean constants, and logical operators. Of course, we are not able to write any useful program. In the coming sections we will grow our language until we reach Lua.

Figure 4 introduces the typical operational semantics for the conditional statement, modeled with the $\rightarrow^{\textsf{s}}$ relation between stateless statements. The first rule states that, in a boolean context (the conditional of the if), any value different from nil  and false is considered true, and therefore the then branch is considered. Note that we write above the line the conditions in which the rule applies. When no condition is required, the line will be omitted. The second rule states that, for false or nil, the else branch is considered.

Figure 5 gives the semantics of expressions using a separate $\rightarrow^{e}$ relation. We use an interpretation function $\delta$, as seen in the literature, which provides meaning to operators using denotational semantics. In contrast to the relations over terms presented so far, denotational semantics are not tight to single computation steps. Figure 6 shows (a simplified version of999The actual equations use Lua’s parenthesized expressions, introduced in 4.1.) the $\delta$-equations for boolean operators.

We proceed now to extend the language with imperative features: (local) variables. Statements are enlarged with variable definition and assignment:

[TABLE]

In order to describe its operational semantics, we must introduce a model of the memory store. We model it as a partial function from a set of references to values, denoted as $\sigma$. We refer to $\sigma$ as the “values’ store”, or simply store.

As for references, we will not force any specific representation, just ask them to satisfy some simple properties to ensure the relation modeling the semantics of variables stays decidable. More specifically, we ask the domain of $\sigma$ (referred as to $\mathsf{dom}(\sigma)$) to be a finite set, with elements that must be syntactically represented, but different from any other syntactic object in the language. We further assume it is always possible to obtain a fresh reference from the store.

We extend the grammar of expressions with references:

[TABLE]

References, in contrast to all the language constructs we mentioned so far, do not belong to the Lua source language, i.e., they cannot be written down by a developer. They are run-time constructs: syntactic extensions made to the language for the sole purpose of obtaining a small-step semantics of the language. We will see other examples of such constructs in the coming sections.

Figure 7 describes the semantics for the definition and assignment of local variables. We use a new $\rightarrow^{{\sf s\_\sigma}}$ relation, which maps a pair of a store $\sigma$ and a statement $\mathsf{s}$ with another pair of a new store $\sigma^{\prime}$ and the resulting statement $\mathsf{s}^{\prime}$.

As shown in the rule for the introduction of local variables, when the right side of the definition is a value v we put it in the store with a fresh reference r. Then, we replace each occurrence of variable x in the scope of the local statement by the new reference r.

An important property of this semantics is that variables are never free, as substitution will always replace them by references right before they would become free. This will have an impact on closure creation (see 4.2).

Returning to Figure 7, the second rule shows variable assignment, with $\sigma[\textsf{{r}}:=\textsf{{v}}]$ denoting a store $\sigma^{\prime}$ such that $\mathsf{dom}(\sigma^{\prime})=\mathsf{dom}(\sigma)$, where $\sigma^{\prime}(\textsf{{r}})=\textsf{{v}}$ and $\forall\textsf{{r'}}\in\mathsf{dom}(\sigma^{\prime}),\textsf{{r'}}\neq\textsf{{r}}\Rightarrow\sigma^{\prime}(\textsf{{r'}})=\sigma(\textsf{{r'}})$. Note that the assignment reduces to an empty statement ;, indicating that there is nothing else to do for this particular statement. The third and final rule shows that references appearing in expressions are always implicitly dereferenced.

We have already defined three different relations, each of them computing a bit of a program: $\rightarrow^{\textsf{e}}$ computes an expression, $\rightarrow^{\textsf{s}}$ an stateless statement, and $\rightarrow^{{\sf s\_\sigma}}$ an stateful statement. Now we are ready to combine the three to perform the execution of a full program. To that effect we define the $\mapsto$ relation. This relation will say exactly when each of the previously defined relations will trigger, at the same time defining the order in which statement or expression must be executed next.

Here is where evaluation contexts play a central role. They describe the syntax of the language with the addition of a new construction: a hole, usually denoted as [ ]. Evaluation contexts will play different roles in later sections, but for the moment the (only) hole in a program will be filled in with the next statement or expression to be executed.

Figure 8 defines the evaluation context E for the small subset of Lua we described so far. We can see from the definition the order we expect evaluation to take place: in an if, the guard must be evaluated first. In the definition of variables we evaluate the rvalue for the definition first. In a binary operation, we evaluate the left operand first101010Our definition enforces left-to-right evaluation of expressions. Even if this is left unspecified in Lua’s reference manual, the two most popular implementations of Lua, the reference interpreter and LuaJIT (luajit.org), both evaluate expressions left-to-right..

Figure 9 defines the $\mapsto$ relation. Both $\textsf{{E}}[\![\textsf{{e}}]\!]$ and $\textsf{{E}}[\![\textsf{{s}}]\!]$ denote an evaluation context where the hole is filled with the respective expression or statement, if this yields a syntactically valid term of the language. If the evaluation context is well-defined, together with the relations that describes computation steps, there is a unique decomposition of a valid term into an evaluation context and a subterm, and this subterm will match one and only one of the semantic rules. The subterm that is filling the hole gives the current focus of the computation.

With all of the main ingredients in place, we are now ready to provide semantics to Lua.

4. A formal description of Lua

In this section, we describe the highlights of our formalization of the semantics of Lua, the main contribution of this work. Section 4.1 covers the stateless subset of the language, Section 4.2 covers the imperative subset, Section 4.3 describes the concepts added to support standard library services, Section 4.4 covers the semantics of metatables, and Section 4.5 wraps up with the semantics of complete programs and error handling.

4.1. Stateless Lua

We extend the stateless subset presented in Section 3 with while loops, breaks, composition of statements, and numbers and strings with their corresponding operations (Figure 10).

Correspondingly, we extend the relation $\rightarrow^{\textsf{s}}$ with the semantics of the new statements (Figure 12). First, a while loop begins by wrapping the whole loop in a Break label, changing also the name from while to $iter**. The purpose of the label is to mark the point in which a **break** should continue the execution, and the renaming is necessary to avoid repeatedly unfolding a while and piling up labels. Labels and **$iter are new run-time statements (Figure 11). Then, a loop marked with $iter is unfolded as usual, using the conditional to check the guard and perform a new iteration.

In a composition of two statements, when the one on the left is a skip (;), we continue with the second. More interestingly, when the execution finds a break inside a (labeled) block, the whole code is replaced with a skip, to signal the execution of the break has exited. This is achieved by using a new evaluation context $E_{\textsf{lf}}$, which represents a program in which no other labeled term occur (its definition is elided for brevity). By not having other labels, we know the one surrounding this context is the one we have to break. The last rule removes the label once the execution of a loop reached skip.

Having defined the semantics for statements, we turn our attention to expressions (Figure 14). For brevity we focus only on arithmetic operators, but similar rules exists for strings. The first rule state that, if operands $\textsf{{v}}_{1}$ and $\textsf{{v}}_{2}$ are both numbers, and the operation is relevant to numbers, we delegate the result to the $\delta$ function already introduced in 3. The second rule covers the case where one or both of the operands are not numbers, but can be coerced into a number by the external function tonumber. In that case, we coerce the operands and do the operation. There is similar rule for concatenation, elided for brevity, when one of the operands is a string and the other a number. Finally, the last rule applies when the operands cannot be coerced into numbers. In this case we label the expression with ArithWO (some labels are listed in Figure 13, where WO stands for Wrong Operands), to signal the error. At this point, execution is stuck here, but in Section 4.4 we show how the metatable mechanism handles this erroneous situation.

4.2. Imperative Lua

The imperative subset is made up of functions, function application, tables, field indexing, and field update. Despite being values, Lua functions are in the imperative subset because parameters are mutable variables, so they are allocated in the $\sigma$ store. Tables are mutable objects, and we allocate them in a separate store, denoted with $\theta$. Object references, the domain of $\theta$, are considered values, so are in the image of $\sigma$. We ask form them to satisfy the same properties as asked for references $\sigma$, together with the possibility of distinguish syntactically between each kind of reference. The image of $\theta$ only contains tables.

Functions are labeled so each function in the source program has a unique label l. How labels are represented is not important; as long as they are comparable. This reproduces the correct semantics of function equality in Lua, where two identical functions are not equal if they are defined in different parts of the source file, as shown in the following interaction with the reference interpreter:

⬇

f = function() end

g = function() end

print(f == g)

false

A source function may evaluate to different values during the evaluation of the program, due to different substitutions of their free variables. Our use of substitution and references means that we do not need to have explicit closures, a function definition is itself a closure once the focus of evaluation has reached it.

Figure 16 adds tuples, used for returning multiple values from function application, and for functions that can handle a variable number of arguments (vararg functions) through the … vararg operator. Wrapping an expression in parenthesis has a semantic effect in Lua: if the expression evaluates to a tuple the parenthesis discards all but the first value of the tuple (if the tuple is empty the parenthesized expression evaluates to nil).

Besides being “truncated” to their first value, these tuples can also be concatenated with another tuple, depending on their syntactical place in the program: in an expression list $e_{1},\ldots,e_{n}$ the tuples of expressions $e_{1}$ to $e_{n-1}$ evaluate to their first value, or nil for the empty tuple (the same behavior as parenthesized expressions). These $n-1$ values then form a tuple of their own, which is concatenated with the tuple of $e_{n}$. Semantically, this is done through reductions between tuples that “flattens” tuples of tuples until reaching a tuple where none of the values are another tuple.

The new statements also include multiple variable definition and assignment, which generalizes the single-variable versions introduced in Section 3. The reduction rules for these statements are not shown here for reasons of brevity, but are a straightforward extension of the simpler versions: in case of multiple assignment, the evaluation contexts assure that all lvalues are evaluated before rvalues, and the tuples for both sides are flattened, then lvalues are paired with their corresponding rvalue, with any lvalues that do not have a corresponding rvalue paired with nil.

Figure 17 describes assignment to table fields. It uses some services modeled by the $\delta$ function: $\mathsf{\delta(type,v)}$ is the type of the value; $\delta(\mathsf{rawget},\textsf{{objr}},\textsf{{v}},\theta)$ is primitive table indexing, yielding either the value associated with v in $\theta(\textsf{{objr}})$ or nil if there is no associated value; $\delta(\mathsf{rawset},\textsf{{objr}},\textsf{{v}}_{k},\textsf{{v}},\theta)$ is primitive table update, yielding a new $\theta$ where the table referenced by objr associates v with value $\textsf{{v}}_{k}$.

The rules show field update under 3 different circumstances: when the operation is made over an actual table with an existing key; when the operation is made over an actual table but with an unknown key; and when the assignment is carried over a non-table value. The last two cases just tag the expression with NewIndex, which will be handled by the metatable mechanism explained in Section 4.4. Field access (Figure 18) have similar rules, but tagging exceptional situations with Index.

Figure 19 provides meaning to table constructors. Its complete semantics actually depends upon the meta-function addkeys, which adds absent keys in the constructor (see Figure 15 for the syntax of table constructors). It works by supplying consecutive natural numbers as keys, starting with 1.

Figure 20 shows function and method application, described with a new relation $\rightarrow^{\mathsf{funcall}}$. Formal parameters are mutable variables, so a fresh reference is allocated for each parameter. The first rule covers all the cases involving the application of a non-vararg function: when it is applied to the same number of arguments as formal parameters, when it is applied to fewer arguments, with unpaired parameters receiving nil, and when it is applied to more arguments, with extra arguments silently ignored. Similar to what we did with while loops in the previous section, we also label the body with Return, to indicate the point to which a return statement must go. It is, roughly speaking, the syntactic equivalent to the return address saved in an activation frame.

Returning to Figure 20, the second rule shows the case of a vararg function call: the difference just resides on what is done with surplus arguments: in this case, they are put into a tuple expression, which replaces the vararg expression (…) in the body of the function.

The third rule has to do with one of the exceptional situations that can be managed by the metatable mechanism: a function call over a non-function value. Again, at this point we just label the whole expression with a tag that indicates what happened. The last rule shows how the method invocation is translated into a table look-up, with the object being injected as the first argument of the function.

Figure 21 shows the semantics of the return statement as well as implicitly returning by reaching the end of the function. The ideas used in this rules are analogous to the ones expressed when defining the semantics of the break statement, in Section 4.1.

4.3. Built-in services

In Lua, built-in services offered by Lua’s standard library are stored in the execution environment, a table named $\mathsf{\_ENV}$, where the keys are the names of the services and the values are their definitions. For instance, when we access the table field named “type”, we access the function that given an element provides its type (as a string):

⬇

print(type({}))

table

(Remember from Section 2: using an identifier not in scope is equal to accessing $\mathsf{\_ENV}$.) We can override its definition and obtain a different behavior:

⬇

type = function () return ’not a type’ end

print(type({}))

not a type

However, the original $\mathsf{type}$ function is still accessible from other services in the library. We can see this when we call $\mathsf{next}$, the function that iterates over the fields in a table:

⬇

next(1)

stdin:1: bad argument (table expected, got number)

In order to model this behavior, prior to the execution of a program the $\mathsf{\_ENV}$ table must be populated with the functions from the standard library. But these functions are just wrappers for a special (run-time) expression $builtIn**. When evaluated, this expression calls the$\delta$function with the actual definition of the function. Built-in services, like$\mathsf{next}$, might call other services through the **$builtIn term instead of ordinary function application, effectively reproducing the early binding that is required.

While it might sound a bit intricate, this design gives the formalization several desirable properties: compliance with the semantics as defined in the reference manual and the reference interpreter, and a modular way of tackling the formalization of built-in services. And at the level of the mechanization, it allows us to experiment and test against different implementations of these services with minimal changes in the rest of the formalization.

Figure 22 gives the semantics of **$builtIn** using three rules, corresponding to three different kinds of services: services that do not operate on tables, so do not need to access the object store$\theta$, services that read from tables, and services that update existing tables, yielding a new$\theta$. The antecedents of the first two rules show just some of the services that are in each category; the actual list of services includes almost all the built-in basic functions of the Lua language, together with services from the libraries math, string and table.

The $\delta$ function defines, in a denotational way, the actual fundamental details of the semantics of the built-in services and the primitive operators of the language. In the rest of this section we discuss an interesting example: the pairs built-in function (Figure 23).

The built-in service pairs is used to iterate a table using a for loop. It must return three values: an iterator function, the object to index, and the first index. According to the equations in the figure, there are three different scenarios: In the first case, when the table objr has a custom handler h in the __pairs key of its metatable, calls this handler to get the iterator triplet. The metafunction indexmetatable queries the metatable (metatables are discussed further in Section 4.4). Also note that we let $\delta$ yield not only values but any valid expression.

It might look odd why we create a function whose body calls h, instead of directly returning it. The reason is twofold: First, according to the manual, we must only return the first three values returned by the indexed function. This is achieved by creating three variables, one for each value, and return only those. If there are more values, they are discarded. Second, since local and return are not valid expressions, and in this case, $\boldsymbol{\delta}$ must return an expression (not necessarily a value!), we wrap this code in a closure.

In the second case, when the table has no metatable or no handler for __pairs, the reference manual indicates that pairs(t) returns “the next function, the table t, and nil”. This case models this behavior by wrapping a call to $builtIn next, as mentioned earlier in this section. The label of this function guarantees that it will be the same function that is bound to next in our initial environment.

The third and final case of pairs constructs an expression that will assemble an error message and then throw an error using the error built-in primitive. As with next, we cannot look-up the error built-in function in the environment, as it could have been rebound by the programmer.

Before concluding this section, we note that we let the interpretation function define the meaning of every primitive operator and library service using a denotational approach, following the tradition of Landin’s ISWIM language (700, ). Several of these primitives could also be given operational semantics, however, we decide to stick to ISWIM’s philosophy, which prioritizes cohesion, modularity, and expressivity.

4.4. Metatables

The most notable feature of Lua is its metaprogramming mechanism, metatables, that lets the programmer adapt the language to specific domains. With metatables Lua can maintain its original design decision to “keep the language simple and small”(evlua, ), while still being able to cope with a variety of programming concepts111111See section “Code Structure / Programming Paradigms” at lua-users.org/wiki/LuaDirectory.

Briefly, metatables let the programmer specify fallbacks for certain operations: arithmetic over non-numeric values, concatenation over non-string values, equality between objects that do not have the same identity, application over values that are not functions, indexing or updating a field over a value that is not a table, etc.

Metatables are plain tables, and the fallbacks that a particular metatable supports are typically functions associated with a unique string key for each operation (e.g. __add for the fallback to the plus operator, or __newindex for the fallback to field update). Lua libraries are free to extend this mechanism with their own fallbacks (like the pairs built-in function of the previous section, which can look up __pairs in the metatable, if it exists). Each Lua table can have its own metatable, while values of other types share a single metatable for each type.

We have shown in previous sections that regular semantics of operations just labels the expression or statement involved when it reaches a case where a fallback in a metatable could be used. This approach simplifies the regular semantics, and improves the modularity of the formalization. The relations $\rightarrow^{\textsf{e\_metatable}}$ and $\rightarrow^{\textsf{s\_metatable}}$ that we show in this section take these labeled terms and act accordingly.

Figure 24 shows how $\rightarrow^{\textsf{e\_metatable}}$ resolves arithmetic operations over operands of unexpected type, a condition labeled with ArithWO. The metafunction getbinhandler is analogous to those described in the Lua reference manual: it looks for a handler first in the metatable of the left operand $\textsf{{v}}_{1}$, then the metatable of the right one $\textsf{{v}}_{2}$, by looking into the corresponding field in those metatables. We also abstract the mapping between binary operators and their metatable keys with the metafunction binopeventkey.

Looking up a fallback in a metatable is guaranteed to either return the fallback or return nil because of two invariants: a metatable is always a table, and the metatable of a metatable, it if exists, is ignored for this look-up. This means that abstracting this look-up with a metafunction does not compromise the small-step nature of our semantics.

The first rule of Figure 24 shows how the operation is rewritten as an application of the handler on the two operands as arguments. If the handler is not a function this may trigger yet another fallback. The second rule shows what happens when no handler is found: an error is thrown using the error built-in service. We also abstract the construction of error messages with the #errmessage metafunction.

Figure 25 describes how the metatable mechanism works for field updates over a non-table value or a missing key. Again, we make use of metafunctions that abstracts the inner workings of the metatable mechanism: indexmetatable, which looks for the metatable of its first argument and looks up the fallback with the key passed as its second argument.

The first two rules of Figure 25 shows how this case resolves differently depending on whether the handler is a function or not (typically, in the second case the handler will be a table). The last two rules show how the absence of handler has different results depending on whether the original value is a table or not.

4.5. Semantics of programs and error handling

The definition of the $\mapsto$ reduction relation that describes the full semantics of Lua is essentially a straightforward extension of the simpler relation given in Figure 10. The domain now includes $\theta$, and maps the relations that were described in previous sections. Each of these relations is extended with the $\theta$ and $\sigma$ stores as needed. We omit these definitions for brevity.

More interestingly, in order to model the semantics of Lua’s exception handling, we must extend this relation. Lua’s exception handling consist of two built-in functions: error, which throws an error (any Lua value, usually a string), and pcall, which executes a function in protected mode. As any other built-in function, this behavior can be override by a developer.

Normally an error aborts the program, but if it is thrown in the context of a pcall, it is caught. In that case, pcall returns false and the error, otherwise, it returns true and the values returned by the function called.

Figure 26 describes the part of the $\mapsto$ relation that models error propagation and handling. For it, two new run-time constructs are added: **$err** to denote an error, and$\llparenthesis{;\textsf{{s}};}\rrparenthesis_{{\footnotesize\textsc{ProtMD}}}$to denote code that must be executed in protected mode. In the first rule, the evaluation context *Enp* is identical to *E*, except that there are no occurrences of$\llparenthesis{;\textsf{{E}};}\rrparenthesis_{{\footnotesize\textsc{ProtMD}}}$. The rule essentially aborts the whole program if there is no protected context around the error. The second rule aborts up to the first occurrence of a protected mode label, if there is one. The other three rules transition out of protected mode whether an error occurred or not.

The astute reader might wonder why we are modeling error handling here, and not as its own relation as we did with all the other parts of the semantics. The reason is merely technical: the rule that aborts the whole program, if isolated in its own relation, would break the unique decomposition property of evaluation contexts, in which there is a single way for decomposing a term into an evaluation context and the contents of its hole. We could have put just this rule explicitly in $\mapsto$ while having the others in an hypothetical $\rightarrow^{\textsf{error}}$ relation, but decided to keep all aspects of a feature together.

5. Mechanization

The formalization of the semantics was carried in parallel with its mechanization in PLT Redex (plt, ). This tool helped us recognize problems in our first attempts at formalizing Lua, and allowed us to experiment with new ideas before adding them to the formalization. It also allowed us to execute part of test suite of the reference interpreter of the language121212Available at https://www.lua.org/tests/., providing evidence that our semantics is in compliance with it.

Naturally, we could not use the whole test suite, for the following reasons:

• •

Language features not covered by our formalization: coroutines, the goto statement, and garbage collection;

• •

Standard library functions not covered in our formalization: file handling and string pattern matching. The Lua standard library is implemented in C, so the parts we cover were ported by hand to the subset of Lua covered by the formalization;

• •

The debug library, as it is heavily tied with the implementation details of the interpreter;

• •

Several other tests that test implementation details of the interpreter, and not the language. According to the Lua authors, the goal of the test suite is to test their reference implementation of Lua, and not to serve as a conformance test for alternative implementations131313 https://www.lua.org/wshop15/Ierusalimschy.pdf.

In practice, from the 25 .lua files present in the test suite, which actually test some feature of the language, we are able to port and run 9 against our PLT Redex mechanization. Figure 27 shows the percentage of LOCs actually tested from each of these remaining files, totaling 1256 LOCs successfully tested. It is important to remark that every file and line not tested is for the reasons explained above, and every line (in the 9 files tested) that fall within the scope of this work successfully passes the test. We take this as strong evidence that the mechanization of our formal semantics behaves exactly the same as the reference Lua interpreter.

Unfortunately, we do not have the space to discuss the code, which we plan to do in an extended version of this article. For the moment, we refer the reader to the documentation accompanying the code attached as supplementary material.

Dynamic loading of code

By implementing our parser directly in Racket (the language upon which PLT Redex is based), we mechanized easily the load service: Lua’s compiler available at runtime.

There are several details to mention related to the solution implemented, but for reasons of space we point out the most prominent:

• •

It covers the two modes: when the program to be compiled is passed as a string, or when it is a function from which the service obtains the program’s string.

• •

It can handle the compilation of code on a modified global environment.

• •

For completion, we emulate the case of the compilation of binary chunks (that is, a pre-compiled version of the code). This feature is implemented in conjunction with the service string.dump, which returns a string containing a binary representation of a function, given as a parameter.

6. Related work

We mentioned already that the present development is inspired by the work done in (guh, ; get, ; python, ), although not without its differences. For a start, we tackle a smaller language, and use this fact to model its semantics without shrinking much the core language. As a result, we do not need to consider an external parser or interpreter: everything is coded in Racket, and we can call the parser as a service of the library (Section 5). Another important difference is that we keep a certain distance between formalization and mechanization: this is why our model of memory is kept abstract, and why we put special emphasis in distinguishing the run-time constructs from the constructs of the source language.

Specific to Lua’s semantics, (flua, ) presents the operational semantics of Featherweight Lua, a minimal core of Lua. It considers a subset of features from the ones presented here, and as such cannot be tested against realistic Lua code. The mechanization is implemented in Haskell, which is not as flexible as PLT Redex to extend and change the model to cover a more realistic subset of the language.

There are several other important works related with real programming languages’ semantics. K-Java (kjava, ) features a complete formal semantics for Java, which is also mechanized using the tool $\mathbb{K}$. JavaScript, in turn, has several formal semantics: besides the aforementioned (guh, ; get, ), JSCert (tjs, ) is a formalization of the language in the Coq proof assistant, together with an interpreter extracted from the formalization. (opjs, ) also introduces a small-step operational semantics for JavaScript, including proofs of several properties of the model.

7. Conclusion

We give a small-step operational semantics for a large subset of the Lua programming language, as specified both by its informal reference manual and by its reference implementation. We did so with little or no syntactical differences between the actual syntax of Lua and the terms on which we define the semantics, therefore having a lightweight elaboration or desugaring step.

The semantics tackles complex Lua features such as its metatable mechanism, dynamic execution of source code, error handling, and a several standard library functions that have no Lua implementation. It is defined in a modular way, and could be extended to tackle absent features such as coroutines and the goto statement without modifying what is already specified.

We also provide a mechanization of the formal semantics in PLT Redex. A large part of the test suite for the reference interpreter has been successfully tested against this mechanization, providing evidence that we are successfully modeling the behavior of the language. As a plus, since our language resembles closely Lua, the traces of execution keep the code almost as written by the developer.

The development of the formal semantics, its mechanization, and its test suite make up a tool that both semanticists and Lua developers can use for understanding and extending the features of the language.

There are several further avenues for development. Besides adding missing features such as coroutines and goto statements, and the new operators and large integer types of version 5.3, the formal semantics and its mechanization can provide a basis for specifying, implementing, and formally proving correct static analyses for Lua programs.