Combinatorial generation via permutation languages. I. Fundamentals

TL;DR

This paper introduces a versatile algorithmic framework for exhaustively generating combinatorial objects via permutation encoding, unifying many known results and enabling new Gray codes for permutations, geometric structures, and lattice congruences.

Contribution

The work presents a unified permutation-based framework for generating diverse combinatorial objects and proves new Gray codes for pattern-avoiding permutations and related structures.

Findings

Four classical Gray codes for permutations, bitstrings, trees, and set partitions.

New Gray codes for pattern-avoiding permutations and their bijections.

Hamiltonian paths on quotientopes demonstrating optimal Gray code generation.

Abstract

In this work we present a general and versatile algorithmic framework for exhaustively generating a large variety of different combinatorial objects, based on encoding them as permutations. This approach provides a unified view on many known results and allows us to prove many new ones. In particular, we obtain four classical Gray codes for permutations, bitstrings, binary trees and set partitions as special cases. We present two distinct applications for our new framework: The first main application is the generation of pattern-avoiding permutations, yielding new Gray codes for different families of permutations that are characterized by the avoidance of certain classical patterns, (bi)vincular patterns, barred patterns, boxed patterns, Bruhat-restricted patterns, mesh patterns, monotone and geometric grid classes, and many others. We also obtain new Gray codes for all the combinatorial objects that are in bijection to these permutations, in particular for five different types of geometric rectangulations, also known as floorplans, which are divisions of a square into $n$ rectangles subject to certain restrictions. The second main application of our framework are lattice congruences of the weak order on the symmetric group $S_n$. Recently, Pilaud and Santos realized all those lattice congruences as $(n-1)$-dimensional polytopes, called quotientopes, which generalize hypercubes, associahedra, permutahedra etc. Our algorithm generates the equivalence classes of each of those lattice congruences, by producing a Hamilton path on the skeleton of the corresponding quotientope, yielding a constructive proof that each of these highly symmetric graphs is Hamiltonian. We thus also obtain a provable notion of optimality for the Gray codes obtained from our framework: They translate into walks along the edges of a polytope.

Figures

Tables

Table 1. Table 1. Tame permutation patterns and corresponding combinatorial objects and orderings generated by Algorithm J. Some patterns are interesting in their own right and have no ‘natural’ associated combinatorial objects, in which case the first two columns are merged. The patterns with underscores, bars, boxes, and those drawn as grids with some shaded cells, are defined and explained in Sections  4.3 – 4.8 . See Table  2 for more patterns.

Tame patterns	Combinatorial objects and ordering	References/OEIS [oei20]
none	permutations by adjacent	[Joh63, Tro62], A000142
	transpositions → plain change order
$231=2\underset{¯}{31}$	Catalan families	A000108
	• binary trees by rotations → Lucas-	[LRR93]
	-Roelants van Baronaigien-Ruskey order
	• triangulations by edge flips
	• Dyck paths by hill flips
$1\underset{¯}{32}$	Bell families	A000110
	• set partitions by element	[Kay76, Wil13]
	exchanges → Kaye’s order
$132\wedge 231=\underset{¯}{13}2\wedge 2\underset{¯}{31}$:	binary strings by bitflips	[Gra53], A011782
permutations without peaks	→ reflected Gray code order (BRGC)
$1342$	forests of $\beta (0,1)$-trees	[Bón97, AKPV16],
		A022558
$2143$: vexillary permutations		[LS85], A005802
conjunction of $v_k$ tame patterns with $v_2=35,v_3=91,v_4=2346$		[BP14], A224318,
(see [Bil13]): $k$-vexillary permutations $\left(k\ge 1\right)$		A223034, A223905
$2143\wedge 3412$: skew-merged permutations		[Sta94, Atk98], A029759
$2143\wedge 2413\wedge 3142$		[DMR10, SV14], A033321
$2143\wedge 2413\wedge 3142\wedge 3412$: X-shaped permutations		[Wat07, Eli11], A006012
$2413\wedge 3142$:	Schröder families	A006318
separable permutations	• slicing floorplans (=guillotine	[AN81, BBL98, ABP06]
	partitions)
	• topological drawings of $K_{2,n}$	[CF18]
$2\underset{¯}{41}3\wedge 3\underset{¯}{14}2$: Baxter	mosaic floorplans (=diagonal	[YCCG03, ABP06]
$2\underset{¯}{41}3\wedge 3\underset{¯}{41}2$: twisted Baxter	rectangulations=R-equivalent	[LR12, CSS18]
$2\underset{¯}{14}3\wedge 3\underset{¯}{14}2$	rectangulations)	A001181
$2\underset{¯}{14}3\wedge 3\underset{¯}{41}2$	S-equivalent rectangulations	[ABBM+13], A214358
$2\underset{¯}{14}3\wedge 3\underset{¯}{41}2\wedge 2413\wedge 3142$	S-equivalent guillotine rectangulations	[ABBM+13], A078482
$3\underset{¯}{51}24\wedge 3\underset{¯}{51}42\wedge 24\underset{¯}{51}3\wedge 42\underset{¯}{51}3$:	generic rectangulations	[Rea12b]
2-clumped permutations	(=rectangular drawings)
conjunction of $c_k$ tame patterns with $c_k=2\left(k/2\right)!\left(k/2+1\right)!$ for $k$ even		[Rea12b]
and $c_k=2{\left(\left(k+1\right)/2\right)!}^2$ for $k$ odd: $k$-clumped permutations
$12543\wedge 13254\wedge 13524\wedge 13542\wedge 21543\wedge 125364\wedge 125634\wedge 215364\wedge 215634$		[FMD21]
$\wedge 315264\wedge 315624\wedge 315642$: permutations with 0-1 Schubert polynomial
$2143\wedge 2413\wedge 3412\wedge 314562\wedge 412563\wedge 415632\wedge 431562\wedge 512364$		[BEV18]
$\wedge 512643\wedge 516432\wedge 541263\wedge 541632\wedge 543162$: widdershins permutations
$243\overline{1}$ (A051295), $25\overline{3}14$ (A117106), $352\overline{4}1$ (A137534), $4\overline{2}513$ (A137535)		[Pud10] (OEIS
$425\overline{1}3$ (A110447), $\overline{4}2153$ (A137536), $25\overline{1}34$ (A137538), $\overline{4}1523$ (A137539)		shown on the left)
$\overline{4}1253$ (A137540), $3524\overline{1}$ (A137542)
$3\overline{1}52\overline{4}=3\overline{1}42\wedge 241\overline{3}$		[Pud08, BMCDK10],
		A098569
$2143$, $3142$		[AKV13]
$\text{<svg><g><g><path></path></g><g><path></path></g><g><path></path></g><g><path></path></g><path></path><path></path><path></path><path></path><path></path><path></path><path></path><path></path><path></path><path></path></g></svg>}\wedge \text{<svg><g><g><path></path></g><g><path></path></g><path></path><path></path><path></path><path></path><path></path><path></path><path></path><path></path><path></path><path></path></g></svg>}$  : permutations that characterize Schubert varieties which are Gorenstein		[WY06], A097483
	$\left(\mathrm{𝟐}+\mathrm{𝟐}\right)$-free posets	[Par09, BMCDK10] A022493

Table 2. Table 2. Permutation patterns that become tame through elementary transformations, plus corresponding combinatorial objects.

Tame patterns	Combinatorial objects	References/OEIS [oei20]
$\mathrm{rot}\left(2341\wedge 3\overline{5}241\right)=1432\wedge 1352\overline{4}$:	rooted non-separable planar maps	[Wes90, Zei92, GW96]
2-stack sortable permutations,		[DGG98], A000139
$2413\wedge 41\overline{3}52$, $2413\wedge 45\overline{3}12$,		[DGW96]
$2413\wedge 21\overline{3}54$, $3241\wedge \overline{2}4153$,
${\mathrm{rot}}^{-1}\left(2413\wedge 5\overline{1}324\right)=2413\wedge 1534\overline{2}$,
$\mathrm{cpl}\left(2413\wedge \overline{4}2315\right)=3142\wedge \overline{2}4351$,
$\mathrm{cpl}\left(2314\wedge \overline{4}2513\right)=3241\wedge \overline{2}4153$,
$\mathrm{cpl}\left(3214\wedge \overline{2}4135\right)=2341\wedge \overline{4}2531$,
$\mathrm{rev}\left(2413\wedge 41\overline{3}52\right)=3142\wedge 2\underset{¯}{41}3$		[CKS09]
conjunction of 20 patterns ${\tau }_i$ with tame $\mathrm{cpl}\left({\tau }_i\right)$:		[Eld06], A245233
permutations generated by a stack of depth two and an infinite stack
$\mathrm{inv}\left(132\wedge 312\right)=132\wedge 231$: Gilbreath permutations		[Vel03, DG12]
$\mathrm{rot}\left(3\overline{1}\underset{¯}{42}\wedge 3\overline{1}\underset{¯}{24}\right)=\mathrm{rot}\left(\text{<svg><g><g><path></path></g><g><path></path></g><g><path></path></g><g><path></path></g><path></path><path></path><path></path><path></path><path></path><path></path></g></svg>}\wedge \text{<svg><g><g><path></path></g><g><path></path></g><path></path><path></path><path></path><path></path><path></path><path></path></g></svg>}\right)=\text{<svg><g><g><path></path></g><g><path></path></g><path></path><path></path><path></path><path></path><path></path><path></path></g></svg>}\wedge \text{<svg><g><g><path></path></g><g><path></path></g><path></path><path></path><path></path><path></path><path></path><path></path></g></svg>}$  :		[Lan07, BL10], A129698
permutations that uniquely encode pile configurations in patience sorting

Table 3. Table 3. Illustration of Lemma  19 .

$\tau$	${\tau }^{\prime }$	${\tau }^{-}$	$r$	$s$	$\sigma$
$\overline{1}45632$	145632	34521	1	1
$1\overline{4}5632$	145632	14532	2	4
$14\overline{5}632$	145632	14532	2	4
$145\overline{6}32$	145632	14532	2	4
$1456\overline{3}2$	145632	13452	5	6
$14563\overline{2}$	145632	13452	5	6

Equations

$$J(L_{n}):=\reflectbox{$\vv{\reflectbox{$c$}}$}(\pi_{1}),\vv{c}(\pi_{2}),\reflectbox{$\vv{\reflectbox{$c$}}$}(\pi_{3}),\vv{c}(\pi_{4}),\ldots,$$

$$f(x_{2}\cdots x_{n}):=\begin{cases}c_{n}\big{(}f(x_{2}\cdots x_{n-1})\big{)}&\text{ if }x_{n}=0,\\ c_{1}\big{(}f(x_{2}\cdots x_{n-1})\big{)}&\text{ if }x_{n}=1,\end{cases}$$

$$\begin{array}{rl}{S}_n(F\wedge G)& :=S_n(F)\cap S_n(G),\\ S_n(F\vee G)& :=S_n(F)\cup S_n(G).\end{array}$$

$$S_{n}(P)=\bigcap_{x\in L(P)}S_{n}(\operatorname{inv}(x))=S_{n}\Big{(}\bigwedge\nolimits_{x\in L(P)}\operatorname{inv}(x)\Big{)}.$$

$$(P_{1},c)=\big{(}(132,c)\wedge(231,0)\big{)}\vee\big{(}(132,c-1)\wedge(231,1)\big{)}\vee\cdots\vee\big{(}(132,0)\wedge(231,c)\big{)}$$

$$S_{n}(\tau)=\bigcap_{1\leq i\leq b}S_{n}(\widetilde{\tau}_{i})=S_{n}\Big{(}\bigwedge\nolimits_{1\leq i\leq b}\widetilde{\tau}_{i}\Big{)}.$$

$$C:=\big{\{}\big{(}r+i-1,\tau^{\prime}(r+i)-1\big{)}\mid 0\leq i\leq s-r\big{\}}.$$

$$S_n(\tau ):=\bigcap _{j\in [k]\setminus I}{S}_n^w({\tau }_j),$$

$${\mathrm{Geo}}_n(M)=S_n({\tau }_1\wedge \cdots \wedge {\tau }_{\ell })$$

Full text

We present two distinct applications for our new framework: The first main application is the generation of pattern-avoiding permutations, yielding new Gray codes for different families of permutations that are characterized by the avoidance of certain classical patterns, (bi)vincular patterns, barred patterns, boxed patterns, Bruhat-restricted patterns, mesh patterns, monotone and geometric grid classes, and many others. We also obtain new Gray codes for all the combinatorial objects that are in bijection to these permutations, in particular for five different types of geometric rectangulations, also known as floorplans, which are divisions of a square into $n$ rectangles subject to certain restrictions.

The second main application of our framework are lattice congruences of the weak order on the symmetric group $S_{n}$. Recently, Pilaud and Santos realized all those lattice congruences as $(n-1)$-dimensional polytopes, called quotientopes, which generalize hypercubes, associahedra, permutahedra etc. Our algorithm generates the equivalence classes of each of those lattice congruences, by producing a Hamilton path on the skeleton of the corresponding quotientope, yielding a constructive proof that each of these highly symmetric graphs is Hamiltonian. We thus also obtain a provable notion of optimality for the Gray codes obtained from our framework: They translate into walks along the edges of a polytope.

An extended abstract of this paper appeared in the Proceedings of the 31st Annual ACM-SIAM Symposium on Discrete Algorithms (SODA 2020) [HHMW20].

Torsten Mütze is also affiliated with the Faculty of Mathematics and Physics, Charles University Prague, Czech Republic. He was supported by Czech Science Foundation grant GA 19-08554S, and by German Science Foundation grant 413902284.

1. Introduction

In mathematics and computer science we frequently encounter different kinds of combinatorial objects, such as permutations, binary strings, binary trees, set partitions, spanning trees of a graph, and so forth. There are three recurring fundamental algorithmic tasks that we want to perform with such objects: counting, random generation, and exhaustive generation. For the first two tasks, there are powerful general methods available, such as generating functions [FS09] and Markov chains [Jer03], solving both problems for a large variety of different objects. For the third task, namely exhaustive generation, however, we are lacking such a powerful and unifying theory, even though some first steps in this direction have been made (see Section 1.2 below). Nonetheless, the literature contains a vast number of algorithms that solve the exhaustive generation problem for specific classes of objects, and many of these algorithms are covered in depth in the most recent volume of Knuth’s seminal series ‘The Art of Computer Programming’ [Knu11].

1.1. Overview of our results

The main contribution of this paper is a general and versatile algorithmic framework for exhaustively generating a large variety of different combinatorial objects, which provides a unified view on many known results and allows us to prove many new ones. The basic idea is to encode a particular set of objects as a set of permutations $L_{n}\subseteq S_{n}$, where $S_{n}$ denotes the set of all permutations of $[n]:=\{1,2,\ldots,n\}$, and to use a simple greedy algorithm to generate those permutations by cyclic rotations of substrings, an operation we call a jump. This works under very mild assumptions on the set $L_{n}$, and allows us to generate more than double-exponentially (in $n$) many distinct sets $L_{n}$. Moreover, the jump orderings obtained from our algorithm translate into listings of combinatorial objects where consecutive objects differ by small changes, i.e., we obtain Gray codes [Sav97], and those changes are smallest possible in a provable sense. The main tools of our framework are Algorithm J and Theorem 1 in Section 2. In particular, we obtain the following four classical Gray codes as special cases: (1) the Steinhaus-Johnson-Trotter algorithm to generate all permutations of $[n]$ by adjacent transpositions, also known as plain change order [Tro62, Joh63]; (2) the binary reflected Gray code (BRGC) to generate all binary strings of length $n$ by flipping a single bit in each step [Gra53]; (3) the Gray code for generating all $n$-vertex binary trees by rotations due to Lucas, Roelants van Baronaigien, and Ruskey [LRR93]; (4) the Gray code for generating all set partitions of $[n]$ by exchanging an element in each step due to Kaye [Kay76].

We see two main applications for our new framework: The first application is the generation of pattern-avoiding permutations, yielding new Gray codes for different families of permutations that are characterized by the avoidance of certain classical patterns, vincular and bivincular patterns [BS00, BMCDK10], barred patterns [Wes90], boxed patterns [AKV13], Bruhat-restricted patterns [WY06], mesh patterns [BC11], monotone and geometric grid classes [HV06, AAB*+*13], and many others. We also obtain new Gray codes for all the combinatorial objects that are in bijection to these permutations, in particular for five different types of geometric rectangulations [ABP06, Rea12b, ABBM*+*13, CSS18], also known as floorplans, which are divisions of a square into $n$ rectangles subject to different restrictions. Our results on pattern-avoiding permutations are the focus of this paper.

The second application of our framework are lattice congruences of the weak order on the symmetric group $S_{n}$. This area has beautiful ramifications into groups, posets, polytopes, geometry, and combinatorics, and has been developed considerably in recent years, in particular thanks to Nathan Reading’s works, summarized in [Rea12a, Rea16a, Rea16b]. There are double-exponentially many distinct such lattice congruences, and they generalize many known lattices such as the Boolean lattice, the Tamari lattice [Tam62], and certain Cambrian lattices [Rea06, CP17]. Recently, Pilaud and Santos [PS19] realized all those lattice congruences as $(n-1)$-dimensional polytopes, called quotientopes, which generalize hypercubes, associahedra, permutahedra etc. Our algorithm generates the equivalence classes of each of those lattice congruences, by producing a Hamilton path on the skeleton of the corresponding quotientope, yielding a constructive proof that each of these highly symmetric graphs is Hamiltonian. Our results in this area are presented in part II of this paper series [HM21].

1.2. Related work

Avis and Fukuda [AF96] introduced reverse-search as a general technique for exhaustive generation. Their idea is to consider the set of objects to be generated as the nodes of a graph, and to connect them by edges that model local modification operations (for instance, adjacent transpositions for permutations). The resulting flip graph is equipped with an objective function, and the directed tree formed by the movements of a local search algorithm that optimizes this function is traversed backwards from the optimum node, using an adjacency oracle. The authors applied this technique successfully to derive efficient generation algorithms for a number of different objects; for instance, triangulations of a point set, spanning trees of a graph, etc. Reverse-search is complementary to our permutation based approach, as both techniques use fundamentally different encodings of the objects. The permutation encoding seems to allow for more fine-grained control (optimal Gray codes) and even faster generation algorithms.

Another method for combinatorial counting and exhaustive generation is the ECO framework introduced by Barcucci, Del Lungo, Pergola, and Pinzani [BDPP99]. The main tool is an infinite tree with integer node labels, and a set of production rules for creating the children of a node based on its label. Bacchelli, Barcucci, Grazzini, and Pergola [BBGP04] also used ECO for exhaustive generation, deriving an efficient algorithm for generating the corresponding root-to-node label sequences in the ECO tree in lexicographic order, which was later turned into a Gray code [BGPP07]. Dukes, Flanagan, Mansour, and Vajnovszki [DFMV08], Baril [Bar09], and Do, Tran and Vajnovszki [DTV19] used ECO for deriving Gray codes for different classes of pattern-avoiding permutations, which works under certain regularity assumptions on the production rules. Vajnovszki [Vaj10] also applied ECO for efficiently generating other classes of permutations, such as involutions and derangements. The main difference between ECO and our framework is that the change operations on the label sequences of the ECO tree do not necessarily correspond to Gray-code like changes on the corresponding combinatorial objects. Minimal jumps in a permutation, on the other hand, always correspond to minimal changes on the combinatorial objects in a provable sense, even though they may involve several entries of the permutation.

Li and Sawada [LS09] considered another tree-based approach for generating so-called reflectable languages, yielding Gray codes for $k$-ary strings and trees, restricted growth strings, and open meandric systems (see also [XCU10]). Ruskey, Sawada, and Williams [RSW12, SW12] proposed a generation framework based on binary strings with a fixed numbers of 1s, called bubble languages, which can generate e.g. combinations, necklaces, Dyck words, and Lyndon words. In the resulting cool-lex Gray codes, any two consecutive words differ by cyclic rotation of some prefix.

Pattern avoidance in permutations is a central topic in combinatorics, as illustrated by the books [Kit11, Bón12], and by the conference ‘Permutation Patterns’, held annually since 2003. Given two permutations $\pi$ and $\tau$, we say that $\pi$ contains the pattern $\tau$, if $\pi$ contains a subpermutation formed by (not necessarily consecutive) entries that appear in the same relative order as in $\tau$; otherwise we say that $\pi$ avoids $\tau$. It is well known that many fundamental classes of combinatorial objects are in bijection with pattern-avoiding permutations (see Tables 1 and 2 and [Ten18]). For instance, Knuth [Knu98] first proved that all 123-avoiding and 132-avoiding permutations are counted by the Catalan numbers (see also [CK08]). With regards to counting and exhaustive generation, a few tree-based algorithms for pattern-avoiding permutations have been proposed [Eli07, DFMV08, Bar08, Bar09]. Pattern-avoidance has also been studied extensively from an algorithmic point of view. In fact, testing whether a permutation $\pi$ contains another permutation $\tau$ as a pattern is known to be NP-complete in general [BBL98]. Jelínek and Kynčl [JK17] proved that the problem remains hard even if $\pi$ and $\tau$ have no decreasing subsequence of length 4 and 3, respectively, which is best possible. On the algorithmic side, Guillemot and Marx [GM14] showed that the problem can be solved in time $2^{O(k^{2}\log k)}n\log n$, where $n$ is the length of $\pi$ and $k$ is the length of $\tau$, a considerable improvement over the obvious $O(n^{k})$ algorithm (see also [Koz19]). In particular, for a pattern of constant length $k$ their algorithm runs in almost linear time. It is also known that for a fixed set of forbidden patterns, computing the number of pattern-avoiding permutations is hard [GP16].

1.3. Outline of this and future papers

This is the first in a series of papers where we develop our theory of combinatorial generation via permutation languages. In this first paper we focus on presenting the fundamental algorithmic ideas (Section 2) and show how to derive the four aforementioned classical Gray codes from our framework (Section 3). We also discuss the main applications of our framework to pattern-avoiding permutations (Sections 4 and 5), and its limitations (Section 6). In part II of the series [HM21], we apply the framework to lattice congruences of the weak order on the symmetric group $S_{n}$. In part III we discuss the generation of different types of rectangulations. In part IV we cover general methods to make the generation algorithms derived from our framework efficient, and we discuss the problems of ranking/unranking for the orderings obtained from our framework, which are highly relevant for the task of random generation mentioned in the beginning. Two of the latter topics, namely efficient algorithms and rectangulations, are only very briefly discussed in this paper (see Section 2.6 and Figure 7 below, respectively).

2. Generating permutations by jumps

In this section we present a simple greedy algorithm, Algorithm J, for exhaustively generating a given set $L_{n}\subseteq S_{n}$ of permutations, and we show that the algorithm works successfully under very mild assumptions on the set $L_{n}$ (Theorem 1).

2.1. Preliminaries

We use $S_{n}$ to denote the set of all permutations of $[n]:=\{1,\ldots,n\}$, and we write $\pi\in S_{n}$ in one-line notation as $\pi=\pi(1)\pi(2)\cdots\pi(n)=a_{1}a_{2}\cdots a_{n}$. We use $\operatorname{id}_{n}=12\cdots n$ to denote the identity permutation, and $\varepsilon\in S_{0}$ to denote the empty permutation. For any $\pi\in S_{n-1}$ and any $1\leq i\leq n$, we write $c_{i}(\pi)\in S_{n}$ for the permutation obtained from $\pi$ by inserting the new largest value $n$ at position $i$ of $\pi$, i.e., if $\pi=a_{1}\cdots a_{n-1}$ then $c_{i}(\pi)=a_{1}\cdots a_{i-1}\,n\,a_{i}\cdots a_{n-1}$. Moreover, for $\pi\in S_{n}$, we write $p(\pi)\in S_{n-1}$ for the permutation obtained from $\pi$ by removing the largest entry $n$. Here, $c_{i}$ and $p$ stand for the child and parent of a node in the tree of permutations discussed shortly.

Given a permutation $\pi=a_{1}\cdots a_{n}$ with a substring $a_{i}\cdots a_{j}$ with $a_{i}>a_{i+1},\ldots,a_{j}$, a right jump of the value $a_{i}$ by $j-i$ steps is a cyclic left rotation of this substring by one position to $a_{i+1}\cdots a_{j}a_{i}$. Similarly, given a substring $a_{i}\cdots a_{j}$ with $a_{j}>a_{i},\ldots,a_{j-1}$, a left jump of the value $a_{j}$ by $j-i$ steps is a cyclic right rotation of this substring to $a_{j}a_{i}\cdots a_{j-1}$. For example, a right jump of the value 5 in the permutation $265134$ by 2 steps yields $261354$.

2.2. The basic algorithm

Our approach starts with the following simple greedy algorithm to generate a set of permutations $L_{n}\subseteq S_{n}$. We say that a jump is minimal (w.r.t. $L_{n}$), if every jump of the same value in the same direction by fewer steps creates a permutation that is not in $L_{n}$. Note that each entry of the permutation admits at most one minimal left jump and at most one minimal right jump.

Algorithm J (Greedy minimal jumps). This algorithm attempts to greedily generate a set of permutations $L_{n}\subseteq S_{n}$ using minimal jumps starting from an initial permutation $\pi_{0}\in L_{n}$.

1. J1.

[Initialize] Visit the initial permutation $\pi_{0}$. 2. J2.

[Jump] Generate an unvisited permutation from $L_{n}$ by performing a minimal jump of the largest possible value in the most recently visited permutation. If no such jump exists, or the jump direction is ambiguous, then terminate. Otherwise visit this permutation and repeat J2.

Put differently, in step J2 we consider the entries $n,n-1,\ldots,2$ of the current permutation in decreasing order, and for each of them we check whether it allows a minimal left or right jump that creates a previously unvisited permutation, and we perform the first such jump we find, unless the same entry also allows a jump in the opposite direction, in which case we terminate. If no minimal jump creates an unvisited permutation, we also terminate the algorithm. For example, consider $L_{4}=\{1243,1423,4123,4213,2134\}$. Starting with $\pi_{0}=1243$, the algorithm generates $\pi_{1}=1423$ (obtained from $\pi_{0}$ by a left jump of the value 4 by 1 step), then $\pi_{2}=4123$, then $\pi_{3}=4213$ (in $\pi_{2}$, 4 cannot jump, as $\pi_{0}$ and $\pi_{1}$ have been visited before; 3 cannot jump either to create any permutation from $L_{4}$, so 2 jumps left by 1 step), then $\pi_{4}=2134$, successfully generating $L_{4}$. If instead we initialize with $\pi_{0}=4213$, then the algorithm generates $\pi_{1}=2134$, and then stops, as no further jump is possible. If we choose $\pi_{0}=1423$, then we may jump 4 to the left or right (by 1 step), but as the direction is ambiguous, the algorithm stops immediately. As mentioned before, the algorithm may stop before having visited the entire set $L_{n}$ either because no minimal jump leading to a new permutation from $L_{n}$ is possible, or because the direction of jump is ambiguous in some step. By the definition of step J2, the algorithm will never visit any permutation twice.

2.3. Zigzag languages

The following main result of our paper provides a sufficient condition on the set $L_{n}$ to guarantee that Algorithm J is successful (cf. Section 6). This condition is captured by the following closure property of the set $L_{n}$. A set of permutations $L_{n}\subseteq S_{n}$ is called a zigzag language, if either $n=0$ and $L_{0}=\{\varepsilon\}$, or if $n\geq 1$ and $L_{n-1}:=\{p(\pi)\mid\pi\in L_{n}\}$ is a zigzag language satisfying the following condition:

1. (z)

For every $\pi\in L_{n-1}$ we have $c_{1}(\pi)\in L_{n}$ and $c_{n}(\pi)\in L_{n}$.

Theorem 1**.**

Given any zigzag language of permutations $L_{n}$ and initial permutation $\pi_{0}=\operatorname{id}_{n}$, Algorithm J visits every permutation from $L_{n}$ exactly once.

Remark 2*.*

It is easy to see that the number of zigzag languages is at least $2^{(n-1)!(n-2)}=2^{2^{\Theta(n\log n)}}$, i.e., it is more than double-exponential in $n$. We will see that many of these languages do in fact encode interesting combinatorial objects. Moreover, minimal jumps as performed by Algorithm J always translate to small changes on those objects in a provable sense, i.e., our algorithm defines Gray codes for a large variety of combinatorial objects, and Hamilton paths/cycles on the corresponding flip graphs and polytopes.

Before we present the proof of Theorem 1, we give two equivalent characterizations of zigzag languages.

2.3.1. Characterization via the tree of permutations

There is an intuitive characterization of zigzag languages via the tree of permutations. This is an infinite (unordered) rooted tree which has as nodes all permutations from $S_{n}$ at distance $n$ from the root; see Figure 1. Specifically, the empty permutation $\varepsilon$ is at the root, and the children of any node $\pi\in S_{n-1}$ are exactly the permutations $c_{i}(\pi)$, $1\leq i\leq n$, i.e., the permutations obtained by inserting the new largest value $n$ in all possible positions. Consequently, the parent of any node $\pi^{\prime}\in S_{n}$ is exactly the permutation $p(\pi^{\prime})$ obtained by removing the largest value $n$. In the figure, for any node $\pi\in S_{n-1}$, the nodes representing the children $c_{1}(\pi)$ and $c_{n}(\pi)$ are drawn black, whereas the other children are drawn white. Any zigzag language of permutations can be obtained from this full tree by pruning subtrees, where by condition (z) a subtree may be pruned only if its root $\pi^{\prime}\in S_{n}$ is neither the child $c_{1}(\pi)$ nor the child $c_{n}(\pi)$ of its parent $\pi=p(\pi^{\prime})\in S_{n-1}$, i.e., only subtrees rooted at white nodes may be pruned. For any subtree obtained by pruning according to this rule and for any $n\geq 1$, the remaining permutations of length $n$ form a zigzag language $L_{n}$; see Figure 2.

Consider all nodes in the tree for which the entire path to the root consists only of black nodes. Those nodes never get pruned and are therefore contained in any zigzag language. These are exactly all permutations without peaks. A peak in a permutation $a_{1}\cdots a_{n}$ is a triple $a_{i-1}a_{i}a_{i+1}$ with $a_{i-1}<a_{i}>a_{i+1}$, and the language of permutations without peaks is generated by the recurrence $P_{0}:=\{\varepsilon\}$ and $P_{n}:=\{c_{1}(\pi),c_{n}(\pi)\mid\pi\in P_{n-1}\}$ for $n\geq 1$. It follows that we have $|P_{n}|=2^{n-1}$ and $P_{n}\subseteq L_{n}\subseteq S_{n}$ for any zigzag language $L_{n}$, i.e., $L_{n}$ is sandwiched between the language of permutations without peaks and the language of all permutations.

2.3.2. Characterization via nuts

Given a permutation $\pi$, we may repeatedly remove the largest value from it as long as it is in the leftmost or rightmost position, and obtain what is called the nut of $\pi$. For example, given $\pi=965214378$, we can remove $9,8,7,6,5$, yielding $2143$ as the nut of $\pi$. A left or right jump of some value in a permutation is maximum if there is no left jump or right jump of the same value with more steps. For example, in $\pi=965214378$ a maximum right jump of the value $6$ gives $\pi^{\prime}=952143678$. By unrolling the recursive definition of zigzag languages from before, we obtain that $L_{n}\subseteq S_{n}$ is a zigzag language if and only if for all $\pi\in L_{n}$ both the maximum left jump and the maximum right jump of the value $i$ yield another permutation in $L_{n}$ for all $k\leq i\leq n$, where $k$ is the largest value in $\pi$’s nut (with $k=2$ if the nut is empty).

2.4. Proof of Theorem 1

Given a zigzag language $L_{n}$, we define a sequence $J(L_{n})$ of all permutations from $L_{n}$, and we prove that Algorithm J generates the permutations of $L_{n}$ exactly in this order. For any $\pi\in L_{n-1}$ we let $\vv{c}(\pi)$ be the sequence of all $c_{i}(\pi)\in L_{n}$ for $i=1,2,\ldots,n$, starting with $c_{1}(\pi)$ and ending with $c_{n}(\pi)$, and we let \reflectbox{\vv{\reflectbox{$c$}}}(\pi) denote the reverse sequence, i.e., it starts with $c_{n}(\pi)$ and ends with $c_{1}(\pi)$. In words, those sequences are obtained by inserting into $\pi$ the new largest value $n$ from left to right, or from right to left, respectively, in all possible positions that yield a permutation from $L_{n}$, skipping the positions that yield a permutation that is not in $L_{n}$. The sequence $J(L_{n})$ is defined recursively as follows: If $n=0$ then we define $J(L_{0}):=\varepsilon$, and if $n\geq 1$ then we consider the finite sequence $J(L_{n-1})=:\pi_{1},\pi_{2},\ldots$ and define

[TABLE]

i.e., this sequence is obtained from the previous sequence by inserting the new largest value $n$ in all possible positions alternatingly from right to left, or from left to right; see Figure 2.

Remark 3*.*

Algorithm J thus defines a left-to-right ordering of the nodes at distance $n$ of the root in the tree representation of the zigzag language $L_{n}$ described before, and this ordering is captured by the sequence $J(L_{n})$; see Figure 2. Clearly, the same is true for all the zigzag languages $L_{0},L_{1},\ldots,L_{n-1}$ that are induced by $L_{n}$ through the rule $L_{k-1}:=\{p(\pi)\mid\pi\in L_{k}\}$ for $k=n,n-1,\ldots,1$. The unordered tree is thus turned into an ordered tree, and it is important to realize that pruning operations change the ordering. Specifically, given two zigzag languages $L_{n}$ and $M_{n}$ with $M_{n}\subseteq L_{n}$, the tree for $M_{n}$ is obtained from the tree for $L_{n}$ by pruning, but in general $J(M_{n})$ is not a subsequence of $J(L_{n})$, as shown by the example in Figure 2. This shows that our approach is quite different from the one presented by Vajnovszki and Vernay [VV11], which considers only subsequences of the Steinhaus-Johnson-Trotter order $J(S_{n})$.

Proof of Theorem 1.

For any $\pi\in L_{n}$, we let $J(L_{n})_{\pi}$ denote the subsequence of $J(L_{n})$ that contains all permutations up to and including $\pi$. An immediate consequence of the definition of zigzag language is that $L_{n}$ contains the identity permutation $\operatorname{id}_{n}=c_{n}(\operatorname{id}_{n-1})$. Moreover, the definition (1) implies that $\operatorname{id}_{n}$ is the very first permutation in the sequence $J(L_{n})$.

We now argue by double induction over $n$ and the length of $J(L_{n})$ that Algorithm J generates all permutations from $L_{n}$ exactly in the order described by the sequence $J(L_{n})$, and that when we perform a minimal jump with the largest possible value to create a previously unvisited permutation, then there is only one direction (left or right) to which it can jump. The induction basis $n=0$ is clear. Now suppose the claim holds for the zigzag language $L_{n-1}:=\{p(\pi)\mid\pi\in L_{n}\}$. We proceed to show that it also holds for $L_{n}$.

As argued before, the identity permutation $\operatorname{id}_{n}$ is the first permutation in the sequence $J(L_{n})$, and this is indeed the first permutation visited by Algorithm J in step J1. Now let $\pi\in L_{n}$ be the permutation currently visited by the algorithm in step J2, and let $\pi^{\prime}:=p(\pi)\in L_{n-1}$. If $\pi^{\prime}$ appears at an odd position in $J(L_{n-1})$, then we define \bar{c}:=\reflectbox{\vv{\reflectbox{$c$}}}(\pi^{\prime}) and otherwise we define $\bar{c}:=\vv{c}(\pi^{\prime})$. By (1), we know that $\pi$ appears in the subsequence $\bar{c}$ within $J(L_{n})$. We first consider the case that $\pi$ is not the last permutation in $\bar{c}$. In this case, the permutation $\rho$ succeeding $\pi$ in $J(L_{n})$ is obtained from $\pi$ by a minimal jump (w.r.t. $L_{n}$) of the largest value $n$ in some direction $d$, which is left if \bar{c}=\reflectbox{\vv{\reflectbox{$c$}}}(\pi^{\prime}) and right if $\bar{c}=\vv{c}(\pi^{\prime})$. Now observe that by the definition of $\bar{c}$, all permutations in $L_{n}$ obtained from $\pi$ by jumping $n$ in the direction opposite to $d$ precede $\pi$ in $J(L_{n})$ and have been visited by Algorithm J before by induction. Consequently, to generate a previously unvisited permutation, the value $n$ can only jump in direction $d$ in step J2 of the algorithm. Again by the definition of $\bar{c}$, the permutation $\rho$ is obtained from $\pi$ by a minimal jump (w.r.t. $L_{n}$), so the next permutation generated by the algorithm will indeed be $\rho$.

It remains to consider the case that $\pi$ is the last permutation in the subsequence $\bar{c}$ within $J(L_{n})$. Let $\rho^{\prime}$ be the permutation succeeding $\pi^{\prime}$ in $J(L_{n-1})$. By induction, we have the following property (): $\rho^{\prime}$ is obtained from $\pi^{\prime}$ by a minimal jump (w.r.t. $L_{n-1}$) of the largest possible value $a$ by $k$ steps in some direction $d$ (left or right), and $a$ can jump only into one direction. As $\pi$ is the last permutation in $\bar{c}$, the largest value $n$ of $\pi$ is at the boundary, which is the left boundary if \bar{c}=\reflectbox{\vv{\reflectbox{$c$}}}(\pi^{\prime}) or the right boundary if $\bar{c}=\vv{c}(\pi^{\prime})$. By (1), the permutation $\rho$ succeeding $\pi$ in $J(L_{n})$ also has $n$ at the same boundary, i.e., $\rho$ differs from $\pi$ by a jump of the value $a$ by $k$ steps in direction $d$. Suppose for the sake of contradiction that when transforming the currently visited permutation $\pi$ in step J2, the algorithm does not perform this jump operation, but another one. This could be a jump of a larger value $b>a$ to transform $\pi$ into some permutation $\tau\in L_{n}$ that is different from $\rho$ and not in $J(L_{n})_{\pi}$, or a jump of the value $a$ in the direction opposite to $d$, or a jump of the value $a$ in direction $d$ by fewer than $k$ steps. But in all those cases the permutation $\tau^{\prime}:=p(\tau)\in L_{n-1}$ is different from $\rho^{\prime}$ and not in $J(L_{n-1})_{\pi^{\prime}}$, and it is obtained from $\pi^{\prime}$ by a jump of the value $b>a$, or a jump of the value $a$ in the direction opposite to $d$, or a jump of the value $a$ in direction $d$ by fewer than $k$ steps, respectively, a contradiction to property (). This completes the proof. ∎

2.5. Further properties of Algorithm J

The next lemma captures when Algorithm J generates a cyclic listing of permutations.

Lemma 4**.**

In the ordering of permutations $J(L_{n})$ generated by Algorithm J, the first and last permutation are related by a minimal jump if and only if $|L_{i}|$ is even for all $2\leq i\leq n-1$.

For example, the conditions described by Lemma 4 are satisfied for the zigzag languages $L_{n}=S_{n}$ (all permutations) and $L_{n}=P_{n}$ (permutations without peaks), and the resulting cyclic orderings $J(L_{n})$ are shown in Figures 3 and 4, respectively. Another cyclic Gray code is shown in Figure 7. In contrast to that, the Gray codes shown in Figures 5 and 6 violate the conditions of the lemma and are therefore not cyclic.

Proof.

Let $\pi_{i}$ be the last permutation in the ordering $J(L_{i})$ for all $i=0,1,\ldots,n$. For $i\geq 1$, we see from (1) that $\pi_{i}=c_{i}(\pi_{i-1})$ if $|L_{i-1}|$ is even and $\pi_{i}=c_{1}(\pi_{i-1})$ if $|L_{i-1}|$ is odd. As $|L_{1}|=1$ is odd, we know that 1 and 2 are reversed in $\pi_{n}$, and so all numbers $|L_{i}|$, $2\leq i\leq n-1$, must be even for $\operatorname{id}_{n}$ and $\pi_{n}$ to be related by a minimal jump. ∎

Remark 5*.*

It follows from the proof of Theorem 1 that instead of initializing the algorithm with the identity permutation $\pi_{0}=\operatorname{id}_{n}$, we may use any permutation without peaks as a seed $\pi_{0}$.

2.6. Efficiency considerations

Let us make it very clear that in its stated form, Algorithm J is not an efficient algorithm to actually generate a particular zigzag language of permutations. The reason is that it requires storing the list of all previously visited permutations in order to decide which one to generate next. However, by introducing a few additional arrays, the algorithm can be made memoryless, so that such lookup operations are not needed anymore, and hence no permutations need to be stored at all. The efficiency of the resulting algorithm is then only determined by the efficiency with which we are able to compute minimal jumps with respect to the input zigzag language $L_{n}$ for a given entry of the permutation. This leads to an algorithm that computes the next permutation to be visited in polynomial time. In many cases, this can be improved to a loopless algorithm that generates each new permutation in constant worst-case time. The key insight here is that any jump changes the inversion table of a permutation only in a single entry. By maintaining only the inversion table, jumps can thus be performed efficiently, even if the number of steps is big. This extensive and important discussion, however, is not the main focus here, and is deferred to part IV of this paper series.

2.7. A general recipe

Here is a step-by-step approach to apply our framework to the generation of a given family $X_{n}$ of combinatorial objects. The first step is to establish a bijection $f$ that encodes the objects from $X_{n}$ as permutations $L_{n}\subseteq S_{n}$. If $L_{n}$ is a zigzag language, which can be checked by verifying the closure property, then we may run Algorithm J with input $L_{n}$, and interpret the resulting ordering $J(L_{n})$ in terms of the combinatorial objects, by applying $f^{-1}$ to each permutation in $J(L_{n})$, yielding an ordering on $X_{n}$. We may also apply $f^{-1}$ to Algorithm J directly, which will yield a simple greedy algorithm for generating $X_{n}$. The final step is to make these algorithms efficient, by introducing additional data structures that allow the change operations on $X_{n}$ (which are the preimages of minimal jumps under $f$) as efficiently as possible. In the next section we amply illustrate this approach by four examples.

3. Classical Gray codes from our framework

In this section we derive the four classical Gray codes mentioned in the introduction from our framework in a systematic fashion, following the approach outlined in Section 2.7.

3.1. Permutations (Steinhaus-Johnson-Trotter)

Consider the set $X_{n}=L_{n}=S_{n}$ of all permutations of $[n]$. The bijection $f$ between $X_{n}$ and $L_{n}$ here is simply the identity, i.e., $f=\operatorname{id}$. In this case, each jump is a jump by 1 step, i.e., it is an adjacent transposition. Algorithm J thus yields an ordering of permutations by adjacent transpositions, which coincides with the well-known Steinhaus-Johnson-Trotter order, also known as plain change order [Tro62, Joh63], which can be implemented efficiently [Knu11]. This ordering is shown in Figure 3. Algorithm J translates into the following simple greedy algorithm to describe this order (see [Wil13]): J1. Visit the identity permutation. J2. Perform a transposition of the largest possible value with an adjacent smaller entry that yields a previously unvisited permutation; then visit this permutation and repeat J2.

3.2. Binary strings (BRGC)

Consider the set $X_{n}$ of binary strings of length $n-1$. We map any binary string $x=x_{2}\cdots x_{n}$ to a permutation $f(x)\in S_{n}$ by setting $f(\varepsilon):=1$ and

[TABLE]

i.e., we build the permutation $f(x)$ by inserting the values $i=2,\ldots,n$ one by one, either at the leftmost or rightmost position, depending on the bit $x_{i}$; see Figure 4 for examples. Observe that $f(X_{n})$ is exactly the set of permutations without peaks $P_{n}\subseteq S_{n}$ discussed previously in Section 2.3.1, and a jump of the value $i$ in the permutation translates to flipping the bit $x_{i}$. Moreover, $f^{-1}(J(P_{n}))$ is exactly the well-known reflected Gray code (BRGC) for binary strings of length $n-1$ [Gra53], which can be implemented efficiently [BER76]. This ordering is shown in Figure 4. Applying $f^{-1}$ to Algorithm J yields the following simple greedy algorithm to describe the BRGC (see [Wil13]): J1. Visit the all-zero string. J2. Flip the rightmost bit that yields a previously unvisited string; then visit this string and repeat J2.

3.3. Binary trees (Lucas-Roelants van Baronaigien-Ruskey)

Consider the set $X_{n}$ of binary trees with $n$ nodes, labelled with $n$ distinct integers from $[n]$, that have the search tree property, i.e., for every node, all nodes in the left subtree are smaller than all nodes in the right subtree. We recursively map any such tree $x$ with root node $i$, left subtree $x_{L}$, and right subtree $x_{R}$ to a permutation $f(x)\in S_{n}$ by setting $f(x):=(i,f(x_{L}),f(x_{R}))$ and $f(\emptyset):=\varepsilon$ if $x=\emptyset$ has no nodes; see Figure 5 for examples. By the search tree property, $f(X_{n})$ is exactly the set of permutations that avoid the pattern 231, i.e., we have $f(X_{n})=S_{n}(231)$, which we will prove to be a zigzag language. A jump of the value $i$ in the permutation translates to a tree rotation involving the node $i$. Specifically, a right jump of the value $i$ in the permutation corresponds to a right rotation at node $i$, and a left jump corresponds to a left rotation at the parent of node $i$, where node $i$ is the right child of this parent, and these two operations are inverse to each other. Moreover, $f^{-1}(J(S_{n}(231)))$ is exactly the ordering of binary trees described by Lucas, Roelants van Baronaigien, and Ruskey [LRR93], which they showed can be implemented efficiently. This ordering is shown in Figure 5. Applying $f^{-1}$ to Algorithm J yields the following simple greedy algorithm to describe this order (see [Wil13]): J1. Visit the all-right tree, i.e., every node has exactly one child, namely a right child. J2. Perform a rotation involving the largest possible node that yields a previously unvisited tree; then visit this tree and repeat J2.

Via standard bijections, binary trees are equivalent to many other Catalan families (such as triangulations, Dyck paths, etc.), so we also obtain Gray codes for all these other objects; see Figure 5.

3.4. Set partitions (Kaye)

Consider the set $X_{n}$ of partitions of the set $[n]$ into nonempty subsets. We can represent any such partition $x=\{x_{1},\ldots,x_{k}\}$, $x_{i}\subseteq[n]$, in a canonic way, by sorting all subsets in decreasing order of their minimum element, and the elements of each subset in increasing order. Then $x$ is mapped to a permutation $f(x)\in S_{n}$ by writing out the elements from left to right in this canonic representation of $x$. For instance, the set partition $x=\{\{9\},\{6\},\{3,4,7\},\{1,2,5,8\}\}$ is encoded as the permutation $f(x)=963471258$. Observe that $f(X_{n})$ is the set of permutations with the property that for every descent $a_{i}a_{i+1}$, $a_{i}>a_{i+1}$, in the permutation, no value left of $a_{i}$ is smaller than $a_{i+1}$. This notion of pattern-avoidance can be described concisely by the vincular permutation pattern $1\underline{32}$, i.e., we have $f(X_{n})=S_{n}(1\underline{32})$ (the formal definition of vincular patterns is given in the next section), which we will prove to be a zigzag language. A jump of the value $i$ in the permutation corresponds to moving the element $i$ from its subset to the previous or next subset in the canonic representation, possibly creating a singleton set $\{i\}$. Moreover, $f^{-1}(J(S_{n}(1\underline{32})))$ is exactly the ordering of set partitions described by Kaye [Kay76], which he showed can be implemented efficiently. This ordering is shown in Figure 6. Applying $f^{-1}$ to Algorithm J yields the following simple greedy algorithm to describe this order (see [Wil13]): J1. Visit the set partition $\{\{1,\ldots,n\}\}$. J2. Move the largest possible element from its subset to the previous or next subset so as to obtain a previously unvisited partition; then visit this partition and repeat J2.

4. Pattern-avoiding permutations

The first main application of our framework is the generation of pattern-avoiding permutations. Our main results in this section are summarized in Theorem 8, Theorem 15 (and its corollaries Lemmas 9–14), and in Table 1. We emphasize that all our results can be generalized to bounding the number of appearances of patterns, where the special case with a bound of 0 appearances is pattern-avoidance; see Section 4.10 below.

4.1. Preliminaries

The following simple but powerful lemma follows immediately from the definition of zigzag languages given in Section 2.3. For any set $L_{n}\subseteq S_{n}$ we define $p(L_{n}):=\{p(\pi)\mid\pi\in L_{n}\}$.

Lemma 6**.**

Let $L_{n},M_{n}\subseteq S_{n}$, $n\geq 1$, be two zigzag languages of permutations. Then $L_{n}\cup M_{n}$ and $L_{n}\cap M_{n}$ are also zigzag languages of permutations, and we have $p(L_{n}\cup M_{n})=p(L_{n})\cup p(M_{n})$ and $p(L_{n}\cap M_{n})=p(L_{n})\cap p(M_{n})$.

We say that two sequences of integers $\sigma$ and $\tau$ are order-isomorphic, if their elements appear in the same relative order in both sequences. For instance, $2576$ and $1243$ are order-isomorphic. Given two permutations $\pi\in S_{n}$ and $\tau\in S_{k}$, we say that $\pi$ contains the pattern $\tau$, if and only if $\pi=a_{1}\cdots a_{n}$ contains a subpermutation $a_{i_{1}}\cdots a_{i_{k}}$, $i_{1}<\cdots<i_{k}$, that is order-isomorphic to $\tau$. We refer to such a subpermutation as a match of $\tau$ in $\pi$. If $\pi$ does not contain the pattern $\tau$, then we say that $\pi$ avoids $\tau$. For example, $\pi=6\hbox{\pagecolor{black!30}\!35\!}41\hbox{\pagecolor{black!30}\!2\!}$ contains the pattern $\tau=231$, as the highlighted entries form a match of $\tau$ in $\pi$. On the other hand, $\pi=654123$ avoids $\tau=231$. For any permutation $\tau$, we let $S_{n}(\tau)$ denote all permutations from $S_{n}$ avoiding the pattern $\tau$.

For propositional formulas $F$ and $G$ consisting of logical ANDs $\wedge$, ORs $\vee$, and patterns as variables, we define

[TABLE]

For instance, $S_{n}(\tau_{1}\wedge\cdots\wedge\tau_{\ell})$ is the set of permutations avoiding each of the patterns $\tau_{1},\ldots,\tau_{\ell}$, and $S_{n}(\tau_{1}\vee\cdots\vee\tau_{\ell})$ is the set of permutations avoiding at least one of the patterns $\tau_{1},\ldots,\tau_{\ell}$.

Remark 7*.*

From the point of view of counting, we clearly have $|L_{n}\cup M_{n}|=|L_{n}|+|M_{n}|-|L_{n}\cap M_{n}|$, so the problem of counting the union of two zigzag languages can be reduced to counting the individual languages and the intersection. However, from the point of view of exhaustive generation, we clearly do not want to take this approach, namely generate all permutations in $L_{n}$, all permutations in $M_{n}$, all permutations in $L_{n}\cap M_{n}$, and then combine and reduce those lists. This shows that the problem of generating languages like $S_{n}(\tau_{1}\vee\cdots\vee\tau_{\ell})$ or $S_{n}(F)$ for more general formulas $F$ is genuinely interesting in our context. We will see a few applications of this general setting below, and we feel that this direction of generalization deserves further investigation by the pattern-avoidance community.

4.2. Tame patterns

We say that an infinite sequence of sets $L_{0},L_{1},\ldots$ is hereditary, if $L_{i-1}=p(L_{i})$ holds for all $i\geq 1$. We say that a permutation pattern $\tau$ is tame, if $S_{n}(\tau)$, $n\geq 0$, is a hereditary sequence of zigzag languages. The hereditary property ensures that for a given set $S_{n}(\tau)=:L_{n}$, we can check whether a permutation $\pi$ is in the sets $L_{i-1}:=p(L_{i})$ for $i=n,n-1,\ldots,1$ simply by checking for matches of the pattern $\tau$ in $\pi$. See also the discussion in Section 6.

The following theorem is an immediate consequence of Lemma 6 and the definition (2).

Theorem 8**.**

Let $F$ be an arbitrary propositional formula consisting of logical ANDs $\wedge$, ORs $\vee$, and tame patterns as variables, then $S_{n}(F)$, $n\geq 0$, is a hereditary sequence of zigzag languages. Consequently, all of these languages can be generated by Algorithm J.

In the following we provide simple sufficient conditions guaranteeing that a pattern is tame (cf. Section 6).

Lemma 9**.**

If a pattern $\tau\in S_{k}$, $k\geq 3$, does not have the largest value $k$ at the leftmost or rightmost position, then it is tame.

We prove Lemma 9 in Section 4.9.

Table 1 lists several tame patterns and the combinatorial objects encoded by the corresponding zigzag languages. The bijections between those permutations and the combinatorial objects are well-known and are described in the listed papers (recall also Section 3). The ordering of $231$-avoiding permutations of length $n=4$ generated by Algorithm J, and the corresponding Gray codes for three different Catalan objects are shown in Figure 5. We refer to the permutation patterns discussed so far as classical patterns. In the following we discuss some other important variants of permutation patterns appearing in the literature.

4.3. Vincular patterns

Vincular patterns were introduced by Babson and Steingrímsson [BS00]. In a vincular pattern $\tau$, there is exactly one underlined pair of consecutive entries, with the interpretation that a match of $\tau$ in $\pi$ requires that the underlined entries match adjacent positions in $\pi$. For instance, the permutation $\pi=\hbox{\pagecolor{black!30}\!3\!}1\hbox{\pagecolor{black!30}\!4\!}\hbox{\pagecolor{black!30}\!2\!}$ contains the pattern $\tau=231$, but it avoids the vincular pattern $\tau=\underline{23}1$.

Lemma 10**.**

If a vincular pattern $\tau\in S_{k}$, $k\geq 3$, does not have the largest value $k$ at the leftmost or rightmost position, and the largest value $k$ is part of the vincular pair, then it is tame.

We prove Lemma 10 in Section 4.9.

Table 1 also lists several tame vincular patterns and the combinatorial objects encoded by the corresponding zigzag languages, namely set partitions and different kinds of rectangulations. The ordering of $1\underline{32}$-avoiding permutations of length $n=4$ generated by Algorithm J, and the resulting Gray code for set partitions, is shown in Figure 6. The generated ordering of twisted Baxter permutations of length $n=4$, and the resulting Gray code for diagonal rectangulations, is shown in Figure 7.

4.4. Barred patterns

Barred permutation patterns were first considered by West [Wes90]. A barred pattern is a pattern $\tau$ with a number of overlined entries, e.g., $\tau=25\overline{3}41$. Let $\tau^{\prime}$ be the permutation obtained by removing the bars in $\tau$, and let $\tau^{-}$ be the permutation that is order-isomorphic to the non-barred entries in $\tau$. In our example, we have $\tau^{\prime}=25341$ and $\tau^{-}=2431$. A permutation $\pi$ contains a barred pattern $\tau$ if and only if it contains a match of $\tau^{-}$ that cannot be extended to a match of $\tau^{\prime}$ by adding entries of $\pi$ at the positions specified by the barred entries. For instance, $\pi=\hbox{\pagecolor{black!30}\!35\!}2\hbox{\pagecolor{black!30}\!41\!}$ contains $\tau=25\overline{3}41$, as the highlighted entries form a match of $\tau^{-}=2431$ that cannot be extended to a match of $\tau^{\prime}=25341$. We clearly have $S_{n}(\tau^{-})\subseteq S_{n}(\tau)$.

The following lemma gives a sufficient condition for a single-barred pattern to be tame.

Lemma 11**.**

If for a single-barred pattern $\tau\in S_{k}$, $k\geq 4$, the permutation $\tau^{-}\in S_{k-1}$ does not have the largest value $k-1$ at the leftmost or rightmost position, and the barred entry in $\tau$ is smaller than $k$ or at a position next to the entry $k-1$, then $\tau$ is tame.

We prove Lemma 11 in Section 4.9. See Table 1 for several examples of tame single-barred patterns that were studied by Pudwell [Pud10]. As we will show in Section 5.3 below, in many cases patterns with multiple bars can be reduced to single-barred patterns.

4.5. Boxed patterns

Boxed patterns were introduced in the paper by Avgustinovich, Kitaev, and Valyuzhenich [AKV13]. A permutation $\pi$ contains the boxed pattern $\boxed{\tau}$ if and only if it contains a match of $\tau$ such that no entry of $\pi$ at a position between the matched ones has a value between the smallest and largest value of the match. For example, the permutation $\pi=\hbox{\pagecolor{black!30}\!43\!}1\hbox{\pagecolor{black!30}\!7\!}928\hbox{\pagecolor{black!30}\!6\!}5$ contains the boxed pattern $\boxed{2143}$, as the highlighted entries of $\pi$ form a match of $2143$, and the entries $1,9,2,8$ are either smaller than 3 or larger than 7. On the other hand, the permutation $\pi^{\prime}=\hbox{\pagecolor{black!30}\!3\!}5\hbox{\pagecolor{black!30}\!16\!}2\hbox{\pagecolor{black!30}\!4\!}$ avoids $\boxed{2143}$, as the only possible match of $2143$ is at the highlighted positions in $\pi^{\prime}$, but the entries 5 and 2 are both between 1 and 6.

Lemma 12**.**

Given a boxed pattern $\boxed{\tau}$ with $\tau\in S_{k}$ and $k\geq 3$, if $\tau$ does not have the largest value $k$ at the leftmost or rightmost position, then it is tame.

We prove Lemma 12 in Section 4.9. Table 1 shows two tame boxed patterns studied in [AKV13].

4.6. Patterns with Bruhat restrictions

Patterns with Bruhat restrictions were introduced by Woo and Yong [WY06]. Such a pattern is a pair $(\tau,B)$, where $\tau\in S_{k}$ and $B\subseteq[k]^{2}$ is a set of pairs of indices $(a,b)$ with $a<b$ and $\tau(a)<\tau(b)$ such that for all $i\in\{a+1,\ldots,b-1\}$ we either have $\tau(i)<\tau(a)$ or $\tau(i)>\tau(b)$. A permutation $\pi$ contains this pattern if and only if it contains a match of $\tau$, and for any pair of entries $\pi(i_{a})$ and $\pi(i_{b})$ that are matched by a corresponding pair of entries $\tau(a)$ and $\tau(b)$ with $(a,b)\in B$, we have that $\pi(i)<\pi(i_{a})$ or $\pi(i)>\pi(i_{b})$ for all $i\in\{i_{a}+1,\ldots,i_{b}-1\}$.

Lemma 13**.**

Given a pattern with Bruhat restrictions $(\tau,B)$ with $\tau\in S_{k}$ and $k\geq 3$, if $\tau$ does not have the largest value $k$ at the leftmost or rightmost position, then it is tame.

We prove Lemma 13 in Section 4.9. Note that Lemma 13 does not impose any restrictions on the set $B$, and that it hence generalizes Lemma 9, which corresponds to the case $B=\emptyset$. Table 1 shows two patterns with Bruhat restriction studied in [WY06].

4.7. Bivincular patterns

Bivincular patterns were introduced by Bousquet-Mélou, Claesson, Dukes, and Kitaev [BMCDK10]. Such a pattern is a pair $(\tau,B)$, where $\tau\in S_{k}$ is a vincular pattern and $B\subseteq[k-1]$. A permutation $\pi$ contains this pattern if and only if it contains a match of the vincular pattern $\tau$ (respecting the adjacency condition for the vincular pair), and in this match, the entries at positions $\tau^{-1}(i)$ and $\tau^{-1}(i+1)$ are consecutive values in $\pi$ for all $i\in B$.

Lemma 14**.**

Given a bivincular pattern $(\tau,B)$ with $\tau\in S_{k}$ and $k\geq 3$, if the vincular pattern $\tau$ satisfies the conditions in Lemma 10 and if $k-1\notin B$, then it is tame.

We prove Lemma 14 in Section 4.9. Note that Lemma 14 generalizes Lemma 10, which corresponds to the case $B=\emptyset$. In Table 1, $(\mathbf{2}+\mathbf{2})$-free posets are mentioned as an example of a combinatorial class that is in bijection to permutations avoiding a tame bivincular pattern.

4.8. Mesh patterns

In the following we take a geometric viewpoint on permutations. For any pair of real numbers $P=(a,b)$, we define $P_{x}:=a$ and $P_{y}:=b$. Moreover, for any mapping $\alpha:[n]\rightarrow B$ and any subset $I\subseteq[n]$ with $|I|=k$ we write $\alpha|_{I}:[k]\rightarrow B$ for the function defined by $\alpha|_{I}(i):=\alpha(j)$, where $j$ is the $i$th smallest element in $I$, for all $i\in[k]$.

The grid representation of a permutation $\pi\in S_{n}$ is a mapping $\alpha:[n]\rightarrow\mathbb{Z}^{2}$ such that the sequence $\alpha(1)_{x},\ldots,\alpha(n)_{x}$ is strictly increasing, and the sequence $\alpha(1)_{y},\ldots,\alpha(n)_{y}$ is order-isomorphic to $\pi$. This representation is unique up to shifts that preserve the relative order of the values $\alpha(1)_{y},\ldots,\alpha(n)_{y}$. We can think of the grid representation of $\pi$ as a graphical representation of the permutation matrix. For instance, the permutation $\pi=14352$ has the grid representation shown on the right. A cell in the grid representation $\alpha$ is a connected region from \mathbb{R}^{2}\setminus\big{(}\alpha([n])_{x}\times\mathbb{R}\;\cup\;\mathbb{R}\times\alpha([n])_{y}\big{)}, and we denote these cells by $c_{\alpha}(i,j)$, $i,j\in\{0,\ldots,n\}$, where the first index $i$ increases with $x$, and the second index $j$ increases with $y$, as shown in the figure on the right. By definition, every cell is a Cartesian product of two open intervals. For instance, we have $c_{\alpha}(1,0)=\left]\alpha(1)_{x},\alpha(2)_{x}\right[\times\left]-\infty,\alpha(1)_{y}\right[$. If the grid representation $\alpha$ is clear from the context, we sometimes refer to a cell $c_{\alpha}(i,j)$ simply by its index $(i,j)$.

Mesh patterns were introduced by Brändén and Claesson [BC11], and they generalize all the aforementioned types of patterns.

A mesh pattern is a pair $\sigma=(\tau,C)$, $\tau\in S_{k}$, with $C\subseteq\{0,\ldots,k\}^{2}$. Each pair $(a,b)\in C$ encodes the cell with index $(a,b)$ in the grid representation of $\tau$, and in our figures we visualize $\sigma$ by drawing the grid representation of $\tau$ and by shading exactly the cells indexed by $C$. For instance, the mesh pattern $\sigma=(\tau,C)=(14352,\{(0,1),(4,3)\})$ has the graphical representation shown on the right.

A permutation $\pi\in S_{n}$ contains the mesh pattern $\sigma=(\tau,C)$, $\tau\in S_{k}$, if the grid representation $\alpha$ of $\pi$ admits a subset $I\subseteq[n]$, $|I|=k$, such that $\beta:=\alpha|_{I}$ is the grid representation of $\tau$, and the cell $c_{\beta}(i,j)$ contains no points from $\alpha([n])$ for all $(i,j)\in C$. In this case, we refer to $\beta$ as a match of the mesh pattern $\sigma$ in the grid representation $\alpha$ of $\pi$. Note that the first condition in the definition of mesh pattern containment is equivalent to requiring that the subpermutation of $\pi$ on the indices in $I$ is order-isomorphic to $\tau$, while the second condition requires that the cells of $C$ in the grid representation of $\tau$ in $\pi$ contain no points from $\pi$.

For example, consider the grid representation $\alpha$ of each of the two permutations $\pi=14352$ and $\pi^{\prime}=153642$ shown on the right. While $\pi$ clearly contains the mesh pattern $\sigma$ from before, the permutation $\pi^{\prime}$ avoids it, as the only choice for $I\subseteq[5]$ such that $\beta:=\alpha|_{I}$ is the grid representation of $\tau$ is $I=\{1,2,3,4,6\}$ (marked points in the figure), but then the cell $c_{\beta}(4,3)$ (shaded gray) contains the point $\alpha(5)$ (non-marked).

The following main theorem of this section implies all the lemmas about classical, vincular, barred patterns, etc. stated in the previous sections.

Theorem 15**.**

Let $\sigma=(\tau,C)$, $\tau\in S_{k}$, $k\geq 3$, be a mesh pattern, and let $i$ be the position of the largest value $k$ in $\tau$. If the pattern satisfies each of the following four conditions, then it is tame:

1. (i)

$i$* is different from 1 and $k$.* 2. (ii)

For all $a\in\{0,\ldots,k\}\setminus\{i-1,i\}$, we have $(a,k)\notin C$. 3. (iii)

If $(i-1,k)\in C$, then for all $a\in\{0,\ldots,k\}\setminus\{i-1\}$ we have $(a,k-1)\notin C$ and for all $b\in\{0,\ldots,k-2\}$ we have that $(i,b)\in C$ implies $(i-1,b)\in C$. 4. (iv)

If $(i,k)\in C$, then for all $a\in\{0,\ldots,k\}\setminus\{i\}$ we have $(a,k-1)\notin C$ and for all $b\in\{0,\ldots,k-2\}$ we have that $(i-1,b)\in C$ implies $(i,b)\in C$.

The conditions in Theorem 15 can be understood in the grid representation of $(\tau,C)$ as follows; see the left hand side of Figure 8: Condition (i) asserts that the highest point of $\tau$ must not be the leftmost or rightmost point (the two crossed out grid points in the figure are forbidden). Condition (ii) asserts that none of the cells in the topmost row (above the points) must be shaded, with the possible exception of the cells next to the highest point (solid crossed out cells in the figure). Condition (iii) asserts that if the cell $(i-1,k)$ to the top left of the highest point is shaded (dark gray cell in the figure), then none of the cells in the row below except possibly $(i-1,k-1)$ must be shaded (dotted crossed out cells), and if one of the cells strictly below $(i,k-1)$ is shaded (dark gray questions marks), then the cell to the left of it must also be shaded (indicated by an arrow to the left). Symmetrically, condition (iv) asserts that if the cell $(i,k)$ to the top right of the highest point is shaded (light gray cell in the figure), then none of the cells in the row below except possibly $(i,k-1)$ must be shaded (dashed crossed out cells), and if one of the cells strictly below $(i-1,k-1)$ is shaded (light gray questions marks), then the cell to the right of it must also be shaded (indicated by an arrow to the right).

To illustrate the conditions in Theorem 15 further, consider the following mesh patterns $\sigma_{k}=(3241,C_{k})$, $k=1,\ldots,7$, with different sets $C_{k}$:

$\sigma_{1}$

$\sigma_{2}$

$\sigma_{3}$

$\sigma_{4}$

$\sigma_{5}$

$\sigma_{6}$

$\sigma_{7}$

All of these mesh patterns satisfy condition (i). The pattern $\sigma_{1}$ violates condition (ii) due to the cell $(1,4)\in C_{1}$ (crossed in the figure), while all other patterns satisfy this condition. The pattern $\sigma_{2}$ satisfies conditions (iii) and (iv) trivially, as the premises of each of the two conditions are not satisfied. The pattern $\sigma_{3}$ violates the first part of condition (iii) due to the cell $(1,3)\in C_{3}$ (crossed). The pattern $\sigma_{4}$ satisfies the first part of condition (iii), but violates the second part due to the cells $(3,0)\in C_{4}$ and $(2,0)\notin C_{4}$ (connected by an arrow in the figure). The pattern $\sigma_{5}$ satisfies conditions (iii) and (iv). The pattern $\sigma_{6}$ violates condition (iv) due to the cells $(2,1)\in C_{6}$ and $(3,1)\notin C_{6}$ (arrow). Finally, the pattern $\sigma_{7}$ satisfies conditions (iii) and (iv).

Proof.

We show that if $\sigma=(\tau,C)$ satisfies the four conditions of the theorem, then $S_{n}(\sigma)$, $n\geq 0$, is a hereditary sequence of zigzag languages. We argue by induction on $n$. Note that $S_{0}(\sigma)=S_{0}=\{\varepsilon\}$ is a zigzag language by definition, so the induction basis is clear. For the induction step let $n\geq 1$. We first show that if $\pi\in S_{n-1}(\sigma)$, then $c_{1}(\pi),c_{n}(\pi)\in S_{n}(\sigma)$. As $c_{1}(\pi)$ and $c_{n}(\pi)$ are obtained from $\pi$ by inserting the new largest value $n$ at the leftmost or rightmost position, respectively, the grid representation of these two permutations differs from the grid representation of $\pi$ by adding a new highest point at the leftmost or rightmost position. However, as $\pi$ avoids $\sigma$ by assumption, condition (i) guarantees that both $c_{1}(\pi)$ and $c_{n}(\pi)$ also avoid $\sigma$, which is what we wanted to show.

To complete the induction step, we now show that if $\pi\in S_{n}(\sigma)$, then $p(\pi)\in S_{n-1}(\sigma)$. Recall that $p(\pi)$ is obtained from $\pi$ by removing the largest value $n$, so in the grid representation, we remove the highest point $P$. Our assumption is that $\pi$ avoids the pattern $\sigma$, and we need to show that removing the highest point does not create a match of the pattern $\sigma$. For the sake of contradiction, suppose that removing $P$ creates a match of the pattern $\sigma$ in $p(\pi)$. Let $Q$ be the highest point in this match of the pattern $\sigma$ in $p(\pi)$. By condition (ii), we are in exactly one of the following two symmetric cases: (a) the cell $(i-1,k)$ is in $C$ and $P$ lies inside this cell of $\sigma$ in this match of the pattern; (b) the cell $(i,k)$ is in $C$ and $P$ lies inside this cell of $\sigma$ in this match of the pattern. We first consider case (a), which is illustrated on the right hand side of Figure 8: We claim that we can exchange the point $Q$ for the point $P$ in the match of the pattern $\sigma$, and obtain another match of $\sigma$ in $\pi$, which would contradict the assumption that $\pi$ avoids $\sigma$. Indeed, this exchange operation strictly enlarges only the cells $(a,k-1)$ for all $a\in\{0,\ldots,k\}\setminus\{i-1\}$ and the cells $(i,b)$ for all $b\in\{0,\ldots,k-2\}$. The first set of cells are not in $C$ by the first part of condition (iii). The second set of cells are either not in $C$, or if they are, then the corresponding cells to the left of it are also in $C$ by the second part of condition (iii). Moreover, after the exchange the cell $(i,k)$ contains no point from $\pi$, as $P$ is the highest point (this is of course only relevant if $(i,k)\in C$). Furthermore, after the exchange the cell $(i-1,k-1)$ contains at most those points from $\pi$ that were in the same cell before the exchange (clearly $P$ is the only point inside the cell $(i-1,k)$). So we indeed obtain a match of $\sigma$ in $\pi$, a contradiction.

In the symmetric case (b), we apply the same exchange argument, using condition (iv) instead of (iii). This completes the proof. ∎

4.9. Proof of Lemmas 9–14

With Theorem 15 in hand, the proofs of Lemmas 9–14 are straightforward. As noted before, Lemma 13 generalizes Lemma 9, and Lemma 14 generalizes Lemma 10, so we only need to prove Lemmas 11–14.

Proof of Lemma 11.

Note that a barred pattern $\tau\in S_{k}$ with a single barred entry $b$ at position $a$ corresponds to the mesh pattern $\sigma=(\tau^{-},\{(a-1,b-1)\})$, i.e., in the grid representation of $\sigma$ a single cell is shaded. It follows that conditions (iii) and (iv) of Theorem 15 are trivially satisfied, and conditions (i) and (ii) translate into the conditions in the lemma. ∎

Proof of Lemma 12.

A boxed pattern $\boxed{\tau}$, $\tau\in S_{k}$, corresponds to the mesh pattern $\sigma=(\tau,C)$ with $C:=\{(i,j)\mid 1\leq i,j\leq k-1\}$, i.e., in the grid representation of $\sigma$, all cells inside the bounding box of the points from $\tau$ are shaded. It follows that conditions (ii)–(iv) of Theorem 15 are trivially satisfied, and condition (i) corresponds exactly to the condition in the lemma. ∎

Proof of Lemma 13.

A pattern with Bruhat restrictions $(\tau,B)$ corresponds to the mesh pattern $\sigma=(\tau,C)$ where $C$ is the union of all the sets $R(a,b):=\{(i,j)\mid a\leq i<b\wedge\tau(a)\leq j<\tau(b)\}$ for $(a,b)\in B$, i.e., in the grid representation of $\sigma$, certain rectangles of cells inside the bounding box of the points from $\tau$ are shaded. It follows that conditions (ii)–(iv) of Theorem 15 are trivially satisfied, and condition (i) corresponds exactly to the condition in the lemma. ∎

Proof of Lemma 14.

A vincular pattern $\tau\in S_{k}$ where the entries at positions $a$ and $a+1$ are underlined corresponds to the mesh pattern $\sigma=(\tau,C)$ with $C:=\{a\}\times\{0,\ldots,k\}$, i.e., in the grid representation of $\sigma$, an entire column of cells is shaded. For the bivincular pattern $(\tau,B)$ we also have to add the sets $\{0,\ldots,k\}\times\{b\}$ for all $b\in B$ to the set of cells $C$, i.e., in the grid representation we also have to shade the corresponding rows of cells. By the conditions stated in Lemma 10, conditions (i) and (ii) of Theorem 15 are satisfied. By the condition $k-1\notin B$, conditions (iii) and (iv) of the theorem are also satisfied, proving that the mesh pattern $\sigma$ is tame. ∎

4.10. Patterns with multiplicities

All the aforementioned notions and results in this section generalize straightforwardly to bounding the number of appearances of a pattern. Formally, a counted pattern is a pair $\sigma=(\tau,c)$, where $\tau$ is a mesh pattern, and $c$ is a non-negative integer. Moreover, $S_{n}(\sigma)$ denotes the set of all permutations from $S_{n}$ that contain at most $c$ matches of the pattern $\tau$, where the special case $c=0$ is pattern-avoidance (cf. [NZ96]).

By Theorem 8, we can form propositional formulas $F$ consisting of logical ANDs $\wedge$, ORs $\vee$, and tame counted patterns $(\tau_{i},c_{i})$ as variables, with possibly different counts $c_{i}$ for each variable. The tameness of each $(\tau_{i},c_{i})$ can be checked by verifying whether the patterns $\tau_{i}$ satisfy the conditions stated in Theorem 15 or its corollaries Lemmas 9–14. We then obtain a hereditary zigzag language $S_{n}(F)$ that can be generated by Algorithm J.

A somewhat contrived example for such a language would be F=\big{(}(231,3)\wedge(2143,5)\big{)}\vee(3\underline{14}2,2), the language of permutations that contain at most 3 matches of the pattern $231$ AND at most 5 matches of the pattern $2143$, OR at most 2 matches of the vincular pattern $3\underline{14}2$.

5. Algebra with patterns

In this section we significantly extend the methods described in the previous section, by applying geometric transformations to permutation patterns, and by describing some other types of patterns as conjunctions and disjunctions of suitable mesh patterns (recall Theorem 8). Two particularly relevant additional types of permutations covered in this section are monotone and geometric grid classes; see Theorem 21 below.

5.1. Elementary transformations

We now consider three important elementary transformations of permutations that are important in the context of pattern-avoidance, as they preserve the cardinality of the set $S_{n}(F)$. Each of them corresponds to a geometric transformation of the grid representation of each of the patterns $\tau=a_{1}\cdots a_{k}$ in the formula $F$, and together these transformations form the dihedral group $D_{4}$ of symmetries of a regular 4-gon:

• •

Reversal, defined as $\operatorname{rev}(\tau):=a_{k}\cdots a_{1}$. This corresponds to a vertical reflection of the grid representation.

• •

Complementation, defined as $\operatorname{cpl}(\tau)_{i}=k+1-a_{i}$ for all $i=1,\ldots,k$. This corresponds to a horizontal reflection of the grid representation.

• •

Inversion, defined by $\operatorname{inv}(\tau)_{\tau(i)}=i$ for all $i=1,\ldots,k$. This corresponds to a diagonal reflection of the grid representation along the south-west to north-east diagonal.

Note that a clockwise 90-degree rotation is obtained as $\operatorname{rot}(\tau):=\operatorname{inv}(\operatorname{rev}(\tau))=\operatorname{cpl}(\operatorname{inv}(\tau))$. Clearly, all these operations generalize to mesh patterns $(\tau,C)$, by applying the aforementioned geometric transformations to the grid representation of $(\tau,C)$. These operations and their relations are illustrated in Figure 9 for $(\tau,C)=(14352,\{(1,0),(1,1),(3,3),(4,3)\})$.

The following lemma is immediate.

Lemma 16**.**

Given any composition $h$ of the elementary transformations reversal, complementation and inversion, and any propositional formula $F$ consisting of logical ANDs $\wedge$, ORs $\vee$, and mesh patterns $\tau_{1},\ldots,\tau_{\ell}$ as variables, then the sets of pattern-avoiding permutations $S_{n}(F)$ and $S_{n}(h(F))$ are in bijection under $h$ for all $n\geq 1$, where the formula $h(F)$ is obtained from $F$ by replacing every pattern $\tau_{i}$ by $h(\tau_{i})$ for all $i=1,\ldots,\ell$.

Lemma 16 is very useful for the purpose of exhaustive generation, because even if $\tau_{i}$ is not tame, then maybe $h(\tau_{i})$ is. So even if we cannot apply Algorithm J to generate $S_{n}(\tau)$ directly, we may be able to generate $S_{n}(h(\tau_{i}))$, and then apply $h^{-1}$ to the resulting permutations. For instance, $\tau=213$ is not tame, as the largest entry appears at the rightmost position. However, $\operatorname{cpl}(\tau)=231$ is tame by Lemma 9, and so we can use Algorithm J to generate $S_{n}(\operatorname{cpl}(\tau))$.

As another example, consider so-called 2-stack sortable permutations introduced by West [Wes90] and later counted in [Zei92, GW96, DGG98]. These permutations are characterized by the pattern-avoidance formula $F=\tau_{1}\wedge\tau_{2}$ with $\tau_{1}:=2341$ and $\tau_{2}:=3\overline{5}241$ ($\tau_{2}$ is a barred pattern). Unfortunately, $\tau_{2}$ is not tame (the barred entry $\overline{5}$ is not at a position next to the entry 4; recall Lemma 11), so Algorithm J cannot be used directly for generating $S_{n}(F)$. However, applying rotation, $h(\tau):=\operatorname{rot}(\tau)=\operatorname{inv}(\operatorname{rev}(\tau))$, yields two tame patterns $h(\tau_{1})=1432$ and $h(\tau_{2})=1352\overline{4}$ and the formula $h(F)=h(\tau_{1})\wedge h(\tau_{2})$, which can be used for generating $S_{n}(h(F))$ via Algorithm J:

\tau_{1}=2341$$\tau_{2}=3\overline{5}241$$\wedge$$\wedge$$h=\operatorname{rot}$$\operatorname{rot}(\tau_{1})=1432$$\operatorname{rot}(\tau_{2})=1352\overline{4}

Table 2 lists several more permutations patterns that have been studied in the literature and that can be turned into tame patterns by such elementary transformations.

5.2. Partially ordered patterns

Partially ordered patterns were introduced by Kitaev [Kit05]. A partially ordered pattern (POP) is a partially ordered set $P=([k],\prec)$, and we say that a permutation $\pi$ contains this pattern if and only if it contains a subpermutation $a_{i_{1}}\cdots a_{i_{k}}$, $i_{1}<\cdots<i_{k}$, such that $k\prec l$ in the partial order implies that $a_{i_{k}}<a_{i_{l}}$. In particular, if $\prec$ is a linear order, then this is equivalent to classical pattern avoidance. However, some other constraints can be expressed much more conveniently using POPs. For instance, avoiding the POP

$P_{1}=$213

is equivalent to avoiding peaks in the permutation, so $S_{n}(P_{1})$ is the set of permutations without peaks discussed before, which satisfies $|S_{n}(P_{1})|=2^{n-1}$.

More generally, the POP $P_{k}$ defined by

$P_{1}=$213

, 

$P_{2}=$24135 , 

$P_{3}=$2461357 $,\ldots,$ 

$P_{k}=$24135\ldots$$2k$$2k-1$$2k+1

realizes the language $S_{n}(P_{k})$ of permutations with at most $k-1$ peaks.

We let $L(P)$ denote the set of all linear extensions of the poset $P$, and for any linear extension $x\in L(P)$, we consider the inverse permutation of $x$, as the $i$th entry of $\operatorname{inv}(x)$ denotes the position of $i$ in $x$. Moreover, $\operatorname{inv}(x)\in S_{k}$, so $\operatorname{inv}(x)$ is a classical pattern.

Lemma 17**.**

For any partially ordered pattern $P=([k],\prec)$, we have

[TABLE]

In particular, if the poset $P$ does not have 1 or $k$ as a maximal element, then $P$ is tame.

Proof.

The first part of the lemma follows immediately from the definition of POPs and from (2). To prove the second part, suppose that $P$ does not have 1 or $k$ as a maximal element. Then in any linear extension $x\in L(P)$, 1 and $k$ will not appear at the last position, and so in the inverse permutation $\operatorname{inv}(x)$, the largest entry $k$ will neither be at position 1 nor at position $k$. We can hence apply Lemma 9, and using Theorem 8 we obtain that $P$ is tame. ∎

For instance, for the POP $P_{1}$ from before we have $L(P_{1})=\{132,312\}$, and so $P_{1}=132\wedge 231$, and for the POP $P_{2}$ we have $L(P_{2})=\{13254,13524,13542,15324,15342,31254,\ldots\}$, a set of 16 linear extensions in total, so $P_{2}=13254\wedge 14253\wedge 15243\wedge 14352\wedge 15342\wedge 23145\wedge\cdots$.

Moreover, we can create counted POPs with multiplicity $c$ (recall Section 4.10), by taking the OR of conjunctions of counted classical patterns as described by Lemma 17, over all number partitions of $c$ into the corresponding number of parts. For instance, the counted POP $\sigma=(P_{1},c)$, which realizes the zigzag language $S_{n}(\sigma)$ of permutations with at most $c$ triples of values forming a peak, is obtained by considering the partitions $c=c+0=(c-1)+1=\cdots=1+(c-1)=0+c$, resulting in the formula

[TABLE]

with counted classical patterns on the right-hand side.

5.3. Barred patterns with multiple bars

Some patterns with multiple bars can be reduced to single-barred patterns (to which Lemma 11 applies) as shown by the following lemma.

Lemma 18** (cf. [Úlf11]).**

Let $\tau\in S_{k}$, $k\geq 5$, be a pattern with $b\geq 2$ bars, such that no two barred entries are at neighboring positions or have adjacent values. Let $\widetilde{\tau}_{1},\ldots,\widetilde{\tau}_{b}\in S_{k-b+1}$ be the permutations with a single barred entry that are order-isomorphic to the sequences obtained from $\tau$ by removing all but one barred entry. Then we have

[TABLE]

Consequently, if $\tau^{-}\in S_{k-b}$ does not have the largest value $k-b$ at the leftmost or rightmost position, and the largest barred entry in $\tau$ is smaller than $k$ or at a position next to the entry $k-1$, then $\tau$ is tame.

Proof.

To prove the first part, observe that when no two barred entries are at neighboring positions or have adjacent values, then the definition of barred pattern avoidance is equivalent to avoiding each of the single-barred patterns $\widetilde{\tau}_{1},\ldots,\widetilde{\tau}_{b}$, so the claim follows using (2).

To prove the second part we show that each of the single-barred patterns $\widetilde{\tau}_{1},\ldots,\widetilde{\tau}_{b}$ satisfies the conditions of Lemma 11. Indeed, we know that $\tau^{-}=(\widetilde{\tau}_{i})^{-}\in S_{k-b}$, $1\leq i\leq b$, does not have the largest value $k-b$ at the leftmost or rightmost position. Moreover, if $\widetilde{\tau}_{i}$ is obtained from $\tau$ by removing all but the largest barred entry, then the barred entry in $\widetilde{\tau}_{i}\in S_{k-b+1}$ is either smaller than $k-b+1$ or at a position next to the entry $k-b$. To see this note that if the largest entry $k$ in $\tau$ is barred, then the second largest entry $k-1$ is not barred by the assumption that no two barred entries have adjacent values. For the same reason, if $\widetilde{\tau}_{i}$ is obtained from $\tau$ by removing barred entries including the largest one, then the barred entry in $\widetilde{\tau}_{i}\in S_{k-b+1}$ is smaller than $k-b+1$. Consequently, we can apply Lemma 11 to each of the patterns $\widetilde{\tau}_{1},\ldots,\widetilde{\tau}_{b}$, and complete the proof by applying Theorem 8. ∎

Lemma 18 applies for instance to the tame pattern $3\overline{1}52\overline{4}=3\overline{1}42\wedge 241\overline{3}$ listed in Table 1.

5.4. Weak avoidance of barred patterns and dotted patterns

Weak pattern avoidance and dotted patterns were introduced by Baril [Bar11] (see also [DRT14]). Given a single-barred pattern $\tau$, we define $\tau^{\prime}$ and $\tau^{-}$ as in Section 4.4, and we say that a permutation $\pi$ weakly contains $\tau$, if and only if it contains a match of $\tau^{-}$ that cannot be extended to a match of $\tau^{\prime}$ by adding one entry of $\pi$, not necessarily at the position specified by the barred entry. Otherwise, we say that $\pi$ weakly avoids $\tau$. We let $S_{n}^{w}(\tau)$ denote the set of permutations that weakly avoid $\tau$. For instance, $\pi=1243$ contains the barred pattern $\tau=12\overline{3}$, as for the increasing pair $24$ in $\pi$, there is no entry in $\pi$ to the right that extends this pair to an increasing triple. However, $\pi$ weakly avoids $\tau$, as each of the increasing pairs $12$, $14$, $13$, $24$ and $23$ can be extended to an increasing triple, by adding an entry from $\pi$ to the right, middle, middle, left and left of this pair, respectively.

The relation between weak pattern avoidance and mesh pattern avoidance is captured by the following lemma.

Lemma 19**.**

Let $\tau\in S_{k}$ be a single-barred pattern with barred entry $b$. Consider the longest increasing or decreasing substring of consecutive values in $\tau^{\prime}$ that contains $b$, and let $r$ and $s$ be the start and end indices of this substring. Let $\sigma$ be the mesh pattern defined by $\sigma:=(\tau^{-},C)$ with

[TABLE]

Then we have $S_{n}^{w}(\tau)=S_{n}(\sigma)$.

Table 3 illustrates the mesh pattern $\sigma=(\tau^{-},C)$ defined in Lemma 19 for six different single-barred patterns.

Proof.

We only consider the case that the longest substring of consecutive values in $\tau^{\prime}$ that contains $b$ is increasing, as the other case is symmetric.

We first show that if a permutation $\pi$ weakly contains $\tau$, then it also contains $\sigma$. For this consider a match of $\tau^{-}$ that cannot be extended to a match of $\tau^{\prime}$ in the grid representation of $\pi$; see the left hand side of Figure 10. Consider the $s-r$ points $P_{r},\ldots,P_{s-1}$ of $\pi$ to which the entries at positions $r,\ldots,s-1$ of $\tau^{-}$ are matched. We know that they form an increasing sequence, and all other points in this match are below or above them. Now consider the $s-r+1$ cells in $C$ defined in (4) between and around these points. We need to show that none of them contains any points of $\pi$, demonstrating that this is a match of the mesh pattern $\sigma$. Indeed, if one of these regions did contain a point $Q$ from $\pi$, then $Q$ together with $P_{r},\ldots,P_{s-1}$ would form an increasing sequence of length $s-r+1$, i.e., $Q$ would extend the match of $\tau^{-}$ to a match of $\tau^{\prime}$ in $\pi$, a contradiction.

It remains to show that if a permutation $\pi$ contains the mesh pattern $\sigma$, then it weakly contains $\tau$. For this consider a match of $\sigma=(\tau^{-},C)$ in the grid representation of $\pi$; see the right hand side of Figure 10. We label the points to which the entries of $\tau^{-}$ are matched by $P_{1},\ldots,P_{k-1}$. By the definition of $\sigma$, the points $P_{r},\ldots,P_{s-1}$ form a longest increasing sequence of consecutive values in this match. In particular, $P_{r-1}$ is not located at the bottom left corner of the leftmost cell of $C$, and $P_{s}$ is not located at the top right corner of the rightmost cell of $C$, so there is a point $P_{a}$, $a\in[k-1]\setminus\{r-1,\ldots,s-1\}$, on the same height as the lower boundary of the leftmost cell of $C$, and a point $P_{b}$, $b\in[k-1]\setminus\{r,\ldots,s\}$, on the same height as the upper boundary of the rightmost cell of $C$. We know that no point of $\pi$ lies within any of the cells in $C$, but we also need to show that a point $Q$ of $\pi$ contained in any of the other cells cannot be used to extend this match of $\tau^{-}$ to to a match of $\tau^{\prime}$. For this we distinguish four cases: Suppose that $Q$ is contained in a cell $L$ to the left of the cells in $C$. Then $P_{1},\ldots,P_{k-1}$ together with $Q$ is not a match of $\tau^{\prime}$, as the longest increasing substring of consecutive values $P_{r},\ldots,P_{s-1}$ contains only $s-r$ points. A symmetric argument works if $Q$ is contained in a cell $R$ to the right of the cells in $C$. Now suppose that $Q$ is contained in a cell $A$ above a cell from $C$, but not above the rightmost one, or below a cell from $C$, but not below the leftmost one. In this case the points $P_{r},\ldots,P_{s-1}$ together with $Q$ do not form an increasing substring, so this is not a match of $\tau^{\prime}$. It remains to consider the case that $Q$ is contained in a cell $B$ above the rightmost cell from $C$, or below the leftmost cell from $C$. In this case the points $P_{r},\ldots,P_{s-1}$ together with $Q$ form an increasing substring, but the values are not consecutive, as $Q$ is separated from $P_{r},\ldots,P_{s-1}$ by the point $P_{b}$ or $P_{a}$, respectively, so this is not match of $\tau^{\prime}$ either. This completes the proof.

∎

Baril [Bar11] also introduced the notion of a dotted pattern $\tau\in S_{k}$, which is a pattern with some entries at positions $I\subseteq[k]$ that have a dot above them. A permutation $\pi$ avoids the pattern $\tau$ if and only if $\pi$ weakly avoids every single-barred pattern obtained by putting a bar above every entry not in $I$, i.e., we have

[TABLE]

where $\tau_{j}$ is the barred pattern obtained by putting the bar above entry $j$.

Lemma 20**.**

Let $\tau\in S_{k}$, $k\geq 3$, be a dotted pattern with dots over all positions $I\subseteq[k]$. If $\tau$ does not have the largest value $k$ at the leftmost or rightmost position, and all entries in the longest increasing or decreasing substring of consecutive values including $k$ are dotted, then $\tau$ is tame.

Proof.

Let $r$ and $s$ be the start and end indices of the longest increasing or decreasing substring of consecutive values including $k$. The conditions of the lemma imply that $[r,s]\subseteq I$.

By (5), Theorem 8, and Lemma 19, to prove that $\tau$ is tame it is sufficient to show this for the mesh pattern $\sigma=(\rho^{-},C)$ with $C$ defined in (4) for every single-barred pattern $\rho$ obtained from $\tau^{\prime}$ by placing a bar over an entry at a position $[k]\setminus I$. As $[r,s]\subseteq I$, we obtain in particular that the largest value $k$ in $\rho$ is not barred, and if the second largest entry $k-1$ is next to $k$, then it is also not barred. Combining this with the assumption that the largest value $k$ is not at the leftmost or rightmost position in $\rho$, we obtain that in $\rho^{-}$, the largest entry $k-1$ is not at the leftmost or rightmost position. Moreover, we obtain from the definition (4) that $C$ does not have any cells in the topmost row, i.e., $(i,k-1)\notin C$ for $i=0,\ldots,k-1$. Therefore, applying Theorem 15 shows that $\sigma$ is tame, completing the proof. ∎

5.5. Monotone and geometric grid classes

Monotone grid classes of permutations were introduced by Huczynska and Vatter [HV06]. To define them, we consider a matrix $M$ with entries from $\{0,+1,-1\}$, indexed first by columns from left to right, and then by rows from bottom to top. A permutation $\pi$ of $[n]$ (for any $n\geq 0$) is in the monotone grid class of $M$, denoted $\operatorname{Grid}(M)$, if we can place the points labelled from 1 to $n$ from bottom to top into a rectangular grid that has as many rows and columns as the matrix $M$, and reading the labels from left to right will yield $\pi$, subject to the following two conditions: No two points are placed on the same horizontal or vertical line. Moreover, for each cell $(x,y)$ in the grid, if $M_{x,y}=0$, then the cell contains no points, if $M_{x,y}=+1$, then the points in this cell are increasing, and if $M_{x,y}=-1$, then the points in this cell are decreasing. This definition is illustrated in the top part of Figure 11. Based on this, we define $\operatorname{Grid}_{n}(M):=\operatorname{Grid}(M)\cap S_{n}$. It is an open problem whether $\operatorname{Grid}(M)$ is characterized by finitely many forbidden patterns for any $M$ (cf. [AB16]).

The second type of permutations we shall discuss in this section are geometric grid classes, introduced by Albert, Atkinson, Bouvel, Ruškuc, and Vatter [AAB*+*13]. They are defined using a matrix $M$ with entries from $\{0,+1,-1\}$ as before. A permutation $\pi$ of $[n]$ is in the geometric grid class of $M$, denoted $\operatorname{Geo}(M)$, if it can be drawn in a rectangular grid as described before, with the slightly strengthened conditions that if $M_{x,y}=+1$, then the points in the cell $(x,y)$ lie on the increasing diagonal line through this cell, and if $M_{x,y}=-1$, then the points in the cell $(x,y)$ lie on the decreasing diagonal line through this cell. This definition is illustrated in the bottom part of Figure 11. Similarly to before, we define $\operatorname{Geo}_{n}(M):=\operatorname{Geo}(M)\cap S_{n}$. We clearly have $\operatorname{Geo}(M)\subseteq\operatorname{Grid}(M)$ and $\operatorname{Geo}_{n}(M)\subseteq\operatorname{Grid}_{n}(M)$.

Unlike for monotone grid classes, it was shown in [AAB*+*13] that any geometric grid class $\operatorname{Geo}(M)$ is characterized by finitely many forbidden patterns, i.e.,

[TABLE]

for a suitable set of patterns $\tau_{1},\ldots,\tau_{\ell}$ and for all $n\geq 0$. For instance, X-shaped permutations studied in [Wat07, Eli11] are exactly the permutations in $S_{n}(2143\wedge 2413\wedge 3142\wedge 3412)$. However, the argument given in [AAB*+*13] for the existence of $\tau_{1},\ldots,\tau_{\ell}$ is non-constructive, so there is no procedure known to compute these patterns from the matrix $M$.

Nevertheless, our next theorem provides an easily verifiable sufficient condition for deciding whether $\operatorname{Grid}_{n}(M)$ and $\operatorname{Geo}_{n}(M)$ are zigzag languages, based only on two particular entries of $M$.

Theorem 21**.**

Let $M$ be a matrix with $-1$ in the top-left corner and $+1$ in the top-right corner. Then $\operatorname{Grid}_{n}(M)$, $n\geq 0$, and $\operatorname{Geo}_{n}(M)$, $n\geq 0$, are both hereditary sequences of zigzag languages. Consequently, all of these languages can be generated by Algorithm J.

From the two monotone and geometric grid classes shown in Figure 11, only the left two satisfy the conditions of the theorem.

Proof.

We only prove the theorem for monotone grid classes $\operatorname{Grid}_{n}(M)$. The argument for geometric grid classes $\operatorname{Geo}_{n}(M)$ is completely analogous.

We argue by induction on $n$. Note that $\operatorname{Grid}_{0}(M)=S_{0}=\{\varepsilon\}$ is a zigzag language by definition, so the induction basis is clear. For the induction step let $n\geq 1$. We first show that if $\pi\in\operatorname{Grid}_{n-1}(M)$, then $c_{1}(\pi),c_{n}(\pi)\in\operatorname{Grid}_{n}(M)$. For this argument we use the assumption that the top-left entry of $M$ is $-1$, and the top-right entry of $M$ is $+1$, i.e., $\pi$ can be drawn into a grid so that the points in the top-left cell are decreasing, and the points in the top-right cell are increasing. It follows that we can draw $c_{1}(\pi)$ on the same grid, by extending the drawing of $\pi$ so that the new point $n$ is placed to the top-left of all other points. Similarly, we can draw $c_{n}(\pi)$ on the same grid, by extending the drawing of $\pi$ so that the new point $n$ is placed to the top-right of all other points.

To complete the induction step, we now show that if $\pi\in\operatorname{Grid}_{n}(M)$, then $p(\pi)\in\operatorname{Grid}_{n-1}(M)$. As $\pi\in\operatorname{Grid}_{n}(M)$, we can draw $\pi$ into a grid respecting the monotonicity conditions described by $M$. Clearly, removing any entry from $\pi$, in particular the largest one, maintains this property, i.e., we can draw $p(\pi)$ on the same grid, showing that $p(\pi)\in\operatorname{Grid}_{n-1}(M)$. This completes the proof. ∎

6. Limitations of our approach

Theorem 1 asserts that if $L_{n}$ is a zigzag language, then Algorithm J successfully visits every permutation from $L_{n}$. This condition is not necessary, however. For instance, the set $L_{4}\subseteq S_{4}$ discussed in Section 2.2 is not a zigzag language, and still Algorithm J is successful when initialized suitably. From the proof of Theorem 1 we immediately see that condition (z) in the definition of zigzag language could be weakened as follows, and the same proof would still go through:

1. (z’)

Given the sequence $J(L_{n-1})$, then for every permutation $\pi$ in this sequence there is a direction $\chi(\pi)\in\{\leftarrow,\rightarrow\}$ satisfying the following: For every $\pi$ from $J(L_{n-1})$ we define $I(\pi):=\{i\in[n]\mid c_{i}(\pi)\in L_{n}\}$, and we also define ${\check{\imath}}(\pi):=\max I(\pi)$ and ${\hat{\imath}}(\pi):=\min I(\pi)$ if $\chi(\pi)={}\leftarrow$, and ${\check{\imath}}(\pi):=\min I(\pi)$ and ${\hat{\imath}}(\pi):=\max I(\pi)$ if $\chi(\pi)={}\rightarrow$. Then for any two permutations $\pi,\rho$ that appear consecutively in $J(L_{n-1})$ and that differ in a jump from position $i_{1}$ to position $i_{2}$, we have ${\hat{\imath}}(\pi)={\check{\imath}}(\rho)$ and this number is not in the interval $]\min\{i_{1},i_{2}\},\max\{i_{1},i_{2}\}]$.

Note that condition (z) implies (z’), as for a zigzag language we have $1,n\in I(\pi)$ for all $\pi\in L_{n-1}$, so $\min I(\pi)=1$ and $\max I(\pi)=n$, and the sequence $\chi$ alternates between $\leftarrow$ and $\rightarrow$ in every step. This shows that Algorithm J succeeds to generate a much larger class of languages $L_{n}\subseteq S_{n}$. However, condition (z’) is considerably more complicated, in particular as it depends on the ordering $J(L_{n-1})$ generated for $L_{n-1}$. More importantly, in the context of permutation patterns we do not see any interesting languages $L_{n}$ that would satisfy condition (z’) but not condition (z), which is why we think condition (z) is the ‘correct’ one.

Let us also briefly motivate the definition of tame permutation patterns given in Section 4.2. On the one hand, for a fixed pattern $\tau$, it is certainly reasonable to require that all of the sets $S_{n}(\tau)$, $n\geq 0$, are zigzag languages. This is because for any classical pattern $\tau\in S_{k}$, the set $S_{n}(\tau)=S_{n}$ for $n<k$ is trivially a zigzag language, but a very uninteresting one. On the other hand, now that we are concerned with an infinite sequence of sets $S_{n}(\tau)$, $n\geq 0$, the requirement for the sequence to be hereditary is equally reasonable, as we shall see by considering the problems that arise if we drop this requirement. For this consider the non-tame barred pattern $\tau=132\overline{4}$. We have $132\notin S_{3}(\tau)$, i.e., the permutation $132$ contains the pattern, whereas $c_{4}(132)=1324\in S_{4}(\tau)$ avoids it. Consequently, if we were interested in the language $L_{4}=S_{4}(\tau)$, then the corresponding set $L_{3}:=p(L_{4})$ would be $L_{3}=S_{3}$, and not $S_{3}(\tau)=S_{3}\setminus\{132\}$. In general, when considering $S_{n}(\tau)$ for a particular value of $n$, then the sets $L_{i-1}:=p(L_{i})$ for $i=n,n-1,\ldots,1$ are not necessarily characterized by avoiding the pattern $\tau$. In fact, it is not clear whether they are characterized by any kind of pattern-avoidance. In terms of the tree representation of zigzag languages, if the sequence $S_{n}(\tau)$, $n\geq 0$, is not hereditary, then different values of $n$ correspond to different tree prunings. In the example from before, the node $132$ is pruned from the tree for $S_{3}(\tau)$, but it is not pruned from the tree for $S_{4}(\tau)$. In contrast to that, in a hereditary sequence, all sets $S_{n}(\tau)$, $n\geq 0$, arise from pruning the infinite rooted tree of permutations in a way that is consistent for all $n$. Summarizing, for patterns $\tau$ for which $S_{n}(\tau)$ is not hereditary, our proof of Theorem 1 breaks seriously. Not only that, Algorithm J in general fails to generate $S_{n}(\tau)$. For instance, it fails to generate $S_{4}(\tau)$, $\tau=132\overline{4}$, when initialized with $\operatorname{id}_{4}=1234$, while it succeeds to generate $S_{4}(\tau^{\prime})$ for the non-tame barred pattern $\tau^{\prime}=\overline{4}132$ .

For a classical pattern $\tau\in S_{k}$ that has the largest value $k$ at the leftmost or rightmost position, we have that $S_{k}(\tau)=S_{k}\setminus\{\tau\}$ is not a zigzag language (as $\tau$ equals either $c_{1}(p(\tau))$ or $c_{k}(p(\tau))$), i.e., $\tau$ is not tame. Moreover, in general Algorithm J fails to generate $S_{k}(\tau)$. For instance, running Algorithm J on $S_{3}(321)$ gives only three permutations $123,132,312$, and then the algorithm stops. This is admittedly a very strong limitation of our approach, as many interesting permutation patterns have the largest value at the boundary, such as $321$, which gives rise to an important Catalan family $S_{n}(321)$.

By what we said before, the condition for tameness stated in Lemma 9 is not only sufficient, but also necessary. In a similar way, it can be shown that the conditions stated in Lemmas 10–14 are necessary for tameness. The situation is slightly more complicated for Theorem 15: Conditions (i) and (ii) of the theorem are indeed necessary. Specifically, if condition (i) is violated, then $S_{k}(\sigma)=S_{k}\setminus\{\tau\}$ is not a zigzag language, and if condition (ii) is violated, then $S_{k}(\sigma)\neq p(S_{k+1}(\sigma))$, i.e., the hereditary property is violated. However, conditions (iii) and (iv) are not necessary. Consider for instance the patterns $\sigma_{1},\sigma_{2},\sigma_{3}$ shown below:

$\sigma_{1}$

$\sigma_{2}$

$\sigma_{3}$

They all satisfy conditions (i), (ii) and (iv), but violate condition (iii) due to the cells $(2,3)$ and $(1,3)$, connected by an arrow in the figures. However, the proof of Theorem 15 given in Section 4.8 can be modified to show that $\sigma_{1}$ is tame. The idea is to apply the exchange argument that involves a point in a match of the pattern and that is illustrated on the right hand side of Figure 8 twice instead of only once. This idea can be iterated, and by applying the exchange argument three or four times, respectively, one can show that $\sigma_{2}$ and $\sigma_{3}$ are tame as well. This kind of reasoning apparently leads to combinatorial chaos, depending on the relative location of points and shaded cells in the pattern, and this prevents us from being able to formulate conditions for a mesh pattern that are necessary and sufficient for tameness. This is not an issue from our point of view, because again, we do not see any interesting families of pattern-avoiding permutations that would satisfy such more complicated conditions but not the conditions stated in Theorem 15.

7. Acknowledgments

We thank Michael Albert, Mathilde Bouvel, Sergi Elizalde, Vít Jelínek, Sergey Kitaev, Vincent Vajnovszki, and Vincent Vatter for very insightful feedback on this work, for pointing out relevant references, and for sharing their knowledge about pattern-avoiding permutations. We also thank Jean Cardinal, Vincent Pilaud, and Nathan Reading for several stimulating discussions about lattice congruences of the weak order on the symmetric group. Lastly, we thank the anonymous reviewer who provided very thoughtful feedback and many suggestions that helped improving the manuscript.