Counting 3-Stack-Sortable Permutations

TL;DR

This paper introduces a decomposition lemma to count permutations under the stack-sorting map, providing new proofs, recurrence relations, and bounds for 2- and 3-stack-sortable permutations, and explores related conjectures and properties.

Contribution

It presents a novel decomposition lemma for counting preimages under stack-sorting, extends known sequences, and disproves some existing conjectures while supporting others.

Findings

Derived algebraic equations for generating functions of 2-stack-sortable permutations.

Computed 3-stack-sortable permutations up to length 174, extending previous data.

Provided lower bounds for the growth rate of 3-stack-sortable permutations.

Abstract

We prove a "decomposition lemma" that allows us to count preimages of certain sets of permutations under West's stack-sorting map $s$. As a first application, we give a new proof of Zeilberger's formula for the number of 2-stack-sortable permutations in $S_n$. Our proof generalizes, allowing us to find an algebraic equation satisfied by the generating function that counts 2-stack-sortable permutations according to length, number of descents, and number of peaks. The same method yields a recurrence relation for $W_3(n)$, the number of 3-stack-sortable permutations in $S_n$. We compute $W_3(n)$ for $n\le 174$, extending the 13 terms of this sequence that were known before. We also prove the first nontrivial lower bound for $\lim\limits_{n\to\infty}W_3(n)^{1/n}$. Invoking a result of Kremer, we also prove that $\lim\limits_{n\to\infty}W_t(n)^{1/n}\geq(\sqrt{t}+1)^2$ for all $t\geq 1$, which we use to improve a result of Smith. Our computations allow us to disprove a conjecture of B\'ona, although we do not yet know for sure which one. We can refine our methods to obtain a recurrence for the number of 3-stack-sortable permutations in $S_n$ with $k$ descents and $p$ peaks. This produces a large amount of evidence supporting a real-rootedness conjecture of B\'ona. Using part of the theory of valid hook configurations, we give a new proof of a $\gamma$-nonnegativity result of Br\"and\'en, which in turn implies an older result of B\'ona. We then answer a question of the current author by producing a set $A\subseteq S_{11}$ such that $\sum_{\sigma\in s^{-1}(A)}x^{\text{des}(\sigma)}$ has nonreal roots. We interpret this as partial evidence against the same real-rootedness conjecture of B\'ona that we found evidence supporting. Examining the parities of the numbers $W_3(n)$, we obtain strong evidence against yet another conjecture of B\'ona. We end with some conjectures of our own.

Equations

$$W_t(n)=|{\mathcal{W}}_t(n)|,W_t(n,k)=|{\mathcal{W}}_t(n,k)|,\text{and}{W}_t(n,k,p)=|{\mathcal{W}}_t(n,k,p)|.$$

$$W_1(n)=|{\mathrm{Av}}_n(231)|=C_n,$$

$$W_2(n)=\frac{2}{(n+1)(2n+1)}\left(\genfrac{}{}{0ex}{}{3n}{n}\right).$$

$$W_t(n)\le (t+1{)}^{2n}.$$

$$\underset{n\to \infty }{\lim }{W}_3(n{)}^{1/n}<12.53296\text{and}\underset{n\to \infty }{\lim }{W}_4(n{)}^{1/n}<\mathrm{21.97225.}$$

$$\underset{n\to \infty }{\lim }{W}_t(n{)}^{1/n}\ge 6.75\text{for all}t\ge 2.$$

$$\sum _{\sigma \in s^{-1}(A)}{x}^{\mathrm{des}(\sigma )}=\sum _{m=0}^{\lfloor \frac{n-1}{2}\rfloor }\frac{|\{\sigma \in s^{-1}(A):\mathrm{peak}(\sigma )=m\}|}{2^{n-1-2m}}{x}^m(1+x{)}^{n-1-2m}.$$

$$W_t(n)\le \left(\genfrac{}{}{0ex}{}{(t+1)n}{n}\right).$$

$$23\cdots k1(k+1)\cdots n,12\cdots (k-2)k(k-1)(k+1)\cdots n,\text{and}k12\cdots (k-1)(k+1)\cdots n.$$

$$\sum _{\sigma \in s^{-1}(\pi )}{x}^{\mathrm{des}(\sigma )+1}{y}^{\mathrm{peak}(\sigma )+1}$$

$$=\sum _{H\in {\mathrm{SW}}_d(\pi )}\left(\sum _{\mu \in s^{-1}({\pi }_U^H)}{x}^{\mathrm{des}(\mu )+1}{y}^{\mathrm{peak}(\mu )+1}\right)\left(\sum _{\lambda \in s^{-1}({\pi }_S^H)}{x}^{\mathrm{des}(\lambda )+1}{y}^{\mathrm{peak}(\lambda )+1}\right).$$

$$\sum _{\sigma \in E_j}{x}^{\mathrm{des}(\sigma )+1}{y}^{\mathrm{peak}(\sigma )+1}=\left(\sum _{\mu \in s^{-1}({\pi }_U^H)}{x}^{\mathrm{des}(\mu )+1}{y}^{\mathrm{peak}(\mu )+1}\right)\left(\sum _{\lambda \in s^{-1}({\pi }_S^H)}{x}^{\mathrm{des}(\lambda )+1}{y}^{\mathrm{peak}(\lambda )+1}\right),$$

$$|s^{-1}(\pi )|=\sum _{H\in {\mathrm{SW}}_d(\pi )}|s^{-1}({\pi }_U^H)|\cdot |s^{-1}({\pi }_S^H)|.$$

$$N(r,i)=\frac{1}{r}\left(\genfrac{}{}{0ex}{}{r}{i}\right)\left(\genfrac{}{}{0ex}{}{r}{i-1}\right)\text{and}V(r,j)=2^{r-2j+1}\left(\genfrac{}{}{0ex}{}{r-1}{2j-2}\right)C_{j-1}.$$

$$N_r(x)=\sum _{i=1}^{r}N(r,i)x^i\text{and}{V}_r(y)=\sum _{j=1}^{r}V(r,j)y^j.$$

$$F(w,x,y)=x+wxy+w(F(w,x,y)+1)(F(w,x,y)-x).$$

$$F(w,x,y)=\frac{1-w+wx-\sqrt{(1-w+wx{)}^2-4wx(1-w+wy)}}{2w},$$

$$L(r,i,j)=\frac{1}{r+1-j}\left(\genfrac{}{}{0ex}{}{r-1}{r-j}\right)\left(\genfrac{}{}{0ex}{}{r+1-j}{j}\right)\left(\genfrac{}{}{0ex}{}{r+1-2j}{i-j}\right).$$

$$L_r(x,y)=\sum _{i=1}^r\sum _{j=1}^{r}L(r,i,j)x^i{y}^j$$

$$L_r(x,1)=N_r(x)\text{and}{L}_r(1,y)=V_r(y).$$

$$\sum _{\sigma \in s^{-1}(\pi )}{x}^{\mathrm{des}(\sigma )+1}{y}^{\mathrm{peak}(\sigma )+1}=\sum _{(q_0,\dots ,q_k)\in \mathcal{V}(\pi )}\prod _{t=0}^k{L}_{q_t}(x,y).$$

$$\sum _{\sigma \in s^{-1}(\pi )}{x}^{\mathrm{des}(\sigma )+1}=\sum _{(q_0,\dots ,q_k)\in \mathcal{V}(\pi )}\prod _{t=0}^k{N}_{q_t}(x)$$

$$\sum _{\sigma \in s^{-1}(\pi )}{y}^{\mathrm{peak}(\sigma )+1}=\sum _{(q_0,\dots ,q_k)\in \mathcal{V}(\pi )}\prod _{t=0}^k{V}_{q_t}(y).$$

$$|s^{-1}(\pi )|=\sum _{(q_0,\dots ,q_k)\in \mathcal{V}(\pi )}\prod _{t=0}^k{C}_{q_t}.$$

$$\sum _{\sigma \in s^{-1}(123\cdots n)}{x}^{\mathrm{des}(\sigma )+1}{y}^{\mathrm{peak}(\sigma )+1}=\sum _{\sigma \in {\mathrm{Av}}_n(231)}{x}^{\mathrm{des}(\sigma )+1}{y}^{\mathrm{peak}(\sigma )+1}=L_n(x,y).$$

$${\mathcal{D}}_{\ell }(n)=\{\pi \in {\mathrm{Av}}_{n+\ell }(231):\mathrm{tl}(\pi )=\ell \}\text{and}{\mathcal{D}}_{\ge \ell }(n)=\{\pi \in {\mathrm{Av}}_{n+\ell }(231):\mathrm{tl}(\pi )\ge \ell \}.$$

$$B_{\ell }(n)=|s^{-1}({\mathcal{D}}_{\ell }(n))|\text{and}{B}_{\ge \ell }(n)=|s^{-1}({\mathcal{D}}_{\ge \ell }(n))|.$$

$$B_{\ell }(n+1)=\sum _{i=1}^n\sum _{j=1}^{\ell }{B}_{\ge \ell -j+1}(n-i)B_{\ge j-1}(i).$$

$$G_{\ell }(w)=\sum _{n\ge 0}{B}_{\ge \ell }(n)w^n\text{and}I(w,z)=\sum _{\ell \ge 0}{G}_{\ell }(w)z^{\ell }.$$

$$G_{\ell }(0)=B_{\ge \ell }(0)=|s^{-1}({\mathcal{D}}_{\ge \ell }(0))|=|s^{-1}(123\cdots \ell )|=C_{\ell }$$

Full text

In fact, we can refine our methods to obtain a recurrence for $W_{3}(n,k,p)$, the number of $3$-stack-sortable permutations in $S_{n}$ with $k$ descents and $p$ peaks. This allows us to gain a large amount of evidence supporting a real-rootedness conjecture of Bóna. Using part of the theory of valid hook configurations, we give a new proof of a $\gamma$-nonnegativity result of Brändén, which in turn implies an older result of Bóna. We then answer a question of the current author by producing a set $A\subseteq S_{11}$ such that $\sum_{\sigma\in s^{-1}(A)}x^{\operatorname{des}(\sigma)}$ has nonreal roots. We interpret this as partial evidence against the same real-rootedness conjecture of Bóna that we found evidence supporting. Examining the parities of the numbers $W_{3}(n)$, we obtain strong evidence against yet another conjecture of Bóna. We end with some conjectures of our own.

1. Introduction

1.1. The Stack-Sorting Map

We use the word “permutation” to refer to an ordering of a set of positive integers written in one-line notation. Let $S_{n}$ denote the set of permutations of the set $[n]:=\{1,\ldots,n\}$. If $\pi$ is a permutation of length $n$, then the normalization of $\pi$ is the permutation in $S_{n}$ obtained by replacing the $i^{\text{th}}$-smallest entry in $\pi$ with $i$ for all $i\in[n]$. We say a permutation is normalized if it is equal to its normalization. A descent of a permutation $\pi=\pi_{1}\cdots\pi_{n}$ is an index $i\in[n-1]$ such that $\pi_{i}>\pi_{i+1}$. A peak of $\pi$ is an index $i\in\{2,\ldots,n-1\}$ such that $\pi_{i-1}<\pi_{i}>\pi_{i+1}$. Let $\operatorname{des}(\pi)$ and $\operatorname{peak}(\pi)$ denote the number of descents of $\pi$ and the number of peaks of $\pi$, respectively.

Definition 1.1**.**

Given $\tau\in S_{m}$, we say a permutation $\sigma=\sigma_{1}\cdots\sigma_{n}$ contains the pattern $\tau$ if there exist indices $i_{1}<\cdots<i_{m}$ in $[n]$ such that the normalization of $\sigma_{i_{1}}\cdots\sigma_{i_{m}}$ is $\tau$. We say $\sigma$ avoids $\tau$ if it does not contain $\tau$. Let $\operatorname{Av}(\tau^{(1)},\ldots,\tau^{(r)})$ denote the set of normalized permutations that avoid the patterns $\tau^{(1)},\ldots,\tau^{(r)}$. Let $\operatorname{Av}_{n}(\tau^{(1)},\ldots,\tau^{(r)})=\operatorname{Av}(\tau^{(1)},\ldots,\tau^{(r)})\cap S_{n}$.

The study of permutation patterns is now a major area of research; it began with Knuth’s analysis of a certain “stack-sorting algorithm” [37]. In his dissertation, West [45] defined a deterministic variant of Knuth’s algorithm. This variant is a function, which we call the “stack-sorting map” and denote by $s$, that sends permutations to permutations. The stack-sorting map has now been studied extensively [1, 3, 8, 7, 6, 9, 11, 12, 13, 14, 15, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 32, 33, 34, 36, 40, 42, 43, 44, 45, 47]. The reader seeking further historical background and motivation should see one of the references [3, 8, 20, 23, 26, 27, 28, 29].

To define the function $s$, let us begin with an input permutation $\pi=\pi_{1}\cdots\pi_{n}$. At any point in time during this procedure, if the next entry in the input permutation is smaller than the entry at the top of the stack or if the stack is empty, the next entry in the input permutation is placed at the top of the stack. Otherwise, the entry at the top of the stack is annexed to the end of the growing output permutation. This process terminates when the output permutation has length $n$, and $s(\pi)$ is defined to be this output permutation. The following illustration shows that $s(4162)=1426$.

Definition 1.2**.**

​​​111The permutations that we call $t$-stack-sortable are often called “West $t$-stack-sortable,” but we have dropped the word “West” for brevity. The term “$t$-stack-sortable” is sometimes used to refer to permutations that can be sorted using Knuth’s (nondeterministic) algorithm with $t$ stacks in series; that use of the term is different from ours. We say a permutation $\pi$ is $t$-stack-sortable if $s^{t}(\pi)$ is an increasing permutation, where $s^{t}$ denotes the $t$-fold iterate of $s$. Let $\mathcal{W}_{t}(n)$ be the set of $t$-stack-sortable permutations in $S_{n}$, and let $\mathcal{W}_{t}(n,k)=\{\pi\in\mathcal{W}_{t}(n):\operatorname{des}(\pi)=k\}$ and $\mathcal{W}_{t}(n,k,p)=\{\pi\in\mathcal{W}_{t}(n,k):\operatorname{peak}(\pi)=p\}$. Let

[TABLE]

Knuth simultaneously initiated the study of stack-sorting and the investigation of permutation patterns with the following theorem.

Theorem 1.1** (​​[37]).**

A permutation is $1$-stack-sortable if and only if it avoids the pattern $231$. Furthermore,

[TABLE]

where $C_{n}=\frac{1}{n+1}{2n\choose n}$ is the $n^{\text{th}}$ Catalan number.

In his dissertation, West conjectured a formula for $W_{2}(n)$, which Zeilberger later proved.

Theorem 1.2** (​​[47]).**

We have

[TABLE]

Combinatorial proofs of Zeilberger’s theorem emerged later in [19, 32, 33, 36]. Some authors have investigated the enumeration of $2$-stack-sortable permutations according to various statistics [6, 11, 13, 32]. The articles [31] and [35] give different proofs that new combinatorial objects called “fighting fish” are counted by the numbers $W_{2}(n)$. The authors of [2] studied what they called “$n$-point dominoes,” and they have found that there are $W_{2}(n+1)$ such objects.

There is very little known about $t$-stack-sortable permutations when $t\geq 3$ is fixed. Úlfarsson [44] characterized $3$-stack-sortable permutations in terms of new “decorated patterns,” but the characterization is too unwieldy to yield any additional information. The recent paper [1] shows that for every $t\geq 1$, the set of $t$-stack-sortable permutations can be described by a sentence in a first-order logical theory that the authors call $\mathsf{TOTO}$. The paper [17] also investigates $t$-stack-sortable permutations when $t=n-r$ for some fixed $r$ (focusing on the case in which $r=4$). For fixed $t\geq 3$, the best known general upper bound for $W_{t}(n)$ (see [8, Theorem 3.4]), is the estimate

[TABLE]

The current author showed [27] that

[TABLE]

The limits in (2) are known to exist (see Section 6). Recently, Bóna has obtained a new proof of the first inequality in (2) using “stack words.” It also follows from Theorem 1.2 that

[TABLE]

When $t\geq 3$, we refer to (3) as a “trivial” lower bound for the growth rate of $W_{t}(n)$, even though it relies on the highly nontrivial enumeration of $2$-stack-sortable permutations. Remarkably, (3) was the best known lower bound for $\lim\limits_{n\to\infty}W_{t}(n)^{1/n}$ for all $t\geq 2$ until now.

Bóna [9] proved that the polynomial $\displaystyle\sum_{k=0}^{n-1}W_{t}(n,k)x^{k}=\sum_{\sigma\in\mathcal{W}_{t}(n)}x^{\operatorname{des}(\sigma)}$ is symmetric and unimodal (see Section 8 for the relevant definitions). In fact, his proof actually shows that $\displaystyle\sum_{\sigma\in s^{-1}(A)}x^{\operatorname{des}(\sigma)}$ is symmetric and unimodal for every set $A\subseteq S_{n}$. Brändén strengthened this result with the following theorem (we define $\gamma$-nonnegativity in Section 8).

Theorem 1.3** (​​[14]).**

If $A\subseteq S_{n}$, then

[TABLE]

In particular, $\displaystyle\sum_{\sigma\in s^{-1}(A)}x^{\operatorname{des}(\sigma)}$ is $\gamma$-nonnegative.

In the present article, we concern ourselves with the following four conjectures of Bóna. Recall that a sequence $(a_{n})_{n\geq 1}$ of positive numbers is called log-convex if $(a_{n+1}/a_{n})_{n\geq 1}$ is nondecreasing.

Conjecture 1.1** (​​[3, 8]).**

For all $n,t\geq 1$, we have

[TABLE]

Conjecture 1.2** (​​[5]).**

For every $t\geq 1$, the sequence $(W_{t}(n))_{n\geq 1}$ is log-convex.

Conjecture 1.3** (​​[34]).**

If $t$ is even, then $W_{t}(n)$ is frequently odd. If $t$ is odd, then $W_{t}(n)$ is rarely odd.

Conjecture 1.4** (​​[9]).**

For all $n,t\geq 1$, the polynomial $\displaystyle\sum_{\sigma\in\mathcal{W}_{t}(n)}x^{\operatorname{des}(\sigma)}$ has only real roots.

Remark 1.1**.**

Bóna’s motivation for formulating Conjecture 1.1 came from the idea of encoding elements of $\mathcal{W}_{t}(n)$ as $n$-uniform words over a $(t+1)$-element alphabet (see [7] and [8] for more details). His motivation behind Conjecture 1.2 came from an observation that the sequences $(W_{t}(n))_{n\geq 1}$ appear to be similar to the sequences that enumerate principal permutation classes, which he has also conjectured are log-convex. For example, Bóna has observed that his methods in [4] can be used to show that for fixed $n,t\geq 1$, the number of $t$-stack-sortable permutations of length $n$ with $c$ components is monotonically decreasing as a function of $c$. Similarly, his methods allow one to prove that the generating functions $\sum_{n\geq 1}W_{t}(n)x^{n}$ are not rational. Bóna formulated Conjecture 1.4 after observing that it holds when $t=1$ and when $t=n-1$ (it also holds when $t\geq n$ because this is equivalent to the $t=n-1$ case). Brändén [15] proved this conjecture in the cases $t=2$ and $t=n-2$, but the remaining cases are still open.

Conjecture 1.3 requires some explanation. Using Bóna’s result that $\sum_{\sigma\in\mathcal{W}_{t}(n)}x^{\operatorname{des}(\sigma)}$ is symmetric, one can easily deduce that $W_{t}(n)$ is even whenever $n$ is even. Therefore, it is natural to consider the parity of $W_{t}(n)$ when $n$ is odd. Let $\mathfrak{g}_{t}(m)$ be the number of integers $n$ with $1\leq n\leq m$ such that $W_{t}(n)$ is odd. Let $F_{r}$ denote the $r^{\text{th}}$ Fibonacci number (with $F_{1}=F_{2}=1$). Using Theorems 1.1 and 1.2, one can show that $\mathfrak{g}_{1}(2^{r})=r$ and $\mathfrak{g}_{2}(2^{r})=F_{r}$ for all positive integers $r$. Bóna [34] interpreted this as saying $W_{1}(n)$ is rarely odd while $W_{2}(n)$ is frequently odd, and this led him to formulate Conjecture 1.3. One could formalize this by saying that $W_{t}(n)$ is rarely odd if $\limsup\limits_{m\to\infty}\dfrac{\log\mathfrak{g}_{t}(m)}{\log m}=0$ and is frequently odd $\liminf\limits_{m\to\infty}\dfrac{\log\mathfrak{g}_{t}(m)}{\log m}>0$ (although Bóna did not use this formalism). Bóna’s motivation behind Conjecture 1.3 also came from the idea of encoding $t$-stack-sortable permutations as words.

1.2. Summary of Main Results

In Section 2, we formulate a “decomposition lemma,” which provides a new method for analyzing preimages of permutations under the stack-sorting map. We actually prove a stronger lemma, which we call the refined decomposition lemma, that allows us to take the statistics $\operatorname{des}$ and $\operatorname{peak}$ into account. In Section 3, we briefly review some formulas arising from the theory of new combinatorial objects called “valid hook configurations.” In Section 4, we use the decomposition lemma to give a new proof of Zeilberger’s formula for $W_{2}(n)$. We also use the refined decomposition lemma to find an algebraic equation satisfied by the generating function of the numbers $W_{2}(n,k,p)$. This equation is new.

Our new proof of Zeilberger’s formula is the first one that generalizes to the setting of $3$-stack-sortable permutations. In Section 5, we use the refined decomposition lemma to prove a recurrence relation for the numbers $W_{3}(n,k,p)$. Specializing this theorem gives us a recurrence for $W_{3}(n,k)$, and specializing further gives a recurrence for $W_{3}(n)$. This yields the first polynomial-time algorithm for computing $W_{3}(n)$. According to Wilf [46], we have solved the problem of counting $3$-stack-sortable permutations. More precisely, he would say that we have “$p$-solved” this problem.

Before now, the values of $W_{3}(n)$ were only known up to $n=13$. Indeed, the only algorithm that was used to compute these numbers before now relied on a brute-force approach. Using our recurrence, we have generated the values of $W_{3}(n)$ for $n\leq 174$. We have added these terms to sequence A134664 in the Online Encyclopedia of Integer Sequences [41]. There are two significant theoretical implications of these computations. First, we will see in Section 6 that Bóna’s Conjectures 1.1 and 1.2 cannot both be true. Thus, we have disproven a conjecture of Bóna, although we do not yet know with absolute certainty which one. Let us remark, however, that the data suggests very strongly that Conjecture 1.2 is true while Conjecture 1.1 is false. Furthermore, it appears that our recurrence coupled with sufficient computing time (and clever computing!) should allow one to completely disprove Conjecture 1.1. Second, we will prove that $\lim\limits_{n\to\infty}W_{3}(n)^{1/n}\geq 8.659702$; this is the first nontrivial lower bound for $\lim\limits_{n\to\infty}W_{3}(n)^{1/n}$. In Section 7, we prove that $\lim\limits_{n\to\infty}W_{t}(n)^{1/n}\geq(\sqrt{t}+1)^{2}$ for every $t\geq 1$, yielding the first nontrivial lower bounds for these growth rates for all $t\geq 4$. As a corollary, we improve a result of Smith concerning permutations that can be sorted by $t$ stacks in series using the so-called “left-greedy algorithm” [43]. Although there are multiple ways one could rigorously interpret Bóna’s Conjecture 1.3, we will see in Section 6 that every reasonable interpretation of the conjecture is likely to be false.

We have also computed the numbers $W_{3}(n,k)$ for $n\leq 43$, allowing us to verify Conjecture 1.4 when $t=3$ and $n\leq 43$ (see OEIS sequence A324916 [41]). In Section 8, we show that the formulas from Section 3 easily imply Brändén’s Theorem 1.3. We also provide a two-element set $A\subseteq S_{11}$ such that $\sum_{\sigma\in s^{-1}(A)}x^{\operatorname{des}(\sigma)}$ is not real-rooted. This provides a negative answer to the last part of Question 12.1 in [28], which we interpret as a small amount of evidence against Bóna’s Conjecture 1.4. Section 9 concludes the paper with a new conjecture about $\lim\limits_{n\to\infty}W_{3}(n)^{1/n}$ and several conjectures about the numbers $\mathfrak{g}_{3}(m)$ (defined in Remark 1.1).

Before we proceed, let us make one additional remark about the usefulness of the decomposition lemma that we prove in Section 2. In a subsequent paper [22], we apply this lemma in order to settle several conjectures of the current author from [28]. More precisely, we complete the project of determining $|s^{-1}(\operatorname{Av}_{n}(\tau^{(1)},\ldots,\tau^{(r)}))|$ for every subset $\{\tau^{(1)},\ldots,\tau^{(r)}\}\subseteq S_{3}$ with the exception of the singleton set $\{321\}$. This allows us to enumerate a new permutation class, find a new example of an unbalanced Wilf equivalence, and prove a conjecture of Hossain concerning the so-called “Boolean-Catalan numbers.” Hence, one can even view the decomposition lemma as a bridge that allows one to use the stack-sorting map $s$ as a tool for proving results that were conjectured without any reference to stack-sorting.

2. The Decomposition Lemma

West [45] defined the fertility of a permutation $\pi$ to be $|s^{-1}(\pi)|$, the number of preimages of $\pi$ under $s$. He then performed extensive calculations in order to compute the fertilities of the permutations of the forms

[TABLE]

Bousquet-Mélou [12] found a method for determining whether or not a given permutation is sorted, meaning that its fertility is positive. She then asked for a general method for computing the fertility of any given permutation. The current author achieved this in even greater generality in [26, 27, 28] using new combinatorial objects called “valid hook configurations.” In this section, we prove the refined decomposition lemma and the decomposition lemma, which provide a new method for analyzing fertilities of permutations.

The plot of a permutation $\pi=\pi_{1}\cdots\pi_{n}$ is the figure showing the points $(i,\pi_{i})$ for all $i\in[n]$. For example, the image on the left in Figure 1 is the plot of $3142567$. A hook of $\pi$ is obtained by starting at a point $(i,\pi_{i})$ in the plot of $\pi$, drawing a vertical line segment moving upward, and then drawing a horizontal line segment to the right that connects with a point $(j,\pi_{j})$. In order for this to make sense, we must have $i<j$ and $\pi_{i}<\pi_{j}$. The point $(i,\pi_{i})$ is called the southwest endpoint of the hook, while $(j,\pi_{j})$ is called the northeast endpoint. Let $\operatorname{\mathsf{SW}}_{i}(\pi)$ be the set of hooks of $\pi$ with southwest endpoint $(i,\pi_{i})$. The right image in Figure 1 shows a hook of $3142567$. This hook is in $\operatorname{\mathsf{SW}}_{3}(3142567)$ because its southwest endpoint is $(3,4)$.

Define the tail length of a permutation $\pi=\pi_{1}\cdots\pi_{n}\in S_{n}$, denoted $\operatorname{tl}(\pi)$, to be the smallest nonnegative integer $\ell$ such that $\pi_{n-\ell}\neq n-\ell$. We make the convention that $\operatorname{tl}(1\cdots n)=n$. The tail of $\pi$ is the sequence of points $(n-\operatorname{tl}(\pi)+1,n-\operatorname{tl}(\pi)+1),\ldots,(n,n)$ in the plot of $\pi$. For example, the tail length of the permutation $3142567$ shown in Figure 1 is $3$, and the tail of this permutation is $(5,5),(6,6),(7,7)$. We say a descent $d$ of $\pi$ is tail-bound if every hook in $\operatorname{\mathsf{SW}}_{d}(\pi)$ has its northeast endpoint in the tail of $\pi$. The only tail-bound descent of $3142567$ is $3$.

Suppose $H$ is a hook of a permutation $\pi=\pi_{1}\cdots\pi_{n}$ with southwest endpoint $(i,\pi_{i})$ and northeast endpoint $(j,\pi_{j})$. Let $\pi_{U}^{H}=\pi_{1}\cdots\pi_{i}\pi_{j+1}\cdots\pi_{n}$ and $\pi_{S}^{H}=\pi_{i+1}\cdots\pi_{j-1}$. The permutations $\pi_{U}^{H}$ and $\pi_{S}^{H}$ are called the $H$-unsheltered subpermutation of $\pi$ and the $H$-sheltered subpermutation of $\pi$, respectively. For example, if $\pi=3142567$ and $H$ is the hook shown on the right in Figure 1, then $\pi_{U}^{H}=3147$ and $\pi_{S}^{H}=25$. In all of the cases we consider in this paper, the plot of $\pi_{S}^{H}$ lies completely below the hook $H$ in the plot of $\pi$ (it is “sheltered” by the hook $H$).

Lemma 2.1** (Refined Decomposition Lemma).**

If $d$ is a tail-bound descent of a permutation $\pi\in S_{n}$, then

[TABLE]

[TABLE]

Proof.

If the tail of $\pi$ is empty, then both sides of the desired equation are [math] because $s^{-1}(\pi)$ and $\operatorname{\mathsf{SW}}_{d}(\pi)$ are empty. Hence, we may assume $\operatorname{tl}(\pi)\geq 1$. Let $a=\pi_{d}$. Given $\sigma\in s^{-1}(\pi)$, we let $f_{\sigma}$ be the entry that forces $a$ to leave the stack when we apply the stack-sorting procedure (described in the introduction) to $\sigma$. More precisely, $f_{\sigma}$ is the leftmost entry that appears to the right of $a$ in $\sigma$ and is larger than $a$. Note that $f_{\sigma}$ appears to the right of $a$ in $\pi$. Because $d$ is tail-bound, this means that the point $(f_{\sigma},f_{\sigma})$ is in the tail of $\pi$. Given a point $(j,j)$ in the tail of $\pi$, let $E_{j}$ be the set of permutations $\sigma\in s^{-1}(\pi)$ such that $f_{\sigma}=j$.

Now fix a point $(j,j)$ in the tail of $\pi$, and let $H$ be the hook in $\operatorname{\mathsf{SW}}_{d}(\pi)$ with northeast endpoint $(j,j)$. We will show that

[TABLE]

from which the lemma will follow. We can write $\pi=L\,a\,\pi_{S}^{H}\,j\,R$, where $L=\pi_{1}\cdots\pi_{d-1}$ and $R=(j+1)\cdots n$. Suppose $\sigma\in E_{j}$. Let us write $\sigma=\tau\,j\,\tau^{\prime}$. Because $j=f_{\sigma}$, it follows from the stack-sorting procedure that every entry in $\tau$ that is smaller than $a$ must appear to the left of $a$ in $s(\sigma)=\pi$. This implies that every entry in $\pi_{S}^{H}$ that is smaller than $a$ is in $\tau^{\prime}$. In particular, $\pi_{d+1}$ is in $\tau^{\prime}$ (we know that $a>\pi_{d+1}$ because $d$ is a descent of $\pi$). Now suppose $b$ is an entry in $\pi_{S}^{H}$ that is larger than $a$. If $b$ is in $\tau$, then we can appeal to the stack-sorting procedure again to see that $b$ must appear to the left of $\pi_{d+1}$ in $\pi$. This is impossible, so every entry in $\pi_{S}^{H}$ is in $\tau^{\prime}$. The stack-sorting procedure forces every entry in $L$ to be in $\tau$, so every entry in $\tau^{\prime}$ that is not in $\pi_{S}^{H}$ must be an entry in $R$. Furthermore, an entry in $R$ that is also in $\tau^{\prime}$ cannot appear to the left of one of the entries from $\pi_{S}^{H}$ in $\tau^{\prime}$ (otherwise, $j$ would appear to the right of one of the entries from $\pi_{S}^{H}$ in $\pi$). This proves that we can write $\tau^{\prime}=\lambda\tau^{\prime\prime}$, where $\lambda$ is a permutation of the entries in $\pi_{S}^{H}$. Moreover, every entry in $\tau^{\prime\prime}$ is in $R$.

Now let $\mu=\tau\tau^{\prime\prime}$. One can verify that $s(\mu)=\pi_{U}^{H}$ and $s(\lambda)=\pi_{S}^{H}$. Let $\delta=1$ if $1$ is a descent of $\tau^{\prime\prime}$, and let $\delta=0$ otherwise. Because $j=f_{\sigma}$, the leftmost entry in $\tau^{\prime\prime}$ is the leftmost entry in $\mu$ that appears to the right of $a$ in $\mu$ and is larger than $a$ (if no such entry exists, then $\tau^{\prime\prime}$ is empty). Also, the rightmost entry in $\tau$ is less than $j$. Combining these observations, we find that $\operatorname{des}(\sigma)+1=\operatorname{des}(\tau)+1+\operatorname{des}(\lambda)+\operatorname{des}(\tau^{\prime\prime})+1=\operatorname{des}(\mu)+1+\operatorname{des}(\lambda)+1$ and $\operatorname{peak}(\sigma)+1=\operatorname{peak}(\tau)+1+\operatorname{peak}(\lambda)+\operatorname{peak}(\tau^{\prime\prime})+\delta+1=\operatorname{peak}(\mu)+1+\operatorname{peak}(\lambda)+1$.

We have shown how to take a permutation $\sigma\in E_{j}$ and decompose it into permutations $\mu\in s^{-1}(\pi_{U}^{H})$ and $\lambda\in s^{-1}(\pi_{S}^{H})$ with $\operatorname{des}(\sigma)+1=\operatorname{des}(\mu)+1+\operatorname{des}(\lambda)+1$ and $\operatorname{peak}(\sigma)+1=\operatorname{peak}(\mu)+1+\operatorname{peak}(\lambda)+1$. We can easily reverse this procedure. Namely, if we are given $\mu$ and $\lambda$, we can write $\mu=\tau\tau^{\prime\prime}$ so that the leftmost entry in $\tau^{\prime\prime}$ is the leftmost entry in $\mu$ that appears to the right of $a$ in $\mu$ and is larger than $a$. We then recover $\sigma$ by letting $\sigma=\tau\,j\,\lambda\,\tau^{\prime\prime}$. ∎

Corollary 2.1** (Decomposition Lemma).**

If $d$ is a tail-bound descent of a permutation $\pi\in S_{n}$, then

[TABLE]

Proof.

Set $x=y=1$ in Lemma 2.1. ∎

3. Fertility Formulas

The purpose of this brief section is to establish some terminology and state some formulas from [26] that we will use in Section 8. We will also use a very special consequence of Theorem 3.1 in Section 4 when we analyze the generating function of the numbers $W_{2}(n,k,p)$.

A composition of $b$ into $a$ parts is an $a$-tuple of positive integers that sum to $b$. For example, $(3,4,3,1)$ is a composition of $11$ into $4$ parts. Let $\operatorname{Comp}_{a}(b)$ denote the set of compositions of $b$ into $a$ parts. Let $C_{r}=\frac{1}{r+1}{2r\choose r}$ denote the $r^{\text{th}}$ Catalan number. Let

[TABLE]

Let

[TABLE]

The numbers $N(r,i)$ are called Narayana numbers. They are given in the OEIS sequence A001263 and constitute the most common refinement of the Catalan numbers [41]. The polynomials $N_{r}(x)$ are called Narayana polynomials. Among many other things, the Narayana numbers $N(r,i)$ count binary plane trees with $r$ vertices and $i-1$ right edges. The numbers $V(r,j)$, which count binary plane trees with $r$ vertices and $j$ leaves, are given in the OEIS sequence A091894. Let $L(r,i,j)$ be the number of binary plane trees with $r$ vertices, $i-1$ right edges, and $j$ leaves. Letting $F(w,x,y)=\sum_{r,i,j\geq 0}L(r,i,j)w^{r}x^{i}y^{j}$, we have

[TABLE]

This yields

[TABLE]

from which one obtains

[TABLE]

Let

[TABLE]

so that

[TABLE]

Theorem 3.1** (​​[26]222Strictly speaking, the first statement in Theorem 3.1 has not been stated explicitly before. However, the proofs of Corollary 5.1 and Theorem 5.2 in [26] immediately generalize to yield that statement.).**

If $n\geq 1$ and $\pi=\pi_{1}\cdots\pi_{n}$ has exactly $k$ descents, then there exists a set $\mathcal{V}(\pi)\subseteq\operatorname{Comp}_{k+1}(n-k)$ such that

[TABLE]

In particular,

[TABLE]

and

[TABLE]

Thus,

[TABLE]

Remark 3.1**.**

If $\pi=123\cdots n$, then the above theorem, along with Theorem 1.1, tells us that

[TABLE]

4. A New Proof of the Formula for $W_{2}(n)$

Recall from Section 2 the definition of the tail length $\operatorname{tl}(\pi)$ of a permutation $\pi$. Let $B_{\ell}(n)$ (respectively, $B_{\geq\ell}(n)$) be the number of $2$-stack-sortable permutations $\sigma\in\mathcal{W}_{2}(n+\ell)$ such that $\operatorname{tl}(s(\sigma))=\ell$ (respectively, $\operatorname{tl}(s(\sigma))\geq\ell$). Let

[TABLE]

Because $\mathcal{W}_{2}(n)=s^{-1}(\mathcal{W}_{1}(n))=s^{-1}(\operatorname{Av}_{n}(231))$ by Theorem 1.1, we can write

[TABLE]

Suppose $\pi\in\mathcal{D}_{\ell}(n+1)$ is such that $\pi_{n+1-i}=n+1$ (where $n\geq 0$). Then $n+1-i$ is a tail-bound descent of $\pi$. The decomposition lemma (Corollary 2.1) tells us that $|s^{-1}(\pi)|$ is equal to the number of triples $(H,\mu,\lambda)$, where $H\in\operatorname{\mathsf{SW}}_{n+1-i}(\pi)$, $\mu\in s^{-1}(\pi_{U}^{H})$, and $\lambda\in s^{-1}(\pi_{S}^{H})$. Choosing $H$ amounts to choosing the number $j\in\{1,\ldots,\ell\}$ such that the northeast endpoint of $H$ is $(n+1+j,n+1+j)$. The permutation $\pi$ and the choice of $H$ determine the permutations $\pi_{U}^{H}$ and $\pi_{S}^{H}$. On the other hand, the choices of $H$ and the permutations $\pi_{U}^{H}$ and $\pi_{S}^{H}$ uniquely determine $\pi$. It follows that $B_{\ell}(n+1)$, which is the number of ways to choose an element of $s^{-1}(\mathcal{D}_{\ell}(n+1))$, is also the number of ways to choose the tuple $(j,\pi_{U}^{H},\pi_{S}^{H},\mu,\lambda)$. Let us fix a choice of $j$.

Because $\pi$ avoids $231$, $\pi_{U}^{H}$ must be a permutation of the set $\{1,\ldots,n-i\}\cup\{n+1\}\cup\linebreak\{n+2+j,\ldots,n+\ell+1\}$, while $\pi_{S}^{H}$ must be a permutation of $\{n-i+1,\ldots,n+j\}\setminus\{n+1\}$. Therefore, choosing $\pi_{U}^{H}$ and $\pi_{S}^{H}$ is equivalent to choosing their normalizations. The normalization of $\pi_{U}^{H}$ is in $\mathcal{D}_{\geq\ell-j+1}(n-i)$, while the normalization of $\pi_{S}^{H}$ is in $\mathcal{D}_{\geq j-1}(i)$ (see Figure 2). Any element of $\mathcal{D}_{\geq\ell-j+1}(n-i)$ can be chosen as the normalization of $\pi_{U}^{H}$, and any element of $\mathcal{D}_{\geq j-1}(i)$ can be chosen as the normalization of $\pi_{S}^{H}$. Also, $\pi_{U}^{H}$ and $\pi_{S}^{H}$ have the same fertilities as their normalizations. Combining these facts, we find that the number of choices for the pair $(\pi_{U}^{H},\mu)$ is $|s^{-1}(\mathcal{D}_{\geq\ell-j+1}(n-i))|=B_{\geq\ell-j+1}(n-i)$. Similarly, the number of choices for the pair $(\pi_{S}^{H},\lambda)$ is $B_{\geq j-1}(i)$. Hence,

[TABLE]

Let

[TABLE]

Note that

[TABLE]

by Theorem 1.1. Let $C(z)=\sum_{n\geq 0}C_{n}z^{n}=\dfrac{1-\sqrt{1-4z}}{2z}$ be the generating function of the Catalan numbers. Because $B_{\geq 0}(n)=W_{2}(n)$ is the total number of $2$-stack-sortable permutations in $S_{n}$, our goal is to understand the generating function

[TABLE]

By (13), we have

[TABLE]

[TABLE]

[TABLE]

On the other hand,

[TABLE]

so

[TABLE]

[TABLE]

Combining (14) and (15) yields the equation

[TABLE]

We can now solve (16) for $C(z)$, use the standard Catalan functional equation $zC(z)^{2}+1-C(z)=0$, and clear denominators to obtain a polynomial

[TABLE]

such that

[TABLE]

Let $Q_{u}^{\prime}=\dfrac{\partial}{\partial u}Q(u,v,w,z)$. There is a unique fractional power series (Puiseux series) $Z=Z(w)$ such that $Z(w)=w+O(w^{2})$ and

[TABLE]

Indeed, we can compute the coefficients of $Z(w)$ one at a time from the equation (18) after we have initially computed sufficiently many terms of $I(w,z)$ via its combinatorial definition. Let $\Delta_{u}Q(v,w,z)$ be the discriminant of $Q(u,v,w,z)$ with respect to the variable $u$. A computer can explicitly compute this discriminant as $\Delta_{u}Q(v,w,z)=w^{6}(1-4z)^{2}z^{3}\widehat{Q}(v,w,z)$, where

[TABLE]

At this point, we use Theorem 14333In the notation of [10], we are applying Theorem 14 with $k=1$. Our polynomial $Q(u,v,w,z)$, power series $I(w,z)$, and power series $I(w,0)$ are playing the roles of $P(x_{0},\ldots,x_{k},t,v)$, $F(t,u)$, and $F_{1}(t)$, respectively, from that article. from the paper [10], which allows us to deduce from (17) and (18) that $z=Z(w)$ is a repeated root of $\Delta_{u}Q(I(w,0),w,z)$. Since $Z(w)=w+O(w^{2})$, we know that $w^{6}(1-4Z)^{2}Z^{3}\neq 0$. Therefore, $z=Z(w)$ is a repeated root of $\widehat{Q}(I(w,0),w,z)$. The discriminant of a polynomial with a repeated root must be [math]. This means that $\Delta_{z}\widehat{Q}(I(w,0),w)=0$, where $\Delta_{z}\widehat{Q}(v,w)$ is the discriminant of $\widehat{Q}(v,w,z)$ with respect to $z$. Computing $\Delta_{z}\widehat{Q}(v,w)$ explicitly and ignoring extraneous factors, we find that $\mathcal{R}(I(w,0),w)=0$, where

[TABLE]

To complete our new proof of Theorem 1.2, we follow the proof of Proposition 5.2 in [12]. Namely, we consider the power series $U(w)$ defined by $U(w)=w(1+U(w))^{3}$. We then verify that $\mathcal{R}(1+U(w)-U(w)^{2},w)=0$ and deduce that $I(w,0)=1+U(w)-U(w)^{2}$. Lagrange inversion then completes the proof that

[TABLE]

The above argument generalizes as follows. Let

[TABLE]

Note that $I_{x,y}(w,0)=\sum_{n\geq 0}\sum_{\sigma\in\mathcal{W}_{2}(n)}w^{n}x^{\operatorname{des}(\sigma)+1}y^{\operatorname{peak}(\sigma)+1}$. We make the convention that the empty permutation has [math] descents and $-1$ peaks so that $I_{x,y}(0,0)=x$. Let $F$ be the generating function in (6). If we replace $B_{\ell}(n)$ with $\sum_{\sigma\in s^{-1}(\mathcal{D}_{\ell}(n))}x^{\operatorname{des}(\sigma)+1}y^{\operatorname{peak}(\sigma)+1}$, replace $B_{\geq\ell}(n)$ with $\sum_{\sigma\in s^{-1}(\mathcal{D}_{\geq\ell}(n))}x^{\operatorname{des}(\sigma)+1}y^{\operatorname{peak}(\sigma)+1}$, use the refined decomposition lemma instead of the decomposition lemma, and use Remark 3.1 instead of Theorem 1.1, then the above argument produces the equation

[TABLE]

in place of (16). We then continue the argument, using the functional equation (6) instead of the Catalan functional equation for $C(z)$, in order to arrive at the following theorem concerning the generating function of the numbers $W_{2}(n,k,p)$.

Theorem 4.1**.**

The generating function

[TABLE]

for the numbers $W_{2}(n,k,p)$ satisfies the equation $R(I_{x,y}(w,0),w,x,y)=0$, where

[TABLE]

[TABLE]

[TABLE]

[TABLE]

5. $3$-Stack-Sortable Permutations

In the previous section, we counted $2$-stack-sortable permutations by viewing them as preimages of $231$-avoiding permutations under the stack-sorting map. In doing so, we had to keep track of the tail lengths of the $231$-avoiding permutations under consideration. In this section, we count $3$-stack-sortable permutations by viewing them as preimages of $2$-stack-sortable permutations. We will again keep track of tail lengths, but we will also need an additional new statistic.

Definition 5.1**.**

Given $\pi=\pi_{1}\cdots\pi_{n}\in S_{n}$ and $a\in\{0,\ldots,n\}$, we say the open interval $(a,a+1)$ is a legal space for $\pi$ if there do not exist indices $i_{1}<i_{2}<i_{3}$ such that $\pi_{i_{3}}\leq a<\pi_{i_{1}}<\pi_{i_{2}}$. Let $\operatorname{leg}(\pi)$ be the number of legal spaces of $\pi$.

For example, if $\pi\in S_{n}$, then $\operatorname{leg}(\pi)=n+1$ if and only if $\pi$ avoids $231$. The legal spaces of $145326$ are $(0,1),(1,2),(4,5),(5,6),(6,7)$, so $\operatorname{leg}(145326)=5$. Imagine adding a new point somewhere to the left of all points in the plot of a permutation $\pi$. One can think of the legal spaces of $\pi$ as the vertical positions where the new point can be inserted so as to not form a new $2341$ pattern. This is relevant for us because of the following characterization of $2$-stack-sortable permutations due to West.

Theorem 5.1** (​​[45]).**

A permutation is $2$-stack-sortable if and only if it avoids the pattern $2341$ and also avoids any $3241$ pattern that is not part of a $35241$ pattern.

We are now in a position to state and prove the main theorems of this article. In what follows, let $B_{\geq\ell}^{(g)}(n)$ be the number of $3$-stack-sortable permutations $\sigma\in\mathcal{W}_{3}(n+\ell)$ such that $\operatorname{tl}(s(\sigma))\geq\ell$ and $\operatorname{leg}(s(\sigma))=\ell+g$. Also, recall the definitions from Section 2.

Theorem 5.2**.**

If $n\geq 1$, then

[TABLE]

where the numbers $B_{\geq\ell}^{(g)}(n)$ satisfy the following relations. We have $B_{\geq\ell}^{(0)}(n)=0$ and

[TABLE]

If $n,g\geq 1$ and $\ell\geq 0$, then

[TABLE]

Proof.

The first statement and the fact that $B_{\geq\ell}^{(0)}(n)=0$ are clear from the definitions we have given. The permutations $\sigma$ counted by $B_{\geq\ell}^{(g)}(1)$ are in $S_{\ell+1}$ and satisfy $\operatorname{tl}(s(\sigma))\geq\ell$, so they must actually satisfy $s(\sigma)=123\cdots(\ell+1)$. Since $\operatorname{leg}(123\cdots(\ell+1))=\ell+2$, the formula for $B_{\geq\ell}^{(g)}(1)$ follows from Theorem 1.1.

Now, let $B_{\ell}^{(g)}(n)$ be the number of $3$-stack-sortable permutations $\sigma\in\mathcal{W}_{3}(n+\ell)$ such that $\operatorname{tl}(s(\sigma))=\ell$ and $\operatorname{leg}(s(\sigma))=\ell+g$. Let

[TABLE]

and

[TABLE]

so that

[TABLE]

We have $B_{\geq\ell}^{(g)}(n+1)=B_{\ell}^{(g)}(n+1)+B_{\geq\ell+1}^{(g-1)}(n)$, so we need to show that

[TABLE]

Suppose $\pi\in\mathcal{D}_{\ell}^{(g)}(n+1)$ is such that $\pi_{n+1-i}=n+1$ (where $n\geq 0$). The decomposition lemma (Corollary 2.1) tells us that $|s^{-1}(\pi)|$ is equal to the number of triples $(H,\mu,\lambda)$, where $H\in\operatorname{\mathsf{SW}}_{n+1-i}(\pi)$, $\mu\in s^{-1}(\pi_{U}^{H})$, and $\lambda\in s^{-1}(\pi_{S}^{H})$. Choosing $H$ amounts to choosing the number $j\in\{1,\ldots,\ell\}$ such that the northeast endpoint of $H$ is $(n+1+j,n+1+j)$. The permutation $\pi$ and the choice of $H$ determine the permutations $\pi_{U}^{H}$ and $\pi_{S}^{H}$. On the other hand, the choices of $H$ and the permutations $\pi_{U}^{H}$ and $\pi_{S}^{H}$ uniquely determine $\pi$. It follows that $B_{\ell}^{(g)}(n+1)$, which is the number of ways to choose an element of $s^{-1}(\mathcal{D}_{\ell}^{(g)}(n+1))$, is also the number of ways to choose the tuple $(j,\pi_{U}^{H},\pi_{S}^{H},\mu,\lambda)$. Let us fix a choice of $j$.

Assume for the moment that $i\leq n-1$, and let $r$ be the largest entry appearing to the left of $n+1$ in $\pi$. Because $\pi$ is $2$-stack-sortable, we can use Theorem 5.1 to see that $\pi_{U}^{H}$ is a permutation of the set $\{1,\ldots,n-i-1\}\cup\{r,n+1\}\cup\{n+2+j,\ldots,n+\ell+1\}$ and that $\pi_{S}^{H}$ is a permutation of $\{n-i,\ldots,n+j\}\setminus\{r,n+1\}$. Therefore, choosing $\pi_{U}^{H}$ and $\pi_{S}^{H}$ is equivalent to choosing their normalizations and the value of $r$. The normalization of $\pi_{S}^{H}$ is in $\mathcal{D}_{\geq j-1}^{(a)}(i)$ for some $a\in\{2,\ldots,i+1\}$, while the normalization of $\pi_{U}^{H}$ is in $\mathcal{D}_{\geq\ell-j+1}^{(b)}(n-i)$ for some $b\in\{2,\ldots,n-i+1\}$. Once we have chosen $a$ and $b$, the number of choices for the tuple $(\pi_{U}^{H},\mu,\pi_{S}^{H},\lambda)$ is $B_{\geq j-1}^{(a)}(i)B_{\geq\ell-j+1}^{(b)}(n-i)$.

Suppose we have already chosen the value of $a$. The fact that $\pi$ avoids $2341$ and the definition of a legal space tell us that there are $a$ possible values of $r$, say $\kappa_{1}<\cdots<\kappa_{a}$ (see Example 5.1 for an illustration of this part of the proof). If we choose $r=\kappa_{m}$, then $\pi$ has $a+b-m+1+\ell$ legal spaces. We are assuming that $\operatorname{leg}(\pi)=\ell+g$, so $g=a+b-m+1$. It follows that $2\leq a\leq n$ and $\max\{2,g-a\}\leq b\leq g-1$. Since $a\in\{2,\ldots,i+1\}$ and $b\in\{2,\ldots,n-i+1\}$, we also have the constraint $a-1\leq i\leq n-b+1$. This explains the expression $\sum_{a=2}^{n}\sum_{b=\max\{2,g-a\}}^{g-1}\sum_{i=a-1}^{n-b+1}\sum_{j=1}^{\ell}B_{\geq j-1}^{(a)}(i)B_{\geq\ell-j+1}^{(b)}(n-i)$ in (21).

The expression $\sum_{j=1}^{\ell}B_{\geq j-1}^{(g-1)}(n)C_{\ell-j+1}$ in (21) comes from the case in which $i=n$. In this case, $\pi_{S}^{H}$ is in $\mathcal{D}_{\geq j-1}^{(g-1)}(n)$, and $\pi_{U}^{H}=(n+1)(n+2+j)(n+3+j)\cdots(n+\ell+1)$ is an increasing permutation of length $\ell-j+1$. The number of choices for the pair $(\pi_{S}^{H},\lambda)$ is $B_{\geq j-1}^{(g-1)}(n)$. The number of choices for $\mu$ is $|s^{-1}(\pi_{U}^{H})|=C_{\ell-j+1}$. ∎

Example 5.1**.**

Consider the part of the proof of Theorem 5.2 in which we have already chosen $n,g,\ell,j,i$ and have assumed $i\leq n-1$. Suppose $n=8$, $\ell=5$, $j=2$, and $i=5$. If we choose the normalization of $\pi_{U}^{H}$ to be $24315678$ and choose the normalization of $\pi_{S}^{H}$ to be $315246$, then $a=\operatorname{leg}(315246)-(j-1)=5$ and $b=\operatorname{leg}(24315678)-(\ell-j+1)=4$. The green dots in Figure 3 represent the possible choices for $r$, which are $\kappa_{1}=4$, $\kappa_{2}=5$, $\kappa_{3}=7$, $\kappa_{4}=8$, and $\kappa_{5}=9$. If $r=\kappa_{m}$, then we can refer to this figure to see that $\operatorname{leg}(\pi)=15-m=\ell+a+b-m+1$. Hence, the choice of $r$ is determined by the value of $g$.

The proof of Theorem 5.2 generalizes, allowing us to obtain a recurrence for $W_{3}(n,k,p)$, the number of $3$-stack-sortable permutations in $S_{n}$ with $k$ descents and $p$ peaks. We actually state the following theorem in terms of polynomials, but one can obtain the desired recurrence by comparing coefficients. In what follows, let

[TABLE]

where $\mathcal{D}_{\geq\ell}^{(g)}(n)$ is as in (20). We have suppressed the dependence on $x$ and $y$ in our notation for readability. Let $L_{r}(x,y)$ be as in (8).

Theorem 5.3**.**

If $n\geq 1$, then

[TABLE]

where the polynomials $E_{\geq\ell}^{(g)}(n)$ satisfy the following relations. We have $E_{\geq\ell}^{(0)}(n)=0$ and

[TABLE]

If $n,g\geq 1$ and $\ell\geq 0$, then

[TABLE]

[TABLE]

Proof.

To derive the formula for $E_{\geq\ell}^{(g)}(1)$, we follow the same argument used to find the formula for $B_{\geq\ell}^{(g)}(1)$ in the proof of Theorem 5.2, except we use Remark 3.1 instead of Theorem 1.1. To derive the last statement in this theorem, we follow the rest of the proof of Theorem 5.2, except we invoke the refined decomposition lemma instead of the decomposition lemma and again use Remark 3.1 instead of Theorem 1.1. ∎

6. Data Analysis

The sum of two permutations $\mu$ and $\lambda$, denoted $\mu\oplus\lambda$, is the permutation whose plot is obtained by placing the plot of $\lambda$ above and to the right of the plot of $\mu$. It is easy to check that the sum of two $t$-stack-sortable permutations is $t$-stack-sortable. It follows that $W_{t}(m+n)\geq W_{t}(m)W_{t}(n)$ for all $m,n\geq 1$. We express this by saying the sequence $(W_{t}(n))_{n\geq 1}$ is supermultiplicative. It follows from Fekete’s lemma that

[TABLE]

We have used Theorem 5.2 to compute the numbers $W_{3}(n)$ for $n\leq 174$. We have added these terms to the OEIS sequence A134664. This allows us to prove the first nontrivial lower bound for $\lim\limits_{n\to\infty}W_{3}(n)^{1/n}$. Note that this is better than the lower bound of $(\sqrt{3}+1)^{2}$ obtained in Section 7.

Theorem 6.1**.**

We have

[TABLE]

Proof.

The value of $W_{3}(174)$ is

[TABLE]

and the $174^{\text{th}}$ root of this number is slightly more than $8.659702$. The proof follows from (22). ∎

We can also show that Bóna’s Conjectures 1.1 and 1.2 contradict each other.

Theorem 6.2**.**

If $(W_{3}(n))_{n\geq 1}$ is log-convex, then $W_{3}(n)>{4n\choose n}$ for all sufficiently large $n$.

Proof.

It follows from Stirling’s formula that $\lim\limits_{n\to\infty}{4n\choose n}^{1/n}=256/27\approx 9.4815$. Also, $\dfrac{W_{3}(174)}{W_{3}(173)}\approx 9.4907$. If $(W_{3}(n))_{n\geq 1}$ is log-convex, then $\lim\limits_{n\to\infty}W_{3}(n)^{1/n}=\lim\limits_{n\to\infty}\dfrac{W_{3}(n+1)}{W_{3}(n)}\geq 9.4907>9.4815$. ∎

We now turn our attention to the parity of $W_{3}(n)$ and Bóna’s Conjecture 1.3. Let $\varepsilon_{t}(n)$ be the number in $\{0,1\}$ with the same parity as $W_{t}(n)$. As mentioned in the introduction, $\varepsilon_{t}(n)=0$ whenever $n$ is even. The values of $\varepsilon_{3}(2n+1)$ for $0\leq n\leq 86$ are

[TABLE]

[TABLE]

Letting $\mathfrak{g}_{t}(m)=\sum_{n=1}^{m}\varepsilon_{t}(n)$, we can use this data to find that $\mathfrak{g}_{1}(m)<\mathfrak{g}_{2}(m)\leq\mathfrak{g}_{3}(m)$ whenever $13\leq m\leq 660$. Therefore, it appears that $W_{3}(n)$ is odd more frequently than $W_{2}(n)$! If this is true, then Bóna’s Conjecture 1.3 is certainly false. We state some new conjectures and open problems concerning the parities of the numbers $W_{3}(n)$ in Section 9.

Let us end this section by recording one final proposition, which verifies Bóna’s Conjecture 1.4 in several new cases. We have obtained this proposition by computing several values of $W_{3}(n,k)$ via Theorem 5.3 (setting $y=1$ in that theorem).

Proposition 6.1**.**

If $n\leq 43$, then the polynomial $\displaystyle\sum_{\sigma\in\mathcal{W}_{3}(n)}x^{\operatorname{des}(\sigma)}$ has only real roots.

7. Lower Bounds for $t$-Stack-Sortable Permutations

Let $\Gamma_{t}$ be the set of all $\kappa=\kappa_{1}\cdots\kappa_{t+2}\in S_{t+2}$ such that $\kappa_{t+1}=t+2$ and $\kappa_{t+2}=1$. Let $\operatorname{Av}_{n}(\Gamma_{t})$ be the set of permutations in $S_{n}$ that avoid all of the patterns in $\Gamma_{t}$. After applying a dihedral symmetry to the permutations in $\Gamma_{t}$, we can use a result of Kremer [38, 39] to see that

[TABLE]

Some basic singularity analysis now shows that $\lim\limits_{n\to\infty}|\operatorname{Av}_{n}(\Gamma_{t})|^{1/n}=(\sqrt{t}+1)^{2}$.

We will prove by induction that $\operatorname{Av}_{n}(\Gamma_{t})\subseteq\mathcal{W}_{t}(n)$. Since $\Gamma_{1}=\{231\}$, this is certainly true for $t=1$ (by Theorem 1.1). Now suppose that $t\geq 2$ and that $\operatorname{Av}_{n}(\Gamma_{t-1})\subseteq\mathcal{W}_{t-1}(n)$. Choose a permutation $\pi\in S_{n}\setminus\mathcal{W}_{t}(n)$. This means that $s(\pi)\not\in\mathcal{W}_{t-1}(n)$, so $s(\pi)$ contains a permutation in $\Gamma_{t-1}$. In other words, there exist entries $b_{1},\ldots,b_{t-1},c,a$ that appear in this order in $s(\pi)$ and satisfy $a<b_{j}<c$ for all $j\in\{1,\ldots,t-1\}$. Because $c$ appears to the left of $a$ in $s(\pi)$, there must be an entry $d>c$ that appears to the right of $c$ and to the left of $a$ in $\pi$. The entries $b_{1},\ldots,b_{t-1}$ must appear to the left of $d$ in $\pi$ since they would appear to the right of $c$ in $s(\pi)$ otherwise. The subpermutation of $\pi$ formed by the entries $a,b_{1},\ldots,b_{t-1},c,d$ has a normalization that is in $\Gamma_{t}$, so $\pi\not\in\operatorname{Av}_{n}(\Gamma_{t})$. This completes the induction and proves the following theorem.

Theorem 7.1**.**

For every $t\geq 1$, we have

[TABLE]

In [43], Smith investigated a variant of the stack-sorting map known as the “left-greedy algorithm.” Let $\widehat{\mathcal{W}}_{t}(n)$ be the set of permutations in $S_{n}$ that can be sorted by $t$ stacks in series using the left-greedy algorithm (see her paper for definitions). Smith proved that $\mathcal{W}_{t}(n)\subseteq\widehat{\mathcal{W}}_{t}(n)$ and that $|\widehat{\mathcal{W}}_{t}(n)|\geq\dfrac{t!}{(t+1)^{t}}(t+1)^{n}$ whenever $n\geq t\geq 1$. In terms of exponential growth rates, this shows that $\lim\limits_{n\to\infty}|\widehat{\mathcal{W}}_{t}(n)|^{1/n}\geq t+1$ (using Fekete’s lemma, one can show that this limit exists). The following corollary of Theorem 7.1 improves this estimate.

Corollary 7.1**.**

For every $t\geq 1$, we have $\lim\limits_{n\to\infty}|\widehat{\mathcal{W}}_{t}(n)|^{1/n}\geq(\sqrt{t}+1)^{2}$.

8. Symmetry, Unimodality, $\gamma$-Nonnegativity, Log-Concavity, and Real-Rootedness

We devote this brief section to showing how Brändén’s theorem concerning $\gamma$-nonnegativity (Theorem 1.3) follows easily from Theorem 3.1. We also show that the analogue of that theorem with “$\gamma$-nonnegative” replaced by “real-rooted” is false. Let us begin by recalling some definitions.

A polynomial $p(x)=a_{0}+a_{1}x+\cdots+a_{n}x^{n}\in\mathbb{R}_{\geq 0}[x]$ is called

• •

symmetric if $a_{i}=a_{n-i}$ for all $i\in\{0,\ldots,n\}$; in this case, $n/2$ is called the center of symmetry of $p(x)$;

• •

unimodal if there exists $j\in\{0,\ldots,n\}$ such that $a_{0}\leq a_{1}\leq\cdots\leq a_{j}\geq a_{j+1}\geq\cdots\geq a_{n}$;

• •

log-concave if $a_{i-1}a_{i+1}\leq a_{i}^{2}$ for all $i\in\{1,\ldots,n-1\}$;

• •

real-rooted if all of the complex roots of $p(x)$ are real.

If $p(x)$ is a symmetric polynomial with center of symmetry $n/2$, then it can be written in the form $p(x)=\sum_{m=0}^{\left\lfloor n/2\right\rfloor}\gamma_{m}x^{m}(1+x)^{n-2m}$ for some real numbers $\gamma_{m}$. We then say $p(x)$ is $\gamma$-nonnegative if the numbers $\gamma_{m}$ are all nonnegative. We have the following implications among these properties for polynomials in $\mathbb{R}_{\geq 0}[x]$ [16]:

[TABLE]

[TABLE]

New Proof of Theorem 1.3.

444To deduce Bóna’s symmetry and unimodality result from Theorem 3.1, one simply needs to observe that this theorem tells us that $\sum_{\sigma\in\mathcal{W}_{t}(n)}x^{\operatorname{des}(\sigma)+1}$ is a sum of products of Narayana polynomials with the same center of symmetry and then use the well-known fact that Narayana polynomials are real-rooted.

Note that it suffices to prove Theorem 1.3 in the specific case in which $A=\{\pi\}$ is a singleton set. Indeed, the result for a general set $A\subseteq S_{n}$ then follows by summing over all $\pi\in A$. Thus, let us fix a permutation $\pi\in S_{n}$ with exactly $k$ descents.

Recall the notation from (4) and (5). One can show that

[TABLE]

for all $q\geq 1$. Therefore, for $(q_{0},\ldots,q_{k})\in\operatorname{Comp}_{k+1}(n-k)$, we have

[TABLE]

[TABLE]

Let $\mathcal{V}(\pi)\subseteq\operatorname{Comp}_{k+1}(n-k)$ be the set of compositions from Theorem 3.1. Invoking equation (10) from that theorem, we obtain

[TABLE]

[TABLE]

[TABLE]

where we have made the substitution $m_{i}^{\prime}=m_{i}+1$. It turns out that

[TABLE]

is the coefficient of $y^{m+1}$ in the polynomial on the right-hand side of (11), so it is equal to

[TABLE]

Note that this is [math] if $m>\frac{n-1}{2}$. Hence,

[TABLE]

We now give an example to show that Theorem 1.3 is false if the term “$\gamma$-nonnegative” is replaced by “real-rooted.”

Example 8.1**.**

Let

[TABLE]

We claim that $\sum_{\sigma\in s^{-1}(\{\mu,\mu^{\prime}\})}x^{\operatorname{des}(\sigma)}$ is not real rooted. To see this, we use the fact555The reader interested in seeing why this is the case can refer to [28] for the full definition of $\mathcal{V}(\pi)$ and a description of how to compute it. However, the reader wishing to avoid this definition can still compute $\sum_{\sigma\in s^{-1}(\{\mu,\mu^{\prime}\})}x^{\operatorname{des}(\sigma)}$ using a brute-force computer program that simply finds all of the permutations in $s^{-1}(\{\mu,\mu^{\prime}\})$. A priori, a brute-force computer program would not easily find this example since it would have to search over subsets of $S_{11}$. that $\mathcal{V}(\mu)=\{(4,2,3),(3,3,3)\}$ and $\mathcal{V}(\mu^{\prime})=\{(4,4,1)\}$. Using (10), we find that

[TABLE]

[TABLE]

[TABLE]

and this polynomial is not real-rooted. This example yields a negative answer to the last part of Question 12.1 in [28].

Remark 8.1**.**

Theorem 1.3 diverges from Bóna’s point of view in Conjecture 1.4 by replacing the sum over $s^{-1}(\mathcal{W}_{t-1}(n))$ with a sum over $s^{-1}(A)$ for an arbitrary set $A\subseteq S_{n}$. This different viewpoint suggests that the sets of the form $\mathcal{W}_{t}(n)=s^{-1}(\mathcal{W}_{t-1}(n))$ might not be too special when compared with arbitrary sets of the form $s^{-1}(A)$ for $A\subseteq S_{n}$. If one believes Conjecture 1.4, then the preceding example lends credence to the hypothesis that the sets $\mathcal{W}_{t}(n)$ are special. On the other hand, if one does not believe there is anything special about the sets $\mathcal{W}_{t}(n)$, then this example hints that Conjecture 1.4 might be false.

9. Conjectures and Open Problems

We saw in Theorem 6.2 that Conjectures 1.1 and 1.2 cannot both be true. Our data suggests that Conjecture 1.2 is true. Moreover, by plotting the points $(1/n,W_{3}(n))$ for $1\leq n\leq 174$, we have arrived at the following new conjecture.

Conjecture 9.1**.**

We have

[TABLE]

We also believe that the decomposition lemma could be used (possibly along with a significant amount of work) to find a lower bound for $\lim\limits_{n\to\infty}W_{4}(n)^{1/n}$ that exceeds $9.704$.

Turning back to the parities of the numbers $W_{3}(n)$, we have the following problem.

Problem 9.1**.**

Characterize the positive integers $n$ such that $W_{3}(n)$ is odd.

Problem 9.1 seems more tractable now that we have obtained a recurrence for the numbers $W_{3}(n)$ in Theorem 5.2. Indeed, it appears as though there could be some patterns in the sequence whose initial terms are listed in (23). Solving this problem could require going through the proof of Theorem 5.2 and seeing which terms in the various sums simplify when we reduce modulo $2$.

Recall the definition of $\mathfrak{g}_{t}(m)$ from Section 6. We have the following conjectures. Conjectures 9.3, 9.4, and 9.5 each contradict Bóna’s Conjecture 1.3.

Conjecture 9.2**.**

The limit $\displaystyle\lim_{n\to\infty}\frac{\log\mathfrak{g}_{3}(m)}{\log m}$ exists.

Conjecture 9.3**.**

We have $\displaystyle\liminf_{n\to\infty}\frac{\log\mathfrak{g}_{3}(m)}{\log m}>0$.

Conjecture 9.4**.**

For every integer $m\geq 13$, we have $\mathfrak{g}_{2}(m)\leq\mathfrak{g}_{3}(m)$.

Conjecture 9.5**.**

We have $\lim\limits_{m\to\infty}(\mathfrak{g}_{3}(m)-\mathfrak{g}_{2}(m))=\infty$.

10. Acknowledgments

I would like to express my deepest gratitude to Niven Achenjang, Amanda Burcroff, and Eric Winsor for writing computer programs that calculated the numbers $W_{3}(n)$ much faster than the author’s original program. I would also like to thank Jay Pantone for running one of these programs on his computer for several days and for analyzing the resulting data. The contributions that these people made were paramount to the analysis discussed in Section 6. I thank Miklós Bóna and Doron Zeilberger for helpful conversations. I also thank Caleb Ji, who wrote a poem that inexplicably predicted I would make progress in the study of $3$-stack-sortable permutations. I thank the anonymous referees for helpful comments that improved the presentation of this article.

The author was supported by a Fannie and John Hertz Foundation Fellowship and an NSF Graduate Research Fellowship.