On pattern-avoiding Fishburn permutations

TL;DR

This paper explores pattern avoidance in Fishburn permutations, providing enumerative results for various pattern classes, including those counted by Catalan and binomial transform numbers, and proposing conjectures for further classes.

Contribution

It offers a complete enumeration of classical pattern avoidance in Fishburn permutations for size 3 and 4, introducing new enumerative results and conjectures for other pattern classes.

Findings

Classical pattern avoidance in Fishburn permutations for size 3 fully characterized.

Certain Fishburn permutation classes are enumerated by Catalan numbers.

Additional classes are linked to binomial transforms of Catalan numbers.

Abstract

The class of permutations that avoid the bivincular pattern (231, {1},{1}) is known to be enumerated by the Fishburn numbers. In this paper, we call them Fishburn permutations and study their pattern avoidance. For classical patterns of size 3, we give a complete enumerative picture for regular and indecomposable Fishburn permutations. For patterns of size 4, we focus on a Wilf equivalence class of Fishburn permutations that are enumerated by the Catalan numbers. In addition, we also discuss a class enumerated by the binomial transform of the Catalan numbers and give conjectures for other equivalence classes of pattern-avoiding Fishburn permutations.

Tables

Table 1. Table 1. σ 𝜎 \sigma -avoiding Fishburn permutations.

Pattern $\sigma$	$\left|{ℱ}_n\left(\sigma \right)\right|$	OEIS
123, 132, 213, 312	1, 2, 4, 8, 16, 32, 64, 128, 256, 512, …	A000079
231	1, 2, 5, 14, 42, 132, 429, 1430, 4862, …	A000108
321	1, 2, 4, 9, 22, 57, 154, 429, 1223, 3550, …	A105633

Table 2. Table 2. σ 𝜎 \sigma -avoiding indecomposable Fishburn permutations.

Pattern $\sigma$	$\left|{ℱ}_n^{\text{ind}}\left(\sigma \right)\right|$	OEIS
123	1, 1, 2, 5, 12, 27, 58, 121, 248, 503, …	A000325
132, 213	1, 1, 2, 4, 8, 16, 32, 64, 128, 256, …	A000079
231	1, 1, 2, 5, 14, 42, 132, 429, 1430, 4862, …	A000108
312	1, 1, 1, 1, 1, 1, 1, 1, 1, 1, …	A000012
321	1, 1, 1, 2, 5, 13, 35, 97, 275, 794, …	A082582

Table 3. Table 3. Equivalence classes with a single pattern

Pattern $\sigma$	$\left|{ℱ}_n\left(\sigma \right)\right|$	OEIS
1342	1, 2, 5, 15, 51, 188, 731, 2950, …	A007317
1432	1, 2, 5, 14, 43, 142, 495, 1796, …
2314	1, 2, 5, 15, 52, 200, 827, 3601, …
2341	1, 2, 5, 15, 52, 202, 858, 3910, …
3412	1, 2, 5, 15, 52, 201, 843, 3764, …	A202062(?)
3421	1, 2, 5, 15, 52, 203, 874, 4076, …
4123	1, 2, 5, 14, 42, 133, 442, 1535, …
4231	1, 2, 5, 15, 52, 201, 843, 3765, …
4312	1, 2, 5, 14, 43, 143, 508, 1905, …
4321	1, 2, 5, 14, 45, 162, 639, 2713, …

Table 4. Table 4.

Pattern $\sigma$	$\left|{ℱ}_n\left(\sigma \right)\right|$	OEIS
1234, 1243, 1324, 1423, 2134, 2143, 3124, 3142	1, 2, 5, 14, 42, 132, 429, 1430, 4862, …	A000108

Table 5. Table 5.

Pattern $\sigma$	$\left|{ℱ}_n^{\text{ind}}\left(\sigma \right)\right|$	OEIS
1234	1, 1, 2, 6, 22, 85, 324, 1204,…
1243, 2134	1, 1, 2, 6, 21, 75, 266, 938,…	A289597(?)
1324	1, 1, 2, 6, 22, 84, 317, 1174, …
1342	1, 1, 2, 6, 22, 88, 367, 1568, …	A165538
1423, 3124	1, 1, 2, 6, 20, 68, 233, 805, …	A279557
1432	1, 1, 2, 6, 20, 71, 263, 1002, …
2143	1, 1, 2, 6, 19, 62, 207, 704, …	A026012
2314	1, 1, 2, 6, 23, 99, 450, 2109, …
2341	1, 1, 2, 6, 22, 91, 409, 1955, …
2413, 2431, 3241	1, 1, 2, 6, 22, 90, 395, 1823, …	A165546(?)
3142	1, 1, 2, 5, 14, 42, 132, 429, …	A000108
3214	1, 1, 2, 6, 20, 72, 275, 1096, …
3412	1, 1, 2, 6, 22, 90, 396, 1840, …
3421	1, 1, 2, 6, 22, 92, 423, 2088, …
4123	1, 1, 2, 5, 14, 43, 143, 507, …
4132, 4213	1, 1, 2, 5, 15, 51, 188, 732, …
4231	1, 1, 2, 6, 22, 90, 396, 1841, …
4312	1, 1, 2, 5, 15, 51, 188, 733, …
4321	1, 1, 2, 5, 17, 66, 279, 1256, …

Equations

$$\sum _{n=0}^{\infty }\xi (n)q^n=1+\sum _{n=1}^{\infty }\prod _{j=1}^n(1-(1-q{)}^j).$$

$${\mathcal{F}}_n(231)={\mathsf{Av}}_n(231),\text{ and so }{F}_n(231)=C_n.$$

$$F_n(132)=2F_{n-1}(132)\text{ for }n>1,$$

$$F_n(123)=F_{n-1}(123)+1+\sum _{i=2}^{n-1}\left(\genfrac{}{}{0ex}{}{n-2}{i-2}\right)=F_{n-1}(123)+2^{n-2},$$

$$F_n(213)=1+\sum _{i=1}^{n-1}{F}_{n-i}(213),$$

$$F_n(312)=1+\sum _{i=1}^{n-1}{F}_{n-i}(312),$$

$$F_n(321)=\sum _{j=0}^{\lfloor (n-1)/2\rfloor }\frac{(-1{)}^j}{n-j}\left(\genfrac{}{}{0ex}{}{n-j}{j}\right)\left(\genfrac{}{}{0ex}{}{2n-3j}{n-j+1}\right).$$

$$1+A^{\sigma }(x)=\frac{1}{1-A_I^{\sigma }(x)},\text{ and so }{A}_I^{\sigma }(x)=\frac{A^{\sigma }(x)}{1+A^{\sigma }(x)}.$$

$$1,1,2,6,23,104,534,3051,19155,130997,\dots .$$

$$1n(n-1)\cdots 32,$$

$$21n(n-1)\cdots 3,$$

$$⋮$$

$$(n-1)\cdots 321n.$$

$${\mathcal{F}}_n^{\text{ind}}(132)=\{1\ominus \pi :\pi \in {\mathcal{F}}_{n-1}(132)\}.$$

$${\mathcal{F}}_n^{\text{ind}}(213)=\{1\ominus \pi :\pi \in {\mathcal{F}}_{n-1}(213)\}.$$

$$I{F}^{\sigma }(x)=\frac{F^{\sigma }(x)}{1+F^{\sigma }(x)}.$$

$$a_n=a_{n-1}+\sum _{j=2}^{n-2}{a}_j{a}_{n-j}\text{ for }n\ge 4,$$

$$F_n(1342)=\sum _{k=1}^n\left(\genfrac{}{}{0ex}{}{n-1}{k-1}\right)C_{n-k}.$$

$$|{\mathcal{A}}_n|=\sum _{k=1}^n|{\mathcal{A}}_{n,k}|=\sum _{k=1}^n\left(\genfrac{}{}{0ex}{}{n-1}{k-1}\right)C_{n-k}.$$

$$\pi (1)>\cdots >\pi (j)<\pi (j+1),$$

$$\varphi :{\mathsf{Av}}_n(\tau \oplus 12)\to {\mathsf{Av}}_n(\tau \oplus 21)$$

$$\tilde{\pi }(j)=\pi (j)\text{ if }j\notin \{i_1,\dots ,i_{\ell }\}.$$

$$\tilde{\pi }(i_1)=b_1.$$

$$\tilde{\pi }(i_k)=b_k\text{ for }k=2,\dots ,\ell ,$$

$$\varphi :{\mathcal{F}}_n(\tau \oplus 12)\to {\mathcal{F}}_n(\tau \oplus 21).$$

$$2135476$$

$$\swarrow$$

$$2153476$$

$$\pi (j-1)<\pi (j)\text{ and }\pi (k-1)<\pi (k).$$

$${\mathcal{F}}_n(3142)\stackrel{{\alpha }_1}{⟶}{\mathcal{F}}_n(3124)\text{ and }{\mathcal{F}}_n(3124)\stackrel{{\alpha }_2}{⟶}{\mathcal{F}}_n(1324)$$

$$Q=\{q\in [n]:\tilde{\pi }(i)\le \tilde{\pi }(q)<\tilde{\pi }({\ell }_m)\}.$$

$$|{\mathcal{F}}_n^{\text{ind}}(3142)|=C_{n-1}.$$

Full text

1. Introduction

Motivated by a recent paper by G. Andrews and J. Sellers [1], we became interested in the Fishburn numbers $\xi(n)$, defined by the formal power series

[TABLE]

They are listed as sequence A022493 in [7] and have several combinatorial interpretations. For example, $\xi(n)$ gives the:

• $\triangleright$

number of linearized chord diagrams of degree $n$,

• $\triangleright$

number of unlabeled $(2+2)$-free posets on $n$ elements,

• $\triangleright$

number of ascent sequences of length $n$,

• $\triangleright$

number of permutations in $S_{n}$ that avoid a certain bivincular pattern.111$S_{n}$ denotes the set of permutations on $[n]=\{1,\dots,n\}$.

For more on these interpretations, we refer the reader to [2] and the references there in.

In this note, we are primarily concerned with the aforementioned class of permutations. That they are enumerated by the Fishburn numbers was proved in [2] by Bousquet-Mélou, Claesson, Dukes, and Kitaev, where the authors introduced bivincular patterns (permutations with restrictions on the adjacency of positions and values) and gave a bijection to ascent sequences. More specifically, a permutation $\pi\in S_{n}$ is said to contain the bivincular pattern $(231,\{1\},\{1\})$ if there are positions $i<k$ with $\pi(i)>1$, $\pi(k)=\pi(i)-1$, such that the subsequence $\pi(i)\pi(i+1)\pi(k)$ forms a 231 pattern. Such a bivincular pattern may be visualized by the plot

where bold lines indicate adjacent entries and gray lines indicate an elastic distance between the entries.

We let $\mathscr{F}_{n}$ denote the class of permutations in $S_{n}$ that avoid the pattern , and since $|\mathscr{F}_{n}|=\xi(n)$ (see [2]), we call the elements of $\mathscr{F}=\bigcup_{n}\mathscr{F}_{n}$ Fishburn permutations. Further, we let $\mathscr{F}_{n}(\sigma)$ denote the class of Fishburn permutations that avoid the pattern $\sigma$.

Our goal is to study $F_{n}(\sigma)=\left|\mathscr{F}_{n}(\sigma)\right|$ for classical patterns of size 3 or 4. In Section 2, we give a complete picture for regular and indecomposable Fishburn permutations that avoid a classical pattern of size 3. Table 1 and Table 2 at the end of that section provide a summary of our findings. In Section 3, we discuss patterns of size 4, focusing on a Wilf equivalence class of Fishburn permutations that are enumerated by the Catalan numbers $C_{n}$ (see Table 4). We also prove the formula $F_{n}(1342)=\sum_{k=1}^{n}\binom{n-1}{k-1}C_{n-k}$, and conjecture two other equivalence classes (see Table 3). Finally, in Section 4, we briefly discuss indecomposable Fishburn permutations that avoid a pattern of size 4. In Table 5, we make some conjectures based on our limited preliminary computations.

Basic notation. Permutations will be written in one-line notation. Given two permutations $\sigma$ and $\tau$ of sizes $k$ and $\ell$ respectively, their direct sum $\sigma\oplus\tau$ is the permutation of size $k+\ell$ consisting of $\sigma$ followed by a shifted copy of $\tau$. Similarly, their skew sum $\sigma\ominus\tau$ is the permutation consisting of $\tau$ preceded by a shifted copy of $\sigma$. For example, $312\oplus 21=31254$ and $312\ominus 21=53421$.

A permutation is said to be indecomposable if it cannot be written as a direct sum of two nonempty permutations.

$\mathsf{Av}_{n}(\sigma)$ denotes the class of permutations in $S_{n}$ that avoid the pattern $\sigma$. It is well known that if $\sigma\in S_{3}$ then $|\mathsf{Av}_{n}(\sigma)|=C_{n}$, where $C_{n}$ is the Catalan number $\frac{1}{n+1}\binom{2n}{n}$, see e.g. [4].

2. Avoiding patterns of size 3

Clearly, $\mathsf{Av}_{n}(231)\subset\mathscr{F}_{n}$. Now, since every Fishburn permutation that avoids the classical pattern $231$ is contained in the set of $231$-avoiding permutations, we get

[TABLE]

Enumeration of the Fishburn permutations that avoid the other five classical patterns of size 3 is less obvious.

Theorem 2.1**.**

For $\sigma\in\{123,132,213,312\}$, we have $F_{n}(\sigma)=2^{n-1}$.

Proof.

First of all, note that for every $\sigma$ of size 3, we have $F_{1}(\sigma)=1$ and $F_{2}(\sigma)=2$.

Case $\sigma=132$: If $\pi\in\mathscr{F}_{n-1}(132)$, the permutations $1\ominus\pi$ and $\pi\oplus 1$ are both in $\mathscr{F}_{n}(132)$. On the other hand, if $\tau$ is a permutation in $\mathsf{Av}_{n}(132)$ with $\tau(i)=n$ for some $1<i<n$, then we must have $\tau(j)>\tau(k)$ for every $j\in\{1,\dots,i-1\}$ and $k\in\{i+1,\dots,n\}$. Thus $n-i=\tau(k^{\prime})$ for some $k^{\prime}>i$ and $n-i+1=\tau(j^{\prime})$ for some $j^{\prime}<i$. But this violates the Fishburn condition since $n-i+1$ is the smallest value left from $n$ and must therefore be part of an ascent in $\tau(1)\cdots\tau(i-1)\,n$. In other words, $\mathscr{F}_{n}(132)$ is the disjoint union of the sets $\{1\ominus\pi:\pi\in\mathscr{F}_{n-1}(132)\}$ and $\{\pi\oplus 1:\pi\in\mathscr{F}_{n-1}(132)\}$. Thus

[TABLE]

which implies $F_{n}(132)=2^{n-1}$.

Case $\sigma=123$: For $n>2$, the permutation $(n-1)(n-2)\cdots 21n$ is the only permutation in $\mathscr{F}_{n}(123)$ that ends with $n$, and if $\pi\in\mathscr{F}_{n-1}(123)$, then $1\ominus\pi\in\mathscr{F}_{n}(123)$.

Assume $\tau\in\mathscr{F}_{n}(123)$ is such that $\tau(i)=n$ for some $1<i<n$. Since $\tau$ avoids the pattern 123, we must have $\tau(1)>\tau(2)>\cdots>\tau(i-1)$. Moreover, the Fishburn condition forces $\tau(i-1)=1$, which implies $\tau(i+1)>\tau(i+2)>\cdots>\tau(n)$. In other words, $\tau$ may be any permutation with $\tau(i-1)=1$, $\tau(i)=n$, and such that the entries left from 1 and right from $n$ form two decreasing sequences. There are $\binom{n-2}{i-2}$ such permutations.

In conclusion, we have the recurrence

[TABLE]

which implies $F_{n}(123)=2^{n-1}$.

Case $\sigma=213$: For $n>2$, the permutation $12\cdots n$ is the only permutation in $\mathscr{F}_{n}(213)$ that ends with $n$, and if $\pi\in\mathscr{F}_{n-1}(213)$, then $1\ominus\pi\in\mathscr{F}_{n}(213)$.

Assume $\tau\in\mathscr{F}_{n}(213)$ is such that $\tau(i)=n$ for some $1<i<n$. Since $\tau$ avoids the pattern 213, we must have $\tau(1)<\tau(2)<\cdots<\tau(i-1)$ and the Fishburn condition forces $\tau(j)=j$ for every $j\in\{1,\dots,i-1\}$. Thus $\tau$ must be of the form $\tau=1\cdots(i-1)n|\pi_{R}$, where $\pi_{R}$ may be any element of $\mathscr{F}_{n-i}(213)$. This implies

[TABLE]

and we conclude $F_{n}(213)=2^{n-1}$.

Case $\sigma=312$: If $\pi\in\mathscr{F}_{n-1}(312)$, the permutation $1\oplus\pi$ is in $\mathscr{F}_{n}(312)$. On the other hand, if $\tau$ is a permutation in $\mathsf{Av}_{n}(312)$ with $\tau(i)=1$ for some $1<i\leq n$, then we must have $\tau(j)<\tau(k)$ for every $j\in\{1,\dots,i-1\}$ and $k\in\{i+1,\dots,n\}$. Moreover, the Fishburn condition forces $\tau(j)=i+1-j$ for every $j\in\{1,\dots,i-1\}$. Thus $\tau$ must be of the form $\tau=i\cdots 21|\pi_{R}$, where $\pi_{R}=\emptyset$ if $i=n$, or $\pi_{R}\in\mathscr{F}_{n-i}(312)$ if $i<n$. This implies

[TABLE]

hence $F_{n}(312)=2^{n-1}$. ∎

For our next result, we use a bijection between $\mathsf{Av}_{n}(321)$ and the set of Dyck paths of semilength $n$, via the left-to-right maxima.222This is a slightly different version of a bijection given by Krattenthaler [5]. Here a Dyck path of semilength $n$ is a simple lattice path from $(0,0)$ to $(n,n)$ that stays weakly above the diagonal $y=x$ (with vertical unit steps $U$ and horizontal unit steps $D$). On the other hand, a left-to-right maximum of a permutation $\pi$ is an element $\pi_{i}$ such that $\pi_{j}<\pi_{i}$ for every $j<i$.

The bijective map between $\mathsf{Av}_{n}(321)$ and Dyck paths of semilength $n$ is defined as follows: Given $\pi\in\mathsf{Av}_{n}(321)$, write $\pi=m_{1}w_{1}m_{2}w_{2}\cdots m_{s}w_{s}$, where $m_{1},\dots,m_{s}$ are the left-to-right maxima of $\pi$, and each $w_{i}$ is a subword of $\pi$. Let $|w_{i}|$ denote the length of $w_{i}$. Reading the decomposition of $\pi$ from left to right, we construct a path starting with $m_{1}$ $U$-steps, $|w_{1}|+1$ $D$-steps, and for every other subword $m_{i}w_{i}$, we add $m_{i}-m_{i-1}$ $U$-steps followed by $|w_{i}|+1$ $D$-steps. In short, identify the left-to-right maxima in the plot of $\pi$ and draw your path over them. For example, for $\pi=351264\in\mathsf{Av}_{6}(321)$ we get:

Note that $\pi=351264\not\in\mathscr{F}_{6}$.

Theorem 2.2**.**

The set $\mathscr{F}_{n}(321)$ is in bijection with the set of Dyck paths of semilength $n$ that avoid the subpath $UUDU$. By [6, Prop. 5] we then have

[TABLE]

This is sequence [7, A105633].

Proof.

Under the above bijection, an ascent $\pi_{i}<\pi_{i+1}$ in $\pi\in\mathsf{Av}_{n}(321)$ with $k=\pi_{i+1}-\pi_{i}$ generates the subpath $UDU^{k}$ in the corresponding Dyck path $P_{\pi}$, and if $\pi_{i}-1=\pi_{j}$ for some $j>i+1$, then $P_{\pi}$ must necessarily contain the subpath $UUDU^{k}$. Thus we have that $\pi$ avoids the pattern if and only if $P_{\pi}$ avoids $UUDU$. ∎

Indecomposable permutations

Let $\mathscr{F}_{n}^{\textsf{ind}}(\sigma)$ be the set of indecomposable Fishburn permutations that avoid the pattern $\sigma$, and let $I\!F_{n}(\sigma)$ denote the number of elements in $\mathscr{F}_{n}^{\textsf{ind}}(\sigma)$. Observe that for every $\sigma$ of size $\geq 3$, we have $I\!F_{1}(\sigma)=1$ and $I\!F_{2}(\sigma)=1$.

We start with a fundamental known lemma, see e.g. [3, Lem. 3.1].

Lemma 2.3**.**

If a pattern $\sigma$ is indecomposable, then the sequence $|\mathsf{Av}_{n}(\sigma)|$ is the invert transform of the sequence $|\mathsf{Av}_{n}^{\mathsf{ind}}(\sigma)|$. That is, if $A^{\sigma}(x)$ and $A_{I}^{\sigma}(x)$ are the corresponding generating functions, then

[TABLE]

In particular, since $231$ is indecomposable, these identities are also valid for Fishburn permutations. The sequence $(I\!F_{n})_{n\in\mathbb{N}}$ that enumerates indecomposable Fishburn permutations of size $n$ starts with

[TABLE]

Theorem 2.4**.**

For $n>1$, we have $I\!F_{n}(123)=2^{n-1}-(n-1)$.

Proof.

As discussed in the proof of Theorem 2.1, for $n>2$ the set $\mathscr{F}_{n}(123)$ consists of elements of the form $1\ominus\pi$ with $\pi\in\mathscr{F}_{n-1}(123)$, and elements of the form $\tau=(i-1)\cdots 1n|\pi_{R}$ for some $1<i\leq n$ and $\pi_{R}\in\mathscr{F}_{n-i}(123)$. This forces $\pi_{R}=\emptyset$ if $i=n$, and $\pi_{R}=(n-1)\cdots i$ if $i<n$. Thus the only decomposable elements of $\mathscr{F}_{n}(123)$ are the $n-1$ permutations

[TABLE]

In conclusion, $I\!F_{n}(123)=F_{n}(123)-(n-1)=2^{n-1}-(n-1)$. ∎

Theorem 2.5**.**

For $n>1$ and $\sigma\in\{132,213\}$, we have $I\!F_{n}(\sigma)=2^{n-2}$.

Proof.

From the proof of Theorem 2.1, we know that for $n>1$ every element of $\mathscr{F}_{n}(132)$ must be of the form $1\ominus\pi$ or $\pi\oplus 1$ with $\pi\in\mathscr{F}_{n-1}(132)$. Since $\pi\oplus 1$ is decomposable and $1\ominus\pi$ is indecomposable, we have

[TABLE]

We also know that $\mathscr{F}_{n}(213)$ consists of elements of the form $1\ominus\pi$ with $\pi\in\mathscr{F}_{n-1}(213)$, and elements of the form $\tau=(1\cdots(i-1))\oplus(1\ominus\pi_{R})$ for some $1<i\leq n$ (with $\pi_{R}=\emptyset$ when $i=n$). Thus

[TABLE]

In conclusion, if $\sigma\in\{132,213\}$, we have $I\!F_{n}(\sigma)=F_{n-1}(\sigma)=2^{n-2}$. ∎

Let $F^{\sigma}(x)$ and $I\!F^{\sigma}(x)$ be the generating functions associated with $(F_{n}(\sigma))_{n\in\mathbb{N}}$ and $(I\!F_{n}(\sigma))_{n\in\mathbb{N}}$, respectively.

Theorem 2.6**.**

For $\sigma\in\{231,312,321\}$, we have

[TABLE]

In particular, $I\!F_{n}(231)=C_{n-1}$ and $I\!F_{n}(312)=1$.

Proof.

This follows from (2.1), Theorem 2.1, and Lemma 2.3. ∎

Theorem 2.7**.**

The enumerating sequence $a_{n}=I\!F_{n}(321)$ satisfies the recurrence relation

[TABLE]

with $a_{1}=a_{2}=a_{3}=1$. This is sequence [7, A082582].

Proof.

We use the same Dyck path approach as in the proof of Theorem 2.2. Under this bijection, indecomposable permutations correspond to Dyck paths that do not touch the line $y=x$ except at the end points.

Let $\mathcal{A}_{n}$ be the set of Dyck paths corresponding to $\mathscr{F}_{n}^{\textsf{ind}}(321)$. We will prove that $a_{n}=|\mathcal{A}_{n}|$ satisfies the claimed recurrence relation. Clearly, for $n=1,2,3$, the only indecomposable Fishburn permutations are $1$, $21$, and $312$, which correspond to the Dyck paths $UD$, $U^{2}D^{2}$, and $U^{3}D^{3}$, respectively. Thus $a_{1}=a_{2}=a_{3}=1$.

Note that indecomposable permutations may not start with $1$ or end with $n$. Moreover, every element of $\pi\in\mathscr{F}_{n}^{\textsf{ind}}(321)$ must be of the form $m1\pi(3)\cdots\pi(n)$ with $m\geq 3$. Therefore, the elements of $\mathcal{A}_{n}$ have no peaks at the points $(0,1)$, $(0,2)$, or $(n-1,n)$, and for $n>3$ their first return to the line $y=x+1$ happens at a lattice point $(x,x+1)$ with $x\in[2,n-1]$.

Dyck paths in $\mathcal{A}_{n}$ having $(n-1,n)$ as their first return to $y=x+1$, are in one-to-one correspondence with the elements of $\mathcal{A}_{n-1}$ (just remove the first $U$ and the last $D$ of the longer path). Now, for $j\in\{2,\dots,n-2\}$, the set of paths $P\in\mathcal{A}_{n}$ having first return to $y=x+1$ at the point $(j,j+1)$ corresponds uniquely to the set of all pairs $(P_{j},P_{n-j})$ with $P_{j}\in\mathcal{A}_{j}$ and $P_{j}\in\mathcal{A}_{n-j}$. For example,

,.

This implies that there are $a_{j}a_{n-j}$ paths in $\mathcal{A}_{n}$ having the point $(j,j+1)$ as their first return to the line $y=x+1$. Finally, summing over $j$ gives the claimed identity. ∎

Here is a summary of our enumeration results for patterns of size 3:

3. Avoiding patterns of size 4

In this section, we discuss the enumeration of Fishburn permutations that avoid a pattern of size 4. There are at least 13 Wilf equivalence classes that we break down into three categories: 10 classes with a single pattern, 2 classes with (conjecturally) three patterns each, and a larger class with eight patterns enumerated by the Catalan numbers.

We will provide a proof for the enumeration of the class $\mathscr{F}_{n}(1342)$, but our main focus in this paper will be on the enumeration of the equivalence class given in Table 4.

For the remaining patterns we have the following conjectures.

Conjecture**.**

$\mathscr{F}_{n}(2413)\sim\mathscr{F}_{n}(2431)\sim\mathscr{F}_{n}(3241)$.

Conjecture**.**

$\mathscr{F}_{n}(3214)\sim\mathscr{F}_{n}(4132)\sim\mathscr{F}_{n}(4213)$.

Our first result of this section involves the binomial transform of the Catalan numbers, namely the sequence [7, A007317].

Theorem 3.1**.**

[TABLE]

Proof.

Let $\mathcal{A}_{n,k}$ be the set of all permutations $\pi\in S_{n}$ such that

• $\circ$

$\pi(k)=1$ and $\pi(1)>\pi(2)>\cdots>\pi(k-1)$,

• $\circ$

$\pi(k+1)\cdots\pi(n)\in\mathsf{Av}_{n-k}(231)$,

and let $\mathscr{A}_{n}=\bigcup_{k=1}^{n}\mathcal{A}_{n,k}$. Clearly,

[TABLE]

We will prove the theorem by showing that $\mathscr{A}_{n}=\mathscr{F}_{n}(1342)$.

First of all, since $\mathcal{A}_{n,k}\subset\mathsf{Av}_{n}(1342)$ and $\mathsf{Av}_{n-k}(231)=\mathscr{F}_{n-k}(231)$ for every $n$ and $k$, we have $\mathscr{A}_{n}\subset\mathscr{F}_{n}(1342)$.

Going in the other direction, let $\pi\in\mathscr{F}_{n}(1342)$ and let $k$ be such that $\pi(k)=1$. Thus $\pi$ is of the form $\pi=\pi(1)\cdots\pi(k-1)\,1\,\pi(k+1)\cdots\pi(n)$, which implies $\pi(k+1)\cdots\pi(n)\in\mathsf{Av}_{n-k}(231)$. Now, if there is a $j\in\{1,\dots,k-2\}$ such that

[TABLE]

then $\pi(j)-1=\pi(\ell)$ for some $\ell>j+1$, and the pattern $\pi(j)\pi(j+1)\pi(\ell)$ would violate the Fishburn condition. In other words, the entries left from $\pi(k)=1$ must form a decreasing sequence, which implies $\pi\in\mathcal{A}_{n,k}\subset\mathscr{A}_{n}$. Thus $\mathscr{F}_{n}(1342)\subset\mathscr{A}_{n}$, and we conclude that $\mathscr{A}_{n}=\mathscr{F}_{n}(1342)$. ∎

Catalan equivalence class

The remaining part of this section is devoted to prove that $|\mathscr{F}_{n}(\sigma)|=C_{n}$ for every $\sigma\in\{1234,1243,1324,1423,2134,2143,3124,3142\}$.

Theorem 3.2**.**

We have $\mathscr{F}_{n}(3142)=\mathscr{F}_{n}(231)$, hence $F_{n}(3142)=C_{n}$.

Proof.

Since 3142 contains the pattern 231, we have $\mathscr{F}_{n}(231)\subseteq\mathscr{F}_{n}(3142)$.

To prove the reverse inclusion, suppose there exists $\pi\in\mathscr{F}_{n}(3142)$ such that $\pi$ contains the pattern 231. Let $i<j<k$ be the positions of the most-left closest 231 pattern contained in $\pi$. By that we mean:

• $\circ$

$\pi(k)<\pi(i)<\pi(j)$,

• $\circ$

$\pi(i)$ is the most-left entry of $\pi$ involved in a 231 pattern,

• $\circ$

$\pi(j)$ is the first entry with $j>i$ such that $\pi(i)<\pi(j)$,

• $\circ$

$\pi(k)$ is the largest entry with $k>j$ such that $\pi(k)<\pi(i)$.

In other words, assume the plot of $\pi$ is of the form

where no elements of $\pi$ may occur in the shaded regions. It follows that, if $\ell$ is the position of $\pi(k)+1$, then $i\leq\ell<j$. But this is not possible since, $\pi(\ell)<\pi(\ell+1)$ violates the Fishburn condition, and $\pi(\ell)>\pi(\ell+1)$ implies $\pi(k)>\pi(\ell+1)$ which forces the existence of a 3142 pattern. In conclusion, no permutation $\pi\in\mathscr{F}_{n}(3142)$ is allowed to contain a 231 pattern. Therefore, $\mathscr{F}_{n}(3142)\subseteq\mathscr{F}_{n}(231)$ and we obtain the claimed equality. ∎

Theorem 3.3**.**

$\mathscr{F}_{n}(1234)\sim\mathscr{F}_{n}(1243)$* and $\mathscr{F}_{n}(2134)\sim\mathscr{F}_{n}(2143)$.*

Proof.

In order to prove both Wilf equivalence relations, we use a bijection

[TABLE]

given by West in [8], which we proceed to describe.

For $\pi\in S_{n}$ and $\sigma\in S_{k}$, $k<n$, let $B_{\pi}(\sigma)$ be the set of maximal values of all instance of the pattern $\sigma$ in the permutation $\pi$. For example, $B_{\pi}(\sigma)=\emptyset$ if $\pi$ avoids $\sigma$, and for $\pi=531968274$ we have $B_{\pi}(123)=\{4,7,8\}$.

For $\pi\in\mathsf{Av}_{n}(\tau\oplus 12)$, let $\ell$ be the number of elements in $B_{\pi}(\tau\oplus 1)$. If $\ell=0$, we define $\phi(\pi)=\pi$. If $\ell>0$, we let $i_{1}<\dots<i_{\ell}$ be the positions in $\pi$ of the elements of $B_{\pi}(\tau\oplus 1)$ and define

[TABLE]

Note that $\pi\in\mathsf{Av}_{n}(\tau\oplus 12)$ implies $\pi(i_{1})>\cdots>\pi(i_{\ell})$.

Let $b_{1}$ be the smallest element of $B_{\pi}(\tau\oplus 1)$ such that $\tilde{\pi}(1)\cdots\tilde{\pi}(i_{1}-1)b_{1}$ contains the pattern $\tau\oplus 1$. Define

[TABLE]

Iteratively, for $k=2,\dots,\ell$, we let $b_{k}$ be the smallest element of $B_{\pi}(\tau\oplus 1)\backslash\{b_{1},\dots,b_{k-1}\}$ such that $\tilde{\pi}(1)\cdots\tilde{\pi}(i_{k}-1)b_{k}$ contains the pattern $\tau\oplus 1$. We then define

[TABLE]

to complete the definition of $\tilde{\pi}=\phi(\pi)$.

For example, for \pi=53196{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bf 8}2{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}\bf 74} and $\tau=12$, we have $\ell=3$ and \tilde{\pi}=53196{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bf 7}2{\color[rgb]{1,.5,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,.5,0}\bf 48}.

It is easy to check that the map $\phi$ induces a bijection

[TABLE]

Indeed, if $\pi(i_{k})\in B_{\pi}(\tau\oplus 1)$ is such that $\pi(i_{k}-1)<\pi(i_{k})$, then $\tilde{\pi}(i_{k}-1)=\pi(i_{k}-1)$ and the pair $\tilde{\pi}(i_{k}-1),\tilde{\pi}(i_{k})$ does not create a pattern .

On the other hand, if $\pi(i_{1})>\dots>\pi(i_{k})$ is a maximal descent of elements from $B_{\pi}(\tau\oplus 1)$, and if $\pi(i_{j})-1>0$ (for $j\in\{1,\dots,k\}$) is not part of that descent, then $\pi(i_{j})-1$ must be left from $\pi(i_{1})$ and so any ascent in $\tilde{\pi}(i_{1})\cdots\tilde{\pi}(i_{k})$ cannot create the pattern .

Thus, if $\pi\in\mathsf{Av}_{n}(\tau\oplus 12)$ is Fishburn, so is $\tilde{\pi}=\phi(\pi)\in\mathsf{Av}_{n}(\tau\oplus 21)$. ∎

Theorem 3.4**.**

$\mathscr{F}_{n}(1423)\sim\mathscr{F}_{n}(1243)\sim\mathscr{F}_{n}(1234)\sim\mathscr{F}_{n}(1324)$.

Proof.

Let $\alpha:\mathscr{F}_{n}(1423)\to\mathscr{F}_{n}(1243)$ be the map defined through the following algorithm.

Algorithm $\alpha$: Let $\pi\in\mathscr{F}_{n}(1423)$ and set $\tilde{\pi}=\pi$.

1. Step 1:

If $\tilde{\pi}\not\in\mathsf{Av}_{n}(1243)$, let $i<j<k<\ell$ be the positions of the most-left 1243 pattern contained in $\tilde{\pi}$. Redefine $\tilde{\pi}$ by moving $\tilde{\pi}(k)$ to position $j$, shifting the entries at positions $j$ through $k-1$ one step to the right. 2. Step 2:

If $\tilde{\pi}\in\mathsf{Av}_{n}(1243)$, then return $\alpha(\pi)=\tilde{\pi}$; otherwise go to Step 1.

For example, for $\pi=2135476\in\mathscr{F}(1423)$, the above algorithm yields

[TABLE]

and so $\alpha(\pi)=2175346\in\mathscr{F}(1243)$.

Observe that the map $\alpha$ changes every 1243 pattern into a 1423 pattern. To see that it preserves the Fishburn condition, let $\pi\in\mathscr{F}_{n}(1423)$ be such that $\pi(i)$, $\pi(j)$, $\pi(k)$, $\pi(\ell)$ form a most-left 1243 pattern. Thus, at first, $\pi$ must be of the form

j$$k

where no elements of $\pi$ may occur in the shaded regions. In particular, we must have

[TABLE]

Hence the step of moving $\pi(k)$ to position $j$ does not create a new ascent, and therefore it cannot create a pattern . After one iteration, $\tilde{\pi}$ takes the form

j^{\prime}$$\ell^{\prime}

and if the most-left 1243 pattern $\tilde{\pi}(i)$, $\tilde{\pi}(j)$, $\tilde{\pi}(k)$, $\tilde{\pi}(\ell)$ contained in $\tilde{\pi}$ has its second entry at a position different from $j^{\prime}$, then $\tilde{\pi}$ must satisfy (3.1) and no will be created.

Otherwise, if $j=j^{\prime}$, then either $k=\ell^{\prime}$ or $\ell=\ell^{\prime}$. In the first case, we have $\tilde{\pi}(k-1)<\tilde{\pi}(k)$ and $\tilde{\pi}(k)<\tilde{\pi}(j-1)$, so moving $\tilde{\pi}(k)$ to position $j$ does not create a new ascent. On the other hand, if $\ell=\ell^{\prime}$, then $\tilde{\pi}(k)>\tilde{\pi}(j-1)$ but $\tilde{\pi}(j-1)-1$ must be left from $\tilde{\pi}(i)$. Therefore, also in this case, applying an iteration of $\alpha$ will preserve the Fishburn condition.

We conclude that, if $\pi$ is Fishburn, so is $\alpha(\pi)$.

The reverse map $\beta:\mathscr{F}_{n}(1243)\to\mathscr{F}_{n}(1423)$ is given by the following algorithm.

Algorithm $\beta$: Let $\tau\in\mathscr{F}_{n}(1243)$ and set $\tilde{\tau}=\tau$.

1. Step 1:

If $\tilde{\tau}\not\in\mathsf{Av}_{n}(1423)$, let $i<j<k<\ell$ be the positions of the most-right 1423 pattern contained in $\tilde{\tau}$. Redefine $\tilde{\tau}$ by moving $\tilde{\tau}(j)$ to position $k$, shifting the entries at positions $j+1$ through $k$ one step to the left. 2. Step 2:

If $\tilde{\tau}\in\mathsf{Av}_{n}(1423)$, then return $\beta(\tau)=\tilde{\tau}$; otherwise go to Step 1.

In conclusion, the map $\alpha$ gives a bijection $\mathscr{F}_{n}(1423)\to\mathscr{F}_{n}(1243)$.

With a similar argument, it can be verified that $\alpha$ also maps $\mathscr{F}_{n}(1324)\to\mathscr{F}_{n}(1234)$ bijectively. Finally, the equivalence $\mathscr{F}_{n}(1243)\sim\mathscr{F}_{n}(1234)$ was shown in Theorem 3.3. ∎

Theorem 3.5**.**

$\mathscr{F}_{n}(3142)\sim\mathscr{F}_{n}(3124)\sim\mathscr{F}_{n}(1324)$.

Proof.

We will define two maps

[TABLE]

through algorithms similar to the one introduced in the proof of Theorem 3.4.

Algorithm $\alpha_{1}$: Let $\pi\in\mathscr{F}_{n}(3142)$ and set $\tilde{\pi}=\pi$.

1. Step 1:

If $\tilde{\pi}\not\in\mathsf{Av}_{n}(3124)$, let $i<j<k<\ell$ be the positions of the most-left 3124 pattern contained in $\tilde{\pi}$. Redefine $\tilde{\pi}$ by moving $\tilde{\pi}(\ell)$ to position $k$, shifting the entries at positions $k$ through $\ell-1$ one step to the right. 2. Step 2:

If $\tilde{\pi}\in\mathsf{Av}_{n}(3124)$, then return $\alpha_{1}(\pi)=\tilde{\pi}$; otherwise go to Step 1.

As $\alpha$ in Theorem 3.4, the map $\alpha_{1}$ is reversible and preserves the Fishburn condition. For an illustration of the latter claim, here is a sketch of a permutation $\pi\in\mathscr{F}_{n}(3142)$ having a most-left 3124 pattern, together with the sketch of $\tilde{\pi}$ after one iteration of $\alpha_{1}$:

k$$\ell$$\pi$$k^{\prime}$$\ell^{\prime}$$\tilde{\pi}

where no elements of the permutation $\pi$ may occur in the shaded regions.

Since $\pi(k-1)<\pi(k)$ and $\pi(\ell-1)<\pi(\ell)$, the Fishburn condition of $\pi$ is preserved after the first iteration of $\alpha_{1}$. Further, if $\tilde{\pi}$ has a most-left 3124 pattern with third entry at position $k^{\prime}$, then we must have $\ell>\ell^{\prime}$ and $\tilde{\pi}(\ell)>\tilde{\pi}(i)$. If $\tilde{\pi}(\ell)<\tilde{\pi}(k^{\prime}-1)$, no new ascent can be created when moving $\tilde{\pi}(\ell)$ to position $k^{\prime}$. Otherwise, if $\tilde{\pi}(\ell)>\tilde{\pi}(k^{\prime}-1)$, then either both entries were part of an ascent in the original $\pi$ or every entry between $\tilde{\pi}(\ell^{\prime})$ and $\tilde{\pi}(\ell)$ must be smaller than $\tilde{\pi}(j)$. Since $\pi\in\mathsf{Av}_{n}(3142)$, the latter would imply that $\tilde{\pi}(k^{\prime}-1)-1$ is left from $\tilde{\pi}(j)$. In any case, no pattern will be created in the next iteration of $\alpha_{1}$.

Since any later iteration of $\alpha_{1}$ may essentially be reduced to one of the above cases, we conclude that $\alpha_{1}$ preserves the Fishburn condition.

Algorithm $\alpha_{2}$: Let $\pi\in\mathscr{F}_{n}(3124)$ and set $\tilde{\pi}=\pi$.

1. Step 1:

If $\tilde{\pi}\not\in\mathsf{Av}_{n}(1324)$, let $i<j<k<\ell$ be the positions of the most-left 1324 pattern contained in $\tilde{\pi}$. Redefine $\tilde{\pi}$ by moving $\tilde{\pi}(j)$ to position $i$, shifting the entries at positions $i$ through $j-1$ one step to the right. 2. Step 2:

If $\tilde{\pi}\in\mathsf{Av}_{n}(1324)$, then return $\alpha_{2}(\pi)=\tilde{\pi}$; otherwise go to Step 1.

This map is reversible and preserves the Fishburn condition. As before, we will illustrate the Fishburn property by sketching the plot of a permutation $\pi\in\mathscr{F}_{n}(3124)$ that contains a most-left 1324 pattern $\pi(i)$, $\pi(j)$, $\pi(k)$, $\pi(\ell)$, together with the sketch of the permutation $\tilde{\pi}$ obtained after one iteration of $\alpha_{2}$:

i$$j$$\pi$$i^{\prime}$$\tilde{\pi}

Since no elements of the permutation $\pi$ may occur in the shaded regions, we must have either $i=1$ or $\pi(i-1)>\pi(j)$. Consequently, moving $\pi(j)$ to position $i$ will not create a new ascent and the Fishburn condition will be preserved.

Similarly, if $\tilde{\pi}$ has a most-left 1324 pattern with first entry at a position different from $i^{\prime}$, or if $\tilde{\pi}(i)=\tilde{\pi}(i^{\prime})$ and $\tilde{\pi}(j)<\tilde{\pi}(i^{\prime}-1)$, then no new ascent will be created and the next $\tilde{\pi}$ will be Fishburn. It is not possible to have $\tilde{\pi}(i)=\tilde{\pi}(i^{\prime})$ and $\tilde{\pi}(j)>\tilde{\pi}(i^{\prime}-1)$.

In summary, $\alpha_{1}$ and $\alpha_{2}$ are both bijective maps. ∎

The following theorem completes the enumeration of the Catalan class (see Table 4).

Theorem 3.6**.**

$\mathscr{F}_{n}(3142)\sim\mathscr{F}_{n}(2143)$.

Proof.

Let $\gamma:\mathscr{F}_{n}(3142)\to\mathscr{F}_{n}(2143)$ be the map defined through the following algorithm.

Algorithm $\gamma$: Let $\pi\in\mathscr{F}_{n}(3142)$ and set $\tilde{\pi}=\pi$.

1. Step 1:

If $\tilde{\pi}\not\in\mathsf{Av}_{n}(2143)$, let $i<j<k$ be the positions of the most-left 213 pattern contained in $\tilde{\pi}$ such that $\tilde{\pi}(i)$, $\tilde{\pi}(j)$, $\tilde{\pi}(k)$, $\tilde{\pi}(\ell)$ form a 2143 pattern for some $\ell>k$. Let $\ell_{m}$ be the position of the smallest such $\tilde{\pi}(\ell)$, and let

[TABLE]

Redefine $\tilde{\pi}$ by replacing $\tilde{\pi}(\ell_{m})$ with $\tilde{\pi}(i)$, adding 1 to $\tilde{\pi}(q)$ for every $q\in Q$. 2. Step 2:

If $\tilde{\pi}\in\mathsf{Av}_{n}(2143)$, then return $\gamma(\pi)=\tilde{\pi}$; otherwise go to Step 1.

For example, if $\pi=4312576$, then $\gamma(\pi)=5412673$ (after 2 iterations, see Figure 3).

The map $\gamma$ is reversible. Moreover, observe that:

1. (a)

since $\tilde{\pi}(\ell_{m})$ is the smallest entry such that $\tilde{\pi}(i)<\tilde{\pi}(\ell_{m})<\tilde{\pi}(k)$, replacing $\tilde{\pi}(\ell_{m})$ with $\tilde{\pi}(i)$ (which is equivalent to moving the plot of $\tilde{\pi}(\ell_{m})$ down to height $\tilde{\pi}(i)$) will not create any new ascent at position $\ell_{m}$; 2. (b)

since $\tilde{\pi}(i)$ is chosen to be the first entry of a most-left 2143 pattern, $\tilde{\pi}(i)-1$ must be right from $\tilde{\pi}(i)$. Hence, replacing $\tilde{\pi}(i)$ by $\tilde{\pi}(i)+1$ cannot create a new pattern .

In conclusion, $\gamma$ preserves the Fishburn condition and gives the claimed bijection. ∎

4. Further remarks

In this paper, we have discussed the enumeration of Fishburn permutations that avoid a pattern of size 3 or a pattern of size 4. In Section 2, we offer the complete picture for patterns of size 3, including the enumeration of indecomposable permutations.

Regarding patterns of size 4, we have proved the Wilf equivalence of eight permutation families counted by the Catalan numbers. We have also shown that $\mathscr{F}_{n}(1342)$ is enumerated by the binomial transform of the Catalan numbers. In general, there seems to be 13 Wilf equivalence classes of permutations that avoid a pattern of size 4, some of which appear to be in bijection with certain pattern avoiding ascent sequences ([7, A202061, A202062]). At this point in time, we don’t know how the pattern avoidance of a Fishburn permutation is related to the pattern avoidance of an ascent sequence. It would be interesting to pursue this line of investigation.

Concerning indecomposable permutations, we leave the field open for future research. Note that Theorem 3.2 and Lemma 2.3 imply

[TABLE]

The study of other patterns is unexplored territory, and our preliminary data suggests the existence of 19 Wilf equivalence classes listed in Table 5.

We are particularly curious about the class $\mathscr{F}_{n}^{\textsf{ind}}(2413)$ as it appears (based on limited data) to be equinumerous with the set $\mathsf{Av}_{n-1}(2413,3412)$, cf. [7, A165546].