Pattern avoidance in nonnesting permutations

TL;DR

This paper investigates pattern avoidance in nonnesting permutations, providing enumeration formulas and generating functions for avoiding certain patterns, revealing connections to Catalan and Fibonacci numbers.

Contribution

It introduces the enumeration of nonnesting permutations avoiding multiple patterns, extending previous work on noncrossing permutations with new formulas and combinatorial insights.

Findings

Closed formulas for pattern-avoiding nonnesting permutations

Generating functions involving Catalan and Fibonacci numbers

Bijections and recurrences used in proofs

Abstract

Nonnesting permutations are permutations of the multiset $\{1,1,2,2,\dots,n,n\}$ that avoid subsequences of the form $abba$ for any $a\neq b$. These permutations have recently been studied in connection to noncrossing (also called quasi-Stirling) permutations, which are those that avoid subsequences of the form $abab$, and in turn generalize the well-known Stirling permutations. Inspired by the work by Archer et al. on pattern avoidance in noncrossing permutations, we consider the analogous problem in the nonnesting case. We enumerate nonnesting permutations that avoid each set of two or more patterns of length 3, as well as those that avoid some sets of patterns of length 4. We obtain closed formulas and generating functions, some of which involve unexpected appearances of the Catalan and Fibonacci numbers. Our proofs rely on decompositions, recurrences, and bijections.