Intersection of conjugate solvable subgroups in symmetric groups

TL;DR

This paper proves that for certain solvable subgroups within almost simple symmetric groups, there exist conjugates whose intersection is trivial, advancing understanding of subgroup intersections in group theory.

Contribution

It establishes the existence of conjugates of solvable subgroups in almost simple groups with trivial intersection, a novel result in the structure of symmetric groups.

Findings

Existence of conjugates with trivial intersection

Applicable to solvable subgroups in symmetric groups

Enhances understanding of subgroup intersection properties

Abstract

It is shown that for a solvable subgroup $G$ of an almost simple group $S$ which socle is isomorphic to $A_n$ $ (n\ge5)$ there are $x,y,z,t \in S$ such that $G \cap G^x \cap G^y \cap G^z \cap G^t =1.$

Equations

$$g:(i_1,\dots ,i_k)\mapsto (i_{1}g,\dots ,i_{k}g).$$

$$({\omega }_1,{\omega }_1,{\omega }_1,{\omega }_1,{\omega }_2);$$

$$({\omega }_1,{\omega }_1,{\omega }_1,{\omega }_2,{\omega }_1);$$

$$({\omega }_1,{\omega }_1,{\omega }_2,{\omega }_1,{\omega }_1);$$

$$({\omega }_1,{\omega }_2,{\omega }_1,{\omega }_1,{\omega }_1);$$

$$({\omega }_2,{\omega }_1,{\omega }_1,{\omega }_1,{\omega }_1)$$

$$({\omega }_1,{\omega }_1,{\omega }_1,{\omega }_2,{\omega }_3);$$

$$({\omega }_1,{\omega }_1,{\omega }_2,{\omega }_3,{\omega }_1);$$

$$({\omega }_1,{\omega }_2,{\omega }_3,{\omega }_1,{\omega }_1);$$

$$({\omega }_2,{\omega }_3,{\omega }_1,{\omega }_1,{\omega }_1);$$

$$({\omega }_1,{\omega }_1,{\omega }_2,{\omega }_1,{\omega }_3)$$

$$({\omega }_1,{\omega }_1,{\omega }_2,{\omega }_3,{\omega }_4);$$

$$({\omega }_1,{\omega }_2,{\omega }_3,{\omega }_4,{\omega }_1);$$

$$({\omega }_2,{\omega }_3,{\omega }_4,{\omega }_1,{\omega }_1);$$

$$({\omega }_2,{\omega }_3,{\omega }_1,{\omega }_4,{\omega }_1);$$

$$({\omega }_2,{\omega }_1,{\omega }_3,{\omega }_1,{\omega }_4)$$

$$((K_1\times \dots \times K_l)⋊L)\cap ((K_1\times \dots \times K_l)⋊L{)}^{(1,2,\dots ,n)}$$

$$((K_1\times \dots \times K_l)⋊L)\cap ((K_1\times \dots \times K_l)⋊L{)}^{(1,2,\dots ,n)}\cap ((K_1\times \dots \times K_l)⋊L{)}^{(1,2,\dots ,n{)}^2},$$

$$\{1,2,3\}\{4,5,6\}\dots \{n-2,n-1,n\};$$

$$\{2,3,4\}\{5,6,7\}\dots \{n-1,n,1\};$$

$$\{3,4,5\}\{6,7,8\}\dots \{n,1,2\}.$$

$$R=((K_1\times \dots \times K_l)⋊S_l)\cap ((K_1\times \dots \times K_l)⋊S_l{)}^{(1,2,\dots ,n)}\cap ((K_1\times \dots \times K_l)⋊S_l{)}^{(1,2,\dots ,n{)}^2}.$$

$$G\cap G^{(1,2,\dots ,n)}\cap G^{(1,2,\dots ,n{)}^2}\cap G^{(3,4)}.$$

$$G\cap G^{(1,2,\dots ,n)}\cap G^{(1,2,\dots ,n{)}^2}\cap G^{(3,4)}=1.$$

$$((K_1\times \dots \times K_l)⋊L)\cap ((K_1\times \dots \times K_l)⋊L{)}^{(1,2,\dots ,n{)}^2}$$

$$((K_1\times \dots \times K_l)⋊L)\cap ((K_1\times \dots$$

$$\cap (((K_1\times \dots$$

$$R=((K_1\times \dots \times K_l)⋊S_l)\cap ((K_1\times \dots$$

$$\cap (((K_1\times \dots \times K_l)⋊S_l)$$

$$\{1,2,3,4\}\{5,6,7,8\}\dots \{n-3,n-2,n-1,n\};$$

$$\{2,3,4,5\}\{6,7,8,9\}\dots \{n-2,n-1,n,1\};$$

$$\{3,4,5,6\}\{7,8,9,10\}\dots \{n-1,n,1,2\};$$

$$\{4,5,6,7\}\{8,9,10,11\}\dots \{n,1,2,3\}.$$

$$R=⟨(1,5,\dots ,n-3)(2,6,\dots ,n-2)(3,7,\dots ,n-1)(4,8,\dots n)⟩.$$

$$G\cap G^{(1,2,\dots ,n)}\cap G^{(1,2,\dots ,n{)}^2}\cap G^{(1,2,\dots ,n{)}^3}\cap G^{(4,5)}.$$

$$G\cap G^{(1,2,\dots ,n)}\cap G^{(1,2,\dots ,n{)}^2}\cap G^{(1,2,\dots ,n{)}^3}\cap G^{(4,5)}=1.$$

$$(G,Ga,G{a}^2,G{a}^3,G(4,5));$$

$$(G,Ga,G{a}^2,G{a}^3,G(3,6));$$

$$(G,Ga,G{a}^2,G{a}^3,G(2,7));$$

$$(G,Ga,G{a}^2,G{a}^3,G(1,8));$$

$$(G,Ga,G{a}^2,G{a}^3,G(2,5)),$$

$$G\cap G^a\cap G^{a^2}\cap G^{a^3}=R,$$

Full text

Kay words: symmetric group, solvable group, almost simple group.

Introduction

Assume that a finite group $S$ acts on a set $\Omega.$ An element $x\in\Omega$ is called a $S$-regular point if $|xS|=|S|$, i.e. if the stabilizer of $x$ is trivial. Define the action of the group $S$ on $\Omega^{k}$ by the rule

[TABLE]

If $S$ acts faithfully and transitively on $\Omega$, then the minimal number $k$ such that the set $\Omega^{k}$ contains a $S$-regular point is called the base size of $S$ and is denoted by $b(S).$ For a positive integer $m$ the number of $S$-regular orbits on $\Omega^{m}$ is denoted by $Reg(S,m)$ (this number equals 0 if $m<b(S)$). If $G$ is a subgroup of $S$ and $S$ acts by the right multiplication on the set $\Omega$ of the right cosets of $G$ then $S/G_{S}$ acts faithfully and transitively on $\Omega.$ (Here $G_{S}=\cap_{g\in S}G^{g}$ is the core of $G$.) In this case, we denote $b(S/G_{S})$ and $Reg(S/G_{S},m)$ by $b_{G}(S)$ and $Reg_{G}(S,m)$ respectively. Thus $b_{G}(S)$ is the minimal number $k$ such that there exist

$x_{1},\ldots,x_{k}$ $\in S$ for which $G^{x_{1}}\cap\ldots\cap H^{x_{k}}=G_{S}.$

Consider Problem 17.41 from “Kourovka notebook”[1]:

** Problem 1****.**

Let $G$ be a solvable subgroup of a finite group $S$ and $S$ does not contain nontrivial normal solvable subgroups. Are there always exist five subgroups conjugated with $G$ such that their intersection is trivial?

The problem is reduced to the case when $S$ is almost simple in [2]. Specifically, it is proved that if for each almost simple group $S$ and solvable subgroup $G$ of $S$ condition $Reg_{G}(S,5)\geq 5$ holds then for each finite nonsolvable group $S$ with trivial solvable radical and solvable subgroup $G$ of $S$ condition $Reg_{G}(S,5)\geq 5$ holds.

We have proved the following

** Theorem 1****.**

Let $G$ be a solvable subgroup of an almost simple group $S$ whose socle is isomorphic to $A_{n}$, $n\geq 5.$ Then $Reg_{G}(S,5)\geq 5.$ In particular $b_{G}(S)\leq\leavevmode\nobreak\ 5.$

The proof contains calculations in GAP system. Texts of programs can be found here:

https://goo.gl/rm9l3M

1 Preliminary results

Our totation is standard.

By $A\rtimes B$ we denote the semidirect product of groups $A$ and $B$, where $B$ acts on $A$, by $A\wr B$ the permutation wreath product of groups $A$ and $B$, where $B\leq Sym_{n}$.

If group $G$ acts on the a $\Omega=\Omega_{1}\sqcup\Omega_{2}$ and $\Omega_{i}$ is an invariant subset, then denote projection of $G$ on $\Omega_{i}$ by $G|_{\Omega_{i}}$.

** Lemma 1****.**

[2, Lemma 13] * Let $G$ be a finite group with trivial solvable radical, $M$ be a solvable subgroup of $S_{n}$, and $k$ be a natural number such that for every solvable subgroup $T$ of $G$ the following inequality holds: $Reg_{T}(G,k)\geq 5$. Then the solvable radical of $G\wr M$ is trivial and for every solvable subgroup $S$ of $G\wr M$ the following inequality holds: ${Reg_{S}(G\wr M,k)\geq 5.}$*

** Lemma 2****.**

[5, Theorem 1] * Let $H$ be a maximal primitive subgroup of $G=S_{n}$ or $A_{n}$; and $H\neq A_{n}.$ Then $b_{H}(G)\leq 3$ for $n\geq 11.$*

** Lemma 3****.**

Let $H<G$ and $b_{H}(G)\leq 4$. Then $Reg_{H}(G,5)\geq 5.$

Proof.

If $b_{G}(S)\leq 2$ and $(\omega_{1},\omega_{2}$) is a regular point of $\Omega^{2}$ (and also $\omega_{1}\neq\omega_{2}$), then it is easy to see that points

[TABLE]

are regular in distinct $S$-orbits.

If $b_{G}(S)=3$ and $(\omega_{1},\omega_{2},\omega_{3}$) is a regular point of $\Omega^{3}$(and also $\omega_{i}\neq\omega_{j}$ for $i\neq j,$ otherwise $b_{G}(S)<3$), then it is easy to see that points

[TABLE]

are regular in distinct $S$-orbits.

If $b_{G}(S)=4$ and $(\omega_{1},\omega_{2},\omega_{3},\omega_{4}$) is a regular point of $\Omega^{4}$(and also $\omega_{i}\neq\omega_{j}$ for $i\neq j,$ otherwise $b_{G}(S)<4$), then it is easy to see that points

[TABLE]

are regular in distinct $S$-orbits. ∎

** Lemma 4****.**

Let $G$ be a solvable primitive subgroup of $S$, where $S$ is isomorphic to $S_{n}$ or $A_{n}$, $n\geq 5$. Then $b_{G}(S_{n})\leq 3.$ In particular $Reg_{G}(S_{n},5)\geq 5.$

Proof.

The group $S_{n}$ is known to possess a non-trivial solvable primitive subgroup if and only if $n=p^{k}$ for a prime $p$ and an integer $k$ [7, §4, Theorem 9]. The proof is divided into 3 cases:

Case 1 ($n\geq 11$). Subgroup $G$ lies in a maximal primitive subgroup $M$ of $S_{n}$. Then by Lemma 2 we have $b_{M}(S_{n})\leq 3$, thus $b_{G}(S_{n})\leq 3.$

Case 2 ($n=p$; $p=5$, $p=7$). In view of the known structure of maximal primitive solvable subgroups in $S_{n}$ [7, §4, Theorem 10] in this case $G$ is isomorphic to $\mathbb{Z}_{p}\rtimes\mathbb{Z}_{p-1}$. Up to conjugation $G$ is a semidirect product $A\rtimes B$. Here $A=\langle(1,2,\ldots,p)\rangle$, and $B$ is the stabilizer of point 1 in $G$. It is easy to verify by hand or using GAP, that in this case $G\cap G^{(1,2)}\cap G^{(1,3)}=1.$ Furthermore, $B$ is generated by an odd permutation $b$, i.e. permutations $b(1,2)$ and $b(1,3)$ are even, thus $G\cap G^{b(1,2)}\cap G^{b(1,3)}=1.$ So, we have $b_{G}(S)\leq 3.$

Case 3 ($n=8$, $n=9$). If $n=9$ then $G$ is isomorphic to $\mathbb{Z}_{3}^{2}\rtimes GL_{2}(3).$ If $n=8$ then $G$ is isomorphic to a subgroup of $\mathbb{Z}_{2}^{3}\rtimes GL_{3}(2)$. These two cases can be verified directly by counting orbits using GAP. Note that group $G$ contains an odd permutation, so it suffices to prove that $b_{G}(S_{n})\leq 3.$ ∎

** Lemma 5****.**

Let $G$ be a semiregular subgroup of $S_{n},n\geq 5$. Then there is $x\in S_{n}$ such that $G\cap G^{x}=1.$

Proof.

It suffices to prove the lemma for the case when the group $G$ is regular. Indeed, let $G$ be non-regularly and $\{1,2\ldots,n\}$ is the union of orbits $\Omega_{1},\ldots,\Omega_{k}$. Then $G$ acts regular on each orbit, so $G\leq S(\Omega_{1})\times\ldots\times S(\Omega_{k})$, where $S(\Omega_{i})$ is the permutation group of $\Omega_{i}$. Let $G_{i}$ be the projection of $G$ on $S(\Omega_{i})$. Since $G_{i}$ is regular, there are $x_{i}\in S(\Omega_{i})$ such that $G_{i}\cap G_{i}^{x_{i}}=1.$ Thus $G\cap G^{x_{1}\cdot\ldots\cdot x_{k}}=1.$

Let $G$ be regular. If $G$ is elementary abelian, then by [8, Theorem 1] we obtain the required. Consider the case when $G$ is not elementary abelian.

We show that if $g\in G$ and $g^{-1}\cdot g^{(1,2)}\neq 1$, then $g^{-1}\cdot g^{(1,2)}$ is not contained in $G$. Consider possibilities for the structure of the element $g$. Since $G$ is regular, every non-trivial element of $G$ has no fixed points. There are three cases:

**1. ** Points $1$ and $2$ are contained in distinct independent cycles, and there are more that two independent cycles, $g=(1,i_{1},\ldots,i_{k-1},i_{k})(2,j_{1},\ldots,j_{l})(...).$ Then $g^{-1}\cdot g^{(1,2)}$ fix all points which are not contained in the same cycle as $1$ or $2$, i.e. $g^{-1}\cdot g^{(1,2)}\notin G$.

2. Let $g=(1,i_{1}\ldots,i_{k-1},i_{k})(2,j_{1}\ldots,j_{l}).$ Since $n\geq 5$, the length of at least one of these cycles is greater than two. Without loss of generality, we can assume that it is the first. Then $g^{-1}\cdot g^{(1,2)}$ fixes $i_{k}$, i.e. $g^{-1}\cdot g^{(1,2)}\notin G$.

3. Let $g=(1,i_{1}\ldots i_{k},2,j_{1}\ldots,j_{l-1},j_{l})$. Since $n\geq 5$, we have $k\geq 2$ or $l\geq 2$. Without loss of generality, we can assume that $l\geq 2$. Then $g^{-1}\cdot g^{(1,2)}$ fixes $j_{l}$, i.e. $g^{-1}\cdot g^{(1,2)}\notin G$.

Therefore, $G\cap G^{(1,2)}\subseteq\{g\in G\mid g^{(1,2)}=g\}$. Since $G$ is regular, every non-trivial element of $G$ do not have fixed points. Thus, if $g^{(1,2)}=g$ then $\{1,2\}$ is an orbit of $\langle g\rangle$ and $g^{2}$ fixes points $1$ and $2$, i.e. $g^{2}=1.$ It means that $g=(1,2)(i_{1},i_{2})\ldots(i_{k-1},i_{k})$, $n$ is even and $|G\cap G^{(1,2)}|=2.$ In particular, if $n$ odd then $G\cap G^{(1,2)}=1$.

If $n$ is even then there is an element $g_{1}$ of order greater than two in $G$ because $G$ is not elementary abelian. Up to conjugation in $S_{n}$ we can assume that $g_{1}$ maps the point 1 to point 2, then, due to the regularity of $G$, $g_{1}$ is the only element which maps 1 to 2. Thus $G\cap G^{(1,2)}=1$, because $(1,2)(i_{1},i_{2})\ldots(i_{k-1},i_{k})$ is not contained in $G$. ∎

2 Case of transitive solvable subgroup

** Lemma 6****.**

Let $G$ be a solvable transitive subgroup of $S$, where $S$ is isomorphic to $S_{n}$ or $A_{n}$, $n\geq 5$. Then $Reg_{G}(S,5)\geq 5.$

Proof.

If $G$ is primitive then the lemma follows from Lemma 4. Let $G$ be imprimitive. Then $G$ is contained in $S_{k}\wr S_{l}$, here $n=k\cdot l,$ and $\{1,2,\ldots,n\}$ is partitioned into blocks $\Omega_{1}\cup\ldots\cup\Omega_{l}$. We denote $G_{i}=\{g\in G\mid\Omega_{i}g=\Omega_{i}\},$ $K_{i}=G_{i}|_{\Omega_{i}}$. Let $L$ be the image of $G$ in $S_{l}$. Then $G\leq(K_{1}\times\ldots\times K_{l})\rtimes L$ and group $(K_{1}\times\ldots\times K_{l})\rtimes L$ is solvable. Thus we may assume that $G=(K_{1}\times\ldots\times K_{l})\rtimes L.$

Case $k\geq 5.$

If $S=S_{n}$ then by induction for $K_{i}$ we have $Reg_{K_{i}}(S_{k},5)\geq 5,$ so by Lemma 1 we get $Reg_{G}(S_{k}\wr L,5)\geq 5$, whence $Reg_{G}(S_{n},5)\geq 5$.

Assume that $S=A_{n}.$ If $K_{i}\leq A_{k}$, then $G\leq A_{k}\wr L$ so, by Lemma 1, $Reg_{G}(A_{n},5)\geq 5.$ If $K_{i}$ is not contained in $A_{k}$, then there exist an odd permutation in $K_{i}$ and hence in $G.$ In this case $Reg_{G}(A_{n},5)\geq 5,$ by the fact that $Reg_{G}(S_{n},5)\geq 5$ and $G$ is normalized by an odd permutation.

Now assume that $k$ equals $2$, $3$, or $4$, and $S$ equals $S_{n}.$ Since $S_{n}$ is solvable for $n\leq 4$, we may assume that $G=(K_{1}\times\ldots\times K_{l})\rtimes L$, where each $K_{i}$ is isomorphic to $S_{k}.$ In particular, there is an odd permutation in group $S_{n}$ normalizing $G$. Hence we may assume that $S=S_{n}.$ Moreover, up to conjugation in $S_{n}$, $K_{i}$ acts on $\{k\cdot(l-1)+1,\ldots,k\cdot l\}.$

Case $k=2$. Clearly, the intersection $((K_{1}\times\ldots\times K_{l})\rtimes L)\cap((K_{1}\times\ldots\times K_{l})\rtimes L)^{(1,2,\ldots,n)}$ stabilizes the partitions $\{1,2\}\{3,4\}\ldots\{n-1,n\}$ and $\{2,3\}\{4,5\}\ldots\{n,1\}$. If an element $g$ from the intersection stabilizes a point (Suppose, for definiteness, that $g$ stabilizes point 1), then $g$ fixes point 2, because it is contained in the same block in the first partition. Thus, because the element $g$ fixes point 2, it also fixes point 3, due to the fact that 2 and 3 are contained in the same block in the second partition, etc. So if $g$ fixes the first element in a block of the first partition then it also stabilizes the second point of the block and the first point of the next block of the same partition. Thus $g$ stabilizes all points, hence $g=1,$ i.e. the intersection

[TABLE]

is semiregular and by Lemma 5 we have $b_{G}(S_{n})\leq 4$. So $Reg_{G}(S_{n},5)\geq 5$ by Lemma 3.

Case $k=3$. Consider the group

[TABLE]

which stabilizes the partitions

[TABLE]

If an element $g$ from this group stabilizes a point (for a definiteness, let $g$ stabilizes the point 1) then the point 2 can be moved under the action of $g$ only to 2 or 3, because of the first partition. However, points $n$, 1 and 2 compose a block in the third partition, therefore 2 is fixed by $g$, so the point 3 is also fixed by $g$. Since points 2, 3 and 4 compose a block in the second partition, the element $g$ has to stabilize the point 4. Thus, repeating the preceding argument, we find that element $g$ stabilizes all the points, i.e $g=1,$ which implies that the considered group is semiregular.

Let us find the number of orbits of the group

[TABLE]

Note that the points 1, 2 and 3 lie in distinct orbits. Indeed, let $g\in R$ maps 1 to 2. In this case, points of the first block of the first partition stay in this block, i.e. $g$ maps 3 to 3, or to 1. In the first case 2 moves to 1, because $g$ stabilizes the first partition, but then $g$ does not stabilize the second partition. In the second case $g$ does not stabilize the third partition. Similar arguments show that $g$ can not map 1 to 3. Due to the symmetry of entries 1, 2 and 3 in the above considerations, these points lie in distinct orbits. Thus the number of orbits $\geq 3$, then $|R|\leq n/3.$ Note that the element $(1,4,\ldots,n-2)(2,5,\ldots,n-1)(3,6,\ldots,n)$ of order $n/3$ stabilizer all three partitions, so it lies in $R$, from which we obtain $|R|=n/3,$ $R=\langle(1,4,\ldots,n-2)(2,5,\ldots,n-1)(3,6,\ldots,n)\rangle$.

We have $G\cap G^{(1,2,\ldots,n)}\cap G^{(1,2,\ldots,n)^{2}}\leq R.$ Consider the group

[TABLE]

It have to stabilize the partition $\{1,2,4\}\{3,5,6\}\ldots\{n-2,n-1,n\}$, which, as easy to see, is not stabilized by any non-trivial element of $R$, thus

[TABLE]

Therefore, $b_{G}(S_{n})\leq 4,$ and by Lemma 3 we obtain that $Reg_{G}(S_{n},5)\geq 5.$

Case $k=4$. Note that the group

[TABLE]

stabilizes the partition $\{1,2\}\{3,4\},\ldots,\{n-1,n\}$, which yields by the first case that the group

[TABLE]

is semiregular.

Let us find the number of orbits of the group

[TABLE]

which stabilizes the partitions

[TABLE]

Arguments similar to those of the preceding case ($k=3$) shows that

[TABLE]

We have $G\cap G^{(1,2,\ldots,n)}\cap G^{(1,2,\ldots,n)^{2}}\cap G^{(1,2,\ldots,n)^{3}}\leq R.$ Consider the group

[TABLE]

It have to stabilize the partition $\{1,2,3,5\}\{4,6,7,8\}\ldots\{n-3,n-2,n-1,n\}$, which is not stabilized by the group $R$, and we obtain that

[TABLE]

Consider the points

[TABLE]

where $a={(1,2,\ldots,n)}$. Arguments similar to those given above, show that all these points are $G$-regular. We will show that they all lie in different orbits.

Assume that $(G,Ga,G{a^{2}},G{a^{3}},G{(4,5)})$ and $(G,Ga,G{a^{2}},G{a^{3}},G{(3,6)})$ lie in the same orbit, i.e. there is an element $g\in S_{n}$, which maps $(G,Ga,G{a^{2}},G{a^{3}},G{(4,5)})$ to $(G,Ga,G{a^{2}},G{a^{3}},G{(3,6)})$, then $g,g^{a^{-1}},g^{a^{-2}},g^{a^{-3}},(4,5)g(3,6)\in G$. It is clear that a trivial permutation does not map any of the points (1) – (5) to another, i.e., we can assume that $g\neq 1.$ Then the element $g$ lies in the group

[TABLE]

therefore $g=r^{m}$, where

[TABLE]

The element $(4,5)g(3,6)$ maps 1 to $i_{m}$, and $4$ to $i_{m+1}$, and therefore does not stabilize the partition $\{1,2,3,4\}\{5,6,7,8\}\ldots\{n-3,n-2,n-1,n\},$ i.e. can not lie in $G$, so (1) and (2) lie in distinct orbits. The fact that (1) and (3), (1) and (5), (2) and (4), (2) and (5), (3) and (4), (3) and (5 ), (4) and (5) does not lie in the same orbit can be proved by similar arguments.

Assume that $(G,Ga,G{a^{2}},G{a^{3}},G{(4,5)})$ and $(G,Ga,G{a^{2}},G{a^{3}},G{(1,8)})$ lie in the same orbit, i.e. there is an element $g\in S_{n}$, which maps $(G,Ga,G{a^{2}},G{a^{3}},G{(4,5)})$ to $(G,Ga,G{a^{2}},G{a^{3}},G{(1,8)})$. As above we obtain $g=r^{m}.$ The element $(4,5)g(1,8)$ maps 1 to $i_{m}$. If $i_{m+1}\neq 1,$ then, as above, $(4,5)g(1,8)$ maps 4 to $i_{m+1}$ and $(4,5)g(1,8)$ can not lie in $G$. Let $i_{m+1}=1$, then $(4,5)g(1,8)$ maps 4 to 8. Then, because the element $(4,5)g(1,8)$ lies in $G$ and stabilize the corresponding partition, it is necessary that $i_{m}=5$. Therefore we obtain that $g=r=r^{-1},$ and it is a contradiction for all cases except the case $G\leq S_{4}\wr S_{2}$, for which the statement of the Lemma is known. The fact that (2) and (3) does not lie in the same orbit can be proved by similar arguments.

Therefore, $Reg_{G}(S_{n},5)\geq 5$, as wanted. ∎

3 Proof of the main theorem

** Lemma 7****.**

Let $G$ be a solvable subgroup of $S$, where $S$ is isomorphic to $S_{n}$ or $A_{n}$, $n\geq 5$. Then $Reg_{G}(S,5)\geq 5.$

Proof.

If $G$ is transitive then the lemma follows from Lemma 6. Let $G$ be intransitive, i.e. $G\leq S_{k_{1}}\times\ldots\times S_{k_{l}}.$ If $k_{i}\geq 5$ for all $i$ then by Lemma 6 there are $x_{i},y_{i},z_{i},t_{i}$, such that $G_{i}\cap G_{i}^{x_{i}}\cap G_{i}^{y_{i}}\cap G_{i}^{z_{i}}\cap G_{i}^{t_{i}}=1$, here $G_{i}$ is the projection of the group $G$ on $S_{i},$ hence

[TABLE]

thus $Reg_{G}(S,5)\geq 5.$

Further proof is by induction on the number of orbits. The base of induction is the cases when there are only the orbits of sizes 2, 3 and 4. It suffices to consider the cases $S_{2}\times S_{3}$, $S_{2}\times S_{4}$, $S_{3}\times S_{4}$, $S_{2}\times S_{2}\times S_{2}$, $S_{3}\times S_{3}$, $S_{4}\times S_{4}$. The validity of the statement $Reg_{G}(S,5)\geq 5$ for these groups can be verified directly with the help of GAP.

Assume that $G\leq S_{k}\times S_{n-k}$, $k=2,3,4;n-k\geq 5.$ Now $S_{k}$, up to conjugation in $S_{n}$, acts on $\{1,\ldots,k\}$, while $S_{n-k}$ acts on $\{k+1,\ldots,n\}.$ Let $G_{1}$ be the projection of $G$ on $S_{n-k}$. As earlier we may assume that the projection of $G$ on $S_{k}$ equals $S_{k},$ since $S_{k}$ is solvable for $k\leq 4$. By induction, there exist $x,y,z,t\in S_{n-k}$ such that

[TABLE]

Denote $K=G_{1}^{x}\cap G_{1}^{y}\cap G_{1}^{z}\cap G_{1}^{t}$.

Case $k=2$. Consider the intersection $S_{2}\times G_{1}\cap(S_{2}\times K)^{(2,3)}$, and let $g$ lies in the intersection. Since $g$ lies in $S_{2}\times G_{1}$, it cannot map $3$ to $1$ or $2$. On the other hand, as an element of $(S_{2}\times K)^{(2,3)}$, $g$ can map $3$ only to $1$. So $g$ fixes $3$. Thus it fixes $1$ and hence it fixes $2$, i.e. $g\in K$. We obtain that $g$ is contained in $G_{1}\cap K=1.$

Consider elements $x_{1},y_{1},z_{1},t_{1}\in S_{n-k}$ such that $G_{1}\cap G_{1}^{x_{1}}\cap G_{1}^{y_{1}}\cap G_{1}^{z_{1}}\cap G_{1}^{t_{1}}=1$ and points $(G_{1},G_{1}x,G_{1}y,G_{1}z,G_{1}t)$ and $(G_{1},G_{1}x_{1},G_{1}y_{1},G_{1}z_{1},G_{1}t_{1})$ lie in distinct $S_{n-k}$-orbits. Then, as we prove above,

[TABLE]

and points

[TABLE]

lie in the same $S_{n}$-orbit if and only if there is $g\in S_{n}$ such that

[TABLE]

Suppose that such $g$ exists. Then since $x,y,z,t,x_{1},y_{1},z_{1},t_{1}\in S_{n-k}$, we obtain that

[TABLE]

As shown above, in this case $g\in S_{n-3}$, in particular $g$ map $(G_{1},G_{1}x,G_{1}y,G_{1}z,G_{t})$ to $(G_{1},G_{1}x_{1},G_{1}y_{1},G_{1}z_{1},G_{1}t_{1})$, but this is impossible, since the points are from distinct orbits. Hence $Reg_{G}(S_{n},5)\geq Reg_{G_{1}}(S_{n-k},5)\geq 5.$

Case $k=3$. Consider the group

[TABLE]

Let $g$ be an element of this group. Then $g$ stabilizes the point 3, because $G$ lies in $S_{3}\times G_{1}\cap(S_{3}\times G_{1}^{x})^{(1,4)}\cap(S_{3}\times G_{1}^{y})^{(1,4)(2,5)}$. Since $g\in S_{3}\times G_{1}$, $g$ can map 2 only to 2 or 1, but $g\in(S_{3}\times G_{1}^{x})^{(1,4)}$ so 2 can not be moved to 1, hence $g$ stabilizes 2, thus $g$ also stabilizes 1 and 4. Now since $g\in(S_{3}\times G_{1}^{y})^{(1,4)(2,5)}$, the element $g$ stabilizes 5, and thus lies in $K$. Therefore $g=1.$

Consider the elements $x_{1},y_{1},z_{1},t_{1}\in S_{n-k}$ such that $G_{1}\cap G_{1}^{x_{1}}\cap G_{1}^{y_{1}}\cap G_{1}^{z_{1}}\cap G_{1}^{t_{1}}=1$ and points $(G_{1},G_{1}x,G_{1}y,G_{1}z,G_{1}t)$ $(G_{1},G_{1}x_{1},G_{1}y_{1},G_{1}z_{1},G_{1}t_{1})$ lie in distinct $S_{n-k}$-orbits. Then, by what was proved above we obtain

[TABLE]

and points

[TABLE]

lie in the same $S_{n}$-orbit if and only if there is $g\in S_{n}$ such that

[TABLE]

Suppose that such $g$ exists then we obtain

[TABLE]

since $x,y,z,t,x_{1},y_{1},z_{1},t_{1}\in S_{n-k}$. As it is shown above, in this case $g\in S_{n-5}$, then $g$ maps $(G_{1},G_{1}x,G_{1}y,G_{1}z,G_{1}t)$ to $(G_{1},G_{1}x_{1},G_{1}y_{1},G_{1}z_{1},G_{1}t_{1})$, which contradicts the assumption, since the points are from distinct orbits. Thus $Reg_{G}(S_{n},5)\geq Reg_{G_{1}}(S_{n-k},5)\geq 5.$

Case $k=4$. Consider the group

[TABLE]

Let $g$ be an element of this group. As easy to see, $g$ stabilize the point 4. Since $g\in S_{4}\times G_{1}\cap(S_{4}\times G_{1}^{y})^{(1,5)(2,6)}$, it stabilize the point 3. Hence, because $g\in S_{4}\times G_{1}\cap(S_{4}\times G_{1}^{x})^{(1,5)}$, we obtain that 2 and 5 also fixed by $g$. Thus $g$ also fixes 1 and 6 since $g\in S_{4}\times G_{1}\cap(S_{4}\times G_{1}^{y})^{(1,5)(2,6)}$. Therefore $g$ also stabilizes 7. Thus $g\in K$, i.e. $g=1.$

Consider the elements $x_{1},y_{1},z_{1},t_{1}\in S_{n-k}$ such that $G_{1}\cap G_{1}^{x_{1}}\cap G_{1}^{y_{1}}\cap G_{1}^{z_{1}}\cap G_{1}^{t_{1}}=1$ and points $(G_{1},G_{1}x,G_{1}y,G_{1}z,G_{1}t)$ and $(G_{1},G_{1}x_{1},G_{1}y_{1},G_{1}z_{1},G_{1}t_{1})$ lie in distinct $S_{n-k}$-orbits. Then, by what was proved above, we obtain that

[TABLE]

and points

[TABLE]

lie in the same $S_{n}$-orbit if and only if there is $g\in S_{n}$ such that

[TABLE]

Suppose that such $g$ exists then we obtain that

[TABLE]

since $x,y,z,t,x_{1},y_{1},z_{1},t_{1}\in S_{n-k}$. As it is shown above, in this case $g\in S_{n-7}$, then $g$ maps $(G_{1},G_{1}x,G_{1}y,G_{1}z,G_{1}t)$ to $(G_{1},G_{1}x_{1},G_{1}y_{1},G_{1}z_{1},G_{1}t_{1})$, which contradicts the assumption since the points are from distinct orbits. Thus $Reg_{G}(S_{n},5)\geq Reg_{G_{1}}(S_{n-k},5)\geq 5.$

Note that in these cases the group $G$ is normalized by $(1,2)$, from this property and the fact that $Reg_{G}(S_{n},5)\geq 5$ we obtain that $Reg_{G}(A_{n},5)\geq 5$.

∎

The main theorem for the case of an almost simple group $S$ with socle isomorphic to $A_{n}$, $n\geq 5$, $n\neq 6$ follows from Lemma 7. The main theorem for the case of an almost simple group $S$ with socle isomorphic to $A_{6}\cong PSL_{2}(9)$ follows from [4, Theorem 1.1], if subgroup $G$ does not lie in Aschbacher class $C_{1}$,and from [3, Lemma 8], if subgroup $G$ lies in Aschbacher class $C_{1}$.