Commensurating HNN-extensions: non-positive curvature and biautomaticity

TL;DR

This paper investigates the properties of certain groups, showing that the commensurator of quasiconvex abelian subgroups in biautomatic groups is limited, and constructs examples of CAT(0) groups that are not biautomatic, addressing key questions in geometric group theory.

Contribution

It establishes a new criterion relating commensurators to biautomaticity and provides explicit examples of CAT(0) groups lacking biautomaticity.

Findings

The commensurator of quasiconvex abelian subgroups in biautomatic groups has finite image.

Existence of CAT(0) groups that are not biautomatic.

Resolution of open questions about the relationship between CAT(0) and biautomatic groups.

Abstract

We show that the commensurator of any quasiconvex abelian subgroup in a biautomatic group is small, in the sense that it has finite image in the abstract commensurator of the subgroup. Using this criterion we exhibit groups that are CAT(0) but not biautomatic. These groups also resolve a number of other questions concerning CAT(0) groups.

Equations

$$G_P\text{mathchar}0"303A=⟨a,b,t\Vert [a,b]=1,t{a}^2{b}^{-1}{t}^{-1}=a^{2}b,ta{b}^2{t}^{-1}=a^{-1}{b}^2⟩.$$

$$G_{k,2}\text{mathchar}0"303A=⟨a,b,t\Vert [a,b]=1,ta{t}^{-1}=b,t{b}^2{t}^{-1}=a^{-2}{b}^k⟩.$$

$${\mathrm{Comm}}_G(H)\text{mathchar}0"303A=\{g\in G\mid |H:(H\cap gH{g}^{-1})|<\infty \text{mbox}and|gH{g}^{-1}:(H\cap gH{g}^{-1})|<\infty \}.$$

$$\underset{i\to \infty }{\mathrm{limsup}}{\log }_w(v_i)=\infty .$$

$$\mathcal{B}\text{mathchar}0"303A=\{g^{-1}\mu (a)g^{\prime }\mid a\in \mathcal{A},g,g^{\prime }\in G,|g{|}_{\mathcal{A}},|g^{\prime }{|}_{\mathcal{A}}\le \varkappa \}\cap H.$$

$$|V_i|\ge |g\nu (U_i)e_i{|}_{\mathcal{C}}\ge |\nu (U_i){|}_{\mathcal{C}}-|g{|}_{\mathcal{C}}-|e_i{|}_{\mathcal{C}}\to \infty ,\text{mbox}asi\to \infty .$$

$$⟨b,c{⟩}_L\text{mathchar}0"303A=\frac{1}{2}\left(d((b+c)x,x{)}^2-d(x,bx{)}^2-d(x,cx{)}^2\right)$$

$$⟨\tilde{\varphi }(b),\tilde{\varphi }(c){⟩}_L=\frac{1}{m^2}⟨\varphi (mb),\varphi (mc){⟩}_{L^{\prime \prime }}=\frac{1}{m^2}⟨mb,mc{⟩}_{L^{\prime }}=⟨b,c{⟩}_L,$$

$$G(L,\varphi ,L^{\prime })\text{mathchar}0"303A=⟨L,t\Vert tc{t}^{-1}=\varphi (c),\forall c\in L^{\prime }⟩.$$

$$G(A,L^{\prime })\text{mathchar}0"303A=⟨L,t\Vert tc{t}^{-1}=Ac,\forall c\in L^{\prime }⟩.$$

$$A_{k/m}\text{mathchar}0"303A=\left(\begin{array}{cc}0& -1\\ 1& k/m\end{array}\right),$$

$$A_P\text{mathchar}0"303A=\left(\begin{array}{cc}3/5& -4/5\\ 4/5& 3/5\end{array}\right).$$

$$G_{k,m}\text{mathchar}0"303A=⟨a,b,t\Vert [a,b]=1,ta{t}^{-1}=b,t{b}^m{t}^{-1}=a^{-m}{b}^k⟩.$$

$$G_P=⟨a,b,t\Vert [a,b]=1,t{a}^2{b}^{-1}{t}^{-1}=a^{2}b,ta{b}^2{t}^{-1}=a^{-1}{b}^2⟩,$$

$$G_P^{\prime }=⟨a,b,t\Vert [a,b]=1,t{a}^5{t}^{-1}=a^3{b}^4,t{b}^5{t}^{-1}=a^{-4}{b}^3⟩.$$

$$⟨Aa,Aa{⟩}_L={\lambda }^2,⟨Aa,a{⟩}_L=⟨a,Aa{⟩}_L={\lambda }^2\mathrm{tr}(A)/2,$$

$$mA+ml{A}^{-1}=k{I}_2,$$

$$L^m\subseteq L^{\prime }\text{mbox}and{L}^{ml}\subseteq L^{\prime \prime }.$$

$$t{c}^m{t}^{-2}{c}^{ml}t=c^k\text{mbox}foranyc\in L\text{mbox}inG.$$

$$\varphi (a)\text{mathchar}0"303A=a^m,\varphi (b)\text{mathchar}0"303A=b^m,\varphi (t)\text{mathchar}0"303A=t$$

$$M=⟨L,t\Vert tc{t}^{-1}=(r^{\prime -1}{r}^{-1}{)}^{-1}c(r^{\prime -1}{r}^{-1}),\forall c\in L^{\prime }⟩.$$

$$⟨a,r,c,s\Vert ra{r}^{-1}=a^{-1},sc{s}^{-1}=c^{-1},a^2{r}^{-2}=c,a{r}^4=s^2⟩,$$

$$⟨r,s\Vert r^5{s}^{-2}{r}^3{s}^{-2}=1,s^5{r}^{-10}{s}^3{r}^{-10}=1⟩.$$

Full text

1 Introduction

The theory of automatic and biautomatic groups was developed in the late 1980’s, and described in the book [17] written by Epstein, Cannon, Holt, Levy, Paterson and Thurston. The question of whether there are any automatic groups that are not biautomatic appears in this book as [17, Question 2.5.6] and as Remark 6.19 at the end of the paper of Gersten and Short [20] in which biautomatic groups were introduced. By that time it had been shown that hyperbolic groups are biautomatic (implicit in [17, Theorem 3.4.5]). The definition of biautomaticity has both language-theoretic and geometric aspects, whose interaction is non-trivial. This motivated Alonso and Bridson to introduce, in the early 1990’s, the geometric class of semihyperbolic groups [1]. This class contains all biautomatic groups and all CAT(0) groups. In the mid 1990’s Niblo and Reeves proved that CAT(0) cubical groups are biautomatic [32]. The question of whether all CAT(0) groups are automatic or even biautomatic must have been a motivating question for much of the above work, but the earliest written versions that we were able to find were in the PhD thesis of Elder [16, Open Question 2], written in 2000, and in a problem list compiled by McCammond after the American Institute of Mathematics meeting ‘Problems in Geometric Group Theory’ April 23–27, 2007 [27, Question 13] (see also [19, Section 6.6]). We answer the latter question by constructing the first examples of CAT(0) groups that are not biautomatic.

The groups that we construct are higher-dimensional analogues of Baumslag-Solitar groups [7], in the sense that they are HNN-extensions of free abelian groups of rank greater than one in which the stable letter conjugates two finite-index subgroups. Consider the group $G_{P}$ given by the presentation

[TABLE]

This is an HNN-extension of $L=\langle a,b\rangle\cong\Z^{2}$ in which the stable letter conjugates two subgroups of index five. If we let $L$ act on the Euclidean plane $\mathbb{E}^{2}$ in such a way that $a$ and $b$ act as translations of length one in orthogonal directions, the elements $a^{2}b^{-1}$ and $ab^{2}$ act as translations of length $\sqrt{5}$ in orthogonal directions, as do the elements $a^{2}b$ and $a^{-1}b^{2}$. The action of $\langle a,b\rangle$ on $\mathbb{E}^{2}$ extends to an action of the whole group $G_{P}$, in which the stable letter acts as a rotation through $\arccos(3/5)$. The fact that this action is by isometries is what ensures that $G_{P}$ is CAT(0), while the fact that the rotation through $\arccos(3/5)$ has infinite order is what allows us to show that $G_{P}$ is not virtually biautomatic (i.e., no finite-index subgroup is biautomatic).

The action of $G_{P}$ on the Euclidean plane can be used to show that for any $n\neq 0$, the subgroup $\langle a^{n},t^{n}\rangle<G_{P}$ is not abelian. But also $\langle a^{n},t^{n}\rangle$ contains a finite-index subgroup of $\langle a,b\rangle$, and so it cannot be a free group (see Corollary 9.6 below). Thus the elements $a,t\in G_{P}$ give the first negative answer to a question of Wise concerning a strong version of the Tits Alternative for CAT(0) groups [8, Question 2.7].

The Bass-Serre tree $T$ for $G_{P}$ is a regular tree of valency $10$, and we show that $G_{P}$ acts geometrically on the direct product $\mathbb{E}^{2}\times T$. For $F$ a free group of rank 5, the group $\Z^{2}\times F$ acts geometrically on the same CAT(0) space as $G_{P}$, and it follows that these groups are quasi-isometric to each other. Hence any property that is not shared by $\Z^{2}\times F$ and the group $G_{P}$ cannot be invariant under quasi-isometry, even amongst CAT(0) groups. In particular there are CAT(0) groups that are quasi-isometric to $\Z^{2}\times F$ but are not virtually biautomatic, and hence cannot be virtually cubical. In fact, by a recent result of Huang and Prytuła [22], no finite-index subgroup of $G_{P}$ admits a proper action on a finite-dimensional CAT(0) cube complex by cubical automorphisms (because no positive power of $t$ normalizes a subgroup of finite index in the abelian base group $L$).

Note that although $G_{P}$ is quasi-isometric to $\Z^{2}\times F$ it is not commensurable to it, and so $\Z^{2}\times F$ is not quasi-isometrically rigid, contrary to some claims in the existing literature. Moreover, $G_{P}$ embeds as an irreducible lattice in the group of isometries of $\mathbb{E}^{2}\times T$. There has been some confusion concerning this property in the literature: in particular [13] claims that no such lattices exist (this has recently been rectified in [14]).

By varying the geometry of the free abelian subgroup, one can construct similar examples in which the indices of the subgroups conjugated by the stable letter are smaller. Consider the groups $G_{k,2}$ for $k\in\Z$ with presentations

[TABLE]

In $G_{k,2}$, the stable letter conjugates two index $2$ subgroups of $\langle a,b\rangle\cong\Z^{2}$, and since $tat^{-1}=b$, this relator and the generator $b$ can be eliminated, giving a presentation of $G_{k,2}$ with just two generators and two relators. We show that $G_{k,2}$ is CAT(0) if and only if $-3\leq k\leq 3$ and that $G_{k,2}$ is biautomatic if and only if $k\in\{-2,0,2\}$. Thus the groups $G_{k,2}$ for $k\in\{-3,-1,1,3\}$ are CAT(0) but not biautomatic, and the elements $a,t\in G_{k,2}$ give counterexamples to Wise’s question for these values of $k$ too [8, Question 2.7].

Although our main examples arise already for base groups free abelian of rank $2$, we consider commensurating HNN-extensions of free abelian groups of arbitrary finite rank. Such a group is described by a pair $L^{\prime},L^{\prime\prime}$ of finite-index subgroups of $L=\Z^{n}$, together with a matrix $A\in GL(n,\Q)$ such that multiplication by $A$ defines an isomorphism $A\times:L^{\prime}\to L^{\prime\prime}$. We denote this group by $G(A,L^{\prime})$, because $L^{\prime\prime}=A\,L^{\prime}$ is determined by the pair $(A,L^{\prime})$. Many of the results that we obtain concerning these groups are summarized in the following theorem.

Theorem 1.1**.**

Let $G=G(A,L^{\prime})$ be a group defined above. Then

$G$* is residually finite $\Leftrightarrow$ $G$ is linear $\Leftrightarrow$ either $L^{\prime}=L$ or $A\,L^{\prime}=L$ or $A$ is conjugate in $GL(n,\Q)$ to an element of $GL(n,\Z)$;* 2. 2.

$G$* is CAT(0) $\Leftrightarrow$ $A$ is conjugate in $GL(n,\R)$ to an orthogonal matrix;* 3. 3.

$G$* is biautomatic $\Leftrightarrow$ $G$ is virtually biautomatic $\Leftrightarrow$ $A$ has finite order.*

In the special case when $n=1$ the three parts of Theorem 1.1 are previously known results concerning Baumslag-Solitar groups. Similarly to Baumslag-Solitar groups, many of the groups $G(A,L^{\prime})$ can be shown to be non-Hopfian. We give a criterion for this in Proposition 10.1 which implies that the groups $G_{P}$ and $G_{k,2}$, for $k$ odd, are all non-Hopfian.

The three parts of Theorem 1.1 are proved separately. The characterization of when $G$ is residually finite in claim (1) follows from earlier work in [4], and we use the affine action of $G$ on $L\otimes\Q$ to show that each residually finite $G$ is linear over $\Q$. For claim (2) we introduce an addendum to the Flat Torus Theorem concerning the commensurator of an abelian group of semi-simple isometries of a CAT(0) space, which may be of independent interest. The main result used to prove claim (3) is the next theorem that imposes strong restrictions on the commensurator of a quasiconvex abelian subgroup in a biautomatic group.

Theorem 1.2**.**

Suppose that $G$ is a group with a biautomatic structure $(\mathcal{A},\mathcal{L})$, and $H\leqslant G$ is an $\mathcal{L}$-quasiconvex abelian subgroup. Then the commensurator $\operatorname{Comm}_{G}(H)$, of $H$ in $G$, has finite image in the abstract commensurator $\operatorname{Comm}(H)$. In particular, there is a finite-index subgroup $\operatorname{Comm}_{G}^{0}(H)\lhd\operatorname{Comm}_{G}(H)$ such that every finitely generated subgroup of $\operatorname{Comm}_{G}^{0}(H)$ centralizes a finite-index subgroup of $H$ in $G$.

The key tool used in the proof of Theorem 1.2 is the boundary of an automatic structure, discussed in Section 2. Our strategy is to show that a biautomatic structure on $G$ induces a biautomatic structure on the quasiconvex subgroup $H$, and $\operatorname{Comm}_{G}(H)$ acts on the boundary of this structure in a natural way (see Section 4). In Section 3 we show that the boundary of any (bi)automatic structure on the abelian group $H$ is finite, hence a finite-index subgroup of $\operatorname{Comm}_{G}(H)$ must fix this boundary pointwise. Finally, in Section 5 we apply the latter to prove Theorem 1.2.

The technical heart of this paper is the results concerning biautomaticity, but the other parts of the paper may be read independently of this material. Section 6 contains our addendum to the Flat Torus Theorem. Section 7 introduces the groups $G(A,L^{\prime})$, and characterizes which of them are CAT(0). Section 8 characterizes which of the groups $G(A,L^{\prime})$ are biautomatic. Section 9 considers in more detail the case when $L=\Z^{2}$, and discusses a class of examples which includes the groups $G_{P}$ and $G_{k,2}$ already mentioned above, establishing many of their properties. Section 10 concerns the non-Hopfian property, with results only in the case $L=\Z^{2}$, and residual finiteness, with a more general result. Section 11 shows that many of our examples can be embedded as index two subgroups of free products with amalgamation in which each factor is virtually abelian. This construction gives rise to amalgamated products of virtually abelian groups with surprising properties. Section 12 concludes with a short list of open problems concerning the groups $G(A,L^{\prime})$.

Acknowledgements

Firstly, the authors thank Tomasz Prytuła. He asked the authors for an example of an abelian subgroup of a CAT(0) group, whose commensurator does not normalize any finite-index subgroup, in connection with his work on classifying spaces for families of abelian subgroups [33, 35]. This question was what originally led us to consider the groups $G^{\prime}_{P}$ and $G_{P}$. The authors also thank Martin Bridson, Pierre-Emmanuel Caprace, Derek Holt, Jingyin Huang, Denis Osin and Kevin Whyte for helpful comments on aspects of the work. The first named author was partially supported by a Research Fellowship from the Leverhulme Trust. Most of the work was done in Southampton. However, during the project the first named author spent some weeks at INI, Cambridge, where research was supported by EPSRC grant EP/K032208/1.

2 Background and notation

2.1 Commensurators

If $G$ is a group and $H\leqslant G$ is a subgroup, the commensurator of $H$ in $G$ is the subset defined by

[TABLE]

It is not difficult to see that $\operatorname{Comm}_{G}(H)$ is actually a subgroup of $G$. We will say that $G$ commensurates $H$ if $G=\operatorname{Comm}_{G}(H)$.

The elements of the abstract commensurator of a group $G$, denoted $\operatorname{Comm}(G)$, are equivalence classes of isomorphisms $\phi:H\rightarrow K$, where both $H$ and $K$ are finite-index subgroups of $G$. Two such isomorphisms $\phi_{1}:H_{1}\rightarrow K_{1}$ and $\phi_{2}:H_{2}\rightarrow K_{2}$ are equivalent if there is $H\leqslant H_{1}\cap H_{2}$ also of finite index in $G$ so that $\phi_{1}|_{H}=\phi_{2}|_{H}$. The abstract commensurator is a group, in which the composite of the equivalence classes of $\phi:H\rightarrow K$ and $\psi:L\rightarrow M$ is the class of $\psi\circ\phi:H\cap\phi^{-1}(L)\rightarrow\psi(\phi(H)\cap L)$.

For any group $G$ and a subgroup $H$, there is a natural map from $\operatorname{Comm}_{G}(H)$ to $\operatorname{Comm}(H)$, taking $g\in\operatorname{Comm}_{G}(H)$ to the element of $\operatorname{Comm}(H)$ represented by conjugation by $g$. The kernel of this homomorphism consists of the elements $g\in G$ that centralize some finite-index subgroup of $H$.

It is easy to see that the abstract commensurator $\operatorname{Comm}(\Z^{n})$ can be naturally identified with $GL(n,\Q)$. Equivalently, if $L$ is a finitely generated free abelian group then $\operatorname{Comm}(L)$ is identified with $GL(L\otimes\Q)$, the group of vector space automorphisms of $L\otimes\Q$. There is a coordinate-free description of the homomorphism from the abstract commensurator of $L$ to $GL(L\otimes\Q)$: suppose that $\phi:L^{\prime}\rightarrow L^{\prime\prime}$ is an isomorphism between finite-index subgroups of $L$, and let $i:L^{\prime}\rightarrow L$ and $j:L^{\prime\prime}\rightarrow L$ be the inclusions. Each of $i\otimes 1:L^{\prime}\otimes\Q\rightarrow L\otimes\Q$ and $j\otimes 1:L^{\prime\prime}\otimes\Q\rightarrow L\otimes\Q$ is an isomorphism. The image ${\tilde{\phi}}$ of $(\phi:L^{\prime}\rightarrow L^{\prime\prime})$ in $GL(L\otimes\Q)$ is ${\tilde{\phi}}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=(j\otimes 1)\circ(\phi\otimes 1)\circ(i\otimes 1)^{-1}:L\otimes\Q\rightarrow L\otimes\Q$.

2.2 Automatic structures and the fellow traveller property

In this subsection we will briefly discuss the notions of automatic and biautomatic structures on groups. The reader is referred to [17, Section 2.3, 2.5] for more details and examples.

Let $\mathcal{A}$ be a finite set and let $G$ be a group with a map $\mu:\mathcal{A}\to G$. We will say that $G$ is generated by $\mathcal{A}$ if the extension of $\mu$ to a homomorphism from the free monoid $\mathcal{A}^{*}$ to $G$ is surjective. Elements of $\mathcal{A}^{*}$ will be called words, and if $W\in\mathcal{A}^{*}$ and $g\in G$ are such that $\mu(W)=g$, we will say that $W$ represents $g$ in $G$. Given a word $W$ in $\mathcal{A}^{*}$, $|W|$ will denote its length. We will always assume that $\mathcal{A}$ is closed under inversion, that is there is an involution $\iota:\mathcal{A}\to\mathcal{A}$, where, for each $a\in A$, $\iota(a)$ is denoted $a^{-1}$ and satisfies $\mu(a^{-1})=\mu(a)^{-1}$ in $G$. Any subset $\mathcal{L}\subseteq A^{*}$ will be called a language over $\mathcal{A}$.

We can form the Cayley graph $\Gamma(G,\mathcal{A})$, of $G$ with respect to $\mathcal{A}$ as follows: the vertices are elements of $G$ and for every $g\in G$ and $a\in A$ there is an edge from $g$ to $g\mu(a)$, labelled by $a$. Metrically, every edge in $\Gamma(G,\mathcal{A})$ will be considered as an isometric copy of the interval $[0,1]$.

We will use $\mathrm{d}_{\mathcal{A}}(\cdot,\cdot)$ to denote the standard graph metric on $\Gamma(G,\mathcal{A})$; its restriction to $G$ is the word metric corresponding to the generating set $\mathcal{A}$. For any element $g\in G$ we will use $|g|_{\mathcal{A}}$ to denote $\mathrm{d}_{\mathcal{A}}(1_{G},g)$; in other words, $|g|_{\mathcal{A}}$ is the length of a shortest word in $\mathcal{A}^{*}$ representing $g$ in $G$. Note that $|g|_{\mathcal{A}}=|g^{-1}|_{\mathcal{A}}$ since $\mathcal{A}$ is closed under inversion.

For an edge path $p$ in $\Gamma(G,\mathcal{A})$, $p_{-}$ and $p_{+}$ will denote the start and end vertices of $p$ respectively, and $|p|$ will denote the length of $p$. The label of $p$ is a word from $\mathcal{A}^{*}$ obtained by collating the labels of its edges.

Any edge path $p$ in $\Gamma(G,\mathcal{A})$ can be equipped with the following ray parametrization: $\widehat{p}:[0,\infty)\to\Gamma(G,\mathcal{A})$, where for each $t\in[0,|p|]\cap\Z$, $\widehat{p}(t)$ is the $t$-th vertex of $p$ (so that $\widehat{p}(0)=p_{-}$, $\widehat{p}(|p|)=p_{+}$), and $\widehat{p}(t)=p_{+}$ for all $t>|p|$; for every $s\in[0,|p|-1]\cap\Z$ and each $t\in[s,s+1]$, $\widehat{p}(t)$ is defined so that the restriction of $\widehat{p}$ to $[s,s+1]$ is an isometry with the corresponding edge of $p$ in $\Gamma(G,\mathcal{A})$.

Definition 2.1**.**

Let $p,q$ be two edge paths in $\Gamma(G,\mathcal{A})$, with ray parametrizations $\widehat{p},\widehat{q}:[0,\infty)\to\Gamma(G,\mathcal{A})$ respectively, and let $\zeta\geq 0$ be a constant. We will say that $p$ $\zeta$-follows $q$ if $\mathrm{d}_{\mathcal{A}}(\widehat{p}(t),\widehat{q}(t))\leq\zeta$ for all $t\in[0,|p|]\cap\Z$. The paths $p$ and $q$ are said to *$\zeta$-fellow travel * if $\mathrm{d}_{\mathcal{A}}(\widehat{p}(t),\widehat{q}(t))\leq\zeta$ for all $t\in\Z$.

If $U$ and $V$ are two words from $\mathcal{A}^{*}$, we can consider two edge paths $p$, $q$ in $\Gamma(G,\mathcal{A})$ which start at $1_{G}$ and are labelled by $U$, $V$ respectively. We will say that $U$ $\zeta$-follows $V$ if the path $p$ $\zeta$-follows the path $q$. Similarly, $U$ and $V$ are said to $\zeta$-fellow travel if $p$ and $q$ $\zeta$-fellow travel.

It is easy to see that two edge paths (words) $\zeta$-fellow travel if and only if each of them $\zeta$-follows the other one.

Definition 2.2**.**

Let $G$ be a group. An automatic structure on the group $G$ is a pair $(\mathcal{A},\mathcal{L})$, where $\mathcal{A}$ is a finite generating set of $G$, which comes equipped with a map $\mu:\mathcal{A}\to G$ as above and which is closed under inversion, and $\mathcal{L}\subseteq\mathcal{A}^{*}$ is a language satisfying the following conditions:

• (i)

$\mu(\mathcal{L})=G$;

• (ii)

$\mathcal{L}$ is a regular language, i.e., $\mathcal{L}$ is the accepted language of a finite state automaton $\mathfrak{A}$ over $\mathcal{A}$;

• (iii)

there exists $\zeta\geq 0$ such that for any two edge paths $p,q$ in $\Gamma(G,\mathcal{A})$, labelled by some words from $\mathcal{L}$ and satisfying $p_{-}=q_{-}$ and $\mathrm{d}_{\mathcal{A}}(p_{+},q_{+})\leq 1$, $p$ and $q$ $\zeta$-fellow travel.

$(\mathcal{A},\mathcal{L})$ is a biautomatic structure on $G$, if $\mathcal{L}$ satisfies the conditions (i), (ii) and

• (iii’)

there exists $\zeta\geq 0$ such that for any two edge paths $p,q$ in $\Gamma(G,\mathcal{A})$, labelled by some words from $\mathcal{L}$ and satisfying $\mathrm{d}_{\mathcal{A}}(p_{-},q_{-})\leq 1$ and $\mathrm{d}_{\mathcal{A}}(p_{+},q_{+})\leq 1$, $p$ and $q$ $\zeta$-fellow travel.

The group $G$ is said to be automatic (biautomatic) if it admits an automatic (respectively, biautomatic) structure.

Obviously condition (iii’) is stronger than condition (iii), so every biautomatic structure on a group is also an automatic structure. Also, condition (iii) implies that for any two paths $p,q$ that are labelled by some words from $\mathcal{L}$ and satisfy $p_{-}=q_{-}$ and $\mathrm{d}_{\mathcal{A}}(p_{+},q_{+})\leq C$, for some $C\in\mathbb{N}\cup\{0\}$, $p$ and $q$ $(\zeta\max\{C,1\})$-fellow travel in $\Gamma(G,\mathcal{A})$. And, if (iii’) holds, then the requirement $p_{-}=q_{-}$ can be relaxed to $\mathrm{d}_{\mathcal{A}}(p_{-},q_{-})\leq C$.

An automatic structure $(\mathcal{A},\mathcal{L})$ on a group $G$ is said to be finite-to-one if $|\mu^{-1}(g)\cap\mathcal{L}|<\infty$ for every $g\in G$. It is known (see [17, Theorem 2.5.1]) that every automatic (biautomatic) structure can be refined to a finite-to-one automatic (respectively, biautomatic) structure, hence from now on we will assume that all the automatic and biautomatic structures are finite-to-one. Without loss of generality we will also suppose that all the automata in this paper have no dead states.

It is worth noting that the nowadays standard definition of a biautomatic structure that we give above is not the same as the definition from [17, Definition 2.5.4]: see [2]. However, for finite-to-one structures these definitions are equivalent [2, Theorem 6].

2.3 The boundary of an automatic structure

Definition 2.3**.**

Let $(\mathcal{A},\mathcal{L})$ be an automatic structure on $G$, and let $(W_{i})_{i\in\mathbb{N}}$ be a sequence of words from $\mathcal{L}$. We will say that this sequence tends to infinity if $|W_{i}|\to\infty$ as $i\to\infty$ and there exists $\zeta\geq 0$ such that for any $i,j\in\mathbb{N}$, $W_{i}$ $\zeta$-follows $W_{j}$ whenever $i\leq j$.

Suppose that $(U_{i})_{i\in\mathbb{N}}$ and $(V_{i})_{i\in\mathbb{N}}$ are two sequences of words from $\mathcal{L}$ tending to infinity, and, for each $i\in\mathbb{N}$, $p_{i}$, $q_{i}$ are the edge paths in $\Gamma(G,\mathcal{A})$ starting at $1_{G}$ and labelled by $U_{i}$, $V_{i}$ respectively. We will say that $(U_{i})_{i\in\mathbb{N}}$ is equivalent to $(V_{i})_{i\in\mathbb{N}}$ if the Hausdorff distance between the corresponding sequences of paths $(p_{i})_{i\in\mathbb{N}}$ and $(q_{i})_{i\in\mathbb{N}}$ is at most $\eta$ in $\Gamma(G,\mathcal{A})$. In other words, there must exist $\eta\geq 0$ such that for all $i\in\mathbb{N}$, any vertex of $p_{i}$ is at most $\eta$ away from a vertex of $q_{j}$, for some $j\in\mathbb{N}$, and vice-versa.

Definition 2.3 immediately implies the following.

Remark 2.4*.*

If $(U_{i_{j}})_{j\in\mathbb{N}}$ is a subsequence of a sequence of words $(U_{i})_{i\in\mathbb{N}}$ tending to infinity, then $(U_{i_{j}})_{j\in\mathbb{N}}$ also tends to infinity and is equivalent to $(U_{i})_{i\in\mathbb{N}}$.

Definition 2.5**.**

If $(\mathcal{A},\mathcal{L})$ is an automatic structure on a group $G$, then the boundary, $\partial\mathcal{L}$, of this automatic structure is the set of equivalence classes of sequences of words from $\mathcal{L}$ tending to infinity. If $\alpha\in\partial\mathcal{L}$ is the equivalence class of a sequence $(U_{i})_{i\in\mathbb{N}}$, we will say that this sequence converges to the boundary point $\alpha$.

The first definition of a boundary of an automatic structure was given by Neumann and Shapiro in [31, pp. 459-460]. It is not difficult to see that there is a natural bijection between their boundary and ours. However, our Definition 2.5 is better suited for constructing the action of a group on the boundary of a biautomatic structure (see Section 4 below).

Given an automaton $\mathfrak{A}$, by a path in this automaton we will mean any directed path in the corresponding graph. If $x$ is a state of $\mathfrak{A}$, a cycle based at $x$ is a closed path starting and ending at the state $x$. A path in $\mathfrak{A}$ is simple if it does not pass through the same state twice. A cycle based at a state $x$ in $\mathfrak{A}$ is simple if it passes through $x$ exactly twice (at its beginning and at its end) and does not pass through any other state more than once. A path or a cycle is non-trivial if it has at least one edge.

The following lemma is a straightforward consequence of the definitions.

Lemma 2.6**.**

Suppose that $(\mathcal{A},\mathcal{L})$ is an automatic structure on a group $G$, $\mathfrak{A}$ is a finite state automaton for $\mathcal{L}$ and $W$ is the label of a non-trivial cycle $w$ in $\mathfrak{A}$ based at some state $x$. Let $S$ and $T$ be words from $\mathcal{A}^{*}$ labelling some paths $s$ and $t$ connecting the initial state of $\mathfrak{A}$ with $x$ and $x$ with an accept state of $\mathfrak{A}$ respectively. Then $(SW^{i}T)_{i\in\mathbb{N}}$ is a sequence of words from $\mathcal{L}$ tending to infinity. In particular, if $\mathcal{L}$ is infinite then $\partial\mathcal{L}\neq\emptyset$.

Definition 2.7**.**

In the notation of Lemma 2.6, if the paths $s,t$ and the cycle $w$ are all simple, the sequence of words $(SW^{i}T)_{i\in\mathbb{N}}$ will be called a simple sequence tending to infinity and the corresponding point $\alpha\in\partial\mathcal{L}$ will be called a simple boundary point.

3 The boundary of an automatic structure on an abelian group

Theorem 3.1**.**

Let $H$ be an abelian group with an automatic structure $(\mathcal{B},\mathcal{M})$ and let $\mathfrak{A}$ be a finite state automaton accepting the language $\mathcal{M}$. Then every boundary point $\alpha\in\partial\mathcal{M}$ is simple; in other words, every sequence of words from $\mathcal{M}$ tending to infinity is equivalent to a simple such sequence.

Proof.

Let $w$ be a simple cycle in $\mathfrak{A}$ based at a state $x$. Given a path $v$ in $\mathfrak{A}$, an occurrence of $w$ in $v$ is a subpath of $v$ starting and ending at $x$ and traversing $w$ exactly once. We will use $\log_{w}(v)$ to denote the number of occurrences of $w$ in $v$.

Let $(U_{i})_{i\in\mathbb{N}}$ be a sequence of words from $\mathcal{M}$ tending to infinity, and for each $i\in\mathbb{N}$ fix a path $u_{i}$ from the initial state to an accept state in $\mathfrak{A}$, labelled by $U_{i}$. Since $|u_{i}|=|U_{i}|\to\infty$ as $i\to\infty$ and the automaton $\mathfrak{A}$ is finite, in view of Remark 2.4 we can replace $(U_{i})_{i\in\mathbb{N}}$ by a subsequence so that for all $i\in\mathbb{N}$ $u_{i}$ and $u_{i+1}$ share a common initial subpath $v_{i}$, of length $i$.

Evidently, since $|v_{i}|=i\to\infty$ as $i\to\infty$, there must exist some non-trivial simple cycle $w$, based at a state $x$ in $\mathfrak{A}$, such that

[TABLE]

Choose $I\in\mathbb{N}$ so that $\log_{w}(v_{I})>0$, then there is a path $s$ from the initial state of $\mathfrak{A}$ to $x$ such that $sw$ is an initial subpath of $v_{I}$ (and, hence, of $u_{i}$ for all $i\geq I$). We can also choose a simple path $t$ from $x$ to an accept state of $\mathfrak{A}$, and let $S$, $T$ and $W$ be the labels of $s$, $t$ and $w$ respectively. We will now show that the sequence of words $(SW^{i}T)_{i\in\mathbb{N}}$, which tends to infinity by Lemma 2.6, is equivalent to the original sequence $(U_{i})_{i\in\mathbb{N}}$.

Let $p_{i}$ and $q_{i}$ be the paths in $\Gamma(H,\mathcal{B})$ starting at $1_{H}$ and labelled by the words $U_{i}$ and $SW^{i}T$ respectively, $i\in\mathbb{N}$. For any $i\in\mathbb{N}$, according to (1), there is $j\geq I$ such that $\log_{w}(u_{j})\geq i$, hence $u_{j}$ is the concatenation $u_{j}=sws_{1}ws_{2}\dots ws_{i}$, where $s_{1},\dots,s_{i-1}$ are some paths in $\mathfrak{A}$ from $x$ to itself, and $s_{i}$ is a path from $x$ to an accept state of $\mathfrak{A}$. Let $S_{k}$ be the word in $\mathcal{B}^{*}$ labelling $s_{k}$, $k=1,\dots,i$.

Note that since $H$ is abelian, the word $U_{j}=SWS_{1}\dots WS_{i}$ represents the same element of $H$ as the word $R\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=SW^{i}S_{1}\dots S_{i}$. Moreover, clearly $R$ is accepted by $\mathfrak{A}$ (it labels the path $sw^{i}s_{1}\dots s_{i}$ in $\mathfrak{A}$), by construction, hence $U_{j}$ and $R$ $\varepsilon$-fellow travel in $\Gamma(H,\mathcal{B})$, where $\varepsilon\geq 0$ is the constant from the definition of the automatic structure $(\mathcal{B},\mathcal{M})$. Therefore the word $SW^{i}$ $\varepsilon$-follows $U_{j}$ in $\Gamma(H,\mathcal{B})$, which implies that every vertex of the path $q_{i}$ lies $\eta_{1}$-close to a vertex of $p_{j}$, where $\eta_{1}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\varepsilon+|T|$.

To show the converse, let $l\in\mathbb{N}$ be arbitrary and set $i\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=|U_{l}|$. By (1) there exists $j\geq\max\{l,i,I\}$ such that $u_{j}=sws_{1}ws_{2}\dots ws_{i}$ as before. If $S_{k}$ is the label of $s_{k}$, $k=1,\dots,i$, arguing as above we can conclude that the words $U_{j}$ and $R\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=SW^{i}S_{1}\dots S_{i}$ $\varepsilon$-fellow travel in $\Gamma(H,\mathcal{B})$. Since $(U_{m})_{m\in\mathbb{N}}$ tends to infinity, we know that there is $\zeta\geq 0$, depending only on this sequence, such that $U_{l}$ $\zeta$-follows $U_{j}$, as $l\leq j$. Hence $U_{l}$ $\zeta$-follows the prefix of $U_{j}$, of length $|U_{l}|=i$, which, in its own turn, $\varepsilon$-follows the prefix of $R$, of the same length. Since $SW^{i}$ is a prefix of $R$ of length at least $i$, we can conclude that $U_{l}$ $(\zeta+\varepsilon)$-follows the word $SW^{i}$ in $\Gamma(H,\mathcal{B})$, hence it also $(\zeta+\varepsilon)$-follows the word $SW^{i}T$. Therefore each vertex of the path $p_{l}$ is $\eta_{2}$-close to a vertex of the path $q_{i}$ in $\Gamma(H,\mathcal{B})$, where $\eta_{2}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\zeta+\varepsilon$.

Thus we have shown that the sequences of paths $(p_{i})_{i\in\mathbb{N}}$ and $(q_{i})_{i\in\mathbb{N}}$ lie in $\eta\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\max\{\eta_{1},\eta_{2}\}$-neighborhoods of each other in $\Gamma(H,\mathcal{B})$, yielding that the sequences $(U_{i})_{i\in\mathbb{N}}$ and $(SW^{i}T)_{i\in\mathbb{N}}$ are equivalent, as claimed.

The proof of the theorem is not quite finished yet, as the path $s$, labelled by $S$ in $\mathfrak{A}$, may not be simple. So, choose some simple path $s^{\prime}$, joining the initial state of $\mathfrak{A}$ with the state $x$, at which the simple cycle $w$ is based, and let $S^{\prime}$ be the word labelling $s^{\prime}$. By Lemma 2.6, $(S^{\prime}W^{i}T)_{i\in\mathbb{N}}$ is a simple sequence tending to infinity, and we will complete the proof by showing that this sequence is equivalent to $(SW^{i}T)_{i\in\mathbb{N}}$ (and, hence to $(U_{i})_{i\in\mathbb{N}}$, by transitivity).

Let $q_{i}^{\prime}$ be the path in $\Gamma(H,\mathcal{B})$ starting at $1_{H}$ and labelled by the word $S^{\prime}W^{i}T$, $i\in\mathbb{N}$. Since $H$ is abelian, for all $i\in\mathbb{N}$ the word $(SW^{i}T)^{-1}S^{\prime}W^{i}T$ represents the same element of $H$ as the word $S^{-1}S^{\prime}$. Therefore $\mathrm{d}_{\mathcal{B}}((q_{i})_{+},(q_{i}^{\prime})_{+})\leq\theta$ for all $i\in\mathbb{N}$ , where $\theta\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=|S^{-1}S^{\prime}|$. Since the labels of $q_{i}$ and $q_{i}^{\prime}$ are both in the language $\mathcal{M}$, these paths $\lambda$-fellow travel in $\Gamma(H,\mathcal{B})$ for each $i\in\mathbb{N}$, where $\lambda\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\varepsilon\max\{1,\theta\}$. Hence the Hausdorff distance between the sequences of paths $(q_{i})_{i\in\mathbb{N}}$ and $(q_{i}^{\prime})_{i\in\mathbb{N}}$ is at most $\lambda$, which implies that the sequences of words $(SW^{i}T)_{i\in\mathbb{N}}$ and $(S^{\prime}W^{i}T)_{i\in\mathbb{N}}$ are equivalent. Thus the theorem is proved. ∎

Definition 2.7 implies that any automatic structure admits only finitely many simple sequences of words tending to infinity. Therefore the following statement is a consequence of Theorem 3.1 (cf. [31, Theorem 6.7]).

Corollary 3.2**.**

If $(\mathcal{B},\mathcal{M})$ is an automatic structure on an abelian group then $\partial\mathcal{M}$ is finite.

4 The action of the commensurator on the boundary of a

quasiconvex subgroup

Let $G$ be a group equipped with a (finite-to-one) biautomatic structure $(\mathcal{A},\mathcal{L})$. Recall that a subgroup $H\leqslant G$ is $\mathcal{L}$-quasiconvex if there exists $\varkappa\geq 0$ such that for any path $p$ in $\Gamma(G,\mathcal{A})$ starting at $1_{G}$, ending at some $h\in H$ and labelled by a word $W\in\mathcal{L}$, every vertex of $p$ lies in the $\varkappa$-neighborhood of $H$ (see [20, p. 129]). Given such a quasiconvex subgroup, let $\mathcal{L}^{\prime}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\mathcal{L}\cap\mu^{-1}(H)$ and define a finite subset $\mathcal{B}$ of $H$ by

[TABLE]

Note that $\mathcal{B}=\mathcal{B}^{-1}$ since $\mathcal{A}$ is closed under inversion. Recalling the construction from [20, p. 138], given any word $W=a_{1}\dots a_{n}\in\mathcal{L}^{\prime}$, where $a_{1},\dots,a_{n}\in\mathcal{A}$, the quasiconvexity of $H$ implies that for each $i\in\{1,\dots,n\}$ there is $g_{i}\in G$ such that $|g_{i}|_{\mathcal{A}}\leq\varkappa$ and $\mu(a_{1}\dots a_{i})g_{i}\in H$. Clearly we can choose $g_{n}=1_{G}$ and let $g_{0}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=1_{G}$ so that $\mu(W)=\prod_{i=1}^{n}g_{i-1}^{-1}\mu(a_{i})g_{i}$ in $G$. This allows us to re-write the words from $\mathcal{L}^{\prime}$ as words from $\mathcal{B}^{*}$, possibly in a non-unique fashion due to some freedom in the choice of $g_{i}$. Let $\mathcal{M}\subseteq\mathcal{B}^{*}$ denote the resulting language, consisting of words from $\mathcal{L}^{\prime}$ re-written as words in $\mathcal{B}^{*}$ in such a way (with all possible $g_{i}$’s).

In [20, Theorem 3.1] Gersten and Short proved that $(\mathcal{B},\mathcal{M})$ is a biautomatic structure on $H$. For our purposes it will be convenient to modify the original biautomatic structure on $G$ as follows. Let $\mathcal{C}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\mathcal{A}\sqcup\mathcal{B}$ be the abstract union of the finite sets $\mathcal{A}$ and $\mathcal{B}$. Then $\mathcal{N}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\mathcal{L}\cup\mathcal{M}$ is a language in $\mathcal{C}^{*}$. Obviously $\mathcal{L}$ and $\mathcal{M}$ are still regular in $\mathcal{C}^{*}$, hence $\mathcal{N}$ is also a regular language in $\mathcal{C}^{*}$, as a union of regular languages (see [17, Lemma 1.4.1]).

One can define a map $\nu:\mathcal{C}\to G$, where $\nu|_{\mathcal{A}}=\mu$ and $\nu|_{\mathcal{B}}$ is the identity map on $\mathcal{B}$ (as $\mathcal{B}\subseteq G$, by definition), and extend it to a monoid homomorphism $\nu:\mathcal{C}^{*}\to G$. Clearly $\mathcal{C}$ is a finite generating set of $G$.

Consider any path $p$ in $\Gamma(G,\mathcal{C})$, labelled by a word $U\in\mathcal{M}$. Let $f\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=p_{-}\in G$, then $p_{+}$ belongs to the coset $fH$. By definition, $U$ is obtained from a word $W\in\mathcal{L}^{\prime}=\mathcal{L}\cap\mu^{-1}(H)\subseteq\mathcal{L}$ by applying the re-writing process above. Let $\tilde{p}$ denote the path in $\Gamma(G,\mathcal{C})$ labelled by $W$ and starting at $f=p_{-}$. Then ${\tilde{p}}_{+}=p_{+}$ and $|\tilde{p}|=|p|$. Moreover, the paths $p$ and $\tilde{p}$ $\varkappa$-fellow travel in $\Gamma(G,\mathcal{C})$, by construction (see Figure 1).

Lemma 4.1**.**

$(\mathcal{C},\mathcal{N})$* is a finite-to-one biautomatic structure on $G$.*

Proof.

We have already observed that $\mathcal{N}$ is a regular language in $\mathcal{C}^{*}$; moreover, $\nu(\mathcal{N})=G$ as $\nu(\mathcal{L})=G$ and $\mathcal{L}\subseteq\mathcal{N}$. Let us now check that the fellow travelling property holds.

Since $(\mathcal{A},\mathcal{L})$ is a biautomatic structure on $G$, there is some $\varepsilon\geq 0$ such that two paths in $\Gamma(G,\mathcal{A})$ with labels from $\mathcal{L}$ whose endpoints are at distance at most $1$ from each other $\varepsilon$-fellow travel. Consider any two paths $p$ and $q$ in $\Gamma(G,\mathcal{C})$ labelled by words from $\mathcal{N}$, with $\mathrm{d}_{\mathcal{C}}(p_{-},q_{-})\leq 1$ and $\mathrm{d}_{\mathcal{C}}(p_{+},q_{+})\leq 1$. Define the paths $p^{\prime}$ and $q^{\prime}$ in $\Gamma(G,\mathcal{C})$ as follows. If $p$ is labelled by a word from $\mathcal{L}$, then $p^{\prime}=p$; otherwise, if $p$ is labelled by a word from $\mathcal{M}$ then $p^{\prime}$ is the path $\tilde{p}$ defined above, labelled by a word from $\mathcal{L}$. We construct the path $q^{\prime}$ similarly, and note that since the labels of $p^{\prime}$ and $q^{\prime}$ are in $\mathcal{A}^{*}$, they can be considered as paths in $\Gamma(G,\mathcal{A})$.

Since $|b|_{\mathcal{A}}\leq 2\varkappa+1$ for any $b\in\mathcal{B}$, by (2), $p^{\prime}$ has the same endpoints as $p$, and $q^{\prime}$ has the same endpoints as $q$, we see that $\mathrm{d}_{\mathcal{A}}(p^{\prime}_{-},q^{\prime}_{-})\leq 2\varkappa+1$ and $\mathrm{d}_{\mathcal{A}}(p^{\prime}_{+},q^{\prime}_{+})\leq 2\varkappa+1$. Therefore the paths $p^{\prime}$ and $q^{\prime}$ $\zeta$-fellow travel in $\Gamma(G,\mathcal{A})$, where $\zeta\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\varepsilon(2\varkappa+1)$. Hence, these paths also $\zeta$-fellow travel in $\Gamma(G,\mathcal{C})$, as $\mathcal{A}\subseteq\mathcal{C}$. It follows that the original paths $p$ and $q$ $\lambda$-fellow travel in $\Gamma(G,\mathcal{C})$, where $\lambda\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\zeta+2\varkappa$. Thus $(\mathcal{C},\mathcal{N})$ is a biautomatic structure on $G$.

By our convention, the biautomatic structure $(\mathcal{A},\mathcal{L})$ on $G$ is finite-to-one, and, by the above construction, there are only finitely many possibilities for re-writing each word $W\in\mathcal{L}^{\prime}$ as a word in $\mathcal{B}^{*}$, hence the biautomatic structure $(\mathcal{B},\mathcal{M})$ on $H$ is also finite-to-one. It follows that the biautomatic structure $(\mathcal{C},\mathcal{N})$ on $G$ is finite-to-one as well. ∎

The new biautomatic structure $(\mathcal{C},\mathcal{N})$ on $G$ naturally extends the biautomatic structure $(\mathcal{B},\mathcal{M})$ on $H$, and the Cayley graph $\Gamma(H,\mathcal{B})$ is a subgraph of the Cayley graph $\Gamma(G,\mathcal{C})$. This allows us to define the action of the commensurator $\operatorname{Comm}_{G}(H)$ on the boundary of $\mathcal{M}$ as follows. Let $(U_{i})_{i\in\mathbb{N}}$ be a sequence of words from $\mathcal{M}$ tending to infinity and representing a boundary point $\alpha\in\partial\mathcal{M}$, and let $g\in\operatorname{Comm}_{G}(H)$. Since $H\cap gHg^{-1}$ has finite index in $gHg^{-1}$, there exist $f_{1},\dots,f_{k}\in G$ such that $gHg^{-1}\subseteq Hf_{1}\cup\dots\cup Hf_{k}$, thus, for any $gh\in gH$ there is $e\in G$, with $|e|_{\mathcal{C}}\leq\max\{|f_{j}g|_{\mathcal{C}}\mid j=1,\dots,k\}<\infty$, such that $ghe\in H$.

Note that $g\,\nu(U_{i})\in gH$ for all $i\in\mathbb{N}$, and choose any sequence of elements $e_{i}\in G$, $i\in\mathbb{N}$, such that $\sup\{|e_{i}|_{\mathcal{C}}\mid i\in\mathbb{N}\}<\infty$ and $g\,\nu(U_{i})\,e_{i}\in H$ for all $i\in\mathbb{N}$. Let $V_{i}\in\mathcal{M}$ be any word representing $g\,\nu(U_{i})\,e_{i}$, $i\in\mathbb{N}$. We claim that the sequence of words $(V_{i})_{i\in\mathbb{N}}$ tends to infinity and define its equivalence class in $\partial\mathcal{M}$ to be the result of the action of $g$ on $\alpha$.

Lemma 4.2**.**

The above construction gives a well-defined action of $\operatorname{Comm}_{G}(H)$ on $\partial\mathcal{M}$.

Proof.

Using the notation from the preceding paragraph, let us first check that the sequence of words $(V_{i})_{i\in\mathbb{N}}$ in $\mathcal{M}$ tends to infinity. Note that $\lim_{i\to\infty}|\nu(U_{i})|_{\mathcal{C}}=\infty$, as $\lim_{i\to\infty}|U_{i}|=\infty$, and the biautomatic structure $(\mathcal{C},\mathcal{N})$ is finite-to-one by Lemma 4.1. We can also observe that

[TABLE]

Therefore $\lim_{i\to\infty}|V_{i}|=\infty$.

For each $i\in\mathbb{N}$ let $p_{i}$ be the path in $\Gamma(G,\mathcal{C})$ starting at $g$ and labelled by $U_{i}$. Since the sequence $(U_{i})_{i\in\mathbb{N}}$, of words in $\mathcal{M}\subseteq\mathcal{N}$, tends to infinity, there exists $\zeta\geq 0$ such that $p_{i}$ $\zeta$-follows $p_{j}$ in $\Gamma(G,\mathcal{C})$, for any $i,j\in\mathbb{N}$ with $i\leq j$.

Let $q_{i}$ be the path in $\Gamma(G,\mathcal{C})$ starting at $1_{G}$ and labelled by $V_{i}$, $i\in\mathbb{N}$. Observe that for all $i\in\mathbb{N}$, $\mathrm{d}_{\mathcal{C}}((p_{i})_{-},(q_{i})_{-})=|g|_{\mathcal{C}}$ and $\mathrm{d}_{\mathcal{C}}((p_{i})_{+},(q_{i})_{+})=\mathrm{d}_{\mathcal{C}}(g\,\nu(U_{i}),g\,\nu(U_{i})e_{i})=|e_{i}|_{\mathcal{C}}$. Since $\sup\{|e_{i}|_{\mathcal{C}}\mid i\in\mathbb{N}\}<\infty$ and the structure $(\mathcal{C},\mathcal{N})$ is biautomatic, there exists $\eta\geq 0$ such that the paths $p_{i}$ and $q_{i}$ $\eta$-fellow travel, for all $i\in\mathbb{N}$. Therefore, for $\lambda\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=2\eta+\zeta$, the path $q_{i}$ $\lambda$-follows the path $q_{j}$ in $\Gamma(G,\mathcal{C})$, whenever $i,j\in\mathbb{N}$ and $i\leq j$. Let $\lambda^{\prime}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\max\{|h|_{\mathcal{B}}\mid h\in H,~{}|h|_{\mathcal{C}}\leq\lambda\}<\infty$. Since each $q_{i}$ is labelled by a word from $\mathcal{B}^{*}$, $q_{i}$ $\lambda^{\prime}$-follows $q_{j}$ as paths in $\Gamma(H,\mathcal{B})$, whenever $i\leq j$. Thus the sequence $(V_{i})_{i\in\mathbb{N}}$, of words in $\mathcal{M}$, tends to infinity.

Now, suppose that $(U_{i}^{\prime})_{i\in\mathbb{N}}$ is another sequence of words in $\mathcal{M}$ which tends to infinity and is equivalent to the sequence $(U_{i})_{i\in\mathbb{N}}$. For each $i\in\mathbb{N}$ choose an arbitrary element $e_{i}^{\prime}\in G$ such that $\sup\{|e_{i}^{\prime}|_{\mathcal{C}}\mid i\in\mathbb{N}\}<\infty$ and $g\,\nu(U_{i}^{\prime})\,e_{i}^{\prime}\in H$, and let $V_{i}^{\prime}$ be any word from $\mathcal{M}$ representing the element $g\,\nu(U_{i}^{\prime})\,e_{i}^{\prime}$. Let us show that the sequence $(V_{i}^{\prime})_{i\in\mathbb{N}}$ is equivalent to the sequence $(V_{i})_{i\in\mathbb{N}}$.

Let $p_{i}^{\prime}$ be the path in $\Gamma(G,\mathcal{C})$ starting at $g$ and labelled by $U_{i}^{\prime}$, and let $q_{i}^{\prime}$ be the path in $\Gamma(G,\mathcal{C})$ starting at $1_{G}$ and labelled by $V_{i}^{\prime}$, $i\in\mathbb{N}$. By the assumptions, there exists $\theta\geq 0$ such that the Hausdorff distance between the sequences $(p_{i})_{i\in\mathbb{N}}$ and $(p_{i}^{\prime})_{i\in\mathbb{N}}$ is at most $\theta$. On the other hand, the argument above shows that the paths $p_{i}^{\prime}$ and $q_{i}^{\prime}$ $\eta^{\prime}$-fellow travel, for some $\eta^{\prime}\geq 0$ and all $i\in\mathbb{N}$. It follows that the Hausdorff distance between the sequences $(q_{i})_{i\in\mathbb{N}}$ and $(q_{i}^{\prime})_{i\in\mathbb{N}}$ does not exceed $\theta+\eta+\eta^{\prime}$ in $\Gamma(G,\mathcal{C})$. Since $q_{i}$ and $q_{i}^{\prime}$ are labelled by words from $\mathcal{B}^{*}$, for each $i\in\mathbb{N}$, the Hausdorff distance between the sequences of these paths is also bounded in $\Gamma(H,\mathcal{B})$ by the constant $\max\{|h|_{\mathcal{B}}\mid h\in H,~{}|h|_{\mathcal{C}}\leq\theta+\eta+\eta^{\prime}\}$. Thus the sequences of words $(V_{i})_{i\in\mathbb{N}}$ and $(V_{i}^{\prime})_{i\in\mathbb{N}}$ indeed give rise to the same boundary point in $\partial\mathcal{M}$. This shows that the above action is well-defined.

It remains to check that the axioms of a group action are satisfied. Let $g,g^{\prime}\in\operatorname{Comm}_{G}(H)$ and let $(U_{i})_{i\in\mathbb{N}}$ be a sequence of words from $\mathcal{M}$ converging to a boundary point $\alpha\in\partial\mathcal{M}$. Obviously $1_{G}\,\alpha=\alpha$, so this axiom is satisfied. On the other hand, by the definition of the action, the point $g^{\prime}(g\,\alpha)\in\partial\mathcal{M}$ is obtained from a sequence of words in $\mathcal{M}$ representing the elements $g^{\prime}g\,\nu(U_{i})\,e_{i}e_{i}^{\prime}$, where $e_{i},e_{i}^{\prime}\in G$, $i\in\mathbb{N}$, $\sup\{|e_{i}|_{\mathcal{C}},|e_{i}^{\prime}|_{\mathcal{C}}\mid i\in\mathbb{N}\}<\infty$ and $g\,\nu(U_{i})\,e_{i}\in H$, $g^{\prime}g\,\nu(U_{i})\,e_{i}e_{i}^{\prime}\in H$, for all $i\in\mathbb{N}$. Set $e_{i}^{\prime\prime}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=e_{i}e_{i}^{\prime}\in G$, and observe that since $|e_{i}^{\prime\prime}|_{\mathcal{C}}\leq|e_{i}|_{\mathcal{C}}+|e_{i}^{\prime}|_{\mathcal{C}}$ for all $i\in\mathbb{N}$, $\sup\{|e_{i}^{\prime\prime}|_{\mathcal{C}}\mid i\in\mathbb{N}\}<\infty$. Therefore the boundary point $(g^{\prime}g)\,\alpha$ can be obtained from the same sequence of words in $\mathcal{M}$, representing the same elements $g^{\prime}g\,\nu(U_{i})\,e_{i}^{\prime\prime}$, $i\in\mathbb{N}$, that were used for the point $g^{\prime}(g\,\alpha)$. Thus $(g^{\prime}g)\,\alpha=g^{\prime}(g\,\alpha)$, which completes the proof of the lemma. ∎

5 The case of an abelian subgroup

In this section we will assume that $G$ is a group equipped with a biautomatic structure $(\mathcal{C},\mathcal{N})$, and $H\leqslant G$ is a finitely generated abelian subgroup with a biautomatic structure $(\mathcal{B},\mathcal{M})$, where $\mathcal{B}\subseteq\mathcal{C}$ and $\mathcal{M}=\mathcal{N}\cap\mathcal{B}^{*}$. As explained in Section 4, we can find such biautomatic structures on $G$ and $H$ starting from any biautomatic structure $(\mathcal{A},\mathcal{L})$ on $G$, as long as $H$ is $\mathcal{L}$-quasiconvex. We also define the action of the commensurator $\operatorname{Comm}_{G}(H)$ on the boundary $\partial\mathcal{M}$ as explained in that section. Let $\mu:\mathcal{B}^{*}\to H$ and $\nu:\mathcal{C}^{*}\to G$ denote the monoid homomorphisms sending the words to the group elements they represent.

The next theorem is the main result of this section.

Theorem 5.1**.**

Using the notation from the beginning of the section, suppose that an element $g\in\operatorname{Comm}_{G}(H)$ acts trivially on the boundary $\partial\mathcal{M}$. Then $g$ centralizes a finite-index subgroup of $H$.

The proof of Theorem 5.1 will require two auxiliary statements.

Lemma 5.2**.**

Suppose that $g\in\operatorname{Comm}_{G}(H)$ fixes a simple boundary point $\alpha\in\partial\mathcal{M}$, given by a sequence of words $(SW^{i}T)_{i\in\mathbb{N}}$ in $\mathcal{M}$ tending to infinity, where $S,W,T\in\mathcal{B}^{*}$. Then there exists $m\in\mathbb{N}$ such that $gh^{m}g^{-1}=h^{m}$ in $G$, where $h\in H$ is the element represented by the word $W$.

Proof.

Since $g\in\operatorname{Comm}_{G}(H)$, we can choose elements $e_{i}\in G$, $i\in\mathbb{N}$, so that $\eta\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\sup\{|e_{i}|_{\mathcal{C}}\mid i\in\mathbb{N}\}<\infty$ and $g\,\nu(SW^{i}T)\,e_{i}\in H$ for all $i\in\mathbb{N}$. By the definition of the action, given in Section 4, $g\,\alpha$ is the equivalence class of a sequence of words $(V_{i})_{i\in\mathbb{N}}$, where $V_{i}\in\mathcal{M}$ represents the element $g\,\nu(SW^{i}T)\,e_{i}$, $i\in\mathbb{N}$.

For each $i\in\mathbb{N}$ let $p_{i}$ be the path in $\Gamma(G,\mathcal{C})$ starting at $g$ and labelled by the word $SW^{i}T$, let $q_{i}$ and $r_{i}$ be the paths in $\Gamma(G,\mathcal{C})$ starting at $1_{G}$ and labelled by the words $V_{i}$ and $SW^{i}T$, respectively (see Figure 2). Note that $\mathrm{d}_{\mathcal{C}}((p_{i})_{+},(q_{i})_{+})=|e_{i}|_{\mathcal{C}}\leq\eta$ and, since $g\,\alpha=\alpha$, there are $\theta\geq 0$, independent of $i$, and $j=j(i)\in\mathbb{N}$, such that $(q_{i})_{+}$ is at most $\theta$-away from some vertex of $r_{j}$. Obviously, any vertex of $r_{j}$ is at most $\varkappa$-away from $(r_{k})_{+}$, for some $k\leq j$ and $\varkappa\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=|S|+|W|+|T|$.

Thus for every $i\in\mathbb{N}$ there is $k=k(i)\in\mathbb{N}$ such that $\mathrm{d}_{\mathcal{C}}((p_{i})_{+}),(r_{k})_{+})\leq\eta+\theta+\varkappa$. On the other hand, recall that $\mathrm{d}_{\mathcal{C}}((p_{i})_{-},(r_{k})_{-})=\mathrm{d}_{\mathcal{C}}(g,1_{G})=|g|_{\mathcal{C}}$. Since the paths $p_{i}$ and $r_{k}$ are both labelled by words from $\mathcal{M}\subseteq\mathcal{N}$ and $(\mathcal{C},\mathcal{N})$ is a biautomatic structure on $G$, there is $\lambda\geq 0$ such that these paths $\lambda$-fellow travel in $\Gamma(G,\mathcal{C})$ for all $i\in\mathbb{N}$ and $k=k(i)$.

Now, the word $W$ cannot be empty by the assumptions, and since the structure $(\mathcal{C},\mathcal{N})$ is finite-to-one, $\nu(W)=h$ must have infinite order in $G$. It follows that $\lim_{i\to\infty}\mathrm{d}_{\mathcal{C}}((p_{i})_{-},(p_{i})_{+})=\infty$, hence $k(i)$ must also tend to infinity as $i\to\infty$. Therefore there is $i_{0}\in\mathbb{N}$ such that both $i_{0}$ and $k_{0}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=k(i_{0})$ are greater than $\xi\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=|\{f\in G\mid|f|_{\mathcal{C}}\leq\lambda\}|+1$.

For each $l\in\{1,\dots,\xi\}$ let $u_{l}\in G$ be the vertex of $p_{i_{0}}$ such that the subpath of $p_{i_{0}}$ from $g=(p_{i_{0}})_{-}$ to $u_{l}$ is labelled by $SW^{l}$. Similarly, we define $v_{l}\in G$ to be the vertex of the path $r_{k_{0}}$ such that the subpath of $r_{k_{0}}$ from $1_{G}=(r_{k_{0}})_{-}$ to $v_{l}$ is labelled by $SW^{l}$. By construction, $|v_{l}^{-1}u_{l}|_{\mathcal{C}}=\mathrm{d}_{\mathcal{C}}(u_{l},v_{l})\leq\lambda$ for every $l=1,\dots,\xi$, and the definition of $\xi$ implies that there must exist indices $l_{1},l_{2}$, $1\leq l_{1}<l_{2}\leq\xi$, such that $v_{l_{1}}^{-1}u_{l_{1}}=v_{l_{2}}^{-1}u_{l_{2}}$ in $G$. Set $z\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=v_{l_{1}}^{-1}u_{l_{1}}$, then the quadrilateral in $\Gamma(G,\mathcal{C})$ with vertices $v_{l_{1}}$, $u_{l_{1}}$, $u_{l_{2}}$ and $v_{l_{2}}$ (see Figure 2) gives rise to the equality $z\,\nu(W^{m})\,z^{-1}=\nu(W^{m})$ in $G$, where $m\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=l_{2}-l_{1}\in\mathbb{N}$. Thus $z$ commutes with $h^{m}$ in $G$. On the other hand, the quadrilateral with vertices $1_{G}$, $g$, $u_{l_{1}}$ and $v_{l_{1}}$ in $\Gamma(G,\mathcal{C})$ gives rise to the equality $g=\nu(SW^{l_{1}})\,z\,\nu(SW^{l_{1}})^{-1}$ in $G$. Since $SW^{l_{1}}$ is a word from $\mathcal{B}^{*}$ and $H$ is abelian, the element $\nu(SW^{l_{1}})$ commutes with $h^{m}$ in $G$, therefore $g$ also commutes with $h^{m}$ and the lemma is proved. ∎

Lemma 5.3**.**

Let $\mathfrak{A}$ be a finite state automaton accepting the language $\mathcal{M}$, and let $D$ be the set of elements of $H$ represented by the labels of non-trivial simple cycles in $\mathfrak{A}$. Then $D$ generates a finite-index subgroup of $H$.

Proof.

Let $n$ be the number of states in the automaton $\mathfrak{A}$. We claim that $H\subseteq E\langle D\rangle$, where $E\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\{h\in H\mid|h|_{\mathcal{B}}\leq n\}$.

Indeed, choose any element $f\in H$ and let $V$ be the shortest word from $\mathcal{M}$ representing $f$ in $H$. We will prove that $f\in E\langle D\rangle$ by induction on the length of $V$. If $|V|\leq n$ then $f\in E$. Otherwise, if $|V|>n$, any path $v$ in $\mathfrak{A}$ labelled by $V$, from the initial state to an accept state of $\mathfrak{A}$ (which exists as this automaton accepts $\mathcal{M}$), must contain a non-trivial simple cycle based at some state $x$ and labelled by a word $W$, with $\mu(W)\in D$. Thus $V=SWT$, where $S$, $T$ are labels of some subpaths of $v$ ending and beginning at $x$ respectively. Clearly $ST\in\mathcal{M}$, as this word is accepted by $\mathfrak{A}$, but its length is strictly smaller than the length of $V$. Moreover, since $H$ is abelian, we have $f=\mu(V)=\mu(ST)\mu(W)\in\mu(ST)D$. By the induction hypothesis, $\mu(ST)\in E\langle D\rangle$, so $f\in E\langle D\rangle D=E\langle D\rangle$, and the claim is proved.

Since $E$ is a finite set, by definition, and $\langle D\rangle\leqslant H$, the inclusion $H\subseteq E\langle D\rangle$ implies that $|H:\langle D\rangle|<\infty$, as required. ∎

Proof of Theorem 5.1.

Let $\mathfrak{A}$ be a finite state automaton accepting the language $\mathcal{M}$, and let $\mathcal{W}=\{W_{1},\dots,W_{n}\}$ be the list of the labels of all non-trivial simple cycles in $\mathfrak{A}$. Let $D$ denote the (finite) set of elements of $H$ represented by the words from $\mathcal{W}$.

Take any $h\in D$, then $h=\mu(W_{k})$ for some $k\in\{1,\dots,n\}$. Let $x$ be the state of $\mathfrak{A}$ at which a cycle labelled by $W_{k}$ is based, and choose some simple paths $S$ and $T$ joining the initial state of $\mathfrak{A}$ with $x$ and $x$ with an accept state of $\mathfrak{A}$ respectively. Then, according to Lemma 2.6, the sequence of words $(SW_{k}^{i}T)_{i\in\mathbb{N}}$ converges to some point $\alpha\in\partial\mathcal{M}$. By the assumptions, $g\,\alpha=\alpha$, so we can use Lemma 5.2 to conclude that $gh^{m}g^{-1}=h^{m}$ in $G$, for some $m=m(h)\in\mathbb{N}$. Since the latter holds for every $h\in D$ and $|D|<\infty$, we can find a single $l\in\mathbb{N}$ such that $g$ commutes with $h^{l}$ for all $h\in D$.

Now, the elements $\{h^{l}\mid h\in D\}$ obviously generate a finite-index subgroup $H^{\prime}$ of the finitely generated abelian group $\langle D\rangle$, which itself has finite index in $H$, by Lemma 5.3. Thus $|H:H^{\prime}|<\infty$ and $H^{\prime}$ is centralized by $g$, as required. ∎

We can now prove Theorem 1.2 stated in the introduction.

Proof of Theorem 1.2.

Define a biautomatic structure $(\mathcal{B},\mathcal{M})$ on the subgroup $L$ as in Section 4. This gives rise to an action of $\operatorname{Comm}_{G}(H)$ on the finite set $\partial\mathcal{M}$ (see Lemma 4.2 and Corollary 3.2), and we denote the kernel of this action by $\operatorname{Comm}_{G}^{0}(H)$. Then $\operatorname{Comm}_{G}^{0}(H)\lhd\operatorname{Comm}_{G}(H)$ and $|\operatorname{Comm}_{G}(H):\operatorname{Comm}_{G}^{0}(H)|<\infty$.

By Theorem 5.1, every element of $\operatorname{Comm}_{G}^{0}(H)$ centralizes a finite-index subgroup of $H$, hence $\operatorname{Comm}_{G}^{0}(H)$ lies in the kernel of the homomorphism from $\operatorname{Comm}_{G}(H)$ to $\operatorname{Comm}(H)$. It follows that the image of $\operatorname{Comm}_{G}(H)$ in $\operatorname{Comm}(H)$ is finite. Any finite subset of $\operatorname{Comm}_{G}^{0}(H)$ centralizes a finite-index subgroup of $H$, thus the same holds for any finitely generated subgroup $F\leqslant\operatorname{Comm}_{G}^{0}(H)$. ∎

The main examples of $\mathcal{L}$-quasiconvex subgroups in biautomatic groups are centralizers of finite subsets (see [17, Corollary 8.3.5 and Theorem 8.3.1]). Therefore the following statement is an immediate corollary of Theorem 1.2.

Corollary 5.4**.**

Let $G$ be a biautomatic group and let $X\subseteq G$ be a finite subset such that $H\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\mathrm{C}_{G}(X)$ is abelian. Then there is a finite-index subgroup $\operatorname{Comm}_{G}^{0}(H)\lhd\operatorname{Comm}_{G}(H)$ such that every finitely generated subgroup of $\operatorname{Comm}_{G}^{0}(H)$ centralizes a finite-index subgroup of $H$ in $G$.

Remark 5.5*.*

In [22, Proposition 9.1] Huang and Prytuła use an example from Wise’s thesis [40] to show that there exists a group $G$, acting properly discontinuously, cocompactly and cellularly on a product of two trees, and an infinite cyclic subgroup $H\leqslant G$ such that $\operatorname{Comm}_{G}(H)$ is not finitely generated and does not normalize any finite-index subgroup of $H$.

Since the product of two trees is a CAT(0) cube complex, the group $G$ is biautomatic by [32]. After analysing the construction it becomes clear that one can replace $H$ with a commensurable infinite cyclic subgroup to ensure that $H=\mathrm{C}_{G}(X)$ for some finite subset $X\subseteq G$. Therefore, the examples of $G$ and $H$ show that it is indeed necessary to pass to finitely generated subgroups of $\operatorname{Comm}_{G}(H)$ in Theorem 1.2 and Corollary 5.4.

6 Commensuration and the Flat Torus Theorem

Let us start this section by recalling the Flat Torus Theorem [12, II.7.1]. Throughout this section we will use additive notation for the group operation on a free abelian group $L$.

Theorem 6.1**.**

Let $L$ be a free abelian group of rank $n$ acting properly by semi-simple isometries on a CAT(0) space $X$. Then:

The min set $M$ for $L$ is non-empty and $M=Y\times\mathbb{E}^{n}$. 2. 2.

Every $c\in L$ leaves $M$ invariant, respects the product decomposition, and acts trivially on $Y$ and by translation on $\mathbb{E}^{n}$. 3. 3.

For $y\in Y$, the quotient $(\{y\}\times\mathbb{E}^{n})/L$ is an $n$-torus. 4. 4.

If an isometry of $X$ normalizes $L$ then it preserves $M$ and the direct product decomposition. 5. 5.

If a group $\Gamma$ of isometries of $X$ normalizes $L$, then a finite-index subgroup of $\Gamma$ centralizes $L$. If $\Gamma$ is finitely generated, then $\Gamma L$ has a finite-index subgroup containing $L$ as a direct factor.

We want an analogous statement to (4) above, but for isometries that lie in the commensurator of $L$ rather than in its normalizer. For this it is easier first to describe a different statement that is equivalent to (3).

Recall that a torsor for an abelian group is a non-empty set on which it acts freely and transitively. An affine space is naturally a torsor for its vector space of translations.

Remark 6.2*.*

Let $L$ be a free abelian group of finite rank $n$, and suppose that $L$ acts by translations on a finite-dimensional real affine space $\mathbb{A}$. The following are equivalent:

• •

the action of $L$ is properly discontinuous and cocompact;

• •

the unique affine extension of the action to $L\otimes\R$ makes $\mathbb{A}$ a torsor for $L\otimes\R$.

In fact, the affine extension to $L\otimes\R$ is free if and only if the action of $L$ on $\mathbb{A}$ is properly discontinuous, and in this case $\mathbb{A}$ is a torsor for $L\otimes\R$ if and only if $\mathbb{A}$ has dimension $n$.

Now suppose that one is given a torsor action of a vector space $V$ by translations on the Euclidean space $\mathbb{E}^{n}$. In this case, the Euclidean distance on $\mathbb{E}^{n}$ enables one to define an inner product on $V$, via \displaystyle\langle v,w\rangle\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\frac{1}{2}\bigl{(}d((v+w)\,x,x)^{2}-d(v\,x,x)^{2}-d(w\,x,x)^{2}\bigr{)}, for any $x\in\mathbb{E}^{n}$.

In particular, with hypotheses and notation as in Theorem 6.1, we may define an inner product $\langle\,\cdot\,,\,\cdot\,\rangle_{L}$ on $L\otimes\R$ by setting

[TABLE]

for each $b,c\in L$, and extending linearly to $L\otimes\R$. Here $x\in M$, the min set of $L$, and Theorem 6.1(2) tells us that the definition does not depend on which $x$ we choose. The fact that this is an inner product follows easily from the cosine rule. The following observation is an immediate consequence of the definitions and Theorem 6.1.

Remark 6.3*.*

Let $L$ be a finitely generated free abelian group acting properly by isometries on a CAT(0) space $X$, and let $L^{\prime}\leqslant L$ be any subgroup. Then the min set of $L$ is contained in the min set of $L^{\prime}$, and for all $b,c\in L^{\prime}$ we have $\langle b,c\rangle_{L^{\prime}}=\langle b,c\rangle_{L}$.

We are now ready state our addendum to the Flat Torus Theorem.

Theorem 6.4**.**

Let $L$ be a free abelian group of rank $n$ acting properly by semi-simple isometries on a CAT(0) space $X$. Then:

• (1)

The min set $M$ for $L$ is non-empty and $M=Y\times\mathbb{E}^{n}$.

• (2)

Every $c\in L$ leaves $M$ invariant, respects the product decomposition, and acts trivially on $Y$ and by translation on $\mathbb{E}^{n}$.

• ($3^{\prime}$)

For each $y\in Y$, $\{y\}\times\mathbb{E}^{n}$ is a torsor for $L\otimes\R$ under the affine extension of the action of $L$.

• ($4^{\prime}$)

For any isometry $\varphi$ of $X$ that commensurates $L$, the image of $\varphi$ in $GL(L\otimes\Q)\leqslant GL(L\otimes\R)$ preserves the inner product $\langle\cdot,\cdot\rangle_{L}$.

Proof.

Statements (1) and (2) are parts of the usual Flat Torus Theorem (Theorem 6.1), and are restated here for convenience. By Remark 6.2, ($3^{\prime}$) is equivalent to Theorem 6.1(3). It remains to establish ($4^{\prime}$).

Since $\varphi$ commensurates $L$, conjugation by it in the group of the isometries of $X$ induces an isomorphism $\phi:L^{\prime}\rightarrow L^{\prime\prime}$, for some finite-index subgroups $L^{\prime}$ and $L^{\prime\prime}$ of $L$. Let $M^{\prime}$ and $M^{\prime\prime}$ be the min sets in $X$ for $L^{\prime}$ and $L^{\prime\prime}$ respectively. Note that $M$, the min set for $L$, is contained in both $M^{\prime}$ and $M^{\prime\prime}$, and that $\varphi$ restricts to an isometry from $M^{\prime}$ to $M^{\prime\prime}$. It follows that $\phi:L^{\prime}\rightarrow L^{\prime\prime}$ respects their inner products, in the sense that for each $b,c\in L^{\prime}$, $\langle\phi(b),\phi(c)\rangle_{L^{\prime\prime}}=\langle b,c\rangle_{L^{\prime}}$. Since $L^{\prime}$ and $L^{\prime\prime}$ have finite index in $L$, there exists $m\in\mathbb{N}$ so that for all $b\in L$, $mb\in L^{\prime},L^{\prime\prime}$. Remark 6.3 implies that $\langle\,\cdot\,,\,\cdot\,\rangle_{L}$, $\langle\,\cdot\,,\,\cdot\,\rangle_{L^{\prime}}$ and $\langle\,\cdot\,,\,\cdot\,\rangle_{L^{\prime\prime}}$ are all equal on the finite-index subgroup $L^{\prime}\cap L^{\prime\prime}$ of $L$. Hence for any $b,c\in L$,

[TABLE]

where ${\tilde{\phi}}$ is the image of $\phi$ in $GL(L\otimes\Q)$, as defined in Subsection 2.1. ∎

7 Commensurating HNN-extensions of free abelian groups

Let $L$ be a finitely generated free abelian group, and let $\phi:L^{\prime}\rightarrow L^{\prime\prime}$ be an isomorphism between finite-index subgroups of $L$. Define a group $G(L,\phi,L^{\prime})$ as the HNN-extension of $L$ in which the stable letter conjugates $L^{\prime}$ to $L^{\prime\prime}$ via $\phi$:

[TABLE]

In the case when we are given a basis for $L\cong\Z^{n}$ and $\phi$ is described by a matrix, we simplify the notation slightly. For $A\in GL(n,\Q)$ and $L^{\prime}$ a finite-index subgroup of $L\cap A^{-1}\,L=\Z^{n}\cap A^{-1}\,\Z^{n}$, we write $G(A,L^{\prime})$ for the HNN-extension defined as above:

[TABLE]

If in this case, $L^{\prime}$ is as large as possible, i.e., $L^{\prime}=L\cap A^{-1}\,L$, then we write $G(A)$ instead of $G(A,L^{\prime})$.

When $n=1$, the groups $G(A,L^{\prime})$ are precisely the Baumslag-Solitar groups; if $A$ is a $1\times 1$ matrix with entry $m/d$, then $G(A,d\Z)=BS(m,d)$ and if $(m,d)=1$ then $G(A,d\Z)=G(A)$.

Proposition 7.1**.**

Each group $G=G(A,L^{\prime})$ is free-by-abelian-by-cyclic.

Proof.

There is an affine action of $G$ on $L\otimes\R\cong\R^{n}$ in which elements of $L$ act as translations and $t$ acts as multiplication by the matrix $A$. Let $\alpha:G\to AGL(n,\R)$ denote the resulting homomorphism, where $AGL(n,\R)$ is the group of affine transformations of $\R^{n}$. The subgroup $L$ is in the kernel of the standard map $\beta:AGL(n,\R)\to GL(n,\R)$, and hence the image $(\beta\circ\alpha)(G)$ is cyclic. Since $\ker\beta\cong\R^{n}$, we can deduce that $\alpha(G)$ is abelian-by-cyclic.

Evidently the intersection $\ker\alpha\cap L$ is trivial, which implies that $\ker\alpha$ acts freely on the Bass-Serre tree for $G$ expressed as an HNN-extension of $L$. Hence this kernel is free, so $G$ is free-by-abelian-by-cyclic. ∎

See [23] for a stronger result in the case $n=1$.

Theorem 7.2**.**

The group $G(A,L^{\prime})=G(\Z^{n},A,L^{\prime})$ is a CAT(0) group if and only if the matrix $A$ is conjugate in $GL(n,\R)$ to an orthogonal matrix.

Proof.

Let $L=\Z^{n}$ and consider any $P\in GL(n,\R)$. Let $\Lambda\leqslant\R^{n}$ be the lattice $P\,L$, let $\Lambda^{\prime}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=P\,L^{\prime}$, and let $B\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=PAP^{-1}$. There is a group isomorphism from $G(A,L^{\prime})$ to $H\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=G(\Lambda,B\times,\Lambda^{\prime})$, given by $c\mapsto P\,c$ for $c\in L$, and $t\mapsto t$.

Now suppose that $B$ is an orthogonal matrix. In this case, there is a homomorphism from $H\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=G(\Lambda,B\times,\Lambda^{\prime})$ to the group of isometries of $\mathbb{E}^{n}$, in which elements of $\Lambda$ act naturally as translations and $t$ acts as multiplication by $B$. This action is not properly discontinuous, but its restriction to each conjugate of $\Lambda$ is free and properly discontinuous. Now let $T$ be the Bass-Serre tree for $H$ expressed as an HNN-extension of $\Lambda$. The stabilizer of each vertex of $T$ is a conjugate of $\Lambda$ and the stabilizer of each edge of $T$ is a conjugate of $\Lambda^{\prime}$. Consider the diagonal action of $H$ on the product $\mathbb{E}^{n}\times T$. Since edge and vertex stabilizers for the action of $H$ on $T$ act freely properly discontinuously and cocompactly on $\mathbb{E}^{n}$, it follows that the diagonal action of $H$ on the product $\mathbb{E}^{n}\times T$ is free, properly discontinuous, cocompact and isometric (for the product metric on $\mathbb{E}^{n}\times T$, which is CAT(0) [12, Example II.1.15(3)]). Hence $H$ is a CAT(0) group.

For the converse, if $G=G(A,L^{\prime})$ is a CAT(0) group, then since $t$ is in $\operatorname{Comm}_{G}(L)$, it follows that the action of $t$ preserves an inner product on $L\otimes\R=\R^{n}$ by Theorem 6.4. But this action is just multiplication by $A$. Hence $A$ preserves an inner product on $\R^{n}$ and so (since all $n$-dimensional real inner product spaces are isomorphic), $A$ is conjugate in $GL(n,\R)$ to an orthogonal matrix. ∎

Remark 7.3*.*

There is another way to describe the CAT(0) space constructed in the above proof. Suppose that $B$ is an orthogonal matrix, $\Lambda\leqslant\R^{n}$ is a lattice and multiplication by $B$ induces an isomorphism of finite-index sublattices $B\times:\Lambda^{\prime}\rightarrow\Lambda^{\prime\prime}$. Then multiplication by $B$ induces an isometry of tori from $\mathbb{T}^{\prime}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\R^{n}/\Lambda^{\prime}$ to $\mathbb{T}^{\prime\prime}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\R^{n}/\Lambda^{\prime\prime}$. Take the torus $\mathbb{T}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\R^{n}/\Lambda$, and the direct product $\mathbb{T}^{\prime}\times[0,1]\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\R^{n}/\Lambda^{\prime}\times[0,1]$. Glue the subspace $\mathbb{T}^{\prime}\times\{0\}\cong\mathbb{T}^{\prime}$ to $\mathbb{T}$ via the covering map $\mathbb{T}^{\prime}\rightarrow\mathbb{T}$, which is a local isometry, and glue the subspace $\mathbb{T}^{\prime}\times\{1\}\cong\mathbb{T}^{\prime}$ to $\mathbb{T}$ by the composite of multiplication by $B$ (an isometry $\mathbb{T}^{\prime}\rightarrow\mathbb{T}^{\prime\prime}$) and the covering map $\mathbb{T}^{\prime\prime}\rightarrow\mathbb{T}$. By the gluing lemma [12, II.11.13], the resulting space is locally CAT(0). The universal cover of this space with its group of deck transformations is of course the direct product $\mathbb{E}^{n}\times T$ with the isometric action of $G(\Lambda,B\times,\Lambda^{\prime})$ as described in the proof above.

Corollary 7.4**.**

If $A\in GL(n,\Q)$ is conjugate in $GL(n,\R)$ to an orthogonal matrix and $L^{\prime}\neq L$ then $G(A,L^{\prime})$ is quasi-isometric to $\Z^{n}\times F$, the direct product of a free abelian group and a finite rank non-abelian free group $F$.

Proof.

Let $F$ be the free group of rank $m=|L:L^{\prime}|>1$. Since the determinant of $A$ is $\pm 1$, $m=|L:L^{\prime\prime}|$ too. Hence the Bass-Serre tree $T$, for the decomposition of $G(A,L^{\prime})$ as an HNN-extension, is a regular tree of valency $2m$, and so both $G$ and $\Z^{n}\times F$ have natural isometric geometric actions on $\mathbb{E}^{n}\times T$. By the Švarc-Milnor lemma [12, I.8.19] they are quasi-isometric to each other. ∎

Theorem 7.5**.**

Suppose that $G=G(A,L^{\prime})$ where $A$ has infinite order and is conjugate in $GL(n,\R)$ to an orthogonal matrix. Then $G$ is a lattice in $\mathrm{Isom}(\mathbb{E}^{n})\times\mathrm{Isom}(T)$ whose projections to the factors are not discrete.

Proof.

Since $A$ is conjugate to a matrix in $O(n)$, $G$ acts isometrically on $\mathbb{E}^{n}$, and the action on the Bass-Serre tree $T$ is always isometric. Since $G$ acts freely, properly discontinuously, cocompactly and isometrically on $\mathbb{E}^{n}\times T$, it follows that $G$ is a lattice in $\mathrm{Isom}(\mathbb{E}^{n})\times\mathrm{Isom}(T)$. Since $A$ has infinite order, the element $t\in G$ acts on $\mathbb{E}^{n}$ as an infinite order element of the point stabilizer, which is compact (and isomorphic to $O(n)$), showing that the projection of $G$ to $\mathrm{Isom}(\mathbb{E}^{n})$ is not discrete.

The vertex stabilizers of the action of $G$ on $T$ are conjugates of the subgroup $L\leqslant G$, hence the kernel of this action is the core $c(L)$ of $L$, i.e., the intersection of all conjugates of $L$ in $G$. Provided that $c(L)$ has infinite index in $L$, it will follow that the image of $L$ in $\mathrm{Isom}(T)$ is an infinite subgroup of the vertex stabilizer, which is a compact (profinite) group (because the tree $T$ is locally finite). But $c(L)$ is a normal abelian subgroup of $G$, and so, by the Flat Torus Theorem (Theorem 6.1), there is $k\in\mathbb{N}$ such that $t^{k}$ centralizes $c(L)$, and thus $t^{k}$ acts trivially on $c(L)\otimes\R$. But $t^{k}$ cannot act trivially on $L\otimes\R$ because $A$ has infinite order, and so $c(L)\otimes\R$ must be a proper subspace of $L\otimes\R$. It follows that $|L:c(L)|=\infty$, and so the image of $L$ (and, hence, of $G$) in $\mathrm{Isom}(T)$ is not discrete. ∎

In the case when $n=2$, it follows that $G$ as in the above statement is an irreducible lattice in $\mathrm{Isom}(\mathbb{E}^{2})\times\mathrm{Isom}(T)$, because the matrix $A$ acts irreducibly on $\mathbb{E}^{2}$. This is not necessarily the case for larger $n$. For example, if $G$ satisfies the hypotheses, then so does $\Z\times G\leqslant\mathrm{Isom}(\mathbb{E}^{1})\times\mathrm{Isom}(\mathbb{E}^{n})\times\mathrm{Isom}(T)\leqslant\mathrm{Isom}(\mathbb{E}^{n+1})\times\mathrm{Isom}(T)$. As mentioned in the introduction, the existence of irreducible lattices in $\mathrm{Isom}(\mathbb{E}^{2})\times\mathrm{Isom}(T)$ contradicts [13, Theorem 1.3(i), Proposition 3.6, Theorem 3.8].

8 Characterizing biautomaticity of the groups $G(A,L^{\prime})$

Suppose that $L=\Z^{n}$, for some $n\in\mathbb{N}$, $A\in GL(n,\Q)$ and $L^{\prime}\leqslant L$ is a finite-index subgroup such that $L^{\prime\prime}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=A\,L^{\prime}$ is contained in $L$. In this section we study the (virtual) biautomaticity of the groups $G(A,L^{\prime})$ defined in (4).

Lemma 8.1**.**

If $L^{\prime}$ and $L^{\prime\prime}$ are both proper subgroups of $L$ then $L$ is self-centralizing in $G(A,L^{\prime})$. In particular, if neither $A$ nor $A^{-1}$ is an integer matrix, then $L$ is self-centralizing.

Proof.

Let $T$ be the Bass-Serre tree for $G=G(A,L^{\prime})$ expressed as an HNN-extension of $L$. The centralizer of $L$ will act on the set of $L$-fixed points $T^{L}$, which is a subtree (because the unique geodesic path between two fixed points must also be fixed). The vertex corresponding to the identity coset of $L$ is fixed by $L$, but the hypotheses imply that no edge that is incident with this vertex can be fixed by $L$. Hence the fixed point set for the action of $L$ on $T$ is this single vertex. The first claim follows since $L$ is the full stabilizer of this vertex.

The second claim follows from the first, because the largest possible choices for $L^{\prime}$ and $L^{\prime\prime}$ are $L^{\prime}=A^{-1}\,L\cap L$ and $L^{\prime\prime}=A\,L\cap L$. ∎

Recall that a subgroup $G$ of a direct product $H\times F$, of two groups $H$ and $F$, is said to be a subdirect product if the restrictions to $G$ of the natural projections $H\times F\to H$ and $H\times F\to F$ are surjective.

The following criterion for biautomaticity will be useful.

Proposition 8.2**.**

Let $H$ be a finitely generated virtually abelian group and let $F$ be a biautomatic group. Then every subdirect product $G\leqslant H\times F$ is also biautomatic.

Proof.

Abusing the notation we identify any subgroup $S\leqslant H$ with the subgroup $S\times\{1\}$ of $H\times F$. Since $G\leqslant H\times F$ is subdirect, the subgroup $N\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=G\cap H$ is normal in $H$ (cf. [29, Lemma 2.1]). Now, by [30, Lemma 4.2], there exists a normal subgroup $R\lhd H$ which intersects $N$ trivially and such that $|H:NR|<\infty$. Let $H_{1}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=H/R$ and $\phi:H\times F\to H_{1}\times F$ be the natural homomorphism whose kernel is $R$.

By construction, $R$ has trivial intersection with $G$ in $H\times F$, hence $\phi(G)\cong G$. Evidently $\phi(G)$ is still subdirect in $H_{1}\times F$. Moreover, $\phi(N)\subseteq\phi(G)\cap H_{1}$ has finite index in $H_{1}$ as $|H:NR|<\infty$, which implies that $\phi(G)$ has finite index in $H_{1}\times F$ (see [29, Lemma 2.1]).

Now, $H_{1}$ is finitely generated and virtually abelian, so it is biautomatic by [17, Corollary 4.2.4]. Therefore $H_{1}\times F$ is biautomatic by [17, Theorem 4.1.1], and, hence its finite-index subgroup $\phi(G)\cong G$ is biautomatic by [17, Theorem 4.1.4]. ∎

Theorem 8.3**.**

The group $G=G(A,L^{\prime})$ is biautomatic if and only if $A$ has finite order.

Proof.

Assume that $G$ is biautomatic. If either $L^{\prime}=L$ or $L^{\prime\prime}=L$, then $G$ is an ascending HNN-extension of $L$; in this case Groves and Hermiller [21, Main Theorem] proved that $G$ must be virtually abelian. The latter clearly implies that $A$ has finite order in $GL(n,\Q)$.

Thus we can assume $L^{\prime}$ and $L^{\prime\prime}$ are proper subgroups of $L$. Then $L$ is finitely generated and self-centralizing by Lemma 8.1, and the commensurator $\operatorname{Comm}_{G}(L)$ is the whole of $G$. Therefore, by Corollary 5.4, there is $k\in\mathbb{N}$ such that $t^{k}$ centralizes a finite-index subgroup of $L$. This means that $A^{k}$ is the identity matrix, and so $A$ has order dividing $k$.

Now suppose that $A$ has finite order $k\in\mathbb{N}$, and let $M$ be the intersection of the subgroups $t^{i}L^{\prime}t^{-i}=A^{i}\,L^{\prime}\leqslant L$, $i=0,\dots,k-1$. Then $M$ is a finite-index subgroup of $L$ and is normal in $G$. The quotient group $F=G/M$ is an HNN-extension of the finite group $L/M$, hence it is virtually free. It follows that $F$ is word hyperbolic, and, therefore, biautomatic (this can be easily deduced from [17, Chapter 3]; see [12, III.$\Gamma$.2.20] for an explicit statement). Let $\beta:G\to F$ denote the natural epimorphism with $\ker\beta=M$.

As in the proof of Proposition 7.1, we also have a homomorphism $\alpha:G\to AGL(n,\R)$ which sends $L$ to a subgroup of translations of $\R^{n}$ and $t$ to the linear transformation of $\R^{n}$ corresponding to $A$. Since $A$ has finite order, it is clear that $H\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\alpha(G)$ is virtually abelian; moreover, $L\cap\ker\alpha=\{1\}$ by construction.

Define the homomorphism $\psi:G\to H\times F$ by $\psi(g)=(\alpha(g),\beta(g))$ for all $g\in G$. This homomorphism is injective because the kernels of $\alpha$ and $\beta$ intersect trivially. Since $\alpha(G)=H$ and $\beta(G)=F$, $\psi(G)$ is a subdirect product in $H\times F$. Therefore $G\cong\psi(G)$ is biautomatic by Proposition 8.2. ∎

A well-known open problem (see [17, Open Question 4.1.5]) asks whether a group which has a finite-index biautomatic subgroup must itself be biautomatic. In the remainder of this section we will show that this is indeed the case for our groups: $G(A,L^{\prime})$ is biautomatic if and only if it is virtually biautomatic.

Lemma 8.4**.**

Suppose that $L^{\prime}$ and $L^{\prime\prime}$ are both proper subgroups of $L$ and let $G^{(2)}$ denote the second derived subgroup of $G=G(A,L^{\prime})$. Then $G^{(2)}$ is a non-abelian free group and for any two non-commuting elements $g_{1},g_{2}\in G^{(2)}$, the centralizer $\mathrm{C}_{G}(\{g_{1},g_{2}\})$ is a finite-index subgroup of $fLf^{-1}$, for some $f\in G$.

Proof.

Proposition 7.1 implies that $G^{(2)}$ is free, and since $L^{\prime}$ and $L^{\prime\prime}$ are proper subgroups of $L$, $G$ cannot be soluble (it will contain non-abelian free subgroups being an HNN-extension in which both of the associated subgroups are proper subgroups of the base group), hence $G^{(2)}$ is non-abelian.

Observe that the normal closure $N$, of $L$ in $G$, is generated by the elements $t^{i}ct^{-i}$, where $i\in\Z$ and $c\in L$. Evidently any such element centralizes the finite-index subgroup $t^{i}Lt^{-i}\cap L$, of $L$. Since each $g\in N$ is a product of finitely many such elements, we conclude that $g$ must also centralize a finite-index subgroup of $L$ in $G$.

Consider any two non-commuting elements $g_{1},g_{2}\in G^{(2)}$. Since $G/N$ is cyclic (generated by the image of $t$), $G^{(2)}\subseteq N$, so $g_{1}$ and $g_{2}$ both centralize some finite-index subgroup $K$ of $L$. Let $T$ be the Bass-Serre tree for the splitting of $G$ as an HNN-extension of $L$. Note that the subgroup $\langle g_{1},g_{2}\rangle\subseteq G^{(2)}$ acts freely on $T$ (see the proof of Proposition 7.1), so each $g_{j}$ acts as a hyperbolic isometry of $T$ with an axis $\ell_{j}$, $j=1,2$.

If $\ell_{1}=\ell_{2}$ then the rank $2$ free subgroup $\langle g_{1},g_{2}\rangle$ acts on the simplicial line $\ell_{1}$ by isometries. This action must have a non-trivial kernel, because the group of all simplicial isometries of $\ell_{1}$ is isomorphic to the infinite dihedral group. This means that a non-trivial element of $\langle g_{1},g_{2}\rangle$ fixes $\ell_{1}$ pointwise, contradicting the freeness of the action of this subgroup on $T$.

Hence $\ell_{1}$ and $\ell_{2}$ must be distinct. Since $\mathrm{C}_{G}(\{g_{1},g_{2}\})$ preserves each of these axes setwise, this centralizer must fix a vertex of $T$: if $\ell_{1}\cap\ell_{2}$ is a finite segment, it will fix a vertex of this segment; if $\ell_{1}\cap\ell_{2}$ is an infinite ray, it will fix all of it; finally, if $\ell_{1}\cap\ell_{2}=\emptyset$, it will fix all vertices of the unique geodesic segment connecting these two axes. The vertex stabilizers for the action of $G$ on $T$ are conjugates of $L$, so there exists $f\in G$ such that $\mathrm{C}_{G}(\{g_{1},g_{2}\})\subseteq fLf^{-1}$. Recall that $G$ commensurates $L$, hence $L\cap fLf^{-1}$ has finite index in $fLf^{-1}$. Since $\mathrm{C}_{G}(\{g_{1},g_{2}\})$ contains $K$ and $|L:K|<\infty$, we conclude that $|fLf^{-1}:\mathrm{C}_{G}(\{g_{1},g_{2}\})|<\infty$, as claimed. ∎

Theorem 8.5**.**

If $A$ has infinite order then the group $G=G(A,L^{\prime})$ is not virtually biautomatic.

Proof.

As in the proof of Theorem 8.3, the case when either $L=L^{\prime}$ or $L=L^{\prime\prime}$ follows from the result of Groves-Hermiller [21, Main Theorem], so we assume from now on that both $L^{\prime}$ and $L^{\prime\prime}$ are proper subgroups of $L$.

Let $H$ be a finite-index subgroup of $G$, then $|G^{(2)}:G^{(2)}\cap H|<\infty$, so $G^{(2)}\cap H$ is a non-abelian free subgroup by Lemma 8.4. Choose arbitrary two non-commuting elements $g_{1},g_{2}\in H\cap G^{(2)}$. The same lemma states that $\mathrm{C}_{G}(\{g_{1},g_{2}\})$ is a finite-index subgroup of $fLf^{-1}$ for some $f\in G$. Since $G$ commensurates $L$, it also commensurates $fLf^{-1}$, as well as its finite-index subgroup $M\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\mathrm{C}_{H}(\{g_{1},g_{2}\})=H\cap\mathrm{C}_{G}(\{g_{1},g_{2}\})$. It follows that $M$ is an abelian subgroup commensurated by $H$. Now, Corollary 5.4 implies that $H$ cannot be biautomatic as $ft^{l}f^{-1}\in H$, for some $l\in\mathbb{N}$, and no non-trivial power of this element can centralize a finite-index subgroup of $fLf^{-1}$ (since $A^{l}$ has infinite order). ∎

9 Explicit examples

Throughout this section, it will be sufficient to specialize the groups $G(A,L^{\prime})$, defined in Section 7, to the case when $L=\Z^{2}$ has rank two. We will write $M_{2}(\Q)$ to denote the ring of $2\times 2$ matrices with rational entries. Before starting, we recall that the classification, up to conjugacy, of square matrices over a field $k$ is equivalent to the classification, up to isomorphism, of finitely generated torsion modules for the polynomial ring $k[x]$, which is a principal ideal domain [25, Ch. XI]. In particular, if $f(x)\in k[x]$ is a polynomial that is square-free (i.e., not divisible by the square of any irreducible polynomial) then there is exactly one conjugacy class of square matrices over $k$ with characteristic polynomial $f(x)$: this is the analogue for $k[x]$ of the familiar statement (for $\Z$-modules) that there is exactly one abelian group of order $n$ provided that $n$ is square-free.

Proposition 9.1**.**

If $A\in M_{2}(\Q)$, then $A$ is conjugate to an element of $SO(2)$ in $GL(2,\R)$ if and only if $\det(A)=1$ and either $A=\pm I$ or $-2<{\rm tr}(A)<2$. Such a matrix $A$ has finite order if and only if ${\rm tr}(A)\in\Z$.

Proof.

Matrices in $SO(2)$ have the claimed properties, and these are not changed by conjugation. Conversely, if $A$ has the claimed properties and $A\neq\pm I$, then the characteristic polynomial of $A$ has the form $X^{2}-{\rm tr}(A)X+1$, and is irreducible over $\R$. Any two matrices with this characteristic polynomial are conjugate in $GL(2,\R)$.

If $A$ has finite order, the additive group of the subring of $M_{2}(\Q)$ generated by $A$ is finitely generated, from which it follows that the characteristic polynomial of $A$ must lie in $\Z[X]$. For the converse, the choices of $-2,-1,0,1,2$ for ${\rm tr}(A)$ give rise to elements of order $2,3,4,6,1$ respectively. ∎

Example 9.2*.*

As examples, the matrix

[TABLE]

for $k,m\in\Z$ with $m>0$, is conjugate to a matrix in $SO(2)$ if and only if $-2m<k<2m$, and has infinite order provided that $k\neq-m,0,m$.

Pythagorean triples give rise to matrices of infinite order in $GL(2,\Q)\cap SO(2)$. For example we shall consider the matrix $A_{P}$, defined by

[TABLE]

A combination of Theorem 7.2 with Theorem 8.3 yields the following.

Corollary 9.3**.**

If $A\in GL(2,\Q)$ has infinite order and is conjugate to an element of $SO(2)$ in $GL(2,\R)$ then for any $L^{\prime}$, $G(A,L^{\prime})$ is CAT(0) and is not biautomatic.

Example 9.4*.*

For more concrete examples, consider the groups

[TABLE]

This group is CAT(0) whenever $-2m<k<2m$ and is not biautomatic provided that $k\neq-m,0,m$. The group $G_{k,m}$ is of the form $G(A_{k/m},L^{\prime})$, where $L^{\prime}=\langle(1,0)^{T},(0,m)^{T}\rangle$ has index $m$ in $L$. In the case when ${\rm gcd}(k,m)=1$, $L^{\prime}$ is as large as possible, so $G_{k,m}=G(A_{k/m})$.

The first example of a group of this type that we found was the group $G^{\prime}_{P}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=G(A_{P},(5\Z)^{2})$, where $A_{P}$ is the matrix defined in (6) . Here are presentations for the groups $G_{P}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=G(A_{P})$ mentioned in the introduction (which corresponds to the case $L^{\prime}=L\cap A_{P}^{-1}\,L=\langle(2,-1)^{T},(1,2)^{T}\rangle$) and $G^{\prime}_{P}$.

[TABLE]

[TABLE]

In the case when $G(A,L^{\prime})$ is CAT(0), there is usually only one choice of CAT(0) metric on $L\otimes\R$ up to homothety.

Corollary 9.5**.**

Suppose that $A\in GL(2,\Q)$ has order at least 3 and is conjugate to an element of $SO(2)$ in $GL(2,\R)$. In this case, the inner product $\langle\,\cdot\,,\,\cdot\,\rangle_{L}$ on $L\otimes\R$, defined by (3) when viewing $L$ as a subgroup of the CAT(0) group $G(A,L^{\prime})$, is, up to multiplication by a scalar, the unique inner product that is preserved by $A$.

Proof.

Let $a\in L$ be any non-identity element, and suppose that the CAT(0) structure on $G(A,L^{\prime})$ is chosen so that $a$ acts on $L\otimes\R$ as translation by some distance $\lambda>0$. In this case $\langle a,a\rangle_{L}=\lambda^{2}$, but also

[TABLE]

because $A$ acts on $L\otimes\R$ as rotation through an angle $\theta$ with $2\cos(\theta)={\rm tr}(A)$. The uniqueness follows, because for $A\neq\pm I_{2}$, $a$ and $A\,a$ form a basis of $L\otimes\R$. ∎

Figure 3 below depicts the unique geometries on $L\otimes\R$ for the seven CAT(0) groups $G_{k,2}$ and the CAT(0) group $G_{P}$.

In [8, Question 2.7] D. Wise asked whether every CAT(0) group $G$ has the following property: for any elements $a,b\in G$, there exists $n>0$ so that the subgroup $\langle a^{n},b^{n}\rangle$ is either abelian or free.

Corollary 9.6**.**

If $A$ has infinite order and is conjugate to an element of $SO(2)$ in $GL(2,\R)$ then the group $G=G(A,L^{\prime})$, for any suitable choice of $L^{\prime}$, is CAT(0) but it is not virtually biautomatic and it does not have Wise’s property. In particular, this applies to the groups $G_{P}$, $G^{\prime}_{P}$ and $G_{k,m}$ from Example 9.4, provided $-2m<k<2m$ and $k\neq 0,\pm m$.

Proof.

The group $G$ is CAT(0) by Theorem 7.2 and it is not virtually biautomatic by Theorem 8.5.

Let $a\in L$ be a non-identity element of $L\leqslant G$ and let $t\in G$ be the stable letter from the presentation (4). Set $m\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=|L/L^{\prime}|$, then $b^{m}\in L^{\prime}$ for every $b\in L$. Given any $n\in\mathbb{N}$, the subgroup $H_{n}=\langle a^{n},t^{n}\rangle\leqslant G$ cannot be abelian because $t^{n}$ does not centralize any non-identity element of $L$. On the other hand, $H_{n}$ cannot be free, because it contains the element $t^{n}a^{nm^{n}}t^{-n}\in L$, which together with $a^{n}$ generates a finite-index subgroup of $L$. ∎

Remark 9.7*.*

Although the standard Tits alternative is still unknown for general CAT(0) groups, Proposition 7.1 implies that it does hold for any of the groups $G(A,L^{\prime})$, defined by (4).

Remark 9.8*.*

After hearing the first named author’s talks on the results of this paper, M. Bridson suggested that the methods developed in his paper [9] give an alternative proof that the groups $G(A,L^{\prime})$ from Corollary 9.6 are not biautomatic. Indeed, [9, Proposition 2.2] states that any biautomatic structure on an abelian group can contain only finitely many commensurability classes of quasiconvex subgroups. If the matrix $A$ has infinite order and has no rational eigenvectors, then for any infinite cyclic quasiconvex subgroup $C\leqslant L$, its conjugates $t^{i}Ct^{-i}$, $i\in\mathbb{N}$, will all be quasiconvex, pairwise non-commensurable and will virtually be subgroups of $L$. Moreover, using the work of Bridson and Gilman [11] it may be possible to extend this method to prove the stronger statement that $G(A,L^{\prime})$ does not admit any bounded bicombing such that the corresponding language is context-free.

10 Residual finiteness and non-Hopficity

As mentioned in the introduction, the groups $G(A,L^{\prime})$ are higher dimensional generalizations of the Baumslag-Solitar groups. Originally Baumslag and Solitar introduced their groups in [7] as the first examples of non-Hopfian one-relator groups. It is not hard to see that nearly the same argument shows that many of our groups are non-Hopfian. The following result is closely related to a theorem of D. Meier [28], although neither result is a direct corollary of the other.

Proposition 10.1**.**

Let $A\in GL(2,\Q)$ satisfy $\det A\in\Z$. Suppose that there exists an integer $m>1$ so that $mA$ is an integer matrix and $k\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=m{\rm tr}(A)$ is coprime to $m$. Then the group $G=G(A)$, defined in Section 7, is non-Hopfian.

Proof.

The characteristic polynomial of $A$ is $X^{2}-(k/m)X+l$, where $l\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\det A\in\Z$. This implies that

[TABLE]

in particular, $mlA^{-1}$ is an integer matrix. Since $mA$ is an integer matrix, we have $A\,L^{m}=(mA)\,L\subseteq L$, hence $L^{m}\subseteq A^{-1}\,L\cap L=L^{\prime}$. Similarly, $L^{ml}\subseteq A\,L\cap L=L^{\prime\prime}$. Thus

[TABLE]

Let $a,b\in L\leqslant G$ be generators for $L\cong\Z^{2}$, and let $t\in G$ be the stable letter. Combining (8) with (7), we obtain the identity

[TABLE]

It is easy to check that the map $\phi$ defined by

[TABLE]

extends to an endomorphism $\phi:G\rightarrow G$. Since the image of $\phi$ contains $t$ and $L^{m}=\langle a^{m},b^{m}\rangle$, in view of (9) we see that it also contains $a^{k}$ and $b^{k}$. Recall that $k$ is coprime to $m$ by the assumptions, hence the image of $\phi$ contains $a$ and $b$, so $\phi$ is surjective.

It remains to show that $\phi$ is not injective. The assumptions imply that both $A$ and $A^{-1}$ have some non-integer entries, hence $L^{\prime}$ and $L^{\prime\prime}$ must be proper subgroups of $L$. Choose arbitrary $c\in L-L^{\prime}$ and $d\in L-L^{\prime\prime}$ (since $L$ cannot be the union of the two proper subgroups $L^{\prime}$ and $L^{\prime\prime}$, there are elements of this form with $c=d$). Then the commutator $[tct^{-1},d]$ is non-trivial in $G$ by Britton’s Lemma for HNN-extensions (cf. [24, Section IV.2]), but it is in the kernel of $\phi$ by (8). ∎

By a well-known theorem of Malcev (cf. [26]) the groups from Proposition 10.1 cannot be residually finite. We can actually say more about the finite images of such groups.

Corollary 10.2**.**

Suppose that $G=G(A)$ is a group satisfying the assumptions of Proposition 10.1, and $\phi:G\to G$ is the endomorphism defined in the proof of this proposition. Let $R\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\bigcup_{n=1}^{\infty}\ker(\phi^{n})\lhd G$. Then the quotient $G/R$ is abelian-by-cyclic. In particular, every finite quotient of $G$ is metabelian.

Proof.

Let $N$ be the normal closure of $L$ in $G$, so that $G/N$ is infinite cyclic. Any $g,h\in N$ can be written as products of elements of the form $t^{i}ct^{-i}$, where $c\in L$ and $i\in\Z$. Hence, we can choose $s\in\mathbb{N}$ such that $g^{\prime}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=t^{s}gt^{-s}$ and $h^{\prime}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=t^{s}ht^{-s}$ are products of elements of the form $t^{i}ct^{-i}$, where $c\in L$ and $i>0$.

Note that, in view of (8), if $i\in\mathbb{N}$ then for each $j\geq i$, $\phi^{j}(t^{i}ct^{-i})\in L$ in $G$. Therefore there exists a sufficiently large $n\in\mathbb{N}$ such that $\phi^{n}(g^{\prime})\in L$ and $\phi^{n}(h^{\prime})\in L$. Since $L$ is abelian, it follows that the commutator $[g^{\prime},h^{\prime}]\in\ker\phi^{n}\subseteq R$, so $[g,h]=t^{-s}[g^{\prime},h^{\prime}]t^{s}\in R$, for arbitrary $g,h\in N$. Therefore the image of $N$ in $G/R$ is abelian, so $G/R$ is abelian-by-cyclic.

For the last assertion, recall that the proof of Malcev’s theorem implies that $R$ is contained in the intersection of all finite-index subgroups of $G$. Therefore it is annihilated by every epimorphism $\psi:G\to Q$, with $Q$ finite. It follows that $Q$ is a quotient of $G/R$, so it is also abelian-by-cyclic, as claimed. ∎

Corollary 10.3**.**

Suppose that $G=G_{P}$ or $G=G_{k,m}$, with $m>1$, $-2m<k<2m$ and ${\rm gcd}(k,m)=1$, is a group from Example 9.4. Then $G$ is a CAT(0) group which is not Hopfian and not uniformly non-amenable.

Proof.

The group $G$ is non-Hopfian by Proposition 10.1. The fact that $G$ is not uniformly non-amenable follows from Corollary 10.2 by [5, Corollary 13.2] or [34, Theorem 2.2]. ∎

The fact that the group $G_{P}$ is non-Hopfian can also be derived from Meier’s criterion [28, Lemma 1], however this criterion does not seem to apply to the groups $G_{k,m}$. The first examples of non-Hopfian CAT(0) groups were constructed by Wise in [39].

Using the work of Andreadakis, Raptis and Varsos [4] we can characterize the residual finiteness of groups $G(A,L^{\prime})$ in general.

Proposition 10.4**.**

Suppose that $L=\Z^{n}$, $A\in GL(n,\Q)$ and $L^{\prime}$ is a finite-index subgroup of $L$ such that $A\,L^{\prime}\subseteq L$. Then the group $G=G(A,L^{\prime})$, defined by (4), is residually finite if and only if one of the following conditions holds:

• (i)

$L^{\prime}=L$* or $A\,L^{\prime}=L$;*

• (ii)

$A$* is conjugate in $GL(n,\Q)$ to a matrix from $GL(n,\Z)$.*

Proof.

By [4, Theorem 1] the group $G$, defined by (4), is residually finite if and only if either $L=L^{\prime}$ or $L=A\,L^{\prime}$ (in which case $G$ is metabelian) or $t$ normalizes a finite-index subgroup of $L$. Let us prove that the latter is equivalent to saying that $A$ is conjugate in $GL(n,\Q)$ to a matrix from $GL(n,\Z)$.

Suppose, first, that $tMt^{-1}=M$ for some finite-index subgroup $M\leqslant L$. This implies that $M\subseteq L^{\prime}$, so $tMt^{-1}=A\,M=M$. Evidently $M=B\,L$ for some invertible matrix $B$ with integer entries, thus $(B^{-1}\,A\,B)\,L=L$, i.e., $B^{-1}\,A\,B\in GL(n,\Z)$.

Conversely, assume that $C^{-1}\,A\,C\in GL(n,\Z)$ for some $C\in GL(n,\Q)$. Set $k\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=|L/L^{\prime}|$ and choose $m\in\mathbb{N}$ so that all entries of the matrix $B\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=mC$ are integers divisible by $k$. Then $B$ is invertible, so $M\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=B\,L$ has finite index in $L$; moreover, $M\subseteq L^{\prime}$ by the choice of $m$. Note that $B^{-1}=\frac{1}{m}C^{-1}$, so $B^{-1}\,A\,B=C^{-1}\,A\,C\in GL(n,\Z)$. It follows that $(B^{-1}\,A\,B)\,L=L$, hence $M=A\,M=tMt^{-1}$, as required. ∎

Using the rational canonical form for matrices [25, Chapter XI.4], condition (ii) from Proposition 10.4 can be restated more algebraically.

Remark 10.5*.*

A matrix $A\in GL(n,\Q)$ is conjugate in $GL(n,\Q)$ to some matrix from $GL(n,\Z)$ if and only if $\det(A)=\pm 1$ and all coefficients of the characteristic polynomial of $A$ are integers.

Proposition 10.6**.**

Let $G=G(A,L^{\prime})$, where $L=\Z^{n}$, $A\in GL(n,\Q)$ and $L^{\prime}$ is a finite-index subgroup of $L$ such that $A\,L^{\prime}\subseteq L$. Then $G$ is residually finite if and only if $G$ is linear over $\Q$.

Proof.

Finitely generated linear groups are residually finite by a result of Malcev [26], hence we only need to prove that if $G$ is residually finite then it is isomorphic to a subgroup of $GL(m,\Q)$ for some $m\in\mathbb{N}$.

Suppose, first, that $t$ normalizes a finite-index subgroup $M$ of $L$. Then $M\lhd G$ and $F\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=G/M$ is an HNN-extension of the finite group $L/M$. Thus $F$ is finitely generated and virtually free, so it is linear over $\Q$. Let $\beta:G\to F$ be the natural epimorphism with $\ker\beta=M$.

As before, we also have a homomorphism $\alpha:G\to AGL(n,\Q)$, which comes from the actions of $L$ and $t$ on $\Q^{n}$ by translations and by $A$ respectively. The standard embedding of $AGL(n,\Q)$ in $GL(n+1,\Q)$ shows that it is linear over $\Q$.

Evidently $\ker\alpha\cap L=\{1\}$, hence $\ker\alpha\cap\ker\beta=\{1\}$. Therefore the homomorphism $\psi:G\to AGL_{n}(\Q)\times F$, defined by $\psi(g)=(\alpha(g),\beta(g))$ for all $g\in G$, is injective. It follows that $G$ is linear over $\Q$.

If $t$ does not normalize any finite-index subgroup of $L$, then, by [4, Theorem 1], either $L^{\prime}=L$ or $A\,L^{\prime}=L$. In this case $G$ is an ascending HNN-extension of $L$, which easily yields that $G$ embeds in the direct product $AGL(n,\Q)\times\Z$, where the homomorphism $G\to\Z$, onto the second factor, is given by the natural projection sending $L$ to [math] and $t$ to $1$. This again shows that $G$ is linear over $\Q$. ∎

11 Free products with amalgamation

Just as a cyclic group embeds as an index two subgroup of a dihedral group, many of the groups $G(A,L^{\prime})$ can be embedded as index two subgroups of groups expressed as free products with amalgamation.

Theorem 11.1**.**

Let $L=\Z^{n}$, let $A\in GL(n,\Q)$ and let $L^{\prime}$ be a finite-index subgroup of $L\cap A^{-1}\,L$. Suppose that there is a matrix $R\in GL(n,\Z)$ with the following properties.

• (i)

$R^{2}=I_{n}$;

• (ii)

$RAR=A^{-1}$;

• (iii)

$R\,L^{\prime}=L^{\prime\prime}$, where $L^{\prime\prime}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=A\,L^{\prime}\leqslant L$.

Then the group $G(A,L^{\prime})$, defined by (4), embeds as an index two subgroup of an amalgamated free product $K=H*_{L^{\prime}=L^{\prime}}H^{\prime}$, where $H$ is an index $2$ overgroup of $L$ and $H^{\prime}$ is an index $2$ overgroup of $L^{\prime}$.

If $n=2$ and $A$ is conjugate to an element of $SO(2)$ in $GL(2,\R)$ then $K$ is CAT(0).

Proof.

Define the matrix $R^{\prime}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=RA\in GL(n,\Q)$. From conditions (i)–(iii) it is immediate that $R^{\prime 2}=I_{n}$, and that $R^{\prime}\,L^{\prime}=L^{\prime}$. The group $H$ is defined as an extension with kernel $L$ and the quotient cyclic of order two, generated by $\rho$, say, where conjugation by $\rho$ acts as multiplication by the matrix $R$. Similarly, $H^{\prime}$ is defined as an extension with kernel $L^{\prime}$ and the quotient cyclic of order two, generated by an element $\rho^{\prime}$ that acts on $L^{\prime}$ as multiplication by $R^{\prime}$. Let $K\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=H*_{L^{\prime}=L^{\prime}}H^{\prime}$ be the amalgamated product of $H$ and $H^{\prime}$ along their common subgroup $L^{\prime}$.

Let $r\in H$ and $r^{\prime}\in H^{\prime}$ be some preimages of $\rho$ and $\rho^{\prime}$ respectively. Let us first check that the subgroup $M\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=\langle L,rr^{\prime}\rangle$ has index $2$ in $K$. Evidently $K$ is generated by $M$ and $r$, and $rLr^{-1}=L$, $r(rr^{\prime})r^{-1}=r^{2}r^{\prime 2}r^{\prime-1}r^{-1}\in L(rr^{\prime})^{-1}\subseteq M$, as $r^{2}\in L$ and $r^{\prime 2}\in L^{\prime}\subseteq L$ in $K$. Therefore $rMr^{-1}\subseteq M$, which implies that $M\lhd K$ because $r^{2}\in M$. It follows that $M$ is the kernel of the epimorphism $\eta:K\to\Z/2\Z$, defined by $\eta(r)=\overline{1}$, $\eta(r^{\prime})=\overline{1}$ and $\eta(L)=\{\overline{0}\}$. Thus $|K:M|=2$, as claimed.

Let $T$ be the Bass-Serre tree for the decomposition of $K$ as an amalgamated free product. The vertices of $T$ can be identified with the cosets $cH$ or $cH^{\prime}$, $c\in K$, and the edges correspond to the cosets $cL^{\prime}$, $c\in K$. Let $u$, $v$ be the vertices of $T$ corresponding to $H$ and $H^{\prime}$ respectively; let $e$ be the edge of $T$ corresponding to $L^{\prime}$. Then $e$ joins $u$ with $v$ in $T$, and this edge, together with the endpoints, is a fundamental domain for the action of $K$ on $T$. As a fundamental domain for the induced action of $M$ on $T$ we can take the union $e\cup e^{\prime}$, where $e^{\prime}\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=r^{\prime-1}\,e$ with the vertices $u=e_{-}$, $v=e_{+}=e^{\prime}_{+}$ and $w\mathrel{\mathop{\mathchar 0\relax}"303A}\mkern-1.2mu=e^{\prime}_{-}=r^{\prime-1}\,e_{-}$. Note that $w=r^{\prime-1}\,u=r^{\prime-1}r^{-1}\,u\in M\,u$, since $r\,u=u$ and $rr^{\prime}\in M$. Thus the action of $M$ on $T$ has two orbits of vertices and two orbits of edges, and the quotient graph $M\backslash T$ consists of two vertices and two edges joining them.

Observe that the $M$-stabilizers of $u$, $v$, $e$ and $e^{\prime}$ are $L$, $L^{\prime}$, $L^{\prime}$ and $r^{\prime-1}L^{\prime}r^{\prime}=L^{\prime}$ respectively. We can now apply the Structure Theorem of Bass-Serre theory [36, Section I.5.4] to find that $M$ has the following presentation:

[TABLE]

It remains to observe that for each $c\in L^{\prime}$ we have $(r^{\prime-1}r^{-1})^{-1}c(r^{\prime-1}r^{-1})=(rr^{\prime})c(rr^{\prime})^{-1}=A\,c$, hence the presentation (10), of $M$, coincides with the presentation (4), of $G(A,L^{\prime})$. Thus $M\cong G(A,L^{\prime})$.

Now suppose that $n=2$ and $A\in SO(2)$. The cases $A=\pm I_{2}$ are easily dealt with. Assuming that $A\in SO(2)$ has order at least $3$, a direct computation shows that any matrix $R\in SL^{\pm}(2,\R)$ satisfying $RA=A^{-1}R$ is a reflection matrix from $O(2)$. Hence whenever $A$ preserves some inner product on $L\otimes\R$, that inner product is also preserved by $R$ and $R^{\prime}$. Once one knows this, showing that $K$ is a CAT(0) group is similar to the proof of Theorem 7.2. There are isometric actions of the group on $\mathbb{E}^{2}$ and on $T$, the Bass-Serre tree for the given decomposition as a free product with amalgamation. Furthermore, the diagonal action on $\mathbb{E}^{2}\times T$ is properly discontinuous, cocompact and by isometries. (If we metrize $T$ so that each edge has length $1/2$, the space $\mathbb{E}^{2}\times T$ with its action of $G(A,L^{\prime})$ is equivariantly isometric to the space constructed in Theorem 7.2.) ∎

Remark 11.2*.*

In the case when $L^{\prime}=L\cap A^{-1}\,L$, one has that $R\,L^{\prime}=R\,L\cap RA^{-1}\,L=L\cap AR\,L=L\cap A\,L=A\,L^{\prime}$, and so in this case condition (iii) follows from conditions (i) and (ii).

The possible isomorphism types of the group $H$ arising in Theorem 11.1 depend on the action of $R$ on $L$. To make this precise, we need a further definition.

Let $L$ be a free abelian group of rank two, and let $\rho$ be an involution of $L$ that reverses the orientation of $L$. It can be shown that there are two conjugacy classes of such involutions in $GL(2,\Z)$, depending on whether $L$ has a basis which is permuted by $\rho$ or has a basis of eigenvectors for $\rho$. We refer to the first as the rhombic case and to the second as the rectangular case. If one chooses an inner product on $L\otimes\R$ that is preserved by the action of the involution, the rhombic case corresponds to the existence of a basis for $L$ consisting of vectors of the same length swapped by $\rho$, and the rectangular case corresponds to the existence of an orthogonal basis for $L$ consisting of eigenvectors for $\rho$.

Proposition 11.3**.**

Let $H$ be a group expressed as an extension with kernel $L\cong\Z^{2}$ and quotient cyclic of order two, generated by $\rho$. If the action of $\rho$ on $L$ is rhombic then $H$ is isomorphic to the wreath product $\Z\wr C_{2}$. If the action of $\rho$ on $L$ is rectangular then $H$ is isomorphic either to the direct product $\Z\times D_{\infty}$ or to the Klein bottle group $BS(1,-1)$.

Proof.

Group extensions with quotient cyclic of order two and kernel a given $C_{2}$-module $V$ are classified by $H^{2}(C_{2};V)$. In the rhombic case $L$ is a free $\Z\langle\rho\rangle$-module and so there is only the split extension $\Z\wr C_{2}$. In the rectangular case $H^{2}(\langle\rho\rangle;L)$ has order two. The split extension gives $\Z\times D_{\infty}$ and the non-split extension gives the Klein bottle group $BS(1,-1)$. ∎

Example 11.4*.*

Next we consider how this embedding result applies to the groups $G_{k,m}$, $G_{P}$ and $G^{\prime}_{P}$ which were defined in Example 9.4.

Let $\{a,b\}$ be the standard basis for $L=\Z^{2}$. For each $k$ and $m$, the matrix $A_{k/m}$, defined in (5), is inverted by the matrix $R=\begin{pmatrix}0&1\\ 1&0\\ \end{pmatrix}$. If $L^{\prime}$ is the submodule of $L$ spanned by $a$ and $mb$, then $RA\,a=a$ and $RA\,mb=-mb+ka$. A calculation shows that the action of $R$ on $L=\Z^{2}$ is rhombic and the action of $RA$ on $L^{\prime}$ is rhombic in the case when $k$ is odd and is rectangular in the case when $k$ is even. Thus in each case the group $G_{k,m}$ embeds as an index two subgroup of a group $H*_{L^{\prime}=L^{\prime}}H^{\prime}$ as in the statement of Theorem 11.1, where $H\cong\Z\wr C_{2}$. If $k$ is odd, then $H^{\prime}$ is also isomorphic to $\Z\wr C_{2}$, whereas if $k$ is even, then $H^{\prime}$ may be taken to be either $\Z\times D_{\infty}$ or $BS(1,-1)$.

Since the matrix $A_{P}$, defined in (6), is already in $SO(2)$, there are four possible choices for $R\in O(2)\cap GL(2,\Z)$: $\begin{pmatrix}0&1\\ 1&0\end{pmatrix}$ or $\begin{pmatrix}0&-1\\ -1&0\end{pmatrix}$ (rhombic case), or $\begin{pmatrix}1&0\\ 0&-1\end{pmatrix}$ or $\begin{pmatrix}-1&0\\ 0&1\end{pmatrix}$ (rectangular case). In the case when $L^{\prime}=\langle(2,-1)^{T},(1,2)^{T}\rangle$ is as large as possible, a calculation shows that the action of $RA_{P}$ on $L^{\prime}$ and the action of $R$ on $L$ both have the same type. Thus we obtain five potentially different amalgamated products of the form $H*_{L^{\prime}=L^{\prime}}H^{\prime}$ that contain $G_{P}$ as an index two subgroup, including one torsion-free group in which each of $H$ and $H^{\prime}$ is isomorphic to $BS(1,-1)$. None of the possible choices for $R$ swaps $(5\Z)^{2}$ and its image under $A_{P}$, so we cannot construct a group of this form containing $G^{\prime}_{P}$ as an index two subgroup.

Example 11.5*.*

Let us give an explicit presentation for the torsion-free amalgamated product $K=H*_{L^{\prime}=L^{\prime}}H^{\prime}$, where $A=A_{P}=\begin{pmatrix}3/5&-4/5\\ 4/5&3/5\end{pmatrix}$, $L^{\prime}=\langle(2,-1)^{T},(1,2)^{T}\rangle$ and $R=\begin{pmatrix}-1&0\\ 0&1\\ \end{pmatrix}$ (rectangular case). Then the group $G_{P}$ from Example 9.4 embeds in $K$ as a subgroup of index $2$, and $K\cong BS(1,-1)*_{\Z^{2}}BS(1,-1)$ has the presentation

[TABLE]

which can be transformed to the $2$-generator and $2$-relator presentation

[TABLE]

Remark 11.6*.*

Suppose that the hypotheses of both Proposition 10.1 and Theorem 11.1 apply, and we are in the split case (i.e., when $r^{2}=r^{\prime 2}=1$, $H=L\rtimes\langle r\rangle$ and $H^{\prime}=L^{\prime}\rtimes\langle r^{\prime}\rangle$). Then the endomorphism $\phi$ constructed in the proof of Proposition 10.1 extends to the amalgamated product $H*_{L^{\prime}=L^{\prime}}H^{\prime}$ via $\phi(r)=r$ and $\phi(r^{\prime})=r^{\prime}$. This gives examples of free products of virtually $\Z^{2}$ groups amalgamating abelian subgroups that are non-Hopfian, and, in particular, not residually finite.

Corollary 11.7**.**

In addition to the assumptions of Theorem 11.1, suppose that $A$ has infinite order. Then the amalgamated product $K=H*_{L^{\prime}=L^{\prime}}H^{\prime}$ is not virtually biautomatic.

Proof.

Indeed, the group $G=G(A,L^{\prime})$ is not virtually biautomatic by Theorem 8.5 and, since $G$ has index $2$ in $K$, $K$ cannot be virtually biautomatic by [17, Theorem 4.1.4]. ∎

Baumslag, Gersten, Shapiro and Short [6] proved that a free product of two finitely generated free abelian groups amalgamating any subgroup is always automatic. However, Bridson [10, Remark 3] noted that amalgamated products of finitely generated virtually abelian groups can be much wilder; in particular they may not even be asynchronously automatic. Corollary 11.7 together with Example 11.4 produce examples of similar spirit.

12 Closing remarks and open questions

Motivated by the question whether all automatic groups are biautomatic, we tried checking automaticity of the groups $G_{P}$ and $G_{k,m}$ from Example 9.4. As part of this effort, we used the GAP implementation of Holt’s KBMAG software [18, 37] to search for automatic structures on the groups $G_{k,2}$ as defined in Example 9.4 for various choices of $k$. In the cases $k\in\{-2,0,2\}$ it easily found an automatic structure, while for $k\in\{-3,-1,1,3,4,5\}$ it failed to find any automatic structure.

Throughout the rest of this section we assume that $n\in\mathbb{N}$, $L=\Z^{n}$, $A\in GL(n,\Q)$ and $L^{\prime}$ is a finite-index subgroup of $L$ such that $A\,L^{\prime}\subseteq L$. The questions below concern the groups $G(A,L^{\prime})$, defined in (4).

Question 12.1**.**

Can the group $G(A,L^{\prime})$ be automatic when $A$ has infinite order?

As explained in the introduction, the groups $G(A,L^{\prime})$ are higher-dimensional analogues of Baumslag-Solitar groups $BS(k,l)$, which cannot be subgroups of biautomatic groups unless $|k|=|l|$ [20, Proposition 6.7]. This yields the following question, suggested to the authors by K.-U. Bux:

Question 12.2**.**

Suppose that $G(A,L^{\prime})$ embeds as a subgroup in a biautomatic group. Does it follow that $A$ has finite order?111A positive answer to this question has recently been announced by M. Valiunas, arXiv:2104.13688.

In [19, Question 43] Farb, Hruska and Thomas asked whether every group which acts properly and cocompactly on a CAT(0) piecewise Euclidean $2$-complex is biautomatic. This question remains open, though one can show that our groups $G_{P}$ and $G_{k,m}$, $-2m<k<2m$, admit geometric actions on CAT(0) piecewise Euclidean $3$-complexes.

Andreadakis, Raptis and Varsos [3] described a necessary and sufficient criterion for an HNN-extension of $\Z^{n}$ to be Hopfian. However, it is not obvious how to check this criterion for any given group $G(A,L^{\prime})$. It would be interesting to characterize the Hopficity of $G(A,L^{\prime})$ only in terms of the matrix $A$ and the finite-index subgroup $L^{\prime}$ of $L=\Z^{n}$ (in the spirit of the residual finiteness criterion from Proposition 10.4).

Question 12.3**.**

Classify the groups $G(A,L^{\prime})$ up to isomorphism222This has recently been completed by M. Valiunas in arXiv:2011.08143., commensurability and quasi-isometry.

We expect that the classification up to isomorphism should be straightforward, while the classification up to commensurability will be more challenging (it has only recently been completed by Casals-Ruiz, Kazachkov and Zakharov [15] for the Baumslag-Solitar groups, which correspond to the case $n=1$). For $n=1$ the quasi-isometry classification of Baumslag-Solitar groups was done by Whyte [38]. Suppose that $A$ is conjugate to an orthogonal matrix in $GL(n,\R)$. Then, in view of Corollary 7.4, $G(A,L^{\prime})$ is quasi-isometric to the direct product of $\Z^{n}$ with a free group. So, for a fixed $n\in\mathbb{N}$, there are exactly $2$ quasi-isometry classes for such $A$: in the first class $L^{\prime}=L$, $A$ has finite order and $G(A,L^{\prime})$ is virtually $\Z^{n+1}$, and in the second class $L^{\prime}$ is a proper subgroup of $L$ and $G(A,L^{\prime})$ is quasi-isometric to $\Z^{n}\times F_{2}$, where $F_{2}$ is the free group of rank $2$. In the latter case $G(A,L^{\prime})$ is commensurable with $\Z^{n}\times F_{2}$ if and only if $A$ has finite order.

Finally, an observant reader may notice that, according to Theorem 1.1, if the group $G(A,L^{\prime})$ is both CAT(0) and residually finite then it is biautomatic. Thus our methods leave the following question open.

Question 12.4**.**

Is every residually finite CAT(0) group biautomatic?