Embeddings of shearlet coorbit spaces into Sobolev spaces

TL;DR

This paper studies how shearlet coorbit spaces relate to Sobolev spaces in three dimensions, revealing how group features affect smoothness properties and embedding behavior.

Contribution

It provides a new perspective by describing coorbit spaces as decomposition spaces, enabling complete embedding characterizations and insights into shearlet group influences.

Findings

Complete characterization of embeddings for certain integrability exponents

Identification of dilation group features affecting embeddings

Insights into shearlet coorbit spaces as smoothness spaces

Abstract

We investigate the existence of embeddings of shearlet coorbit spaces associated to weighted mixed $L^p$-spaces into classical Sobolev spaces in dimension three by using the description of coorbit spaces as decomposition spaces. This different perspective on these spaces enables the application of embedding results that allow the complete characterization of embeddings for certain integrability exponents, and thus provides access to a deeper understanding of the smoothness properties of coorbit spaces, and of the influence of the choice of shearlet groups on these properties. We give a detailed analysis, identifying which features of the dilation groups have an influence on the embedding behavior, and which do not. Our results also allow to comment on the validity of the interpretation of shearlet coorbit spaces as smoothness spaces.

Equations

$$f=\frac{1}{C_{\psi }}{\int }_G{W}_{\psi }f(x,h)\pi (x,h)\psi \mathrm{d}(x,h),$$

$$H^{{\lambda }_1,{\lambda }_2}$$

$$H^{\delta }$$

$$\left\{f:G\to \mathbb{C}:{\int }_H{\left({\int }_{{\mathbb{R}}^d}{\left|f(x,h)\right|}^{p}v(x,h{)}^p\mathrm{d}x\right)}^{q/p}\frac{\mathrm{d}h}{|\det (h)|}<\infty \right\},$$

$$\left\{f\in {(\mathcal{H}}_w^1{)}^{\neg }:W_{\psi }f\in W({\mathrm{L}}_v^{p,q}(G))\right\}$$

$$\mathcal{D}(\mathcal{Q},{\mathrm{L}}^p,{\ell }_u^q):=\{f\in {\mathcal{D}}^{\prime }(\mathcal{O}):\Vert f{\Vert }_{\mathcal{D}(\mathcal{Q},{\mathrm{L}}^p,{\ell }_u^q)}<\infty \}$$

$$\Vert f{\Vert }_{\mathcal{D}(\mathcal{Q},{\mathrm{L}}^p,{\ell }_u^q)}:={\Vert {\left(u_i\cdot \Vert {\mathcal{F}}^{-1}({\phi }_{i}f){\Vert }_{{\mathrm{L}}^p({\mathbb{R}}^d)}\right)}_{i\in I}\Vert }_{{\ell }^q(I)}.$$

$$B_{n,m_1,m_2,ϵ}^{{\lambda }_1,{\lambda }_2}:=ϵ\left(\begin{array}{ccc}{2}^n& m_1{2}^n& m_2{2}^n\\ 0& 2^{n{\lambda }_1}& 0\\ 0& 0& 2^{n{\lambda }_2}\end{array}\right)\in H^{{\lambda }_1,{\lambda }_2}$$

$$U^{{\lambda }_1,{\lambda }_2}$$

$$B_{n,m_1,m_2,ϵ}^{\delta }:=ϵ\left(\begin{array}{ccc}{2}^n& m_1{2}^n& m_2{2}^n\\ 0& 2^{n(1-\delta )}& m_1{2}^{n(1-\delta )}\\ 0& 0& 2^{n(1-2\delta )}\end{array}\right)\in H^{\delta }$$

$$U^{\delta }$$

$$\displaystyle W^{k,q}(\mathbb{R}^{d}):=\Set{f\in\mathrm{L}^{q}(\mathbb{R}^{d})}{\begin{array}[]{l}\partial^{\alpha}f\in\mathrm{L}^{q}(\mathbb{R}^{d})\text{ for all }\\ \alpha\in\mathbb{N}_{0}^{d},\ |\alpha|_{1}\leq k\end{array}}$$

$$\displaystyle W^{k,q}_{*}(\mathbb{R}^{d}):=\Set{(\partial^{\alpha}f)_{\alpha\in\mathbb{N}_{0}^{d},\lvert\alpha\rvert_{1}\leq k}}{\begin{array}[]{l}f\in C^{\infty}(\mathbb{R}^{d}),\ \partial^{\alpha}f\in\mathrm{L}^{q}(\mathbb{R}^{d})\\ \text{for all }\alpha\in\mathbb{N}_{0}^{d},\ |\alpha|_{1}\leq k\end{array}}$$

$${\partial }_{\ast }^{\alpha }:\mathcal{D}(\mathcal{Q},{\mathrm{L}}^p,{\ell }_u^r)$$

$$f$$

$${\iota }_q^{(k)}:$$

$$f$$

$${\iota }_q^{(k)}:$$

$$f$$

$$\mathrm{Co}\left({\mathrm{L}}_m^{p,q}({\mathbb{R}}^d⋊H)\right)$$

$$f$$

$$w_i^{\{q\}}:={\left|\det \left(T_i\right)\right|}^{\frac{1}{p}-\frac{1}{q}}\left(1+|b_i{|}^k+{\Vert T_i\Vert }^k\right),$$

$$\frac{w^{\{q\}}}{u}\in {\ell }^{q\cdot {\left(r/q\right)}^{\prime }}(I)\text{ for }q<\infty \text{ and}\frac{w^{\{q\}}}{u}\in {\ell }^{r^{\prime }}(I)\text{ for }q=\infty .$$

$$\mathcal{D}(\mathcal{Q},{\mathrm{L}}^p,{\ell }_u^r)\hookrightarrow W^{k,q}({\mathbb{R}}^d)$$

$$\mathcal{F}:\mathrm{Co}({\mathrm{L}}_v^{p,r}({\mathbb{R}}^3⋊H^{{\lambda }_1,{\lambda }_2}))\to \mathcal{D}({\mathcal{C}}^{{\lambda }_1,{\lambda }_2},{\mathrm{L}}^p,{\ell }_u^r)$$

$$u_{n,m_1,m_2,ϵ}$$

$$u_{n,m_1,m_2,ϵ}^{(\alpha ,\beta )}$$

$$=2^{-n(1+{\lambda }_1+{\lambda }_2)(\frac{1}{2}-\frac{1}{r})}{2}^{-n\alpha }{\Vert A_{n,m_1,m_2,ϵ}^{{\lambda }_1,{\lambda }_2}\Vert }^{\beta }$$

$$w_{n,m_1,m_2,ϵ}^{\{q\}}$$

$$=2^{n(1+{\lambda }_1+{\lambda }_2)(\frac{1}{p}-\frac{1}{q})}\left(1+{\Vert A_{n,m_1,m_2,ϵ}^{{\lambda }_1,{\lambda }_2}\Vert }^k\right).$$

$${\zeta }_{n,m_1,m_2,ϵ}^{{\lambda }_1,{\lambda }_2}$$

$$=2^{n\left[\alpha +(1+{\lambda }_1+{\lambda }_2)(\frac{1}{2}-\frac{1}{r}+\frac{1}{p}-\frac{1}{q})\right]}\left({\Vert A_{n,m_1,m_2,ϵ}^{{\lambda }_1,{\lambda }_2}\Vert }^{-\beta }+{\Vert A_{n,m_1,m_2,ϵ}^{{\lambda }_1,{\lambda }_2}\Vert }^{k-\beta }\right).$$

$${\zeta }^{{\lambda }_1,{\lambda }_2}\in {\ell }^{\theta }(I)⟺{\psi }^{(a,\beta )},{\psi }^{(a,\beta -k)}\in {\ell }^{\theta }({\mathbb{Z}}^3).$$

$$M_1^{+}$$

Full text

{history}\published

(Day Month Year)

\ccode

AMS Subject Classification:

1 Introduction

This paper is a study of approximation-theoretic properties of generalized wavelet systems arising from the action of certain matrix groups by dilation, combined with arbitrary translations. Starting with the paper by Murenzi [26], soon after generalized by Bernier and Taylor [2], it was realized that the theory of square-integrable group representations provides access to a large variety of possible wavelet constructions, see e.g. Refs. \refciteMR1419174,MR1633179. An important addition to this class were the shearlets introduced for dimension two in Ref. \refciteMR2543193, and for higher dimensions in Ref. \refciteMR2643586. It was later realized that in dimensions $\geq 3$, several distinct choices of shearing operations can be employed, by introducing the Toeplitz shearlet construction [6]. A general scheme for the construction of shearlet dilation groups, which leads to a vast choice of different groups in higher dimensions, was then developed in Refs. \refciteAlbertiEtAl2017,FuRe.

The interest in shearlets comes from the fact that the combination of anisotropic scaling and shearing results in a system of functions that is better equipped for the resolution of oriented singularities such as edges in images. This statement can be formalized by showing that the wavefront set of a signal can be characterized in terms of shearlet coefficients, which was first shown for two-dimensional wavelets in Ref. \refciteKuLa, and later extended to more general shearlet groups in arbitrary dimensions [1].

An alternative way of understanding how coefficient decay and smoothness of the analyzed signal are connected uses the theory of coorbit spaces. These spaces, introduced by Feichtinger and Gröchenig [11, 12], are based on the idea of introducing norms that quantify coefficient decay of a signal $f$ with respect to a given generalized wavelet system, and it is known that this theory applies to shearlet dilation groups in arbitrary dimensions [19]. Hence, each of these groups induces its own scale of coorbit spaces, defined in terms of the speed of coefficient decay. In view of the large pool of possible choices of such groups, this raises the question of analyzing and understanding coorbit spaces associated to a given shearlet dilation group, or more pointedly, understanding the influence that the choice of shearlet dilation group has on its scale of coorbit spaces. This paper can be seen as a case study for such an endeavor: We consider two families of shearlet dilation groups in dimension three, and analyze in a systematical fashion how coorbit spaces associated to weighted mixed $L^{p}$-spaces over these groups embed into Sobolev spaces. This question is interesting for several reasons. The first reason originates from the interpretation of coorbit spaces as smoothness spaces, as done, e.g., for shearlets in the introduction of Ref. \refciteMR2643586. This point of view seems natural given the fact that, for all shearlet dilation groups, there exist shearlet systems consisting of compactly supported, smooth functions [8, 19]. Hence, one might expect that the elements of a coorbit space requiring fast decay of the coefficients inherit fast decay and smoothness properties from the elements of the shearlet system that efficiently approximate them. Clearly, studying the embedding behaviour of coorbit spaces into Sobolev spaces is one way of putting this general intuition to the test, and our analysis will reveal the extent to which it is justified, and how different features of the dilation groups influence its validity.

Furthermore, the project of understanding the relationship between shearlet coorbit spaces and classical smoothness spaces is also motivated by work analyzing Fourier integral or pseudo-differential operators using shearlets [21, 29], with a view to characterizing the mapping properties of these operators on the various function spaces. The embeddings of the type studied here fit well into this general endeavor.

The last source of motivation that we want to mention comes from the method of proof, which largely relies on the machinery of decomposition spaces. These spaces were first introduced by Feichtinger and Gröbner [10] as a unified approach to both Besov and modulation spaces, with the scale of $\alpha$-modulation spaces as intermediate construction. Decomposition space applications and techniques were later extended by Borup and Nielsen [3], who pointed out (among other things) that curvelets could also be included in this setting. These ideas were further developed by the work of Voigtlaender, who introduced a powerful embedding theory between decomposition spaces of different kinds [30], and of decomposition spaces into Sobolev spaces [31]. The scope of these results is truly remarkable: Among the function spaces that have a decomposition space description are (homogeneous) isotropic Besov spaces, or more generally, $\alpha$-modulation spaces [10], inhomogeneous Besov spaces [30] and anisotropic Besov spaces (both homogeneous and inhomogeneous) [4]. Another class of examples, which is of particular relevance for this paper, are the coorbit spaces associated to general dilation groups, introduced in full generality in Ref. \refciteFuehCooSpaAndWavCoeDecOveGenDilGro, and identified as decomposition spaces in Refs. \refciteMR3345605,Voigtlaender2015PHD. In particular, all of the previously mentioned shearlet coorbit spaces fall in this category.

Hence the embedding theory developed in Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces is applicable to our problem, and our paper is both a sample application of the methods developed in the cited paper, and an illustration of the remarkable power of these methods. Prior to the work of Voigtlaender, an analysis of comparable depth and scope had simply been out of reach.

Overview and summary of the paper

The paper is structured as follows: Sections 2 through 4 introduce the objects and results necessary to formulate and prove our main result. The class of shearlet transforms that we study is introduced in Section 2. Coorbit spaces, and their decomposition space description, are introduced and explained in Section 3. Voigtlaender’s embedding result is then formulated in Section 4. Theorem 4.7 shows that a decomposition space of the type $\mathcal{D}(\mathcal{P},L^{p},\ell^{q}_{v})$ embeds into a Sobolev space if a certain sequence, that is explicitly derived from the data $\mathcal{P},p,q,v$ entering the definition of the decomposition space, is summable in a suitable sense. For summability indices $p,q\leq 2$, this statement in fact becomes an equivalence. The systematic application of this result to the setting of shearlet coorbit spaces is then the subject of Section 5. The main technical results of our paper are the Theorems 5.10, containing a precise and exhaustive characterization of embeddings into Sobolev spaces for coorbit spaces associated to the standard shearlet groups in dimension three, and Theorem 5.12, which formulates an analog for Toeplitz shearlet groups. For two-dimensional shearlet groups, this analysis had been performed in Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces, but for the three-dimensional cases, the results are completely new. They are also substantially more complicated than those for the two-dimensional case, due to the additional parameter describing the anisotropic scaling in the third dimension, and the additional variation in the shearing subgroup of the Toeplitz shearlet group.

In particular, while the Theorems 5.10 and 5.12 contain essentially complete information, the interpretation of these results, say with a view to investigating the smoothness space interpretation of coorbit spaces, and the influence of the dilation group on this issue, becomes a separate nontrivial problem, which we address in Section 6. We restrict the discussion to certain pertinent subcases, and analyze more closely the influence of the different components of the shearlet dilation group on the embedding behavior. Specifically, we investigate the role of the exponents describing the scaling subgroup on the one hand, and the choice of shearing subgroup on the other.

This allows to draw the following conclusions: For the important subcase of coorbit spaces of the kind $Co(L^{p})$, where $0<p<2$, there exists no embedding into Sobolev spaces with nontrivial smoothness exponent $k>0$, regardless of the choice of shearlet dilation group; see Corollary 6.3. Elements of these spaces can be understood as functions in $L^{2}$ with a non-trivial approximation rate – in the $L^{2}$-norm – with respect to any discrete shearlet system obtained by discretizing the continuous shearlet transform; see e.g. the discussion in Ref. \refcite[Section 1.1]MR3452925. Our observation makes clear that this type of decay alone does not guarantee smoothness. We then resort to (mostly) analyzing embeddings of $Co(L^{p}_{v})$ for suitable weights, and $0<p<2$. Here, it turns out that the embedding behavior depends on certain features of the shearlet dilation groups, and is independent of others. More precisely, the shearing subgroup has no influence (Corollary 6.5), whereas the scaling subgroup is critically influential (Theorem 6.8). The fact that the shearing subgroup has no influence is interesting because the coorbit spaces associated to different shearing subgroups do not coincide, by the results in Ref. \refciteRKDoktorarbeit. Thus the embedding behaviour into Sobolev spaces does not allow to distinguish different scales of shearlet coorbit spaces. This fact could probably have been expected, but it has been brought to light and rigourously established by our analysis.

As a further byproduct of our analysis, we obtain that anisotropic scaling is required to guarantee the existence of embeddings into Sobolev spaces with nontrivial smoothness parameter. This means that using multiples of the identity operator as the scaling subgroup never works, see Corollary 6.12 and the following Remark 6.14 (2). On the one hand, this observation is slightly surprising, as the target spaces $W^{k,p}$ of the embedding results exhibit no anisotropies. On the other hand, it is well in line with the fact that anisotropy is needed for the study of singularities, such as the wavefront set, via the decay behaviour of shearlet transforms. Here, anisotropic scaling is generally necessary [13], and – with additional restrictions – also sufficient [1].

Finally, we study for which groups the smoothness of elements of coorbit spaces improve as the decay requirements imposed by the coorbit spaces become more restrictive. This can be done by asking how the best possible parameter $k$ in the embedding $Co(L_{v}^{p})\hookrightarrow W^{k,q}$ scales as $p$ goes to zero. Again, our results show that this can be attributed to properties of the scaling subgroup alone; see Corollary 6.15 and the subsequent Remark.

2 Generalized Wavelet Transform and Shearlet Groups

In this section, we recall basic definitions underlying the continuous wavelet transform and generalized shearlet dilation groups.

2.1 Generalized Wavelet Transform

For a closed matrix group $H\leq\mathrm{GL}(\mathbb{R}^{d})$, which we also call dilation group in the following, we define the group $G:=\mathbb{R}^{d}\rtimes H$ generated by dilations with elements of $H$ and translations with the group law $(x,h)\circ(y,g):=(x+hy,hg)$. We denote integration with respect to a left Haar measure on $H$ with $\mathrm{d}h$, the associated left Haar measure on $G$ is then given by $d(x,h)=\lvert\det h\rvert^{-1}\mathrm{d}x\mathrm{d}h$. The Lebesgue spaces on $G$ are always defined through integration with respect to a Haar measure. The group $G$ acts on the space $\mathrm{L}^{2}(\mathbb{R}^{d})$ through the quasi-regular representation $\pi$ defined by $[\pi(x,h)f](y):=\lvert\det h\rvert^{-1/2}f(h^{-1}(y-x))$ for $f\in\mathrm{L}^{2}(\mathbb{R}^{d})$. The generalized continuous wavelet transform (with respect to $\psi\in\mathrm{L^{2}(\mathbb{R}^{d})}$) of $f$ is then given as the function $W_{\psi}f:G\to\mathbb{C}:(x,h)\mapsto\braket{\psi,\pi(x,h)f}.$ Important properties of the map $W_{\psi}:f\mapsto W_{\psi}f$ depend on $H$ and the chosen $\psi$. If the quasi-regular representation is square-integrable, which means that there exists a $0\neq\psi$ with $W_{\psi}\psi\in\mathrm{L}^{2}(G)$, and irreducible, then we call $H$ admissible and the map $W_{\psi}:\mathrm{L}^{2}(\mathbb{R}^{d})\to\mathrm{L}^{2}(G)$ is a multiple of an isometry, which gives rise to the (weak-sense) inversion formula

[TABLE]

i.e., each $f\in\mathrm{L}^{2}(\mathbb{R}^{d})$ is a continuous superposition of the wavelet system. According to results in Refs. \refciteMR1419174, \refciteMR2652610, the admissibility of $H$ can be characterized by the dual action defined by $G\mapsto\mathbb{R}^{d},(\xi,h)\mapsto p_{\xi}(h):=h^{-T}\xi$, where $p_{\xi}$ denotes the associated orbit map. In fact, $H$ is admissible iff the dual action has a single open orbit $\mathcal{O}:=H^{-T}\xi_{0}\subset\mathbb{R}^{d}$ of full measure for some $\xi_{0}\in\mathbb{R}^{d}$ and additionally the isotropy group $H_{\xi_{0}}:=\Set{h:p_{\xi_{0}}(h)=\xi_{0}}$ is compact.

2.2 Shearlet Groups

In order to state the definition given in Ref. \refciteAlbertiEtAl2017, we use the notation of $\mathfrak{gl}(\mathbb{R}^{d})$ for the set of all $d\times d$-matrices and let $\exp:\mathfrak{gl}(\mathbb{R}^{d})\to\mathrm{GL}(\mathbb{R}^{d})$ be the exponential map defined by the series $\exp(A):=\sum_{k=0}^{\infty}\frac{A^{k}}{k!}$ for every $A\in\mathfrak{gl}(\mathbb{R}^{d})$. We consider convergence of this series with respect to the norm $\lVert A\rVert_{\mathrm{op}}:=\sup_{\lvert x\rvert\leq 1}\lvert Ax\rvert$. Furthermore, we denote with $T(\mathbb{R}^{d})$ the set of upper triangular $d\times d$-matrices with one on their diagonals.

Definition 2.1** (Ref. \refciteAlbertiEtAl2017 Definition 1.).**

Let $H\subset\mathrm{GL}(\mathbb{R}^{d})$ be a closed, admissible dilation group. The group $H$ is called generalized shearlet dilation group if there exist two closed subgroups $S,D\subset\mathrm{GL}(\mathbb{R}^{d})$ such that

1. [i)]

$S$* is a connected abelian subgroup of $T(\mathbb{R}^{d})$,* 3. 2.

$D=\Set{\exp(rY)}{r\in\mathbb{R}}$* is a one-parameter group, where $Y\in\mathfrak{gl}(\mathbb{R}^{d})$ is a diagonal matrix and* 4. 3.

every $h\in H$ has a unique representation as $h=\pm ds$ for some $d\in D$ and $s\in S$.

$S$* is called the shearing subgroup of $H$, $D$ is called the scaling subgroup of $H$, and $Y$ is called the infinitesimal generator of $D$.*

We denote the canonical basis of $\mathbb{R}^{d}$ with $e_{1},\ldots,e_{d}$ and the identity matrix in $\mathrm{GL}(\mathbb{R}^{d})$ with $E_{d}$ or just $E$ if the dimension is clear from the context. The next result contains information about the structure of shearing subgroups. All generalized shearlet dilation groups in dimension $d$ share the same open dual orbit and isotropy group.

Lemma 2.2** (Ref. \refciteAlbertiEtAl2017 Proposition 11.).**

For a generalized shearlet dilation group $H$, the unique open dual orbit of $H$ is given by $\mathcal{O}=\mathbb{R}^{*}\times\mathbb{R}^{d-1}$ and the isotropy group of $\xi\in\mathcal{O}$ with respect to the dual action is given by $H_{\xi}=\Set{E_{d}}$.

Now, we introduce some concrete classes of shearlet groups, which we will further investigate in the next sections. The class of standard shearlet groups

[TABLE]

for $\delta\in\mathbb{R}$.

Remark 2.3**.**

In dimension three, these are the only possible generalized shearlet dilation groups (see Ref. \refciteAlbertiEtAl2017 remark 19).

3 Coorbit Spaces and Decompositions Spaces

Coorbit spaces are defined in terms of decay behavior of generalized wavelet transforms. To give a precise definition, we introduce weighted mixed $\mathrm{L}^{p}$-spaces on $G$, denoted by $\mathrm{L}^{p,q}_{v}(G)$ . By definition, this space is the set of functions

[TABLE]

with natural (quasi-)norm $\|\cdot\|_{\mathrm{L}^{p,q}_{v}}$. This definition is valid for $0<p,q<\infty$, for $p=\infty$ or $q=\infty$ the essential supremum has to be taken at the appropriate place instead. The function $v:G\to\mathbb{R}^{>0}$ is a weight function that fulfills the condition $v(ghk)\leq v_{0}(g)v(h)v_{0}(k)$ for some submultiplicative weight $v_{0}$. If the last condition is satisfied, we call $v$ left- and right moderate with respect to $v_{0}$. Thus, the expression $\|W_{\psi}f\|_{\mathrm{L}^{p,q}_{v}}$ can be read as a measure of wavelet coefficient decay of $f$. We will exclusively consider weights which only depend on $H$. The coorbit space $\mathrm{Co}\left(\mathrm{L}^{p,q}_{v}(\mathbb{R}^{d}\rtimes H)\right)$ is then defined as the space

[TABLE]

for some suitable wavelet $\psi$ and some control weight $w$ associated to $v$. The space $(\mathcal{H}^{1}_{w})^{\neg}$ denotes the space of antilinear functionals on $\mathcal{H}^{1}_{w}:=\left\{f\in\mathrm{L}^{2}(\mathbb{R}^{d}):W_{\psi}f\in\mathrm{L}^{1}_{w}(G)\right\}$ and $W(Y)$ for a function space $Y$ on $G$ denotes the Wiener amalgam space defined by $W_{Q}(Y):=\{f\in\mathrm{L}^{\infty}_{\text{loc}}(G)|M_{Q}f\in Y\}$ with quasi-norm $\|f\|_{W_{Q}(Y)}:=\|M_{Q}f\|_{Y}$ for $f\in W_{Q}(Y)$, where the maximal function $M_{Q}f$ for some suitable unit neighborhood $Q\subset G$ is $M_{Q}f:G\to[0,\infty],\ x\mapsto\operatorname*{ess\ sup}_{y\in xQ}|f(y)|.$

The appearance of the Wiener amalgam space in (2) is necessary to guarantee consistently defined quasi-Banach spaces in the case $\{p,q\}\cap(0,1)\neq\emptyset$, see Ref. \refciteRauCooSpaTheForQuaBanSpa and Ref. \refciteVoigtlaender2015PHD. In the classical coorbit theory for Banach spaces, which was developed in Refs. \refciteFeiGroeI, \refciteFeiGroeII, the Wiener amalgam space can be replaced by the simpler space $\mathrm{L}^{p,q}_{v}(G)$, see Ref. \refciteRauCooSpaTheForQuaBanSpa.

Many useful properties of these spaces are known and hold in the quasi-Banach space case as well as in the Banach space case. The most prominent examples of coorbit spaces associated to generalized wavelet transforms are the homogeneous Besov spaces and the modulation spaces. However, each shearlet group gives rise to its scale of coorbit spaces, as well; see Refs. \refciteDahHaeuTesCooSpaTheForTheToeSheTra, \refciteFuehCooSpaAndWavCoeDecOveGenDilGro, \refciteKLShearlets.

The starting point for the definition of decomposition spaces is the notion of an admissible covering $\mathcal{Q}=(Q_{i})_{i\in I}$ of some open set $\mathcal{O}\subset\mathbb{R}^{d}$ (Ref. \refciteFeichtingerGroebnerBanachSpacesOfDistributionsI) which is a family of nonempty sets such that

1. [i)]

$\bigcup_{i\in I}Q_{i}=\mathcal{O}$ and 3. 2.

$\sup_{i\in I}|\{j\in I:Q_{i}\cap Q_{j}\neq\emptyset\}|<\infty$.

The main tool for the localization is a special partition of unity $\Phi=(\varphi_{i})_{i\in I}$ subordinate to $\mathcal{Q}$, also called $\mathrm{L}^{p}$-BAPU (bounded admissible partition of unity), with the following properties

1. [i)]

$\varphi_{i}\in C_{c}^{\infty}(\mathcal{O})\quad\forall i\in I$, 3. 2.

$\sum_{i\in I}\varphi_{i}(x)=1\quad\forall x\in\mathcal{O}$, 4. 3.

$\varphi_{i}(x)=0$ for $x\in\mathbb{R}^{d}\setminus Q_{i}$ and $i\in I$, 5. 4.

if $1\leq p\leq\infty$: $\sup_{i\in I}\|\mathcal{F}^{-1}\varphi_{i}\|_{\mathrm{L}^{1}}<\infty$

if $0<p<1$: $\sup_{i\in I}|\det(T_{i})|^{\frac{1}{p}-1}\|\mathcal{F}^{-1}\varphi_{i}\|_{\mathrm{L}^{p}}<\infty$,

where we have to further assume in the case $0<p<1$ that the covering $\mathcal{Q}$ has the structure $Q_{i}=T_{i}Q+b_{i}$ with $T_{i}\in\mathrm{GL}(\mathbb{R}^{d})$, $b_{i}\in\mathbb{R}^{d}$ and an open, precompact set $Q$ ($\mathcal{Q}$ is then called a structured admissible covering). The definition of decomposition spaces requires one last ingredient, namely a weight $(u_{i})_{i\in I}$ such that there exists $C>0$ with $u_{i}\leq Cu_{j}$ for all $i,j\in I:Q_{i}\cap Q_{j}\neq\emptyset$, a weight with this property is also called $\mathcal{Q}$-moderate. The interpretation of this property is that the value of $(u_{i})_{i\in I}$ is comparable for indices corresponding to sets which are ”close” to each other. Finally, we define the decomposition space with respect to the covering $\mathcal{Q}$ and the weight $(u_{i})_{i\in I}$ with integrability exponents $0<p,q\leq\infty$ as

[TABLE]

for

[TABLE]

As the notation suggests, the decomposition spaces are independent of the precise choice of $\Phi$ (Ref. \refciteVoigtlaender2015PHD Corollary 3.4.11).

In order to describe coorbit spaces as decomposition spaces, we need to associate a covering of the frequencies to a given dilation group. This is done using the dual action $H\times\mathbb{R}^{d}\ni(h,\xi)\mapsto h^{-T}\xi$.

We then pick a well-spread family in $H$, i.e. a family of elements $(h_{i})_{i\in I}$ with the properties

1. [i)]

there exists a relatively compact neighborhood $U\subset H$ of the identity such that $\bigcup_{i\in I}h_{i}U=H$ – we say $(h_{i})_{i\in I}$ is $U$-dense in this case – and 3. 2.

there exists a neighborhood $V\subset H$ of the identity such that $h_{i}V\cap h_{j}V=\emptyset$ for $i\neq j$ – we say $(h_{i})_{i\in I}$ is $V$-separated in this case.

The dual covering induced by $H$ is then given by the family $\mathcal{Q}=(Q_{i})_{i\in I}$, where $Q_{i}=p_{\xi_{0}}(h_{i}U)$ for some $\xi_{0}$ with $H^{-T}\xi_{0}=\mathcal{O}$. It can be shown that well-spread families always exist, and that the induced covering is indeed an admissible covering in the sense of decomposition space theory, for which $\mathrm{L}^{p}$-BAPUs exist according to Ref. \refciteVoigtlaender2015PHD. Furthermore, there always exist induced coverings consisting of open and connected sets, an additional feature which can facilitate the investigations in some cases, see Ref. \refciteRKDoktorarbeit Corollary 2.5.9.

There always exists a discretization of the weight $v$, which enables a decomposition space description of the coorbit space.

Definition 3.1** (Ref. \refciteVoigtlaender2015PHD Definition 4.5.3.).**

For $q\in(0,\infty]$ and a weight $v:H\to(0,\infty)$, we define the weight $v^{(q)}:H\to(0,\infty),\ h\mapsto\lvert\det(h)\rvert^{\frac{1}{2}-\frac{1}{q}}v(h).$ Here, we set $\frac{1}{\infty}:=0$.

Theorem 3.2** (Ref. \refciteVoigtlaender2015PHD Theorem 4.6.3).**

Let $\mathcal{Q}$ be a covering of the dual orbit $\mathcal{O}$ induced by $H$, $0<p,q\leq\infty$ and $u=(u_{i})_{i\in I}$ a suitable weight, then the Fourier transform $\mathcal{F}:\mathrm{Co}\left(\mathrm{L}^{p,q}_{v}(\mathbb{R}^{d}\rtimes H)\right)\to\mathcal{D}(\mathcal{Q},\mathrm{L}^{p},\ell^{q}_{u})$ is an isomorphism of (quasi-) Banach spaces. The weight $(u_{i})_{i\in I}$ can be chosen as $u_{i}:=v^{(q)}(h_{i})$, where $(h_{i})_{i\in I}$ is the well-spread family used in the construction of $\mathcal{Q}$ and we call such a weight a $\mathcal{Q}-$discretization of $v$.

In order to apply the embedding results in Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces, we need explicitly given well-spread families in standard and Toeplitz shearlet groups.

Lemma 3.3**.**

Define

[TABLE]

for $n,m_{1},m_{2}\in\mathbb{Z}$ and $\epsilon\in\{\pm 1\}$. Then the family $\Gamma^{\lambda_{1},\lambda_{2}}:=\left(B^{\lambda_{1},\lambda_{2}}_{n,m_{1},m_{2},\epsilon}\right)_{(n,m_{1},m_{2},\epsilon)\in I}$ for $I:=\mathbb{Z}^{3}\times\{\pm 1\}$ is $U^{\lambda_{1},\lambda_{2}}$ -dense and -separated, with

[TABLE]

Furthermore, this well-spread family induces a covering $\mathcal{C}^{\lambda_{1},\lambda_{2}}$ of the associated dual orbit. For later reference, we define $A^{\lambda_{1},\lambda_{2}}_{n,m_{1},m_{2},\epsilon}:=\left(B^{\lambda_{1},\lambda_{2}}_{n,m_{1},m_{2},\epsilon}\right)^{-T}$.

Lemma 3.4**.**

Define

[TABLE]

for $n,m_{1},m_{2}\in\mathbb{Z}$ and $\epsilon\in\{\pm 1\}$. Then the family $\Gamma^{\delta}:=\left(B^{\delta}_{n,m_{1},m_{2},\epsilon}\right)_{(n,m_{1},m_{2},\epsilon)\in I}$ is $U^{\delta}$ -dense and -separated, with

[TABLE]

Furthermore, this well-spread family induces a covering $\mathcal{C}^{\delta}$ of the associated dual orbit. For later reference, we define $A^{\delta}_{n,m_{1},m_{2},\epsilon}:=\left(B^{\delta}_{n,m_{1},m_{2},\epsilon}\right)^{-T}$.

For the following investigation of the existence of embeddings of the associated shearlet coorbit into Sobolev spaces, it is not necessary to have an explicit description of these induced coverings, but the reader can find one in Ref. \refciteRKDoktorarbeit.

4 Embeddings into Sobolev spaces

Our goal in this section is to study the embedding behavior of the coorbit spaces $\mathrm{Co}\left(\mathrm{L}^{p,q}_{m}(\mathbb{R}^{3}\rtimes H^{\delta})\right)$ and $\mathrm{Co}\left(\mathrm{L}^{p,q}_{m}(\mathbb{R}^{3}\rtimes H^{\lambda_{1},\lambda_{2}})\right)$ associated to three dimensional shearlet groups into Sobolev spaces for $p,q\in(0,\infty]$ and a specific class of weights $m$ on $\mathrm{GL}(\mathbb{R}^{3})$.

This section is based on methods developed in Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces. There, the two dimensional case was considered and characterized to a large extent. Similar questions related to the embedding of (subspaces) of certain shearlet coorbit spaces into classical smoothness spaces were also investigated in Ref. \refciteDahlkeShearletCoorbitSpacesCompactlySupported (also in two dimensions).

At first, we introduce the necessary tools, in particular, a definition of Sobolev spaces for integrability exponents $0<p<1$, and another type of partition of unity. Afterwards, we explain what we precisely mean by an embedding of a coorbit space into a Sobolev space, which will depend on the identification of the coorbit space with a suitable decomposition space, and present the result in Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces which is a crucial tool in this chapter.

In the following sections, we apply the result and will compare coorbit spaces associated to different groups with regards to their embedding behavior. Surprisingly, the embedding behavior of shearlet groups in dimension three into Sobolev spaces is determined by the scaling subgroup of the group, which means the shearing part has no influence on the embedding behavior for parameters in a suitable range and the class of weights we consider.

We will also see how the group parameters influence the smoothness of the elements in the associated coorbit spaces.

We start by giving a definition of Sobolev spaces, which is completely classical in the Banach space case. The definition in the quasi-Banach space case used in Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces is inspired by the definition by Peetre [27] and the resulting spaces exhibit quite unexpected properties. For example, the definition as tuple of functions is motivated by the fact that $(\partial^{\alpha}f)_{\alpha\in\mathbb{N}_{0}^{d},\lvert\alpha\rvert_{1}\leq k}\mapsto f_{0}$ is not injective in general. Since we heavily rely on the results in Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces , we adhere to the definition employed there.

Definition 4.1** (Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces Subsection 2.1).**

We define

[TABLE]

for $q\in[1,\infty]$, where $\partial^{\alpha}f$ denotes the weak partial derivative of $f$ and for $0<q<1$, let $W^{k,q}(\mathbb{R}^{d})$ be the closure of

[TABLE]

in the product $\prod_{\alpha\in\mathbb{N}_{0}^{d},\ |\alpha|_{1}\leq k}L^{q}(\mathbb{R}^{d})$.

We will define suitable differentiation operators on decomposition spaces by resorting to special partitions of unity.

Definition 4.2** (Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces Definition 2.4.).**

Let $\mathcal{Q}=(T_{i}Q+b_{i})_{i\in I}$ be a structured admissible covering of some open set $\mathcal{O}\subset\mathbb{R}^{d}$ and let $(\phi_{i})_{i\in I}$ be a partition of unity subordinate to $\mathcal{Q}$ with $\phi_{i}\in C_{c}^{\infty}(\mathcal{O})$. The normalized version of $\phi_{i}$ is given by $\phi_{i}^{\#}:\mathbb{R}^{d}\to\mathbb{C},\ x\mapsto\phi_{i}(T_{i}x+b_{i})$ for $i\in I$. Additionally, we say that $(\phi_{i})_{i\in I}$ is a regular partition of unity if $\sup_{i\in I}\left\lVert\partial^{\alpha}\phi_{i}^{\#}\right\rVert_{\mathrm{sup}}<\infty$ for all $\alpha\in\mathbb{N}_{0}^{d}$.

The connection between regular partitions of unity and $L^{p}$-BAPUs as well as their existence for our preferred type of covering are given by the next lemmata.

Lemma 4.3** (Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces Corollary 2.7.).**

If $(\phi_{i})_{i\in I}$ is a regular partition of unity subordinate to a structured admissible covering $\mathcal{Q}$, then $(\phi_{i})_{i\in I}$ is an $L^{p}$-BAPU for every $p\in(0,\infty]$.

Lemma 4.4** (Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces Theorem 2.8.).**

Every structured admissible covering admits a subordinate regular partition of unity.

These preparations allow us to define differentiation operators on decomposition spaces and to specify what we mean by an embedding of a decomposition space or coorbit space into a Sobolev space.

Definition 4.5**.**

Let $\mathcal{Q}=(Q_{i})_{i\in I}$ be a structured admissible covering of some open set $\mathcal{O}\subset\mathbb{R}^{d}$ and $(u_{i})_{i\in I}$ a $\mathcal{Q}$-moderate weight on $I$. For $p,r\in(0,\infty]$, let $(\phi_{i})_{i\in I}$ be a regular partition of unity for $\mathcal{Q}$.

1. [i)]

We say that $\mathcal{D}(\mathcal{Q},\mathrm{L}^{p},\ell^{r}_{u})$ admits a partial differential operator (with respect to $q$) for $\alpha\in\mathbb{N}_{0}^{k}$ with $k\in\mathbb{N}_{0}$ if the map

[TABLE]

is well-defined, bounded, with unconditional convergence of the series. 3. 2.

We write $\mathcal{D}(\mathcal{Q},\mathrm{L}^{p},\ell^{r}_{u})\hookrightarrow W^{k,q}(\mathbb{R}^{d})$ if $\mathcal{D}(\mathcal{Q},\mathrm{L}^{p},\ell^{r}_{u})$ admits a partial differential operator in the sense of i) and

[TABLE]

is well-defined, bounded and injective for $q\geq 1$.

In the case $0<q<1$, we require that the map

[TABLE]

is well-defined and bounded. 4. 3.

We write $\mathrm{Co}\left(\mathrm{L}^{p,r}_{m}(\mathbb{R}^{d}\rtimes H)\right)\hookrightarrow W^{k,q}(\mathbb{R}^{d})$ for $r,q\in(0,\infty]$, where $H$ is an admissible group with dual orbit $\mathcal{O}$ and $m$ is a weight on $H$ that is right moderate with respect to a locally bounded weight on $H$ if the associated isomorphic decomposition space in the sense of Theorem 3.2 satisfies $\mathcal{D}(\mathcal{Q},\mathrm{L}^{p},\ell^{r}_{u})\hookrightarrow W^{k,q}(\mathbb{R}^{d}).$

Remark 4.6**.**

1. [i)]

This definition is inspired by Theorem 3.4. and Corollary 4.5. in Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces. We had to adapt the definition of the partial differential operator slightly (by changing $\hat{f}$ to $f$) because Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces works with space-side decomposition spaces. Despite this change, all the results and proofs in Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces can be applied to our setting by just interchanging $\hat{f}$ to $f$ at the appropriate place. 3. 2.

A motivation for this definition is the fact that if $\mathcal{D}(\mathcal{Q},\mathrm{L}^{p},Y)$ admits a partial differential operator $\partial_{*}^{\alpha}$, then $\partial_{*}^{\alpha}f=\partial^{\alpha}\left(\mathcal{F}^{-1}f\right)$ for all $f\in\mathcal{D}(\mathcal{Q},\mathrm{L}^{p},Y)\cap C_{c}^{\infty}(\mathcal{O})$, according to Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces Theorem 3.4. Furthermore, $\mathcal{D}(\mathcal{Q},\mathrm{L}^{p},Y)\hookrightarrow W^{k,q}(\mathbb{R}^{d})$ implies $\partial^{\alpha}(\iota_{q}^{(k)}f)=\partial_{*}^{\alpha}f$ for $f\in\mathcal{D}(\mathcal{Q},\mathrm{L}^{p},Y)$ and $|\alpha|_{1}\leq k$, according to Corollary 3.5 in Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces. Hence, $\mathrm{Co}(\mathrm{L}^{p,q}_{m}(\mathbb{R}^{d}\rtimes H)\hookrightarrow W^{k,q}(\mathbb{R}^{d})$ implies

[TABLE]

for $f\in\mathrm{Co}(\mathrm{L}^{p,q}_{m}(\mathbb{R}^{d}\rtimes H)\cap\mathcal{F}^{-1}(C_{c}^{\infty}(\mathcal{O}))$ and $|\alpha|\leq k$, where we used that $\mathcal{F}f=\widehat{f}$ for $f\in\mathcal{F}^{-1}(C_{c}^{\infty}(\mathcal{O}))\subset\mathrm{L}^{2}(\mathbb{R}^{d})$, according to Ref. \refciteVoigtlaender2015PHD Theorem 3.2.1. Here, $\widehat{f}$ denotes the usual Fourier transform of a function. 4. 3.

The phrase the associated isomorphic decomposition space in (3) is justified, because the same reasoning as in the proof of Corollary 3.6.4 in Ref. \refciteRKDoktorarbeit shows that this space is uniquely determined.

We will employ the following theorem to obtain sufficient and necessary conditions for the embedding of shearlet coorbit spaces in three dimensions into Sobolev spaces in the sense of the last definition. Here, we define $p^{\triangledown}:=\min\left\{p,p^{\prime}\right\}$, where $p^{\prime}=\infty$ for $0<p<1$ and $p^{\prime}$ is the usual conjugate exponent for $p\geq 1$.

Theorem 4.7** (Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces Corollary 5.2).**

Let $\mathcal{Q}=(T_{i}Q+b_{i})_{i\in I}$ be a structured admissible covering of some open set $\mathcal{O}\subset\mathbb{R}^{d}$. Let $p,q,r\in(0,\infty],\ k\in\mathbb{N}_{0}$ and let $u=(u_{i})_{i\in I}$ be a $\mathcal{Q}$-moderate weight on $I$. Define the weight

[TABLE]

for $i\in I$, where $\left\lVert\cdot\right\rVert$ is some norm on $\mathrm{GL}(\mathbb{R}^{d})$. Then the following hold:

1. [i)]

If $p\leq q$ and $\frac{w^{\{q\}}}{u}\in\ell^{q^{\triangledown}\cdot\left(r/q^{\triangledown}\right)^{\prime}}(I),$ then $\mathcal{D}(\mathcal{Q},\mathrm{L}^{p},\ell^{r}_{u})\hookrightarrow W^{k,q}(\mathbb{R}^{d}).$ 3. 2.

If $\mathcal{D}(\mathcal{Q},\mathrm{L}^{p},\ell^{r}_{u})\hookrightarrow W^{k,q}(\mathbb{R}^{d})$, then $p\leq q$ and

[TABLE]

Remark 4.8**.**

Since the necessary and sufficient conditions coincide for $q\in(0,2]\cup\{\infty\}$, this theorem provides a characterization for the embedding

[TABLE]

for $q$ in this range.

Our computations will make repeated use of the symbol $\asymp$ denoting equivalence between families of scalars. More precisely, $(a_{i})_{i\in I}\asymp(b_{i})_{i\in I}$ means there exist constants $C,c>0$ such that $ca_{i}\leq b_{i}\leq Ca_{i}$ for all $i\in I$. We will write this as $a_{i}\asymp b_{i}$ if the set $I$ is clear from the context. Moreover, we denote with $\lfloor a\rfloor$ and $\lceil b\rceil$ the biggest integer smaller than $a$ and the smallest integer bigger than $b$, respectively.

We note the following elementary but useful observations. The proof is omitted.

Lemma 4.9**.**

We have the asymptotic relation $\sum_{m=m_{0}}^{\infty}m^{\rho}\asymp m_{0}^{1+\rho}$ for $m_{0}\in\mathbb{N}$ and $\rho<-1$.

If $(a_{i})_{i\in I}$ is a family of real numbers with the property that there exists $\delta\geq 0$ such that $a_{i}\geq\delta$ for all $i\in I$, then

1. [i)]

$\lfloor a_{i}\rfloor+1\asymp a_{i}+1$, 3. 2.

if $\delta>0$, then $\lceil a_{i}\rceil^{s}\asymp a_{i}^{s}$ for all $s\in\mathbb{R}$, 4. 3.

if $\delta>0$, then $\lfloor a_{i}\rfloor+1\asymp a_{i}$, 5. 4.

if $\delta\geq 1$, then $\lfloor a_{i}\rfloor^{s}\asymp a_{i}^{s}$ for all $s\in\mathbb{R}$, 6. 5.

if $\delta\geq 1$, then $\left(\lfloor a_{i}\rfloor+1\right)\lfloor a_{i}\rfloor\asymp\lfloor a_{i}\rfloor^{2}\asymp a_{i}^{2}$.

The next lemma is also easily verified.

Lemma 4.10**.**

Let $(a_{i})_{i\in I}$, $(b_{i})_{i\in I}$ and $(c_{i})_{i\in I}$ be families of nonnegative numbers with $b_{i},c_{i}\leq a_{i}$ for all $i\in I$. If there exist sets $I_{1},I_{2}\subset I$ with $I_{1}\cup I_{2}=I$, then $a_{i}\asymp b_{i}$ on $I_{1}$ and $a_{i}\asymp c_{i}$ on $I_{2}$ imply $a_{i}\asymp b_{i}+c_{i}$ on $I$ and if $a_{i}>0$ for all $i\in I$, then $a_{i}^{s}\asymp(b_{i}+c_{i})^{s}$ for $i\in I$ and all $s\in\mathbb{R}$

Furthermore, under the above assumptions, we have $\sum_{i\in I}a_{i}<\infty\Leftrightarrow\sum_{i\in I}(b_{i}+c_{i})<\infty,$ and if $a_{i}>0$ for all $i\in I$, then $\sum_{i\in I}a_{i}^{s}<\infty\Leftrightarrow\sum_{i\in I}(b_{i}+c_{i})^{s}<\infty$ for all $s\in\mathbb{R}$.

5 Embeddings of shearlet coorbit spaces into Sobolev spaces

Now that all preliminaries are dealt with, we can take up the task of applying the general results described in the previous sections to the special setup of shearlet dilation groups in dimension three. Following the programme developed above, we now need to use, for each group under consideration, the well spread families in Lemma 3.3 and Lemma 3.4, in order to apply Theorem 4.7. We will treat the case of the standard shearlet groups in more or less full detail. By comparison, our treatment of the Toeplitz shearlet groups is less complete. Here the central estimates, and the way they are obtained, turn out to be very similar to the calculations made for the standard shearlet case, which is why we refrain from including all the details. These can be found in Ref. \refciteRKDoktorarbeit.

5.1 The standard shearlet groups

In this subsection, we consider the class of standard shearlet groups $H^{\lambda_{1},\lambda_{2}}$ from section 2 and their associated coorbit spaces in dimension three. Our first task is to prepare the application of Theorem 4.7.

According to Theorem 3.2 and Lemma 3.3, the map

[TABLE]

is an isomorphism for $p,r\in(0,\infty]$ and any weight $v$ on $H^{\lambda_{1},\lambda_{2}}$ that is right moderate with respect to a locally bounded weight on $H^{\lambda_{1},\lambda_{2}}$ if $u$ is a $\mathcal{C}^{\lambda_{1},\lambda_{2}}$-discretization of $v$. Here, we set $I:=\mathbb{Z}^{3}\times\{\pm 1\}$ and

[TABLE]

where the matrices $A^{\lambda_{1},\lambda_{2}}_{n,m_{1},m_{2},\epsilon}$ were defined in Lemma 3.3 and the weight $v^{(r)}$ is defined as in Definition 3.1.

As in Ref. \refciteVoigtlaenderEmbeddingsOfDecSpInSobolebAndBVSPaces, we will restrict attention to weights $v^{(\alpha,\beta)}:h\mapsto\lvert h_{1,1}\rvert^{\alpha}\left\lVert h^{-T}\right\rVert^{\beta},$ where $h_{1,1}$ denotes the component in the first row and first column of $h$ and $\alpha\in\mathbb{R},\beta\geq 0$. Informally speaking, this means that $\alpha$ influences how we gauge the scaling factor of the matrix and $\beta$ how we weigh the shearing part. This weight is a generalization of the weight that is considered in Ref. \refciteDahlkeShearletCoorbitSpacesCompactlySupported in the two-dimensional case.

For this specific weight, the associated weight $u^{(\alpha,\beta)}$ for the decomposition space is given by

[TABLE]

according to Definition 3.1, where one has to keep in mind that the matrices $A^{\lambda_{1},\lambda_{2}}_{n,m_{1},m_{2},\epsilon}$ are the inverse transposes of a well spread family.

The weight $w^{\{q\}}$ in Theorem 4.7 is in this setting given by

[TABLE]

An application of Theorem 4.7 boils down to the study of the sequence $\zeta^{\lambda_{1},\lambda_{2}}$ defined by

[TABLE]

More precisely, we want to characterize $\zeta^{\lambda_{1},\lambda_{2}}\in\ell^{\theta}(I)$ for $I=\mathbb{Z}^{3}\times\{\pm 1\}$ and $\theta\in(0,\infty]$. Since $\zeta_{n,m_{1},m_{2},1}^{\lambda_{1},\lambda_{2}}=\zeta_{n,m_{1},m_{2},-1}^{\lambda_{1},\lambda_{2}}\geq 0$, it is sufficient to characterize $(\zeta_{n,m_{1},m_{2},1}^{\lambda_{1},\lambda_{2}})_{n,m_{1},m_{2}}\in\ell^{\theta}(\mathbb{Z}^{3})$. Furthermore, $\zeta_{n,m_{1},m_{2},1}^{\lambda_{1},\lambda_{2}}=\psi_{n,m_{1},m_{2}}^{(a,\beta)}+\psi_{n,m_{1},m_{2}}^{(a,\beta-k)}$ with $\psi_{n,m_{1},m_{2}}^{(a,\beta)}:=2^{na}\left\lVert A^{\lambda_{1},\lambda_{2}}_{n,m_{1},m_{2},1}\right\rVert^{-\beta}$ and $a:=\alpha+(1+\lambda_{1}+\lambda_{2})\left(\frac{1}{2}-\frac{1}{r}+\frac{1}{p}-\frac{1}{q}\right).$

Note that $\psi_{n,m_{1},m_{2}}^{(a,b)}\geq 0$, which implies the equivalence

[TABLE]

In summary, our task is reduced to the investigation of the sequence $\psi^{(a,b)}$ for $a,b\in\mathbb{R}$ with $b\geq 0$ and to finding a characterization in terms of $a,b,\theta,\lambda_{1},\lambda_{2}$ for $\psi^{(a,b)}\in\ell^{\theta}(\mathbb{Z}^{3})$ with $a,\lambda_{1},\ \lambda_{2}\in\mathbb{R},\ b\geq 0\text{ and }\theta\in(0,\infty].$ Since all norms on $\mathrm{GL}(\mathbb{R}^{3})$ are equivalent, and equivalent norms lead to the same conditions for membership of the considered sequences in $\ell^{\theta}$-spaces, from now on, we will consider the norm $\left\lVert h\right\rVert=\sum_{1\leq i,j\leq 3}|h_{i,j}|,$ where $h_{i,j}$ are the components of the matrix $h\in\mathrm{GL}(\mathbb{R}^{3})$. The relation $\psi^{(a,b)}_{0,m_{1},m_{2}}=\left\lVert A^{\lambda_{1},\lambda_{2}}_{0,m_{1},m_{2},1}\right\rVert^{-b}=(3+|m_{1}|+|m_{2}|)^{-b}$ shows that $b\geq 0$ is necessary for $\psi^{(a,b)}\in\ell^{\theta}(\mathbb{Z}^{3})\subset\ell^{\infty}(\mathbb{Z}^{3})$. This is also the reason why we restrict the weights $v^{(\alpha,\beta)}$ to the range of parameters $\alpha\in\mathbb{R}$ and $\beta\geq 0$.

In the following subsections, we will focus on the case $\lambda_{1}\leq\lambda_{2}$ and associated sub-cases. Since $\left\lVert A^{\lambda_{1},\lambda_{2}}_{n,m_{1},m_{2},1}\right\rVert=2^{n}+2^{n\lambda_{1}}+2^{n\lambda_{2}}+2^{\lambda_{1}}|m_{1}|+2^{\lambda_{2}}|m_{2}|$ is invariant under a change of the index $1$ to $2$ and vice versa, the results for the case $\lambda_{2}\leq\lambda_{1}$ follow by interchanging $\lambda_{1}$ and $\lambda_{2}$ in the conditions of the appropriate sub-case.

5.2 Standard Shearlet Group: case $1\leq\lambda_{1}\leq\lambda_{2}$

The general proof strategy consists in determining the asymptotic behavior of $\left\lVert A^{\lambda_{1},\lambda_{2}}_{n,m_{1},m_{2},1}\right\rVert$ on suitable subsets of $\mathbb{Z}^{3}$, and then combining the different conditions on the exponents and weights that arise from the requirement that summation over each subset converges. More precisely, we break $\mathbb{Z}^{3}$ down into two discrete half spaces and then further into eight octants, and study summation over these subsets.

Definition 5.1**.**

We define the subsets of $\mathbb{Z}^{3}$

[TABLE]

Furthermore, we let $M^{\dagger}=\bigcup_{i=1,\ldots,4}M_{i}^{\dagger}$, for $\dagger\in\{+,-\}$.

Note that the union of all sets in the above definition is $\mathbb{Z}^{3}$. We introduce these sets because we want to use $\psi^{(a,b)}\in\ell^{\theta}(\mathbb{Z}^{3})\Longleftrightarrow\psi^{(a,b)}\in\ell^{\theta}(M^{\dagger}_{i})\text{ for all }\dagger\in\{\pm\},i\in\{1,\ldots,4\}.$

Lemma 5.2**.**

The following asymptotic relations hold for $(n,m_{1},m_{2})$ in the given sets:

[TABLE]

Proof 5.3**.**

We have $\left\lVert A^{\lambda_{1},\lambda_{2}}_{n,m_{1},m_{2},1}\right\rVert=2^{n}+2^{n\lambda_{1}}+2^{n\lambda_{2}}+2^{n\lambda_{1}}|m_{1}|+2^{n\lambda_{2}}|m_{2}|.$ Hence, if $n\geq 0$, the inequality $2^{n\lambda_{2}}\leq 2^{n}+2^{n\lambda_{1}}+2^{n\lambda_{2}}\leq 3\cdot 2^{n\lambda_{2}}$ shows that

[TABLE]

holds and for $n<0$, the inequality $2^{n}\leq 2^{n}+2^{n\lambda_{1}}+2^{n\lambda_{2}}\leq 3\cdot 2^{n}$ implies $\left\lVert A^{\lambda_{1},\lambda_{2}}_{n,m_{1},m_{2},1}\right\rVert\asymp 2^{n}+2^{\lambda_{1}}|m_{1}|+2^{\lambda_{2}}|m_{2}|$. The rest follows by observing that the sets $M_{i}^{\dagger}$ for $i\in\{1,\ldots,4\}$ and $\dagger\in\{\pm\}$ are precisely chosen in a way to ensure that one term dominates the other terms in these asymptotic norm expressions.

The next step consists in using these asymptotic relations in order to characterize $\psi^{(a,b)}\in\ell^{\theta}(\mathbb{Z}^{3})$. We first consider $\theta=\infty$.

Lemma 5.4**.**

For $1\leq\lambda_{1}\leq\lambda_{2}$, $a\in\mathbb{R}$ and $b\geq 0$, we have

[TABLE]

Proof 5.5**.**

We will determine successively necessary and sufficient conditions for $\psi^{(a,b)}\in\ell^{\infty}(M_{i}^{\dagger})$ for $i\in\{1,\ldots,4\}$ and $\dagger\in\{\pm\}$.

1. [i)]

The set $M_{1}^{+}$: In this case, we have

[TABLE]

if and only if $a-b\lambda_{2}\leq 0$. In $(\ast)$ we used that the expression $|m_{2}|^{-b}$ achieves its maximum with respect to $(m_{1},m_{2})$ with the given restriction for the choice $(m_{1},m_{2})=(0,1)$. 3. 2.

The set $M_{2}^{+}$: In this case, we have

[TABLE]

if and only if $a-b\lambda_{2}\leq 0$. In $(\ast)$ we used that the expression we take the supremum of is independent of $(m_{1},m_{2})$. 4. 3.

The set $M_{3}^{+}$: In this case, we have

[TABLE]

if and only if $a-b\lambda_{2}\leq 0$. In $(\ast)$ we used that the expression $|m_{1}|^{-b}$ achieves its maximum with respect to $(m_{1},m_{2})$ with the given restriction for the choice $(m_{1},m_{2})=(\lceil 2^{n(\lambda_{2}-\lambda_{1})}\rceil,0)$. 5. 4.

The set $M_{4}^{+}$: In this case, we have

[TABLE]

if and only if $a-b\lambda_{2}\leq 0$. In $(\ast)$ we used that the expression we take the supremum of is independent of $(m_{1},m_{2})$. 6. 5.

The set $M^{+}$: The previous four cases and Lemma 4.10 combined with Lemma 5.2 lead to

[TABLE]

if and only if $a-b\lambda_{2}\leq 0$. 7. 6.

The set $M_{1}^{-}$: In this case, we have

[TABLE]

if and only if $a-b\geq 0$. In $(\ast)$ we used that the expression $|m_{2}|^{-b}$ achieves its maximum with respect to $(m_{1},m_{2})$ with the given restriction for the choice $(m_{1},m_{2})=(0,\lceil 2^{n(1-\lambda_{2})}\rceil)$. 8. 7.

The set $M_{2}^{-}$: In this case, we have

[TABLE]

if and only if $a-b\geq 0$. In $(\ast)$ we used that the expression we take the supremum of is independent of $(m_{1},m_{2})$. 9. 8.

The set $M_{3}^{-}$: In this case, we have

[TABLE]

if and only if $a-b\geq 0$. In $(\ast)$ we used that the expression $|m_{1}|^{-b}$ achieves its maximum with respect to $(m_{1},m_{2})$ with the given restriction for the choice $(m_{1},m_{2})=(\lceil 2^{n(1-\lambda_{1})}\rceil,0)$. 10. 9.

The set $M_{4}^{-}$: In this case, we have

[TABLE]

if and only if $a-b\geq 0$. In $(\ast)$ we used that the expression we take the supremum of is independent of $(m_{1},m_{2})$. 11. 10.

The set $M^{-}$: The previous four cases and Lemma 4.10 combined with Lemma 5.2 lead to

[TABLE]

if and only if $a-b\geq 0$. 12. 11.

The set $\mathbb{Z}^{3}$: To summarize our results, we have $\psi^{(a,b)}\in\ell^{\infty}(\mathbb{Z}^{3})<\infty$ if and only if $b\leq a\leq b\lambda_{2}$.

The next step is to consider the remaining $\theta<\infty$.

Lemma 5.6**.**

For $1\leq\lambda_{1}\leq\lambda_{2}$, $a\in\mathbb{R}$ and $b\geq 0$, and $\theta\in(0,\infty)$, we have $\psi^{(a,b)}\in\ell^{\theta}(\mathbb{Z}^{3})$ if and only if $b\theta>2$ and $\frac{\lambda_{1}+\lambda_{2}-2}{\theta}+b<a<\frac{\lambda_{1}-\lambda_{2}}{\theta}+b\lambda_{2}.$

Proof 5.7**.**

We proceed as in the previous proof:

1. [i)]

The set $M_{1}^{+}$: In this case, we have

[TABLE]

if and only if $b\theta>2$ and $\theta(a-b\lambda_{2})+\lambda_{2}-\lambda_{1}<0$. 3. 2.

The set $M_{2}^{+}$: We can decompose the set $M_{2}^{+}$ as $M_{2}^{+}=M_{2}^{+{{}^{\prime\prime}}}\cup M_{2}^{+{{}^{\prime}}},$ where

[TABLE]

Since $M_{2}^{+{{}^{\prime\prime}}}\subset M_{1}^{+}$, the conditions $b\theta>2$ and $\theta(a-b\lambda_{2})+\lambda_{2}-\lambda_{1}<0$ ensure also $\left\lVert\psi^{(a,b)}\right\rVert_{\ell^{\theta}(M_{2}^{+{{}^{\prime\prime}}})}<\infty$. For the set $M_{2}^{+{{}^{\prime}}}$, we get

[TABLE]

if and only if $\theta(a-b\lambda_{2})<0$, which is weaker condition than $\theta(a-b\lambda_{2})+\lambda_{2}-\lambda_{1}<0$ since $\lambda_{2}-\lambda_{1}\geq 0$. 4. 3.

The set $M_{3}^{+}$: In this case, we have for $b\theta>2$ the following additional condition

[TABLE]

if and only if $\theta(a-b\lambda_{2})+\lambda_{2}-\lambda_{1}<0$. 5. 4.

The set $M_{4}^{+}$: In this case, we have

[TABLE]

if and only if $\theta(a-b\lambda_{2})+\lambda_{2}-\lambda_{1}<0$. In $(\ast)$ we used that for $0\leq m_{1}\leq\lfloor 2^{n(\lambda_{2}-\lambda_{1})}\rfloor$ the inequality $0\leq m_{2}\leq\lfloor 2^{n(\lambda_{1}-\lambda_{2})}m_{1}\rfloor$ implies $m_{2}\in\{0,1\}$, which means that the value of the sum we omitted in this step is in $\{1,2\}$. 6. 5.

The set $M^{+}$: The previous four cases and Lemma 4.10 combined with Lemma 5.2 lead to

[TABLE]

if and only if $\theta(a-b\lambda_{2})+\lambda_{2}-\lambda_{1}<0$ and $b\theta>2$ for the set $M^{+}:=\cup_{i=1}^{4}M_{i}^{+}$. 7. 6.

The set $M_{1}^{-}$: In this case, we have under the already established condition $b\theta>2$

[TABLE]

if and only if and $\theta(a-b)+2-\lambda_{2}-\lambda_{1}>0$. 8. 7.

The set $M_{2}^{-}$: In this case, we have

[TABLE]

We expand the sum and consider the two following series separately, first

[TABLE]

if and only if $\theta(a-b)+2-\lambda_{2}-\lambda_{1}>0$ and $\theta(a-b)+1-\lambda_{1}>0,$ where the first inequality implies the second inequality because $\lambda_{2}-1\geq 0$ and in $(\ast)$ we used $\sum_{m=1}^{n}m=\frac{1}{2}n(n+1).$

Now, we consider the remaining series

[TABLE]

if and only if $\theta(a-b)+1-\lambda_{2}>0$, which is implied by the already established inequality $\theta(a-b)+2-\lambda_{2}-\lambda_{1}>0$ since $\lambda_{1}-1\geq 0$.

In summary, we have $\left\lVert\psi^{(a,b)}\right\rVert_{\ell^{\theta}(M^{-}_{2})}^{\theta}<\infty\Longleftrightarrow\theta(a-b)+2-\lambda_{2}-\lambda_{1}>0.$ 9. 8.

The set $M_{3}^{-}$: In this case, we have under the already established condition $b\theta>2$

[TABLE]

if and only if $\theta(a-b)+2-\lambda_{2}-\lambda_{1}>0$. 10. 9.

The set $M_{4}^{-}$: In this case, we have

[TABLE]

and this sum is finite if and only if the following two sums are finite. The first sum is

[TABLE]

if and only if $\theta(a-b)+2-\lambda_{2}-\lambda_{1}>0$, where we used in $(\ast)$ again $\sum_{m=1}^{n}m=\frac{1}{2}n(n+1)$. The second sum is

[TABLE]

if and only if $\theta(a-b)+1-\lambda_{1}>0$, which is a weaker condition than for the first sum because $\lambda_{2}\geq 1$. 11. 10.

The set $M^{-}$: The previous four cases and Lemma 4.10 combined with Lemma 5.2 lead to

[TABLE]

if and only if $\theta(a-b)+2-\lambda_{2}-\lambda_{1}>0$ and $b\theta>2$ for the set $M^{-}:=\cup_{i=1}^{4}M_{i}^{-}$. 12. 11.

The set $\mathbb{Z}^{3}$: To summarize our results, we have $\psi^{(a,b)}\in\ell^{\theta}(M^{-}\cup M^{+})=\ell^{\theta}(\mathbb{Z}^{3})<\infty$ if and only if $b\theta>2$ and

[TABLE]

If we put the last results together, we get a complete characterization for these $1\leq\lambda_{1}\leq\lambda_{2}$.

Corollary 5.8**.**

The following are equivalent to $\psi^{(a,b)}\in\ell^{\theta}(\mathbb{Z}^{3})$ for $1\leq\lambda_{1}\leq\lambda_{2}$, $a\in\mathbb{R}$ and $b\geq 0$:

[TABLE]

5.3 Standard Shearlet Group: remaining cases

By proceeding in a completely analogous manner in the cases $\lambda_{1}\leq\lambda_{2}\leq 1$ and $\lambda_{1}\leq 1\leq\lambda_{2}$, we achieve for these remaining cases similar results, which we summarize in the next corollary.

Corollary 5.9**.**

The following are equivalent to $\psi^{(a,b)}\in\ell^{\theta}(\mathbb{Z}^{3})$ for $a\in\mathbb{R}$ and $b\geq 0$:

1. [i)]

If $1\leq\lambda_{1}\leq\lambda_{2}$:

[TABLE] 3. 2.

If $\lambda_{1}\leq\lambda_{2}\leq 1$:

[TABLE] 4. 3.

If $\lambda_{1}\leq 1\leq\lambda_{2}$:

[TABLE]

And the case for $\lambda_{2}\leq\lambda_{1}$ corresponds to the respective case for $\lambda_{1}\leq\lambda_{2}$ by interchanging $\lambda_{1}$ and $\lambda_{2}$.

Recall that we were actually interested in conditions that characterize the finiteness of the norm of $\zeta_{n,m_{1},m_{2},1}^{\lambda_{1},\lambda_{2}}=\psi_{n,m_{1},m_{2}}^{(a,\beta)}+\psi_{n,m_{1},m_{2}}^{(a,\beta-k)}$ in order to apply Theorem 4.7 to decide and characterize the existence of an embedding $\mathrm{Co}(L^{p,r}_{m}(\mathbb{R}^{d}\rtimes H^{\lambda_{1},\lambda_{2}}))\hookrightarrow W^{k,q}(\mathbb{R}^{d})$ in the sense of Definition 4.5.

Theorem 5.10**.**

Let $p,q,r\in(0,\infty]$, $k\in\mathbb{N}_{0}$ and set $\gamma:=\left(\frac{1}{2}-\frac{1}{r}+\frac{1}{p}-\frac{1}{q}\right).$ The embedding $\mathrm{Co}(\mathrm{L}^{p,r}_{v^{(\alpha,\beta)}}(\mathbb{R}^{3}\rtimes H^{\lambda_{1},\lambda_{2}}))\hookrightarrow W^{k,q}(\mathbb{R}^{3})$ holds for $q\in(0,2]\cup\{\infty\},$ if and only if $p\leq q$ and

1. [i)]

for $r\leq q^{\triangledown}$

1. (a)

the inequalities

[TABLE]

hold if $1\leq\lambda_{1}\leq\lambda_{2}$, 2. (b)

the inequalities

[TABLE]

hold if $\lambda_{1}\leq\lambda_{2}\leq 1$, 3. (c)

the inequalities

[TABLE]

hold if $\lambda_{1}\leq 1\leq\lambda_{2}$. 3. 2.

for $r>q^{\triangledown}$ the inequality $\beta>k+2\left(\frac{1}{q^{\triangledown}}-\frac{1}{r}\right)$ and

1. (a)

the inequalities

[TABLE]

hold if $1\leq\lambda_{1}\leq\lambda_{2}$, 2. (b)

the inequalities

[TABLE]

hold if $\lambda_{1}\leq\lambda_{2}\leq 1$, 3. (c)

the inequalities

[TABLE]

hold if $\lambda_{1}\leq 1\leq\lambda_{2}$.

For the remaining case $q\in(2,\infty)$, the given conditions are sufficient for the embedding $\mathrm{Co}(\mathrm{L}^{p,r}_{v^{(\alpha,\beta)}}(\mathbb{R}^{3}\rtimes H^{\lambda_{1},\lambda_{2}})\hookrightarrow W^{k,q}(\mathbb{R}^{3})$, and necessary for this embedding if one replaces $q^{\triangledown}$ with $q$ in the inequalities. The case for $\lambda_{2}\leq\lambda_{1}$ corresponds to the respective case for $\lambda_{1}\leq\lambda_{2}$ by interchanging $\lambda_{1}$ and $\lambda_{2}$.

Proof 5.11**.**

Essentially, this is an application of Theorem 4.7 and the prior discussion in Section 5.1. Since $\zeta^{\lambda_{1},\lambda_{2}}\in\ell^{\theta}(\mathbb{Z}^{3}\times\{\pm 1\})\quad\Longleftrightarrow\quad\psi^{(a,\beta)},\ \psi^{(a,\beta-k)}\in\ell^{\theta}(\mathbb{Z}^{3})$ with $a=\alpha+(1+\lambda_{1}+\lambda_{2})\gamma$. And the condition $\zeta^{\lambda_{1},\lambda_{2}}\in\ell^{q^{\triangledown}\left(\frac{r}{q^{\triangledown}}\right)^{\prime}}(\mathbb{Z}^{3})$ is sufficient for the existence of the embedding $\mathrm{Co}(\mathrm{L}^{p,r}_{v^{(\alpha,\beta)}}(\mathbb{R}^{3}\rtimes H^{\lambda_{1},\lambda_{2}}))\hookrightarrow W^{k,q}(\mathbb{R}^{3})$ for $q\in(2,\infty)$ and equivalent to it for $q\in(0,2]\cup\{\infty\}$.

We just have to set $\theta:=q^{\triangledown}\left(\frac{r}{q^{\triangledown}}\right)^{\prime}$ and notice that $q^{\triangledown}\left(\frac{r}{q^{\triangledown}}\right)^{\prime}=\infty\Longleftrightarrow r\leq q^{\triangledown}$ and that in the case $r>q^{\triangledown}$, we get $q^{\triangledown}\left(\frac{r}{q^{\triangledown}}\right)^{\prime}=q^{\triangledown}\frac{\frac{r}{q^{\triangledown}}}{\frac{r}{q^{\triangledown}}-1}=\frac{1}{\frac{1}{q^{\triangledown}}-\frac{1}{r}}.$ Hence $\theta^{-1}=\frac{1}{q^{\triangledown}}-\frac{1}{r}$, where all these computations also hold for $q=\infty$ or $r=\infty$ or $p=\infty$ with $\frac{1}{\infty}=0$. The application of Corollary 5.9 with these parameters then leads to the given conditions in the given cases.

5.4 Embeddings for Toeplitz shearlet groups

In this section, we consider the class of Toeplitz shearlet groups $H^{\delta}$ from section 2 and their associated coorbit spaces in dimension three. Again, according to Theorem 3.2 and Lemma 3.4, the map $\mathcal{F}:\mathrm{Co}(L^{p,r}_{v}(\mathbb{R}^{3}\rtimes H^{\delta}))\to\mathcal{D}(\mathcal{C}^{\delta},L^{p},\ell^{r}_{u})$ is an isomorphism for $p,r\in(0,\infty]$ and any weight $v:H^{\delta}\to(0,\infty)$ that is right moderate with respect to a locally bounded weight on $H^{\delta}$ if $u:I\to(0,\infty)$ is a $\mathcal{C}^{\delta}$-discretization of a decomposition weight for the coorbit space $\mathrm{Co}(L^{p,r}_{v}(\mathbb{R}^{3}\rtimes H^{\delta}))$. The associated weight has the following form $u_{n,m_{1},m_{2},\epsilon}:=v^{(r)}\left(A^{\delta}_{n,m_{1},m_{2},\epsilon}\right)=\left\lvert\det\left(\left(A^{\delta}_{n,m_{1},m_{2},\epsilon}\right)^{-T}\right)\right\rvert^{\frac{1}{2}-\frac{1}{r}}v\left(\left(A^{\delta}_{n,m_{1},m_{2},\epsilon}\right)^{-T}\right),$ where the matrices $A^{\delta}_{n,m_{1},m_{2},\epsilon}$ were defined in Lemma 3.4.

As in the standard shearlet case, we restrict attention to weights $v^{(\alpha,\beta)}:h\mapsto\lvert h_{1,1}\rvert^{\alpha}\left\lVert h^{-T}\right\rVert^{\beta},$ where $\alpha\in\mathbb{R}$ and $\beta\geq 0$. For this specific weight, the associated weight $u^{(\alpha,\beta)}$ for the decomposition space is given by

[TABLE]

according to Definition 3.1, where one has to keep in mind that the matrices $A^{\delta}_{n,m_{1},m_{2},\epsilon}$ are the inverse transposes of a well spread family.

The weight $w^{\{q\}}$ in Theorem 4.7 is in this setting given by

[TABLE]

An application of Theorem 4.7 boils down to the study of the sequence $\zeta^{\delta}$ defined by

[TABLE]

More precisely, we want to characterize $\zeta^{\delta}\in\ell^{\theta}(I)$ for $I=\mathbb{Z}^{3}\times\{\pm 1\}$ and $\theta\in(0,\infty]$. Since $\zeta_{n,m_{1},m_{2},1}^{\delta}=\zeta_{n,m_{1},m_{2},-1}^{\delta}\geq 0$, it is sufficient to characterize $(\zeta_{n,m_{1},m_{2},1}^{\delta})_{n,m_{1},m_{2}}\in\ell^{\theta}(\mathbb{Z}^{3})$. Furthermore, $\zeta_{n,m_{1},m_{2},1}^{\delta}=\psi_{n,m_{1},m_{2}}^{(a,\beta)}+\psi_{n,m_{1},m_{2}}^{(a,\beta-k)}$ with $\psi_{n,m_{1},m_{2}}^{(a,\beta)}:=2^{na}\left\lVert A^{\delta}_{n,m_{1},m_{2},1}\right\rVert^{-\beta}$ and $a:=\alpha+3(1-\delta)\left(\frac{1}{2}-\frac{1}{r}+\frac{1}{p}-\frac{1}{q}\right).$

Note that $\psi_{n,m_{1},m_{2}}^{(a,b)}\geq 0$, which implies the equivalence

[TABLE]

Hence, our task is again reduced to the investigation of the sequence $\psi^{(a,b)}$ for $a,b\in\mathbb{R}$, and finding a characterization in terms of $a,b,\theta,\delta$ for $\psi^{(a,b)}\in\ell^{\theta}(\mathbb{Z}^{3})$ with $a,\delta\in\mathbb{R},b\geq 0$ and $\theta\in(0,\infty]$. We consider again the norm $\lVert h\rVert=\sum_{1\leq i,j\leq 3}|h_{i,j}|.$ For the same reason as in the standard shearlet case, we restrict the weights $v^{\alpha,\beta}$ to the range of parameters $\alpha\in\mathbb{R}$ and $\beta\geq 0$.

The details of this calculation will not be presented here. Somewhat surprisingly, it turns out that no additional, extensive computations of the type presented in Subsection 5.2 have to be performed. The results for the Toeplitz case can be traced back to already established estimates for the standard shearlet case and the choice $\lambda_{1}:=1-\delta,\lambda_{2}:=1-2\delta$ or variations of this choice. Despite the different shearing behavior of these two classes of groups, the process of applying the embedding results is very similar. For details, we refer to Ref. \refciteRKDoktorarbeit.

Corollary 5.12**.**

The following are equivalent to $\psi^{(a,b)}\in\ell^{\theta}(\mathbb{Z}^{3})$ for $a\in\mathbb{R}$ and $b\geq 0$: if $\delta\geq 0$

[TABLE]

if $\delta<0$

[TABLE]

These conditions can again be used to get an embedding result.

Theorem 5.13**.**

Let $p,q,r\in(0,\infty]$, $\alpha\in\mathbb{R}$, $\beta\geq 0$, $k\in\mathbb{N}_{0}$ and set

[TABLE]

The embedding $\mathrm{Co}(\mathrm{L}^{p,r}_{v^{(\alpha,\beta)}}(\mathbb{R}^{3}\rtimes H^{\delta}))\hookrightarrow W^{k,q}(\mathbb{R}^{3})$ holds for $q\in(0,2]\cup\{\infty\},$ if and only if $p\leq q$ and

1. [i)]

for $r\leq q^{\triangledown}$

1. (a)

the inequalities

[TABLE]

hold if $\delta\geq 0$, 2. (b)

the inequalities

[TABLE]

hold if $\delta<0$, 3. 2.

for $r>q^{\triangledown}$ the inequality $\beta>k+2\left(\frac{1}{q^{\triangledown}}-\frac{1}{r}\right)$ and

1. (a)

*the inequalities *

[TABLE]

hold if $\delta\geq 0$, 2. (b)

the inequalities

[TABLE]

hold if $\delta<0$.

For the remaining case $q\in(2,\infty)$, the given conditions are sufficient for the embedding $\mathrm{Co}(\mathrm{L}^{p,r}_{v^{(\alpha,\beta)}}(\mathbb{R}^{3}\rtimes H^{\delta})\hookrightarrow W^{k,q}(\mathbb{R}^{3})$, and necessary for this embedding if one replaces $q^{\triangledown}$ with $q$ in the inequalities.

6 Interpretation of embedding results. Coorbit spaces as smoothness spaces

In this section, we give a sample application of the results attained in Subsection 5.1 and Section 5.4 and compare the embedding behavior of the spaces

[TABLE]

for $q\in(0,2]$. We choose $p=r$ in this section to illustrate some consequences of these embedding results in a particularly transparent case.

To make precise what we mean by the embedding behavior of different groups, we make the following definition.

Definition 6.1**.**

We say that two shearlet dilation groups $H_{1},H_{2}\subset\mathrm{GL}(\mathbb{R}^{3})$ have the same embedding behavior if for all $q\in(0,2],\ p\in(0,\infty],\ \alpha,\beta\in\mathbb{R}$ with $\beta\geq 0$ and $k\in\mathbb{N}_{0}$, the following equivalence holds

[TABLE]

If this does not hold, we say $H_{1},H_{2}$ have different embedding behavior.

Remark 6.2**.**

1. [i)]

We restrict this definition to $q\in(0,2]$ because the inequalities in Theorem 5.10 and Theorem 5.13 provide sufficient and necessary conditions for the existence of such an embedding. It is worth noting again that $p$ has no influence on the existence of such an embedding for $q$ in this range, except that the condition $p\leq q$ has to be fulfilled. 3. 2.

The relation of having the same embedding behavior is clearly an equivalence relation on the set of shearlet dilation groups in three dimensions.

As simple example for the kind of result we are interested in establishing in this section, we show that there does not hold an embedding into Sobolev spaces with at least one weak derivative for constant weights (i.e. $\alpha=\beta=0$).

Corollary 6.3**.**

Let $p,q\in(0,2]$, $k\in\mathbb{N}_{0}$ be arbitrary. Necessary for the embedding

[TABLE]

is $\beta\geq k$.

Proof 6.4**.**

This follows from the earlier observation regarding the sequence $\zeta^{\lambda_{1},\lambda_{2}}$ after the equivalence in (5).

The above Corollary has the consequence that in the unweighted case $v\equiv 1$, there are not any smooth functions in the space $\mathrm{Co}(L^{p}(\mathbb{R}^{3}\rtimes H^{\lambda_{1},\lambda_{2}}))$ if we restrict the search for smooth functions to $W^{k,q}(\mathbb{R}^{3})$ for $p,q\in(0,2]$ and $k\geq 1$. This is somewhat remarkable since smaller integrability exponents guarantee a sharper decay condition on the coefficients in the atomic decomposition of $f\in\mathrm{Co}(L^{p}(\mathbb{R}^{3}\rtimes H^{\lambda_{1},\lambda_{2}}))$ (see Ref. \refciteVoigtlaender2015PHD Theorem 2.4.19) and this leads in many instances to higher regularity properties of the functions in the space. However, for some parameters, the intuition that smaller $p$ leads to higher regularity is indeed true as we see in Remark 6.16.

6.1 Characterizing groups by their embedding behavior

In this subsection, we clarify when coorbit spaces associated to two different shearlet dilation groups have the same embedding behavior. This question is interesting for three reasons: It allows to determine which components of shearlet groups are truly relevant for embedding theorems; as it turns out, the shearing parts are not relevant (see the remark below). Secondly, these results contrast nicely to the fact, established in Ref. \refciteRKDoktorarbeit, that for any pair of distinct shearlet groups in dimension three, the induced scales of coorbit spaces do differ. Hence the embedding behavior into Sobolev spaces does not generally allow to distinguish between shearlet groups. Thirdly, understanding when two shearlet groups induce the same embedding behavior simplifies the subsequent further discussion, by allowing to concentrate on certain subcases.

We first observe that we can actually restrict the question of different embedding behaviors to the class of standard shearlet groups.

Corollary 6.5**.**

The groups $H^{\lambda_{1},\lambda_{2}}$ and $H^{\lambda_{2},\lambda_{1}}$ have the same embedding behavior and the groups $H^{\delta}$ and $H^{1-\delta,1-2\delta}$ have the same embedding behavior for $\lambda_{1},\lambda_{2},\delta\in\mathbb{R}$.

Proof 6.6**.**

This follows immediately by comparing the inequalities given in Theorem 5.10 and Theorem 5.13.

Remark 6.7**.**

1. [i)]

To phrase this differently, the embedding behavior of shearlet dilation groups in dimension three into Sobolev spaces in our setting is completely determined by the diagonal entries of the group elements. The different shearing parts of Toeplitz and standard shearlet groups have no effect on the existence of embeddings. 3. 2.

This also means that we can restrict our attention to the group $H^{\lambda_{1},\lambda_{2}}$ with $\lambda_{1}\leq\lambda_{2}$, when we discuss the embedding behaviour of shearlet coorbit spaces in more detail. The next result shows that within this smaller class, different groups have different embedding behaviors.

Theorem 6.8**.**

For $\lambda_{1},\lambda_{2},\lambda_{1}^{\prime},\lambda_{2}^{\prime}\in\mathbb{R}$, the groups $H^{\lambda_{1},\lambda_{2}}$ and $H^{\lambda_{1}^{\prime},\lambda_{2}^{\prime}}$ have the same embedding behavior if and only if $\{\lambda_{1},\lambda_{2}\}=\{\lambda_{1}^{\prime},\lambda_{2}^{\prime}\}$.

Proof 6.9**.**

We already know that the embedding behavior is the same if $\{\lambda_{1},\lambda_{2}\}=\{\lambda_{1}^{\prime},\lambda_{2}^{\prime}\}$, according to Corollary 6.5 and that it suffices to consider the case $\lambda_{1}\leq\lambda_{2}$ and $\lambda_{1}^{\prime}\leq\lambda_{2}^{\prime}$. Let $p\in(0,\infty]$, $q\in(0,2]$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $k\in\mathbb{N}_{0}$ and $\gamma=1/2-1/q$ in this proof. According to Theorem 5.10, the embedding

[TABLE]

holds if and only if $p\leq q$ and

[TABLE]

for $1\leq\lambda_{1}\leq\lambda_{2}$, denote this inequality with $I_{1}(\lambda_{1},\lambda_{2})$ and

[TABLE]

for $\lambda_{1}\leq\lambda_{2}\leq 1$, denote this inequality with $I_{2}(\lambda_{1},\lambda_{2})$ and

[TABLE]

for $\lambda_{1}\leq 1\leq\lambda_{2}$, denote this inequality with $I_{3}(\lambda_{1},\lambda_{2})$.

Assume that the groups $H^{\lambda_{1},\lambda_{2}}$ and $H^{\lambda_{1}^{\prime},\lambda_{2}^{\prime}}$ have the same embedding behavior and let always be $p\leq q$.

1. [i)]

If $1\leq\lambda_{1}\leq\lambda_{2}$ and $1\leq\lambda_{1}^{\prime}\leq\lambda_{2}^{\prime}$, this entails that the inequalities $I_{1}(\lambda_{1},\lambda_{2})$ and $I_{1}(\lambda_{1}^{\prime},\lambda_{2}^{\prime})$ are equivalent for all choices of parameters $p\in(0,\infty]$, $q\in(0,2]$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $k\in\mathbb{N}_{0}$. This implies the equalities $\beta-(1+\lambda_{1}+\lambda_{2})\gamma=\beta-(1+\lambda_{1}^{\prime}+\lambda_{2}^{\prime})\gamma$ and $(\beta-k)\lambda_{2}-(1+\lambda_{1}+\lambda_{2})\gamma=(\beta-k)\lambda_{2}^{\prime}-(1+\lambda_{1}^{\prime}+\lambda_{2}^{\prime})\gamma.$ for all $p\in(0,\infty]$, $q\in(0,2]$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $k\in\mathbb{N}_{0}$ (the lower and upper bounds of the inequalities for $\alpha$ have to coincide). The second equality implies for $q=2$ ($\gamma=0$) and $\beta-k=1\neq 0$ the equality $\lambda_{2}=\lambda_{2}^{\prime}$. Then the first equality implies for some $q\neq 2$ ($\gamma\neq 0$) the equality $\lambda_{1}=\lambda_{1}^{\prime}$. 3. 2.

If $1\leq\lambda_{1}\leq\lambda_{2}$ and $\lambda_{1}^{\prime}\leq\lambda_{2}^{\prime}\leq 1$, this entails that the inequalities $I_{1}(\lambda_{1},\lambda_{2})$ and $I_{2}(\lambda_{1}^{\prime},\lambda_{2}^{\prime})$ are equivalent for all choices of parameters $p\in(0,\infty]$, $q\in(0,2]\cup$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $k\in\mathbb{N}_{0}$. This implies the equalities $\beta-(1+\lambda_{1}+\lambda_{2})\gamma=\max\{\beta\lambda_{1}^{\prime},(\beta-k)\lambda_{1}^{\prime}\}-(1+\lambda_{1}^{\prime}+\lambda_{2}^{\prime})\gamma$and$(\beta-k)\lambda_{2}-(1+\lambda_{1}+\lambda_{2})\gamma=(\beta-k)-(1+\lambda_{1}^{\prime}+\lambda_{2}^{\prime})\gamma.$ for all $p\in(0,\infty]$, $q\in(0,2]$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $k\in\mathbb{N}_{0}$ (the lower and upper bounds of the inequalities for $\alpha$ have to coincide). The second equality implies for $q=2$ ($\gamma=0$) and $\beta-k=1\neq 0$ the equality $\lambda_{2}=1$. The first equality implies for $q=2$ ($\gamma=0$) and some $\beta>0$ first the inequality $\lambda_{1}^{\prime}\geq 0$ and then $\lambda_{1}^{\prime}=1$. But this implies already $\lambda_{1}=\lambda_{1}^{\prime}=\lambda_{2}=\lambda_{2}^{\prime}=1.$ 4. 3.

If $1\leq\lambda_{1}\leq\lambda_{2}$ and $\lambda_{1}^{\prime}\leq 1\leq\lambda_{2}^{\prime}$, this entails that the inequalities $I_{1}(\lambda_{1},\lambda_{2})$ and $I_{3}(\lambda_{1}^{\prime},\lambda_{2}^{\prime})$ are equivalent for all choices of parameters $p\in(0,\infty]$, $q\in(0,2]$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $k\in\mathbb{N}_{0}$. This implies the equalities $\beta-(1+\lambda_{1}+\lambda_{2})\gamma=\max\{\beta\lambda_{1}^{\prime},(\beta-k)\lambda_{1}^{\prime}\}-(1+\lambda_{1}^{\prime}+\lambda_{2}^{\prime})\gamma$ and $(\beta-k)\lambda_{2}-(1+\lambda_{1}+\lambda_{2})\gamma=(\beta-k)\lambda_{2}^{\prime}-(1+\lambda_{1}^{\prime}+\lambda_{2}^{\prime})\gamma.$ for all $p\in(0,\infty]$, $q\in(0,2]$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $k\in\mathbb{N}_{0}$ (the lower and upper bounds of the inequalities for $\alpha$ have to coincide). The second equality implies for $q=2$ ($\gamma=0$) and $\beta-k=1\neq 0$ the equality $\lambda_{2}=\lambda_{2}^{\prime}$. If we then cancel $(\beta-k)\lambda_{2}$ on both sides of the second equality and consider some $q\neq 2$ ($\gamma\neq 0$), we get $\lambda_{1}=\lambda_{1}^{\prime}$. 5. 4.

If $\lambda_{1}\leq\lambda_{2}\leq 1$ and $1\leq\lambda_{1}^{\prime}\leq\lambda_{2}^{\prime}$, we can argue as in case (2). 6. 5.

If $\lambda_{1}\leq\lambda_{2}\leq 1$ and $\lambda_{1}^{\prime}\leq\lambda_{2}^{\prime}\leq 1$, this entails that the inequalities $I_{2}(\lambda_{1},\lambda_{2})$ and $I_{2}(\lambda_{1}^{\prime},\lambda_{2}^{\prime})$ are equivalent for all choices of parameters $p\in(0,\infty]$, $q\in(0,2]$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $k\in\mathbb{N}_{0}$. This implies the equalities $\max\{\beta\lambda_{1},(\beta-k)\lambda_{1}\}-(1+\lambda_{1}+\lambda_{2})\gamma\\ =\max\{\beta\lambda_{1}^{\prime},(\beta-k)\lambda_{1}^{\prime}\}-(1+\lambda_{1}^{\prime}+\lambda_{2}^{\prime})\gamma$and $(\beta-k)-(1+\lambda_{1}+\lambda_{2})\gamma=(\beta-k)-(1+\lambda_{1}^{\prime}+\lambda_{2}^{\prime})\gamma.$ for all $p\in(0,\infty]$, $q\in(0,2]$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $k\in\mathbb{N}_{0}$ (the lower and upper bounds of the inequalities for $\alpha$ have to coincide). The first equality implies for $q=2$ ($\gamma=0$) and $\beta\neq 0$ with $\beta-k\neq 0$ first that $\lambda_{1}$ and $\lambda_{1}^{\prime}$ have the same sign and then that they are actually equal. The second equality then implies for $q\neq 2$ ($\gamma\neq 0$) the equality $\lambda_{2}=\lambda_{2}^{\prime}$. 7. 6.

If $\lambda_{1}\leq\lambda_{2}\leq 1$ and $\lambda_{1}^{\prime}\leq 1\leq\lambda_{2}^{\prime}$, this entails that the inequalities $I_{2}(\lambda_{1},\lambda_{2})$ and $I_{3}(\lambda_{1}^{\prime},\lambda_{2}^{\prime})$ are equivalent for all choices of parameters $p\in(0,\infty]$, $q\in(0,2]$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $k\in\mathbb{N}_{0}$. This implies the equalities $\max\{\beta\lambda_{1},(\beta-k)\lambda_{1}\}-(1+\lambda_{1}+\lambda_{2})\gamma\\ =\max\{\beta\lambda_{1}^{\prime},(\beta-k)\lambda_{1}^{\prime}\}-(1+\lambda_{1}^{\prime}+\lambda_{2}^{\prime})\gamma$and $(\beta-k)-(1+\lambda_{1}+\lambda_{2})\gamma=(\beta-k)\lambda_{2}^{\prime}-(1+\lambda_{1}^{\prime}+\lambda_{2}^{\prime})\gamma.$ for all $p\in(0,\infty]$, $q\in(0,2]$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $k\in\mathbb{N}_{0}$ (the lower and upper bounds of the inequalities for $\alpha$ have to coincide). The second equality implies for $q=2$ ($\gamma=0$) and $\beta-k=1\neq 0$ the equality $\lambda_{2}^{\prime}=1$. The first equality implies for $q=2$ ($\gamma=0$) and $\beta-k>0$ first that $\lambda_{1}^{\prime}$ and $\lambda_{1}$ have the same sign and in a next step that they are equal. The second equality lastly implies for some $q\neq 2$ ($\gamma\neq 0$) that $\lambda_{2}=\lambda_{2}^{\prime}$. 8. 7.

If $\lambda_{1}\leq 1\leq\lambda_{2}$ and $1\leq\lambda_{1}^{\prime}\leq\lambda_{2}^{\prime}$, we can argue as in case (3). 9. 8.

If $\lambda_{1}\leq 1\leq\lambda_{2}$ and $\lambda_{1}^{\prime}\leq\lambda_{2}^{\prime}\leq 1$, we can argue as in case (6). 10. 9.

If $\lambda_{1}\leq 1\leq\lambda_{2}$ and $\lambda_{1}^{\prime}\leq 1\leq\lambda_{2}^{\prime}$, this entails that the inequalities $I_{3}(\lambda_{1},\lambda_{2})$ and $I_{3}(\lambda_{1}^{\prime},\lambda_{2}^{\prime})$ are equivalent for all choices of parameters $p\in(0,\infty]$, $q\in(0,2]$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $k\in\mathbb{N}_{0}$. This implies the equalities $\max\{\beta\lambda_{1},(\beta-k)\lambda_{1}\}-(1+\lambda_{1}+\lambda_{2})\gamma=\max\{\beta\lambda_{1}^{\prime},(\beta-k)\lambda_{1}^{\prime}\}-(1+\lambda_{1}^{\prime}+\lambda_{2}^{\prime})\gamma$ and $(\beta-k)\lambda_{2}-(1+\lambda_{1}+\lambda_{2})\gamma=(\beta-k)\lambda_{2}^{\prime}-(1+\lambda_{1}^{\prime}+\lambda_{2}^{\prime})\gamma.$ for all $p\in(0,\infty]$, $q\in(0,2]$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $k\in\mathbb{N}_{0}$ (the lower and upper bounds of the inequalities for $\alpha$ have to coincide). The second equality implies for $q=2$ ($\gamma=0$) and $\beta-k=1\neq 0$ the equality $\lambda_{2}^{\prime}=\lambda_{2}$. The first equality implies for $q=2$ ($\gamma=0$) and $\beta-k>0$ first that $\lambda_{1}^{\prime}$ and $\lambda_{1}$ have the same sign and in a next step that they are equal.

For Toeplitz shearlet groups, this implies the following.

Corollary 6.10**.**

For $\delta,\delta^{\prime}\in\mathbb{R}$, the groups $H^{\delta}$ and $H^{\delta^{\prime}}$ have the same embedding behavior if and only if $\delta=\delta^{\prime}$.

Proof 6.11**.**

The group $H^{\delta}$ has the same embedding behavior as the group $H^{1-\delta,1-2\delta}$ and $H^{\delta^{\prime}}$ has the same embedding behavior as $H^{1-\delta^{\prime},1-2\delta^{\prime}}$, according to Corollary 6.5. Hence, Corollary 6.8 implies that $H^{\delta}$ and $H^{\delta^{\prime}}$ have the same embedding behavior if and only if $\{1-\delta,1-2\delta\}=\{1-\delta^{\prime},1-2\delta^{\prime}\}$. If $1-\delta=1-\delta^{\prime}$, then $\delta=\delta^{\prime}$ and if $1-\delta=1-2\delta^{\prime}$, then also $1-2\delta=1-\delta^{\prime}$, which implies $\delta=\delta^{\prime}=0$.

6.2 Influence of the group on the embedding behavior

In the last section of this chapter, we apply the results in Section 5.1 and Section 5.4 to investigate which groups allow embeddings into Sobolev spaces of higher smoothness and what the influence of the different parameters on the existence of such an embedding is. Corollary 6.5 allows us to focus on standard shearlet groups with $\lambda_{1}\leq\lambda_{2}$.

The next result is an immediate consequence of Theorem 5.10.

Corollary 6.12**.**

Let $p\in(0,\infty]$, $q\in(0,2]$, $k\in\mathbb{N}_{0}$, $\beta\geq 0$ with $p\leq q$ and $\lambda_{1}\leq\lambda_{2}$. Necessary and sufficient for the existence of $\alpha\in\mathbb{R}$ such that the embedding $\mathrm{Co}(\mathrm{L}^{p}_{v^{(\alpha,\beta)}}(\mathbb{R}^{3}\rtimes H^{\lambda_{1},\lambda_{2}}))\hookrightarrow W^{k,q}(\mathbb{R}^{3})$ holds is

1. [i)]

the inequality $\beta\leq(\beta-k)\lambda_{2}$ if $1\leq\lambda_{1}\leq\lambda_{2}$, 3. 2.

the inequality $\max\{\beta\lambda_{1},(\beta-k)\lambda_{1}\}\leq\beta-k$ if $\lambda_{1}\leq\lambda_{2}\leq 1$, 4. 3.

the inequality $\max\{\beta\lambda_{1},(\beta-k)\lambda_{1}\}\leq(\beta-k)\lambda_{2}$ if $\lambda_{1}\leq 1\leq\lambda_{2}$.

Proof 6.13**.**

If the embedding holds, then the respective inequality has to hold, according to Theorem 5.10. On the other hand, if the inequalities in $(1)$, $(2)$, or $(3)$ hold, then the choice $\alpha:=(\beta-k)\lambda_{2}-(1+\lambda_{1}+\lambda_{2})(1/2-1/q)$ ensures the embedding in case $(1)$ and $(3)$ and the choice $\alpha:=\beta-k-(1+\lambda_{1}+\lambda_{2})(1/2-1/q)$ ensures the embedding in case $(2)$, according to Theorem 5.10.

Remark 6.14**.**

1. [i)]

If $\beta=k=0$, then all these inequalities are satisfied, which means that for arbitrary parameters with $p\leq q$, there exists always one $\alpha$ such that the embedding $\mathrm{Co}(\mathrm{L}^{p}_{v^{(\alpha,0)}}(\mathbb{R}^{3}\rtimes H^{\lambda_{1},\lambda_{2}}))\hookrightarrow L^{q}(\mathbb{R}^{3})$ holds. 3. 2.

Since none of these inequalities hold for $\lambda_{1}=\lambda_{2}=1$ and $k>0$, we infer that coorbit spaces associated to shearlet groups with isotropic scaling matrices never allow an embedding into Sobolev spaces with nontrivial smoothness requirement ($k>0$). 4. 3.

If $\beta=k>0$, then no embedding exists for $\lambda_{1}>0$, but for every $\lambda_{1}\leq 0$. 5. 4.

If $\beta-k>0$, then we can always choose in case $(1)$ a sufficiently large $\lambda_{2}$ such that the inequality is fulfilled, in case (2) a sufficiently small $\lambda_{1}>0$ or $\lambda_{1}\leq 0$ such that the inequality is fulfilled and in case (3) a sufficiently large $\lambda_{2}$ or a sufficiently small $\lambda_{1}>0$ or $\lambda_{1}\leq 0$ such that the inequality holds. To phrase this more informally, it is easier to find smoother functions in the space $\mathrm{Co}(\mathrm{L}^{p}_{v^{(\alpha,\beta)}}(\mathbb{R}^{3}\rtimes H^{\lambda_{1},\lambda_{2}}))$ the more different the pair $(\lambda_{1},\lambda_{2})$ is from $(1,1)$.

Next, we consider the case $p=q$. An application of Theorem 5.10 for this choice of parameters leads to the next result.

Corollary 6.15**.**

Let $p\in(0,2]$, $k\in\mathbb{N}_{0}$, $\alpha\in\mathbb{R}$, $\beta\geq 0$ and $\lambda_{1}\leq\lambda_{2}$. Necessary and sufficient for the embedding $\mathrm{Co}(\mathrm{L}^{p}_{v^{(\alpha,\beta)}}(\mathbb{R}^{3}\rtimes H^{\lambda_{1},\lambda_{2}}))\hookrightarrow W^{k,p}(\mathbb{R}^{3})$ are

1. [i)]

the inequalities $\beta\leq\alpha+(1+\lambda_{1}+\lambda_{2})\left(\frac{1}{2}-\frac{1}{p}\right)\leq(\beta-k)\lambda_{2}$ if $1\leq\lambda_{1}\leq\lambda_{2}$, 3. 2.

the inequalities $\max\{\beta\lambda_{1},(\beta-k)\lambda_{1}\}\leq\alpha+(1+\lambda_{1}+\lambda_{2})\left(\frac{1}{2}-\frac{1}{p}\right)\leq\beta-k$ if $\lambda_{1}\leq\lambda_{2}\leq 1$, 4. 3.

the inequalities $\max\{\beta\lambda_{1},(\beta-k)\lambda_{1}\}\leq\alpha+(1+\lambda_{1}+\lambda_{2})\left(\frac{1}{2}-\frac{1}{p}\right)\leq(\beta-k)\lambda_{2}$ if $\lambda_{1}\leq 1\leq\lambda_{2}$.

Remark 6.16**.**

1. [i)]

In the first case, the right inequality leads to the condition

[TABLE]

for $k$. If we fix all parameters except $p$ and $k$, this shows that for smaller $p$ this upper bound for $k$ increases, which means that the functions in $\mathrm{Co}(L^{p}_{v^{(\alpha,\beta)}}(\mathbb{R}^{3}\rtimes H^{\lambda_{1},\lambda_{2}}))$ are smoother for smaller $p$. 3. 2.

In the second case with $\lambda_{1}>0$, the right inequality leads to the similar condition

[TABLE]

for $k$. If we fix all parameters except $p$ and $k$, this shows that for smaller $p$ this upper bound for $k$ increases, which means that in this case the functions in $\mathrm{Co}(L^{p}_{v^{(\alpha,\beta)}}(\mathbb{R}^{3}\rtimes H^{\lambda_{1},\lambda_{2}}))$ are smoother for smaller $p$ as well. 4. 3.

In the third case with $\lambda_{1}>0$, the right inequality leads to the same condition

[TABLE]

for $k$. If we fix all parameters except $p$ and $k$, this shows that for smaller $p$ this upper bound for $k$ increases, which means that in this case the functions in $\mathrm{Co}(L^{p}_{v^{(\alpha,\beta)}}(\mathbb{R}^{3}\rtimes H^{\lambda_{1},\lambda_{2}}))$ are smoother for smaller $p$ as well.

7 Conclusion

This paper provides a case study how the decomposition space approach can be systematically employed to study embedding properties of coorbit spaces. The complexity of the characterizations we obtained also highlight that even though the approach developed by Voigtlaender often reduces complex questions of containment between function spaces into a mere combinatorial problem, understanding the latter in more conceptual terms remains a challenge.