# Hybridization and embryological patterns underpinning reproductive barriers in Dactylorhiza (Orchidaceae)

## TL;DR

This study explores how hybridization and embryological patterns create reproductive barriers in the Dactylorhiza orchid genus.

## Contribution

The research provides new insights into postzygotic barriers and allopolyploid formation in Dactylorhiza.

## Key findings

- D. majalis showed significantly lower seed viability compared to diploids and D. maculata.
- Fluorescent in situ hybridization confirmed the allopolyploid origin of D. majalis.
- Maternal influence was observed in hybrid genotypes based on microsatellite marker analysis.

## Abstract

Pre- and postmating barriers are strongly associated with embryological processes occurring in developing flower buds and flowers. Although the embryology of Dactylorhiza representatives has been briefly described, comprehensive cytoembryological studies are scarce. In this study, the mechanisms of reproductive isolation within the Dactylorhiza incarnata/maculata complex were examined using cytological, embryological, and molecular approaches. Four taxa were analyzed: the diploid ancestors D. incarnata and D. fuchsii, the autotetraploid D. maculata, and the allotetraploid D. majalis. Fluorescent in situ hybridization revealed species-specific rDNA patterns, with D. majalis exhibiting rDNA patterns consistent with the hypothesis of its allopolyploid origin. Embryological assessments under controlled pollination conditions demonstrated variable gametophyte development and seed viability, with diploids and D. maculata showing 73–85% seed viability compared to 28% for D. majalis. Interspecific crosses between diploid progenitors yielded asymmetrical hybrids, with crosses between D. fuchsii × D. incarnata showing a significantly higher success rate. Cytological analysis revealed meiotic abnormalities, including chromosome segregation disruptions during megasporogenesis in D. incarnata and D. maculata, suggesting that cytogenetic mechanisms may reduce seed fertility and viability. Analysis of microsatellite markers of F1 hybrids revealed maternal influence on hybrid genotypes. All these findings suggest that postzygotic barriers dominate over prezygotic mechanisms in this complex, challenging previous assumptions about the direction of allopolyploids formation within the group. Our results shed new light on the reproductive mechanisms underlying the recurrent hybridization and polyploidization events that have shaped diversity in the Dactylorhiza genus.

The online version contains supplementary material available at 10.1038/s41598-025-24871-2.