Multi-Lingual Implicit Discourse Relation Recognition with Multi-Label Hierarchical Learning

TL;DR

This paper presents a multi-lingual, multi-label model for implicit discourse relation recognition that leverages hierarchical dependencies, outperforming large language models and setting new state-of-the-art results on multiple datasets.

Contribution

Introduces HArch, the first multi-lingual, multi-label hierarchical model for implicit discourse relation recognition, with superior performance over LLM prompting and previous benchmarks.

Findings

RoBERTa-HArch excels in English

XLM-RoBERTa-HArch performs best multi-lingually

Fine-tuned models outperform GPT-4o and Llama-4-Maverick

Abstract

This paper introduces the first multi-lingual and multi-label classification model for implicit discourse relation recognition (IDRR). Our model, HArch, is evaluated on the recently released DiscoGeM 2.0 corpus and leverages hierarchical dependencies between discourse senses to predict probability distributions across all three sense levels in the PDTB 3.0 framework. We compare several pre-trained encoder backbones and find that RoBERTa-HArch achieves the best performance in English, while XLM-RoBERTa-HArch performs best in the multi-lingual setting. In addition, we compare our fine-tuned models against GPT-4o and Llama-4-Maverick using few-shot prompting across all language configurations. Our results show that our fine-tuned models consistently outperform these LLMs, highlighting the advantages of task-specific fine-tuning over prompting in IDRR. Finally, we report SOTA results on the DiscoGeM 1.0 corpus, further validating the effectiveness of our hierarchical approach.

Tables

Table 1. Table 1: Distribution of level-1 senses per language in the DiscoGeM 2.0 corpus. Each value represents the sum of the corresponding sense in all the multi-label distributions of implicit discourse relations in the specific language.

Level-1	English	German	French	Czech	All
Temporal	$556.8$	$438.4$	$400.7$	$458.8$	$1,854.6$
Contingency	$1,695.4$	$772.3$	$811.0$	$681.1$	$3,959.8$
Comparison	$796.3$	$346.5$	$525.0$	$227.1$	$1,894.9$
Expansion	$2,798.6$	$1,030.8$	$891.3$	$632.9$	$5,353.7$
Total	$5,847.1$	$2,588.0$	$2,628.0$	$1,999.9$	$13,063$

Table 2. Table 2: Experimental results using HArch with different PLM encoders. The results are the average JS distance of three different runs on the test split of DiscoGeM 2.0. Results are reported using the English language and all languages simultaneously. Lower scores indicate better performance. Values in bold show the best score at each level in each language setting. Rows shaded in gray present the results of LLMs with few-shot prompting. The final column reports the approximate number of parameters of each model.

Language		Model	Level-1	Level-2	Level-3	Params
Test	Train
Eng	Eng	Costa and Kosseim (2025)	$0.329\pm 0.005$	$0.498\pm 0.004$	$0.569\pm 0.005$	125M
		RoBERTa-HArch	$0.327\pm 0.004$	$0.478\pm 0.005$	$0.541\pm 0.005$	125M
		ModernBERT-HArch	$0.364\pm 0.004$	$0.507\pm 0.005$	$0.592\pm 0.006$	149M
		XLM-RoBERTa-HArch	$0.343\pm 0.004$	$0.487\pm 0.006$	$0.564\pm 0.005$	125M
		GPT-4o	$0.373\pm 0.002$	$0.559\pm 0.003$	$0.657\pm 0.003$	200B
		Llama-4-Maverick	$0.423\pm 0.004$	$0.641\pm 0.006$	$0.711\pm 0.007$	400B
All	All	XLM-RoBERTa-HArch	$0.356\pm 0.005$	$0.531\pm 0.006$	$0.605\pm 0.006$	125M
		EuroBERT-HArch	$0.414\pm 0.006$	$0.569\pm 0.007$	$0.641\pm 0.006$	210M
		Flan-T5-HArch	$0.364\pm 0.003$	$0.550\pm 0.003$	$0.623\pm 0.004$	220M
		GPT-4o	$0.438\pm 0.002$	$0.671\pm 0.002$	$0.709\pm 0.003$	200B
		Llama-4-Maverick	$0.534\pm 0.004$	$0.774\pm 0.004$	$0.839\pm 0.005$	400B

Table 3. Table 3: Final results showing the average JS distance of three different runs for each model on the test split of DiscoGeM 1.0. Lower scores indicate better performance. Values in bold show the best score at each level.

Model	Level-1	Level-2	Level-3
Costa and Kosseim (2025)	$0.299\pm 0.002$	$0.446\pm 0.003$	$0.523\pm 0.002$
RoBERTa-HArch	$0.302\pm 0.003$	$0.435\pm 0.003$	$0.506\pm 0.004$

Table 4. Table 4: Final results showing the average JS distance of three different runs for each model on the test split of DiscoGeM 2.0. Results are reported using all languages simultaneously and each language individually. Lower scores indicate better performance. Values in bold show the best score at each level in each tested language setting. Rows shaded in gray present the results of LLMs with few-shot prompting.

Language		Model	Level-1	Level-2	Level-3
Test	Train
All	All	XLM-RoBERTa-HArch	$0.353\pm 0.005$	$0.529\pm 0.005$	$0.606\pm 0.04$
	$-$	GPT-4o	$0.434\pm 0.002$	$0.678\pm 0.003$	$0.705\pm 0.003$
	$-$	Llama-4-Maverick	$0.529\pm 0.004$	$0.769\pm 0.005$	$0.830\pm 0.005$
Eng	Eng	RoBERTa-HArch	$0.331\pm 0.004$	$0.477\pm 0.004$	$0.548\pm 0.005$
	All	XLM-RoBERTa-HArch	$0.349\pm 0.004$	$0.493\pm 0.006$	$0.568\pm 0.005$
	Eng		$0.347\pm 0.005$	$0.486\pm 0.006$	$0.561\pm 0.006$
	$-$	GPT-4o	$0.368\pm 0.001$	$0.551\pm 0.001$	$0.647\pm 0.002$
	$-$	Llama-4-Maverick	$0.420\pm 0.001$	$0.634\pm 0.002$	$0.702\pm 0.002$
Ger	All	XLM-RoBERTa-HArch	$0.360\pm 0.007$	$0.566\pm 0.006$	$0.630\pm 0.006$
	Ger		$0.393\pm 0.006$	$0.586\pm 0.006$	$0.656\pm 0.007$
	$-$	GPT-4o	$0.411\pm 0.002$	$0.644\pm 0.002$	$0.725\pm 0.002$
	$-$	Llama-4-Maverick	$0.509\pm 0.003$	$0.732\pm 0.006$	$0.809\pm 0.007$
Fre	All	XLM-RoBERTa-HArch	$0.367\pm 0.006$	$0.553\pm 0.005$	$0.628\pm 0.006$
	Fre		$0.382\pm 0.004$	$0.576\pm 0.006$	$0.655\pm 0.007$
	$-$	GPT-4o	$0.406\pm 0.001$	$0.611\pm 0.001$	$0.689\pm 0.002$
	$-$	Llama-4-Maverick	$0.473\pm 0.003$	$0.667\pm 0.003$	$0.748\pm 0.004$
Cze	All	XLM-RoBERTa-HArch	$0.371\pm 0.010$	$0.555\pm 0.009$	$0.647\pm 0.009$
	Cze		$0.404\pm 0.009$	$0.584\pm 0.011$	$0.689\pm 0.010$
	$-$	GPT-4o	$0.549\pm 0.011$	$0.820\pm 0.010$	$0.854\pm 0.012$
	$-$	Llama-4-Maverick	$0.613\pm 0.005$	$0.864\pm 0.004$	$0.890\pm 0.005$

Table 5. Table 5: Results showing the contribution of multi-task learning. The results are the average JS distance of three different runs on the test split of DiscoGeM 2.0. Results are reported using the English language and all languages simultaneously. Lower scores indicate better performance. Values in bold show the best score at each level in each language setting. The final column reports the total time required to compute all of the averaged results in each row.

Language		Model	Level-1	Level-2	Level-3	Time
Test	Train
Eng	Eng	RoBERTa-HArch	$0.331\pm 0.004$	$0.477\pm 0.004$	$0.548\pm 0.005$	9h 09m
		RoBERTa-Individual	$0.342\pm 0.004$	$0.513\pm 0.004$	$0.589\pm 0.005$	21h 56m
All	All	XLM-RoBERTa-HArch	$0.353\pm 0.005$	$0.529\pm 0.005$	$0.606\pm 0.04$	20h 31m
		XLM-RoBERTa-Individual	$0.365\pm 0.004$	$0.552\pm 0.005$	$0.641\pm 0.004$	59h 14m

Table 6. Table 6: Distribution of level-2 and level-3 senses per language in the DiscoGeM 2.0 corpus. Each value represents the sum of the corresponding sense in all the multi-label distributions of implicit discourse relations in the specific language subset. Senses at level-3 marked with * represent level-2 senses that were projected to level-3 since they do not exist in the PDTB 3.0.

Level-2	English	German	French	Czech	All	Level-3	English	German	French	Czech	All
Synchronous	$99.4$	$191.2$	$139.8$	$254.6$	$685.0$	Synchronous*	$99.4$	$191.2$	$139.8$	$254.6$	$685.0$
Asynchronous	$457.4$	$247.1$	$260.9$	$204.2$	$1169.6$	Precedence	$420.7$	$222.7$	$240.0$	$193.7$	$1,077.1$
						Succession	$36.7$	$24.4$	$20.9$	$10.5$	$92.5$
Cause	$1,690.7$	$457.5$	$608.6$	$576.7$	$3,333.5$	Reason	$386.9$	$243.9$	$237.6$	$281.3$	$1,149.8$
						Result	$1,303.8$	$213.6$	$371.0$	$295.4$	$2,183.7$
Condition	$1.2$	$126.1$	$93.2$	$42.0$	$262.5$	ARG1-as-Cond	$0.1$	$109.4$	$87.9$	$25.1$	$222.4$
						ARG2-as-Cond	$1.1$	$16.7$	$5.4$	$16.9$	$40.1$
Neg-Condition	$1.1$	$14.0$	$14.0$	$12.9$	$42.0$	ARG1-as-NegCond	$1.0$	$7.8$	$10.9$	$10.9$	$30.6$
						ARG2-as-NegCond	$0.1$	$6.2$	$3.1$	$2.0$	$11.5$
Purpose	$2.3$	$174.8$	$95.2$	$49.5$	$321.8$	ARG1-as-Goal	$1.4$	$114.8$	$91.6$	$46.5$	$254.2$
						ARG2-as-Goal	$0.9$	$60.0$	$3.6$	$3.1$	$67.6$
Concession	$505.1$	$161.3$	$261.3$	$160.9$	$1088.6$	ARG1-as-Denier	$167.5$	$61.9$	$91.3$	$39.9$	$360.5$
						ARG2-as-Denier	$337.7$	$99.4$	$169.9$	$121.0$	$728.1$
Contrast	$187.4$	$98.7$	$173.3$	$40.8$	$500.2$	Contrast*	$187.4$	$98.7$	$173.3$	$40.8$	$500.2$
Similarity	$103.7$	$86.5$	$90.4$	$25.5$	$306.1$	Similarity*	$103.7$	$86.5$	$90.4$	$25.5$	$306.1$
Conjunction	$1,298.8$	$260.0$	$121.8$	$155.9$	$1,836.5$	Conjunction*	$1,298.8$	$260.0$	$121.8$	$155.9$	$1,836.5$
Disjunction	$2.7$	$6.5$	$3.0$	$3.1$	$15.3$	Disjunction*	$2.7$	$6.5$	$3.0$	$3.1$	$15.3$
Equivalence	$19.0$	$154.7$	$101.5$	$57.4$	$332.6$	Equivalence*	$19.0$	$154.7$	$101.5$	$57.4$	$332.6$
Exception	$1.8$	$26.8$	$41.0$	$6.8$	$76.4$	ARG1-as-Exception	$0.3$	$14.7$	$30.7$	$2.8$	$48.5$
						ARG2-as-Exception	$1.4$	$12.2$	$10.3$	$4.0$	$27.8$
Instantiation	$352.4$	$99.3$	$124.2$	$84.4$	$660.3$	ARG1-as-Instance	$16.5$	$0.0$	$35.6$	$25.9$	$78.0$
						ARG2-as-Instance	$335.9$	$99.3$	$88.6$	$58.5$	$582.3$
Level-of-Detail	$1079.1$	$393.2$	$363.1$	$278.8$	$2114.1$	ARG1-as-Detail	$154.9$	$88.7$	$51.8$	$110.1$	$405.6$
						ARG2-as-Detail	$924.1$	$304.5$	$311.3$	$168.6$	$1708.5$
Manner	$4.5$	$33.7$	$124.0$	$37.2$	$199.4$	ARG1-as-Manner	$1.3$	$26.6$	$112.3$	$22.7$	$163.0$
						ARG2-as-Manner	$3.1$	$7.0$	$11.7$	$14.6$	$36.5$
Substitution	$40.5$	$56.6$	$12.7$	$9.3$	$119.1$	ARG1-as-Substitution	$0.0$	$7.6$	$6.0$	$1.3$	$14.9$
						ARG2-as-Substitution	$40.5$	$49.0$	$6.7$	$8.0$	$104.1$

Equations

$${\mathbf{W}}_{\alpha }=\alpha \cdot {\mathbf{h}}_{{\text{Aug}}_1}+(1-\alpha )\cdot {\mathbf{h}}_{\text{Dropout}}$$

$$\begin{array}{rl}{\mathbf{W}}_{{\beta }_1,{\beta }_2}=& {\beta }_1\cdot {\mathbf{h}}_{{\text{Aug}}_1}+{\beta }_2\cdot {\mathbf{h}}_{{\text{Aug}}_2}+\\ & (1-{\beta }_1-{\beta }_2)\cdot {\mathbf{h}}_{\text{Dropout}}\end{array}$$

Full text

**Multi-Lingual Implicit Discourse Relation Recognition

with Multi-Label Hierarchical Learning**

Nelson Filipe Costa  and Leila Kosseim

Computational Linguistics at Concordia (CLaC) Laboratory

Department of Computer Science and Software Engineering

Concordia University, Montréal, Québec, Canada

nelsonfilipe.costa@mail.concordia.ca

leila.kosseim@concordia.ca

1 Introduction

Discourse analysis explores how textual segments are connected through relations of coherence. One of the most widely adopted frameworks for computational discourse analysis is the Penn Discourse Treebank (PDTB) (Miltsakaki et al., 2004; Prasad et al., 2008a). In the PDTB framework, a discourse relation is established between two textual segments (referred to as arguments) either explicitly, through the presence of a discourse connective (e.g., but and because), or implicitly when no connective is present. The sense of the relation is then classified using a defined list of discourse senses, which are organized hierarchically across three levels (Webber et al., 2019). The task of identifying the sense of an implicit relation is known as implicit discourse relation recognition (IDRR).

Research on IDRR has mostly been conducted on the English annotated PDTB corpora (Prasad et al., 2008b, 2019). However, the subjective nature of discourse interpretation poses challenges for single-label annotation schemes, where each instance is assigned only one sense label  (Stede, 2008; Scholman and Demberg, 2017; Hoek et al., 2021). The ambiguity in the annotation of implicit relations has been further highlighted by the challenges in mapping them across discourse frameworks (Demberg et al., 2019; Costa et al., 2023). As a response, recent works have advocated for multi-label annotation, recognizing that multiple discourse senses can often co-occur within a single relation (Yung et al., 2019; Pyatkin et al., 2020; Scholman et al., 2022a, b; Pyatkin et al., 2023; Yung et al., 2024b; Yung and Demberg, 2025). This shift in perspective has led to the development of the multi-label annotated DiscoGeM 1.0 corpus (Scholman et al., 2022a) for implicit discourse relations.

This perspective shift toward multi-label annotation has stirred an initial wave of research in multi-label IDRR (Costa and Kosseim, 2024; Long et al., 2024; Costa and Kosseim, 2025). However, as with the PDTB corpora, the DiscoGeM 1.0 corpus is limited to the English language and most work in the field of discourse continues to focus mostly on the English language. While single-label discourse annotated corpora following the PDTB framework exist in other languages, such as the German Potsdam Commentary Corpus 2.2 (Bourgonje and Stede, 2020) and the Czech Prague Discourse Treebank 3.0 (Synková et al., 2024), multi-lingual corpora remain scarce. For instance, the TED-MDB corpus (Zeyrek et al., 2024) provides parallel annotations of single-label discourse relations across seven languages. However, its size remains relatively small for the training of IDRR models - it contains $1,774$ implicit relations. To address this gap and enable cross-linguistic research in IDRR, the recently released DiscoGeM 2.0 corpus (Yung et al., 2024b) introduces parallel multi-label annotations of implicit discourse relations following the PDTB 3.0 sense hierarchy in four languages: English, German, French and Czech.

In this paper, we leverage the DiscoGeM 2.0 corpus to present the first multi-lingual and multi-label classification model for IDRR. Our main contributions are as follows:

• •

We propose a hierarchical multi-task architecture, HArch, that leverages dependencies across all three sense levels in the PDTB 3.0 framework, improving on prior multi-label approaches (Costa and Kosseim, 2025).

• •

We conduct the first evaluation of multi-label IDRR on the DiscoGeM 2.0 corpus in the English, German, French and Czech languages individually and in a multi-lingual setting.

• •

We compare different pre-trained encoder backbones and find that RoBERTa-HArch and XLM-RoBERTa-HArch achieve the best results in the English-only and multi-lingual settings, respectively.

• •

We evaluate our fine-tuned models against LLMs, including GPT-4o and Llama-4-Maverick, using few-shot prompting and show that our models consistently outperform them in multi-label IDRR.

• •

We benchmark our best-performing model on the English-only DiscoGeM 1.0 corpus and achieve new SOTA results.

2 Previous Work

The advent of transformer-based models (Vaswani et al., 2017) has led to substantial progress in natural language understanding. Nevertheless, IDRR remains one of the most challenging tasks in computational discourse analysis. Traditionally formulated as a single-label classification problem, most approaches have addressed IDRR in English either by fine-tuning (Long and Webber, 2022; Liu and Strube, 2023) or prompt-tuning (Chan et al., 2023; Zhao et al., 2023; Zeng et al., 2024; Long and Webber, 2024) pre-trained language models (PLMs). Another line of research has also explored directly prompting large language models (LLMs) through prompt-engineering to solve single-label IDRR (Chan et al., 2024; Yung et al., 2024a). However, results from both work indicate that zero-shot and few-shot prompting of LLMs continues to significantly underperform when compared to the results obtained through fine-tuning and prompt-tuning PLMs. The same outcome has been observed in other IDRR work focusing on different languages (Saeed et al., 2025; Ruby et al., 2025).

The recent move toward multi-label annotation in discourse data has led to initial efforts in multi-label IDRR. Long et al. (2024) used the 4.9% of implicit discourse relations in the PDTB 3.0 corpus that are annotated with two senses to build a model capable of predicting up to two senses per instance. However, given that most of PDTB 3.0 annotations remain single-label, their model predominantly produces single-label predictions. In contrast, Yung et al. (2022) and Costa and Kosseim (2024) used the multi-label DiscoGeM 1.0 corpus but convert its annotations into a single-label format during training. Finally, Costa and Kosseim (2025) present the first results in multi-label IDRR by training a model on DiscoGeM 1.0 that jointly predicts sense distributions across all three levels in the PDTB 3.0 framework. Rather than adapting multi-label data to a single-label setting or modifying a single-label corpus for multi-label use, they preserve the inherent multi-label structure of DiscoGeM 1.0 throughout training. However, like most previous work in IDRR, their model is limited to the English language.

3 Our Approach

We propose a hierarchical multi-task classification model, HArch, to predict probability distributions across all three sense levels in the PDTB 3.0 framework. We also propose a prompting framework to compare our HArch model against SOTA LLMs through few-shot learning.

3.1 HArch Model

Our HArch model improves on the model of Costa and Kosseim (2025) by explicitly modeling hierarchical dependencies across sense levels. Figure 1 shows the architecture of our model. The input of the model is a concatenated pair of discourse arguments (ARG1 + ARG2), which is encoded using a pre-trained language model. The resulting contextual representation is fed to a shared linear transformation followed by a dropout layer. This shared representation is then passed to three separate classification heads - one for each sense level. The level-1 classification head predicts a probability distribution over the 4 possible level-1 senses using a linear layer followed by a softmax activation - as shown in red on the bottom-right side of Figure 1. This 4-dimensional output vector is then projected onto the same embedding space of the pre-trained language model through an augmentation block. As shown in blue on the upper-right side of Figure 1, this augmentation is done in two steps: the output of the classification head is first mapped to a hidden dimension of half the size of the embedding space and then, in a second step, further projected to match the full embedding size. A GELU (Hendrycks and Gimpel, 2016) activation function after each incremental step introduces non-linearity, while helping with sparsity and preventing exploding activations in the dimensionality projection. Based on early experiments, this incremental dimensional projection ensured a smoother transition in feature space. The augmented level-1 output is then combined with the shared representation through a weighted sum defined by Equation 1, where $\alpha$ is a learnable parameter of the model, to yield the input of the level-2 classification head.

[TABLE]

The level-2 classification head then predicts a probability distribution over the 17 possible level-2 senses. Similar to level-1, the level-2 output is passed through a dimensionality augmentation block to project it into the same embedding space of the pre-trained language model. The augmented level-1 and level-2 outputs are then added to the shared representation through a second weighted sum defined by Equation 2, where $\beta_{1}$ and $\beta_{2}$ are learnable parameters. The output of this weighted sum is then fed into the level-3 classification head, which predicts a probability distribution over the level-3 senses.

[TABLE]

This cascading of information ensures that the predictions of lower-level classification heads are used to help inform the prediction of higher-level classification heads that require more fine-grained information. Finally, we use the Adam optimization algorithm Kingma and Ba (2015) to minimize the loss of the model, which we calculate as the sum of the losses of each classification head. We use the Mean Absolute Error function to calculate the loss of each classification head as it was shown to lead to better performance in in multi-label classification (Costa and Kosseim, 2025). To evaluate our model, we use Jensen-Shannon (JS) distance to measure the similarity between the predicted and the reference probability distributions, similarly to other multi-label classification works in NLP (Pyatkin et al., 2023; Yung et al., 2024b; van der Meer et al., 2024; Costa and Kosseim, 2025).

3.2 Few-Shot Prompting

Despite prior work showing poor results in IDRR through prompt engineering (Chan et al., 2024; Yung et al., 2024a), we compared our HArch model to LLMs in multi-label IDRR via direct prompting with few-shot learning. We tested GPT-4o (OpenAI, 2024) and Llama-4-Maverick with the prompt structure described in Appendix B. Depending on the language evaluated, the prompt was translated and examples were drawn from the corresponding language subset of DiscoGeM 2.0. When considering all languages simultaneously in a multi-lingual setting, we kept the structure of the prompt in English and provided examples in the different languages. All of the examples were taken from the training split of the corpus. To ensure consistency, we set the temperature to [math] and allowed up to $5$ retries per instance in cases where prompt outputs did not conform to the expected format. Prompting costs were approximately 5.00\$$ for GPT-4o and 0.19-0.49$$ for Llama-4-Maverick per million input tokens.

The prompt was designed to closely replicate the annotation methodology of DiscoGeM 2.0. When crowdsourcing the annotation of the corpus, the authors provided the annotators a list of non-ambiguous connectives (Yung et al., 2024b) that represented each of the possible senses in the PDTB 3.0 framework. This list of connectives was adapted for each language. The annotators were then asked to choose the connective that best expresses the semantic relation between the arguments of the relation being annotated, irrespective of whether syntax would need to be adjusted. By mirroring this annotation process in our prompting template, we were able to directly evaluate the performance of both LLMs on multi-label IDRR using the test split of DiscoGeM 2.0.

4 Data Preparation

To train our model on multi-lingual IDRR, we use the recently released DiscoGeM 2.0 corpus (Yung et al., 2024b). The corpus comprises a total of 13,063111In their paper they report a total of 12,834 implicit discoutse relations, out of which 5,618 are in English. However, we counted 5,847 discourse relations in English in the corpus. parallel annotated implicit discourse relations across four languages: 5,847 in English, 2,588 in German, 2,628 in French, and 2,000 in Czech. All annotations follow the PDTB 3.0 discourse framework (Webber et al., 2019). While its predecessor, DiscoGeM 1.0 Scholman et al. (2022a), provided the first corpus of multi-label implicit discourse relations, DiscoGeM 2.0 is the first corpus to provide multi-lingual parallel annotation of multi-label implicit discourse relations.

Each implicit discourse relation in the corpus was annotated by at least 10 crowdworkers. Each annotator selected the sense they considered most appropriate for each relation through a proxy task and then all of the annotated senses of each relation were averaged to produce a multi-label sense distribution over 28 different senses in the PDTB 3.0 framework - the Belief and the Speech-Act senses were not annotated. For each language, the authors of the corpus prepared a list of discourse connectives (one representing each of the available senses) and asked the annotators to select for each relation the discourse connective that best expresses the semantic relation between its two arguments. From the chosen connectives, the authors were able to infer the possible multiple senses of each implicit discourse relation.

To ensure reproducibility of our results, we split the DiscoGeM 2.0 corpus following the proposed training, validation and test splits (Yung et al., 2024b). Contrary to the common approach in traditional single-label IDRR, we follow the methodology of Costa and Kosseim (2025) and consider all three sense levels in the PDTB 3.0 framework when possible. Table 1 shows the distribution of level-1 senses and Table 6 in Appendix A shows the distribution of level-2 and level-3 senses per language in the DiscoGeM 2.0 corpus. We replaced the non-existent PDTB 3.0 level-3 senses with their corresponding level-2 sense.

5 Encoder Selection

In this section, we compare the performance of different PLMs as encoders in our HArch model (see Figure 1) for multi-label IDRR in both multi-lingual and English-only settings, using the DiscoGeM 2.0 corpus. Table 2 summarizes the performance of each configuration on the validation split of the corpus. Each model was trained for 10 epochs with a batch size of 16 and the results are reported as the average Jensen-Shannon (JS) distance across three independent runs. In addition to experimenting with different PLM encoders, we also report results obtained using LLMs via few-shot prompting. All of the code and prompt templates are available on GitHub222https://github.com/nelsonfilipecosta/Multi-Lingual-Implicit-Discourse-Relation-Recognition.

To evaluate the impact of incorporating hierarchical learning in multi-label IDRR, we first compare our proposed HArch model against the non-hierarchical model proposed by Costa and Kosseim (2025). For a fair comparison, both models use the same pre-trained RoBERTa${}_{\text{base}}$ encoder (Liu et al., 2019) and are trained under identical conditions with a learning rate of $1e^{-5}$. To our knowledge, Costa and Kosseim (2025) is the only existing approach modeling multi-label IDRR with probability distributions over the full set of PDTB 3.0 sense labels. Since their model was only trained in the English language, we compared the performance of our models using only the English subset of DiscoGeM 2.0. As shown in Table 2, our model RoBERTa-HArch, which incorporates hierarchical dependencies across sense levels, achieves lower JS distances at level-2 and level-3 compared to the non-hierarchical model of Costa and Kosseim (2025). At level-1, the performance of both models is comparable, which is to be expected as level-1 predictions do not benefit from lower-level inputs within the hierarchical structure.

Next, we explored the impact of using different PLMs as encoder backbones for our HArch hierarchical model. In particular, we experimented with ModernBERT${}_{\text{base}}$ (Warner et al., 2024), a recently introduced encoder claimed to outperform RoBERTa, and the multi-lingual XLM-RoBERTa${}_{\text{base}}$ (Conneau et al., 2020). Contrary to expectations, ModernBERT-HArch consistently underperformed across all three sense levels, as shown in Table 2. Even XLM-RoBERTa-HArch, despite its multi-lingual design, achieved lower JS distances than ModernBERT-HArch - though, it still fell short of the performance achieved by RoBERTa-HArch. In addition to fine-tuned models, we also evaluated LLMs via few-shot prompting. As shown in Table 2, GPT-4o substantially outperforms Llama-4-Maverick in terms of JS distance across all levels, replicating a pattern already observed between GPT-3.5 and LLaMa (Touvron et al., 2023) in a similar setting (Yung et al., 2024a). However, despite its stronger performance in a few-shot setting, GPT-4o still does not surpass our smaller fine-tuned RoBERTa-HArch model, which remains the best-performing system overall in this multi-label IDRR task.

In the multi-lingual setting, we evaluated PLMs pre-trained on multiple languages as encoder backbones for HArch. Specifically, we experimented with XLM-RoBERTa${}_{\text{base}}$, FLAN-T5${}_{\text{base}}$ (Chung et al., 2024) and the recently released EuroBERT${}_{\text{210m}}$ (Boizard et al., 2025). We also adapted the prompt template described in Appendix B to include examples in all four languages covered by DiscoGeM 2.0. Given the relatively limited number of annotated instances in German, French, and Czech within the DiscoGeM 2.0 corpus (see Table 1), we did not experiment with language specific PLMs for these languages. Instead, we infer model performance in these languages from the results in the multi-lingual setting. As shown in Table 2, using XLM-RoBERTa as the encoder backbone of HArch achieves the best overall performance in the multi-lingual configuration, even outperforming the more recent EuroBERT encoder model. Finally, we observe a surprising drop in performance in the LLM models when prompted in the multi-lingual setting compared to when prompted in the English language alone.

6 Results

In this section, we report the performance of our HArch models on the test splits of both DiscoGeM 1.0 (see Table 3) and DiscoGeM 2.0 (see Table 4) in a multi-lingual setting, where all languages are used simultaneously, and separately for each language. These evaluations allow us to assess the generalizability of our models across the different languages of the corpus.

We report results on DiscoGeM 1.0 to benchmark our approach with the only other existing model for multi-label IDRR that predicts probability distributions over the full set of PDTB 3.0 sense labels (Costa and Kosseim, 2025). Since DiscoGeM 1.0 contains only English data, we report the performance of our best-performing English model in Table 2 - RoBERTa-HArch. As shown in Table 3, RoBERTa-HArch achieves lower JS distances at level-2 and level-3 compared to the non-hierarchical model of Costa and Kosseim (2025), while maintaining comparable performance at level-1. These results provide further empirical support for our hypothesis that incorporating hierarchical dependencies improves the ability of the model to generate more accurate probability distributions at finer-grained sense levels.

Table 4 presents the results of our models across different language configurations. Since XLM-RoBERTa-HArch achieved the best overall performance in the multi-lingual setting with the validation split of DiscoGeM 2.0 (see Table 2), we report its results on the test split of the corpus when trained both on all languages and individually on each language. For English, we also include results from RoBERTa-HArch, which was fine-tuned exclusively on English data alone. In addition, we evaluate GPT-4o and Llama-4-Maverick using language-specific versions of the prompt detailed in Appendix B. As shown in Table 4, our HArch models consistently outperform both LLMs across all languages and all sense levels. The performance gap is particularly pronounced in all settings with the exception of the English language. Notably, XLM-RoBERTa-HArch achieves substantially lower JS distances than both LLMs at level-2 and level-3 in the multi-lingual and Czech language. Consistent with earlier results in Table 2, GPT-4o outperforms Llama-4-Maverick across all configurations, but still lags behind our smaller fine-tuned HArch models. These findings reinforce the idea that fine-tuning smaller encoder-based models can still outperform prompting large language models in specific tasks (Bucher and Martini, 2024; Chan et al., 2024; Yung et al., 2024a).

Focusing on our fine-tuned models, Table 4 shows that for German, French and Czech, XLM-RoBERTa-HArch achieves lower JS distances when trained on all languages compared to when trained on the test language alone. This suggests that multi-lingual training helps compensate for the limited annotated data available in these languages. In contrast, for English, XLM-RoBERTa-HArch performs slightly better when fine-tuned exclusively on the English subset of DiscoGeM 2.0. However, in both cases, it fails to outperform RoBERTa-HArch - which remains the strongest model for English overall. When comparing the performance of RoBERTa-HArch on DiscoGeM 1.0 in Table 3 and DiscoGeM 2.0 in Table 4, we observe a drop across all sense levels on the latter. This discrepancy can be attributed to the reduced label set used for evaluation on DiscoGeM 1.0. To allow a fair comparison with Costa and Kosseim (2025), we adopted the same adapted set of 14 level-2 sense labels proposed by Kim et al. (2020). This reduced set of level-2, and consequently level-3, sense labels lowers the complexity of the multi-label IDRR task on DiscoGeM 1.0 when compared to DiscoGeM 2.0, where we considered all sense labels on the corpus.

7 Multi-Task Contribution

To evaluate the benefit of jointly modeling all three sense levels in a multi-task setting, we conducted an ablation study by training three individual models - each dedicated to predicting probability distributions for a single sense level. This study was conducted in the English and multi-lingual setting. To ensure a fair comparison, each individual model tries to replicate the architecture of the multi-task model (see Figure 1), with the exception that it includes only one classification head and no augmentation blocks. Specifically, each individual model encodes a concatenated pair of discourse arguments using the corresponding PLM, which it then passes to a linear transformation and dropout layer, before feeding the representation to a classification head — similar to the level-1 prediction path in the multi-task model. Table 5 compares the results of the three individual models for each sense level against RoBERTa-HArch in the English setting and XLM-RoBERTa-HArch in the multi-lingual setting, using the test split of DiscoGeM 2.0.

As shown in Table 5, the HArch model trained in a multi-task setting consistently outperforms the single-level models across all three sense levels. The performance gains are especially pronounced at level-2 and level-3, suggesting that these finer-grained senses benefit significantly from the shared representations and hierarchical cues provided by joint training. While level-1 predictions also improve slightly, the main advantage of multi-task learning emerges from its ability to propagate information downward through the sense hierarchy. This ablation study isolates the impact of our architecture and confirms that jointly modeling the hierarchy enhances multi-label IDRR performance. In addition to yielding better performance, HArch is also more efficient - as shown by the total time it took to calculate the averaged results in each row of Table 5 with a 32-core compute node with 512GB of RAM. This highlights both the effectiveness and efficiency of our proposed architecture.

8 Conclusion

In this work, we presented the first multi-lingual and multi-label classification model for IDRR. We modeled hierarchical dependencies between discourse senses to predict probability distributions across all three sense levels in the PDTB 3.0 framework. We introduced the first evaluation on the DiscoGeM 2.0 corpus across all languages and compared different pre-trained language models as encoder backbones within our proposed HArch model architecture. Results show that RoBERTa-HArch achieves the best performance in English, while XLM-RoBERTa-HArch performs best in the multi-lingual setting. We further evaluated our models against GPT-4o and Llama-4-Maverick using few-shot prompting across all language configurations and found that the HArch models consistently obtained lower JS distances. These findings demonstrate that fine-tuning smaller encoder-based models can outperform prompting large language models in the context of multi-label IDRR. Lastly, we benchmarked RoBERTa-HArch on the English-only DiscoGeM 1.0 corpus and achieved SOTA results compared to existing work.

These findings not only advance the field of IDRR, but also highlight the advantages of fine-tuning smaller encoder-based models over prompting large language models — particularly, given the significantly higher monetary and environmental costs associated with the latter. Looking ahead, we plan to investigate how discourse sense predictions vary across languages in DiscoGeM 2.0 for parallel discourse relations. In particular, we are interested in exploring how translation influences discourse reasoning and label distribution across these four languages. Additionally, we aim to compare the performance of using XLM-RoBERTa as the encoder in our HArch model in the German, French and Czech languages against encoders pre-trained specifically in these languages. This line of research may shed light on cross-linguistic challenges in discourse analysis and lead to more robust multi-lingual IDRR models.

9 Limitations

The PDTB 3.0 framework defines a total of 22 level-2 senses and 36 level-3 senses (after projection from level-2 senses). However, the Belief and SPEECH-ACT senses were not annotated in the DiscoGeM 2.0 corpus. As a result, our HArch model could only be trained on 17 level-2 and 28 level-3 senses (after projection from level-2 senses). Should these additional senses be annotated in future versions of the dataset, our model could be readily extended to incorporate them.

Another limitation of this work concerns the choice of PLMs used to generate embeddings for the pair of discourse arguments. Due to computational and time constraints, we did not experiment with larger models such as LLaMA 3 (Grattafiori et al., 2024), which might have led to improved performance. Although our experiments were conducted using a high-performance computing infrastructure, we relied on relatively smaller models, such as RoBERTa, to reduce resource demands. Nevertheless, the environmental impact of such experiments remains non-negligible. Future work should take into account the energy consumption associated with fine-tuning PLMs and prompting LLMs and consider it as an additional metric when evaluating model performance.

Acknowledgements

The authors would like to thank the anonymous reviewers for their comments. This work was financially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC).

Appendix A Data Statistics

Table 6 shows the distribution of level-2 and level-3 senses per language in the DiscoGeM 2.0 corpus. We replaced the non-existent PDTB 3.0 level-3 senses with their corresponding level-2 sense and marked them with * in the Level-3 column. Each value represents the sum of the corresponding sense in all the multi-label distributions of implicit discourse relations in the specific language subset.

Appendix B Prompt Template

Figure 2 illustrates the prompt-response template used to evaluate GPT-4o and LLaMA-4-Maverick on multi-label IDRR using the English subset of DiscoGeM 2.0. For evaluations in other languages, we translated the prompt template and selected examples from the corresponding language subset of the corpus. In the multi-lingual setting, where all four languages are used simultaneously, we kept the prompt structure in English and provided examples in all four languages. Due to space constraints, Figure 2 only shows two examples of discourse relations in the initial User Message prompt. However, we included five examples when prompting the models. All of the examples were taken from the training split of DiscoGeM 2.0.

We designed the prompt to closely replicate the annotation methodology of DiscoGeM 2.0. When crowdsourcing the annotation of the corpus, the authors provided the annotators a list of non-ambiguous connectives (Yung et al., 2024b) that represented each of the possible senses in the PDTB 3.0 framework. This list of connectives was adapted for each language. The annotators were then asked to choose the connective that best expresses the semantic relation between the arguments of the relation being annotated, irrespective of whether syntax would need to be adjusted. By mirroring this annotation process in our prompting template, we were able to directly evaluate the performance of both LLMs on multi-label IDRR using the test split of DiscoGeM 2.0.