CNRL at SemEval-2020 Task 5: Modelling Causal Reasoning in Language with Multi-Head Self-Attention Weights based Counterfactual Detection

TL;DR

This paper presents a novel method for detecting counterfactuals in text by leveraging multi-head self-attention weights from transformer models, combining convolutional layers and fine-tuning for improved causal reasoning detection.

Contribution

It introduces a new approach that uses self-attention weights and convolutional layers for counterfactual detection, with a fine-tuning strategy for related sub-tasks.

Findings

Transformer models effectively extract contextual features.

Self-attention weights provide interpretability of model dynamics.

Fine-tuning improves counterfactual detection performance.

Abstract

In this paper, we describe an approach for modelling causal reasoning in natural language by detecting counterfactuals in text using multi-head self-attention weights. We use pre-trained transformer models to extract contextual embeddings and self-attention weights from the text. We show the use of convolutional layers to extract task-specific features from these self-attention weights. Further, we describe a fine-tuning approach with a common base model for knowledge sharing between the two closely related sub-tasks for counterfactual detection. We analyze and compare the performance of various transformer models in our experiments. Finally, we perform a qualitative analysis with the multi-head self-attention weights to interpret our models' dynamics.