# Mining Discourse Markers for Unsupervised Sentence Representation Learning

## TL;DR

This paper introduces a method to automatically extract discourse markers from large unannotated text data to improve unsupervised sentence embedding learning, achieving state-of-the-art results in transfer tasks.

## Contribution

It presents a novel automatic extraction technique for discourse markers and demonstrates their effectiveness in training transferable sentence representations.

## Key findings

- Discovered 174 discourse markers with over 10,000 examples each.
- Achieved state-of-the-art results in various transfer learning tasks.
- Datasets are publicly available for further research.

## Abstract

Current state of the art systems in NLP heavily rely on manually annotated datasets, which are expensive to construct. Very little work adequately exploits unannotated data -- such as discourse markers between sentences -- mainly because of data sparseness and ineffective extraction methods. In the present work, we propose a method to automatically discover sentence pairs with relevant discourse markers, and apply it to massive amounts of data. Our resulting dataset contains 174 discourse markers with at least 10k examples each, even for rare markers such as coincidentally or amazingly We use the resulting data as supervision for learning transferable sentence embeddings. In addition, we show that even though sentence representation learning through prediction of discourse markers yields state of the art results across different transfer tasks, it is not clear that our models made use of the semantic relation between sentences, thus leaving room for further improvements. Our datasets are publicly available (https://github.com/synapse-developpement/Discovery)