When Do Discourse Markers Affect Computational Sentence Understanding?

TL;DR

This paper evaluates how well NLP systems understand discourse connectives, revealing that their processing varies by connective type and that explicit connective knowledge improves system accuracy.

Contribution

It provides a systematic assessment of NLP systems' ability to process discourse connectives and highlights the importance of explicit connective understanding for improved NLP performance.

Findings

NLP systems do not process all discourse connectives equally.

Processing complexity of connectives varies and is not always aligned with human perception.

Explicit knowledge of connectives enhances NLP system accuracy.

Abstract

The capabilities and use cases of automatic natural language processing (NLP) have grown significantly over the last few years. While much work has been devoted to understanding how humans deal with discourse connectives, this phenomenon is understudied in computational systems. Therefore, it is important to put NLP models under the microscope and examine whether they can adequately comprehend, process, and reason within the complexity of natural language. In this chapter, we introduce the main mechanisms behind automatic sentence processing systems step by step and then focus on evaluating discourse connective processing. We assess nine popular systems in their ability to understand English discourse connectives and analyze how context and language understanding tasks affect their connective comprehension. The results show that NLP systems do not process all discourse connectives equally well and that the computational processing complexity of different connective kinds is not always consistently in line with the presumed complexity order found in human processing. In addition, while humans are more inclined to be influenced during the reading procedure but not necessarily in the final comprehension performance, discourse connectives have a significant impact on the final accuracy of NLP systems. The richer knowledge of connectives a system learns, the more negative effect inappropriate connectives have on it. This suggests that the correct explicitation of discourse connectives is important for computational natural language processing.