Indirectly Supervised English Sentence Break Prediction Using Paragraph Break Probability Estimates

TL;DR

This paper introduces a novel sentence break prediction method using paragraph break probability estimates, achieving high accuracy with minimal annotated data, and further improved by combining signals with a support vector machine classifier.

Contribution

It presents an indirectly supervised approach to sentence break prediction that relies mainly on paragraph break probabilities, reducing the need for extensive annotated data.

Findings

High accuracy achieved with minimal annotated data

Combining paragraph probabilities with SVM improves results

Effective across both in-domain and out-of-domain texts

Abstract

This report explores the use of paragraph break probability estimates to help predict the location of sentence breaks in English natural language text. We show that a sentence break predictor based almost solely on paragraph break probability estimates can achieve high accuracy on this task. This sentence break predictor is trained almost entirely on a large amount of naturally occurring text without sentence break annotations, with only a small amount of annotated data needed to tune two hyperparameters. We also show that even better results can be achieved across in-domain and out-of-domain test data, if paragraph break probability signals are combined with a support vector machine classifier trained on a somewhat larger amount of sentence-break-annotated data. Numerous related issues are addressed along the way.