Enhanced Integrated Scoring for Cleaning Dirty Texts

TL;DR

This paper improves the ISSAC system for cleaning and preprocessing dirty texts from online sources, achieving higher accuracy in spelling correction, abbreviation expansion, and case restoration for ontology engineering.

Contribution

The paper introduces an enhanced version of ISSAC that significantly improves text preprocessing accuracy in ontology engineering applications.

Findings

Achieved 98% accuracy with enhanced ISSAC on chat records.

Outperformed basic ISSAC and Aspell in cleaning accuracy.

Demonstrated effectiveness on 700 chat records.

Abstract

An increasing number of approaches for ontology engineering from text are gearing towards the use of online sources such as company intranet and the World Wide Web. Despite such rise, not much work can be found in aspects of preprocessing and cleaning dirty texts from online sources. This paper presents an enhancement of an Integrated Scoring for Spelling error correction, Abbreviation expansion and Case restoration (ISSAC). ISSAC is implemented as part of a text preprocessing phase in an ontology engineering system. New evaluations performed on the enhanced ISSAC using 700 chat records reveal an improved accuracy of 98% as compared to 96.5% and 71% based on the use of only basic ISSAC and of Aspell, respectively.