LFTK: Handcrafted Features in Computational Linguistics

TL;DR

LFTK is an open-source, systematically organized multilingual system that extracts over 220 handcrafted linguistic features, aiding various NLP tasks and addressing previous inconsistencies and lack of standardization.

Contribution

It introduces a comprehensive categorization, correlation analysis, and an expandable multilingual extraction system for handcrafted linguistic features.

Findings

Identified over 220 useful handcrafted features

Provided correlation insights for task-specific applications

Developed and released the largest open-source feature extraction system

Abstract

Past research has identified a rich set of handcrafted linguistic features that can potentially assist various tasks. However, their extensive number makes it difficult to effectively select and utilize existing handcrafted features. Coupled with the problem of inconsistent implementation across research works, there has been no categorization scheme or generally-accepted feature names. This creates unwanted confusion. Also, most existing handcrafted feature extraction libraries are not open-source or not actively maintained. As a result, a researcher often has to build such an extraction system from the ground up. We collect and categorize more than 220 popular handcrafted features grounded on past literature. Then, we conduct a correlation analysis study on several task-specific datasets and report the potential use cases of each feature. Lastly, we devise a multilingual handcrafted linguistic feature extraction system in a systematically expandable manner. We open-source our system for public access to a rich set of pre-implemented handcrafted features. Our system is coined LFTK and is the largest of its kind. Find it at github.com/brucewlee/lftk.