Pre-train, Interact, Fine-tune: A Novel Interaction Representation for Text Classification

TL;DR

This paper introduces a novel interaction-based text representation called Hybrid Interaction Representation (HIR) and a Pre-train, Interact, Fine-tune (PIF) architecture, which together improve text classification accuracy across multiple datasets.

Contribution

The paper proposes a new interaction representation capturing local and global word interactions, combined with a PIF architecture that integrates feature-based and fine-tuning methods for better text classification.

Findings

Ensemble PIF outperforms state-of-the-art baselines by 2.03% to 3.15% in error rate.

The proposed method maintains performance improvements regardless of text length.

The hybrid interaction representation effectively captures semantic relationships in text.

Abstract

Text representation can aid machines in understanding text. Previous work on text representation often focuses on the so-called forward implication, i.e., preceding words are taken as the context of later words for creating representations, thus ignoring the fact that the semantics of a text segment is a product of the mutual implication of words in the text: later words contribute to the meaning of preceding words. We introduce the concept of interaction and propose a two-perspective interaction representation, that encapsulates a local and a global interaction representation. Here, a local interaction representation is one that interacts among words with parent-children relationships on the syntactic trees and a global interaction interpretation is one that interacts among all the words in a sentence. We combine the two interaction representations to develop a Hybrid Interaction Representation (HIR). Inspired by existing feature-based and fine-tuning-based pretrain-finetuning approaches to language models, we integrate the advantages of feature-based and fine-tuning-based methods to propose the Pre-train, Interact, Fine-tune (PIF) architecture. We evaluate our proposed models on five widely-used datasets for text classification tasks. Our ensemble method, outperforms state-of-the-art baselines with improvements ranging from 2.03% to 3.15% in terms of error rate. In addition, we find that, the improvements of PIF against most state-of-the-art methods is not affected by increasing of the length of the text.