TRIM: Token Reduction and Inference Modeling for Cost-Effective Language Generation

TL;DR

TRIM reduces inference costs in large language models by omitting redundant words during generation and reconstructing the full answer with a smaller model, maintaining accuracy while saving tokens.

Contribution

The paper introduces TRIM, a novel pipeline that combines token reduction with inference modeling to improve efficiency in language generation tasks.

Findings

19.4% token savings on average with GPT-4o

Tiny decrease in evaluation metrics

Effective balance of efficiency and accuracy

Abstract

The high inference cost of Large Language Models (LLMs) poses challenges, especially for tasks requiring lengthy outputs. However, natural language often contains redundancy, which presents an opportunity for optimization. We have observed that LLMs can generate distilled language (i.e., concise outputs that retain essential meaning) when prompted appropriately. We propose TRIM, a pipeline for saving computational cost in which the LLM omits a predefined set of semantically irrelevant and easily inferable words based on the context during inference. Then, a specifically trained smaller language model with lower inference cost reconstructs the distilled answer into the ideal answer. Our experiments show promising results, particularly on the proposed NaLDA evaluation dataset focused on the reconstruction task, with 19.4% saved tokens on average for GPT-4o and only a tiny decrease in evaluation metrics. This suggests that the approach can effectively balance efficiency and accuracy in language processing tasks.