An Effective Framework to Help Large Language Models Handle Numeric-involved Long-context Tasks

TL;DR

This paper introduces a workflow that decomposes numeric-involved long-context tasks into subtasks, enabling smaller models and code-based calculations to improve accuracy and reduce costs in large language models.

Contribution

The proposed workflow effectively handles numeric-involved long-context tasks by decomposing them and leveraging code generation, addressing limitations of current LLMs.

Findings

Improved accuracy on numeric long-context benchmarks.

Significant reduction in API call costs.

Effective use of smaller models for subtask processing.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities in handling long texts and have almost perfect performance in traditional retrieval tasks. However, their performance significantly degrades when it comes to numerical calculations in the long-context. Numeric-involved long-context tasks typically cannot be addressed by current LLMs in normal settings due to their inherent limitations in simultaneously handling complex and massive information. Some CoT like prompting methods can improve accuracy but demands massive output tokens, which is costly and slow. To address this issue, we propose a workflow, which decompose a numeric-involved long-context task into 4 low-level subtasks: judging, extracting and processing with code and conclusion. The former 2 subtasks is relatively simple, which allows us to use smaller models for efficiently processing long context. When numerical calculations are required, we use code generated by LLMs to avoid the disadvantage of LLM not being good at calculations. The results in 2 numeric-involved long-context benchmarks demonstrate our workflow can not only improve accuracy, but also significantly reduce the cost of API calls.