Learning to Solve Domain-Specific Calculation Problems with Knowledge-Intensive Programs Generator

TL;DR

This paper introduces KIPG, a pipeline that generates knowledge-intensive programs to solve complex domain-specific calculation problems, demonstrated effectively in the legal domain and adaptable to others without retraining.

Contribution

It proposes a novel knowledge-intensive programs generator pipeline that improves solving complex domain-specific problems by extracting key variables and iteratively aligning code logic with domain knowledge.

Findings

KIPG effectively solves complex legal domain problems.

The code generator adapts to new domains without retraining.

Experimental results validate the pipeline's effectiveness.

Abstract

Domain Large Language Models (LLMs) are developed for domain-specific tasks based on general LLMs. But it still requires professional knowledge to facilitate the expertise for some domain-specific tasks. In this paper, we investigate into knowledge-intensive calculation problems. We find that the math problems to be challenging for LLMs, when involving complex domain-specific rules and knowledge documents, rather than simple formulations of terminologies. Therefore, we propose a pipeline to solve the domain-specific calculation problems with Knowledge-Intensive Programs Generator more effectively, named as KIPG. It generates knowledge-intensive programs according to the domain-specific documents. For each query, key variables are extracted, then outcomes which are dependent on domain knowledge are calculated with the programs. By iterative preference alignment, the code generator learns to improve the logic consistency with the domain knowledge. Taking legal domain as an example, we have conducted experiments to prove the effectiveness of our pipeline, and extensive analysis on the modules. We also find that the code generator is also adaptable to other domains, without training on the new knowledge.