Applying the Chinese Wall Reverse Engineering Technique to Large Language Model Code Editing

TL;DR

This paper introduces a novel application of the Chinese Wall reverse engineering technique to enhance the performance of weaker code language models by leveraging high-quality models for instruction generation, improving their ability to perform complex tasks.

Contribution

The paper proposes a new method applying the Chinese Wall technique to improve weaker code LLMs using high-quality model instructions, addressing ethical and copyright concerns.

Findings

Improved Comma v0.1 1T performance by over 66% on CanItEdit benchmark.

Enhanced Starcoder2 Instruct performance by roughly 20%.

Demonstrated practical limitations due to lack of public domain training data.

Abstract

Large language models for code (Code LLM) are increasingly utilized in programming environments. Despite their utility, the training datasets for top LLM remain undisclosed, raising concerns about potential copyright violations. Some models, such as Pleias and Comma put emphasis on data curation and licenses, however, with limited training data these models are not competitive and only serve as proof of concepts. To improve the utility of these models, we propose an application of the "Chinese Wall" technique, inspired by the reverse engineering technique of the same name -- a high quality model is used to generate detailed instructions for a weaker model. By doing so, a weaker but ethically aligned model may be used to perform complicated tasks that, otherwise, can only be completed by more powerful models. In our evaluation, we've found that this technique improves Comma v0.1 1T's performance in CanItEdit benchmark by over 66%, and Starcoder2 Instruct by roughly 20% compared to when running the same model on the benchmark alone. The practical application of this technique today, however, may be limited due to the lack of models trained on public domain content without copyright restrictions.