An Empirical Study on Strong-Weak Model Collaboration for Repo-level Code Generation

TL;DR

This paper investigates cost-effective collaboration strategies between strong and weak language models for repository-level code generation, demonstrating that certain methods can match strong model performance at significantly reduced costs.

Contribution

It provides a comprehensive evaluation of various collaboration strategies and offers practical guidelines for optimizing cost and performance in code generation tasks.

Findings

Pipeline-based methods are most efficient.

Collaboration can reduce costs by 40%.

Performance comparable to strong models achieved.

Abstract

We study cost-efficient collaboration between strong and weak language models for repository-level code generation, where the weak model handles simpler tasks at lower cost, and the most challenging tasks are delegated to the strong model. While many works propose architectures for this task, few analyze performance relative to cost. We evaluate a broad spectrum of collaboration strategies: context-based, pipeline-based, and dynamic, on GitHub issue resolution. Our most effective collaborative strategy achieves equivalent performance to the strong model while reducing the cost by 40%. Based on our findings, we offer actionable guidelines for choosing collaboration strategies under varying budget and performance constraints. Our results show that strong-weak collaboration substantially boosts the weak model's performance at a fraction of the cost, pipeline and context-based methods being most efficient. We release the code for our work at https://github.com/shubhamrgandhi/codegen-strong-weak-collab.