UCoder: Unsupervised Code Generation by Internal Probing of Large Language Models

TL;DR

This paper presents UCoder, an unsupervised code generation framework leveraging internal probing of large language models, reducing reliance on labeled data while maintaining competitive performance.

Contribution

Introduces IPC, an unsupervised internal probing method for code generation, enabling training of UCoder without external datasets or labeled code snippets.

Findings

Unsupervised methods achieve competitive results on code benchmarks.

Internal model states contain valuable signals for code quality assessment.

Effective unsupervised training reduces dependence on labeled data.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in code generation tasks. However, their effectiveness heavily relies on supervised training with extensive labeled (e.g., question-answering pairs) or unlabeled datasets (e.g., code snippets), which are often expensive and difficult to obtain at scale. To address this limitation, this paper introduces a method IPC, an unsupervised framework that leverages Internal Probing of LLMs for Code generation without any external corpus, even unlabeled code snippets. We introduce the problem space probing, test understanding probing, solution space probing, and knowledge consolidation and reinforcement to probe the internal knowledge and confidence patterns existing in LLMs. Further, IPC identifies reliable code candidates through self-consistency mechanisms and representation-based quality estimation to train UCoder (coder with unsupervised learning). We validate the proposed approach across multiple code benchmarks, demonstrating that unsupervised methods can achieve competitive performance compared to supervised approaches while significantly reducing the dependency on labeled data and computational resources. Analytic experiments reveal that internal model states contain rich signals about code quality and correctness, and that properly harnessing these signals enables effective unsupervised learning for code generation tasks, opening new directions for training code LLMs in resource-constrained scenarios.