C2LLM Technical Report: A New Frontier in Code Retrieval via Adaptive Cross-Attention Pooling

TL;DR

C2LLM introduces a novel code embedding approach using adaptive cross-attention pooling, achieving state-of-the-art results on code retrieval benchmarks with flexible, high-quality sequence representations.

Contribution

The paper presents C2LLM, a new code embedding model leveraging a multihead attention pooling module to improve sequence representation and retrieval performance.

Findings

C2LLM-7B ranks 1st on MTEB-Code leaderboard.

Models trained on three million data achieve record performance.

The pooling method effectively utilizes causal representations and aggregates token information.

Abstract

We present C2LLM - Contrastive Code Large Language Models, a family of code embedding models in both 0.5B and 7B sizes. Building upon Qwen-2.5-Coder backbones, C2LLM adopts a Pooling by Multihead Attention (PMA) module for generating sequence embedding from token embeddings, effectively 1) utilizing the LLM's causal representations acquired during pretraining, while also 2) being able to aggregate information from all tokens in the sequence, breaking the information bottleneck in EOS-based sequence embeddings, and 3) supporting flexible adaptation of embedding dimension, serving as an alternative to MRL. Trained on three million publicly available data, C2LLM models set new records on MTEB-Code among models of similar sizes, with C2LLM-7B ranking 1st on the overall leaderboard.