asLLR: LLM based Leads Ranking in Auto Sales

TL;DR

This paper introduces asLLR, a novel LLM-based approach for ranking sales leads in auto sales, effectively combining natural language and tabular data modeling to improve prediction accuracy and sales outcomes.

Contribution

The study presents a new LLM-based model integrating CTR and QA losses for better lead ranking, with a custom dataset and significant real-world sales improvements.

Findings

Achieved an AUC of 0.8127, surpassing traditional methods by 0.0231.

Enhanced CTR models with text features by 0.0058.

Increased sales volume by approximately 9.5% in real-world testing.

Abstract

In the area of commercial auto sales system, high-quality lead score sequencing determines the priority of a sale's work and is essential for optimizing the efficiency of the sales system. Since CRM (Customer Relationship Management) system contains plenty of textual interaction features between sales and customers, traditional techniques such as Click Through Rate (CTR) prediction struggle with processing the complex information inherent in natural language features, which limits their effectiveness in sales lead ranking. Bridging this gap is critical for enhancing business intelligence and decision-making. Recently, the emergence of large language models (LLMs) has opened new avenues for improving recommendation systems, this study introduces asLLR (LLM-based Leads Ranking in Auto Sales), which integrates CTR loss and Question Answering (QA) loss within a decoder-only large language model architecture. This integration enables the simultaneous modeling of both tabular and natural language features. To verify the efficacy of asLLR, we constructed an innovative dataset derived from the customer lead pool of a prominent new energy vehicle brand, with 300,000 training samples and 40,000 testing samples. Our experimental results demonstrate that asLLR effectively models intricate patterns in commercial datasets, achieving the AUC of 0.8127, surpassing traditional CTR estimation methods by 0.0231. Moreover, asLLR enhances CTR models when used for extracting text features by 0.0058. In real-world sales scenarios, after rigorous online A/B testing, asLLR increased the sales volume by about 9.5% compared to the traditional method, providing a valuable tool for business intelligence and operational decision-making.