Improving the Capabilities of Large Language Model Based Marketing Analytics Copilots With Semantic Search And Fine-Tuning

TL;DR

This paper enhances large language models for marketing analytics by combining semantic search, prompt engineering, and fine-tuning, improving their accuracy in domain-specific tasks like SQL generation and data analysis.

Contribution

It introduces a novel approach integrating semantic search and fine-tuning to improve LLM performance in marketing analytics tasks.

Findings

Semantic search significantly improves model accuracy.

Fine-tuning enhances domain-specific question-answering.

Open-source models perform comparably to proprietary ones in tests.

Abstract

Artificial intelligence (AI) is widely deployed to solve problems related to marketing attribution and budget optimization. However, AI models can be quite complex, and it can be difficult to understand model workings and insights without extensive implementation teams. In principle, recently developed large language models (LLMs), like GPT-4, can be deployed to provide marketing insights, reducing the time and effort required to make critical decisions. In practice, there are substantial challenges that need to be overcome to reliably use such models. We focus on domain-specific question-answering, SQL generation needed for data retrieval, and tabular analysis and show how a combination of semantic search, prompt engineering, and fine-tuning can be applied to dramatically improve the ability of LLMs to execute these tasks accurately. We compare both proprietary models, like GPT-4, and open-source models, like Llama-2-70b, as well as various embedding methods. These models are tested on sample use cases specific to marketing mix modeling and attribution.