ColBERT Retrieval and Ensemble Response Scoring for Language Model Question Answering

TL;DR

This paper presents a question answering system for telecom domain using ColBERT retrieval and ensemble scoring, significantly improving accuracy for small language models in specialized technical questions.

Contribution

It introduces a novel combination of ColBERT retrieval and ensemble response scoring tailored for small language models in domain-specific QA tasks.

Findings

Achieved 81.9% accuracy with Phi-2

Achieved 57.3% accuracy with Falcon-7B

Publicly released code and models

Abstract

Domain-specific question answering remains challenging for language models, given the deep technical knowledge required to answer questions correctly. This difficulty is amplified for smaller language models that cannot encode as much information in their parameters as larger models. The "Specializing Large Language Models for Telecom Networks" challenge aimed to enhance the performance of two small language models, Phi-2 and Falcon-7B in telecommunication question answering. In this paper, we present our question answering systems for this challenge. Our solutions achieved leading marks of 81.9% accuracy for Phi-2 and 57.3% for Falcon-7B. We have publicly released our code and fine-tuned models.