Language Models as a Service: Overview of a New Paradigm and its Challenges

TL;DR

This paper discusses the challenges of evaluating and trusting proprietary language models accessed via services, and provides a comprehensive overview of current LMaaS systems, their licenses, and capabilities.

Contribution

It systematically analyzes the challenges of LMaaS, reviews existing solutions, and offers recommendations and a detailed overview of major LMaaS platforms.

Findings

Identified key challenges in evaluating LMaaS

Reviewed existing solutions and their limitations

Provided a comprehensive overview of major LMaaS platforms

Abstract

Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how the aforementioned challenges act as impediments to the accessibility, replicability, reliability, and trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information about language models for each of these four aspects. We conduct a detailed analysis of existing solutions and put forth a number of considered recommendations, and highlight the directions for future advancements. On the other hand, it serves as a comprehensive resource for existing knowledge on current, major LMaaS, offering a synthesized overview of the licences and capabilities their interfaces offer.