Tele-FLM Technical Report

TL;DR

This paper introduces Tele-FLM, a 52-billion-parameter multilingual large language model with a stable, efficient pre-training method, demonstrating strong multilingual and English/Chinese capabilities while sharing detailed training insights.

Contribution

The paper presents Tele-FLM, a novel open-source multilingual LLM with a stable training paradigm and competitive performance, addressing the lack of scalable methodologies for models beyond 50B parameters.

Findings

Tele-FLM achieves superior multilingual modeling abilities.

It performs comparably to larger models like Llama2-70B and DeepSeek-67B.

The report shares core design and training practices for community benefit.

Abstract

Large language models (LLMs) have showcased profound capabilities in language understanding and generation, facilitating a wide array of applications. However, there is a notable paucity of detailed, open-sourced methodologies on efficiently scaling LLMs beyond 50 billion parameters with minimum trial-and-error cost and computational resources. In this report, we introduce Tele-FLM (aka FLM-2), a 52B open-sourced multilingual large language model that features a stable, efficient pre-training paradigm and enhanced factual judgment capabilities. Tele-FLM demonstrates superior multilingual language modeling abilities, measured by BPB on textual corpus. Besides, in both English and Chinese foundation model evaluation, it is comparable to strong open-sourced models that involve larger pre-training FLOPs, such as Llama2-70B and DeepSeek-67B. In addition to the model weights, we share the core designs, engineering practices, and training details, which we expect to benefit both the academic and industrial communities.