Apertus: A Fully Open, Multilingual LLM

Apertus is a fully open, multilingual large language model with 70B and 8B parameters, supporting over 1000 languages and long context. Trained on 15T tokens of fully compliant, open data, it achieves performance comparable to closed-source models. Apertus uses a novel xIELU activation function and the AdEMAMix optimizer, undergoing supervised fine-tuning and QRPO alignment. Its weights, data, and training details are publicly available, respecting data owner opt-out consent and avoiding memorization of training data. Integrated into the transformers library, Apertus supports various deployment methods. While powerful, users should be aware of potential inaccuracies and biases in its output.