> For the complete documentation index, see [llms.txt](https://brindha.gitbook.io/mylearning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://brindha.gitbook.io/mylearning/tools/mpt.md).

# MPT

#### MPT: MosaicML Pretrained Transformer

**What is MPT?**

MPT — MosaicML Pretrained Transformer — is a family of open-source language models released by MosaicML starting in May 2023. MPT was notable at the time of release as one of the first high-quality, commercially usable open-source LLMs that matched LLaMA's quality without the non-commercial restrictions that limited LLaMA derivatives. MosaicML was subsequently acquired by Databricks in June 2023 for $1.3 billion, and MPT became part of the Databricks AI portfolio, later succeeded by DBRX.

***

**Training**

MPT-7B was trained entirely from scratch on 1 trillion tokens of text and code using the MosaicML platform, completing in 9.5 days at a cost of approximately $200,000 with zero human intervention during training. The pretraining data came from a curated mix of ten open-source text corpora, tokenized using the EleutherAI GPT-NeoX-20B tokenizer, and included elements of the RedPajama dataset for up-to-date web and Wikipedia content.

***

**Architecture Innovations**

MPT incorporated several technical improvements over earlier open models. Flash Attention was used for training and inference efficiency. ALiBi (Attention with Linear Biases) replaced standard positional embeddings, enabling the model to extrapolate to much longer context lengths than it was trained on — MPT was trained on contexts up to 65,000 tokens and could handle inputs up to 84,000 tokens, far exceeding the 2,000 to 4,000 token limits of other open models of the era. Additional stability improvements were implemented to prevent loss spikes during training.

***

**Model Variants**

MPT-7B was the initial release and the first commercially licensed model in the family. MPT-7B-Instruct, MPT-7B-Chat, and MPT-7B-StoryWriter were fine-tuned variants optimized for instruction following, conversational AI, and long-form story generation respectively. MPT-30B followed in June 2023, a significantly larger model that MosaicML claimed outperformed the original GPT-3 on benchmarks, also released for commercial use.

***

**Commercial License**

A defining feature of MPT was its Apache 2.0 license for commercial use — a meaningful differentiation from LLaMA-based models at the time, which carried non-commercial restrictions. This made MPT one of the few genuinely commercially deployable open-weight models in mid-2023.

***

**Legacy**

After Databricks acquired MosaicML, the MPT project was gradually superseded by DBRX, which applied many of the same training techniques and infrastructure at a much larger scale with a MoE architecture. The Mosaic AI training platform and open-source training libraries developed for MPT — LLM Foundry, Composer, and Streaming — became core infrastructure for DBRX development.

***

**Strengths**

* Commercially usable Apache 2.0 license at a time when most competitors were non-commercial
* ALiBi enabling very long context extrapolation (up to 84K tokens)
* Fully documented training process with open weights
* Trained from scratch rather than fine-tuned — a genuine foundation model
* Strong performance matching LLaMA-7B at launch

**Limitations**

* Superseded by DBRX and other more capable models
* No longer actively developed
* 200K training cost limits reproducibility for academic researchers
* Less multilingual depth than purpose-built multilingual models