> For the complete documentation index, see [llms.txt](https://brindha.gitbook.io/mylearning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://brindha.gitbook.io/mylearning/tools/pythia.md).

# Pythia

#### Pythia: EleutherAI's Suite for Studying Language Model Training

**What is Pythia?**

Pythia is a suite of large language models released by EleutherAI in April 2023, designed not primarily as production-ready assistants but as scientific tools for studying how language models develop during training. Pythia's defining feature is that it releases not just the final model weights but the full sequence of intermediate training checkpoints — allowing researchers to observe how model behavior, capabilities, and representations evolve throughout the training process. It is the most comprehensively documented suite of language models ever released for research purposes.

***

**Research Mission**

EleutherAI designed Pythia to enable reproducible research into language model training dynamics. Questions that Pythia was designed to help answer include: how do models acquire factual knowledge over training? How does memorization develop? How do biases emerge and evolve? How does model behavior change as a function of training tokens seen? When do different capabilities emerge? All of these questions require access to models at different stages of training — something no other public model family provided at the time.

***

**Model Sizes and Checkpoints**

Pythia was released in eight sizes: 70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, and 12B parameters. Each model was trained on the same data in the same order, and 154 intermediate checkpoints are available for each model, spaced approximately every 2 billion tokens throughout training. This design allows direct comparison across scales — researchers can study the same training dynamics at different parameter counts with controlled variables.

***

**Training Data**

All Pythia models were trained on The Pile — a diverse, curated 825GB open-source text dataset assembled by EleutherAI containing 22 diverse sources including books, academic papers, code, web content, and specialized datasets. The Pile and the order in which it was presented to the models were both publicly released, making Pythia one of the most reproducible model suites in existence. Deduplicated and non-deduplicated versions of Pythia were released to study the effects of data deduplication on model behavior.

***

**Architecture**

Pythia uses a GPT-NeoX style decoder-only transformer architecture with Flash Attention and rotary positional embeddings. All models share the same architecture design, differing only in scale, to facilitate clean comparisons across parameter counts.

***

**Influence**

Beyond its research contributions, Pythia became foundational infrastructure for other projects. Databricks used EleutherAI's Pythia-12B as the base for Dolly 2.0, and the GPT-NeoX training framework developed by EleutherAI was used by numerous subsequent open-source model training efforts including RedPajama and StableLM. EleutherAI's broader contributions to open LLM infrastructure — The Pile, lm-evaluation-harness, and GPT-NeoX — have become standard tools in the open research ecosystem.

***

**License**

Pythia is released under Apache 2.0, making it fully open for both research and commercial use.

***

**Strengths**

* Unique research value: 154 intermediate checkpoints per model enable training dynamics studies impossible with other model families
* Eight model sizes with controlled variables for scaling studies
* Fully open: weights, checkpoints, training data, and data ordering all public
* Apache 2.0 commercial license
* Foundational infrastructure for Dolly, StableLM, and many other projects
* The Pile dataset remains one of the most studied pretraining datasets in open research

**Limitations**

* Not optimized for instruction following or conversational use out of the box
* Surpassed in performance by more recently trained models
* Primarily English from The Pile
* No multimodal capabilities
* Intended as a research tool rather than a deployment model