> For the complete documentation index, see [llms.txt](https://brindha.gitbook.io/mylearning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://brindha.gitbook.io/mylearning/tools/bloom.md).

# Bloom

#### BLOOM: BigScience's Multilingual Foundation Model

**What is BLOOM?**

BLOOM — BigScience Large Open-science Open-access Multilingual Language Model — is a 176 billion parameter open-access language model released in July 2022 by BigScience, a collaborative research initiative involving over 1,000 researchers from more than 60 countries and 250 institutions, coordinated by Hugging Face. BLOOM was the largest openly accessible language model at the time of its release and represented a landmark achievement in international, community-driven AI research.

***

**The BigScience Initiative**

BigScience was established as a one-year open science collaboration with the specific goal of creating a large, openly accessible multilingual language model as an alternative to closed proprietary models. The project was supported by the French national computing infrastructure GENCI and the Jean Zay supercomputer, where training took place over approximately 3.5 months using 384 A100 80GB GPUs. The collaborative structure, bringing together academic institutions, research labs, and independent researchers from around the world, was itself a significant experiment in distributed scientific collaboration.

***

**Architecture**

BLOOM uses a decoder-only transformer architecture based on Megatron-LM with several modifications. It employs ALiBi positional embeddings (similar to MPT), allowing attention to be more robust at varying context lengths. It uses learned positional embeddings, full attention without sparse patterns, and embedding layer norm for training stability. The tokenizer was trained specifically on the multilingual training data, with a vocabulary of 250,680 tokens designed to serve the model's broad language coverage.

***

**Multilingual Training Data**

BLOOM's defining characteristic is its multilingual scope. The training dataset, ROOTS, contains 498 sources across 46 natural languages and 13 programming languages, totaling 1.61 trillion tokens. Languages include French, Spanish, Portuguese, Arabic, Catalan, Chinese, English, Indonesian, and many others, with deliberate effort to include lower-resource languages underrepresented in most large models. All data sources were openly curated by BigScience working groups, and the dataset curation process itself was published.

***

**Model Family**

Alongside the flagship 176B model, BigScience released BLOOM in smaller sizes: 560M, 1.1B, 1.7B, 3B, and 7.1B parameters — all trained on the same ROOTS dataset. BLOOMZ, a fine-tuned instruction-following version of BLOOM, was also released, along with mT0 for multilingual task generalization.

***

**License**

BLOOM is released under the Responsible AI License (RAIL), which is broadly permissive for research and commercial use but includes acceptable use provisions prohibiting specific harmful applications. This made BLOOM more accessible than LLaMA for commercial use at a time when most alternatives had stronger restrictions.

***

**Strengths**

* Largest openly accessible model at the time of release
* Exceptional multilingual coverage across 46 natural languages
* Fully open: model weights, training data, code, and methodology all publicly available
* Produced by genuinely diverse international research community
* Pioneered collaborative open-science LLM development at scale

**Limitations**

* 176B parameter size requires substantial GPU memory for deployment
* Performance on English-only benchmarks trails models of similar size trained primarily on English
* Architecture is now dated compared to more recent models with improved designs
* No longer a frontier model — significantly surpassed by newer releases
* RAIL license has nuances requiring review before certain commercial deployments