> For the complete documentation index, see [llms.txt](https://brindha.gitbook.io/mylearning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://brindha.gitbook.io/mylearning/tools/redpajama.md). # RedPajama #### RedPajama: Open Dataset and Models for Transparent LLM Development **What is RedPajama?** RedPajama is an open-science initiative launched in April 2023 by Together AI in collaboration with Ontocord.ai, ETH DS3Lab, Stanford CRFM, Hazy Research, and MILA Québec AI Institute. The project was motivated by a clear problem: Meta's LLaMA models had demonstrated powerful performance but carried non-commercial licenses, and the training data that produced them was not publicly available. RedPajama set out to reproduce the LLaMA training recipe with fully open, reproducible data — enabling anyone to train genuinely open-source models with complete transparency about what data they were trained on. *** **RedPajama-Data-1T** The first release was RedPajama-Data-1T — a 1.2 trillion token dataset following the exact recipe described in the LLaMA paper. It covered the same sources as LLaMA's training data: CommonCrawl (processed through the CCNet pipeline with quality filtering for Wikipedia-like pages), C4, GitHub, Wikipedia, Books, ArXiv, and StackExchange. All data preprocessing code and quality filters were released openly on GitHub, making the dataset fully reproducible from scratch. At approximately 5TB uncompressed, it was the largest fully open pretraining dataset at the time of release. *** **RedPajama-Data-V2** RedPajama-V2, released in October 2023, expanded dramatically in scope to 30 trillion tokens — at the time the largest publicly available dataset for LLM training. It was constructed from 84 CommonCrawl crawls and covered English, German, French, Spanish, and Italian. The key innovation was the inclusion of over 40 quality signal annotations per document — computed by multiple ML classifiers covering content quality, minhash signatures for fuzzy deduplication, and heuristic quality filters. These annotations allow developers to filter and weight the data according to their own quality criteria rather than receiving a single pre-filtered dataset. RedPajama-V2 has over 100 trillion tokens in total when counting annotated documents, with quality filtering reducing this to a high-quality subset. *** **RedPajama Models — RedPajama-INCITE** Alongside the datasets, Together trained and released a full suite of language models on the RedPajama-Data-1T dataset called RedPajama-INCITE (trained on the INCITE supercomputer at Oak Ridge National Laboratory). Available in 3B and 7B parameter sizes, both in base and instruction-tuned variants. The instruction-tuned models used hundreds of thousands of high-quality human instructions from the OpenChatKit project. All RedPajama-INCITE models were released under Apache 2.0 for commercial use. *** **Impact on the Ecosystem** RedPajama datasets have been adopted as training data components by major projects beyond the RedPajama models themselves. Snowflake Arctic, Salesforce XGen, and AI2's OLMo all incorporated RedPajama data. StableLM and MPT also used RedPajama data components. The project directly addressed academic exclusion from foundation model research — as Together's researchers noted, the combination of data scale and compute cost had effectively locked academic institutions out of training large models, and RedPajama's fully open data was designed to restore that access. *** **Strengths** * Fully reproducible dataset with published preprocessing code and quality filters * V2's 40+ quality annotations give developers granular control over data filtering * Apache 2.0 models for commercial use * Foundational dataset adopted by many major open LLM projects * Addresses the data transparency gap that LLaMA left open * Supports academic and community-based research at scale **Limitations** * RedPajama-INCITE models are not frontier models — surpassed by later generations * V2's 30T token scale requires significant infrastructure to process and use * Coverage in V2 limited to five European languages * Models less capable than contemporaries at comparable parameter counts * Less active development focus on the model side compared to the dataset side *** *** ### STABLELM #### StableLM: Stability AI's Open Language Model Family **What is StableLM?** StableLM is a family of open-source language models developed by Stability AI — the company best known for Stable Diffusion image generation — as part of its effort to extend open AI capabilities from images to language. Released beginning in April 2023, StableLM aimed to provide a commercially deployable, openly accessible foundation model for language tasks. Stability AI's entry into language modeling represented a significant expansion of its open-source AI portfolio. *** **StableLM Alpha (2023)** The initial StableLM-Alpha models were released in 3B and 7B parameter sizes, trained on a diverse data mix drawing from The Pile, RedPajama-Data, Falcon's RefinedWeb extract, and internal datasets — with web text sampled at a high rate (35% to 71% depending on the training stage). The second training stage incorporated StarCoder data for improved code capabilities. StableLM-Tuned-Alpha instruction-following variants were fine-tuned using a combination of five instruction datasets: Stanford Alpaca, GPT4All, ShareGPT52K, Databricks Dolly, and Anthropic's HH dataset. *** **StableLM-3B-4E1T** A technically significant variant trained on 1 trillion tokens for 4 epochs at 3B parameters, achieving state-of-the-art performance among 3B scale models at the time of its September 2023 release and competitive with many contemporary 7B models. It was trained on 256 NVIDIA A100 40GB GPUs and used the NeoX tokenizer with a vocabulary of 50,257 tokens. *** **StableLM 2 (2024)** StableLM 2 represented a significant quality improvement over the Alpha series. Available in 1.6B and 12B parameter sizes, StableLM 2 used an improved data mix adding multilingual content from the CulturaX dataset covering OSCAR corpora in multiple languages. It used the Arcade100k tokenizer with a larger vocabulary of 100,352 tokens optimized for multilingual content. The 12B model was trained on 384 NVIDIA H100 GPUs on AWS P5 instances, incorporating Flash Attention 2, SwiGLU activations, and Rotary Embeddings. *** **Architecture** All StableLM models use a decoder-only transformer architecture inspired by LLaMA's design, incorporating rotary positional embeddings, SwiGLU activations, RMSNorm, and Flash Attention. Training used a fork of EleutherAI's GPT-NeoX codebase with 2D data and tensor parallelism with ZeRO-1 optimizer state sharding. *** **StableVicuna** A notable derivative, StableVicuna, was produced by Stability AI's CarperAI team — an RLHF fine-tuned version of Vicuna-13B, which itself was a LLaMA fine-tune. StableVicuna was an early experiment in applying reinforcement learning from human feedback to open-source models, though it was limited to non-commercial use due to inheriting LLaMA's license restrictions. *** **License** StableLM models are released under the Stability AI Community License. Commercial use requires a commercial license from Stability AI, available through a membership arrangement, rather than unrestricted Apache 2.0 access. *** **Context** Stability AI went through significant corporate turbulence in 2023 and 2024, including the departure of its founder Emad Mostaque in March 2024 and subsequent restructuring. The StableLM project continued under new leadership but with reduced development momentum compared to the early 2023 period of rapid releases. *** **Strengths** * Early entrant in commercially-oriented open language models from a company known for open image generation * StableLM 2 improved multilingual coverage through CulturaX training data * Multiple size options from 1.6B to 12B covering edge to server deployment * Strong code capabilities from StarCoder data incorporation * Technically sound architecture with modern efficiency techniques **Limitations** * Commercial license requires Stability AI membership rather than unrestricted use * Corporate instability at Stability AI affected development continuity * Surpassed by more capable models at comparable parameter counts * Less ecosystem tooling and community support than models from larger organizations * StableLM 2 12B is the ceiling of the family — no very large model variant