> For the complete documentation index, see [llms.txt](https://brindha.gitbook.io/mylearning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://brindha.gitbook.io/mylearning/tools/smollm.md).

# SmolLM

**What it is**

SmolLM is a family of state-of-the-art small language models developed by Hugging Face, aimed at delivering high performance with lower computational costs and improved user privacy. The three models in the SmolLM family are available in 135M, 360M, and 1.7B parameters. 

**Training data**

The training corpus combines Cosmopedia v2 for synthetic textbook content, Python-Edu for programming knowledge, and FineWeb-Edu with 220 billion tokens of deduplicated educational web content. Dataset deduplication removes repetitive or near-identical content, giving SmolLM broader topic coverage than raw web data of equivalent size. 

**Architecture**

The smaller 135M and 360M models incorporate Grouped-Query Attention and prioritize depth over width in their architecture, while the larger 1.7B parameter model uses a more traditional design. 

**Performance**

Despite being trained on fewer tokens, SmolLM-135M surpassed MobileLM-125M, the current best model with less than 200M parameters. SmolLM-360M and SmolLM-1.7B outperformed all other models with less than 500M and 2B parameters respectively. 

**SmolLM2 advances**

The SmolLM2 1.7B variant demonstrates significant advances over its predecessor, particularly in instruction following, knowledge, reasoning, and mathematics. It was trained on 11 trillion tokens using a diverse dataset combination including FineWeb-Edu, DCLM, and The Stack, along with new mathematics and coding datasets. 

**Latest generation: SmolLM3**

SmolLM3 represents the latest generation with advanced capabilities including dual-mode reasoning, support for 6 languages (English, French, Spanish, German, Italian, and Portuguese), and extended context length up to 128K tokens. 

**Multimodal extension**

SmolVLM combines SmolLM2-1.7B with vision processing capabilities to handle arbitrary sequences of image and text inputs for multimodal tasks including visual question answering, image description, and visual storytelling.