> For the complete documentation index, see [llms.txt](https://brindha.gitbook.io/mylearning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://brindha.gitbook.io/mylearning/tools/tinyllama.md).

# TinyLlama

**What it is**

TinyLlama is a compact 1.1B language model pre-trained on around 1 trillion tokens for 3 epochs. It is built on the architecture and tokenizer of Llama 2 and leverages various advances contributed by the open-source community. 

**Who built it**

TinyLlama was developed by Associate Professor Lu Wei of Singapore University of Technology and Design, along with research assistant Zhang Peiyuan and PhD students Zeng Guangtao and Wang Tianduo. 

**Architecture**

TinyLlama is not merely a downscale of Llama 2. Its architecture is deliberately set up with 22 transformer layers, a hidden dimension of 2048, and 32 attention heads, incorporating FlashAttention-2 and Grouped-Query Attention for efficiency. 

**Key strengths**

The 4-bit quantized TinyLlama-1.1B's weight takes up only 637 MB, making it deployable on edge devices with restricted memory and computational capacities, enabling functionalities like real-time machine translation without an internet connection. 

**Use cases**

Use cases include low-resource language research, educational tools in settings with limited computing access, content generation for small businesses, and prototyping and experimentation for developers without access to high-powered computing resources. 

**License:** Apache 2.0, allowing both commercial and non-commercial use.