> For the complete documentation index, see [llms.txt](https://brindha.gitbook.io/mylearning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://brindha.gitbook.io/mylearning/tools/alpaca.md).

# Alpaca

Stanford Alpaca: Instruction-Following LLM Research at Minimal Cost

**What is Alpaca?**

Stanford Alpaca is an instruction-following language model released in March 2023 by researchers at Stanford University's Center for Research on Foundation Models (CRFM). It is a fine-tuned version of Meta's LLaMA 7B model, trained on 52,000 instruction-following examples generated by OpenAI's text-davinci-003 model using the Self-Instruct methodology. Alpaca demonstrated that capable instruction-following behavior could be produced with as little as $600 of compute — a landmark result that opened the door to an entire wave of low-cost open-source LLM fine-tuning.

***

**Training Approach**

The Self-Instruct method used to create Alpaca works by prompting a large, capable model — in this case text-davinci-003 — with a small set of seed instructions and asking it to generate new instruction-response pairs. Stanford generated 52,000 such examples at a cost of less than $500 using the OpenAI API, then used these to fine-tune LLaMA 7B through supervised instruction fine-tuning. Total fine-tuning compute cost was approximately $100, bringing the full project budget to around $600.

***

**Context Length and Architecture**

Alpaca used LLaMA's base architecture with a context length of 512 tokens — shorter than Vicuna's 2,048 tokens, which limited its ability to handle long conversations or documents. It was trained only on the 7B parameter LLaMA, without larger variants, prioritizing accessibility and reproducibility over maximum performance.

***

**Performance**

On qualitative benchmarks at the time of release, Alpaca performed comparably to OpenAI's text-davinci-003 despite being dramatically smaller and cheaper to produce. Subsequent models like Vicuna and WizardLM surpassed Alpaca's performance, but its contribution was not primarily about being the best model — it was about demonstrating the feasibility of low-cost instruction tuning, which became foundational to open-source LLM development in 2023.

***

**Research Significance**

Alpaca's primary contribution was conceptual and practical: it proved that the gap between general-purpose pretrained models and instruction-following assistants could be bridged cheaply and openly using synthetic data from stronger models. This sparked an enormous wave of similar projects including Vicuna, WizardLM, Koala, GPT4All, and many others. The Alpaca dataset and training code were released publicly, enabling the community to replicate and build upon the approach freely.

***

**License**

Like Vicuna, Alpaca is licensed for non-commercial research use only, inheriting restrictions from both the LLaMA base model license and terms around using OpenAI's API outputs for model training.

***

**Strengths**

* Historically pivotal: proved instruction tuning is achievable at minimal cost
* Fully documented training process, data, and code released openly
* Inspired the entire wave of 2023 open-source instruction-tuned models
* Simple and reproducible methodology accessible to academic researchers

**Limitations**

* Short 512-token context window
* Non-commercial license
* Surpassed quickly by more capable fine-tunes like Vicuna and WizardLM
* No longer actively developed
* Based on LLaMA 7B — smaller and weaker than subsequently released models