> For the complete documentation index, see [llms.txt](https://brindha.gitbook.io/mylearning/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://brindha.gitbook.io/mylearning/tools/wizardlm.md).

# WizardLM

#### WizardLM: Microsoft Research's Evol-Instruct Approach

**What is WizardLM?**

WizardLM is a family of instruction-following language models developed by Microsoft Research, first released in April 2023. Built on Meta's LLaMA base model, WizardLM introduced a novel automated instruction generation technique called Evol-Instruct — a method for systematically evolving simple instructions into more complex, diverse, and challenging variants. This approach addressed a recognized limitation of earlier instruction-tuned models: their training data, while large in volume, was often too simple and uniform to teach models to handle genuinely difficult tasks.

***

**Evol-Instruct Methodology**

The core innovation of WizardLM is Evol-Instruct, which uses a language model to iteratively rewrite and complicate existing instructions through a series of evolutionary steps. These steps include deepening instructions by adding more constraints, broadening them by requiring more knowledge, increasing their reasoning complexity, and generating entirely new instruction types. The result is a training dataset with dramatically higher diversity and difficulty than manually curated or simple self-instruct datasets. This automated process also scales well, producing high-quality instruction data at scale without proportional human effort.

***

**Model Versions**

WizardLM was initially released as a 7B model fine-tuned from LLaMA. Subsequent versions expanded to 13B and 70B sizes and transitioned to LLaMA 2 as the base model. WizardLM V1.1 was released in July 2023 with significantly improved performance. WizardLM-2 was released in April 2024 as a major update claiming state-of-the-art performance among open instruction-tuned models, available in 7B and 8x22B (Mixtral-based MoE) variants.

The WizardLM family also spawned specialized variants: WizardCoder, applying Evol-Instruct to code-focused data for code generation tasks, and WizardMath, applying the approach to mathematical problem-solving data.

***

**Performance**

WizardLM was trained on 2 trillion tokens of data and demonstrated strong performance on complex instruction-following tasks, outperforming Vicuna and approaching GPT-4 performance on certain benchmarks at the time of its release. The 70B variant was competitive with significantly larger and more expensive proprietary models. WizardLM's particular strength was on tasks requiring multi-step reasoning, complex constraints, and instruction adherence — precisely the areas that Evol-Instruct was designed to improve.

***

**License**

WizardLM models are released under a non-commercial license, restricting their use to research purposes.

***

**Strengths**

* Evol-Instruct is a genuinely novel and influential contribution to instruction data generation
* Strong performance on complex, multi-constraint tasks relative to model size
* Specialized variants for code and math extend applicability
* WizardLM-2 at 8x22B scale is highly capable for an open research model
* Automated and scalable instruction data generation methodology

**Limitations**

* Non-commercial license
* Evol-Instruct can occasionally produce instructions that are complex but poorly specified
* Specialized variants require separate models for different task domains
* Less emphasis on safety and alignment than some other model families
* No longer a frontier model relative to more recent releases