大模型量化简介

本文记录下学习大模型量化过程中的过程，本文不涉及高深的各种量化策略以及量化效果对比，只是记录对量化过程的探索。量化的本质上很简单，即将模型的存储数据从浮点转换为整数，从而提高文章的使用，这个过程中必然有数值的转换、模型保存、填充。但是在研究量化的过程中所时间超过了之前的LoRA，因为实际过程中有很多坑，光网上是不会理解的，具体来说，起有下面几个：

Transformer本身支持的量化方案，根本load_in_8bit参数需要依赖bitsandbytes，这个库在Windows上支持很不友好，也依赖GPU。
直接使用quanto库进行量化有问题，需要optimum-quanto
量化效果不显着，是因为很多量化通常只说量化Linear以及Norm层，很多层都不量化，比如用GPT2测试，自然有量化层不会量化
使用Quanto的quantize进行量化之后，还需要freeze才会将权重改成INT8类型

下面开始对量化的探索。

确定简介

大模型量化指的是高精度模型参数从高精度（比如FP32, 4字节）转换为低精度（比如INT8, 1字节）的方法。下面这张图（来自：Huggingface的量化基础）展示了量化的核心思想：

可以看到，确定之后使用的内存减少到了四分之一。

量化的核心就是怎么做这个映射，使得低精度的模型精度变得准确。这里只介绍最简单的精确量化absmax。下面的图（来自：大模型量化总结）展示了基本的思想，本质上就是将输入的梯度的绝对值映射到127上，从而将整个数据映射到-127到127之间，而原始的0映射到低其精度的0。

在反判断之后可以看到，精度有所损失。

手动测量

参考大语言模型量化原理和GPT-2量化实战中的示例，这里手动进行abxmax量化实验。

highlight 复制代码

<span style="color:#333333"><span style="background-color:#ffffff"><span style="color:#333333"><span style="background-color:#000000"><code>
<span style="color:#e28964">def</span><span style="color:#ffffff"> absmax_quantize</span><span style="color:#ffffff">(</span><span style="color:#ffffff">X</span><span style="color:#ffffff">):</span>
    <span style="color:#aeaeae"><em># 计算比例因子</em></span><span style="color:#ffffff">
    scale </span><span style="color:#ffffff">=</span> <span style="color:#3387cc">127</span> <span style="color:#ffffff">/</span><span style="color:#ffffff"> torch</span><span style="color:#ffffff">.</span><span style="color:#ffffff">max</span><span style="color:#ffffff">(</span><span style="color:#ffffff">torch</span><span style="color:#ffffff">.</span><span style="color:#ffffff">abs</span><span style="color:#ffffff">(</span><span style="color:#ffffff">X</span><span style="color:#ffffff">))</span>
    <span style="color:#aeaeae"><em># 量化</em></span><span style="color:#ffffff">
    X_quant </span><span style="color:#ffffff">=</span> <span style="color:#ffffff">(</span><span style="color:#ffffff">scale </span><span style="color:#ffffff">*</span><span style="color:#ffffff"> X</span><span style="color:#ffffff">).</span><span style="color:#ffffff">round</span><span style="color:#ffffff">()</span>
    <span style="color:#aeaeae"><em># 反量化</em></span><span style="color:#ffffff">
    X_dequant </span><span style="color:#ffffff">=</span><span style="color:#ffffff"> X_quant </span><span style="color:#ffffff">/</span><span style="color:#ffffff"> scale
    </span><span style="color:#e28964">return</span><span style="color:#ffffff"> X_quant</span><span style="color:#ffffff">.</span><span style="color:#ffffff">to</span><span style="color:#ffffff">(</span><span style="color:#ffffff">torch</span><span style="color:#ffffff">.</span><span style="color:#ffffff">int8</span><span style="color:#ffffff">),</span><span style="color:#ffffff"> X_dequant

</span><span style="color:#e28964">from</span><span style="color:#ffffff"> transformers </span><span style="color:#e28964">import</span> <span style="color:#89bdff">AutoModelForCausalLM</span><span style="color:#ffffff">,</span> <span style="color:#89bdff">AutoTokenizer</span>
<span style="color:#e28964">import</span><span style="color:#ffffff"> torch
torch</span><span style="color:#ffffff">.</span><span style="color:#ffffff">manual_seed</span><span style="color:#ffffff">(</span><span style="color:#3387cc">0</span><span style="color:#ffffff">)</span>

<span style="color:#aeaeae"><em># 设置设备为cpu</em></span><span style="color:#ffffff">
device </span><span style="color:#ffffff">=</span> <span style="color:#65b042">'cpu'</span>
<span style="color:#aeaeae"><em># 加载模型和分词器</em></span><span style="color:#ffffff">
model_id </span><span style="color:#ffffff">=</span> <span style="color:#65b042">'gpt2'</span><span style="color:#ffffff">
model </span><span style="color:#ffffff">=</span> <span style="color:#89bdff">AutoModelForCausalLM</span><span style="color:#ffffff">.</span><span style="color:#ffffff">from_pretrained</span><span style="color:#ffffff">(</span><span style="color:#ffffff">model_id</span><span style="color:#ffffff">).</span><span style="color:#ffffff">to</span><span style="color:#ffffff">(</span><span style="color:#ffffff">device</span><span style="color:#ffffff">)</span><span style="color:#ffffff">
tokenizer </span><span style="color:#ffffff">=</span> <span style="color:#89bdff">AutoTokenizer</span><span style="color:#ffffff">.</span><span style="color:#ffffff">from_pretrained</span><span style="color:#ffffff">(</span><span style="color:#ffffff">model_id</span><span style="color:#ffffff">)</span>

<span style="color:#aeaeae"><em># 打印模型大小</em></span>
<span style="color:#e28964">print</span><span style="color:#ffffff">(</span><span style="color:#ffffff">f</span><span style="color:#65b042">"模型的大小: {model.get_memory_footprint():,} bytes"</span><span style="color:#ffffff">)</span>


<span style="color:#aeaeae"><em># 提取第一个注意力层的权重</em></span><span style="color:#ffffff">
weights </span><span style="color:#ffffff">=</span><span style="color:#ffffff"> model</span><span style="color:#ffffff">.</span><span style="color:#ffffff">transformer</span><span style="color:#ffffff">.</span><span style="color:#ffffff">h</span><span style="color:#ffffff">[</span><span style="color:#3387cc">0</span><span style="color:#ffffff">].</span><span style="color:#ffffff">attn</span><span style="color:#ffffff">.</span><span style="color:#ffffff">c_attn</span><span style="color:#ffffff">.</span><span style="color:#ffffff">weight</span><span style="color:#ffffff">.</span><span style="color:#ffffff">data
</span><span style="color:#e28964">print</span><span style="color:#ffffff">(</span><span style="color:#65b042">"原始权重:"</span><span style="color:#ffffff">)</span>
<span style="color:#e28964">print</span><span style="color:#ffffff">(</span><span style="color:#ffffff">weights</span><span style="color:#ffffff">)</span>

<span style="color:#aeaeae"><em># 用 absmax 方法量化</em></span><span style="color:#ffffff">
weights_abs_quant</span><span style="color:#ffffff">,</span><span style="color:#ffffff"> _ </span><span style="color:#ffffff">=</span><span style="color:#ffffff"> absmax_quantize</span><span style="color:#ffffff">(</span><span style="color:#ffffff">weights</span><span style="color:#ffffff">)</span>
<span style="color:#e28964">print</span><span style="color:#ffffff">(</span><span style="color:#65b042">"\nAbsmax量化权重:"</span><span style="color:#ffffff">)</span>
<span style="color:#e28964">print</span><span style="color:#ffffff">(</span><span style="color:#ffffff">weights_abs_quant</span><span style="color:#ffffff">)</span>
</code></span></span></span></span>

下面的输出可以看到，模型已经从坐标转换为低精度了。

下面直接使用GPT2进行量化：

highlight 复制代码

<span style="color:#333333"><span style="background-color:#ffffff"><span style="color:#333333"><span style="background-color:#000000"><code>
<span style="color:#e28964">import</span><span style="color:#ffffff"> numpy </span><span style="color:#e28964">as</span><span style="color:#ffffff"> np
</span><span style="color:#e28964">from</span><span style="color:#ffffff"> copy </span><span style="color:#e28964">import</span><span style="color:#ffffff"> deepcopy

</span><span style="color:#aeaeae"><em># 保存原始权重</em></span><span style="color:#ffffff">
weights </span><span style="color:#ffffff">=</span> <span style="color:#ffffff">[</span><span style="color:#ffffff">param</span><span style="color:#ffffff">.</span><span style="color:#ffffff">data</span><span style="color:#ffffff">.</span><span style="color:#ffffff">clone</span><span style="color:#ffffff">()</span> <span style="color:#e28964">for</span><span style="color:#ffffff"> param </span><span style="color:#e28964">in</span><span style="color:#ffffff"> model</span><span style="color:#ffffff">.</span><span style="color:#ffffff">parameters</span><span style="color:#ffffff">()]</span>

<span style="color:#aeaeae"><em># 创建量化模型副本</em></span><span style="color:#ffffff">
model_abs </span><span style="color:#ffffff">=</span><span style="color:#ffffff"> deepcopy</span><span style="color:#ffffff">(</span><span style="color:#ffffff">model</span><span style="color:#ffffff">)</span>

<span style="color:#aeaeae"><em># 量化所有模型权重</em></span><span style="color:#ffffff">
weights_abs </span><span style="color:#ffffff">=</span> <span style="color:#ffffff">[]</span>
<span style="color:#e28964">for</span><span style="color:#ffffff"> param </span><span style="color:#e28964">in</span><span style="color:#ffffff"> model_abs</span><span style="color:#ffffff">.</span><span style="color:#ffffff">parameters</span><span style="color:#ffffff">():</span><span style="color:#ffffff">
    _</span><span style="color:#ffffff">,</span><span style="color:#ffffff"> dequantized </span><span style="color:#ffffff">=</span><span style="color:#ffffff"> absmax_quantize</span><span style="color:#ffffff">(</span><span style="color:#ffffff">param</span><span style="color:#ffffff">.</span><span style="color:#ffffff">data</span><span style="color:#ffffff">)</span><span style="color:#ffffff">
    param</span><span style="color:#ffffff">.</span><span style="color:#ffffff">data </span><span style="color:#ffffff">=</span><span style="color:#ffffff"> dequantized
    weights_abs</span><span style="color:#ffffff">.</span><span style="color:#ffffff">append</span><span style="color:#ffffff">(</span><span style="color:#ffffff">dequantized</span><span style="color:#ffffff">)</span><span style="color:#ffffff">

weights </span><span style="color:#ffffff">=</span><span style="color:#ffffff"> model</span><span style="color:#ffffff">.</span><span style="color:#ffffff">transformer</span><span style="color:#ffffff">.</span><span style="color:#ffffff">h</span><span style="color:#ffffff">[</span><span style="color:#3387cc">0</span><span style="color:#ffffff">].</span><span style="color:#ffffff">attn</span><span style="color:#ffffff">.</span><span style="color:#ffffff">c_attn</span><span style="color:#ffffff">.</span><span style="color:#ffffff">weight</span><span style="color:#ffffff">.</span><span style="color:#ffffff">data
</span><span style="color:#e28964">print</span><span style="color:#ffffff">(</span><span style="color:#65b042">"原始权重:"</span><span style="color:#ffffff">)</span>
<span style="color:#e28964">print</span><span style="color:#ffffff">(</span><span style="color:#ffffff">weights</span><span style="color:#ffffff">)</span><span style="color:#ffffff">

weights1 </span><span style="color:#ffffff">=</span><span style="color:#ffffff"> model_abs</span><span style="color:#ffffff">.</span><span style="color:#ffffff">transformer</span><span style="color:#ffffff">.</span><span style="color:#ffffff">h</span><span style="color:#ffffff">[</span><span style="color:#3387cc">0</span><span style="color:#ffffff">].</span><span style="color:#ffffff">attn</span><span style="color:#ffffff">.</span><span style="color:#ffffff">c_attn</span><span style="color:#ffffff">.</span><span style="color:#ffffff">weight</span><span style="color:#ffffff">.</span><span style="color:#ffffff">data
</span><span style="color:#e28964">print</span><span style="color:#ffffff">(</span><span style="color:#65b042">"量化之后的权重:"</span><span style="color:#ffffff">)</span>
<span style="color:#e28964">print</span><span style="color:#ffffff">(</span><span style="color:#ffffff">weights1</span><span style="color:#ffffff">)</span>
</code></span></span></span></span>

注意absmax_quantize返回值的第二部分是校准后反校准得到的精度的值，与原始的精度值会有一些区别，可以看到使用校准后的INT8类型再反校准得到的精度与原始精度有一些误差。

使用Quanto库测量

下面研究使用 Quanto 库进行量化

highlight 复制代码

<span style="color:#333333"><span style="background-color:#ffffff"><span style="color:#333333"><span style="background-color:#000000"><code><span style="color:#ffffff">
pip install optimum</span><span style="color:#ffffff">-</span><span style="color:#ffffff">quanto
</span></code></span></span></span></span>

量化并保存量化之后的模型，这里用Qwen2-0.5B-Instruct测试。

highlight 复制代码

<span style="color:#333333"><span style="background-color:#ffffff"><span style="color:#333333"><span style="background-color:#000000"><code>
<span style="color:#e28964">from</span><span style="color:#ffffff"> transformers </span><span style="color:#e28964">import</span> <span style="color:#89bdff">AutoModelForCausalLM</span>
<span style="color:#e28964">from</span><span style="color:#ffffff"> optimum</span><span style="color:#ffffff">.</span><span style="color:#ffffff">quanto </span><span style="color:#e28964">import</span> <span style="color:#89bdff">QuantizedModelForCausalLM</span><span style="color:#ffffff">,</span><span style="color:#ffffff"> qint8

model </span><span style="color:#ffffff">=</span> <span style="color:#89bdff">AutoModelForCausalLM</span><span style="color:#ffffff">.</span><span style="color:#ffffff">from_pretrained</span><span style="color:#ffffff">(</span><span style="color:#65b042">'Qwen/Qwen2-0.5B-Instruct'</span><span style="color:#ffffff">)</span><span style="color:#ffffff">
qmodel </span><span style="color:#ffffff">=</span> <span style="color:#89bdff">QuantizedModelForCausalLM</span><span style="color:#ffffff">.</span><span style="color:#ffffff">quantize</span><span style="color:#ffffff">(</span><span style="color:#ffffff">model</span><span style="color:#ffffff">,</span><span style="color:#ffffff"> weights</span><span style="color:#ffffff">=</span><span style="color:#ffffff">qint8</span><span style="color:#ffffff">,</span><span style="color:#ffffff"> exclude</span><span style="color:#ffffff">=</span><span style="color:#65b042">'lm_head'</span><span style="color:#ffffff">)</span><span style="color:#ffffff">
qmodel</span><span style="color:#ffffff">.</span><span style="color:#ffffff">save_pretrained</span><span style="color:#ffffff">(</span><span style="color:#65b042">'./Qwen2-0.5B-Instruct-quantized'</span><span style="color:#ffffff">)</span>
</code></span></span></span></span>

加载评估模型，并对比输出。

highlight 复制代码

<span style="color:#333333"><span style="background-color:#ffffff"><span style="color:#333333"><span style="background-color:#000000"><code>
<span style="color:#e28964">from</span><span style="color:#ffffff"> optimum</span><span style="color:#ffffff">.</span><span style="color:#ffffff">quanto </span><span style="color:#e28964">import</span> <span style="color:#89bdff">QuantizedModelForCausalLM</span>
<span style="color:#e28964">from</span><span style="color:#ffffff"> transformers </span><span style="color:#e28964">import</span> <span style="color:#89bdff">AutoTokenizer</span><span style="color:#ffffff">,</span> <span style="color:#89bdff">AutoModelForCausalLM</span><span style="color:#ffffff">

tokenizer </span><span style="color:#ffffff">=</span> <span style="color:#89bdff">AutoTokenizer</span><span style="color:#ffffff">.</span><span style="color:#ffffff">from_pretrained</span><span style="color:#ffffff">(</span><span style="color:#65b042">"Qwen/Qwen2-0.5B-Instruct"</span><span style="color:#ffffff">)</span><span style="color:#ffffff">

model </span><span style="color:#ffffff">=</span> <span style="color:#89bdff">AutoModelForCausalLM</span><span style="color:#ffffff">.</span><span style="color:#ffffff">from_pretrained</span><span style="color:#ffffff">(</span><span style="color:#65b042">'Qwen/Qwen2-0.5B-Instruct'</span><span style="color:#ffffff">)</span><span style="color:#ffffff">
qmodel </span><span style="color:#ffffff">=</span> <span style="color:#89bdff">QuantizedModelForCausalLM</span><span style="color:#ffffff">.</span><span style="color:#ffffff">from_pretrained</span><span style="color:#ffffff">(</span><span style="color:#65b042">'./Qwen2-0.5B-Instruct-quantized'</span><span style="color:#ffffff">)</span><span style="color:#ffffff">

input_text </span><span style="color:#ffffff">=</span> <span style="color:#65b042">"Hello, who are you? "</span><span style="color:#ffffff">
input_ids </span><span style="color:#ffffff">=</span><span style="color:#ffffff"> tokenizer</span><span style="color:#ffffff">(</span><span style="color:#ffffff">input_text</span><span style="color:#ffffff">,</span><span style="color:#ffffff"> return_tensors</span><span style="color:#ffffff">=</span><span style="color:#65b042">"pt"</span><span style="color:#ffffff">).</span><span style="color:#ffffff">input_ids

outputs </span><span style="color:#ffffff">=</span><span style="color:#ffffff"> qmodel</span><span style="color:#ffffff">.</span><span style="color:#ffffff">generate</span><span style="color:#ffffff">(</span><span style="color:#ffffff">input_ids</span><span style="color:#ffffff">)</span>
<span style="color:#e28964">print</span><span style="color:#ffffff">(</span><span style="color:#ffffff">tokenizer</span><span style="color:#ffffff">.</span><span style="color:#ffffff">decode</span><span style="color:#ffffff">(</span><span style="color:#ffffff">outputs</span><span style="color:#ffffff">[</span><span style="color:#3387cc">0</span><span style="color:#ffffff">]))</span><span style="color:#ffffff">


outputs2 </span><span style="color:#ffffff">=</span><span style="color:#ffffff"> model</span><span style="color:#ffffff">.</span><span style="color:#ffffff">generate</span><span style="color:#ffffff">(</span><span style="color:#ffffff">input_ids</span><span style="color:#ffffff">)</span>
<span style="color:#e28964">print</span><span style="color:#ffffff">(</span><span style="color:#ffffff">tokenizer</span><span style="color:#ffffff">.</span><span style="color:#ffffff">decode</span><span style="color:#ffffff">(</span><span style="color:#ffffff">outputs2</span><span style="color:#ffffff">[</span><span style="color:#3387cc">0</span><span style="color:#ffffff">]))</span>

</code></span></span></span></span>

这里我们先看看量化之后在模型文件中是怎么提现的，然后分析量化模型的加载过程。从大小来看，数量确实小了，但也没有小多少，因为量化之后的文件还要存放额外的数据。

注意，量化之后的目录里面还有个quanto_qmap.json，这个文件保存了各个权重的配置，或者使用的量化方案。

打开量化前后的model.saftensors文件，可以看到，量化前是BF16，量化后等于了I8

可以看到权重的dtype变了，而且所在的位置也不一样，一个直接是xxx.weight，一个是xxx.weight._data。那为啥模型的大小没有小多少呢，可以看到量化之后的模型文件有很多的input_scale和output_scale，这个是在加载量化模型的时候需要的。当然这只是量化这个库的行为，其他的量化方案可能采用不同的存储方案。