【LLM】qwen2本地部署显存占用情况（base3080-12G）

不求上进的鱼仔2024-09-04 15:06

避免辣鸡网站隐藏后文，先上结论

model_name	memory
qwen2-7b-int8	11.6G
qwen2-7b-int4	8.5G
qwen2-1.5b	4.2G
qwen2-1.5b-int8	3G
qwen2-1.5b-int4	2.5G

btw: ollama部署的qwen2-1.5b只需要0.9G ，vllm需要4G，不知道是不是量化差异
btw: ollama部署qwen2-1.5b模型是0.9G，显存占用是2G，qwen2-7b模型4G，显存5G，在1070显卡也能跑，但是7b会比较慢

测试环境

windows11
python310
torch2.1
cuda12.1
显卡：RTX-3080-12G

模型来源
https://hf-mirror.com/

测试代码
参考的官方调用

python 复制代码

from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda"  # the device to load the model onto


model_path = 'path/to/your/model'
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype="auto",
    device_map="auto"
)
model = model.bfloat16()  # 解决量化模型报错
print('model ok')

tokenizer = AutoTokenizer.from_pretrained(model_path)
print('tokenizer ok')

prompt = "Give me a short introduction to large language model. response using chinese"
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
print(prompt)

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(device)

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)