运行调试大语言模型

ggaofeng2026-01-11 14:28

很多LLM是开源的，是可以自己下载模型，运行调试的。

下载模型：https://www.modelscope.cn/models/Qwen/Qwen3-0.6B/files

代码：上面只是模型的权重，词表等文件，代码是在transform库中的

首先，查找模型的class名称

bash 复制代码

Qwen3-8B# cat config.json 
{
  "architectures": [
    "Qwen3ForCausalLM"
  ],

然后在python 包中找这个类

bash 复制代码

(gaofeng1120) qwen3$ pwd
/home/gaofeng/anaconda3/envs/gaofeng1120/lib/python3.10/site-packages/transformers/models/qwen3
(gaofeng1120) qwen3$ ls
configuration_qwen3.py  __init__.py  modeling_qwen3.py  modular_qwen3.py  __pycache__
(gaofeng1120) qwen3$ cat modeling_qwen3.py |grep "class Qwen3ForCausalLM"
class Qwen3ForCausalLM(Qwen3PreTrainedModel, GenerationMixin):

调试：

例如使用Hugging Face的Transformers库, 在cpu下也可以调试

python 复制代码

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen3-0.6B"
model_name = "/home/qwen3-0.6B"
# load the tokenizer and the model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map="auto"
)

while True:
    prompt = "勾股定理有多少种证明方法？"
    #prompt = input("please input:")
    if prompt == 'exit':
        break

    messages = [
        {"role": "user", "content": prompt}
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
        enable_thinking=True  # Switches between thinking and non-thinking modes. Default is True.
    )
    model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

    # conduct text completion
    generated_ids = model.generate(
        **model_inputs,
        max_new_tokens=32768
    )
    output_ids = generated_ids[0][len(model_inputs.input_ids[0]):].tolist()

    # parsing thinking content
    try:
        # rindex finding 151668 (</think>)
        index = len(output_ids) - output_ids[::-1].index(151668)
    except ValueError:
        index = 0

    thinking_content = tokenizer.decode(output_ids[:index], skip_special_tokens=True).strip("\n")
    content = tokenizer.decode(output_ids[index:], skip_special_tokens=True).strip("\n")

    print("thinking content:", thinking_content)
    print("content:", content)