ollama 直接运行 gguf 模型踩坑细节（io timeout以及System message...）

视频链接：https://www.bilibili.com/video/BV1o7Kz6REES/?vd_source=5ba34935b7845cd15c65ef62c64ba82f

仓库链接：https://github.com/LitchiCheng/DGX-Spark

在 ollama 上有一些模型没有，需要在hf或者modelscope上下载，但只支持gguf，因为底层是llamacpp，过程中碰到如下两个问题，可供大家参考。

pull 时出现 i/o timeout

复制代码

ollama pull hf.co/CodeFault/Nvidia-Qwen3.6-27B-NVFP4-GGUF

出现

pulling 7e77ebeecf77: 100% ▕██████████████████████████████████████████████▏ 17 GB

Error: max retries exceeded: Get "https://huggingface.co/v2/CodeFault/Nvidia-Qwen3.6-27B-NVFP4-GGUF/blobs/sha256:48ed978f59cbc824c41ec7c78ba1f7ae0e52bc0c9286b95b7067d56fdef4df04?__sign=eyJhbGciOiJFZERTQSIsImtpZCI6Im1sR1pFYktIVHRVSGRhX1RRdEczQ0N1S3k5ME9Qd25DNHY5elRjM3FVYzgifQ.eyJyZWFkIjp0cnVlLCJwZXJtaXNzaW9ucyI6eyJyZXBvLmNvbnRlbnQucmVhZCI6dHJ1ZX0sImlhdCI6MTc4Mzk1MTQxOCwianRpIjoiMGUzMTZmZjgtMjZiMS00OTVlLTlkZDQtYmU5M2I1ZmQ5MDExIiwic3ViIjoiL0NvZGVGYXVsdC9OdmlkaWEtUXdlbjMuNi0yN0ItTlZGUDQtR0dVRiIsImV4cCI6MTc4Mzk1MjAxOCwiaXNzIjoiaHR0cHM6Ly9odWdnaW5nZmFjZS5jbyJ9.7ue22DbG7bBcK_8i5iqtkt2u_fRAZ2EsL1W07dRnSe_xAuDwcrg0chP2bkX08ckCs-J4AmM60O98VVjeLVwqBw": dial tcp $2a03:2880:f130:83:face:b00c:0:25de$ :443: i/o timeout

运行出现Error: Jinja Exception: System message

API Error: 400 {"error":{"code":400,"message":"Unable to generate parser for this template. Automatic parser generation failed: \n------------\nWhile executing CallExpression at line 85, column 32 in source:\n...first %}↵ {{- raise_exception('System message must be at the beginnin...\n ^\nError: Jinja Exception: System message must be at the beginning.","type":"invalid_request_error"}}

https://github.com/lmstudio-ai/lmstudio-bug-tracker/issues/1999

使用如下指令测试下，看下问题是否出在模板上

复制代码

curl -s http://localhost:11434/api/chat -d '{
"model": "qwen3.6-27b-nvfp4-fixed",
"messages": [
{"role": "system", "content": "初始系统提示"},
{"role": "user", "content": "你好"},
{"role": "system", "content": "这是中间插入的系统消息"},
{"role": "user", "content": "请回答"}
],
"stream": false
}' | jq '.message.content // .error'
"{"error":{"code":400,"message":"Unable to generate parser for this template. Automatic parser generation failed: \n------------\nWhile executing CallExpression at line 85, column 32 in source:\n...first %}↵            {{- raise_exception('System message must be at the beginnin...\n                                           ^\nError: Jinja Exception: System message must be at the beginning.","type":"invalid_request_error"}}"

可以看到出现System message限制的提示，接下来就是需要修改modelfile，首先查看当前ollama下载的位置

复制代码

ollama show hf.co/mudler/Qwen3.6-35B-A3B-NVFP4-GGUF:latest --modelfile

B-NVFP4-GGUF:latest --modelfile

Modelfile generated by "ollama show"

To build a new Modelfile based on this, replace FROM with:

FROM hf.co/mudler/Qwen3.6-35B-A3B-NVFP4-GGUF:latest

FROM /usr/share/ollama/.ollama/models/blobs/sha256-1690d0424e232527b8bb135a38033e4699ad11817677eebacd40349020faea52

TEMPLATE {{ .Prompt }}

创建如下 patch_template.py，移除system message限制

复制代码

#!/usr/bin/env python3
"""极简版：移除 Qwen GGUF 中的 system message 限制"""

import sys
import re
from pathlib import Path

# 确保安装了 gguf
try:
    from gguf import GGUFReader, GGUFWriter, GGUFValueType, Keys
except ImportError:
    import subprocess
    subprocess.run([sys.executable, "-m", "pip", "install", "gguf"], check=True)
    from gguf import GGUFReader, GGUFWriter, GGUFValueType, Keys


def patch_gguf(input_path: Path, output_path: Path = None):
    input_path = Path(input_path)
    if output_path is None:
        output_path = input_path.with_suffix('.fixed.gguf')
    else:
        output_path = Path(output_path)
    
    # 读取
    reader = GGUFReader(str(input_path), 'r')
    arch = reader.get_field(Keys.General.ARCHITECTURE).contents()
    
    # 提取并修改模板
    old_template = reader.get_field(Keys.Tokenizer.CHAT_TEMPLATE).contents()
    
    # 移除 raise_exception
    new_template = re.sub(
        r"{{-?\s*raise_exception\s*\(\s*['\"]System message must be at the beginning.*?['\"]\s*\)\s*}}",
        "{# system check removed #}",
        old_template,
        flags=re.IGNORECASE
    )
    
    # 也处理 if 包裹的版本
    new_template = re.sub(
        r"{%\s*if\s+.*?%\s*}{{-?\s*raise_exception\s*\([^}]*\)\s*}}.*?{%\s*endif\s*%}",
        "",
        new_template,
        flags=re.IGNORECASE | re.DOTALL
    )
    
    print(f"原模板长度: {len(old_template)}")
    print(f"新模板长度: {len(new_template)}")
    
    # 写入新文件
    writer = GGUFWriter(str(output_path), arch=arch, endianess=reader.endianess)
    
    # 复制所有字段，跳过旧 template
    for field in reader.fields.values():
        if field.name in (Keys.General.ARCHITECTURE,) or field.name.startswith('GGUF.'):
            continue
        if field.name.startswith(Keys.Tokenizer.CHAT_TEMPLATE):
            continue
            
        val_type = field.types[0]
        sub_type = field.types[-1] if val_type == GGUFValueType.ARRAY else None
        writer.add_key_value(field.name, field.contents(), val_type, sub_type=sub_type)
    
    # 添加新 template
    writer.add_chat_template(new_template)
    
    # 复制张量
    for tensor in reader.tensors:
        writer.add_tensor_info(
            tensor.name, tensor.data.shape, tensor.data.dtype,
            tensor.data.nbytes, tensor.tensor_type
        )
    
    # 写入
    writer.write_header_to_file()
    writer.write_kv_data_to_file()
    writer.write_ti_data_to_file()
    for tensor in reader.tensors:
        writer.write_tensor_data(tensor.data, tensor_endianess=reader.endianess)
    writer.close()
    
    print(f"✓ 已保存到: {output_path}")
    return output_path


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("用法: python patch_qwen.py input.gguf [output.gguf]")
        sys.exit(1)
    
    patch_gguf(sys.argv[1], sys.argv[2] if len(sys.argv) > 2 else None)

创建好如上的脚本后，执行如下指令

复制代码

python patch_template.py Nvidia-Qwen3.6-27B-NVFP4-GGUF.gguf Nvidia-Qwen3.6-27B-NVFP4-GGUF-fixed.gguf

# 然后创建模型 FROM 修改后的文件
cat > Modelfile << 'EOF'
FROM /home/litchi/dev/hf-model/Nvidia-Qwen3.6-27B-NVFP4-GGUF-fixed.gguf
EOF

ollama create Nvidia-Qwen3.6-27B-NVFP4-GGUF-fixed -f Modelfile