今天我们从cursor手搓cursor进入下一个篇章，开始向泛化的agent去开发agent，进一步开始研究agent，向RSI进发。
首先想把llama.cpp的技术都熟悉一遍。于是挑了一个相对好写一些的语言golang，准备针对性地单独跑模型；现在用得最多的快速模型还是q wen3.6:35b，那就写一个引擎运行这个模型吧。本来打算用gemini直接写，这货给我写了个request来了，用if然后生成template返回static string！于是用qwen3.8:flash-next让它从0开始抄llama.cpp和ds4的代码，手搓metal和golang，大概搓了两天，先把bge-m3和embedding跑通了，qwen3.6:35b只是开了个头。之后勉强用gemini 3.8这种也比较弱的模型fix了一些收尾调试验证工作，于是qwen3.6:35b MXFP4 (qwen35moe)最终是跑起来了。
一开始我以为MXFP4和Q4都很纯粹，让它只实现Q4, MXFP4, BF16，结果跑不起来。实现了gguf的dump以后，各个层可能用的量化都不一样，所以要从Q4, Q5, Q6, Q8, ...全部要实现一下。最初的时候是只用 GEMV，速度是 prefill 20 token/s, generate 20 token/s。跑了下llama.cpp，多少？prefill 400 token/s, generate 55 token/s。这差的有点多了吧，于是让gemini读llama.cpp的代码，它改进了下prefill和generate都能到40 token/s。但是到这里似乎就卡住了。于是正好测下自己的agent，用llama.cpp运行的qwen3.6:35b去读我们的代码和llama.cpp的代码，让它分析下瓶颈在哪里。附录给出了它的结论，虽然比较粗，但是还是可以说明些问题的，之后当然是把这个结论直接交给qwen3.8:flash-next，让它边读代码，边改进；于是它将一部分GEMV实现变成了GEMM的batch实现，这个时候prefill变成了80 token/s，generate却减到了30 token/s。看来metal的调试后面是要自己参与学习和深入调优了。现在的引擎还没到能彻底实战的时候。
下面的计划是，死磕一段时间performance，把prefill和generate都提高到和llama.cpp差不多的水平，之后开始研究下hook LoRA训练一下router+专项动态激活LoRA，主要也是想沿着engram的思路，尝试从参数层面探索外置知识。
其实整个大模型的世界可以归结为两种递归收敛：一个是训练过程，或者程序员更熟悉的compile time，在训练过程中，使用loss，让它收敛到一个比较泛化的智能水准。第二种当然是runtime的收敛，就是比如我用thinking，一直think一直产生tokens，这些tokens会将整个vector拉向一个方向从而加强某些方面的信号；agent或者说harness/loop其实也是这个思路，在runtime使用逐步的形式加强某些方面的信号，让它收敛到我们预期的水准输出。这个就是大模型使用的控制。所以我们要compile time和runtime都体验下，感觉下下一个方向是什么。现在的大模型思路太快了，比如我们写代码，我们是可以问问题然后一步一步说我要写什么函数如何实现整体功能，大模型目前是看到目标，直接想一步就完成，最后造成眼高手低的崩坏。
附录：
Performance Analysis: Golang Metal vs. llama.cpp Metal
The significant performance disparity between your Golang implementation (~40 t/s for both prefill and generation) and llama.cpp (~400 t/s prefill, ~55 t/s generation) is caused by inefficient GPU kernel design in the Golang Metal shaders. Specifically, the Golang implementation suffers from low GPU occupancy , lack of shared memory optimization , and inefficient kernel dispatch strategies.
Below is the detailed technical breakdown of the bottlenecks in both phases.
1. Generation Phase (40 t/s vs. 55 t/s)
Primary Bottleneck: Low GPU Occupancy and Inefficient GEMV Kernels
Golang Implementation (metal/shaders/40_gemv.metal)
- Tiny Thread Groups: The GEMV kernel uses a thread group size of 32 threads to compute only 1 output row. Apple Silicon (M-series) GPUs have wide SIMD widths (16 lanes). A 32-thread group only utilizes 2 SIMD groups, leaving the majority of the GPU's compute units idle.
- No Shared Memory: The kernel does not use
threadgroupmemory to cache weights or inputs. Every weight chunk and input vector element is fetched from global memory for every row, saturating memory bandwidth. - Inefficient Reduction: It relies solely on
simd_sumwithin the small 32-thread group, which does not effectively aggregate results across the larger GPU chip.
llama.cpp Implementation (local/llama.cpp/ggml/src/ggml-metal/kernels/mul_mv.metal)
- High Occupancy: llama.cpp uses large thread groups configured via function constants (e.g., processing
NR0rows simultaneously with multiple SIMD groups). This allows hundreds of threads per kernel launch, maximizing GPU utilization. - Shared Memory Reduction: It uses
threadgroupmemory (shmem) and a dedicated reduction helper (helper_mv_reduce_and_write) to cache intermediate results and efficiently reduce across SIMD groups, minimizing global memory writes. - Vectorized Dequantization: It employs optimized inline dequantization functions that use bitwise operations and
selectinstructions, which are much faster and more branch-efficient than the generic dequantization calls in the Golang code.
2. Prefill Phase (40 t/s vs. 400 t/s)
Primary Bottleneck: Single-Query Dispatch and Lack of KV Tiling
Golang Implementation (metal/shaders/60_attn.metal)
- Single-Query Kernel Dispatch: The attention kernel computes one (query head, query token) pair per thread group launch. For a prefill sequence of NNN tokens and HHH heads, this results in N×HN \times HN×H separate kernel launches. This creates massive kernel launch overhead and serialization bottlenecks.
- Low Occupancy: Like the generation phase, it uses only 32 threads per thread group, resulting in extremely low GPU occupancy.
- No KV Cache Tiling: While the Query (Q) is loaded into shared memory, the Key (K) and Value (V) matrices are accessed directly from global memory in a sequential loop (
for (int j = 0; j < kv_len; ++j)). This leads to strided, uncoalesced global memory accesses and poor bandwidth utilization. - No Quantization Support: The kernel assumes float inputs, bypassing any potential optimizations for quantized KV caches (GGUF format) that llama.cpp handles efficiently.
llama.cpp Implementation (local/llama.cpp/ggml/src/ggml-metal/kernels/fa.metal)
- Modular and Fused Kernels: llama.cpp uses a modular pipeline with separate, optimized kernels for dequantization (
kernel_flash_attn_ext_kv_f16), padding (kernel_flash_attn_ext_pad), and mask scanning (kernel_flash_attn_ext_blk). This allows specialized processing for each step. - Shared Memory Tiling: The main attention kernel (
kernel_flash_attn_ext_impl) extensively usesthreadgroupmemory (shmem_f16) to tile Q, K, V, and intermediate results. This minimizes global memory accesses and maximizes memory bandwidth reuse. - High Occupancy Block Processing: It processes blocks of queries and KV cache items simultaneously in large thread groups, ensuring high GPU occupancy and efficient parallelism.
- Quantization Optimization: It has native, efficient support for GGUF quantized types (Q4_K, Q5_K, etc.) with specialized dequantization kernels that feed directly into the attention calculation.
Summary of Key Differences
| Feature | Golang Implementation | llama.cpp | Impact |
|---|---|---|---|
| Thread Group Size | 32 threads (Very Low) | 256+ threads (High) | Major: Golang leaves GPU cores idle. |
| Occupancy | Low (1-4 rows per group) | High (Multiple rows/blocks) | Major: Poor utilization of M-series silicon. |
| Shared Memory | None for KV/Weights | Extensive Tiling (Q,K,V,O,S) | Major: Golang saturates memory bandwidth; llama.cpp hides latency. |
| Dispatch Strategy | Per-Token/Per-Row (Many launches) | Batched/Block-based (Few launches) | High: Golang has high CPU/GPU kernel launch overhead. |
| Dequantization | Generic, branch-heavy | Bitwise, SIMD-optimized | Moderate: llama.cpp processes quantized data faster. |
| Attention Algo | Sequential KV access | Tiled KV access | High: llama.cpp maximizes bandwidth reuse during prefill. |
Recommendation
To bridge the performance gap, the Golang implementation needs to be refactored to match llama.cpp's Metal shader architecture:
- Increase Thread Group Sizes: Increase thread groups from 32 to at least 256+ threads to improve occupancy.
- Implement Shared Memory Tiling: Use
threadgroupmemory to cache K and V blocks during attention and weights during GEMV. - Batch Kernel Dispatch: Change the prefill kernel to process multiple tokens/heads per launch rather than one-by-one.
- Optimize Dequantization: Adopt bitwise and SIMD-optimized dequantization techniques similar to llama.cpp.