aisimulate分析(一)——算子workload建模

AISimulate

 AISimulate predicts LLM serving behavior and searches for strong deployment configurations offline, without bringing up a GPU serving cluster.

 AISimulate is the successor to the AIConfigurator (AIC) repository.

算子时延估计模式

DatabaseMode

pub enum DatabaseMode {
    Silicon,
    Hybrid,
    Empirical,
    Sol,
    /// Python-only per-call diagnostic retained on the wire for compatibility.
    /// Engine builders reject this as a database's default mode.
    SolFull,
}

SILICON (default, uses collected silicon data),

HYBRID (uses silicon data when available, otherwise SOL+empirical),

EMPIRICAL (SOL+empirical for all)

SOL (speed-of-light only) 使用roofline模型，根据计算量和仿存量估计算子运行时延。

roofline模型下的算子时延计算

gemm

query_gemm_table

gemm_sol_with_flops

/// Speed-of-light GEMM roofline components, from a pre-resolved `tc_flops`.
///
/// Mirrors Python's `GEMM._query_gemm_table::get_sol`:
/// - `sol_math = 2 * m * n * k / tc_flops * 1000`
/// - `sol_mem  = quant.memory * (m*n + m*k + n*k) / mem_bw * 1000`
/// - `sol      = max(sol_math, sol_mem)`
pub(crate) fn gemm_sol_with_flops(
    spec: &SystemSpec,
    quant: GemmQuantMode,
    tc_flops: f64,
    m: f64,
    n: f64,
    k: f64,
) -> SolComponents {
    let mapping = quant.mapping();
    let sol_math = 2.0 * m * n * k / tc_flops * 1000.0;
    let sol_mem = mapping.memory * (m * n + m * k + n * k) / spec.gpu.mem_bw * 1000.0;
    SolComponents::new(sol_math, sol_mem)
}

 ops=2.0 * m * n * k。GEMM 的总 FLOPs。m·n 是输出元素数，每个元素要做 k 次 MAC（乘加），每个 MAC = 2 FLOP。

 mem_bytes = mapping.memory * (m * n + m * k + n * k)。

输入：m * k + n * k。输出：m * n。

MoE

moe_sol

fn moe_sol(
    spec: &SystemSpec,
    quant: MoeQuantMode,
    num_gemms: u64,
    num_tokens: u32,
    hidden_size: u32,
    inter_size: u32,
    topk: u32,
    num_experts: u32,
    moe_tp_size: u32,
    moe_ep_size: u32,
    tc_flops: f64,
) -> SolComponents {
    let total_tokens = num_tokens as u64 * topk as u64;
    let moe_expert_compute = (moe_ep_size as u64).max(1);
    let moe_tp = (moe_tp_size as u64).max(1);
    let h = hidden_size as u64;
    let inter = inter_size as u64;
    let ne = num_experts as u64;

    let ops = total_tokens * h * inter * num_gemms * 2 / moe_expert_compute / moe_tp;
    let mem_bytes_int = total_tokens / moe_expert_compute * h * 2 // input + output
        + total_tokens / moe_expert_compute * inter * num_gemms / moe_tp // intermediate
        + h * inter * num_gemms / moe_tp
            * std::cmp::min(ne / moe_expert_compute, total_tokens / moe_expert_compute);
    let mem_bytes = (mem_bytes_int as f64) * quant.mapping().memory;

    // `tc_flops` is pre-resolved by the caller via `quant_tc_flops`
    // (strict per-dtype lookup; a missing `*_tc_flops` entry errors there).
    let sol_math = (ops as f64) / tc_flops * 1000.0;
    let sol_mem = mem_bytes / spec.gpu.mem_bw * 1000.0;
    SolComponents::new(sol_math, sol_mem)
}

ops（单卡浮点运算次数）计算原理

核心公式：单卡 FLOPs = 总激活Token数 × 隐藏维度 × 中间维度 × GEMM数量 × 2 ÷ EP ÷ TP

1. 基础矩阵乘法规则（* 2 的来源）：

   全连接层 Y = XW（形状 M,K × K,N）中，输出矩阵的每个元素需要执行 K 次乘法和 K 次加法。因此运算量为 2 × M × K × N。代码中的 hidden_size（K）和 inter_size（N）直接套用此公式。

2. MoE 结构叠加（num_gemms）：

   MoE 的 FFN 由多个全连接层堆叠：SwiGLU 包含 3 个 GEMM（Up, Gate, Down），ReLU 包含 2 个（FC1, FC2）。因此基础总运算量直接乘以 num_gemms。

3. 并行切分的数学映射（÷ EP ÷ TP）：

   除以 EP（专家并行）：Token 在专家间均匀切分，每个 GPU 只处理 1/EP 的 Token，运算量与 Token 数成正比，故直接除以 EP。

   除以 TP（张量并行）：权重矩阵沿 inter_size（输出维度 N）切分，每个 GPU 只计算该矩阵的 1/TP 列，运算量也相应除以 TP。

4. 忽略专家总数（num_experts 不参与运算）：

   这是 MoE 稀疏性的核心体现。计算量仅由被激活的路径决定，无论总专家数有多少，每个 Token 只经过 topk 个专家（已体现在 total_tokens 放大里）。1000 个 Token 均匀分布在 1000 个专家上，与 1000 个 Token 全部落在 1 个专家上的总运算量完全相等。

mem_bytes（单卡显存搬运字节数）计算原理

核心公式：单卡访存量 = 量化系数 × ( I/O激活 + 中间激活 + 稀疏化权重 )

1. 第一项：输入+输出激活（total_tokens/EP × hidden_size × 2）

   物理含义：每个 Token 进入专家前需读取 hidden_size 的输入向量，离开专家后需写回 hidden_size 的输出向量。并行：除以 EP（Token 切分）。乘以 2 代表读入和写回两次操作。

   第二项：中间层激活（total_tokens/EP × inter_size × num_gemms / TP）

2. 物理含义：在 SwiGLU 中，前两个 GEMM（Up/Gate）产生 T, I 的中间结果，第三个 GEMM（Down）需要读取该中间结果。总共涉及 num_gemms 次对 T, I 大小张量的读写访问。

   并行：除以 EP（Token 切分），再除以 TP（因为 TP 将 inter_size 维度切分到各卡）。

3. 第三项：模型权重（H × I × num_gemms / TP × min(专家总数/EP, Token总数/EP)）

   物理含义：这是 MoE 访存优化的精髓。权重的总参数量为 H × I × num_gemms。受 TP 影响，单卡存有 1/TP 的权重切片。

4. 量化系数（quant.mapping().memory）：

   以上三项统计的是"元素个数"，乘以此系数（如 FP16=2, INT8=1）才转化为真正的字节数。

Multihead Attention

query_generation_attention_table

generation_attention_sol

pub(crate) fn generation_attention_sol(
    spec: &SystemSpec,
    n_kv_lookup: u32,
    head_size: u32,
    window_size: u32,
    kv_quant: KvCacheQuantMode,
    n: f64,
    b: f64,
    s: f64,
    attn_flops: f64,
) -> SolComponents {
    let n_kv = if n_kv_lookup == 0 {
        n
    } else {
        n_kv_lookup as f64
    };
    let kv_len = if window_size > 0 {
        (s - 1.0).min(window_size as f64)
    } else {
        s - 1.0
    };
    let h = head_size as f64;
    let kv_mem = kv_quant.mapping().memory;
    let ops = 2.0 * b * n * h * 2.0 * kv_len;
    let mem_bytes = b * (n * h * 2.0 + 2.0 * n_kv * kv_len * h * kv_mem + n * h * 2.0);
    let sol_math = ops / attn_flops * 1000.0;
    let sol_mem = mem_bytes / spec.gpu.mem_bw * 1000.0;
    SolComponents::new(sol_math, sol_mem)
}

 ops 的计算： 2.0 * b * n * h * 2.0 * kv_len

• 2.0（第一个）：每个乘加（MAC）算 2 次浮点运算（1 次乘 + 1 次加）。
• b： batch size，并行处理的序列数。
• n： num attention heads。
• h： head size（每个头的维度 d_head）。
• 2.0（第二个）：attention 分两步 ------ (1) QK^T 打分，(2) 分数加权求和 V。

 mem_bytes的计算：b * (n * h * 2.0 + 2.0 * n_kv * kv_len * h * kv_mem + n * h * 2.0)

• 读 Q： n * h * 2.0, n 个头 × h 维 × 2 字节/元素（fp16）。
• 读 KV cache： 2.0 * n_kv * kv_len * h * kv_mem， 2.0 因为 K 和 V 各一份， kv_mem 是量化后每元素字节数（fp16=2，int8=1）。
• 写输出： n * h * 2.0，n 头 × h 维 × 2 字节（fp16）。

Multi-Head Latent Attention(MLA)

query_generation_mla_module_table

generation_mla_module_sol

/// Generation MLA module SOL in ms. Mirrors
/// `MLAModule._query_generation_mla_module_table::get_sol`: the generation
/// MLA SOL plus BMM pre+post terms folded into sol_math / sol_mem BEFORE the
/// max (NOT `max(attn) + max(bmm)`). `attn_flops` is the caller-resolved
/// decode-attention TC-FLOPS (`generation_attn_flops(spec, kv_quant)`);
/// `bmm_flops` the gemm-quant TC-FLOPS
/// (`quant_tc_flops(spec, gemm_quant.mapping())`):
/// - attn: `ops = 2*b*n*1088*s`, `mem = b*(n*1088*2 + (s-1)*576*kv.memory)`
/// - bmm:  `ops = 2*2*b*n*128*512`, `mem = 2*n*(b*640 + 128*512)*gemm.memory`
/// - `sol_math = attn_ops/attn_flops + bmm_ops/bmm_flops`
/// - `sol_mem  = (attn_mem + bmm_mem) / mem_bw`
/// - `sol      = max(sol_math, sol_mem)`
#[allow(clippy::too_many_arguments)]
pub(crate) fn generation_mla_module_sol(
    spec: &SystemSpec,
    kv_quant: KvCacheQuantMode,
    gemm_quant: GemmQuantMode,
    n: f64,
    b: f64,
    s: f64,
    attn_flops: f64,
    bmm_flops: f64,
) -> SolComponents {
    // MLA attention ops
    let attn_ops = 2.0 * b * n * 1088.0 * s;
    let mem_bytes = b * (n * 1088.0 * 2.0 + (s - 1.0) * 576.0 * kv_quant.mapping().memory);
    let mut sol_math = attn_ops / attn_flops * 1000.0;
    let mut sol_mem = mem_bytes / spec.gpu.mem_bw * 1000.0;
    // Add BMM pre + post SOL (same as query_mla_bmm)
    let bmm_ops = 2.0 * 2.0 * b * n * 128.0 * 512.0; // pre + post
    let bmm_mem = 2.0 * n * (b * 640.0 + 128.0 * 512.0) * gemm_quant.mapping().memory;
    let bmm_math = bmm_ops / bmm_flops * 1000.0;
    let bmm_mem_time = bmm_mem / spec.gpu.mem_bw * 1000.0;
    sol_math += bmm_math;
    sol_mem += bmm_mem_time;
    SolComponents::new(sol_math, sol_mem)
}

 在decode阶段，MLA计算采用矩阵吸收形式，本质上是multi query Attention。

 超细图解MLA计算流&吸收矩阵对比分析。

 Q的shape：b, n, kv_lora_rank + qk_rope_head_dim, n代表 num_attention_heads。

 K的shape：b, context_len, kv_lora_rank + qk_rope_head_dim

 V的shape：b, context_len, kv_lora_rank

 kv_lora_rank=512, qk_rope_head_dim=64。

QK^T的计算量：2 * b * n * (kv_lora_rank + qk_rope_head_dim) * context_len 。得到score的shape为：b, n, context_len。

 score 和 V的计算量：2 * b * n * context_len * kv_lora_rank。

 mqa_ops = 2 * b * n * context_len * (kv_lora_rank + qk_rope_head_dim + kv_lora_rank) = 2.0 * b * n * 1088.0 * context_len

 mqa_ops的计算可以同shallowsim相互印证。

 上面的推导同代码中的attn_ops就对上了。

 attn_ops的计算：2.0 * b * n * 1088.0 * s。

 MQA中，Q的输入dim为kv_lora_rank + qk_rope_head_dim = 576。

 参考MHA，mem_bytes = b * (n * 576 * 2.0 + (s - 1.0) * 576.0 * kv_quant.mapping().memory + n * 512 * 2.0)。

• 读 Q： n * 576 * 2.0 , n 个头 × h 维 × 2 字节/元素（fp16）。
• 写输出：n * 512 * 2.0，n 头 × h 维 × 2 字节（fp16）。

bmm_ops / bmm_mem 对应的是 Q 侧投影 + Attention 输出投影。

BMM pre代表q_absorb的计算，权重矩阵W shape, n, qk_nope_head_dim, kv_lora_rank。

BMM post代表out absorb的计算，权重矩阵 W shape, n, kv_lora_rank, v_head_dim。

qk_nope_head_dim = v_head_dim = 128。

bmm_ops = 2.0 * b * n * qk_nope_head_dim * kv_lora_rank + 2.0 * b * n * v_head_dim* kv_lora_rank。

逐矩阵算 mem（权重 + 激活读写）

q_absorb（pre）

权重: n · 128 · 512
激活: 读 [b,n,128] + 写 [b,n,512] = b·n·128 + b·n·512 = b·n·(128+512) = b·n·640
pre_mem = [ n·128·512 + b·n·640 ] · gemm.memory

out_absorb（post）

权重: n · 512 · 128
激活: 读 [b,n,512] + 写 [b,n,128] = b·n·512 + b·n·128 = b·n·640
post_mem = [ n·512·128 + b·n·640 ] · gemm.memory

deepseek v4 attention层的SOL时延估计

prefill (context)

query_context

decode (generation)

query_generation

dsv4_attention_sol_ms

pub(crate) fn dsv4_attention_sol_ms(
    spec: &SystemSpec,
    dims: &Dsv4SolDims,
    compress_ratio: i64,
    is_context: bool,
    kv_quant: KvCacheQuantMode,
    fmha_quant: FmhaQuantMode,
    gemm_quant: GemmQuantMode,
    b: i64,
    s: i64,
    prefix: i64,
    local_heads: i64,
    flops: Dsv4SolFlops,
) -> f64 {
    let local_o_groups = dims.local_o_groups.max(1);

    let (b, s, prefix) = (b as i128, s as i128, prefix as i128);
    let nh = local_heads as i128;
    let h = dims.hidden_size as i128;
    let qlr = dims.q_lora_rank as i128;
    let olr = dims.o_lora_rank as i128;
    let hd = dims.head_dim as i128;
    let rope_hd = dims.rope_head_dim as i128;
    let inh = dims.index_n_heads as i128;
    let ihd = dims.index_head_dim as i128;
    let topk = dims.index_topk as i128;
    let window = dims.window_size as i128;
    let cr = compress_ratio as i128;
    let lg = local_o_groups as i128; // Python local_groups = max(1, o_groups)

    let tokens = if is_context { b * s } else { b };
    let kv_len = if is_context {
        prefix + s
    } else {
        (s - 1).max(0)
    };

    let gemm_projection_ops = 2 * tokens * h * qlr
        + 2 * tokens * qlr * nh * hd
        + 2 * tokens * h * hd
        + 2 * tokens * lg * olr * h;
    let output_absorption_ops = 2 * tokens * nh * hd * olr;

    let compressor_mult: i128 = if cr == 4 { 2 } else { 1 };
    let mut compressor_ops: i128 = 0;
    if cr != 0 {
        compressor_ops = 4 * tokens * h * compressor_mult * hd + 2 * tokens * compressor_mult * hd;
        if cr == 4 {
            let indexer_compressor_mult: i128 = 2;
            compressor_ops += 4 * tokens * h * indexer_compressor_mult * ihd;
            compressor_ops += 2 * tokens * indexer_compressor_mult * ihd;
        }
    }

    let (window_pairs, compressed_pairs) = if is_context {
        let wp = causal_limited_pairs(b, s, prefix, window);
        let cp = if cr != 0 {
            let limit = if cr == 4 { topk } else { (kv_len / cr).max(0) };
            compressed_context_pairs(b, s, prefix, cr, limit)
        } else {
            0
        };
        (wp, cp)
    } else {
        let wp = b * kv_len.min(window);
        let cp = if cr != 0 {
            let limit = if cr == 4 { topk } else { (kv_len / cr).max(0) };
            b * (kv_len / cr).min(limit)
        } else {
            0
        };
        (wp, cp)
    };

    let attention_pairs = window_pairs + compressed_pairs;
    let attention_ops = 4 * nh * hd * attention_pairs;

    let mut indexer_ops: i128 = 0;
    let mut indexer_bfloat16_ops: i128 = 0;
    let mut indexer_cache_bytes: f64 = 0.0;
    if cr == 4 {
        let compressed_len = kv_len / cr;
        let indexer_query_tokens = if is_context { b * s } else { b };
        let indexer_pairs = indexer_query_tokens * compressed_len;
        indexer_ops = 2 * indexer_query_tokens * qlr * inh * ihd + 2 * indexer_pairs * inh * ihd;
        indexer_bfloat16_ops = 2 * indexer_query_tokens * h * inh;
        // Python: b * compressed_len * deepseek_v4_indexer_cache_entry_bytes(ihd)
        // where the entry is index_head_dim * 0.5 (FP4).
        indexer_cache_bytes = (b * compressed_len) as f64 * (dims.index_head_dim as f64 * 0.5);
    }

    let gemm_mem = gemm_quant.mapping().memory;
    let bf16_mem = GemmQuantMode::Bfloat16.mapping().memory;
    let mut gemm_weight_bytes = (h * qlr + qlr * nh * hd + h * hd + lg * olr * h) as f64 * gemm_mem;
    let mut bfloat16_weight_bytes = (nh * hd * olr) as f64 * bf16_mem;
    if cr != 0 {
        gemm_weight_bytes += (2 * h * compressor_mult * hd) as f64 * gemm_mem;
    }
    if cr == 4 {
        gemm_weight_bytes += (qlr * inh * ihd) as f64 * gemm_mem;
        bfloat16_weight_bytes += (h * inh) as f64 * bf16_mem;
    }

    let activation_bytes = (tokens * (h + qlr + nh * hd + hd + lg * olr)) as f64 * gemm_mem;
    let kv_cache_bytes = (attention_pairs * hd) as f64 * kv_quant.mapping().memory;
    let rope_bytes = (tokens * nh * rope_hd) as f64 * fmha_quant.mapping().memory;

    let sol_math = ((gemm_projection_ops + compressor_ops) as f64 / flops.gemm
        + (output_absorption_ops + indexer_bfloat16_ops) as f64 / flops.bf16
        + indexer_ops as f64 / flops.fp8
        + attention_ops as f64 / flops.attn)
        * 1000.0;
    let sol_mem = (gemm_weight_bytes
        + bfloat16_weight_bytes
        + activation_bytes
        + kv_cache_bytes
        + indexer_cache_bytes
        + rope_bytes)
        / spec.gpu.mem_bw
        * 1000.0;
    sol_math.max(sol_mem)
}

 此代码是从早期python代码修改来的。_deepseek_v4_attention_sol

def _deepseek_v4_attention_sol(
    database: PerfDatabase,
    *,
    is_context: bool,
    b: int,
    s: int,
    prefix: int,
    num_heads: int,
    hidden_size: int,
    q_lora_rank: int,
    o_lora_rank: int,
    head_dim: int,
    rope_head_dim: int,
    index_n_heads: int,
    index_head_dim: int,
    index_topk: int,
    window_size: int,
    compress_ratio: int,
    o_groups: int,
    kvcache_quant_mode: common.KVCacheQuantMode,
    fmha_quant_mode: common.FMHAQuantMode,
    gemm_quant_mode: common.GEMMQuantMode,
) -> tuple[float, float, float]:
    """Shared SOL formula for both context and generation phases.

    Verbatim port of the legacy ``PerfDatabase._deepseek_v4_attention_sol``
    body. Reads ``database.system_spec``, ``database._causal_limited_pairs``,
    ``database._compressed_context_pairs``, and ``GEMM._get_quant_tc_flops``.
    """
    from aiconfigurator_core.sdk.operations.gemm import GEMM

    def _tc_flops(quant_mode):
        return GEMM._get_quant_tc_flops(database.system_spec, quant_mode)

    tokens = b * s if is_context else b
    kv_len = prefix + s if is_context else max(0, s - 1)
    local_groups = max(1, o_groups)

    gemm_projection_ops = (
        2 * tokens * hidden_size * q_lora_rank
        + 2 * tokens * q_lora_rank * num_heads * head_dim
        + 2 * tokens * hidden_size * head_dim
        + 2 * tokens * local_groups * o_lora_rank * hidden_size
    )
    output_absorption_ops = 2 * tokens * num_heads * head_dim * o_lora_rank

    compressor_mult = 2 if compress_ratio == 4 else 1
    compressor_ops = 0.0
    if compress_ratio:
        compressor_ops = 4 * tokens * hidden_size * compressor_mult * head_dim
        compressor_ops += 2 * tokens * compressor_mult * head_dim
        if compress_ratio == 4:
            indexer_compressor_mult = 2
            compressor_ops += 4 * tokens * hidden_size * indexer_compressor_mult * index_head_dim
            compressor_ops += 2 * tokens * indexer_compressor_mult * index_head_dim

    if is_context:
        window_pairs = database._causal_limited_pairs(b, s, prefix, window_size)
        if compress_ratio:
            compressed_limit = index_topk if compress_ratio == 4 else max(0, kv_len // compress_ratio)
            compressed_pairs = database._compressed_context_pairs(b, s, prefix, compress_ratio, compressed_limit)
        else:
            compressed_pairs = 0
    else:
        window_pairs = b * min(kv_len, window_size)
        if compress_ratio:
            compressed_limit = index_topk if compress_ratio == 4 else max(0, kv_len // compress_ratio)
            compressed_pairs = b * min(kv_len // compress_ratio, compressed_limit)
        else:
            compressed_pairs = 0

    attention_pairs = window_pairs + compressed_pairs
    attention_ops = 4 * num_heads * head_dim * attention_pairs

    indexer_ops = 0.0
    indexer_bfloat16_ops = 0.0
    indexer_cache_bytes = 0.0
    if compress_ratio == 4:
        compressed_len = kv_len // compress_ratio
        if is_context:
            indexer_query_tokens = b * s
        else:
            indexer_query_tokens = b
        indexer_pairs = indexer_query_tokens * compressed_len
        indexer_ops = (
            2 * indexer_query_tokens * q_lora_rank * index_n_heads * index_head_dim
            + 2 * indexer_pairs * index_n_heads * index_head_dim
        )
        indexer_bfloat16_ops = 2 * indexer_query_tokens * hidden_size * index_n_heads
        indexer_cache_bytes = b * compressed_len * common.deepseek_v4_indexer_cache_entry_bytes(index_head_dim)

    gemm_weight_bytes = (
        hidden_size * q_lora_rank
        + q_lora_rank * num_heads * head_dim
        + hidden_size * head_dim
        + local_groups * o_lora_rank * hidden_size
    ) * gemm_quant_mode.value.memory
    bfloat16_weight_bytes = num_heads * head_dim * o_lora_rank * common.GEMMQuantMode.bfloat16.value.memory
    if compress_ratio:
        gemm_weight_bytes += 2 * hidden_size * compressor_mult * head_dim * gemm_quant_mode.value.memory
    if compress_ratio == 4:
        gemm_weight_bytes += q_lora_rank * index_n_heads * index_head_dim * gemm_quant_mode.value.memory
        bfloat16_weight_bytes += hidden_size * index_n_heads * common.GEMMQuantMode.bfloat16.value.memory

    activation_bytes = (
        tokens
        * (hidden_size + q_lora_rank + num_heads * head_dim + head_dim + local_groups * o_lora_rank)
        * gemm_quant_mode.value.memory
    )
    # DeepSeek-V4 attention uses compressed (MLA / MQA-equivalent) KV cache: each
    # cache entry stores a single ``head_dim``-sized vector that is shared by all
    # ``num_heads`` query heads (see ``DeepSeekV4Model.get_kvcache_bytes_per_sequence``
    # which derives storage as ``head_dim * kvcache_quant_mode.value.memory``).
    # The previous formula multiplied by ``num_heads``, which double-counted KV
    # traffic by a factor of ``num_heads`` (128x for DSv4-Pro). The inflated
    # ``sol_mem`` pinned SOL TPOT above HYBRID at non-trivial decode batch sizes,
    # reproducible with a single CLI invocation:
    #
    #   aiconfigurator cli estimate \
    #     --model-path deepseek-ai/DeepSeek-V4-Pro --system gb300 \
    #     --backend sglang --estimate-mode static_gen \
    #     --isl 8192 --osl 1024 --batch-size 570 --tp 1 --dp 12 --ep 12 \
    #     --database-mode HYBRID --detail all
    #
    # whose "Latency Summary" reported HYBRID at 99.3 ms vs SOL at 216.7 ms
    # (HYBRID ~2.2x faster than SOL) and the ``generation_attention`` op alone
    # at HYBRID 52.0 s vs SOL 197.8 s (3.8x violation), an impossible ordering
    # since SOL is the per-op roofline lower bound. The corrected formula reads
    # each KV entry once (pairs * head_dim), matching the storage layout and
    # the underlying kernel's MQA broadcast pattern.
    kv_cache_bytes = attention_pairs * head_dim * kvcache_quant_mode.value.memory
    rope_bytes = tokens * num_heads * rope_head_dim * fmha_quant_mode.value.memory

    sol_math = (
        (gemm_projection_ops + compressor_ops) / _tc_flops(gemm_quant_mode)
        + (output_absorption_ops + indexer_bfloat16_ops) / _tc_flops(common.GEMMQuantMode.bfloat16)
        + indexer_ops / _tc_flops(common.GEMMQuantMode.fp8)
        + attention_ops / _tc_flops(fmha_quant_mode)
    ) * 1000
    sol_mem = (
        (
            gemm_weight_bytes
            + bfloat16_weight_bytes
            + activation_bytes
            + kv_cache_bytes
            + indexer_cache_bytes
            + rope_bytes
        )
        / database.system_spec["gpu"]["mem_bw"]
        * 1000
    )
    return max(sol_math, sol_mem), sol_math, sol_mem

sol_math（单卡计算时延）的计算原理

 核心逻辑：总计算耗时 = ∑ \sum ∑ (各类算子的 FLOPs / 该类算子的峰值 TFLOPS)。

代码将不同精度的算子分开，因为 GPU 在不同精度（FP8、BF16、FP32）下的理论算力（_tc_flops）完全不同。

gemm_projection_ops

gemm_projection_ops = (
    2 * tokens * hidden_size * q_lora_rank
    + 2 * tokens * q_lora_rank * num_heads * head_dim
    + 2 * tokens * hidden_size * head_dim
    + 2 * tokens * local_groups * o_lora_rank * hidden_size
)

2 * tokens * hidden_size * q_lora_rank：Q 的降维投影（H -> Q_lora）。

2 * tokens * q_lora_rank * num_heads * head_dim：Q 的升维投影（Q_lora -> 多头维度）。

2 * tokens * hidden_size * head_dim：K/V 的原始投影（输入到压缩前的隐空间）。

2 * tokens * local_groups * o_lora_rank * hidden_size：输出 O 投影。

2的来源：输出矩阵的每个元素需要执行 K 次乘法和 K 次加法。

compressor_ops

compressor_mult = 2 if compress_ratio == 4 else 1
compressor_ops = 0.0
if compress_ratio:
    compressor_ops = 4 * tokens * hidden_size * compressor_mult * head_dim
    compressor_ops += 2 * tokens * compressor_mult * head_dim
    if compress_ratio == 4:
        indexer_compressor_mult = 2
        compressor_ops += 4 * tokens * hidden_size * indexer_compressor_mult * index_head_dim
        compressor_ops += 2 * tokens * indexer_compressor_mult * index_head_dim

deepseek v4 compressor 代码

kv = self.wkv(x)      # [b, s, coff * head_dim]
score = self.wgate(x) # [b, s, coff * head_dim]

 这两个 GEMM 贡献了 4 * tokens * hidden_size * mult * head_dim 的算力消耗（两个投影，每个含乘加）。

kv 同 score逐元素相乘，再求和

计算量：2 * tokens * compressor_mult * head_dim

1. prefill阶段

kv = (kv * score.softmax(dim=2)).sum(dim=2)

 张量形状：重塑后，kv 和 score 的形状均为 B, S/ratio, ratio, mult\*head_dim。

 步骤一（逐元素乘法）：kv * softmax(score)。

每个元素执行 1 次浮点乘法。总乘法次数 = (S/ratio) * ratio * (mult*head_dim) = S * mult * head_dim。

 步骤二（归约求和）：.sum(dim=2) 将 ratio 维度压缩。

每个输出元素需要 (ratio - 1) 次浮点加法，近似为 ratio 次，总加法次数 ≈ (S/ratio) * ratio * (mult*head_dim) = S * mult * head_dim。

 总计：乘法 + 加法 ≈ 2 * tokens * mult * head_dim。

2. decode 阶段

kv = (self.kv_state[:bsz] * self.score_state[:bsz].softmax(dim=1)).sum(dim=1, keepdim=True)

 kv_state:bsz：形状为 bsz, mult \* ratio, head_dim

 score_state:bsz：形状为 bsz, mult \* ratio, head_dim

 逐元素乘法计算量：bsz * (mult * ratio) * head_dim

 归约求和计算量：bsz * (mult * ratio - 1) * head_dim ≈ \approx ≈ bsz * (mult * ratio) * head_dim。

 总计：乘法 + 加法 ≈ 2 * bsz * (mult * ratio) * head_dim。

 每 ratio 步触发一次，平均到每步 = 2 * bsz * mult * head_dim。

deepseekv4使用两次Compressor，一次用于attention做kv压缩，一次在indexer做kv压缩。

Indexer（压缩比=4), 使用了compressor，因此：

if compress_ratio == 4:
     indexer_compressor_mult = 2
     compressor_ops += 4 * tokens * hidden_size * indexer_compressor_mult * index_head_dim
     compressor_ops += 2 * tokens * indexer_compressor_mult * index_head_dim

attention_ops

attention_pairs = window_pairs + compressed_pairs
attention_ops = 4 * num_heads * head_dim * attention_pair

 attention_pairs = window_pairs + compressed_pairs，即滑动窗口内的 Token 个数 + 压缩后的全局 Token 个数。

 标准 Flash Attention 的 FLOPs 约为 4 * 序列长度 * 头数 * 头维度（包含 QK 打分和 PV的乘加）。

测试命令

模型支持检查

aiconfigurator cli support --model-path deepseek-ai/DeepSeek-V3 --system h200_sxm

检查deepseek v4 pro

aiconfigurator cli support --model-path deepseek-ai/DeepSeek-V4-Pro --system h200_sxm --backend=vllm  --backend-version 0.24.0

检查deepseek v4 flash

aiconfigurator cli support --model-path deepseek-ai/DeepSeek-V4-Flash --system h200_sxm --backend=vllm  --backend-version 0.24.0

模型支持矩阵：https://ai-dynamo.github.io/aiconfigurator/support-matrix/

评估示例

aiconfigurator cli default \
  --model-path deepseek-ai/DeepSeek-V3 \
  --total-gpus 32 \
  --system h200_sxm \
  --backend vllm \
  --prefix 0 \
  --isl 2048 \
  --osl 512 \
  --ttft 1500 \
  --tpot 25 \
  --save-dir ./deepseek_eval_results

windows平台运行前，需要配置读取文件编码格式

set PYTHONIOENCODING=utf-8
set PYTHONUTF8=1

测试命令

aiconfigurator cli default --model-path deepseek-ai/DeepSeek-V3  --total-gpus 32   --system h200_sxm   --backend vllm   --prefix 0   --isl 2048   --osl 512   --ttft 1500   --tpot 25   --save-dir ./deepseek_eval_results

https://github.com/ai-dynamo/aiconfigurator/blob/v0.9.0/src/aiconfigurator/cli/api.py#L793

Reference

AIConfigurator: Lightning-Fast Configuration Optimization for Multi-Framework LLM Serving

AIConfigurator 深度解析：从算子数据库到 disagg 配置搜索的性能估算

Roofline Model与深度学习模型的性能分析