5. light wam 中video pre阶段

ZHOU_WUYI2026-08-01 16:43

总的video pre阶段

#mermaid-svg-RDWJ6uiEnX8GU3vN{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-RDWJ6uiEnX8GU3vN .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-RDWJ6uiEnX8GU3vN .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-RDWJ6uiEnX8GU3vN .error-icon{fill:#552222;}#mermaid-svg-RDWJ6uiEnX8GU3vN .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-RDWJ6uiEnX8GU3vN .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-RDWJ6uiEnX8GU3vN .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-RDWJ6uiEnX8GU3vN .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-RDWJ6uiEnX8GU3vN .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-RDWJ6uiEnX8GU3vN .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-RDWJ6uiEnX8GU3vN .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-RDWJ6uiEnX8GU3vN .marker{fill:#333333;stroke:#333333;}#mermaid-svg-RDWJ6uiEnX8GU3vN .marker.cross{stroke:#333333;}#mermaid-svg-RDWJ6uiEnX8GU3vN svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-RDWJ6uiEnX8GU3vN p{margin:0;}#mermaid-svg-RDWJ6uiEnX8GU3vN .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-RDWJ6uiEnX8GU3vN .cluster-label text{fill:#333;}#mermaid-svg-RDWJ6uiEnX8GU3vN .cluster-label span{color:#333;}#mermaid-svg-RDWJ6uiEnX8GU3vN .cluster-label span p{background-color:transparent;}#mermaid-svg-RDWJ6uiEnX8GU3vN .label text,#mermaid-svg-RDWJ6uiEnX8GU3vN span{fill:#333;color:#333;}#mermaid-svg-RDWJ6uiEnX8GU3vN .node rect,#mermaid-svg-RDWJ6uiEnX8GU3vN .node circle,#mermaid-svg-RDWJ6uiEnX8GU3vN .node ellipse,#mermaid-svg-RDWJ6uiEnX8GU3vN .node polygon,#mermaid-svg-RDWJ6uiEnX8GU3vN .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-RDWJ6uiEnX8GU3vN .rough-node .label text,#mermaid-svg-RDWJ6uiEnX8GU3vN .node .label text,#mermaid-svg-RDWJ6uiEnX8GU3vN .image-shape .label,#mermaid-svg-RDWJ6uiEnX8GU3vN .icon-shape .label{text-anchor:middle;}#mermaid-svg-RDWJ6uiEnX8GU3vN .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-RDWJ6uiEnX8GU3vN .rough-node .label,#mermaid-svg-RDWJ6uiEnX8GU3vN .node .label,#mermaid-svg-RDWJ6uiEnX8GU3vN .image-shape .label,#mermaid-svg-RDWJ6uiEnX8GU3vN .icon-shape .label{text-align:center;}#mermaid-svg-RDWJ6uiEnX8GU3vN .node.clickable{cursor:pointer;}#mermaid-svg-RDWJ6uiEnX8GU3vN .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-RDWJ6uiEnX8GU3vN .arrowheadPath{fill:#333333;}#mermaid-svg-RDWJ6uiEnX8GU3vN .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-RDWJ6uiEnX8GU3vN .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-RDWJ6uiEnX8GU3vN .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-RDWJ6uiEnX8GU3vN .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-RDWJ6uiEnX8GU3vN .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-RDWJ6uiEnX8GU3vN .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-RDWJ6uiEnX8GU3vN .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-RDWJ6uiEnX8GU3vN .cluster text{fill:#333;}#mermaid-svg-RDWJ6uiEnX8GU3vN .cluster span{color:#333;}#mermaid-svg-RDWJ6uiEnX8GU3vN div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-RDWJ6uiEnX8GU3vN .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-RDWJ6uiEnX8GU3vN rect.text{fill:none;stroke-width:0;}#mermaid-svg-RDWJ6uiEnX8GU3vN .icon-shape,#mermaid-svg-RDWJ6uiEnX8GU3vN .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-RDWJ6uiEnX8GU3vN .icon-shape p,#mermaid-svg-RDWJ6uiEnX8GU3vN .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-RDWJ6uiEnX8GU3vN .icon-shape .label rect,#mermaid-svg-RDWJ6uiEnX8GU3vN .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-RDWJ6uiEnX8GU3vN .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-RDWJ6uiEnX8GU3vN .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-RDWJ6uiEnX8GU3vN :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 3. Decode Tokens (_decode_video_tokens)
2. Backbone Forward (DiT)

  1. Build Video Pre (_build_video_pre)
    Inputs to _predict_video_only
    video_pre (dict)
    grid_size=(3, 7, 14)
    None
    False
    latents_video

shape=(2, 16, 3, 14, 28)
timestep_video

shape=(2,)
context

shape=(2, 129, 4096)
context_mask

shape=(2, 129)
action

shape=(2, 32, 7)
fuse_vae_embedding_in_latents

True
apply_spatial_downsample

False
restore_spatial_resolution

False
self._build_video_pre(...)

Patchify & Create Embeddings
tokens

shape=(2, 294, 1536)
freqs

shape=(294, 1, 64)
t

shape=(2, 294, 1536)
t_mod

shape=(2, 294, 6, 1536)
context

shape=(2, 129, 1536)
context_mask

shape=(2, 294, 129)
meta

grid_size, tokens_per_frame, batch_size
compression_meta

None
self.video_expert.forward_backbone(video_pre)
video_tokens

shape=(2, 294, 1536)
self._decode_video_tokens(...)

Unpatchify back to spatial latents
pred_video

shape=(2, 16, 3, 14, 28)
Return: pred_video

shape=(2, 16, 3, 14, 28)

  • 变化前：
bash 复制代码 
video_supervision_latents   shape=(2, 16, 3, 28, 56)
context                     shape=(2, 129, 4096)
context_mask                shape=(2, 129)
action                      shape=(2, 32, 7)
action_is_pad               shape=(2, 32)
image_is_pad                shape=(2, 9)
fuse_flag                   True
inputs['first_frame_latents'] shape=(2, 16, 1, 28, 56)
latents_video                 shape=(2, 16, 3, 14, 28)
target_video                  shape=(2, 16, 3, 14, 28)
  • 变化后：
bash 复制代码 
pred_video                    shape=(2, 16, 3, 14, 28)

_build_video_pre 阶段

#mermaid-svg-tPcCASN5kaRRRWQD{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-tPcCASN5kaRRRWQD .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-tPcCASN5kaRRRWQD .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-tPcCASN5kaRRRWQD .error-icon{fill:#552222;}#mermaid-svg-tPcCASN5kaRRRWQD .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-tPcCASN5kaRRRWQD .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-tPcCASN5kaRRRWQD .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-tPcCASN5kaRRRWQD .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-tPcCASN5kaRRRWQD .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-tPcCASN5kaRRRWQD .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-tPcCASN5kaRRRWQD .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-tPcCASN5kaRRRWQD .marker{fill:#333333;stroke:#333333;}#mermaid-svg-tPcCASN5kaRRRWQD .marker.cross{stroke:#333333;}#mermaid-svg-tPcCASN5kaRRRWQD svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-tPcCASN5kaRRRWQD p{margin:0;}#mermaid-svg-tPcCASN5kaRRRWQD .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-tPcCASN5kaRRRWQD .cluster-label text{fill:#333;}#mermaid-svg-tPcCASN5kaRRRWQD .cluster-label span{color:#333;}#mermaid-svg-tPcCASN5kaRRRWQD .cluster-label span p{background-color:transparent;}#mermaid-svg-tPcCASN5kaRRRWQD .label text,#mermaid-svg-tPcCASN5kaRRRWQD span{fill:#333;color:#333;}#mermaid-svg-tPcCASN5kaRRRWQD .node rect,#mermaid-svg-tPcCASN5kaRRRWQD .node circle,#mermaid-svg-tPcCASN5kaRRRWQD .node ellipse,#mermaid-svg-tPcCASN5kaRRRWQD .node polygon,#mermaid-svg-tPcCASN5kaRRRWQD .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-tPcCASN5kaRRRWQD .rough-node .label text,#mermaid-svg-tPcCASN5kaRRRWQD .node .label text,#mermaid-svg-tPcCASN5kaRRRWQD .image-shape .label,#mermaid-svg-tPcCASN5kaRRRWQD .icon-shape .label{text-anchor:middle;}#mermaid-svg-tPcCASN5kaRRRWQD .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-tPcCASN5kaRRRWQD .rough-node .label,#mermaid-svg-tPcCASN5kaRRRWQD .node .label,#mermaid-svg-tPcCASN5kaRRRWQD .image-shape .label,#mermaid-svg-tPcCASN5kaRRRWQD .icon-shape .label{text-align:center;}#mermaid-svg-tPcCASN5kaRRRWQD .node.clickable{cursor:pointer;}#mermaid-svg-tPcCASN5kaRRRWQD .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-tPcCASN5kaRRRWQD .arrowheadPath{fill:#333333;}#mermaid-svg-tPcCASN5kaRRRWQD .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-tPcCASN5kaRRRWQD .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-tPcCASN5kaRRRWQD .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-tPcCASN5kaRRRWQD .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-tPcCASN5kaRRRWQD .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-tPcCASN5kaRRRWQD .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-tPcCASN5kaRRRWQD .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-tPcCASN5kaRRRWQD .cluster text{fill:#333;}#mermaid-svg-tPcCASN5kaRRRWQD .cluster span{color:#333;}#mermaid-svg-tPcCASN5kaRRRWQD div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-tPcCASN5kaRRRWQD .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-tPcCASN5kaRRRWQD rect.text{fill:none;stroke-width:0;}#mermaid-svg-tPcCASN5kaRRRWQD .icon-shape,#mermaid-svg-tPcCASN5kaRRRWQD .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-tPcCASN5kaRRRWQD .icon-shape p,#mermaid-svg-tPcCASN5kaRRRWQD .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-tPcCASN5kaRRRWQD .icon-shape .label rect,#mermaid-svg-tPcCASN5kaRRRWQD .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-tPcCASN5kaRRRWQD .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-tPcCASN5kaRRRWQD .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-tPcCASN5kaRRRWQD :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Returned video_pre (dict)
video_expert.pre_dit
_build_video_pre
Inputs to _build_video_pre
Here apply_spatial_downsample=False

(already downsampled in caller)
seq_len / num_groups
tokens_per_frame
latents_video

shape=(2, 16, 3, 14, 28)
context

shape=(2, 129, 4096)
context_mask

shape=(2, 129)
action

shape=(2, 32, 7)
timestep_video
Spatial Downsample (if True)
latents_for_backbone

shape=(2, 16, 3, 14, 28)
compression_meta

None
Time Embedding

seperated_timestep + fuse_vae
t

shape=(2, 294, 1536)
t_mod

shape=(2, 294, 6, 1536)
Patchify

(patch_h=2, patch_w=2)
x_tokens

shape=(2, 294, 1536)
freqs (RoPE)

shape=(294, 1, 64)
meta: grid_size=(3, 7, 14)

tokens_per_frame=98

batch_size=2
Text Embedding

(Linear)
Action Embedding

(Linear + PosEmb)
Concat Text & Action
context

shape=(2, 129, 1536)
Expand Mask & Add Action Causal Mask
context_mask

shape=(2, 294, 129)
tokens

shape=(2, 294, 1536)
freqs

shape=(294, 1, 64)
t

shape=(2, 294, 1536)
t_mod

shape=(2, 294, 6, 1536)
context

shape=(2, 129, 1536)
context_mask

shape=(2, 294, 129)
meta

grid_size, tokens_per_frame...

  • 变化前：
bash 复制代码 
    video_supervision_latents   shape=(2, 16, 3, 28, 56)
    context                     shape=(2, 129, 4096)
    context_mask                shape=(2, 129)
    action                      shape=(2, 32, 7)
    action_is_pad               shape=(2, 32)
    image_is_pad                shape=(2, 9)
    fuse_flag                   True
    inputs['first_frame_latents'] shape=(2, 16, 1, 28, 56)
    latents_video                 shape=(2, 16, 3, 14, 28)
    target_video                  shape=(2, 16, 3, 14, 28)
  • 变化后：
bash 复制代码 
    tokens: shape=(2, 294, 1536)
    freqs: shape=(294, 1, 64)
    t: shape=(2, 294, 1536)
    t_mod: shape=(2, 294, 6, 1536)
    context: shape=(2, 129, 1536)
    context_mask: shape=(2, 294, 129)
    meta: dict with keys ['grid_size', 'tokens_per_frame', 'batch_size']

forward_backbone 阶段

#mermaid-svg-2abIXkDqMCKqjnFV{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-2abIXkDqMCKqjnFV .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-2abIXkDqMCKqjnFV .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-2abIXkDqMCKqjnFV .error-icon{fill:#552222;}#mermaid-svg-2abIXkDqMCKqjnFV .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-2abIXkDqMCKqjnFV .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-2abIXkDqMCKqjnFV .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-2abIXkDqMCKqjnFV .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-2abIXkDqMCKqjnFV .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-2abIXkDqMCKqjnFV .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-2abIXkDqMCKqjnFV .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-2abIXkDqMCKqjnFV .marker{fill:#333333;stroke:#333333;}#mermaid-svg-2abIXkDqMCKqjnFV .marker.cross{stroke:#333333;}#mermaid-svg-2abIXkDqMCKqjnFV svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-2abIXkDqMCKqjnFV p{margin:0;}#mermaid-svg-2abIXkDqMCKqjnFV .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-2abIXkDqMCKqjnFV .cluster-label text{fill:#333;}#mermaid-svg-2abIXkDqMCKqjnFV .cluster-label span{color:#333;}#mermaid-svg-2abIXkDqMCKqjnFV .cluster-label span p{background-color:transparent;}#mermaid-svg-2abIXkDqMCKqjnFV .label text,#mermaid-svg-2abIXkDqMCKqjnFV span{fill:#333;color:#333;}#mermaid-svg-2abIXkDqMCKqjnFV .node rect,#mermaid-svg-2abIXkDqMCKqjnFV .node circle,#mermaid-svg-2abIXkDqMCKqjnFV .node ellipse,#mermaid-svg-2abIXkDqMCKqjnFV .node polygon,#mermaid-svg-2abIXkDqMCKqjnFV .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-2abIXkDqMCKqjnFV .rough-node .label text,#mermaid-svg-2abIXkDqMCKqjnFV .node .label text,#mermaid-svg-2abIXkDqMCKqjnFV .image-shape .label,#mermaid-svg-2abIXkDqMCKqjnFV .icon-shape .label{text-anchor:middle;}#mermaid-svg-2abIXkDqMCKqjnFV .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-2abIXkDqMCKqjnFV .rough-node .label,#mermaid-svg-2abIXkDqMCKqjnFV .node .label,#mermaid-svg-2abIXkDqMCKqjnFV .image-shape .label,#mermaid-svg-2abIXkDqMCKqjnFV .icon-shape .label{text-align:center;}#mermaid-svg-2abIXkDqMCKqjnFV .node.clickable{cursor:pointer;}#mermaid-svg-2abIXkDqMCKqjnFV .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-2abIXkDqMCKqjnFV .arrowheadPath{fill:#333333;}#mermaid-svg-2abIXkDqMCKqjnFV .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-2abIXkDqMCKqjnFV .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-2abIXkDqMCKqjnFV .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2abIXkDqMCKqjnFV .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-2abIXkDqMCKqjnFV .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2abIXkDqMCKqjnFV .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-2abIXkDqMCKqjnFV .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-2abIXkDqMCKqjnFV .cluster text{fill:#333;}#mermaid-svg-2abIXkDqMCKqjnFV .cluster span{color:#333;}#mermaid-svg-2abIXkDqMCKqjnFV div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-2abIXkDqMCKqjnFV .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-2abIXkDqMCKqjnFV rect.text{fill:none;stroke-width:0;}#mermaid-svg-2abIXkDqMCKqjnFV .icon-shape,#mermaid-svg-2abIXkDqMCKqjnFV .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-2abIXkDqMCKqjnFV .icon-shape p,#mermaid-svg-2abIXkDqMCKqjnFV .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-2abIXkDqMCKqjnFV .icon-shape .label rect,#mermaid-svg-2abIXkDqMCKqjnFV .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-2abIXkDqMCKqjnFV .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-2abIXkDqMCKqjnFV .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-2abIXkDqMCKqjnFV :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Output
Transformer Blocks Loop
Initialization
Input: pre_state Dict
video_seq_len=294
Enter Layer
Hidden States
Perturbed or Unchanged
Next Layer
All Layers Finished
x_tokens

shape=(2, 294, 1536)
context_emb

shape=(2, 129, 1536)
t_mod

shape=(2, 294, 6, 1536)
freqs

shape=(294, 1, 64)
context_attn_mask

shape=(2, 294, 129)
meta

tokens_per_frame=98
self.reset_wam_adapter_cache()
build_video_to_video_mask()

if not bidirectional
self_attn_mask

None or Mask Tensor
Loop Condition
block(x_tokens, context_emb, t_mod, freqs, ...)

w/ or w/o gradient checkpointing
self.apply_wam_adapter(layer_idx, x_tokens)
Update x_tokens
return x_tokens

shape=(2, 294, 1536)

  • 变化前：
bash 复制代码 
    tokens: shape=(2, 294, 1536)
    freqs: shape=(294, 1, 64)
    t: shape=(2, 294, 1536)
    t_mod: shape=(2, 294, 6, 1536)
    context: shape=(2, 129, 1536)
    context_mask: shape=(2, 294, 129)
    meta: dict with keys ['grid_size', 'tokens_per_frame', 'batch_size']
  • 变化后：
bash 复制代码 
    video_tokens: shape=(2, 294, 1536)

_decode_video_tokens 阶段

#mermaid-svg-YSi577Wqp5lc4Ovl{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-YSi577Wqp5lc4Ovl .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-YSi577Wqp5lc4Ovl .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-YSi577Wqp5lc4Ovl .error-icon{fill:#552222;}#mermaid-svg-YSi577Wqp5lc4Ovl .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-YSi577Wqp5lc4Ovl .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-YSi577Wqp5lc4Ovl .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-YSi577Wqp5lc4Ovl .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-YSi577Wqp5lc4Ovl .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-YSi577Wqp5lc4Ovl .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-YSi577Wqp5lc4Ovl .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-YSi577Wqp5lc4Ovl .marker{fill:#333333;stroke:#333333;}#mermaid-svg-YSi577Wqp5lc4Ovl .marker.cross{stroke:#333333;}#mermaid-svg-YSi577Wqp5lc4Ovl svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-YSi577Wqp5lc4Ovl p{margin:0;}#mermaid-svg-YSi577Wqp5lc4Ovl .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-YSi577Wqp5lc4Ovl .cluster-label text{fill:#333;}#mermaid-svg-YSi577Wqp5lc4Ovl .cluster-label span{color:#333;}#mermaid-svg-YSi577Wqp5lc4Ovl .cluster-label span p{background-color:transparent;}#mermaid-svg-YSi577Wqp5lc4Ovl .label text,#mermaid-svg-YSi577Wqp5lc4Ovl span{fill:#333;color:#333;}#mermaid-svg-YSi577Wqp5lc4Ovl .node rect,#mermaid-svg-YSi577Wqp5lc4Ovl .node circle,#mermaid-svg-YSi577Wqp5lc4Ovl .node ellipse,#mermaid-svg-YSi577Wqp5lc4Ovl .node polygon,#mermaid-svg-YSi577Wqp5lc4Ovl .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-YSi577Wqp5lc4Ovl .rough-node .label text,#mermaid-svg-YSi577Wqp5lc4Ovl .node .label text,#mermaid-svg-YSi577Wqp5lc4Ovl .image-shape .label,#mermaid-svg-YSi577Wqp5lc4Ovl .icon-shape .label{text-anchor:middle;}#mermaid-svg-YSi577Wqp5lc4Ovl .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-YSi577Wqp5lc4Ovl .rough-node .label,#mermaid-svg-YSi577Wqp5lc4Ovl .node .label,#mermaid-svg-YSi577Wqp5lc4Ovl .image-shape .label,#mermaid-svg-YSi577Wqp5lc4Ovl .icon-shape .label{text-align:center;}#mermaid-svg-YSi577Wqp5lc4Ovl .node.clickable{cursor:pointer;}#mermaid-svg-YSi577Wqp5lc4Ovl .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-YSi577Wqp5lc4Ovl .arrowheadPath{fill:#333333;}#mermaid-svg-YSi577Wqp5lc4Ovl .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-YSi577Wqp5lc4Ovl .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-YSi577Wqp5lc4Ovl .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-YSi577Wqp5lc4Ovl .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-YSi577Wqp5lc4Ovl .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-YSi577Wqp5lc4Ovl .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-YSi577Wqp5lc4Ovl .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-YSi577Wqp5lc4Ovl .cluster text{fill:#333;}#mermaid-svg-YSi577Wqp5lc4Ovl .cluster span{color:#333;}#mermaid-svg-YSi577Wqp5lc4Ovl div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-YSi577Wqp5lc4Ovl .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-YSi577Wqp5lc4Ovl rect.text{fill:none;stroke-width:0;}#mermaid-svg-YSi577Wqp5lc4Ovl .icon-shape,#mermaid-svg-YSi577Wqp5lc4Ovl .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-YSi577Wqp5lc4Ovl .icon-shape p,#mermaid-svg-YSi577Wqp5lc4Ovl .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-YSi577Wqp5lc4Ovl .icon-shape .label rect,#mermaid-svg-YSi577Wqp5lc4Ovl .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-YSi577Wqp5lc4Ovl .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-YSi577Wqp5lc4Ovl .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-YSi577Wqp5lc4Ovl :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Output
Spatial Resolution Logic
self.video_expert.post_dit
Inputs to _decode_video_tokens
video_pre (Dict)
x

shape=(2, 294, 16*patch_h*patch_w)
grid_size=(f, h, w)
True (Return early)
False (Proceed)
Upsampled Tensor
video_tokens

shape=(2, 294, 1536)
t

shape=(2, 294, 1536)
meta: grid_size=(3, 14, 28)

※ Note: grid_size depends on patchify
compression_meta

None
restore_spatial_resolution

(False)
self.head(x_tokens, t)

Linear Projection
self.unpatchify(x, grid_size)

Reshape 1D sequence back to 3D spatial
pred_video

shape=(2, 16, 3, 14, 28)
if not restore_spatial_resolution:
self._restore_video_prediction_spatial_resolution(...)

Upsample feature map
Return: pred_video

shape=(2, 16, 3, 14, 28)

  • 变化前：
bash 复制代码 
    video_tokens: shape=(2, 294, 1536)
  • 变化后：
bash 复制代码 
    pred_video: shape=(2, 16, 3, 14, 28)
相关推荐
汇智信科16 分钟前
全域感知，协同智管 —— 智能矿山综合管控平台
人工智能·汇智信科
熊猫钓鱼>_>22 分钟前
免费域名的隐秘陷阱与网站稳定部署实战：火山引擎到底行不行？
人工智能·ai·llm·域名·火山引擎·引擎·火山
甲维斯25 分钟前
鬼故事 ！DeepSeek4.1 比Opus5 GPT5.6还强！
人工智能
xsd2024111826 分钟前
Pdf-Inspector开源工具：用Rust实现毫秒级PDF分类、文本提取与Markdown转换
人工智能
Yanjun2i26 分钟前
Agent学习记录五：Pydantic验证
人工智能·python·学习
找方案29 分钟前
AI视频生成对决：Sora vs 可灵 vs Veo，谁能定义未来
人工智能·机器学习·音视频
AI_Auto1 小时前
架构视角看数字化转型｜核心架构：大共享平台+小应用，从按需走向适变
大数据·人工智能·架构·制造
小蒋观天下1 小时前
社区AI智能摄像头完整选型指南
大数据·人工智能·安全·计算机视觉·语音识别·ai大模型
xsd202411181 小时前
雷达点云与海康摄像机数据融合全解析：从标定同步到工程实践的技术指南
人工智能
冬奇Lab1 小时前
DeepSeek Harness 系列（03）：工具系统——给 Agent 装上手
人工智能·deepseek
热门推荐
01国内可直接用、免费额度/永久免费的大模型API清单（含 SiliconFlow、火山、阿里、智谱、百度、Kimi、DeepSeek、DMXAPI 等）02如何新建文件夹？ 电脑新建文件夹的4种方法03GitHub 镜像站点04历年考研数学一、数学二、数学三真题试卷及答案PDF05蜘蛛磁力 搜索引擎大全，如何使用蜘蛛磁力查找磁力链接06微信历史版本含下载地址（ Windows PC | 安卓 | MAC ）及设置微信不更新072026 国产 AI 大模型横评：DeepSeek、通义千问、Kimi、文心一言、星火、豆包谁更能打？08ZCode 周末送额度活动开启：3 亿 Token 免费领取09【一等奖版】2026 国赛 A题 药材的烘干问题10Agnes AI 免费 API 接入指南：文本、生图、生视频，一套接口全免费