The two-phase process behind LLMs’ responses

This relatively easier post will be the opportunity to warm up by getting back to the basics of the Transformer architecture and text generation using Transformer-based decoders. Most importantly, I will establish the vocabulary I will use throughout the series. I highlight in bold the terms I personally favor. You will in particular learn about the two phases of text generation: the initiation phase and the generation (or decoding) phase.

First, a little Transformer refresher. For simplicity, let's assume that we process a single sequence at a time (i.e. batch size is 1). In the figure below I pictured the main layers of a vanilla Transformer-based decoder (Figure 1) used to generate an output token from a sequence of input tokens.

Figure 1 ---Outline of a Transformer decoder model

Notice that the decoder itself does not output tokens but logits (as many as the vocabulary size). By the way, the last layer outputting the logits is often called the language model head or LM head . Deriving the token from the logits is the job of a heuristic called (token) search strategy, generation strategy or decoding strategy. Common decoding strategies include:

  • Greedy decoding which simply consists of picking the token with the largest logit, possibly after altering the logits using transformations such as a repetition penalty.
  • Sampling decoding which consists of using the logits as a multinomial distribution to sample from. In other words, we pick a token from the vocabulary by sampling. The distribution we sample from can first be warped using simple transformations such as temperature scaling, top-k and top-p to mention the most well known.
  • More complex heuristics such as beam search, contrastive decoding¹, etc.

For the sake of simplicity, we will assume the decoding strategy to be part of the model (Figure 2). This mental model is actually useful in the context of LLM serving solutions where such entities that take a sequence of tokens as input and return a corresponding output token are usually called an execution engine or an inference engine.

Figure 2 --- An overly simplified Transformer decoder model

And what about generating more than one token? Generating text (commonly named completion ) from an input text sequence (commonly named prompt) using a Transformer-based decoder basically consists of the following steps:

  1. Loading the model weights to GPU

  2. Tokenizing the prompt on CPU and transferring the token tensor to GPU (Figure 3)

Figure 3 --- Tokenization step

  1. Generating the first token of the completion by running the tokenized prompt through the network.

This single-step phase is typically called the initiation phase . In the next post, we will see that it is also often called the pre-fill phase.

  1. Appending the generated token to the sequence of input tokens and using it as a new input to generate the second token of the completion. Then, repeat this process until either a stop sequence has been generated (e.g. a single end-of-sequence (EOS) token) or the configured maximum sequence length has been reached (Figure 4).

This multi-step phase is usually called the generation phase , the decoding phase , the auto-regressive phase or even the incremental phase.

Both step 3 and 4 are illustrated in the figure below (Figure 4).

Figure 4 --- Initiation and decoding phases of the token generation process

  1. Fetching the the completion's tokens to CPU and detokenize them to get your generated text (Figure 5).

Figure 5 --- Detokenization step

Notice: Recent and more advanced techniques aiming at achieving lower latency such as speculative sampling² or lookahead decoding³ don't exactly follow the simple algorithm described above.

At that point you should be either disappointed, confused or both. You could ask me: so what is the actual difference between the initiation phase and the decoding phase? It seems artificial at best at this point. The initiation phase feels indeed as special as the initialization step of a while loop and we essentially do the same in both phases: on each iteration we apply a forward pass to a sequence of tokens which gets one token larger every time.

You would actually be right. At that point, there is indeed no difference on how the computations are run on the hardware and therefore nothing special about either phase in that regard. However, and as we will see in the next post, this setup involves expensive computations (scaling quadratically in the total sequence length), a lot of which being actually and fortunately redundant. An obvious way to alleviate this is to cache what we could spare recomputing. This optimization is known as KV caching and introduces this critical difference I keep hinting at. See you on the next post!

1: A Contrastive Framework for Neural Text Generation (Su et al., 2022)

2: Fast Inference from Transformers via Speculative Decoding (Leviathan et al., 2022)

3: Breaking the Sequential Dependency of LLM Inference Using Lookahead Decoding (Fu et al. 2023)

Llm

Large Language Models

相关推荐
user44651179179120 分钟前
Agent 也能「过安检」:AgentScope 权限系统设计拆解
人工智能
烟小栀34 分钟前
Windows下的深度学习环境配置参考及配置教程
人工智能
tellmewhoisi1 小时前
深度学习0:数学知识:最大似然推导过程
人工智能·深度学习·概率论
oil欧哟1 小时前
从产品经理视角拆解 DeepSeek Super-agent:双层路由与异步调度架构
人工智能
张张123y1 小时前
个人助手型 Agent 的六个工程细节:Hermes 与 OpenClaw 的审批、委派、上下文与调度
网络·人工智能·开源·后端开发·ai agent
Dawson Zhu2 小时前
【模型量化⑦】W4A8与GPTQ-Int4:极限压缩与MLC-LLM部署实战
人工智能·语言模型·架构·aigc·agi
千寻技术帮2 小时前
50055_基于AI的互帮互助小程序(源码+文档+数据库脚本+部署教程)
数据库·人工智能·小程序·毕业设计·springboot·课程设计
天树星智源2 小时前
KV 剪枝的同一个误差,模型确定时只改 1.4% 的 token,不确定时改 65%:接上不确定性,AIME25 端到端快 1.27 倍
人工智能·大模型
AI英德西牛仔2 小时前
PC 用户的效率革命:腾讯元宝导出 Word 指令与“AI 导出鸭”的底层逻辑
人工智能·word·excel·deepseek·ai导出鸭
Flynn Liang3 小时前
多智能体协同不再复杂:Redis Stream 消费者组、幂等与故障恢复
数据库·人工智能·redis·缓存·ai agent·智能体