关于Transformer的理解

关于Transformer, QKV的意义表示其更像是一个可学习的查询系统,或许以前搜索引擎的算法就与此有关或者某个分支的搜索算法与此类似。


Can anyone help me to understand this image? - #2 by J_Johnson - nlp - PyTorch Forums

Embeddings - these are learnable weights where each token(token could be a word, sentence piece, subword, character, etc) are converted into a vector, say, with 500 values between 0 and 1 that are trainable.

Positional Encoding - for each token, we want to inform the model where it's located, orderwise. This is because linear layers are not ideal for handling sequential information. So we manually pass this in by adding a vector of sine and cosine values on the first 2 elements in the embedding vector.

This sequence of vectors goes through an attention layer, which basically is like a learnable digitized database search function with keys, queries and values. In this case, we are "searching" for the most likely next token.

The Feed Forward is just a basic linear layer, but is applied across each embedding in the sequence separately(i.e. 3 dim tensor instead of 2 dim).

Then the final Linear layer is where we want to get out our predicted next token in the form of a vector of probabilities, which we apply a softmax to put the values in the range of 0 to 1.

There are two sides because when that diagram was developed, it was being used in language translations. But generative language models for next token prediction just use the Transformer decoder and not the encoder.

Here is a PyTorch tutorial that might help you go through how it works.

Language Modeling with nn.Transformer and torchtext --- PyTorch Tutorials 2.0.1+cu117 documentation


相关推荐
Csvn9 小时前
同一个问题问 100 遍,钱就扣 100 次?精确缓存让它只扣一次(C02)
人工智能·aigc·agent
罗西的思考9 小时前
【轻量化复刻大模型 / KIMI K3】Multi-Token Prediction(MTP / EAGLE-3)(2)
人工智能·算法·机器学习
罗西的思考9 小时前
【轻量化复刻大模型 / KIMI K3】Multi-Token Prediction(MTP / EAGLE-3)(1)
人工智能·算法·机器学习
汤姆宝库10 小时前
BPE分词器的完整实现:字节对合并与词表构建
开发语言·人工智能·c#
岁月宁静10 小时前
从零拆解 AI 语音模拟面试系统
人工智能·python·agent
武子康11 小时前
H3-World 接入 Game Agent:谁来判断门真的打开了?
人工智能·算法·机器学习
可乐ea11 小时前
MCP 和 CLI 差 42 倍:一次 GitHub 任务的上下文成本实测
人工智能·github·工具调用·ai智能体·openrouter·mcp协议·上下文优化
kyriewen12 小时前
编辑弹窗点了取消,列表里的数据怎么还是变了?
前端·javascript·人工智能
泯泷12 小时前
Multica 真正打动我的,不是更多 Agent,而是一种新的团队协作方式
人工智能·llm·agent
电科_银尘12 小时前
【深度学习项目实战·22】灰度图像着色:用小型U-Net学习从灰度到RGB
人工智能·深度学习·学习