【Transformer系列(2)】Multi-head self-attention 多头自注意力

一、多头自注意力

多头自注意力机制与自注意力机制的区别在于,Q,K,V向量被分为了num_heads份。

实现流程

(1)根据num_heads参数将单头变成多头,获取多头注意力中的各个头的Q,K,V值

(2)Q叉乘K的转置,再使用softmax,获取attention

(3)attention叉乘V,得到输出

二、代码实现

(1)根据num_heads参数将单头变成多头,获取多头注意力中的各个头的Q,K,V值

cpp 复制代码
# 每个token(Q,K,V)的尺寸
values_length = 33
# 原始单头长度
hidden_size = 768
# 单头qkv
# [33,768]
Query = np.random.rand(values_length, hidden_size)
Key = np.random.rand(values_length, hidden_size)
Value = np.random.rand(values_length, hidden_size)

# 单头 -> 分组为8个头
# [33,768] -> [33,8,96]
# 8个头
num_attention_heads = 8
# 原始单头拆分为多头后,我们单头的长度
attention_head_size = hidden_size // num_attention_heads
Query = np.reshape(Query, [values_length, num_attention_heads, attention_head_size])
Key = np.reshape(Key, [values_length, num_attention_heads, attention_head_size])
Value = np.reshape(Value, [values_length, num_attention_heads, attention_head_size])

# [33,8,96] -> [8,33,96] 头放最前面 M,H*W,C
Query = np.transpose(Query, [1, 0, 2])
Key = np.transpose(Key, [1, 0, 2])
Value = np.transpose(Value, [1, 0, 2])

(2)Q叉乘K的转置,再使用softmax,获取attention

cpp 复制代码
# qv -> attention
# [8,33,96] @ [8,96,33] -> [8,33,33] [m1,n] @ [n,m2] -> [m1,m2]
scores = Query @ np.transpose(Key, [0, 2, 1])
print(np.shape(scores))
# qv+softmax -> attention
scores = soft_max(scores)
print(np.shape(scores))

(3)attention叉乘V,得到输出

cpp 复制代码
# attention+v -> output
# [8,33,33] @ [8,33,96] -> [8,33,96] [m1,n] @ [n,m2] -> [m1,m2]
out = scores @ Value
print(np.shape(out))
# [8,33,96] -> [33,8,96]
out = np.transpose(out, [1, 0, 2])
print(np.shape(out))
# [33,8,96] -> [33,768]
out = np.reshape(out, [values_length , 768])
print(np.shape(out))

三、完整代码

cpp 复制代码
# multi-head self-attention #
# by liushuai #
# 2024/2/6 #

import numpy as np

def soft_max(z):
    t = np.exp(z)
    a = np.exp(z) / np.expand_dims(np.sum(t, axis=-1), -1)
    return a

# 每个token(Q,K,V)的尺寸
# 相当于H*W
values_length = 33
# 原始单头深度
# 相当于Channels
hidden_size = 768
# 单头qkv
# [33,768]
Query = np.random.rand(values_length, hidden_size)
Key = np.random.rand(values_length, hidden_size)
Value = np.random.rand(values_length, hidden_size)

# 单头 -> 分组为8个头
# [33,768] -> [33,8,96]
# 8个头
num_attention_heads = 8
# 原始单头拆分为多头后,我们单头的深度
attention_head_size = hidden_size // num_attention_heads
Query = np.reshape(Query, [values_length, num_attention_heads, attention_head_size])
Key = np.reshape(Key, [values_length, num_attention_heads, attention_head_size])
Value = np.reshape(Value, [values_length, num_attention_heads, attention_head_size])

# [33,8,96] -> [8,33,96] 头放最前面 M,H*W,C
Query = np.transpose(Query, [1, 0, 2])
Key = np.transpose(Key, [1, 0, 2])
Value = np.transpose(Value, [1, 0, 2])

# qv -> attention
# [8,33,96] @ [8,96,33] -> [8,33,33] [m1,n] @ [n,m2] -> [m1,m2]
scores = Query @ np.transpose(Key, [0, 2, 1])
print(np.shape(scores))
# qv+softmax -> attention
scores = soft_max(scores)
print(np.shape(scores))

# attention+v -> output
# [8,33,33] @ [8,33,96] -> [8,33,96] [m1,n] @ [n,m2] -> [m1,m2]
out = scores @ Value
print(np.shape(out))
# [8,33,96] -> [33,8,96]
out = np.transpose(out, [1, 0, 2])
print(np.shape(out))
# [33,8,96] -> [33,768]
out = np.reshape(out, [values_length , 768])
print(np.shape(out))
相关推荐
Json_几秒前
Vue 实例方法
前端·vue.js·深度学习
mosquito_lover115 分钟前
矿山边坡监测预警系统设计
人工智能·python·深度学习·神经网络·视觉检测
船长@Quant15 分钟前
PyTorch量化进阶教程:第二章 Transformer 理论详解
pytorch·python·深度学习·transformer·量化交易·sklearn·ta-lib
契合qht53_shine16 分钟前
OpenCV 从入门到精通(day_03)
人工智能·opencv·计算机视觉
Json_28 分钟前
实例入门 实例属性
前端·深度学习
Json_29 分钟前
JS中的apply和arguments小练习
前端·javascript·深度学习
Json_1 小时前
Vue Methods Option 方法选项
前端·vue.js·深度学习
Naomi5211 小时前
Trustworthy Machine Learning
人工智能·机器学习
刘 怼怼1 小时前
使用 Vue 重构 RAGFlow 实现聊天功能
前端·vue.js·人工智能·重构
程序员安仔1 小时前
每天学新 AI 工具好累?我终于发现了“一键全能且免费不限量”的国产终极解决方案
人工智能