TVM: An Automated End-to-End Optimizing Compiler for Deep Learning

TVM can process graph-level and operator-level optimization.

graph-level optimization

As for the graph-level optimization, it can do operator fusion, constant folding, static memory pre-allocation, and data transformate pass.

operator fusion

Now I want emphasis operator fusion, It split the operator into 4 type:

  • injective (one on one map, e.g., add)
  • reduction
  • complex out fusable(can fuse element-wise op to output)
  • opaque(can't be fused, e.g., sort)

TVM will fuse as much as possible.

These optimization methods are very common.

Operator-level Optimization

TVM seperate schedule and compute. So it can detribute different devices. There are 3 schedule primitives in TVM, Special Memory Scope, Tensorizaiton, Latency Hiding.

  • Special Memory Scope, to utilize maxmium the shaped memory in GPU.
  • Tensorization, spliting a bigger data into micro-data to fully utiize the vectorization.
  • Latency Hiding. Overlaping the computation and transition. On CPU, it is achieving by using multi-threading or hardward prefetching. GPU relys on repid context switching of many wraps of threads.

Automating Optimization

How to find the optimal parameter is very important. It proposed a ML-based cost model, which is a gradient tree boosting model based on XGboost, to predict these prameters by giving the loop pragram in the kernel, which include the memory access count, and the resue ratio of each memory buffer, as well as one-hot encoding of loop annotation such as "vectorize", "unroll" and "parallel". As shown in the following graph, the collected data can be train the model again. So the TVM matainer will updated this model periodicly.

Consequently, TVM lowers the threshold for writing a relavely high-performance kernel. I think there are 2 points deserved us to learn more, which are the schedule primitive and the prediction model.

相关推荐
Flynn Liang2 分钟前
多智能体协同不再复杂:Redis Stream 消费者组、幂等与故障恢复
数据库·人工智能·redis·缓存·ai agent·智能体
VIP_CQCRE9 小时前
GPT Image 2.6 商品图换背景后悬浮?接触阴影、透明区域与反射验收
图像处理·人工智能·ai·电商
泯泷9 小时前
别再傻傻地喂文件了,大厂都在偷偷给AI建这种“专属图书馆”
人工智能
oak隔壁找我10 小时前
一款运行在 DSH(DeepSeek Harness)中的类 Postman HTTP 接口调试插件
人工智能·程序员
高洁0110 小时前
数字孪生与计算机视觉:让机器看懂现场
人工智能·python·机器学习·transformer·迁移学习
Csvn11 小时前
如果明天供应商涨价 3 倍?三层解耦,换模型只改一行配置(C05)
人工智能·aigc·agent
在所不辞兄11 小时前
为什么数字孪生驱动的产线标定需要在线更新机制,而不能一次标定后长期沿用?
人工智能·神经网络·其他·算法·机器学习
锋行天下12 小时前
langgraph 全栈项目(三)
前端·人工智能·后端
黄啊码12 小时前
【黄啊码】最近面了十几个想转 FDE 的程序员,一个都没招
人工智能
锋行天下13 小时前
langgraph 全栈项目(二)
前端·人工智能·后端