transformers in tabular tiny survey 2024.4.8

推荐阅读

TabLLM

pmlr2023,

Few-shot Classification of Tabular Data with Large Language Models

方法

使用把tabular数据序列化成文字的方法进行classification。
使用的序列化方法有几个,有人工也有AI生成。

效果

做few shot learning的效果
看上去一般。

TransTab

Learning Transferable Tabular Transformers Across Tables

方法

属于transfer learning的方法。对category、binary和numeric值进行embedding后再进行transformers最后进行classification。

使用场景

原文:

  • S(1) Transfer learning . We collect data tables from multiple cancer trials for testing the efficacy

of the same drug on different patients. These tables were designed independently with overlapping

columns. How do we learn ML models for one trial by leveraging tables from all trials?

  • S(2) Incremental learning . Additional columns might be added over time. For example, additional

features are collected across different trial phases. How do we update the ML models using tables

from all trial phases?

  • S(3) Pretraining+Finetuning . The trial outcome label (e.g., mortality) might not be always available

from all table sources. Can we benefit pretraining on those tables without labels? How do we finetune

the model on the target table with labels?

  • S(4) Zero-shot inference . We model the drug efficacy based on our trial records. The next step is to

conduct inference with the model to find patients that can benefit from the drug. However, patient

tables do not share the same columns as trial tables so direct inference is not possible.

效果

具体看原文吧,与当时的baseline比有提升。

MET

Masked Encoding for Tabular Data

tabtransformer

2020年,arxiv,TabTransformer: Tabular Data Modeling Using Contextual Embeddings

方法

transformer无监督训练,mlp监督训练。

原文

we introduce a pre-training procedure to train the Transformer layers using unlabeled data . This is followed by fine-tuning of the pre-trained Transformer layers along with the top MLP layer using the labeled data

效果

跟mlp

跟其他模型

tabnet

2020, arxiv,Google Cloud AI,Attentive Interpretable Tabular Learning, 封装的非常好,都可以当工具包使用了。

方法

跟transformer没关系的。
feature selection用的是17年的某个选择模型,最后agg一下做predict。

相关推荐
Csvn6 小时前
同一个问题问 100 遍,钱就扣 100 次?精确缓存让它只扣一次(C02)
人工智能·aigc·agent
罗西的思考7 小时前
【轻量化复刻大模型 / KIMI K3】Multi-Token Prediction(MTP / EAGLE-3)(2)
人工智能·算法·机器学习
罗西的思考7 小时前
【轻量化复刻大模型 / KIMI K3】Multi-Token Prediction(MTP / EAGLE-3)(1)
人工智能·算法·机器学习
汤姆宝库8 小时前
BPE分词器的完整实现:字节对合并与词表构建
开发语言·人工智能·c#
岁月宁静8 小时前
从零拆解 AI 语音模拟面试系统
人工智能·python·agent
武子康8 小时前
H3-World 接入 Game Agent:谁来判断门真的打开了?
人工智能·算法·机器学习
可乐ea9 小时前
MCP 和 CLI 差 42 倍:一次 GitHub 任务的上下文成本实测
人工智能·github·工具调用·ai智能体·openrouter·mcp协议·上下文优化
kyriewen9 小时前
编辑弹窗点了取消,列表里的数据怎么还是变了?
前端·javascript·人工智能
泯泷9 小时前
Multica 真正打动我的,不是更多 Agent,而是一种新的团队协作方式
人工智能·llm·agent
电科_银尘9 小时前
【深度学习项目实战·22】灰度图像着色:用小型U-Net学习从灰度到RGB
人工智能·深度学习·学习