MapReduce Join 操作：大表关联小表用 Map 端，大表关联大表用 Reduce 端
数据处理中，多表关联是家常便饭------订单关联用户、日志关联商品、销售关联库存。在 SQL 里一个 JOIN 就搞定，在 MapReduce 里得自己实现。
MapReduce 提供了两种 Join 方案：
- Reduce 端 Join：通用，任何规模都能用，但要走 Shuffle，性能一般
- Map 端 Join：快，但要求一张表足够小，能塞进内存
怎么选？就一句话：小表能装进内存就用 Map 端 Join，装不下就用 Reduce 端 Join。
Join 的核心难点：怎么让相同 Key 的数据到同一个地方
两个表关联，关键是把相同关联键的数据凑到一起。
- 订单表有商品 ID → 需要找到商品表里相同商品 ID 的信息
- 问题是：订单数据分散在不同节点，商品数据也分散在不同节点
- 怎么让同一个商品 ID 的订单数据和商品数据落到同一个节点？
这就是 Join 的核心挑战。MapReduce 用两种方式解决：
| 方案 | 怎么凑到一起 | 前提 |
|---|---|---|
| Reduce 端 Join | 通过 Shuffle，相同 Key 自动分到同一个 Reduce | 无限制 |
| Map 端 Join | 小表提前加载到每个 Map 任务的内存里 | 小表能装进内存 |
Reduce 端 Join：通用方案，任何规模都能用
实现原理
#mermaid-svg-DCWcdcFvCUVc8DXR{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-DCWcdcFvCUVc8DXR .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-DCWcdcFvCUVc8DXR .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-DCWcdcFvCUVc8DXR .error-icon{fill:#552222;}#mermaid-svg-DCWcdcFvCUVc8DXR .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-DCWcdcFvCUVc8DXR .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-DCWcdcFvCUVc8DXR .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-DCWcdcFvCUVc8DXR .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-DCWcdcFvCUVc8DXR .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-DCWcdcFvCUVc8DXR .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-DCWcdcFvCUVc8DXR .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-DCWcdcFvCUVc8DXR .marker{fill:#333333;stroke:#333333;}#mermaid-svg-DCWcdcFvCUVc8DXR .marker.cross{stroke:#333333;}#mermaid-svg-DCWcdcFvCUVc8DXR svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-DCWcdcFvCUVc8DXR p{margin:0;}#mermaid-svg-DCWcdcFvCUVc8DXR .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-DCWcdcFvCUVc8DXR .cluster-label text{fill:#333;}#mermaid-svg-DCWcdcFvCUVc8DXR .cluster-label span{color:#333;}#mermaid-svg-DCWcdcFvCUVc8DXR .cluster-label span p{background-color:transparent;}#mermaid-svg-DCWcdcFvCUVc8DXR .label text,#mermaid-svg-DCWcdcFvCUVc8DXR span{fill:#333;color:#333;}#mermaid-svg-DCWcdcFvCUVc8DXR .node rect,#mermaid-svg-DCWcdcFvCUVc8DXR .node circle,#mermaid-svg-DCWcdcFvCUVc8DXR .node ellipse,#mermaid-svg-DCWcdcFvCUVc8DXR .node polygon,#mermaid-svg-DCWcdcFvCUVc8DXR .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-DCWcdcFvCUVc8DXR .rough-node .label text,#mermaid-svg-DCWcdcFvCUVc8DXR .node .label text,#mermaid-svg-DCWcdcFvCUVc8DXR .image-shape .label,#mermaid-svg-DCWcdcFvCUVc8DXR .icon-shape .label{text-anchor:middle;}#mermaid-svg-DCWcdcFvCUVc8DXR .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-DCWcdcFvCUVc8DXR .rough-node .label,#mermaid-svg-DCWcdcFvCUVc8DXR .node .label,#mermaid-svg-DCWcdcFvCUVc8DXR .image-shape .label,#mermaid-svg-DCWcdcFvCUVc8DXR .icon-shape .label{text-align:center;}#mermaid-svg-DCWcdcFvCUVc8DXR .node.clickable{cursor:pointer;}#mermaid-svg-DCWcdcFvCUVc8DXR .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-DCWcdcFvCUVc8DXR .arrowheadPath{fill:#333333;}#mermaid-svg-DCWcdcFvCUVc8DXR .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-DCWcdcFvCUVc8DXR .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-DCWcdcFvCUVc8DXR .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-DCWcdcFvCUVc8DXR .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-DCWcdcFvCUVc8DXR .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-DCWcdcFvCUVc8DXR .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-DCWcdcFvCUVc8DXR .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-DCWcdcFvCUVc8DXR .cluster text{fill:#333;}#mermaid-svg-DCWcdcFvCUVc8DXR .cluster span{color:#333;}#mermaid-svg-DCWcdcFvCUVc8DXR div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-DCWcdcFvCUVc8DXR .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-DCWcdcFvCUVc8DXR rect.text{fill:none;stroke-width:0;}#mermaid-svg-DCWcdcFvCUVc8DXR .icon-shape,#mermaid-svg-DCWcdcFvCUVc8DXR .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-DCWcdcFvCUVc8DXR .icon-shape p,#mermaid-svg-DCWcdcFvCUVc8DXR .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-DCWcdcFvCUVc8DXR .icon-shape .label rect,#mermaid-svg-DCWcdcFvCUVc8DXR .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-DCWcdcFvCUVc8DXR .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-DCWcdcFvCUVc8DXR .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-DCWcdcFvCUVc8DXR :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Map 阶段
Map 阶段
表 A 数据
打标签
表 B 数据
打标签
Shuffle 按 Key 分组
Reduce 阶段合并
输出关联结果
思路：
- Map 阶段：从不同表读数据，统一输出
<关联键, 来源标记 + 数据> - Shuffle 阶段：相同关联键自动分到同一个 Reduce
- Reduce 阶段：把同一 Key 的两张表数据合并
关键点：Map 阶段一定要标记数据来源。 Reduce 阶段收到的是同一个 Key 的 value 列表，不知道哪些来自订单表、哪些来自商品表，所以需要标记（比如用 "order" 和 "product" 前缀）。
代码骨架：
Map 阶段：
java
public void map(LongWritable key, Text value, Context context) {
FileSplit split = (FileSplit) context.getInputSplit();
String fileName = split.getPath().getName();
String[] fields = value.toString().split(",");
if (fileName.contains("order")) {
// 订单表：关联键是商品ID（第3列）
outKey.set(fields[2]); // 商品ID
outValue.set("order," + fields[0] + "," + fields[1] + "," + fields[3]);
context.write(outKey, outValue);
} else if (fileName.contains("product")) {
// 商品表：关联键是商品ID（第1列）
outKey.set(fields[0]); // 商品ID
outValue.set("product," + fields[1]); // 商品名称
context.write(outKey, outValue);
}
}Reduce 阶段：
java
public void reduce(Text key, Iterable<Text> values, Context context) {
List<String> orders = new ArrayList<>();
String productName = "";
for (Text value : values) {
String[] parts = value.toString().split(",");
if (parts[0].equals("order")) {
orders.add(parts[1] + "," + parts[2] + "," + parts[3]); // 订单ID,用户ID,金额
} else if (parts[0].equals("product")) {
productName = parts[1];
}
}
// 内连接：只输出有商品信息的订单
for (String order : orders) {
if (!productName.isEmpty()) {
context.write(null, new Text(order + "," + productName));
}
}
}优缺点：
| 优点 | 缺点 |
|---|---|
| 数据量无限制，什么都能 Join | 所有数据都要走 Shuffle，网络传输大 |
| 支持内连接、左连接、外连接 | Reduce 端需要排序，开销大 |
| 实现直观，好理解 | 大表 Join 大表时，Reduce 是瓶颈 |
Map 端 Join：快，但小表得能装进内存
实现原理
#mermaid-svg-EjmRvvjXraaoTpvd{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-EjmRvvjXraaoTpvd .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-EjmRvvjXraaoTpvd .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-EjmRvvjXraaoTpvd .error-icon{fill:#552222;}#mermaid-svg-EjmRvvjXraaoTpvd .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-EjmRvvjXraaoTpvd .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-EjmRvvjXraaoTpvd .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-EjmRvvjXraaoTpvd .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-EjmRvvjXraaoTpvd .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-EjmRvvjXraaoTpvd .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-EjmRvvjXraaoTpvd .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-EjmRvvjXraaoTpvd .marker{fill:#333333;stroke:#333333;}#mermaid-svg-EjmRvvjXraaoTpvd .marker.cross{stroke:#333333;}#mermaid-svg-EjmRvvjXraaoTpvd svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-EjmRvvjXraaoTpvd p{margin:0;}#mermaid-svg-EjmRvvjXraaoTpvd .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-EjmRvvjXraaoTpvd .cluster-label text{fill:#333;}#mermaid-svg-EjmRvvjXraaoTpvd .cluster-label span{color:#333;}#mermaid-svg-EjmRvvjXraaoTpvd .cluster-label span p{background-color:transparent;}#mermaid-svg-EjmRvvjXraaoTpvd .label text,#mermaid-svg-EjmRvvjXraaoTpvd span{fill:#333;color:#333;}#mermaid-svg-EjmRvvjXraaoTpvd .node rect,#mermaid-svg-EjmRvvjXraaoTpvd .node circle,#mermaid-svg-EjmRvvjXraaoTpvd .node ellipse,#mermaid-svg-EjmRvvjXraaoTpvd .node polygon,#mermaid-svg-EjmRvvjXraaoTpvd .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-EjmRvvjXraaoTpvd .rough-node .label text,#mermaid-svg-EjmRvvjXraaoTpvd .node .label text,#mermaid-svg-EjmRvvjXraaoTpvd .image-shape .label,#mermaid-svg-EjmRvvjXraaoTpvd .icon-shape .label{text-anchor:middle;}#mermaid-svg-EjmRvvjXraaoTpvd .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-EjmRvvjXraaoTpvd .rough-node .label,#mermaid-svg-EjmRvvjXraaoTpvd .node .label,#mermaid-svg-EjmRvvjXraaoTpvd .image-shape .label,#mermaid-svg-EjmRvvjXraaoTpvd .icon-shape .label{text-align:center;}#mermaid-svg-EjmRvvjXraaoTpvd .node.clickable{cursor:pointer;}#mermaid-svg-EjmRvvjXraaoTpvd .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-EjmRvvjXraaoTpvd .arrowheadPath{fill:#333333;}#mermaid-svg-EjmRvvjXraaoTpvd .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-EjmRvvjXraaoTpvd .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-EjmRvvjXraaoTpvd .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-EjmRvvjXraaoTpvd .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-EjmRvvjXraaoTpvd .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-EjmRvvjXraaoTpvd .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-EjmRvvjXraaoTpvd .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-EjmRvvjXraaoTpvd .cluster text{fill:#333;}#mermaid-svg-EjmRvvjXraaoTpvd .cluster span{color:#333;}#mermaid-svg-EjmRvvjXraaoTpvd div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-EjmRvvjXraaoTpvd .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-EjmRvvjXraaoTpvd rect.text{fill:none;stroke-width:0;}#mermaid-svg-EjmRvvjXraaoTpvd .icon-shape,#mermaid-svg-EjmRvvjXraaoTpvd .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-EjmRvvjXraaoTpvd .icon-shape p,#mermaid-svg-EjmRvvjXraaoTpvd .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-EjmRvvjXraaoTpvd .icon-shape .label rect,#mermaid-svg-EjmRvvjXraaoTpvd .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-EjmRvvjXraaoTpvd .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-EjmRvvjXraaoTpvd .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-EjmRvvjXraaoTpvd :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} Map 初始化
Map 阶段
小表数据
加载至内存 Map
大表数据
读取大表记录，通过内存 Map 关联
直接输出关联结果
思路： 把小表加载到每个 Map 任务的内存里，Map 读大表时直接在内存里查，不需要 Shuffle 和 Reduce。
前提条件： 小表必须能装进 MapTask 内存（通常 < 1GB）。如果装不下，Map 端 Join 不适用。
代码骨架：
setup 阶段加载小表：
java
private HashMap<String, String> productMap = new HashMap<>();
@Override
protected void setup(Context context) {
// 从分布式缓存读取小表
URI[] cacheFiles = context.getCacheFiles();
Path productPath = new Path(cacheFiles[0]);
FileSystem fs = FileSystem.get(context.getConfiguration());
BufferedReader reader = new BufferedReader(new InputStreamReader(fs.open(productPath)));
String line;
while ((line = reader.readLine()) != null) {
String[] fields = line.split(",");
productMap.put(fields[0], fields[1]); // 商品ID → 商品名称
}
reader.close();
}Map 阶段查内存：
java
public void map(LongWritable key, Text value, Context context) {
String[] fields = value.toString().split(",");
String productId = fields[2]; // 订单表的商品ID
String productName = productMap.get(productId);
if (productName != null) {
String result = fields[0] + "," + fields[1] + "," + fields[3] + "," + productName;
context.write(null, new Text(result));
}
}Driver 里把小表放进分布式缓存：
java
job.addCacheFile(new URI("hdfs:///input/product.txt"));
job.setNumReduceTasks(0); // 不需要 Reduce优缺点：
| 优点 | 缺点 |
|---|---|
| 无 Shuffle，无 Reduce，性能极高 | 小表必须能装进内存，否则 OOM |
| 网络传输少 | 只适合"一大一小"的场景 |
| 代码比 Reduce 端 Join 简单 | 不支持大表 Join 大表 |
实际项目中的经验：
- 小表 < 100MB → 无脑 Map 端 Join
- 小表 100MB - 500MB → Map 端 Join，把 MapTask 内存调到 2GB 以上
- 小表 > 500MB → 先用 Map 端 Join 试，不行换 Reduce 端
- 两张都大 → Reduce 端 Join，同时用 Combiner 和压缩优化