在多渠道公开商业信号采集与检索系统中，同一条真实采购需求往往会在短时间内被十几个不同站点镜像转载：省级招采门户首发后，市级分站、行业协会网站、招投标聚合页乃至垂直社区都会陆续出现标题略有增删（例如加上【转载】或地区前缀）、企业名称写法不一（全角括号、简称、地区分公司后缀混用）的副本。
如果入库管线缺乏高吞吐的近似去重与企业实体规范化能力，会同时引发两个工程痛点：
- 检索结果严重刷屏：一线业务人员搜索某个细分设备词，首屏十条结果里八条都是同一个项目的转载副本，白白消耗核验工时；
- 简单去重抹掉关键出处：如果用粗暴的标题哈希覆盖去重，又可能把同一家集团同期发布的"一期工程"和"二期工程"误合并，或者丢掉首发官方门户的权威原文链接。
本文分享一套在日增十万级异构文本流下稳定运行的工程方案：企业总分实体规范化 + 64-bit SimHash（4×16-bit 抽屉分桶索引）召回 + 信用代码与标段边界守卫 + 3-gram 字符片 Jaccard 精排校验 + 只追加（Append-Only）不可变证据链存储，并附经 Python 3.11 实跑验证的标准库参考代码。
一、为什么单靠 SimHash 或单靠全文检索都不够？
在中文短中长度公文与商业询盘（通常 100 至 1,500 字）场景下，我们对常见去重方案做过实测对比：
| 方案 | 时间复杂度 | 转载标题加前缀鲁棒性 | 同业主不同标段区分度 | 主要工程短板 |
|---|---|---|---|---|
| 原文精确 SHA-256 | O(1) |
极差（多一个感叹号即失效） | 高 | 无法识别任何改写转载 |
| 纯 64-bit SimHash（汉明距离<=3） | 分桶后近 O(1) |
高 | 一般（模板公文套话占比高时易误撞） | 长模板公文头部相同时，仅靠哈希易把一二标段合并 |
| 纯 Shingle Jaccard 两两计算 | O(N^2) |
高 | 高 | 存量百万级文档下无法实时在线比对 |
| SimHash 4段分桶召回 + 信用代码/标段守卫 + 3-gram Jaccard 精排 | 近 O(1) 召回 + 极小候选精排 |
高 | 高 | 兼顾高吞吐与高精度，实现复杂度适中 |
根据鸽巢原理（Pigeonhole Principle），把 64-bit 指纹均分为 4 个互不重叠的 16-bit 分桶后，仅当两个指纹的汉明距离 <= 3 时，才能 100% 保证至少有一个 16-bit 段完全相等（至多 3 个差异比特放入 4 个抽屉，必有至少 1 个抽屉 0 翻转）。必须注意两个工程边界：
4桶索引只保证<= 3的汉明确定性召回 ：一旦两条文本的指纹汉明距离>= 4（例如 4 个翻转位恰好均匀落在 4 个段中各占 1 位，或仅抽取三十余字极短摘要导致前置【转载】标签翻转超过 3 位），4 个分桶可能出现0共享桶，候选集为空，后续精排根本不会执行。因此构造指纹的输入必须保留足够完整的项目要素（编号、工程范围、核心设备参数、限价、工期与资质，通常100字以上）；若业务容忍更大改写幅度、需要确定性召回汉明距离<= 5的样本，则必须按鸽巢原理扩充至6个分桶或多探针位翻转。- 同桶命中必须走"且（AND）"精排与业务边界守卫 ：共享 1 个
16-bit分桶既可能是1/65536的随机碰撞，也可能是同一业主套用长招标模板同期发布的"一标段"与"二标段"。因此候选召回后，必须依次通过统一社会信用代码、分支机构、标段/期次硬边界检查，并同时满足汉明距离 <= 3与3-gram Jaccard >= 阈值方可归并。
二、中文企业实体规范化：母公司与属地分公司的双层保留
在 B2B 场景中，采购主体经常写作 华南智造新材料股份有限公司（东莞分公司） 或 华南智造新材料股份有限公司东莞分公司。如果把它们当成两个完全无关的字符串，不仅集团历史履约记录无法打通，还会干扰特征切片；如果直接截断丢掉 东莞分公司，负责珠三角片区的属地销售又会错失具体的交付所在地信息。
我们的规范化与边界判定遵循三步确定性规则：
- 执行
NFKCUnicode 标准折叠，统一全半角字母、数字、空白符并将全角括号（）折叠为半角()； - 用受控组织形式正则（
股份有限公司、有限责任公司、集团有限公司等）定位法人主体右边界，同时兼容带括号的(东莞分公司)与裸后缀/连字符形式的东莞分公司、-东莞办事处，统一将法人母公司写入canonical_hq_name、将不带括号的纯净分支名写入branch_name； - 当两条记录均携带非空统一社会信用代码（
credit_code）且彼此不一致时，或均携带明确的分支机构、标段/期次标识（如一标段与二标段、一期与二期）且相互冲突时，无论文本相似度多高均强制阻断合并，防止不同法人套用同一招标代理模板串单或同业主多标段误并。
三、只追加（Append-Only）证据链存储 Schema 设计
去重绝不意味着"删掉副本只留一条摘要"。在关系型数据库（如 PostgreSQL）中，我们采用主从双表结构保留完整证据链：
sql
CREATE TABLE demand_clusters (
cluster_id VARCHAR(32) PRIMARY KEY,
canonical_hq_name TEXT NOT NULL,
branch_name TEXT,
credit_code CHAR(18),
scope_marker VARCHAR(64),
-- Python uint64 按 64 位二补码无损映射为有符号 BIGINT（u64 >= 2^63 时写入 u64 - 2^64）
rep_simhash BIGINT NOT NULL,
-- 16-bit 无符号分桶值范围 0..65535，超出 SMALLINT 上限(32767)，须采用 INTEGER 配合范围约束
band_0 INTEGER NOT NULL CHECK (band_0 BETWEEN 0 AND 65535),
band_1 INTEGER NOT NULL CHECK (band_1 BETWEEN 0 AND 65535),
band_2 INTEGER NOT NULL CHECK (band_2 BETWEEN 0 AND 65535),
band_3 INTEGER NOT NULL CHECK (band_3 BETWEEN 0 AND 65535),
first_seen_at TIMESTAMPTZ NOT NULL,
latest_seen_at TIMESTAMPTZ NOT NULL
);
CREATE INDEX idx_clusters_band0 ON demand_clusters (band_0);
CREATE INDEX idx_clusters_band1 ON demand_clusters (band_1);
CREATE INDEX idx_clusters_band2 ON demand_clusters (band_2);
CREATE INDEX idx_clusters_band3 ON demand_clusters (band_3);
CREATE TABLE demand_evidence_records (
evidence_id BIGSERIAL PRIMARY KEY,
cluster_id VARCHAR(32) NOT NULL REFERENCES demand_clusters(cluster_id),
source_id VARCHAR(64) NOT NULL,
source_url TEXT NOT NULL,
captured_at TIMESTAMPTZ NOT NULL,
raw_org_name TEXT NOT NULL,
raw_excerpt TEXT NOT NULL,
UNIQUE (source_url)
);落地 PostgreSQL DDL 时需要特别处理两处无符号整数类型映射：
- 16 位分桶列不能使用
SMALLINT：PostgreSQL 没有无符号整型，SMALLINT（int2）的有符号上限仅为32767（-32768..32767），而(fp >> (16 * i)) & 0xFFFF产生的16-bit无符号分桶值范围为0..65535（例如实跑样例中的42745、61776），使用SMALLINT会直接触发溢出报错，因此改为INTEGER并附加CHECK (band_i BETWEEN 0 AND 65535)； - 64 位无符号 SimHash 到
BIGINT的二补码映射 ：Python 生成的64-bit指纹位于0..2^64-1，当最高位bit 63 = 1（u64 >= 2^63）时超出有符号BIGINT上限2^63-1。通过uint64_to_pg_bigint（当u64 >= 2^63时减去2^64）做二补码无损转换，读取时再用i64 & 0xFFFFFFFFFFFFFFFF还原，不仅避免范围溢出，还完整保持全部 64 个比特位不变------在 PostgreSQL 14+ 中可直接用bit_count((rep_simhash # :incoming_simhash)::bit(64)) <= 3完成库内汉明距离核验。
同时注意 rep_simhash 在簇创建时即固定锁定，不随新合并副本动态重算均值------这能有效防止**链式漂移（Chaining Drift）**把语义逐步偏移的无关记录吸入同一个簇。
四、可独立运行的 Python 3 参考实现
以下脚本仅使用 Python 3 标准库（兼容 Python 3.10+，已通过 Python 3.11 实跑验证）实现完整的 4×16-bit 分桶索引、括号与裸后缀双形态分公司剥离、标段/信用代码边界守卫、PostgreSQL BIGINT 二补码映射与多源证据归并：
python
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Reference Implementation: Two-Stage Near-Duplicate Signal Clustering (64-bit SimHash
with 4x16-bit Pigeonhole Bands + Boundary Guards + 3-gram Jaccard) & Enterprise Entity
Canonicalization with Immutable Evidence-Chain Storage.
Python 3.10+ standard library only (verified on Python 3.11).
"""
from __future__ import annotations
import hashlib
import json
import re
import unicodedata
from collections import defaultdict
from dataclasses import dataclass, field
from typing import Dict, FrozenSet, Iterable, List, Optional, Set, Tuple
CORP_SUFFIX_RE = re.compile(
r"(有限责任公司|股份有限公司|集团有限公司|科技有限公司|工程有限公司|有限公司)"
r"(?:[----·]*(?:\(([^()]+?(?:分公司|办事处))\)|([^()]+?(?:分公司|办事处))))?$"
)
CN_NUM_MAP = {
"一": "1",
"二": "2",
"三": "3",
"四": "4",
"五": "5",
"六": "6",
"七": "7",
"八": "8",
"九": "9",
"十": "10",
}
LOT_MARKER_RE = re.compile(
r"第?([一二三四五六七八九十\d]+)(?:标段|包段|分包)|标段([一二三四五六七八九十A-Z\d]+)"
)
PHASE_MARKER_RE = re.compile(r"第?([一二三四五六七八九十\d]+)期")
UINT64_MASK = (1 << 64) - 1
INT64_SIGN_BIT = 1 << 63
UINT64_MOD = 1 << 64
def uint64_to_pg_bigint(u64: int) -> int:
"""
Lossless two's-complement mapping from unsigned 64-bit integer [0, 2^64-1]
to PostgreSQL signed BIGINT [-2^63, 2^63-1], preserving all 64 bits for XOR/bit_count.
"""
u64 &= UINT64_MASK
return u64 - UINT64_MOD if u64 >= INT64_SIGN_BIT else u64
def pg_bigint_to_uint64(i64: int) -> int:
"""Restores unsigned 64-bit SimHash integer from PostgreSQL signed BIGINT."""
return i64 & UINT64_MASK
def canonicalize_org_name(raw_name: str) -> Tuple[str, Optional[str]]:
"""
Normalizes full-width punctuation/brackets via NFKC and separates legal parent entity
from regional branch suffix (both '(XX分公司)' and bare 'XX分公司' / '-XX办事处').
"""
norm = unicodedata.normalize("NFKC", raw_name).strip()
norm = re.sub(r"\s+", "", norm)
norm = re.sub(r"^\([A-Z0-9]+\)", "", norm) # strip leading stock/tag prefix if any
m = CORP_SUFFIX_RE.search(norm)
if m:
branch = m.group(2) or m.group(3)
if branch:
hq = norm[: m.end(1)]
return hq, branch
return norm, None
def extract_scope_markers(text: str) -> Dict[str, FrozenSet[str]]:
"""Extracts explicit lot/section (标段/分包) and phase (期次) identifiers."""
norm = unicodedata.normalize("NFKC", text)
lots = {
CN_NUM_MAP.get(m.group(1) or m.group(2), m.group(1) or m.group(2))
for m in LOT_MARKER_RE.finditer(norm)
}
phases = {CN_NUM_MAP.get(m.group(1), m.group(1)) for m in PHASE_MARKER_RE.finditer(norm)}
markers: Dict[str, FrozenSet[str]] = {}
if lots:
markers["lot"] = frozenset(lots)
if phases:
markers["phase"] = frozenset(phases)
return markers
def has_scope_conflict(a: Dict[str, FrozenSet[str]], b: Dict[str, FrozenSet[str]]) -> bool:
"""Returns True if both carry explicit lot or phase markers and disagree."""
for key in ("lot", "phase"):
if key in a and key in b and a[key] != b[key]:
return True
return False
def format_scope_markers(markers: Dict[str, FrozenSet[str]]) -> Optional[str]:
if not markers:
return None
parts = [f"{k}={','.join(sorted(markers[k]))}" for k in ("phase", "lot") if k in markers]
return ";".join(parts) if parts else None
def char_shingles(text: str, k: int = 3) -> Set[str]:
cleaned = re.sub(r"[\s\W_]+", "", unicodedata.normalize("NFKC", text))
if len(cleaned) <= k:
return {cleaned} if cleaned else set()
return {cleaned[i : i + k] for i in range(len(cleaned) - k + 1)}
def compute_simhash64(tokens: Iterable[str]) -> int:
weights = [0] * 64
for tok in sorted(tokens):
digest = hashlib.blake2b(tok.encode("utf-8"), digest_size=8).digest()
h = int.from_bytes(digest, "big")
for bit in range(64):
if (h >> bit) & 1:
weights[bit] += 1
else:
weights[bit] -= 1
fp = 0
for bit, w in enumerate(weights):
if w > 0:
fp |= 1 << bit
return fp
def hamming_distance64(a: int, b: int) -> int:
return (a ^ b).bit_count()
def jaccard_similarity(a: Set[str], b: Set[str]) -> float:
if not a and not b:
return 1.0
union = len(a | b)
return len(a & b) / union if union else 0.0
@dataclass(frozen=True)
class EvidenceRecord:
source_id: str
source_url: str
captured_at: str
raw_excerpt: str
raw_org_name: str
credit_code: Optional[str] = None
@dataclass
class CanonicalDemandCluster:
cluster_id: str
canonical_hq_name: str
branch_name: Optional[str]
credit_code: Optional[str]
scope_markers: Dict[str, FrozenSet[str]]
representative_simhash: int
shingles: Set[str] = field(repr=False)
evidence_chain: List[EvidenceRecord] = field(default_factory=list)
class SignalDeduplicator:
"""
4 bands x 16-bit inverted buckets guarantee 100% recall ONLY for Hamming distance <= 3
(by Pigeonhole Principle: <= 3 bit flips across 4 disjoint 16-bit bands leaves >= 1 band
identical). Recalled candidates must pass credit-code, branch, and lot/phase boundary
guards, plus Hamming distance <= max_hamming AND 3-gram Jaccard >= min_jaccard checks.
"""
def __init__(self, max_hamming: int = 3, min_jaccard: float = 0.62) -> None:
self.max_hamming = max_hamming
self.min_jaccard = min_jaccard
self.clusters: Dict[str, CanonicalDemandCluster] = {}
self.band_index: List[Dict[int, List[str]]] = [defaultdict(list) for _ in range(4)]
@staticmethod
def bands_u16(fp: int) -> List[int]:
"""Returns four unsigned 16-bit band values in 0..65535 (stored as INTEGER in PostgreSQL)."""
return [(fp >> (16 * i)) & 0xFFFF for i in range(4)]
def ingest(self, ev: EvidenceRecord) -> Tuple[str, str]:
hq_name, branch_name = canonicalize_org_name(ev.raw_org_name)
scope = extract_scope_markers(ev.raw_excerpt)
shingles = char_shingles(f"{hq_name}:{ev.raw_excerpt}", k=3)
fp = compute_simhash64(shingles)
fp_bands = self.bands_u16(fp)
candidate_ids: Set[str] = set()
for band_idx, band_val in enumerate(fp_bands):
candidate_ids.update(self.band_index[band_idx].get(band_val, ()))
best_cid: Optional[str] = None
for cid in sorted(candidate_ids):
cluster = self.clusters[cid]
# 1. Unified social credit code boundary: distinct legal entities never merge
if ev.credit_code and cluster.credit_code and ev.credit_code != cluster.credit_code:
continue
# 2. Regional branch boundary: distinct regional branches do not merge
if branch_name and cluster.branch_name and branch_name != cluster.branch_name:
continue
# 3. Lot / phase boundary: distinct lots (一标段 vs 二标段) of same template do not merge
if has_scope_conflict(scope, cluster.scope_markers):
continue
# 4. Joint SimHash Hamming filter + exact 3-gram Jaccard verification
hdist = hamming_distance64(fp, cluster.representative_simhash)
if hdist > self.max_hamming:
continue
jacc = jaccard_similarity(shingles, cluster.shingles)
if jacc >= self.min_jaccard:
best_cid = cid
break
if best_cid is not None:
target = self.clusters[best_cid]
if not target.credit_code and ev.credit_code:
target.credit_code = ev.credit_code
if not target.branch_name and branch_name:
target.branch_name = branch_name
for k, v in scope.items():
if k not in target.scope_markers:
target.scope_markers[k] = v
if all(e.source_url != ev.source_url for e in target.evidence_chain):
target.evidence_chain.append(ev)
return best_cid, "MERGED_INTO_EXISTING_CLUSTER"
new_cid = f"CLU-{len(self.clusters) + 1:04d}"
new_cluster = CanonicalDemandCluster(
cluster_id=new_cid,
canonical_hq_name=hq_name,
branch_name=branch_name,
credit_code=ev.credit_code,
scope_markers=dict(scope),
representative_simhash=fp,
shingles=shingles,
evidence_chain=[ev],
)
self.clusters[new_cid] = new_cluster
for band_idx, band_val in enumerate(fp_bands):
self.band_index[band_idx][band_val].append(new_cid)
return new_cid, "CREATED_NEW_CLUSTER"
if __name__ == "__main__":
samples = [
EvidenceRecord(
source_id="provincial_portal_01",
source_url="https://portal-a.example.org/202611/notice_901.html",
captured_at="2026-11-14T09:10:00+08:00",
raw_org_name="华南智造新材料股份有限公司（东莞分公司）",
credit_code="91440000MA5123456X",
raw_excerpt=(
"项目编号：HNZZ-DG-202611；项目名称：松山湖基地二期涂布车间一标段有机废气RTO蓄热催化燃烧环保治理改造工程；"
"建设地点：广东省东莞市松山湖高新区工业北路88号；采购内容：设计风量60000m3/h三室RTO蓄热式热力氧化炉1套、"
"LEL在线可燃气体监测连锁装置2套、防爆变频离心引风机2台、阻火器及PLC防爆控制柜交钥匙工程；"
"排放标准：非甲烷总烃排放浓度低于20mg/m3；预算上限135万元，工期45日历天，须具备环保工程专业承包二级及以上资质。"
),
),
EvidenceRecord(
source_id="municipal_mirror_02",
source_url="https://mirror-b.example.org/detail/88312.html",
captured_at="2026-11-14T10:05:00+08:00",
raw_org_name="华南智造新材料股份有限公司东莞分公司",
credit_code=None,
raw_excerpt=(
"【转载】招标公告：项目编号HNZZ-DG-202611，项目名称：松山湖基地二期涂布车间一标段有机废气RTO蓄热催化燃烧环保治理改造工程，"
"建设地点：广东省东莞市松山湖高新区工业北路88号，采购内容：设计风量60000m3/h三室RTO蓄热式热力氧化炉1套、"
"LEL在线可燃气体监测连锁装置2套、防爆变频离心引风机2台、阻火器及PLC防爆控制柜交钥匙工程，"
"排放标准：非甲烷总烃排放浓度低于20mg/m3，预算上限135万元，工期45日历天，须具备环保工程专业承包二级及以上资质（转自省招采网）。"
),
),
EvidenceRecord(
source_id="provincial_portal_03",
source_url="https://portal-a.example.org/202611/notice_902.html",
captured_at="2026-11-14T10:20:00+08:00",
raw_org_name="华南智造新材料股份有限公司（东莞分公司）",
credit_code="91440000MA5123456X",
raw_excerpt=(
"项目编号：HNZZ-DG-202611；项目名称：松山湖基地二期涂布车间二标段有机废气RTO蓄热催化燃烧环保治理改造工程；"
"建设地点：广东省东莞市松山湖高新区工业北路88号；采购内容：设计风量60000m3/h三室RTO蓄热式热力氧化炉1套、"
"LEL在线可燃气体监测连锁装置2套、防爆变频离心引风机2台、阻火器及PLC防爆控制柜交钥匙工程；"
"排放标准：非甲烷总烃排放浓度低于20mg/m3；预算上限128万元，工期45日历天，须具备环保工程专业承包二级及以上资质。"
),
),
EvidenceRecord(
source_id="agency_bulletin_04",
source_url="https://agency-c.example.org/bid/202611_551.html",
captured_at="2026-11-14T11:00:00+08:00",
raw_org_name="深圳华南智造新材料股份有限公司",
credit_code="91440300MA5987654D",
raw_excerpt=(
"项目编号：HNZZ-DG-202611；项目名称：松山湖基地二期涂布车间一标段有机废气RTO蓄热催化燃烧环保治理改造工程；"
"建设地点：广东省东莞市松山湖高新区工业北路88号；采购内容：设计风量60000m3/h三室RTO蓄热式热力氧化炉1套、"
"LEL在线可燃气体监测连锁装置2套、防爆变频离心引风机2台、阻火器及PLC防爆控制柜交钥匙工程；"
"排放标准：非甲烷总烃排放浓度低于20mg/m3；预算上限135万元，工期45日历天，须具备环保工程专业承包二级及以上资质。"
),
),
EvidenceRecord(
source_id="industry_board_05",
source_url="https://bbs.example.org/thread-44291.html",
captured_at="2026-11-14T11:30:00+08:00",
raw_org_name="苏州精密冷链设备有限公司",
credit_code="91320500MA1987654K",
raw_excerpt="公开询价采购昆山高新区生鲜物流冷库聚氨酯保温板与双级变频冷风机组安装配套工程，总面积约1800平米，预算85万元，下周进场勘测。",
),
]
engine = SignalDeduplicator()
decisions = [engine.ingest(rec) for rec in samples]
summary = [
{
"cluster_id": c.cluster_id,
"canonical_hq_name": c.canonical_hq_name,
"branch_name": c.branch_name,
"credit_code": c.credit_code,
"scope_marker": format_scope_markers(c.scope_markers),
"simhash_hex": f"0x{c.representative_simhash:016x}",
"rep_simhash_pg_bigint": uint64_to_pg_bigint(c.representative_simhash),
"bands": SignalDeduplicator.bands_u16(c.representative_simhash),
"evidence_count": len(c.evidence_chain),
"sources": [e.source_id for e in c.evidence_chain],
"evidence_urls": [e.source_url for e in c.evidence_chain],
}
for c in engine.clusters.values()
]
print(json.dumps({"decisions": decisions, "clusters": summary}, ensure_ascii=False, indent=2))五、运行验证输出
执行 python3.11 examples/signal_entity_dedup.py 的真实终端输出如下：
json
{
"decisions": [
[
"CLU-0001",
"CREATED_NEW_CLUSTER"
],
[
"CLU-0001",
"MERGED_INTO_EXISTING_CLUSTER"
],
[
"CLU-0002",
"CREATED_NEW_CLUSTER"
],
[
"CLU-0003",
"CREATED_NEW_CLUSTER"
],
[
"CLU-0004",
"CREATED_NEW_CLUSTER"
]
],
"clusters": [
{
"cluster_id": "CLU-0001",
"canonical_hq_name": "华南智造新材料股份有限公司",
"branch_name": "东莞分公司",
"credit_code": "91440000MA5123456X",
"scope_marker": "phase=2;lot=1",
"simhash_hex": "0x71ae7abe03f1a6f9",
"rep_simhash_pg_bigint": 8191619728762054393,
"bands": [
42745,
1009,
31422,
29102
],
"evidence_count": 2,
"sources": [
"provincial_portal_01",
"municipal_mirror_02"
],
"evidence_urls": [
"https://portal-a.example.org/202611/notice_901.html",
"https://mirror-b.example.org/detail/88312.html"
]
},
{
"cluster_id": "CLU-0002",
"canonical_hq_name": "华南智造新材料股份有限公司",
"branch_name": "东莞分公司",
"credit_code": "91440000MA5123456X",
"scope_marker": "phase=2;lot=2",
"simhash_hex": "0x718e7abe02f1a6d9",
"rep_simhash_pg_bigint": 8182612529490536153,
"bands": [
42713,
753,
31422,
29070
],
"evidence_count": 1,
"sources": [
"provincial_portal_03"
],
"evidence_urls": [
"https://portal-a.example.org/202611/notice_902.html"
]
},
{
"cluster_id": "CLU-0003",
"canonical_hq_name": "深圳华南智造新材料股份有限公司",
"branch_name": null,
"credit_code": "91440300MA5987654D",
"scope_marker": "phase=2;lot=1",
"simhash_hex": "0x71ae7ebe03f5a6f9",
"rep_simhash_pg_bigint": 8191624126808827641,
"bands": [
42745,
1013,
32446,
29102
],
"evidence_count": 1,
"sources": [
"agency_bulletin_04"
],
"evidence_urls": [
"https://agency-c.example.org/bid/202611_551.html"
]
},
{
"cluster_id": "CLU-0004",
"canonical_hq_name": "苏州精密冷链设备有限公司",
"branch_name": null,
"credit_code": "91320500MA1987654K",
"scope_marker": null,
"simhash_hex": "0x9e4f2aed3553f150",
"rep_simhash_pg_bigint": -7039360494266486448,
"bands": [
61776,
13651,
10989,
40527
],
"evidence_count": 1,
"sources": [
"industry_board_05"
],
"evidence_urls": [
"https://bbs.example.org/thread-44291.html"
]
}
]
}可以看到：
- 同项目跨站转载召回后精排合并，完整保留双 URL ：省级门户首发公告（
provincial_portal_01）与地方镜像站的【转载】副本（municipal_mirror_02），将带全角括号的华南智造新材料股份有限公司（东莞分公司）与不带括号的华南智造新材料股份有限公司东莞分公司统一规范化为母公司华南智造新材料股份有限公司+branch_name = "东莞分公司"；两条指纹汉明距离仅为1（<= 3），共享band_0、band_1、band_3三个分桶召回后，经3-gramJaccard 精排（0.9301 >= 0.62）归并入CLU-0001，保留信用代码91440000MA5123456X及首发、镜像两个原始 URL； - 同业主不同标段不误并 ：同一业主发布的
二标段公告（provincial_portal_03）因沿用长招标模板，与CLU-0001汉明距离为3且共享band_2被召回（Jaccard 达0.9369），在精排阶段由标段边界守卫（phase=2;lot=2对比phase=2;lot=1）准确隔离为独立簇CLU-0002； - 不同统一社会信用代码不误并 ：另一独立法人套用同一招标代理模板发布的公告（
agency_bulletin_04，信用代码91440300MA5987654D），虽共享band_0、band_3召回且汉明距离为2、Jaccard 达0.9908，被信用代码强边界拦截为独立簇CLU-0003； - PostgreSQL 字段映射实证 ：
CLU-0001的band_0 = 42745及华东冷链询盘CLU-0004（industry_board_05）的band_0 = 61776、band_3 = 40527均大于32767（验证INTEGER分桶列必要性），且CLU-0004指纹0x9e4f2aed3553f150最高位为1，二补码映射为有符号BIGINT-7039360494266486448后可直接入库。
这套轻量二段式架构无需引入笨重的向量数据库即可在单机进程内轻松支撑十万级日增公开文本的毫秒级去重归并，非常适合对原文可追溯性要求极高的 B2B 情报工程场景。