凌晨 3 点，一条评论出现在目标账号的评论区："你们这个多少钱一平？家里 120 平，想做全屋定制。"这条评论的存活时间只有几十分钟，如果没人接住，用户可能已经去问下一家了。
评论区获客的工程问题很简单：信息采集要快、要稳、要防封。本文记录一套评论区 RPA 采集与处理的工程实践。
01 评论区采集的三个工程难点
做过评论区抓取的人都会遇到三个问题。
翻页与增量。评论区是瀑布流加载，采集需要模拟滚动，还要记住已采集位置，避免重复抓取。增量策略是记录每账号每条评论的唯一ID，每次任务只拉取ID列表里最新的部分。
频率与封禁。同一账号短时间高频访问会被风控。实测某平台单账号访问频率超过每分钟20次后，开始出现验证码和临时封禁。
反爬对抗。页面结构会不定期调整，选择器失效是常态。采集代码要设计成可配置的选择器，而不是写死。
02 采集模块的整体设计
采集模块拆成三层，各层职责分离：
| 层 | 职责 | 关键点 |
|---|---|---|
| 调度层 | 分配采集任务、控制频率 | 每账号维护独立限速器 |
| 采集层 | 执行页面操作、解析数据 | 选择器可配置，支持热更新 |
| 存储层 | 写入队列、去重 | 评论ID为主键，幂等写入 |
调度层用令牌桶限速，每账号每60秒放行一次采集窗口，窗口内最多拉取一页。频率参数可配置，新账号从保守值起步，观察无风控信号后再调高。
采集层基于浏览器自动化框架封装，页面操作统一走"打开评论区、滚动到底、解析DOM、提取结构化字段"四步。结构化字段包括：评论ID、用户ID、正文、时间戳、图片URL、回复数。
采集模块的整体流程图如下：
#mermaid-svg-hrSmsvdcb9pCrrYR{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-hrSmsvdcb9pCrrYR .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-hrSmsvdcb9pCrrYR .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-hrSmsvdcb9pCrrYR .error-icon{fill:#552222;}#mermaid-svg-hrSmsvdcb9pCrrYR .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-hrSmsvdcb9pCrrYR .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-hrSmsvdcb9pCrrYR .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-hrSmsvdcb9pCrrYR .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-hrSmsvdcb9pCrrYR .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-hrSmsvdcb9pCrrYR .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-hrSmsvdcb9pCrrYR .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-hrSmsvdcb9pCrrYR .marker{fill:#333333;stroke:#333333;}#mermaid-svg-hrSmsvdcb9pCrrYR .marker.cross{stroke:#333333;}#mermaid-svg-hrSmsvdcb9pCrrYR svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-hrSmsvdcb9pCrrYR p{margin:0;}#mermaid-svg-hrSmsvdcb9pCrrYR .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-hrSmsvdcb9pCrrYR .cluster-label text{fill:#333;}#mermaid-svg-hrSmsvdcb9pCrrYR .cluster-label span{color:#333;}#mermaid-svg-hrSmsvdcb9pCrrYR .cluster-label span p{background-color:transparent;}#mermaid-svg-hrSmsvdcb9pCrrYR .label text,#mermaid-svg-hrSmsvdcb9pCrrYR span{fill:#333;color:#333;}#mermaid-svg-hrSmsvdcb9pCrrYR .node rect,#mermaid-svg-hrSmsvdcb9pCrrYR .node circle,#mermaid-svg-hrSmsvdcb9pCrrYR .node ellipse,#mermaid-svg-hrSmsvdcb9pCrrYR .node polygon,#mermaid-svg-hrSmsvdcb9pCrrYR .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-hrSmsvdcb9pCrrYR .rough-node .label text,#mermaid-svg-hrSmsvdcb9pCrrYR .node .label text,#mermaid-svg-hrSmsvdcb9pCrrYR .image-shape .label,#mermaid-svg-hrSmsvdcb9pCrrYR .icon-shape .label{text-anchor:middle;}#mermaid-svg-hrSmsvdcb9pCrrYR .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-hrSmsvdcb9pCrrYR .rough-node .label,#mermaid-svg-hrSmsvdcb9pCrrYR .node .label,#mermaid-svg-hrSmsvdcb9pCrrYR .image-shape .label,#mermaid-svg-hrSmsvdcb9pCrrYR .icon-shape .label{text-align:center;}#mermaid-svg-hrSmsvdcb9pCrrYR .node.clickable{cursor:pointer;}#mermaid-svg-hrSmsvdcb9pCrrYR .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-hrSmsvdcb9pCrrYR .arrowheadPath{fill:#333333;}#mermaid-svg-hrSmsvdcb9pCrrYR .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-hrSmsvdcb9pCrrYR .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-hrSmsvdcb9pCrrYR .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hrSmsvdcb9pCrrYR .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-hrSmsvdcb9pCrrYR .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hrSmsvdcb9pCrrYR .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-hrSmsvdcb9pCrrYR .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-hrSmsvdcb9pCrrYR .cluster text{fill:#333;}#mermaid-svg-hrSmsvdcb9pCrrYR .cluster span{color:#333;}#mermaid-svg-hrSmsvdcb9pCrrYR div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-hrSmsvdcb9pCrrYR .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-hrSmsvdcb9pCrrYR rect.text{fill:none;stroke-width:0;}#mermaid-svg-hrSmsvdcb9pCrrYR .icon-shape,#mermaid-svg-hrSmsvdcb9pCrrYR .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-hrSmsvdcb9pCrrYR .icon-shape p,#mermaid-svg-hrSmsvdcb9pCrrYR .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-hrSmsvdcb9pCrrYR .icon-shape .label rect,#mermaid-svg-hrSmsvdcb9pCrrYR .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-hrSmsvdcb9pCrrYR .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-hrSmsvdcb9pCrrYR .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-hrSmsvdcb9pCrrYR :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 存储层
评论ID去重
幂等写入
采集层
打开评论区
滚动到底
解析DOM
提取结构化字段
调度层
分配采集任务
令牌桶限速
下面是采集层四步流程的 Python 实战示例，选择器集中配置，页面结构变化时只需更新 SELECTORS 字典：
python
import time
import random
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# 选择器集中配置：页面改版时只改这里，提取逻辑保持不变
SELECTORS = {
"comment_item": "div.comment-item", # 单条评论容器
"comment_id": ".comment-item", # 评论 ID，走 data 属性
"user_id": ".user-info .user-id", # 用户 ID
"content": ".comment-content", # 评论正文
"time": ".comment-time", # 发布时间
"images": ".comment-images img", # 评论图片
"reply_count": ".reply-count", # 回复数
}
MAX_SCROLL_ROUNDS = 20 # 最大滚动轮数，防止无限循环
SCROLL_WAIT_SECONDS = (0.8, 2.0) # 滚动后随机等待，模拟阅读行为
def open_comment_area(driver, url):
"""打开目标页面并进入评论区。"""
driver.get(url)
wait = WebDriverWait(driver, 15)
# 等待评论区容器出现，确认页面评论模块已加载
wait.until(
EC.presence_of_element_located((By.CSS_SELECTOR, SELECTORS["comment_item"]))
)
def scroll_to_bottom(driver):
"""滚动到底部并等待懒加载，返回是否到达底部。"""
no_change_rounds = 0
last_height = 0
for _ in range(MAX_SCROLL_ROUNDS):
current_height = driver.execute_script(
"return document.body.scrollHeight;"
)
# 如果连续两轮高度都不变，说明评论区已无新内容
if current_height == last_height:
no_change_rounds += 1
if no_change_rounds >= 2:
return True
else:
no_change_rounds = 0
last_height = current_height
# 触发懒加载：滚动到当前页面底部
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(random.uniform(*SCROLL_WAIT_SECONDS))
return False
def _safe_text(element):
"""缺失字段返回空字符串，避免单条评论异常中断整个任务。"""
try:
return element.text.strip()
except Exception:
return ""
def extract_fields(item):
"""从单个评论容器中提取结构化字段。"""
comment_id = item.get_attribute("data-cid") or ""
user_id = item.get_attribute("data-uid") or ""
content_el = item.find_elements(By.CSS_SELECTOR, SELECTORS["content"])
content = _safe_text(content_el[0]) if content_el else ""
time_el = item.find_elements(By.CSS_SELECTOR, SELECTORS["time"])
timestamp = time_el[0].get_attribute("datetime") if time_el else ""
if not timestamp and time_el:
timestamp = _safe_text(time_el[0])
# 图片 URL 可能有多张，先取全部，避免只取第一张导致数据丢失
image_els = item.find_elements(By.CSS_SELECTOR, SELECTORS["images"])
image_urls = [
img.get_attribute("src") for img in image_els if img.get_attribute("src")
]
reply_els = item.find_elements(By.CSS_SELECTOR, SELECTORS["reply_count"])
reply_text = _safe_text(reply_els[0]) if reply_els else "0"
try:
reply_count = int(reply_text)
except ValueError:
reply_count = 0
return {
"comment_id": comment_id,
"user_id": user_id,
"content": content,
"timestamp": timestamp,
"image_urls": image_urls,
"reply_count": reply_count,
}
def collect_comments(driver):
"""滚动到底后解析 DOM，并提取全部评论字段。"""
scroll_to_bottom(driver)
items = driver.find_elements(By.CSS_SELECTOR, SELECTORS["comment_item"])
return [extract_fields(item) for item in items]
if __name__ == "__main__":
driver = webdriver.Chrome() # 实际项目可替换为无头模式
try:
open_comment_area(driver, "https://example.com/video/123")
comments = collect_comments(driver)
print(comments[:3]) # 预览前三条结构化结果
finally:
driver.quit()03 防封与频率控制的实测参数
防封的核心不是"慢"，而是"像人"。
实测一组频率参数的效果：
| 策略 | 每分钟请求 | 连续运行时长 | 触发风控 |
|---|---|---|---|
| 无限制 | 60+ | 约40分钟 | 触发验证码 |
| 固定间隔 | 10 | 约3小时 | 触发临时封禁 |
| 随机间隔+行为模拟 | 8-12随机 | 48小时+ | 未触发 |
固定间隔依然会被识别------风控模型检测的是行为的规律性，不是频率高低。随机间隔加行为模拟（滚动、悬停、随机阅读时长）才能稳定运行。
另一个关键参数是账号的日采集上限。实测单账号日采集超过 800 条后，次日大概率被要求验证。控制在 500 条以内，账号长期稳定。
04 异常处理与断点续采
采集中断是常态，不是异常。断网、验证码、页面改版，任何一步都可能挂掉。设计上必须支持断点续采。
实现方案：
python
def run_with_resume(task):
checkpoint = load_checkpoint(task.id) # 断点状态
cursor = checkpoint.get("cursor", 0)
while cursor < task.total:
try:
page = fetch_page(task, cursor)
except CaptchaError:
wait_for_human() # 触发验证码，等待人工处理
continue
except FrequencyLimit:
backoff(task.account) # 频率限制，指数退避
continue
save_batch(page) # 每页落库后再推进游标
save_checkpoint(task.id, cursor + page.size)
cursor += page.size异常处理与断点续采的流程如下：
#mermaid-svg-UKo5PJLskcjlndNl{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-UKo5PJLskcjlndNl .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-UKo5PJLskcjlndNl .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-UKo5PJLskcjlndNl .error-icon{fill:#552222;}#mermaid-svg-UKo5PJLskcjlndNl .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-UKo5PJLskcjlndNl .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-UKo5PJLskcjlndNl .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-UKo5PJLskcjlndNl .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-UKo5PJLskcjlndNl .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-UKo5PJLskcjlndNl .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-UKo5PJLskcjlndNl .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-UKo5PJLskcjlndNl .marker{fill:#333333;stroke:#333333;}#mermaid-svg-UKo5PJLskcjlndNl .marker.cross{stroke:#333333;}#mermaid-svg-UKo5PJLskcjlndNl svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-UKo5PJLskcjlndNl p{margin:0;}#mermaid-svg-UKo5PJLskcjlndNl .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-UKo5PJLskcjlndNl .cluster-label text{fill:#333;}#mermaid-svg-UKo5PJLskcjlndNl .cluster-label span{color:#333;}#mermaid-svg-UKo5PJLskcjlndNl .cluster-label span p{background-color:transparent;}#mermaid-svg-UKo5PJLskcjlndNl .label text,#mermaid-svg-UKo5PJLskcjlndNl span{fill:#333;color:#333;}#mermaid-svg-UKo5PJLskcjlndNl .node rect,#mermaid-svg-UKo5PJLskcjlndNl .node circle,#mermaid-svg-UKo5PJLskcjlndNl .node ellipse,#mermaid-svg-UKo5PJLskcjlndNl .node polygon,#mermaid-svg-UKo5PJLskcjlndNl .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-UKo5PJLskcjlndNl .rough-node .label text,#mermaid-svg-UKo5PJLskcjlndNl .node .label text,#mermaid-svg-UKo5PJLskcjlndNl .image-shape .label,#mermaid-svg-UKo5PJLskcjlndNl .icon-shape .label{text-anchor:middle;}#mermaid-svg-UKo5PJLskcjlndNl .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-UKo5PJLskcjlndNl .rough-node .label,#mermaid-svg-UKo5PJLskcjlndNl .node .label,#mermaid-svg-UKo5PJLskcjlndNl .image-shape .label,#mermaid-svg-UKo5PJLskcjlndNl .icon-shape .label{text-align:center;}#mermaid-svg-UKo5PJLskcjlndNl .node.clickable{cursor:pointer;}#mermaid-svg-UKo5PJLskcjlndNl .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-UKo5PJLskcjlndNl .arrowheadPath{fill:#333333;}#mermaid-svg-UKo5PJLskcjlndNl .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-UKo5PJLskcjlndNl .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-UKo5PJLskcjlndNl .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-UKo5PJLskcjlndNl .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-UKo5PJLskcjlndNl .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-UKo5PJLskcjlndNl .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-UKo5PJLskcjlndNl .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-UKo5PJLskcjlndNl .cluster text{fill:#333;}#mermaid-svg-UKo5PJLskcjlndNl .cluster span{color:#333;}#mermaid-svg-UKo5PJLskcjlndNl div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-UKo5PJLskcjlndNl .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-UKo5PJLskcjlndNl rect.text{fill:none;stroke-width:0;}#mermaid-svg-UKo5PJLskcjlndNl .icon-shape,#mermaid-svg-UKo5PJLskcjlndNl .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-UKo5PJLskcjlndNl .icon-shape p,#mermaid-svg-UKo5PJLskcjlndNl .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-UKo5PJLskcjlndNl .icon-shape .label rect,#mermaid-svg-UKo5PJLskcjlndNl .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-UKo5PJLskcjlndNl .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-UKo5PJLskcjlndNl .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-UKo5PJLskcjlndNl :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 否
是
CaptchaError
FrequencyLimit
无异常
加载 checkpoint
获取游标 cursor
游标是否未到总数
任务完成
fetch_page 拉取一页
是否异常
暂停任务
推送人工队列
指数退避 backoff
save_batch 落库
推进游标
保存 checkpoint
关键点：游标只在数据落库之后推进。如果数据还没写库就推进游标，中断后会丢数据；如果写库后不推进，中断后会重复采集。以"落库即推进"为原则，采集是幂等可恢复的。
验证码处理单独设计：检测到验证码后暂停该账号全部任务，推送人工队列，人工处理后恢复。不要尝试自动过验证码，风险不可控。
05 采集与判定模块的衔接
采集模块和意图判定模块之间用消息队列解耦。采集层只负责"把评论变成结构化数据"，判定层只消费队列里的数据，互不阻塞。
采集与判定模块的衔接关系如下：
#mermaid-svg-SKnfVablwctDCnNk{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;fill:#333;}@keyframes edge-animation-frame{from{stroke-dashoffset:0;}}@keyframes dash{to{stroke-dashoffset:0;}}#mermaid-svg-SKnfVablwctDCnNk .edge-animation-slow{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 50s linear infinite;stroke-linecap:round;}#mermaid-svg-SKnfVablwctDCnNk .edge-animation-fast{stroke-dasharray:9,5!important;stroke-dashoffset:900;animation:dash 20s linear infinite;stroke-linecap:round;}#mermaid-svg-SKnfVablwctDCnNk .error-icon{fill:#552222;}#mermaid-svg-SKnfVablwctDCnNk .error-text{fill:#552222;stroke:#552222;}#mermaid-svg-SKnfVablwctDCnNk .edge-thickness-normal{stroke-width:1px;}#mermaid-svg-SKnfVablwctDCnNk .edge-thickness-thick{stroke-width:3.5px;}#mermaid-svg-SKnfVablwctDCnNk .edge-pattern-solid{stroke-dasharray:0;}#mermaid-svg-SKnfVablwctDCnNk .edge-thickness-invisible{stroke-width:0;fill:none;}#mermaid-svg-SKnfVablwctDCnNk .edge-pattern-dashed{stroke-dasharray:3;}#mermaid-svg-SKnfVablwctDCnNk .edge-pattern-dotted{stroke-dasharray:2;}#mermaid-svg-SKnfVablwctDCnNk .marker{fill:#333333;stroke:#333333;}#mermaid-svg-SKnfVablwctDCnNk .marker.cross{stroke:#333333;}#mermaid-svg-SKnfVablwctDCnNk svg{font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:16px;}#mermaid-svg-SKnfVablwctDCnNk p{margin:0;}#mermaid-svg-SKnfVablwctDCnNk .label{font-family:"trebuchet ms",verdana,arial,sans-serif;color:#333;}#mermaid-svg-SKnfVablwctDCnNk .cluster-label text{fill:#333;}#mermaid-svg-SKnfVablwctDCnNk .cluster-label span{color:#333;}#mermaid-svg-SKnfVablwctDCnNk .cluster-label span p{background-color:transparent;}#mermaid-svg-SKnfVablwctDCnNk .label text,#mermaid-svg-SKnfVablwctDCnNk span{fill:#333;color:#333;}#mermaid-svg-SKnfVablwctDCnNk .node rect,#mermaid-svg-SKnfVablwctDCnNk .node circle,#mermaid-svg-SKnfVablwctDCnNk .node ellipse,#mermaid-svg-SKnfVablwctDCnNk .node polygon,#mermaid-svg-SKnfVablwctDCnNk .node path{fill:#ECECFF;stroke:#9370DB;stroke-width:1px;}#mermaid-svg-SKnfVablwctDCnNk .rough-node .label text,#mermaid-svg-SKnfVablwctDCnNk .node .label text,#mermaid-svg-SKnfVablwctDCnNk .image-shape .label,#mermaid-svg-SKnfVablwctDCnNk .icon-shape .label{text-anchor:middle;}#mermaid-svg-SKnfVablwctDCnNk .node .katex path{fill:#000;stroke:#000;stroke-width:1px;}#mermaid-svg-SKnfVablwctDCnNk .rough-node .label,#mermaid-svg-SKnfVablwctDCnNk .node .label,#mermaid-svg-SKnfVablwctDCnNk .image-shape .label,#mermaid-svg-SKnfVablwctDCnNk .icon-shape .label{text-align:center;}#mermaid-svg-SKnfVablwctDCnNk .node.clickable{cursor:pointer;}#mermaid-svg-SKnfVablwctDCnNk .root .anchor path{fill:#333333!important;stroke-width:0;stroke:#333333;}#mermaid-svg-SKnfVablwctDCnNk .arrowheadPath{fill:#333333;}#mermaid-svg-SKnfVablwctDCnNk .edgePath .path{stroke:#333333;stroke-width:2.0px;}#mermaid-svg-SKnfVablwctDCnNk .flowchart-link{stroke:#333333;fill:none;}#mermaid-svg-SKnfVablwctDCnNk .edgeLabel{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-SKnfVablwctDCnNk .edgeLabel p{background-color:rgba(232,232,232, 0.8);}#mermaid-svg-SKnfVablwctDCnNk .edgeLabel rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-SKnfVablwctDCnNk .labelBkg{background-color:rgba(232, 232, 232, 0.5);}#mermaid-svg-SKnfVablwctDCnNk .cluster rect{fill:#ffffde;stroke:#aaaa33;stroke-width:1px;}#mermaid-svg-SKnfVablwctDCnNk .cluster text{fill:#333;}#mermaid-svg-SKnfVablwctDCnNk .cluster span{color:#333;}#mermaid-svg-SKnfVablwctDCnNk div.mermaidTooltip{position:absolute;text-align:center;max-width:200px;padding:2px;font-family:"trebuchet ms",verdana,arial,sans-serif;font-size:12px;background:hsl(80, 100%, 96.2745098039%);border:1px solid #aaaa33;border-radius:2px;pointer-events:none;z-index:100;}#mermaid-svg-SKnfVablwctDCnNk .flowchartTitleText{text-anchor:middle;font-size:18px;fill:#333;}#mermaid-svg-SKnfVablwctDCnNk rect.text{fill:none;stroke-width:0;}#mermaid-svg-SKnfVablwctDCnNk .icon-shape,#mermaid-svg-SKnfVablwctDCnNk .image-shape{background-color:rgba(232,232,232, 0.8);text-align:center;}#mermaid-svg-SKnfVablwctDCnNk .icon-shape p,#mermaid-svg-SKnfVablwctDCnNk .image-shape p{background-color:rgba(232,232,232, 0.8);padding:2px;}#mermaid-svg-SKnfVablwctDCnNk .icon-shape .label rect,#mermaid-svg-SKnfVablwctDCnNk .image-shape .label rect{opacity:0.5;background-color:rgba(232,232,232, 0.8);fill:rgba(232,232,232, 0.8);}#mermaid-svg-SKnfVablwctDCnNk .label-icon{display:inline-block;height:1em;overflow:visible;vertical-align:-0.125em;}#mermaid-svg-SKnfVablwctDCnNk .node .label-icon path{fill:currentColor;stroke:revert;stroke-width:revert;}#mermaid-svg-SKnfVablwctDCnNk :root{--mermaid-font-family:"trebuchet ms",verdana,arial,sans-serif;} 评论ID去重
失败重试3次
指数退避
仍失败
采集模块
结构化数据
消息队列
判定模块
意图判定
死信队列
人工处理
衔接上的两个细节：
去重在采集侧做，判定侧不重复去重。评论 ID 在主键写入时去重，判定模块消费时假设数据已去重。
失败重试在消费侧做。判定接口偶发超时，消费端设置重试 3 次、指数退避，重试仍失败则进入死信队列人工处理。实测判定接口的可用性从 99.2% 提升到 99.8%。
最后说两句
评论区 RPA 的稳定性工程，优先级高于功能工程。一个能稳定跑48小时的采集模块，价值远大于一个功能全但三天两头挂的模块。断点续采、随机行为模拟、落库即推进，这三件事做好，评论区获客的采集侧就稳了。