1. 选型依据
PDD 公开数据分布在移动端 API（mobile.yangkeduo.com）与 H5（mobile.pinduoduo.com）。当采集规模上升，手写 requests + 线程池在三个方面迅速失效：
- 调度：限流、重试、去重需自行实现，易误伤出口 IP；
- 对抗耦合：签名、UA、Cookie、代理切换逻辑与业务混杂，难以演进；
- 可观测：缺乏统一日志、统计与失败队列。
Scrapy 以「中间件 + 调度器 + Pipeline」的声明式结构内建上述能力。本框架的核心设计原则是 将反爬对抗逻辑全部收敛至 Downloader/Spider Middleware，业务 Spider 仅负责请求编排与解析。
轻量一次性抓取用
httpx + parsel即可；需限流、代理池与长期运行的采集，Scrapy 为更优解。
2. 反爬机制与应对矩阵
| 维度 | 表现 | 应对 |
|---|---|---|
| 请求签名 | 移动端强制 anti_content（设备指纹 + 时间戳 + 请求体哈希） |
签名算法中间件化，或接入合规签名 SDK |
| 频率限制 | 同 IP/设备高频触发 460/限流 | 代理 IP 池 + AUTOTHROTTLE |
| 设备指纹 | client_info/device_id 行为关联 |
多套设备参数轮换 |
| 行为验证 | 风险后返回滑块/验证码 | 降速、换出口 IP、降并发 |
| 登录态 | 部分接口需有效 Cookie | 账号池 + Cookie 持久化 |
anti_content 是移动端第一道校验门。社区已有开源还原实现，但随版本迭代维护成本高；生产环境建议将其封装为独立签名服务，由中间件调用，隔离算法失效风险。
3. 框架结构
plain
scrapy-pinduoduo/
├── scrapy_pinduoduo/
│ ├── settings.py
│ ├── items.py # dataclass 数据模型
│ ├── middlewares/
│ │ ├── proxy_middleware.py # 隧道代理（亿牛云）
│ │ ├── sign_middleware.py # anti_content 注入
│ │ └── ua_middleware.py # UA/设备指纹轮换
│ ├── spiders/
│ │ ├── goods_search.py
│ │ └── goods_detail.py
│ └── pipelines/
│ └── save_pipeline.py
├── tests/
├── pyproject.toml
└── scrapy.cfg依赖隔离使用 uv，对抗逻辑与业务 Spider 解耦，更换签名方案或代理供应商时仅改对应 middleware。
4. 初始化
bash
uv venv && source .venv/bin/activate
uv add scrapy pydantic httpx
scrapy genspider goods_search mobile.yangkeduo.compyproject.toml：
toml
[project]
name = "scrapy-pinduoduo"
version = "0.1.0"
requires-python = ">=3.10"
dependencies = ["scrapy>=2.11", "pydantic>=2.0", "httpx>=0.27"]5. 商品搜索 Spider
scrapy_pinduoduo/spiders/goods_search.py：
python
from __future__ import annotations
import scrapy
from scrapy_pinduoduo.items import GoodsItem
class GoodsSearchSpider(scrapy.Spider):
"""拼多多商品搜索爬虫（移动端 H5 接口，分页采集）。"""
name = "goods_search"
allowed_domains = ["mobile.yangkeduo.com"]
api_template = "https://mobile.yangkeduo.com/proxy/api/api/aristotle/search?q={keyword}&page={page}"
custom_settings = {
"DOWNLOAD_DELAY": 1.5,
"CONCURRENT_REQUESTS": 4,
"AUTOTHROTTLE_ENABLED": True,
}
def __init__(self, keyword: str = "手机壳", max_page: int = 10, *args, **kwargs):
super().__init__(*args, **kwargs)
self.keyword = keyword
self.max_page = int(max_page)
def start_requests(self) -> scrapy.Request:
for page in range(1, self.max_page + 1):
url = self.api_template.format(keyword=self.keyword, page=page)
yield scrapy.Request(
url,
callback=self.parse,
meta={"need_sign": True, "page": page},
errback=self.on_error,
)
def parse(self, response: scrapy.http.Response):
data = response.json()
items = data.get("goods_list", []) or data.get("items", [])
for row in items:
yield GoodsItem(
goods_id=row.get("goods_id"),
title=row.get("goods_name") or row.get("title"),
price=row.get("price") / 100 if row.get("price") else None,
sales=row.get("sales_num") or row.get("cnt"),
shop_id=row.get("mall_id"),
page=response.meta.get("page"),
)
def on_error(self, failure):
self.logger.error("请求失败: %s | %s", failure.request.url, failure.value)items.py：
python
from __future__ import annotations
from dataclasses import dataclass
@dataclass
class GoodsItem:
goods_id: int | None
title: str | None
price: float | None
sales: int | None
shop_id: int | None
page: int | None6. 代理中间件
PDD 对出口 IP 敏感，规模化采集依赖代理池打散 IP 维度。本例采用 亿牛云代理 隧道代理------服务端按请求轮询出口 IP，无需自维护 IP 池，契合 Scrapy 每请求换 IP 的高并发模型。
scrapy_pinduoduo/middlewares/proxy_middleware.py：
python
from __future__ import annotations
import base64
from scrapy import signals
from scrapy.http import Request, Response
class YiniuProxyMiddleware:
"""亿牛云隧道代理中间件：固定入口 + Basic 鉴权，按请求自动换 IP。"""
PROXY_URL = "http://tunnel.16yun.cn:10000"
PROXY_AUTH = "用户名:密码" # 替换为亿牛云后台鉴权信息
@classmethod
def from_crawler(cls, crawler):
m = cls()
crawler.signals.connect(m.spider_opened, signal=signals.spider_opened)
return m
def process_request(self, request: Request, spider) -> None:
if request.meta.get("use_proxy", True):
request.meta["proxy"] = self.PROXY_URL
token = base64.b64encode(self.PROXY_AUTH.encode()).decode()
request.headers["Proxy-Authorization"] = f"Basic {token}"
def process_response(self, request: Request, response: Response, spider):
if response.status in (429, 460, 503):
spider.logger.warning("代理限流 %s，触发重试", response.status)
request.dont_filter = True
return request
return response
def spider_opened(self, spider) -> None:
spider.logger.info("亿牛云代理中间件已启用")settings.py 中间件注册（数值为执行优先级）：
python
DOWNLOADER_MIDDLEWARES = {
"scrapy_pinduoduo.middlewares.ua_middleware.UAMiddleware": 300,
"scrapy_pinduoduo.middlewares.proxy_middleware.YiniuProxyMiddleware": 350,
"scrapy_pinduoduo.middlewares.sign_middleware.SignMiddleware": 400,
}隧道代理相较私密 IP 池的优势在于内置轮询，规避因轮换策略不当导致的 IP 复用封禁；中小规模采集性价比最高。
7. 签名中间件
scrapy_pinduoduo/middlewares/sign_middleware.py：
python
from __future__ import annotations
from scrapy import Request
class SignMiddleware:
"""anti_content 注入。签名实现下沉至独立服务/SDK，本类仅做调用。"""
def process_request(self, request: Request, spider) -> None:
if not request.meta.get("need_sign"):
return
body = request.body.decode() if request.body else ""
request.headers["anti-content"] = self.generate_anti_content(request.url, body)
@staticmethod
def generate_anti_content(url: str, body: str) -> str:
raise NotImplementedError("接入 anti_content 签名实现（开源还原或合规 SDK）")8. 落盘 Pipeline
scrapy_pinduoduo/pipelines/save_pipeline.py：
python
from __future__ import annotations
import csv
import json
from pathlib import Path
from scrapy_pinduoduo.items import GoodsItem
class SavePipeline:
"""双格式落盘：JSON Lines（流式入库）+ CSV（运营分析）。"""
def open_spider(self, spider):
self.out_dir = Path("output")
self.out_dir.mkdir(exist_ok=True)
self.jsonl = open(self.out_dir / f"{spider.name}.jsonl", "w", encoding="utf-8")
self.csv_file = open(self.out_dir / f"{spider.name}.csv", "w", encoding="utf-8-sig", newline="")
self.writer = csv.DictWriter(
self.csv_file,
fieldnames=["goods_id", "title", "price", "sales", "shop_id", "page"],
)
self.writer.writeheader()
def close_spider(self, spider):
self.jsonl.close()
self.csv_file.close()
def process_item(self, item: GoodsItem, spider) -> GoodsItem:
row = item.__dict__
self.jsonl.write(json.dumps(row, ensure_ascii=False) + "\n")
self.writer.writerow(row)
return item9. 限速与重试
settings.py：
python
CONCURRENT_REQUESTS = 8
DOWNLOAD_DELAY = 1.0
RANDOMIZE_DOWNLOAD_DELAY = True # 抖动延迟，规避固定节奏
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 1.0
AUTOTHROTTLE_MAX_DELAY = 10.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 4.0
RETRY_ENABLED = True
RETRY_TIMES = 3
RETRY_HTTP_CODES = [429, 460, 500, 502, 503, 504]
DUPEFILTER_CLASS = "scrapy.dupefilters.RFPDupeFilter"10. 分布式扩展
规模超出单机时：
scrapy-redis：start_urls与去重指纹下沉至 Redis，多节点协同消费；- 能力下沉：签名服务、代理网关独立为 HTTP 微服务，Spider 无状态水平扩容。
注意：PDD 对「同设备指纹的多机并发」同样聚类风控，设备参数池须随节点数线性扩张，否则扩机器反而加速封禁。