Python爬虫实战: 爬虫常用到的技术及方案详解

爬虫是获取网络数据的重要工具，Python因其丰富的库生态系统而成为爬虫开发的首选语言。下面我将详细介绍Python爬虫的常用技术和方案。

一、基础技术栈

1. 请求库

Requests - 同步HTTP请求库

python 复制代码

import requests

# 基本GET请求
response = requests.get('https://httpbin.org/get')
print(response.status_code)
print(response.text)

# 带参数的请求
params = {'key1': 'value1', 'key2': 'value2'}
response = requests.get('https://httpbin.org/get', params=params)

# 带请求头的请求
headers = {'User-Agent': 'Mozilla/5.0'}
response = requests.get('https://httpbin.org/get', headers=headers)

# POST请求
data = {'key': 'value'}
response = requests.post('https://httpbin.org/post', data=data)

aiohttp - 异步HTTP请求库

python 复制代码

import aiohttp
import asyncio

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    async with aiohttp.ClientSession() as session:
        html = await fetch(session, 'http://example.com')
        print(html)

# Python 3.7+
asyncio.run(main())

2. 解析库

BeautifulSoup - HTML/XML解析库

python 复制代码

from bs4 import BeautifulSoup
import requests

html_doc = """
<html><head><title>The Dormouse's story</title></head>
<body>
<p class="title"><b>The Dormouse's story</b></p>
<p class="story">Once upon a time there were three little sisters...</p>
</body>
</html>
"""

soup = BeautifulSoup(html_doc, 'html.parser')

# 查找元素
title_tag = soup.title
print(title_tag.string)  # 输出: The Dormouse's story

# 查找所有p标签
p_tags = soup.find_all('p')
for tag in p_tags:
    print(tag.get('class'), tag.text)

# CSS选择器
title = soup.select_one('.title b')
print(title.text)

lxml - 高性能解析库

python 复制代码

from lxml import html
import requests

response = requests.get('http://example.com')
tree = html.fromstring(response.content)

# XPath选择元素
title = tree.xpath('//title/text()')[0]
print(title)

# CSS选择器
paragraphs = tree.cssselect('p')
for p in paragraphs:
    print(p.text_content())

pyquery - jQuery风格的解析库

python 复制代码

from pyquery import PyQuery as pq

d = pq('<html><body><p class="content">Hello World!</p></body></html>')
print(d('p.content').text())  # 输出: Hello World!

3. 数据存储

CSV文件存储

python 复制代码

import csv

data = [
    ['Name', 'Age', 'City'],
    ['Alice', '25', 'Beijing'],
    ['Bob', '30', 'Shanghai']
]

with open('data.csv', 'w', newline='', encoding='utf-8') as f:
    writer = csv.writer(f)
    writer.writerows(data)

JSON文件存储

python 复制代码

import json

data = {
    'name': 'Alice',
    'age': 25,
    'cities': ['Beijing', 'Shanghai']
}

with open('data.json', 'w', encoding='utf-8') as f:
    json.dump(data, f, ensure_ascii=False, indent=2)

数据库存储（SQLite示例）

python 复制代码

import sqlite3

# 创建数据库和表
conn = sqlite3.connect('data.db')
cursor = conn.cursor()
cursor.execute('''
CREATE TABLE IF NOT EXISTS users (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    name TEXT NOT NULL,
    age INTEGER
)
''')

# 插入数据
cursor.execute('INSERT INTO users (name, age) VALUES (?, ?)', ('Alice', 25))
conn.commit()
conn.close()

二、高级爬虫框架

1. Scrapy框架

Scrapy是一个强大的爬虫框架，提供了完整的爬虫开发解决方案。

安装Scrapy

bash 复制代码

pip install scrapy

创建Scrapy项目

bash 复制代码

scrapy startproject myproject
cd myproject
scrapy genspider example example.com

示例Spider

python 复制代码

import scrapy

class ExampleSpider(scrapy.Spider):
    name = 'example'
    allowed_domains = ['example.com']
    start_urls = ['http://example.com/']

    def parse(self, response):
        # 提取数据
        title = response.css('title::text').get()
        yield {
            'title': title,
            'url': response.url
        }
        
        # 跟踪链接
        for next_page in response.css('a::attr(href)').getall():
            yield response.follow(next_page, self.parse)

运行Scrapy爬虫

bash 复制代码

scrapy crawl example -o output.json

2. Scrapy-Redis分布式爬虫

对于大规模爬取，可以使用Scrapy-Redis实现分布式爬虫。

python 复制代码

# settings.py中添加配置
SCHEDULER = "scrapy_redis.scheduler.Scheduler"
DUPEFILTER_CLASS = "scrapy_redis.dupefilter.RFPDupeFilter"
REDIS_URL = 'redis://localhost:6379'

# Spider继承RedisSpider
from scrapy_redis.spiders import RedisSpider

class MyDistributedSpider(RedisSpider):
    name = 'distributed_spider'
    redis_key = 'myspider:start_urls'

三、反爬虫应对策略

1. User-Agent轮换

python 复制代码

import random
import requests

user_agents = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15',
    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36'
]

headers = {
    'User-Agent': random.choice(user_agents)
}

response = requests.get('https://example.com', headers=headers)

2. IP代理池

python 复制代码

import requests

proxies = {
    'http': 'http://10.10.1.10:3128',
    'https': 'http://10.10.1.10:1080',
}

response = requests.get('https://example.com', proxies=proxies)

3. 验证码处理

python 复制代码

# 使用第三方服务处理验证码，如打码平台
def solve_captcha(image_data):
    # 调用打码平台API
    # 返回识别结果
    return "captcha_text"

4. 浏览器自动化

Selenium示例

python 复制代码

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
options.add_argument('--headless')  # 无头模式
driver = webdriver.Chrome(options=options)

try:
    driver.get('https://example.com')
    # 等待元素加载
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.ID, "myDynamicElement"))
    )
    # 执行JavaScript
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    # 获取页面源码
    html = driver.page_source
finally:
    driver.quit()

四、数据处理与清洗

1. 数据清洗

python 复制代码

import re
from bs4 import BeautifulSoup

def clean_html(html):
    # 移除脚本和样式
    soup = BeautifulSoup(html, 'html.parser')
    for script in soup(["script", "style"]):
        script.decompose()
    
    # 获取文本
    text = soup.get_text()
    
    # 清理空白字符
    lines = (line.strip() for line in text.splitlines())
    chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
    text = ' '.join(chunk for chunk in chunks if chunk)
    
    return text

def extract_emails(text):
    # 使用正则表达式提取邮箱
    email_pattern = r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b'
    emails = re.findall(email_pattern, text)
    return emails

2. 数据去重

python 复制代码

# 使用布隆过滤器进行高效去重
from pybloom_live import BloomFilter

# 创建布隆过滤器
bf = BloomFilter(capacity=100000, error_rate=0.001)

urls_to_crawl = ['http://example.com/1', 'http://example.com/2']
for url in urls_to_crawl:
    if url not in bf:
        bf.add(url)
        # 爬取这个URL

五、爬虫架构设计

1. 基本爬虫架构

python 复制代码

爬虫调度器 → URL管理器 → 网页下载器 → 网页解析器 → 数据存储器

2. 分布式爬虫架构

python 复制代码

主节点(调度) → Redis(URL队列) → 多个爬虫节点 → 数据库/文件存储

六、法律与道德考虑

遵守robots.txt：尊重网站的爬虫协议
控制请求频率：避免对目标网站造成过大压力
尊重版权：不随意传播抓取的受版权保护内容
隐私保护：不抓取和传播个人隐私信息

七、最佳实践

设置合理的请求间隔

python 复制代码

import time
import random

def crawl_with_delay(url):
    response = requests.get(url)
    # 随机延迟1-3秒
    time.sleep(random.uniform(1, 3))
    return response

使用会话保持

python 复制代码

session = requests.Session()
session.headers.update({'User-Agent': 'My Crawler'})
response = session.get('https://example.com')

错误处理与重试

python 复制代码

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=4, max=10))
def request_with_retry(url):
    return requests.get(url, timeout=5)

日志记录

python 复制代码

import logging

logging.basicConfig(
    level=logging.INFO,
    format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)

logger = logging.getLogger(__name__)
logger.info('Starting crawler')

总结

Python爬虫技术涵盖了从简单的网页请求到复杂的分布式系统设计。选择合适的工具和技术取决于你的具体需求：

对于简单任务：Requests + BeautifulSoup/Lxml
对于复杂项目：Scrapy框架
对于JavaScript渲染的页面：Selenium/Playwright
对于大规模爬取：分布式架构+代理池

无论使用哪种技术，都应始终遵守法律法规和道德准则，尊重网站的资源和使用条款。