下面以"抓取 Bing 搜索结果"为例，写一段根据关键词抓取的 Python 代码。可以把 URL 和解析规则换成你要抓取的目标网站。
安装依赖
```bash
pip install requests beautifulsoup4
```
示例代码
```python
import time
import csv
import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin
HEADERS = {
"User-Agent": (
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/124.0 Safari/537.36"
),
"Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8",
}
def scrape_bing(keyword: str, pages: int = 1, delay: float = 1.0):
"""
根据关键词抓取 Bing 搜索结果。
pages: 抓取页数
delay: 每页请求间隔，避免请求过快
"""
results = \[\]
for page in range(pages):
first = page * 10 + 1 # Bing 每页约 10 条
url = "https://www.bing.com/search"
params = {
"q": keyword,
"first": first,
}
try:
resp = requests.get(
url,
params=params,
headers=HEADERS,
timeout=15
)
resp.raise_for_status()
except requests.RequestException as e:
print(f"第 {page + 1} 页请求失败：{e}")
continue
soup = BeautifulSoup(resp.text, "html.parser")
items = soup.select("li.b_algo")
if not items:
print(f"第 {page + 1} 页未解析到结果，可能被反爬或页面结构已变化")
continue
for item in items:
a = item.select_one("h2 a")
if not a:
continue
title = a.get_text(" ", strip=True)
link = a.get("href", "")
link = urljoin("https://www.bing.com", link)
summary_tag = (
item.select_one(".b_caption p")
or item.select_one(".b_lineclamp2")
or item.select_one(".b_algoSlug")
)
summary = summary_tag.get_text(" ", strip=True) if summary_tag else ""
results.append({
"title": title,
"link": link,
"summary": summary,
})
time.sleep(delay)
return results
def save_to_csv(data, filename="results.csv"):
"""保存为 CSV 文件"""
if not data:
print("没有数据可保存")
return
with open(filename, "w", newline="", encoding="utf-8-sig") as f:
writer = csv.DictWriter(
f,
fieldnames="title", "link", "summary"
)
writer.writeheader()
writer.writerows(data)
print(f"已保存到 {filename}")
if name == "main":
keyword = input("请输入关键词：").strip()
if not keyword:
print("关键词不能为空")
else:
data = scrape_bing(keyword, pages=1)
for i, item in enumerate(data, 1):
print(f"{i}. {item'title'}")
print(f" 链接：{item'link'}")
print(f" 摘要：{item'summary'}")
print()
save_to_csv(data)
```
说明
· 该代码会访问 Bing 搜索页，提取标题、链接和摘要。
· 如果要抓取其他网站，只需要修改：
· 请求 URL
· 请求参数
· soup.select(...) 中的 CSS 选择器
· 如果目标网站需要登录、验证码或 JavaScript 渲染，requests 可能抓不到，需要改用 Selenium、Playwright 或官方 API。
· 请遵守目标网站的 robots.txt、服务条款和当地法律，不要高频请求，不要抓取隐私或非公开数据。
此处为参考学习用。