关于scrapy模块中setting.py文件的介绍

作用

在Scrapy框架中,settings.py 文件起着非常重要的作用,它用于配置和控制整个Scrapy爬虫项目的行为、性能和功能。

setting.py文件的介绍

python 复制代码
# Scrapy settings for haodaifu project
#
# For simplicity, this file contains only settings considered important or
# commonly used. You can find more settings consulting the documentation:
#
#     https://docs.scrapy.org/en/latest/topics/settings.html
#     https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
#     https://docs.scrapy.org/en/latest/topics/spider-middleware.html

# 设置爬虫的名称
BOT_NAME = "spidername"

# 指定包含的爬虫代码的模块
SPIDER_MODULES = ["spidername.spiders"]
NEWSPIDER_MODULE = "spidername.spiders"

# 设置用户代理,用于模拟浏览器或特定的爬虫身份
# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"

# 配置爬虫是否遵循robots.txt柜子
# Obey robots.txt rules
ROBOTSTXT_OBEY = False

# 控制并发请求的数量(默认为16)
# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 64

# 设置下载延迟,控制请求之间的时间间隔,以避免对目标服务器造成过大负载
# Configure a delay for requests for the same website (default: 0)
# See https://docs.scrapy.org/en/latest/topics/settings.html#download-delay
# See also autothrottle settings and docs
DOWNLOAD_DELAY = 3

# 配置每个域名的最大并发请求数
# The download delay setting will honor only one of:
CONCURRENT_REQUESTS_PER_DOMAIN = 16

# 设置每个IP地址的最大并发请求数
CONCURRENT_REQUESTS_PER_IP = 16

# 启用或禁用cookies
# Disable cookies (enabled by default)
COOKIES_ENABLED = False

# 启用或禁用Telnet控制台
# Disable Telnet Console (enabled by default)
TELNETCONSOLE_ENABLED = False

# 默认的请求头
# Override the default request headers:
DEFAULT_REQUEST_HEADERS = {

    'Referer': 'https://www.xxx.com/',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36',
}

# 启用或禁用爬虫中间件
# Enable or disable spider middlewares
# See https://docs.scrapy.org/en/latest/topics/spider-middleware.html
# 爬虫中间件及其顺序
SPIDER_MIDDLEWARES = {
    "spidername.middlewares.SpidernameSpiderMiddleware": 543,
}

# 启用或禁用下载中间件
# Enable or disable downloader middlewares
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html
# 下载中间件及其顺序
DOWNLOADER_MIDDLEWARES = {
    "spidername.middlewares.SpidernameDownloaderMiddleware": 543,
}

# 启用和配置scrapy扩展
# Enable or disable extensions
# See https://docs.scrapy.org/en/latest/topics/extensions.html
EXTENSIONS = {
    "scrapy.extensions.telnet.TelnetConsole": None,
}

# 启用或禁用项目管道
# Configure item pipelines
# See https://docs.scrapy.org/en/latest/topics/item-pipeline.html
# 项目管道
ITEM_PIPELINES = {
   "spidername.pipelines.spidernamePipeline": 300,
}

# Enable and configure the AutoThrottle extension (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/autothrottle.html
# 是否启用自动限速
AUTOTHROTTLE_ENABLED = True

# 初始化下载延迟(秒) 
# The initial download delay
AUTOTHROTTLE_START_DELAY = 5

# 最大下载延迟(秒)
# The maximum download delay to be set in case of high latencies
AUTOTHROTTLE_MAX_DELAY = 60

# The average number of requests Scrapy should be sending in parallel to
# each remote server
# scrapy 的目标并发请求数
AUTOTHROTTLE_TARGET_CONCURRENCY = 1.0

# Enable showing throttling stats for every response received:
# 是否启用自动限速调试模式
AUTOTHROTTLE_DEBUG = False


# 启用和配置http缓存
# Enable and configure HTTP caching (disabled by default)
# See https://docs.scrapy.org/en/latest/topics/downloader-middleware.html#httpcache-middleware-settings
# 是否启用HTTP缓存
HTTPCACHE_ENABLED = True

# 缓存超时时间(秒)
HTTPCACHE_EXPIRATION_SECS = 0

# 缓存目录
HTTPCACHE_DIR = "httpcache"

# 忽略缓存的HTTP状态码列表
HTTPCACHE_IGNORE_HTTP_CODES = []

HTTPCACHE_STORAGE = "scrapy.extensions.httpcache.FilesystemCacheStorage"

# Set settings whose default value is deprecated to a future-proof value
REQUEST_FINGERPRINTER_IMPLEMENTATION = "2.7"
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

# 输出feed的编码
FEED_EXPORT_ENCODING = "utf-8"
相关推荐
我就说好玩7 天前
使用Scrapy框架爬取博客信息
scrapy·pycharm·网络爬虫·spider
孤客网络科技工作室9 天前
深入学习 Scrapy 框架:从入门到精通的全面指南
学习·scrapy
虞书欣的611 天前
Python小游戏14——雷霆战机
python·scrapy·游戏·pycharm·pygame
胜天半月子13 天前
Scrapy | 通过爬取豆瓣Top250电影信息来学习在中间件中应用随机请求头和代理ip
selenium·scrapy·中间件·代理ip
微刻时光16 天前
影刀RPA实战:网页爬虫之桌面壁纸图片
爬虫·python·scrapy·机器人·rpa·影刀·影刀rpa
胜天半月子16 天前
Scrapy | Scrapy框架中管道的使用
数据库·python·scrapy·管道
人生の三重奏17 天前
scrapy案例——当当网的爬取一
爬虫·scrapy·xpath
聪明的墨菲特i17 天前
Python中3个常见网络爬虫库
数据库·后端·爬虫·python·selenium·scrapy·requests
人生の三重奏17 天前
爬虫——scrapy的基本使用
爬虫·scrapy
胜天半月子18 天前
Scrapy | 爬取笑话网来认识继承自Spider的crawlspider爬虫类
爬虫·scrapy·crawlspider