scrapy---爬虫中间件和下载中间件

爬虫中间件

python 复制代码
# 爬虫中间件 (了解) middlewares.py
class MysfirstscrapySpiderMiddleware:

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    def process_spider_input(self, response, spider):
        return None

    def process_spider_output(self, response, result, spider):
        for i in result:
            yield i

    def process_spider_exception(self, response, exception, spider):

        pass

    def process_start_requests(self, start_requests, spider):

        for r in start_requests:
            yield r

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)

下载中间件

-进来request对象

-加代理

-加cookie

-加请求头

-出去response对象

-修改响应对象,最后进入到爬虫的parser中就是修改后的response

python 复制代码
# 下载中间件
class MysfirstscrapyDownloaderMiddleware:

    @classmethod
    def from_crawler(cls, crawler):
        # This method is used by Scrapy to create your spiders.
        s = cls()
        crawler.signals.connect(s.spider_opened, signal=signals.spider_opened)
        return s

    # 请求来了执行
    def process_request(self, request, spider):
        # 返回值可以是如下
        # return None:继续处理本次请求,执行执行下一个中间件的process_request
        #return Response:执行当前中间件的process_response回去,进入到引擎,被调度,进入第6步,返回到爬虫的解析方法中
        # return a Request:直接返回,给引擎,被调度,进入第2步,进入调度器等待下次被调度爬取
        # raise IgnoreRequest:执行 process_exception
        return None

    # 请求走了
    def process_response(self, request, response, spider):

        # 返回如下
        # return Response :继续往后走,进入到引擎,被调度到爬虫中解析
        # return Request :进入到引擎,被调度进调度器
        # - or raise IgnoreRequest:会执行process_exception
        return response

    def process_exception(self, request, exception, spider):
        # Called when a download handler or a process_request()
        # (from other downloader middleware) raises an exception.

        # Must either:
        # - return None: continue processing this exception
        # - return a Response object: stops process_exception() chain
        # - return a Request object: stops process_exception() chain
        pass

    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)

        
# 在配置文件中配置

1.加代理

python 复制代码
# 在下载中间件的def process_request(self, request, spider):写代码

# 第一步:
	-在下载中间件写process_request方法
   	 def get_proxy(self):
        import requests
        res = requests.get('http://127.0.0.1:5010/get/').json()
        if res.get('https'):
            return 'https://' + res.get('proxy')
        else:
            return 'http://' + res.get('proxy')
     def process_request(self, request, spider):
            request.meta['proxy'] = self.get_proxy()
            return None
        
# 第二步:代理可能不能用,会触发process_exception,在里面写
    def process_exception(self, request, exception, spider):
        print('-----',request.url)  # 这个地址没有爬
        return request

2.加cookie,修改请求头,随机生成UserAgent

2.1加cookie
python 复制代码
def process_request(self, request, spider):
    print(request.cookies)
    request.cookies['name']='lqz'
    return None
2.2 修改请求头
python 复制代码
  def process_request(self, request, spider):
        print(request.headers)
        request.headers['referer'] = 'http://www.lagou.com'
        return None
2.3 动态生成User-agent使用

需要先安装模块

python 复制代码
pip insttall fake_useragent
python 复制代码
 def process_request(self, request, spider):
        # fake_useragent模块
        from fake_useragent import UserAgent
        ua = UserAgent()
        request.headers['User-Agent']=str(ua.random)
        print(request.headers)
        return None
相关推荐
weixin_456904272 天前
跨域(CORS)和缓存中间件(Redis)深度解析
redis·缓存·中间件
万粉变现经纪人3 天前
如何解决pip安装报错ModuleNotFoundError: No module named ‘cuml’问题
python·scrapy·beautifulsoup·pandas·ai编程·pip·scipy
吴秋霖3 天前
主流反爬虫、反作弊防护与风控对抗手段
爬虫·算法·反爬虫技术
hui函数3 天前
scrapy框架-day02
后端·爬虫·python·scrapy
用户051610461674 天前
爬虫 API 技术全解析:从原理到实战的高效数据采集指南
爬虫·api
优秀的老黄4 天前
Docker部署RabbitMQ
linux·运维·docker·中间件·容器·centos·rabbitmq
xiaoxiongip6665 天前
动态ip适合挂什么项目
网络·爬虫·python·网络协议·tcp/ip·ip
专注API从业者5 天前
Scrapy 集成 JD API:一种高效且合规的商品数据采集方案
大数据·数据库·scrapy
q567315236 天前
自动化拨号爬虫体系:虚拟机集群部署与增量管理
运维·爬虫·网络协议·自动化
Thuni_soft6 天前
华宇TAS应用中间件与瀚高股份两款产品完成兼容互认证
中间件