python爬虫进阶篇:Scrapy中使用Selenium+Firefox浏览器爬取沪深A股股票行情

一、前言

上篇记录了Scrapy搭配selenium的使用方法,有了基本的了解后我们可以将这项技术落实到实际需求中。目前很多股票网站的行情信息都是动态数据,我们可以用Scrapy+selenium对股票进行实时采集并持久化,再进行数据分析、邮件通知等操作。

二、环境搭建

详情请看上篇笔记

三、代码实现

  • items
python 复制代码
class StockSpiderItem(scrapy.Item):
    # define the fields for your item here like:
    # name = scrapy.Field()
    # 股票代码
    stock_code = scrapy.Field()
    # 股票名称
    stock_name = scrapy.Field()
    # 最新价
    last_price = scrapy.Field()
    # 涨跌幅
    rise_fall_rate = scrapy.Field()
    # 涨跌额
    rise_fall_price = scrapy.Field()
  • middlewares
python 复制代码
	
    def __init__(self):
        # ----------------firefox的设置------------------------------- #
        self.options = firefox_options()
    def spider_opened(self, spider):
        spider.logger.info('Spider opened: %s' % spider.name)
        spider.driver = webdriver.Firefox(options=self.options)  # 指定使用的浏览器
        
    def process_request(self, request, spider):
        # Called for each request that goes through the downloader
        # middleware.

        # Must either:
        # - return None: continue processing this request
        # - or return a Response object
        # - or return a Request object
        # - or raise IgnoreRequest: process_exception() methods of
        #   installed downloader middleware will be called

        spider.driver.get("https://quote.eastmoney.com/center/gridlist.html#hs_a_board")
        return None
        
    def process_response(self, request, response, spider):
        # Called with the response returned from the downloader.

        # Must either;
        # - return a Response object
        # - return a Request object
        # - or raise IgnoreRequest
        response_body = spider.driver.page_source

        return HtmlResponse(url=request.url, body=response_body, encoding='utf-8', request=request)
  • settings设置
python 复制代码
# See https://doc.scrapy.org/en/latest/topics/spider-middleware.html
SPIDER_MIDDLEWARES = {
   'stock_spider.middlewares.StockSpiderSpiderMiddleware': 543,
}

# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
   'stock_spider.middlewares.StockSpiderDownloaderMiddleware': 543,
}
  • spider文件
python 复制代码
    def parse(self, response):
        # 股票代码
        stock_code = response.css("table.table_wrapper-table tbody tr td:nth-child(2) a::text").extract()
        # 股票名称
        stock_name = response.css("table.table_wrapper-table tbody tr td:nth-child(3) a::text").extract()
        # 最新价
        last_price = response.css("table.table_wrapper-table tbody tr td:nth-child(5) span::text").extract()
        # 涨跌幅
        rise_fall_rate = response.css("table.table_wrapper-table tbody tr td:nth-child(6) span::text").extract()
        # 涨跌额
        rise_fall_price = response.css("table.table_wrapper-table tbody tr td:nth-child(7) span::text").extract()

        for i in range(len(stock_code)):
            item = StockSpiderItem()
            item["stock_code"] = stock_code[i]
            item["stock_name"] = stock_name[i]
            item["last_price"] = last_price[i]
            item["rise_fall_rate"] = rise_fall_rate[i]
            item["rise_fall_price"] = rise_fall_price[i]
            yield item

    def close(self, spider):
        spider.driver.quit()
  • pipelines持久化
python 复制代码
    def process_item(self, item, spider):
        """
        接收到提交过来的对象后,写入csv文件
        """
        filename = f'stock_info.csv'

        with open(filename, 'a+', encoding='utf-8') as f:
            line = item["stock_code"] + "," + item["stock_name"] + "," + item["last_price"] + "," + \
                   item["rise_fall_rate"] + "," + item["rise_fall_price"] + "\n"
            f.write(line)

        return item
  • readme文件
python 复制代码
1.安装依赖包 
- python 3.0+
- pip install -r requirements.txt

2.将最第二层stock_spider文件夹设置为根目录

3.将firefox驱动程序包放到python环境的Scripts文件夹里

4.必须要安装firefox浏览器才会调用到浏览器

5.执行spider_main.py文件启动爬虫
相关推荐
万粉变现经纪人2 小时前
如何解决 pip install -r requirements.txt 私有索引未设为 trusted-host 导致拒绝 问题
开发语言·python·scrapy·flask·beautifulsoup·pandas·pip
查士丁尼·绵3 小时前
笔试-九宫格三阶积幻方
python·九宫格·三阶积幻方
l1t5 小时前
DeepSeek辅助利用搬移底层xml实现快速编辑xlsx文件的python程序
xml·开发语言·python·xlsx
大飞记Python5 小时前
部门管理|“编辑部门”功能实现(Django5零基础Web平台)
前端·数据库·python·django
查士丁尼·绵7 小时前
笔试-羊狼过河
python
摸鱼的老谭7 小时前
构建Agent该选Python还是Java ?
java·python·agent
鄃鳕8 小时前
python 字典 列表 类比c++【python】
c++·python
可触的未来,发芽的智生8 小时前
新奇特:黑猫警长的纳米世界,忆阻器与神经网络的智慧
javascript·人工智能·python·神经网络·架构
程序员三藏8 小时前
Jmeter接口测试与压力测试
自动化测试·软件测试·python·测试工具·jmeter·接口测试·压力测试
烛阴8 小时前
用 Python 揭秘 IP 地址背后的地理位置和信息
前端·python