爬虫案例---抓取小米商店应用
代码如下：
抓取第一页的内容
import requests
from lxml import etree
url = 'https://app.mi.com/catTopList/0?page=1'
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36'
}
获取响应内容
res = requests.get(url, headers=headers)
content = res.content.decode('utf-8')
解析响应内容
tree = etree.HTML(content)
应用排行榜应用程序的名称和超链接xpath路径
path = '//li/h5/a'
a_list = tree.xpath(path)
for a in a_list:
print(a.xpath('./text()')0, end=' ')
print('https://app.mi.com' + a.xpath('./@href')0)
```shell
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3 /Users/bruce_liu/PycharmProjects/路飞/day04/requests/爬虫案例1.py
王者荣耀-S34赛季云梦有灵 https://app.mi.com/details?id=com.tencent.tmgp.sgame
夸克-新生代智能搜索 https://app.mi.com/details?id=com.quark.browser
抖音 https://app.mi.com/details?id=com.ss.android.ugc.aweme
钉钉-让进步发生 https://app.mi.com/details?id=com.alibaba.android.rimet
作业帮-作业检查和辅导工具 https://app.mi.com/details?id=com.baidu.homework
QQ https://app.mi.com/details?id=com.tencent.mobileqq
快对-原快对作业 https://app.mi.com/details?id=com.kuaiduizuoye.scan
UC浏览器-好搜好看好好用 https://app.mi.com/details?id=com.UCMobile
快手 https://app.mi.com/details?id=com.smile.gifmaker
拼多多 https://app.mi.com/details?id=com.xunmeng.pinduoduo
快手极速版 https://app.mi.com/details?id=com.kuaishou.nebula
哔哩哔哩 https://app.mi.com/details?id=tv.danmaku.bili
京东 https://app.mi.com/details?id=com.jingdong.app.mall
腾讯会议 https://app.mi.com/details?id=com.tencent.wemeet.app
小红书 https://app.mi.com/details?id=com.xingin.xhs
虎牙直播 https://app.mi.com/details?id=com.duowan.kiwi
地铁跑酷-开年即开跑 https://app.mi.com/details?id=com.kiloo.subwaysurf
QQ浏览器 https://app.mi.com/details?id=com.tencent.mtt
迅雷 https://app.mi.com/details?id=com.xunlei.downloadprovider
知乎 https://app.mi.com/details?id=com.zhihu.android
抖音极速版 https://app.mi.com/details?id=com.ss.android.ugc.aweme.lite
淘宝 https://app.mi.com/details?id=com.taobao.taobao
和平精英 https://app.mi.com/details?id=com.tencent.tmgp.pubgmhd
美图秀秀 https://app.mi.com/details?id=com.mt.mtxx.mtxx
百度 https://app.mi.com/details?id=com.baidu.searchbox
番茄免费小说 https://app.mi.com/details?id=com.dragon.read
剪映 https://app.mi.com/details?id=com.lemon.lv
微信 https://app.mi.com/details?id=com.tencent.mm
闲鱼 https://app.mi.com/details?id=com.taobao.idlefish
腾讯视频-黑土无言全网独播 https://app.mi.com/details?id=com.tencent.qqlive
微博 https://app.mi.com/details?id=com.sina.weibo
爱奇艺-大江大河3首播 https://app.mi.com/details?id=com.qiyi.video
迷你世界 https://app.mi.com/details?id=com.minitech.miniworld.TMobile.mi
美团-美好生活小帮手 https://app.mi.com/details?id=com.sankuai.meituan
优酷视频-你也有今天全网独播 https://app.mi.com/details?id=com.youku.phone
百度网盘 https://app.mi.com/details?id=com.baidu.netdisk
Process finished with exit code 0