python爬虫题目

网站

https://project-iprj6705f17ebcfad66461658c5c-8000.preview.node01.inscode.run/

第一道题爬取api并且保存

python 复制代码
import requests,re
import json
url = "https://project-iprj6705f17ebcfad66461658c5c-8000.preview.node01.inscode.run/tasks/api/"
headers= {

'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36'
}

res = requests.get(url,headers=headers).json()
with open('1.json','w') as f:
    f.write(json.dumps(res,ensure_ascii=False))

第二道爬取所有图片

python 复制代码
from urllib.parse import urljoin
import requests,re
from urllib.parse import urlparse
import json
url = "https://project-iprj6705f17ebcfad66461658c5c-8000.preview.node01.inscode.run/tasks/api/"
headers= {

'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36'
}

res = requests.get(url,headers=headers).json()
list1 = res['articles']
list2=[]
for i in list1:
    list2.append(i['image'])
base_url ="https://"+urlparse(url).netloc

for image in list2:
    image_url = urljoin(base_url,image)
    img = requests.get(image_url).content
    img_name = image.split("/")[-1]
    with open(img_name,'wb') as f:
        f.write(img)

第三道 爬取题目和摘要

python 复制代码
import requests,csv
from lxml import etree
with open("data.csv","w",newline='',encoding='gbk') as f:
    writer = csv.writer(f)
    writer.writerow(["题目","再要"])
url = "https://project-iprj6705f17ebcfad66461658c5c-8000.preview.node01.inscode.run/tasks/article/list/"
headers= {

'user-agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36'
}

res = requests.get(url,headers=headers)
html = etree.HTML(res.text)
wen_zhang = html.xpath('//div[@class="lab-block"]//a//@href')
with open("data.csv","w",newline='',encoding='gbk') as f:
    writer = csv.writer(f)
    writer.writerow(["ti","zai"])



for i in wen_zhang:
    url_l = "https://project-iprj6705f17ebcfad66461658c5c-8000.preview.node01.inscode.run/"+i
    result = requests.get(url_l,headers=headers)
    select = etree.HTML(result.text)
    timu = select.xpath('//h2/text()')[0]
    zaiyao = select.xpath('//p//text()')
    result = "".join(zaiyao)
    with open("data.csv", "a", newline='',encoding='utf-8') as f:
        writer = csv.writer(f)
        writer.writerow([timu, result])
相关推荐
学无止境小袁同学1 小时前
LlamaIndex Workflows 学习
python·llm·tensorflow·llamaindex·agent开发·多agent·workflows
Evand J1 小时前
【MATLAB例程】基于APF的多无人机动态路径规划与避障仿真。避碰及安全性能分析。附完整代码的下载链接
开发语言·matlab·无人机
鲸渔2 小时前
【AI Agent系列课程13-Python 基础13】JSON 读写:结构化数据交换基础
开发语言·python·json·ai编程
Highcharts.js7 小时前
Highcharts漏斗图表使用配置指南
java·开发语言·javascript·信息可视化·highcharts·漏斗图
wuyk55512 小时前
Python办公自动化-第01讲-环境搭建与自动化认知
python·自动化·pyqt
高洁0113 小时前
数字孪生与计算机视觉:让机器看懂现场
人工智能·python·机器学习·transformer·迁移学习
ComPDFKit13 小时前
RAG 回答只显示 PDF 文件名?用 Python 保留引用页码
开发语言·python·pdf·rag
FRing0313 小时前
vue3干货总结2(vue3生命周期、父子通信、模版引用、provide和inject)
开发语言·前端·vue.js·vue3
yz_aiks14 小时前
Termux 使用教程:把安卓手机变成随身 Linux(装 Python、SSH 远程、跑自动化脚本)
android·linux·python·智能手机·ssh·termux
琳少_15 小时前
java程序开发笔记
java·开发语言·数据库·笔记