Python 爬虫基础

Python 爬虫基础

1.1 理论

在浏览器通过网页拼接【/robots.txt】来了解可爬取的网页路径范围

例如访问: https://www.csdn.net/robots.txt

User-agent: *

Disallow: /scripts

Disallow: /public

Disallow: /css/

Disallow: /images/

Disallow: /content/

Disallow: /ui/

Disallow: /js/

Disallow: /scripts/

Disallow: /article_preview.html*

Disallow: /tag/

Disallow: /?

Disallow: /link/

Disallow: /tags/

Disallow: /news/

Disallow: /xuexi/

通过Python Requests 库发送HTTP【Hypertext Transfer Protocol "超文本传输协议"】请求

通过Python Beautiful Soup 库来解析获取到的HTML内容

HTTP请求

HTTP响应

1.2 实践代码 【获取价格&书名】

python 复制代码
import requests
# 解析HTML
from bs4 import BeautifulSoup

# 将程序伪装成浏览器请求
head = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
}
requests = requests.get("http://books.toscrape.com/",headers= head)
# 指定编码
# requests.encoding= 'gbk'
if requests.ok:
    # file = open(r'C:\Users\root\Desktop\Bug.html', 'w')
    # file.write(requests.text)
    # file.close
    content =  requests.text
    ## html.parser 指定当前解析 HTML 元素
    soup = BeautifulSoup(content, "html.parser")
    
    ## 获取价格
    all_prices = soup.findAll("p", attrs={"class":"price_color"})
    for price in all_prices:
        print(price.string[2:])

    ## 获取名称
    all_title = soup.findAll("h3")
    for title in all_title:
        ## 获取h3下面的第一个a元素
        print(title.find("a").string)
else:
    print(requests.status_code)

1.3 实践代码 【获取 Top250 的电影名】

python 复制代码
import requests
# 解析HTML
from bs4 import BeautifulSoup
head = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
}
# 获取 TOP 250个电影名
for i in range(0,250,25):
    response = requests.get(f"https://movie.douban.com/top250?start={i}", headers= head)
    if response.ok:
        content =  response.text
        soup = BeautifulSoup(content, "html.parser")
        all_titles = soup.findAll("span", attrs={"class": "title"})
        for title in all_titles:
            if "/" not in title.string:
                print(title.string) 
    else:
        print(response.status_code)

1.4 实践代码 【下载图片】

python 复制代码
import requests
# 解析HTML
from bs4 import BeautifulSoup

head = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
}
response = requests.get(f"https://www.maoyan.com/", headers= head)
if response.ok:
    soup = BeautifulSoup(response.text, "html.parser")
    for img in soup.findAll("img", attrs={"class": "movie-poster-img"}):
        img_url = img.get('data-src')
        alt = img.get('alt')
        path = 'img/' + alt + '.jpg'
        res = requests.get(img_url)
        with open(path, 'wb') as f:
            f.write(res.content)
else:
    print(response.status_code)

1.5 实践代码 【千图网图片 - 爬取 - 下载图片】

python 复制代码
import requests
# 解析HTML
from bs4 import BeautifulSoup


# 千图网图片 - 爬取
head = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"
}
# response = requests.get(f"https://www.58pic.com/piccate/53-0-0.html", headers= head)
# response = requests.get(f"https://www.58pic.com/piccate/53-598-2544.html", headers= head)
response = requests.get(f"https://www.58pic.com/piccate/53-527-1825.html", headers= head)
if response.ok:
    soup = BeautifulSoup(response.text, "html.parser")
    for img in soup.findAll("img", attrs={"class": "lazy"}):
        img_url = "https:" + img.get('data-original')
        alt = img.get('alt')
        path = 'imgqiantuwang/' + str(alt) + '.jpg'
        res = requests.get(img_url)
        with open(path, 'wb') as f:
            f.write(res.content)
else:
    print(response.status_code)
相关推荐
Bruce小鬼8 分钟前
QT文件基本操作
开发语言·qt
2202_7544215414 分钟前
生成MPSOC以及ZYNQ的启动文件BOOT.BIN的小软件
java·linux·开发语言
我只会发热21 分钟前
Java SE 与 Java EE:基础与进阶的探索之旅
java·开发语言·java-ee
LZXCyrus22 分钟前
【杂记】vLLM如何指定GPU单卡/多卡离线推理
人工智能·经验分享·python·深度学习·语言模型·llm·vllm
Enougme25 分钟前
Appium常用的使用方法(一)
python·appium
懷淰メ30 分钟前
PyQt飞机大战游戏(附下载地址)
开发语言·python·qt·游戏·pyqt·游戏开发·pyqt5
hummhumm44 分钟前
第 22 章 - Go语言 测试与基准测试
java·大数据·开发语言·前端·python·golang·log4j
宁静@星空1 小时前
006-自定义枚举注解
java·开发语言
hummhumm1 小时前
第 28 章 - Go语言 Web 开发入门
java·开发语言·前端·python·sql·golang·前端框架
武子康1 小时前
Java-07 深入浅出 MyBatis - 一对多模型 SqlMapConfig 与 Mapper 详细讲解测试
java·开发语言·数据库·sql·mybatis·springboot