爬虫 crawler 入门爬取不设防网页 并实现无限增生

基础版本

爬取网页后直接将前端html代码不加处理的输出

python 复制代码
# pip3 install requests
import requests

# request the target URL
def crawler():
    response = requests.get("https://www.scrapingcourse.com/ecommerce/")
    response.raise_for_status()
    print(response.text)

# execute the crawler
crawler()

无限增生的爬虫

从第一个链接开始,记录已经遍历过的链接;
并且从这个链接爬取的html代码中记录 ahref 的链接,存储到将要遍历的列表;
对于已经爬取的链接,直接continue处理

python 复制代码
# pip3 install requests
import requests

def crawler():
    while urls_to_visit:

        # get the page to visit from the list
        current_url = urls_to_visit.pop(0)
        print(current_url)
        if current_url in visited_urls:
            continue
        # 记录访问过的url到列表中
        visited_urls.add(current_url)

        try:
            response = requests.get(current_url, timeout=5)  # 设置超时时间,避免死循环
            response.raise_for_status()  # 检查请求是否成功
        except requests.RequestException as e:
            print(f"请求失败: {current_url}, 错误: {e}")
            continue

        # parse the HTML
        soup = BeautifulSoup(response.text, "html.parser")

        # collect all the links
        link_elements = soup.select("a[href]")
        for link_element in link_elements:
            url = link_element["href"]

            if url.startswith("#"):
                continue  # ignore internal links

            # convert links to absolute URLs
            if not url.startswith("http"):
                absolute_url = requests.compat.urljoin(target_url, url)
            else:
                absolute_url = url

            # ensure the crawled link belongs to the target domain and hasn't been visited
            if (
                absolute_url.startswith(target_url)
                and absolute_url not in urls_to_visit
            ):
                urls_to_visit.append(url)

# pip3 install requests beautifulsoup4

from bs4 import BeautifulSoup



target_url = "https://www.scrapingcourse.com/ecommerce/"
# initialize the list of discovered URLs
urls_to_visit = [target_url]
visited_urls = set()  # 记录已访问的 URL,防止重复爬取
# execute the crawler
crawler()

无限增生的效果

部分链接爬取失败后会返回错误信息

相关推荐
wuyk5551 天前
Spider-第16讲-线程池与竞态问题
大数据·爬虫·python·elasticsearch·搜索引擎
wuyk5553 天前
Spider-第14讲-表单提交与CSRF
前端·爬虫·python·elasticsearch·csrf
wuyk5555 天前
Spider-第11讲-工程化改造与可维护性
大数据·爬虫·elasticsearch·搜索引擎
wuyk5555 天前
Spider-第12讲-asyncio异步并发爬取
大数据·爬虫·elasticsearch·搜索引擎
小宋10215 天前
AI 爬虫把大学图书馆站点拖垮:按行为指纹给爬虫分类、按类发配额、用边缘缓存分层,以带宽成本结构和真实用户 P95 延迟定档
人工智能·爬虫·分类
wuyk5556 天前
Spider-第10讲-SQLite持久化与去重
数据库·爬虫·sqlite
wuyk5556 天前
Spider-第09讲-三格式落盘CSV-Excel-JSON
爬虫·python·json·excel
wuyk5557 天前
Spider-第07讲-JSON接口与正则提取
爬虫·python·json
feasibility.7 天前
一物降一物?AI逆向没有终局:REA拆App,BLA设套,LASIK清场
人工智能·爬虫·github·逆向·skill·rea
wuyk5559 天前
Spider-第03讲-HTTP速通与读懂响应
网络·爬虫·网络协议·http