Web爬虫应用功能及需求设计

秋月霜风2024-09-24 11:20

Web爬虫应用功能及需求设计

1. 概述

这是一个基于PyQt5的图形用户界面(GUI)网络爬虫应用程序。

应用允许用户输入URL,选择输出目录,并爬取指定网站的内容。提供了一个用户友好的界面来控制爬虫过程,并显示爬取进度和状态。

2. 主要功能

2.1 URL输入与历史记录

  • 功能:允许用户输入URL,并保存最近使用的URL历史记录。
  • 设计理念:提高用户体验,方便重复使用常用URL。
  • 相关代码:
python 复制代码 
class MainWindow(QMainWindow):
    def __init__(self):
        # ...
        self.url_combo = QComboBox()
        self.url_combo.setEditable(True)
        self.url_combo.setInsertPolicy(QComboBox.InsertAtTop)
        self.url_combo.editTextChanged.connect(self.on_url_changed)
        # ...

    def on_url_changed(self, text):
        if text and text not in [self.url_combo.itemText(i) for i in range(self.url_combo.count())]:
            self.url_combo.insertItem(0, text)
            self.save_url_to_history()

    def save_url_to_history(self):
        urls = [self.url_combo.itemText(i) for i in range(self.url_combo.count())]
        urls = urls[:10]  # Only keep the 10 most recent URLs
        self.settings.setValue("url_history", urls)
        self.settings.sync()

    def load_url_history(self):
        urls = self.settings.value("url_history", [])
        self.url_combo.addItems(urls)

2.2 输出目录选择

  • 功能:允许用户选择爬取内容的保存位置。
  • 设计理念:提供直观的文件对话框,方便用户选择目录。
  • 相关代码:
python 复制代码 
def select_output_directory(self):
    directory = QFileDialog.getExistingDirectory(self, "Select Output Directory")
    if directory:
        self.output_input.setText(directory)

2.3 爬虫控制

  • 功能:启动和控制爬虫过程。
  • 设计理念:使用多线程确保GUI响应性,同时提供进度和状态更新。
  • 相关代码:
python 复制代码 
def start_crawling(self):
    url = self.url_combo.currentText()
    output_dir = self.output_input.text()

    if not url or not output_dir:
        self.status_label.setText("Please enter URL and select output directory")
        return

    self.crawler_thread = CrawlerThread(url, output_dir, self.browser_settings)
    self.crawler_thread.progress_signal.connect(self.update_progress)
    self.crawler_thread.status_signal.connect(self.update_status)
    self.crawler_thread.finished_signal.connect(self.crawling_finished)
    self.crawler_thread.start()

    self.crawl_button.setEnabled(False)

2.4 浏览器设置

  • 功能:允许用户自定义爬虫的浏览器相关设置。
  • 设计理念:提供灵活性,使用户能够根据需要调整爬虫行为。
  • 相关代码:
python 复制代码 
class BrowserSettingsDialog(QDialog):
    def __init__(self, parent=None):
        # ...
        self.ua_input = QLineEdit()
        self.timeout_input = QLineEdit()
        self.robots_checkbox = QCheckBox("Follow robots.txt")
        self.retries_input = QComboBox()
        # ...

    def get_settings(self):
        return {
            "user_agent": self.ua_input.text(),
            "timeout": int(self.timeout_input.text() or 30),
            "follow_robots": self.robots_checkbox.isChecked(),
            "max_retries": int(self.retries_input.currentText())
        }

2.5 进度和状态显示

  • 功能:实时显示爬虫的进度和状态信息。
  • 设计理念:提供清晰的反馈,让用户了解爬虫的工作状态。
  • 相关代码:
python 复制代码 
def update_progress(self, value):
    self.progress_bar.setValue(value)

def update_status(self, status):
    self.status_label.setText(status)

3. robots.txt 遵守功能

3.1 功能描述

该爬虫程序包含了检查网站的robots.txt文件并遵守其协议的功能。这个功能被集成在浏览器设置中,用户可以选择是否启用此功能。

3.2 设计理念

  • 尊重网站所有者：通过遵守robots.txt,我们尊重网站管理员对其网站内容的访问控制。
  • 合法合规：许多国家的法律要求网络爬虫遵守robots.txt协议。
  • 可配置性：允许用户根据需要启用或禁用此功能,提供灵活性。

3.3 实现细节

  1. 在浏览器设置中添加选项：
python 复制代码 
class BrowserSettingsDialog(QDialog):
    def __init__(self, parent=None):
        # ...
        self.robots_checkbox = QCheckBox("Follow robots.txt")
        layout.addWidget(self.robots_checkbox)
        # ...

    def get_settings(self):
        return {
            # ...
            "follow_robots": self.robots_checkbox.isChecked(),
            # ...
        }
  1. 在爬虫逻辑中实现检查：
python 复制代码 
class CrawlerThread(QThread):
    # ...
    def crawl_directory(self, url: str, base_path: str) -> None:
        # ...
        # Check robots.txt if enabled
        if self.browser_settings['follow_robots']:
            rp = robotparser.RobotFileParser()
            rp.set_url(urljoin(url, "/robots.txt"))
            rp.read()
            if not rp.can_fetch(self.browser_settings['user_agent'], url):
                logging.warning(f"Skipping {url} as per robots.txt")
                self.status_signal.emit(f"Skipping {url} as per robots.txt")
                return
        # ...

3.4 用户界面交互

用户可以在"Browser Settings"对话框中选择是否遵守robots.txt:

  1. 点击主界面上的"Browser Settings"按钮。
  2. 在弹出的对话框中,勾选或取消勾选"Follow robots.txt"选项。
  3. 点击"OK"保存设置。

3.5 优势

  1. 合规性：确保爬虫行为符合网络道德和法律要求。
  2. 减少被封禁风险：遵守robots.txt可以降低爬虫被网站封禁的风险。
  3. 灵活性：用户可以根据具体需求选择是否启用此功能。

3.6 todo

  1. 缓存robots.txt：为了提高效率,可以考虑缓存已经读取过的robots.txt文件内容。
  2. 更详细的robots.txt解析：添加对更复杂robots.txt指令的支持,如爬取频率限制。
  3. 可视化robots.txt规则：在UI中显示当前网站的robots.txt规则,帮助用户理解爬取限制。

4. 技术实现

4.1 多线程爬虫

使用QThread实现爬虫功能,确保GUI的响应性。

python 复制代码 
class CrawlerThread(QThread):
    progress_signal = pyqtSignal(int)
    status_signal = pyqtSignal(str)
    finished_signal = pyqtSignal()

    def __init__(self, url: str, base_path: str, browser_settings: dict):
        # ...

    def run(self):
        self.crawl_directory(self.url, self.base_path)
        self.finished_signal.emit()

    def crawl_directory(self, url: str, base_path: str) -> None:
        # 爬虫逻辑实现
        # ...

4.2 设置持久化

使用QSettings保存和加载URL历史记录和浏览器设置。

python 复制代码 
def save_browser_settings(self):
    for key, value in self.browser_settings.items():
        self.settings.setValue(f"browser_settings/{key}", value)
    self.settings.sync()

def load_browser_settings(self):
    return {
        "user_agent": self.settings.value("browser_settings/user_agent", ""),
        "timeout": int(self.settings.value("browser_settings/timeout", 30)),
        "follow_robots": self.settings.value("browser_settings/follow_robots", True, type=bool),
        "max_retries": int(self.settings.value("browser_settings/max_retries", 3))
    }

5. 未来扩展

代码中包含了一个占位符用于未来扩展:

python 复制代码 
self.extension_placeholder = QLabel("Future extensions will be added here")
layout.addWidget(self.extension_placeholder)

这为将来添加新功能预留了空间,如:

  • 高级过滤选项
  • 爬取深度控制
  • 导出爬取结果为不同格式，格式优化等
  • 爬取调度功能

6. 总结

这个Web爬虫应用程序提供了一个用户友好的界面,允许用户轻松地爬取网站（主要基于ftp的文件批量下载）内容，结合了多线程爬虫、自定义浏览器设置和持久化存储等功能,为用户提供了一个功能丰富且灵活的工具。通过清晰的进度和状态显示,用户可以轻松监控爬虫的工作状态，预留了一定的扩展空间。

完整代码：

python 复制代码 
import sys
import requests
from bs4 import BeautifulSoup
import os
from urllib.parse import urljoin, urlparse
import logging
from typing import Optional, Set, List, Tuple
from urllib import robotparser
from PyQt5.QtWidgets import (QApplication, QMainWindow, QWidget, QVBoxLayout, QHBoxLayout,
                             QPushButton, QLineEdit, QTextEdit, QProgressBar, QLabel,
                             QFileDialog, QComboBox, QDialog, QDialogButtonBox, QCheckBox)
from PyQt5.QtGui import QIntValidator
from PyQt5.QtCore import QThread, pyqtSignal, Qt, QSettings


logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(levelname)s - %(message)s')


class CrawlerThread(QThread):
    progress_signal = pyqtSignal(int)
    status_signal = pyqtSignal(str)
    finished_signal = pyqtSignal()

    def __init__(self, url: str, base_path: str, browser_settings: dict):
        super().__init__()
        self.url = url
        self.base_path = base_path
        self.browser_settings = browser_settings
        self.session = requests.Session()
        self.visited = set()

        # Apply browser settings
        self.session.headers.update({'User-Agent': self.browser_settings['user_agent']})
        self.session.mount('https://',
                           requests.adapters.HTTPAdapter(max_retries=self.browser_settings['max_retries']))
        self.session.mount('http://',
                           requests.adapters.HTTPAdapter(max_retries=self.browser_settings['max_retries']))

    def run(self):
        self.crawl_directory(self.url, self.base_path)
        self.finished_signal.emit()

    def crawl_directory(self, url: str, base_path: str) -> None:
        if url in self.visited:
            logging.warning(f"Skipping already visited URL: {url}")
            return
        self.visited.add(url)

        try:
            response = self.session.get(url, timeout=self.browser_settings['timeout'])
            response.raise_for_status()
        except requests.RequestException as e:
            logging.error(f"Failed to fetch {url}: {e}")
            self.status_signal.emit(f"Error: Failed to fetch {url}")
            return

        # Check robots.txt if enabled
        if self.browser_settings['follow_robots']:
            rp = robotparser.RobotFileParser()
            rp.set_url(urljoin(url, "/robots.txt"))
            rp.read()
            if not rp.can_fetch(self.browser_settings['user_agent'], url):
                logging.warning(f"Skipping {url} as per robots.txt")
                self.status_signal.emit(f"Skipping {url} as per robots.txt")
                return

        soup = BeautifulSoup(response.text, 'html.parser')
        links = soup.find_all('a', href=True)

        for link in links:
            href = link['href']
            full_url = urljoin(url, href)
            if full_url.startswith(url) and full_url not in self.visited:
                file_name = os.path.join(base_path, urlparse(full_url).path.lstrip('/'))
                os.makedirs(os.path.dirname(file_name), exist_ok=True)

                try:
                    with open(file_name, 'w', encoding='utf-8') as f:
                        f.write(requests.get(full_url).text)
                    self.status_signal.emit(f"Saved: {full_url}")
                except Exception as e:
                    logging.error(f"Failed to save {full_url}: {e}")
                    self.status_signal.emit(f"Error: Failed to save {full_url}")

                self.crawl_directory(full_url, base_path)

        self.progress_signal.emit(len(self.visited))


class BrowserSettingsDialog(QDialog):
    def __init__(self, parent=None):
        super().__init__(parent)
        self.setWindowTitle("Browser Settings")
        self.setModal(True)

        layout = QVBoxLayout(self)

        # User-Agent
        ua_layout = QHBoxLayout()
        ua_layout.addWidget(QLabel("User-Agent:"))
        self.ua_input = QLineEdit()
        ua_layout.addWidget(self.ua_input)
        layout.addLayout(ua_layout)

        # Timeout
        timeout_layout = QHBoxLayout()
        timeout_layout.addWidget(QLabel("Timeout (seconds):"))
        self.timeout_input = QLineEdit()
        self.timeout_input.setValidator(QIntValidator(1, 60))
        timeout_layout.addWidget(self.timeout_input)
        layout.addLayout(timeout_layout)

        # Follow robots.txt
        self.robots_checkbox = QCheckBox("Follow robots.txt")
        layout.addWidget(self.robots_checkbox)

        # Max retries
        retries_layout = QHBoxLayout()
        retries_layout.addWidget(QLabel("Max retries:"))
        self.retries_input = QComboBox()
        self.retries_input.addItems([str(i) for i in range(6)])
        retries_layout.addWidget(self.retries_input)
        layout.addLayout(retries_layout)

        # Buttons
        self.button_box = QDialogButtonBox(QDialogButtonBox.Ok | QDialogButtonBox.Cancel)
        self.button_box.accepted.connect(self.accept)
        self.button_box.rejected.connect(self.reject)
        layout.addWidget(self.button_box)

    def get_settings(self):
        return {
            "user_agent": self.ua_input.text(),
            "timeout": int(self.timeout_input.text() or 30),
            "follow_robots": self.robots_checkbox.isChecked(),
            "max_retries": int(self.retries_input.currentText())
        }

    def set_settings(self, settings):
        self.ua_input.setText(settings.get("user_agent", ""))
        self.timeout_input.setText(str(settings.get("timeout", 30)))
        self.robots_checkbox.setChecked(settings.get("follow_robots", True))
        self.retries_input.setCurrentText(str(settings.get("max_retries", 3)))


class MainWindow(QMainWindow):
    def __init__(self):
        super().__init__()
        self.setWindowTitle("Web Crawler")
        self.setGeometry(100, 100, 800, 600)

        self.settings = QSettings("YourCompany", "WebCrawler")

        central_widget = QWidget()
        self.setCentralWidget(central_widget)

        layout = QVBoxLayout()
        central_widget.setLayout(layout)

        # URL input with history
        url_layout = QHBoxLayout()
        self.url_combo = QComboBox()
        self.url_combo.setEditable(True)
        self.url_combo.setInsertPolicy(QComboBox.InsertAtTop)
        self.url_combo.editTextChanged.connect(self.on_url_changed)
        url_layout.addWidget(QLabel("URL:"))
        url_layout.addWidget(self.url_combo)
        layout.addLayout(url_layout)

        # Output directory selection
        output_layout = QHBoxLayout()
        self.output_input = QLineEdit()
        self.output_button = QPushButton("Select")
        self.output_button.clicked.connect(self.select_output_directory)
        output_layout.addWidget(QLabel("Output:"))
        output_layout.addWidget(self.output_input)
        output_layout.addWidget(self.output_button)
        layout.addLayout(output_layout)

        # Crawl button
        self.crawl_button = QPushButton("Start Crawling")
        self.crawl_button.clicked.connect(self.start_crawling)
        layout.addWidget(self.crawl_button)

        # Settings button
        self.settings_button = QPushButton("Browser Settings")
        self.settings_button.clicked.connect(self.open_browser_settings)
        layout.addWidget(self.settings_button)

        # Progress bar
        self.progress_bar = QProgressBar()
        layout.addWidget(self.progress_bar)

        # Status label
        self.status_label = QLabel()
        layout.addWidget(self.status_label)

        # Load saved URL history
        self.load_url_history()

        # Load browser settings
        self.browser_settings = self.load_browser_settings()

        # Placeholder for future extensions
        self.extension_placeholder = QLabel("Future extensions will be added here")
        layout.addWidget(self.extension_placeholder)

    def on_url_changed(self, text):
        if text and text not in [self.url_combo.itemText(i) for i in range(self.url_combo.count())]:
            self.url_combo.insertItem(0, text)
            self.save_url_to_history()

    def select_output_directory(self):
        directory = QFileDialog.getExistingDirectory(self, "Select Output Directory")
        if directory:
            self.output_input.setText(directory)

    def start_crawling(self):
        url = self.url_combo.currentText()
        output_dir = self.output_input.text()

        if not url or not output_dir:
            self.status_label.setText("Please enter URL and select output directory")
            return

        # Ensure current URL is at the top of the list
        self.url_combo.removeItem(self.url_combo.findText(url))
        self.url_combo.insertItem(0, url)
        self.url_combo.setCurrentIndex(0)

        # Save URL history
        self.save_url_to_history()

        self.crawler_thread = CrawlerThread(url, output_dir, self.browser_settings)
        self.crawler_thread.progress_signal.connect(self.update_progress)
        self.crawler_thread.status_signal.connect(self.update_status)
        self.crawler_thread.finished_signal.connect(self.crawling_finished)
        self.crawler_thread.start()

        self.crawl_button.setEnabled(False)

    def update_progress(self, value):
        self.progress_bar.setValue(value)

    def update_status(self, status):
        self.status_label.setText(status)

    def crawling_finished(self):
        self.crawl_button.setEnabled(True)
        self.status_label.setText("Crawling finished")

    def save_url_to_history(self):
        urls = [self.url_combo.itemText(i) for i in range(self.url_combo.count())]
        urls = urls[:10]  # Only keep the 10 most recent URLs
        self.settings.setValue("url_history", urls)
        self.settings.sync()

    def load_url_history(self):
        urls = self.settings.value("url_history", [])
        self.url_combo.addItems(urls)

    def open_browser_settings(self):
        dialog = BrowserSettingsDialog(self)
        dialog.set_settings(self.browser_settings)
        if dialog.exec_() == QDialog.Accepted:
            self.browser_settings = dialog.get_settings()
            self.save_browser_settings()

    def save_browser_settings(self):
        for key, value in self.browser_settings.items():
            self.settings.setValue(f"browser_settings/{key}", value)
        self.settings.sync()

    def load_browser_settings(self):
        return {
            "user_agent": self.settings.value("browser_settings/user_agent", ""),
            "timeout": int(self.settings.value("browser_settings/timeout", 30)),
            "follow_robots": self.settings.value("browser_settings/follow_robots", True, type=bool),
            "max_retries": int(self.settings.value("browser_settings/max_retries", 3))
        }


if __name__ == "__main__":
    app = QApplication(sys.argv)
    window = MainWindow()
    window.show()
    sys.exit(app.exec_())
相关推荐
请叫我欧皇i9 分钟前
html本地离线引入vant和vue2（详细步骤）
开发语言·前端·javascript
nuclear201112 分钟前
使用Python 在Excel中创建和取消数据分组 - 详解
python·excel数据分组·创建excel分组·excel分类汇总·excel嵌套分组·excel大纲级别·取消excel分组
躺平的花卷12 分钟前
Python爬虫案例八：抓取597招聘网信息并用xlutils进行excel数据的保存
爬虫·excel
533_12 分钟前
[vue] 深拷贝 lodash cloneDeep
前端·javascript·vue.js
guokanglun18 分钟前
空间数据存储格式GeoJSON
前端
Lucky小小吴27 分钟前
有关django、python版本、sqlite3版本冲突问题
python·django·sqlite
zhang-zan1 小时前
nodejs操作selenium-webdriver
前端·javascript·selenium
GIS 数据栈1 小时前
每日一书 《基于ArcGIS的Python编程秘笈》
开发语言·python·arcgis
爱分享的码瑞哥1 小时前
Python爬虫中的IP封禁问题及其解决方案
爬虫·python·tcp/ip
猫爪笔记1 小时前
前端：HTML （学习笔记）【2】
前端·笔记·学习·html
热门推荐
01(欧拉)openEuler系统添加网卡文件配置流程、(欧拉)openEuler系统手动配置ipv6地址流程、(欧拉)openEuler系统网络管理说明02PyTorch机器学习实现液态神经网络03【HarmonyOS】HUAWEI DevEco Studio 下载地址汇总04玄机平台应急响应—webshell查杀05Coze扣子平台完整体验和实践（附国内和国际版对比）06Ubuntu 20.04使用Livox mid 360 测试 FAST_LIO07【目标跟踪】相机运动补偿08Unity中PICO实现 隔空取物 和 接触抓取物体09RAG 实践- Ollama+RagFlow 部署本地知识库10怎样让音频速度变慢？请跟随以下方法进行操作