生成网站sitemap.xml地图教程

要生成 sitemap.xml 文件,需要通过爬虫程序抓取网站的所有有效链接。以下是完整的解决方案:


步骤 1:安装必要的 Python 库

  • ounter(line

    pip install requests beautifulsoup4 lxml


步骤 2:创建 Python 爬虫脚本 (sitemap_generator.py)

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

  • ounter(line

    import requestsfrom bs4 import BeautifulSoupfrom urllib.parse import urljoin, urlparseimport xml.etree.ElementTree as ETfrom datetime import datetime
    def get_all_links(base_url): # 存储已访问和待访问的链接 visited = set() queue = [base_url] all_links = set()
    while queue: url = queue.pop(0) if url in visited: continue
    try: headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36' } response = requests.get(url, headers=headers, timeout=10) if response.status_code != 200: continue
    # 添加到已访问列表 visited.add(url) all_links.add(url) print(f"Crawled: {url}")
    # 解析 HTML 获取新链接 soup = BeautifulSoup(response.text, 'lxml') for link in soup.find_all('a', href=True): href = link['href'].strip() full_url = urljoin(url, href)
    # 过滤无效链接 parsed = urlparse(full_url) if parsed.scheme not in ('http', 'https'): continue if not parsed.netloc.endswith('91kaiye.cn'): # 仅限本站链接 continue if '#' in full_url: # 忽略锚点 full_url = full_url.split('#')[0]
    # 添加到待访问队列 if full_url not in visited: queue.append(full_url)
    except Exception as e: print(f"Error crawling {url}: {str(e)}")
    return all_links
    def create_sitemap(links, filename='sitemap.xml'): root = ET.Element('urlset', xmlns='http://www.sitemaps.org/schemas/sitemap/0.9') for link in sorted(links): url_elem = ET.SubElement(root, 'url') ET.SubElement(url_elem, 'loc').text = link ET.SubElement(url_elem, 'lastmod').text = datetime.now().strftime('%Y-%m-%d') ET.SubElement(url_elem, 'changefreq').text = 'daily' ET.SubElement(url_elem, 'priority').text = '0.8' tree = ET.ElementTree(root) tree.write(filename, encoding='utf-8', xml_declaration=True) print(f"\nSitemap generated: {filename} with {len(links)} URLs")
    if name == 'main': base_url = 'https://www.91kaiye.cn/' print("Starting crawl...") links = get_all_links(base_url) create_sitemap(links)


步骤 3:运行脚本

  • ounter(line

    python sitemap_generator.py


执行说明:

  1. 爬虫逻辑

    • 从首页 https://www.91kaiye.cn/ 开始广度优先搜索

    • 自动过滤非本站链接、锚点和无效 URL

    • 记录每个页面的最后修改日期(默认当天)

    • 设置更新频率为 daily,优先级为 0.8

  2. 输出文件

    • 生成的 sitemap.xml 格式如下:
      • ounter(line

      • ounter(line

      • ounter(line

      • ounter(line

      • ounter(line

      • ounter(line

      • ounter(line

      • ounter(line

      • ounter(line

      • ounter(line

        <?xml version='1.0' encoding='utf-8'?><urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> <url> <loc>https://www.91kaiye.cn/page1</loc> <lastmod>2023-10-05</lastmod> <changefreq>daily</changefreq> <priority>0.8</priority> </url> ...</urlset>

注意事项:

  1. 反爬措施

    • 如果网站有反爬机制,可能需要:
      • 添加 time.sleep(1) 延迟请求

      • 使用代理 IP

      • 设置更真实的请求头

  2. 动态内容

    • 对于 JavaScript 渲染的页面(如 Vue/React),需改用 SeleniumPlaywright
  3. 优化建议


替代方案:使用在线工具

如果不想运行代码,可用在线服务生成:

  1. XML-Sitemaps.com

  2. Screaming Frog SEO Spider(桌面工具)


生成后请将 sitemap.xml 上传到网站根目录,并通过百度/Google站长工具提交。

相关推荐
qq_318693011 天前
XML 站点地图制作与提交全流程,新站收录效率提升 50%
xml·数据库·mysql
卷Java1 天前
WXML 编译错误修复总结
xml·java·前端·微信小程序·uni-app·webview
武子康2 天前
Java-131 深入浅出 MySQL MyCat 深入解析 schema.xml 配置详解:逻辑库、逻辑表、数据节点全攻略
xml·java·数据库·mysql·性能优化·系统架构·mycat
fendouweiqian2 天前
pom.xml 不在根目录,idea无法识别项目处理方案
xml·java·intellij-idea
虾说羊3 天前
Spring注解驱动开发:从XML到注解的完美蜕变
xml·sql·spring
l1t11 天前
利用美团龙猫用libxml2编写XML转CSV文件C程序
xml·c语言·libxml2·解析器
Luna-player15 天前
某个工程 pom.xml
xml
l1t15 天前
对expat库XML_Parse函数调用优化的测试
xml·c语言·解析器·expat
l1t15 天前
利用美团龙猫添加xlsx的sheet.xml读取sharedStrings.xml中共享字符串输出到csv功能
xml·c语言·数据结构·人工智能·算法·解析器
叶 落15 天前
[Maven 基础课程]pom.xml
xml·pom.xml 常见配置项·maven 的 pom.xml