爬虫案例：使用webpack爬取批量数据

T - mars2025-04-05 17:26

网站地址：（有基础的可以看懂网址）aHR0cHM6Ly93d3cuZGlnaWtleS5jbi96aC9wcm9kdWN0cy9maWx0ZXIvY29heGlhbC1jYWJsZXMtcmYvNDc1P3M9TjRJZ3JDQmNvQTVRTEFHaERPa0JNWUMlMkJXZw==

1.使用普通js逆向（不使用webpack版本）

  • 找js源代码方法：路径搜索（打断点）
  • 使用request请求工具生成代码：没有基础的可以去学习一下
python 复制代码 
# -*- coding: utf-8 -*-
# coding=utf-8

import requests

# from curl_cffi import requests

cookies = {
    '_evga_8292': 'a55c176dad51e758.',
    '_pxvid': 'c529250c-0ae9-11f0-9527-33aaac40f4a7',
    'OptanonAlertBoxClosed': '2025-03-27T08:59:26.297Z',
    '_ga': 'GA1.1.223144337.1743065961',
    '__ssid': '41460e75e455f9a227d9d3719897aed',
    '_pxhd': '405212efbeae18f381aca0b6fba33fde81488eb67b455ceb4f3bf1c0295267d3:c529250c-0ae9-11f0-9527-33aaac40f4a7',
    'dkLangPref': 'zh',
    'cur': 'USD',
    '_cs_c': '0',
    'dk_ref_click_data': 'ref_click_element=button#header-search-button|div.searchbox-inner-searchbutton|div.searchbox-inner dd-privacy-allow|div.header__searchbar header-search-panel|div.header__search|div.header__top|header#header',
    'digikey_theme': 'dklt',
    'sid': '180144056265390070xQ3W8NBXK7S95POMVBQL7WNRVYQZGZ6FT9IAVDJPK6N8F6UV6XJMJFUQI0KHYPXI7',
    'ping-accept-language': 'zh-CN',
    'search': '%7B%22id%22%3A%22959984db-d1d5-4eba-a1a1-ef34dae4c8d3%22%2C%22usage%22%3A%7B%22dailyCount%22%3A2%2C%22lastRequest%22%3A%222025-03-28T09%3A12%3A33.663Z%22%7D%2C%22version%22%3A1.1%7D',
    'sacn': '%7B%22token%22%3A%22eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJSZXF1ZXN0SWQiOiJjMTAxYjY0Ny1mMTU4LTQyOWMtYWZkYS0yNmIyMzYyMmRlOTMiLCJyZWZyZXNoIjoiTmpNNE56ZzNOVEExTlRRd09ESTRNakkwIiwibmJmIjoxNzQzMTUzMTU0LCJleHAiOjE3NDMxNTM3NTQsImlhdCI6MTc0MzE1MzE1NCwiaXNzIjoiaHR0cHM6Ly93d3cuZGlnaWtleS5jb20iLCJhdWQiOiJ1c2VyIn0.JO3dhyqxVUnLxHohATio4g9Nkxn0iF4z8nMaflWGKdQ%22%2C%22updated%22%3A1743153154108%7D',
    'TS01173021': '01ec0843e36f746aef95028224c636c1f2b3c9bced4fc8e1ea0a92b5b54c82039ab2c3d13f27142558974ae30def52bd8eff83dfef',
    'dk_sft_ses': '7447bef7-6548-4b60-8c67-029069937bfe',
    'pxcts': 'ca929e02-0bb4-11f0-97b2-e289aec77fc9',
    'dkuhint': 'false',
    'ps-eudo-sid': '%7b%22CustomerId%22%3a0%2c%22LoginId%22%3a0%2c%22CustomerClass%22%3a0%2c%22Currency%22%3a%22USD%22%2c%22OrderModel%22%3a-1%2c%22UserIdentity%22%3a0%2c%22RootId%22%3a0%7d',
    'TS01d90084': '01c72bed214e6ccc0d43453b9a7304e38d3f03aea252d9eddd273f5ac46e8b6962ec4854ae1cfff1aa7edc85194ccec24bd81e9465',
    'dk_tagxi': 'undefined.0.undefined',
    'dk_item_data': 'vil=ps-fam_475',
    'dk_ref_data_x': 'ref_page_type=PS&ref_page_sub_type=FAM&ref_page_id=475&ref_page_url=https://www.digikey.cn/zh/products/filter/coaxial-cables-rf/475&ref_content_search_keywords=undefined',
    'dk_loginid': 'undefined',
    '_cs_cvars': '%7B%221%22%3A%5B%22Page%20Title%22%2C%22Part%20Search%20-%20Family%20Page%22%5D%2C%222%22%3A%5B%22Page%20Site%22%2C%22CN%22%5D%2C%223%22%3A%5B%22Page%20Type%22%2C%22PS%22%5D%2C%224%22%3A%5B%22Page%20Sub%20Type%22%2C%22FAM%22%5D%2C%225%22%3A%5B%22Page%20Content%20Group%22%2C%22Part%20Search%22%5D%2C%226%22%3A%5B%22PageContentSubGroup%22%2C%22Category%20Page%22%5D%2C%227%22%3A%5B%22Page%20ID%22%2C%22475%22%5D%2C%2210%22%3A%5B%22Customer%20Dimension%22%2C%22%7B%5C%22%5C%5C%5C%22PLS%5C%5C%5C%22%5C%22%3A%5C%22Scrolling%5C%22%7D%22%5D%2C%2215%22%3A%5B%22Page%20State%22%2C%22Sort%20Order%20Test%20-%20Default%22%5D%2C%2216%22%3A%5B%22L1%20cat%22%2C%22%E7%94%B5%E7%BC%86%EF%BC%8C%E7%94%B5%E7%BA%BF%22%5D%7D',
    'TS019f5e6c': '01c72bed215a0fe2217a10b310a869712751e6f6642589799e8b8345f61dc7677b6065cd754ce0abefd700fe3abe503f8004f2e80a',
    'TSbafe380b027': '08a1509f8aab200070a74a5fffaf43e32d7e616708b0c903de04eae0bea5a99bef9029865e06796008a108128a113000eb6b42528f072df333b7fd975c8a1d397ddb0fcdf26d5dcaceab711aee82ee21e2d87983748b3579e3e727e8e470b1e2',
    'TS01c02def': '01dea9a590755b9251a71eb73d8c7635f179fbf70a4b03c2d1ad4e82d05cddde3dd3c09e4fe5d45c1b65a83464eba3d88632d5b8f1',
    'TS01b6e805': '01dea9a590755b9251a71eb73d8c7635f179fbf70a4b03c2d1ad4e82d05cddde3dd3c09e4fe5d45c1b65a83464eba3d88632d5b8f1',
    'TS809e22c5027': '081244aa86ab200080c1c3570a7d480b1e09f9abb68aeb679289785afe1e0bc10fa5f367e3caafb9087bf994ce11300062e39ca948225bdacc80901ef5c507d2fbc25298df837051c30cf33d3a84f4f3003aa9a91661eb98092f1c02a3f69973',
    'TS01ee79e7': '01dea9a590b0ad51e36edb18cc462226f1bd9ae61a7abdfd79609f5725f133445b7dda4e8268c29c83a188a7cbb4e89de7e485fc11',
    'TS01cba742': '01c72bed21aed56b051401d0c6bc258111c2457485be977d198d78eeeae1ab041c21024fc1be15cd7aa252e01415c9bf312a4f0765',
    'TS01bd62a8': '01c72bed21aed56b051401d0c6bc258111c2457485be977d198d78eeeae1ab041c21024fc1be15cd7aa252e01415c9bf312a4f0765',
    'TS605a4192027': '08a1509f8aab200027627f5dae374ce5f0942b550374707b4459a5237dce0133f4055ccf668ccde6084ecbbc8b113000091680c3585f315633b7fd975c8a1d39fb226408e0fcc8e0fb000dba3dbb03e3afe3b0e8b27f0421b4d13f158cd31850',
    'TS011a64f6': '01c72bed214e7a5e21fe83957af1179f3aada75991a7a6fb9327e8cfdc9d3ce4b1723b92f7d24d621f55d415034a590fc46242d210',
    'OptanonConsent': 'isGpcEnabled=0&datestamp=Fri+Mar+28+2025+17%3A12%3A45+GMT%2B0800+(GMT%2B08%3A00)&version=202407.1.0&browserGpcFlag=0&isIABGlobal=false&hosts=&consentId=c9441d6a-ce08-469e-945a-91a553ba1bf7&interactionCount=1&isAnonUser=1&landingPath=NotLandingPage&groups=C0001%3A1%2CC0003%3A1%2CC0002%3A1%2CC0004%3A1&intType=1&geolocation=CN%3BGS&AwaitingReconsent=false',
    '_ga_276R9HWFF5': 'GS1.1.1743153159.3.1.1743153165.54.0.0',
    '_uetsid': 'c962e6c00ae911f0bc3ff3499240a18f',
    '_uetvid': 'c96313200ae911f099814fb331ec3d30',
    '_uetmsclkid': '_uet35336ee3ffa21b89af650906ecf1c1f3',
    'TScb802422027': '089265d43dab2000f9cfc0602b46efa108c07276acda2614feccdc74e129c6938d3117444978b30e0844907dfd113000cadbf4d26afc110b23438b43804bfc1f2c50613b8d0bece6082dc5e160554f877c8092e1d6fb1436dd2de465365d1fe6',
    '_cs_id': '24965e33-8dc9-a2d0-bf6d-d19fb121eb43.1743067565.3.1743153191.1743153161.1742214720.1777231565987.1.x',
    'QSI_HistorySession': '',
    '_px2': 'eyJ1IjoiY2YzNTcwMTAtMGJiNC0xMWYwLTkyNDEtNmZiYmRlOWRhYmFhIiwidiI6ImM1MjkyNTBjLTBhZTktMTFmMC05NTI3LTMzYWFhYzQwZjRhNyIsInQiOjE3NDMxNTM3MTAxOTQsImgiOiI0Nzg3YWM0M2Q1OTQzMDJiZGZiMWFkZWM0YWQzZDRkOTk0ZjVmNTBhNmVlY2JhYzRkYTgwN2U3YTE2YmRjNDQ1In0=',
    '_pxde': '514f28ad61fd608e49e35bbd0675855de53fe8dbbcddd767f9bacd2d35bd3b4d:eyJ0aW1lc3RhbXAiOjE3NDMxNTM0MTAxOTUsImZfa2IiOjAsImlwY19pZCI6W119',
    '_cs_s': '4.5.0.9.1743155319363',
    'website#lang': 'zh-CN',
    'TS0198bc0d': '01c72bed21cc43640f3632e0efec1dd6d6059fb38611be381638fefa7acc16cb9d767f4391fc952e4200ea7b29df17ac322a2518bc',
    'TSe14c7dc7027': '08a1509f8aab2000bb364f84b6afa238557a0d9aa773100ac8d85401c58bedf005c409bdf56da6d1087de1596e113000e1b172aee13b9a09a3fc5cc150584533baef30d3d9a6560258f58c32c819456b2902355536e399807707f2050df942c7',
    'TS016ca11c': '01c72bed213a14315a168d71bb746592df2dac2aca112f7c39580a07f2ebedbb7580733d7eee112fbbd5e0c721ca90e90dbcc51e89',
    'TSc580adf4027': '08a1509f8aab2000328e6d20fd6a323037eefdedb3ca14559cbbb1561b290587025fd7a03b46899508214bd27f113000c4503a27bd6963bea3fc5cc1505845336903852c092b87d3dbe377b3df6e39094da70237501a153092680ebf0bcd8a0b',
    'TS01afc56e': '01c72bed2147782e410da5e37ca1069ce3eb2d39c353abd360d3769293ecdbe5813340955ceb28d0c77fb6fb575b4dfdc317ef6023',
    'TSf44f2996027': '08a1509f8aab20001d9b40051ced02cfcd6e38571b4b1f9acfad263e8e0c8f5ab5e0c0e5f78f8bbb087bac30bb1130001031c414de540300a3fc5cc150584533fb3753eb178b8218ead8a112aeffe106a4a457d2299bb014b18857ae6449c29f',
    'dkc_tracker': '3674965742303',
    '_dd_s': 'rum=0&expire=1743154426772',
}

headers = {
    'accept': '*/*',
    'accept-language': 'zh-CN,zh;q=0.9,en;q=0.8,en-GB;q=0.7,en-US;q=0.6',
    'authorization': 'Bearer',
    'cache-control': 'no-cache',
    'lang': 'zh',
    'pragma': 'no-cache',
    'priority': 'u=1, i',
    'referer': 'https://www.digikey.cn/zh/products/filter/coaxial-cables-rf/475?s=N4IgrCBcoA5QzAGhDOkBMYC%2BWg',
    'sec-ch-ua': '"Chromium";v="134", "Not:A-Brand";v="24", "Microsoft Edge";v="134"',
    'sec-ch-ua-mobile': '?0',
    'sec-ch-ua-platform': '"Windows"',
    'sec-fetch-dest': 'empty',
    'sec-fetch-mode': 'cors',
    'sec-fetch-site': 'same-origin',
    'site': 'cn',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.0.0 Safari/537.36 Edg/134.0.0.0',
    'x-currency': 'USD',
    'x-request-id': 'c101b647-f158-429c-afda-26b23622de93',
    # 'cookie': '_evga_8292=a55c176dad51e758.; _pxvid=c529250c-0ae9-11f0-9527-33aaac40f4a7; OptanonAlertBoxClosed=2025-03-27T08:59:26.297Z; _ga=GA1.1.223144337.1743065961; __ssid=41460e75e455f9a227d9d3719897aed; _pxhd=405212efbeae18f381aca0b6fba33fde81488eb67b455ceb4f3bf1c0295267d3:c529250c-0ae9-11f0-9527-33aaac40f4a7; dkLangPref=zh; cur=USD; _cs_c=0; dk_ref_click_data=ref_click_element=button#header-search-button|div.searchbox-inner-searchbutton|div.searchbox-inner dd-privacy-allow|div.header__searchbar header-search-panel|div.header__search|div.header__top|header#header; digikey_theme=dklt; sid=180144056265390070xQ3W8NBXK7S95POMVBQL7WNRVYQZGZ6FT9IAVDJPK6N8F6UV6XJMJFUQI0KHYPXI7; ping-accept-language=zh-CN; search=%7B%22id%22%3A%22959984db-d1d5-4eba-a1a1-ef34dae4c8d3%22%2C%22usage%22%3A%7B%22dailyCount%22%3A2%2C%22lastRequest%22%3A%222025-03-28T09%3A12%3A33.663Z%22%7D%2C%22version%22%3A1.1%7D; sacn=%7B%22token%22%3A%22eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJSZXF1ZXN0SWQiOiJjMTAxYjY0Ny1mMTU4LTQyOWMtYWZkYS0yNmIyMzYyMmRlOTMiLCJyZWZyZXNoIjoiTmpNNE56ZzNOVEExTlRRd09ESTRNakkwIiwibmJmIjoxNzQzMTUzMTU0LCJleHAiOjE3NDMxNTM3NTQsImlhdCI6MTc0MzE1MzE1NCwiaXNzIjoiaHR0cHM6Ly93d3cuZGlnaWtleS5jb20iLCJhdWQiOiJ1c2VyIn0.JO3dhyqxVUnLxHohATio4g9Nkxn0iF4z8nMaflWGKdQ%22%2C%22updated%22%3A1743153154108%7D; TS01173021=01ec0843e36f746aef95028224c636c1f2b3c9bced4fc8e1ea0a92b5b54c82039ab2c3d13f27142558974ae30def52bd8eff83dfef; dk_sft_ses=7447bef7-6548-4b60-8c67-029069937bfe; pxcts=ca929e02-0bb4-11f0-97b2-e289aec77fc9; dkuhint=false; ps-eudo-sid=%7b%22CustomerId%22%3a0%2c%22LoginId%22%3a0%2c%22CustomerClass%22%3a0%2c%22Currency%22%3a%22USD%22%2c%22OrderModel%22%3a-1%2c%22UserIdentity%22%3a0%2c%22RootId%22%3a0%7d; TS01d90084=01c72bed214e6ccc0d43453b9a7304e38d3f03aea252d9eddd273f5ac46e8b6962ec4854ae1cfff1aa7edc85194ccec24bd81e9465; dk_tagxi=undefined.0.undefined; dk_item_data=vil=ps-fam_475; dk_ref_data_x=ref_page_type=PS&ref_page_sub_type=FAM&ref_page_id=475&ref_page_url=https://www.digikey.cn/zh/products/filter/coaxial-cables-rf/475&ref_content_search_keywords=undefined; dk_loginid=undefined; _cs_cvars=%7B%221%22%3A%5B%22Page%20Title%22%2C%22Part%20Search%20-%20Family%20Page%22%5D%2C%222%22%3A%5B%22Page%20Site%22%2C%22CN%22%5D%2C%223%22%3A%5B%22Page%20Type%22%2C%22PS%22%5D%2C%224%22%3A%5B%22Page%20Sub%20Type%22%2C%22FAM%22%5D%2C%225%22%3A%5B%22Page%20Content%20Group%22%2C%22Part%20Search%22%5D%2C%226%22%3A%5B%22PageContentSubGroup%22%2C%22Category%20Page%22%5D%2C%227%22%3A%5B%22Page%20ID%22%2C%22475%22%5D%2C%2210%22%3A%5B%22Customer%20Dimension%22%2C%22%7B%5C%22%5C%5C%5C%22PLS%5C%5C%5C%22%5C%22%3A%5C%22Scrolling%5C%22%7D%22%5D%2C%2215%22%3A%5B%22Page%20State%22%2C%22Sort%20Order%20Test%20-%20Default%22%5D%2C%2216%22%3A%5B%22L1%20cat%22%2C%22%E7%94%B5%E7%BC%86%EF%BC%8C%E7%94%B5%E7%BA%BF%22%5D%7D; TS019f5e6c=01c72bed215a0fe2217a10b310a869712751e6f6642589799e8b8345f61dc7677b6065cd754ce0abefd700fe3abe503f8004f2e80a; TSbafe380b027=08a1509f8aab200070a74a5fffaf43e32d7e616708b0c903de04eae0bea5a99bef9029865e06796008a108128a113000eb6b42528f072df333b7fd975c8a1d397ddb0fcdf26d5dcaceab711aee82ee21e2d87983748b3579e3e727e8e470b1e2; TS01c02def=01dea9a590755b9251a71eb73d8c7635f179fbf70a4b03c2d1ad4e82d05cddde3dd3c09e4fe5d45c1b65a83464eba3d88632d5b8f1; TS01b6e805=01dea9a590755b9251a71eb73d8c7635f179fbf70a4b03c2d1ad4e82d05cddde3dd3c09e4fe5d45c1b65a83464eba3d88632d5b8f1; TS809e22c5027=081244aa86ab200080c1c3570a7d480b1e09f9abb68aeb679289785afe1e0bc10fa5f367e3caafb9087bf994ce11300062e39ca948225bdacc80901ef5c507d2fbc25298df837051c30cf33d3a84f4f3003aa9a91661eb98092f1c02a3f69973; TS01ee79e7=01dea9a590b0ad51e36edb18cc462226f1bd9ae61a7abdfd79609f5725f133445b7dda4e8268c29c83a188a7cbb4e89de7e485fc11; TS01cba742=01c72bed21aed56b051401d0c6bc258111c2457485be977d198d78eeeae1ab041c21024fc1be15cd7aa252e01415c9bf312a4f0765; TS01bd62a8=01c72bed21aed56b051401d0c6bc258111c2457485be977d198d78eeeae1ab041c21024fc1be15cd7aa252e01415c9bf312a4f0765; TS605a4192027=08a1509f8aab200027627f5dae374ce5f0942b550374707b4459a5237dce0133f4055ccf668ccde6084ecbbc8b113000091680c3585f315633b7fd975c8a1d39fb226408e0fcc8e0fb000dba3dbb03e3afe3b0e8b27f0421b4d13f158cd31850; TS011a64f6=01c72bed214e7a5e21fe83957af1179f3aada75991a7a6fb9327e8cfdc9d3ce4b1723b92f7d24d621f55d415034a590fc46242d210; OptanonConsent=isGpcEnabled=0&datestamp=Fri+Mar+28+2025+17%3A12%3A45+GMT%2B0800+(GMT%2B08%3A00)&version=202407.1.0&browserGpcFlag=0&isIABGlobal=false&hosts=&consentId=c9441d6a-ce08-469e-945a-91a553ba1bf7&interactionCount=1&isAnonUser=1&landingPath=NotLandingPage&groups=C0001%3A1%2CC0003%3A1%2CC0002%3A1%2CC0004%3A1&intType=1&geolocation=CN%3BGS&AwaitingReconsent=false; _ga_276R9HWFF5=GS1.1.1743153159.3.1.1743153165.54.0.0; _uetsid=c962e6c00ae911f0bc3ff3499240a18f; _uetvid=c96313200ae911f099814fb331ec3d30; _uetmsclkid=_uet35336ee3ffa21b89af650906ecf1c1f3; TScb802422027=089265d43dab2000f9cfc0602b46efa108c07276acda2614feccdc74e129c6938d3117444978b30e0844907dfd113000cadbf4d26afc110b23438b43804bfc1f2c50613b8d0bece6082dc5e160554f877c8092e1d6fb1436dd2de465365d1fe6; _cs_id=24965e33-8dc9-a2d0-bf6d-d19fb121eb43.1743067565.3.1743153191.1743153161.1742214720.1777231565987.1.x; QSI_HistorySession=; _px2=eyJ1IjoiY2YzNTcwMTAtMGJiNC0xMWYwLTkyNDEtNmZiYmRlOWRhYmFhIiwidiI6ImM1MjkyNTBjLTBhZTktMTFmMC05NTI3LTMzYWFhYzQwZjRhNyIsInQiOjE3NDMxNTM3MTAxOTQsImgiOiI0Nzg3YWM0M2Q1OTQzMDJiZGZiMWFkZWM0YWQzZDRkOTk0ZjVmNTBhNmVlY2JhYzRkYTgwN2U3YTE2YmRjNDQ1In0=; _pxde=514f28ad61fd608e49e35bbd0675855de53fe8dbbcddd767f9bacd2d35bd3b4d:eyJ0aW1lc3RhbXAiOjE3NDMxNTM0MTAxOTUsImZfa2IiOjAsImlwY19pZCI6W119; _cs_s=4.5.0.9.1743155319363; website#lang=zh-CN; TS0198bc0d=01c72bed21cc43640f3632e0efec1dd6d6059fb38611be381638fefa7acc16cb9d767f4391fc952e4200ea7b29df17ac322a2518bc; TSe14c7dc7027=08a1509f8aab2000bb364f84b6afa238557a0d9aa773100ac8d85401c58bedf005c409bdf56da6d1087de1596e113000e1b172aee13b9a09a3fc5cc150584533baef30d3d9a6560258f58c32c819456b2902355536e399807707f2050df942c7; TS016ca11c=01c72bed213a14315a168d71bb746592df2dac2aca112f7c39580a07f2ebedbb7580733d7eee112fbbd5e0c721ca90e90dbcc51e89; TSc580adf4027=08a1509f8aab2000328e6d20fd6a323037eefdedb3ca14559cbbb1561b290587025fd7a03b46899508214bd27f113000c4503a27bd6963bea3fc5cc1505845336903852c092b87d3dbe377b3df6e39094da70237501a153092680ebf0bcd8a0b; TS01afc56e=01c72bed2147782e410da5e37ca1069ce3eb2d39c353abd360d3769293ecdbe5813340955ceb28d0c77fb6fb575b4dfdc317ef6023; TSf44f2996027=08a1509f8aab20001d9b40051ced02cfcd6e38571b4b1f9acfad263e8e0c8f5ab5e0c0e5f78f8bbb087bac30bb1130001031c414de540300a3fc5cc150584533fb3753eb178b8218ead8a112aeffe106a4a457d2299bb014b18857ae6449c29f; dkc_tracker=3674965742303; _dd_s=rum=0&expire=1743154426772',
}

params = {
    's': 'N4IgrCBcoA5QLAGhDOkBMY CWg',
}

response = requests.get('https://www.digikey.cn/products/api/v5/filter-page/475', params=params, cookies=cookies,
                        headers=headers)

bs1 = response.text
# print(bs1)

# 忽略无法编码的字符(我们这里获取的是铭文数据说明我们这里，不需要响应解密，只需要请求加密爬取批量数据)
print(bs1.encode('gbk', errors='ignore').decode('gbk'))
  • 在pycharm中我们可以看到输出结果它是铭文的，所以我们只需要解决请求加密（爬取批量数据），不需要响应解密。注意：不要干扰网站的正常运行，不然会是犯法
  • js逆向入口
  • l.default这里就是请求加密，点进去看到下面的图片，加断点，然后再点击c.encodeUrlStateString(t)
  • 点进来看到所圈的地方就是加密的位置，再往进去点
  • 点进来看到一个函数，进行运行调试，看到圈到的方法，点进去再看一下，就是我们怎样进行加密的

  • 这时候我们就确定js请求加密就是这个位置，我们就可以复制js代码中进行调试
javascript 复制代码 
function _compress(e, t, r) {
    if (null == e)
        return "";
    var n, o, i, a = {}, s = {}, u = "", l = "", c = "", f = 2, d = 3, p = 2, g = [], h = 0, v = 0;
    for (i = 0; i < e.length; i += 1)
        if (u = e.charAt(i),
        Object.prototype.hasOwnProperty.call(a, u) || (a[u] = d++,
            s[u] = !0),
            l = c + u,
            Object.prototype.hasOwnProperty.call(a, l))
            c = l;
        else {
            if (Object.prototype.hasOwnProperty.call(s, c)) {
                if (256 > c.charCodeAt(0)) {
                    for (n = 0; n < p; n++)
                        h <<= 1,
                            v == t - 1 ? (v = 0,
                                g.push(r(h)),
                                h = 0) : v++;
                    for (n = 0,
                             o = c.charCodeAt(0); n < 8; n++)
                        h = h << 1 | 1 & o,
                            v == t - 1 ? (v = 0,
                                g.push(r(h)),
                                h = 0) : v++,
                            o >>= 1
                } else {
                    for (n = 0,
                             o = 1; n < p; n++)
                        h = h << 1 | o,
                            v == t - 1 ? (v = 0,
                                g.push(r(h)),
                                h = 0) : v++,
                            o = 0;
                    for (n = 0,
                             o = c.charCodeAt(0); n < 16; n++)
                        h = h << 1 | 1 & o,
                            v == t - 1 ? (v = 0,
                                g.push(r(h)),
                                h = 0) : v++,
                            o >>= 1
                }
                0 == --f && (f = Math.pow(2, p),
                    p++),
                    delete s[c]
            } else
                for (n = 0,
                         o = a[c]; n < p; n++)
                    h = h << 1 | 1 & o,
                        v == t - 1 ? (v = 0,
                            g.push(r(h)),
                            h = 0) : v++,
                        o >>= 1;
            0 == --f && (f = Math.pow(2, p),
                p++),
                a[l] = d++,
                c = String(u)
        }
    if ("" !== c) {
        if (Object.prototype.hasOwnProperty.call(s, c)) {
            if (256 > c.charCodeAt(0)) {
                for (n = 0; n < p; n++)
                    h <<= 1,
                        v == t - 1 ? (v = 0,
                            g.push(r(h)),
                            h = 0) : v++;
                for (n = 0,
                         o = c.charCodeAt(0); n < 8; n++)
                    h = h << 1 | 1 & o,
                        v == t - 1 ? (v = 0,
                            g.push(r(h)),
                            h = 0) : v++,
                        o >>= 1
            } else {
                for (n = 0,
                         o = 1; n < p; n++)
                    h = h << 1 | o,
                        v == t - 1 ? (v = 0,
                            g.push(r(h)),
                            h = 0) : v++,
                        o = 0;
                for (n = 0,
                         o = c.charCodeAt(0); n < 16; n++)
                    h = h << 1 | 1 & o,
                        v == t - 1 ? (v = 0,
                            g.push(r(h)),
                            h = 0) : v++,
                        o >>= 1
            }
            0 == --f && (f = Math.pow(2, p),
                p++),
                delete s[c]
        } else
            for (n = 0,
                     o = a[c]; n < p; n++)
                h = h << 1 | 1 & o,
                    v == t - 1 ? (v = 0,
                        g.push(r(h)),
                        h = 0) : v++,
                    o >>= 1;
        0 == --f && (f = Math.pow(2, p),
            p++)
    }
    for (n = 0,
             o = 2; n < p; n++)
        h = h << 1 | 1 & o,
            v == t - 1 ? (v = 0,
                g.push(r(h)),
                h = 0) : v++,
            o >>= 1;
    for (; ;) {
        if (h <<= 1,
        v == t - 1) {
            g.push(r(h));
            break
        }
        v++
    }
    return g.join("")
}


function compressToEncodedURIComponent(e) {
    return null == e ? "" : _compress(e, 6, function (e) {
        return r.charAt(e)
    })
}
//测试代码
data = '{"5":{"p":15,"pp":25}}'
r = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+-$"
ret = compressToEncodedURIComponent(data)
console.log(ret)

调试成功，这里我们就破解了这个网站的请求加密，我们就可以爬取批量的数据

2.使用普通js逆向第二种方式

  • 继续找到请求加密入口
  • 找到它的核心逻辑，直接copy，放到pycharm中
  • 在pycharm中复现复制过来的代码，哪里报错就添哪里改哪里
  • 完整代码如下：
javascript 复制代码 
function _compress(e, t, r) {
    if (null == e)
        return "";
    var n, o, i, a = {}, s = {}, u = "", l = "", c = "", f = 2, d = 3, p = 2, g = [], h = 0, v = 0;
    for (i = 0; i < e.length; i += 1)
        if (u = e.charAt(i),
        Object.prototype.hasOwnProperty.call(a, u) || (a[u] = d++,
            s[u] = !0),
            l = c + u,
            Object.prototype.hasOwnProperty.call(a, l))
            c = l;
        else {
            if (Object.prototype.hasOwnProperty.call(s, c)) {
                if (256 > c.charCodeAt(0)) {
                    for (n = 0; n < p; n++)
                        h <<= 1,
                            v == t - 1 ? (v = 0,
                                g.push(r(h)),
                                h = 0) : v++;
                    for (n = 0,
                             o = c.charCodeAt(0); n < 8; n++)
                        h = h << 1 | 1 & o,
                            v == t - 1 ? (v = 0,
                                g.push(r(h)),
                                h = 0) : v++,
                            o >>= 1
                } else {
                    for (n = 0,
                             o = 1; n < p; n++)
                        h = h << 1 | o,
                            v == t - 1 ? (v = 0,
                                g.push(r(h)),
                                h = 0) : v++,
                            o = 0;
                    for (n = 0,
                             o = c.charCodeAt(0); n < 16; n++)
                        h = h << 1 | 1 & o,
                            v == t - 1 ? (v = 0,
                                g.push(r(h)),
                                h = 0) : v++,
                            o >>= 1
                }
                0 == --f && (f = Math.pow(2, p),
                    p++),
                    delete s[c]
            } else
                for (n = 0,
                         o = a[c]; n < p; n++)
                    h = h << 1 | 1 & o,
                        v == t - 1 ? (v = 0,
                            g.push(r(h)),
                            h = 0) : v++,
                        o >>= 1;
            0 == --f && (f = Math.pow(2, p),
                p++),
                a[l] = d++,
                c = String(u)
        }
    if ("" !== c) {
        if (Object.prototype.hasOwnProperty.call(s, c)) {
            if (256 > c.charCodeAt(0)) {
                for (n = 0; n < p; n++)
                    h <<= 1,
                        v == t - 1 ? (v = 0,
                            g.push(r(h)),
                            h = 0) : v++;
                for (n = 0,
                         o = c.charCodeAt(0); n < 8; n++)
                    h = h << 1 | 1 & o,
                        v == t - 1 ? (v = 0,
                            g.push(r(h)),
                            h = 0) : v++,
                        o >>= 1
            } else {
                for (n = 0,
                         o = 1; n < p; n++)
                    h = h << 1 | o,
                        v == t - 1 ? (v = 0,
                            g.push(r(h)),
                            h = 0) : v++,
                        o = 0;
                for (n = 0,
                         o = c.charCodeAt(0); n < 16; n++)
                    h = h << 1 | 1 & o,
                        v == t - 1 ? (v = 0,
                            g.push(r(h)),
                            h = 0) : v++,
                        o >>= 1
            }
            0 == --f && (f = Math.pow(2, p),
                p++),
                delete s[c]
        } else
            for (n = 0,
                     o = a[c]; n < p; n++)
                h = h << 1 | 1 & o,
                    v == t - 1 ? (v = 0,
                        g.push(r(h)),
                        h = 0) : v++,
                    o >>= 1;
        0 == --f && (f = Math.pow(2, p),
            p++)
    }
    for (n = 0,
             o = 2; n < p; n++)
        h = h << 1 | 1 & o,
            v == t - 1 ? (v = 0,
                g.push(r(h)),
                h = 0) : v++,
            o >>= 1;
    for (; ;) {
        if (h <<= 1,
        v == t - 1) {
            g.push(r(h));
            break
        }
        v++
    }
    return g.join("")
}

function xxx(e) {
    return "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+-$".charAt(e)
}

//测试代码
data = '{"5":{"p":15,"pp":25}}'
console.log(_compress(data, 6, xxx))

3.使用webpack进行逆向

  • 这时候我们需要看看附近有没有l，往上翻代码，翻到l，我们进行刷新页面进行断点
  • 这时候我们看到平常的webpack是一层，这里怎么还套了一层n（），我们可以点进去看一下或者在控制台打印一下套没套是不是都一样
  • 我们 点进去一看发现e没有做什么返回还是个e，所以我们可以推断出这里没有变化
  • 最后在控制台打印了一下，发现也是一样
  • 这时候我们可以悬浮鼠标点r，点进去就发现了webpack的调度器
  • 点击全选复制这个文件中的所有代码，我们可以看到整个文件的代码才300多行，说明这里的webpack是多文件webpack，我们需要进行解耦合（webpack中的大数组、大列表应该在其他的文件中）
  • 我们可以看到这份代码中调用了另一份代码
  • 我们先运行一下代码，先根据报错进行补环境（下面就是我们需要补的环境），location是我们在浏览器控制台输出获得的本网站的location
python 复制代码 
window = global
document = {}
location = {
    "ancestorOrigins": {},
    "href": "",
    "origin": "",
    "protocol": "https:",
    "host": "",
    "hostname": "",
    "port": "",
    "pathname": "/en/products/filter/coaxial-cables-rf/475",
    "search": "?s=N4IgrCBcoA5QjAdgDQhnSAmMBfHQ",
    "hash": ""
}
  • 我们开始处理webpack，先补上全局变量，再补上日志
  • 这里我们创建环境文件和webpack主函数文件，再创建一个main文件，下面就是main文件，然后打印我们之前逆向到的webpack值，我们进行运行调试
  • 调试之后我们发现，出现了错误，没有找到 87284模块，我们就可以去浏览器去搜索这个
  • 我们搜哦找到之后在进行复制粘贴，创建一个新的文件
  • 复制成功后，先将所有的self改成window，再导入主函数main中进行调试
  • 在浏览器看到代码是属于前端代码，我们可以直接删除这段代码，在进行调试（因为前端代码一般不会参与加密，我们就可以先删除再调试），如果参与加密我们就要慢慢调试，查补环境
  • 然后改完我们就继续执行代码，发现和上面的错误很相似，我们就继续采取上面的方法，进行调试
  • 有时候还会出现以下错误
  • 看到下面这张图结果就已经出来了
  • 最后再强制让程序退出，再到浏览器进行对加密的值验证
  • 最后我们就完成了webpack逆向爬虫，完整代码目录
相关推荐
deckcode11 小时前
xpath定位
爬虫·功能测试·网络爬虫·web测试
程序员一诺16 小时前
【爬虫开发】爬虫开发从0到1全知识教程第14篇：scrapy爬虫框架,介绍【附代码文档】
后端·爬虫·python·数据
数据采集10218 小时前
Python数据采集：从入门到实战，代码全解析！
爬虫
ONE_Gua18 小时前
chromium魔改——绕过无限debugger反调试
chrome·爬虫·浏览器
ONE_Gua1 天前
chromium魔改——CDP（Chrome DevTools Protocol）检测01
前端·后端·爬虫
ONE_Gua2 天前
chromium魔改——navigator.webdriver 检测
前端·后端·爬虫
z_mazin2 天前
JavaScript逆向魔法：Chrome开发者工具探秘之旅
javascript·chrome·爬虫
攻城狮7号2 天前
Python爬虫第2节-网页基础和爬虫基本原理
爬虫·python爬虫
π2702 天前
爬虫：网络请求(通信)步骤，http和https协议
网络·爬虫
热门推荐
01汽车上的各种质量：整备质量、总质量、装载质量、簧上质量、簧下质量02我决定放弃搞 Java 了03如何在WPS和Word/Excel中直接使用DeepSeek功能04第8章：系统质量属性与架构评估05DeepSeek各版本说明与优缺点分析06从零安装 LLaMA-Factory 微调 Qwen 大模型成功及所有的坑07C++11可变参数模板单例模式08RAG 实践- Ollama+RagFlow 部署本地知识库09DeepSeek RAGFlow构建本地知识库系统10分享 3 款基于 .NET 开源且免费的远程桌面工具