Python爬虫中Xpath用法之——normalize-space()

复制代码
 

背景:

今天遇到一个问题:有一个表格,现在要用python获取到其中的内容。每个tr是一行,有9个是取td里包含内容文本,还有2个td里还有一层子元素<a>,取子元素<a>里面的文本内容。如果格式如下图

我用列表推导式写了半天,怎么都不能满足要示。意外发现用item.xpath('normalize-space()')就能很好实现功能。

知识点:

normalize-space(.) 或 normalize-space():获取当前元素‌所有后代文本节点拼接后‌的整体字符串并标准化(适合提取元素内完整可见文本)。

normalize-space(text()): 仅处理当前元素的‌直接文本子节点‌,若存在子元素会丢失嵌套文本或导致节点集转换异常 。‌

原网站数据如图:

爬虫代码如下:

复制代码
from lxml import etree
import requests

target_url='https://nba.hupu.com/stats/players/pts'

response  = requests.get(target_url,headers={'User-Agent':'Mozilla/5.0'}).content
document = etree.HTML(response)

tr_list = document.xpath('//*[@id="data_js"]/div[4]/div/table/tbody/tr')

print(tr_list)

"""

‌若无参数‌:normalize-space() 默认等价于 normalize-space(.),即处理当前节点
关键区别:. 与 text()
normalize-space(.):获取当前元素‌所有后代文本节点拼接后‌的整体字符串并标准化(适合提取元素内完整可见文本)。
normalize-space(text()):仅处理当前元素的‌直接文本子节点‌,若存在子元素会丢失嵌套文本或导致节点集转换异常 。‌
"""
all_list = list()
for item in tr_list:
    td_list = item.xpath('./td')
    #这一句很重要,不然会很麻烦。
    # 因为tr中有9个是直接td里包含内容文本,还有2个是包含一级子元素<a>取里面有内容。
    # normalize-space() 或 normalize-space(.)  normalize-space()
    td_row = [item.xpath('normalize-space()') for item in td_list]
    print(td_row)
print(all_list)

获取的数据如下:

复制代码
['排名', '球员', '球队', '得分', '命中-出手', '命中率', '命中-三分', '三分命中率', '命中-罚球', '罚球命中率', '场次', '上场时间']
['1', '卢卡-东契奇', '湖人', '33.50', '10.80-22.80', '47.6%', '4.00-10.80', '36.6%', '7.90-10.10', '78%', '64', '35.80']
['2', '谢伊-吉尔杰斯-亚历山大', '雷霆', '31.10', '10.80-19.40', '55.3%', '1.70-4.40', '38.6%', '7.90-9.00', '87.9%', '68', '33.20']
['3', '安东尼-爱德华兹', '森林狼', '28.80', '9.90-20.20', '48.9%', '3.40-8.40', '39.9%', '5.70-7.20', '79.6%', '61', '35.00']
['4', '杰伦-布朗', '76人', '28.70', '10.40-21.70', '47.7%', '2.00-5.70', '34.7%', '6.00-7.60', '79.5%', '71', '34.40']
['5', '泰雷斯-马克西', '76人', '28.30', '9.90-21.40', '46.2%', '3.10-8.60', '36.7%', '5.30-6.00', '89.2%', '70', '38.00']
['6', '多诺万-米切尔', '骑士', '27.90', '9.70-20.00', '48.3%', '3.20-8.80', '36.4%', '5.30-6.10', '86.5%', '70', '33.50']
['6', '科怀-伦纳德', '猛龙', '27.90', '9.80-19.40', '50.5%', '2.70-6.80', '38.7%', '5.70-6.40', '89.2%', '65', '32.10']
['8', '尼古拉-约基奇', '掘金', '27.70', '9.90-17.40', '56.9%', '1.70-4.50', '38%', '6.10-7.40', '83.1%', '65', '34.80']
['9', '德文-布克', '太阳', '26.10', '8.50-18.70', '45.6%', '1.90-5.80', '33%', '7.10-8.10', '87.3%', '64', '33.50']
['10', '杰伦-布伦森', '尼克斯', '26.00', '9.30-19.90', '46.7%', '2.60-7.10', '36.9%', '4.80-5.70', '84.1%', '74', '35.00']
['10', '凯文-杜兰特', '火箭', '26.00', '9.20-17.60', '52%', '2.40-5.80', '41.3%', '5.20-6.00', '87.4%', '78', '36.40']
['12', '贾马尔-默里', '掘金', '25.40', '8.80-18.10', '48.3%', '3.30-7.50', '43.5%', '4.60-5.20', '88.7%', '75', '35.40']
['13', '维克托·文班亚马', '马刺', '25.00', '8.60-16.90', '51.2%', '1.90-5.50', '34.9%', '5.80-7.00', '82.7%', '64', '29.20']
['14', '德尼-阿夫迪亚', '开拓者', '24.20', '7.50-16.10', '46.2%', '1.90-5.90', '31.8%', '7.40-9.20', '80.2%', '66', '33.30']
['15', '帕斯卡尔-西亚卡姆', '步行者', '24.00', '9.00-18.60', '48.4%', '1.70-4.70', '35.8%', '4.20-6.10', '69.3%', '62', '33.20']
['16', '凯德-坎宁安', '活塞', '23.90', '8.60-18.60', '46.1%', '2.00-5.70', '34.2%', '4.80-6.00', '81.2%', '64', '33.90']
['17', '杰伦-约翰逊', '老鹰', '22.50', '8.30-17.10', '48.9%', '1.70-4.70', '35.2%', '4.20-5.30', '78.8%', '72', '35.20']
['18', '保罗-班切罗', '魔术', '22.20', '7.30-16.00', '45.9%', '1.20-3.80', '30.5%', '6.30-8.20', '77.5%', '72', '34.80']
['19', '诺曼-鲍威尔', '公牛', '21.70', '7.30-15.40', '47%', '2.70-7.10', '38%', '4.50-5.50', '82.7%', '58', '29.60']
['20', '特雷-墨菲', '鹈鹕', '21.50', '7.50-15.90', '47%', '3.20-8.60', '37.9%', '3.30-3.70', '88.6%', '66', '35.50']
['20', '布兰登-英格拉姆', '快船', '21.50', '8.00-16.70', '47.7%', '1.80-4.60', '38.2%', '3.80-4.60', '82%', '77', '33.80']
['22', '朱利叶斯-兰德尔', '篮网', '21.10', '7.30-15.30', '48.1%', '1.40-4.40', '31.5%', '5.00-6.30', '80.2%', '79', '33.00']
['23', '库珀-弗拉格', '独行侠', '21.00', '8.00-17.10', '46.8%', '1.00-3.50', '29.5%', '4.00-4.90', '82.7%', '70', '33.50']
['23', '蔡恩-威廉森', '鹈鹕', '21.00', '7.80-13.00', '60%', '0.00-0.10', '25%', '5.40-7.50', '71.6%', '62', '29.70']
['25', '勒布朗-詹姆斯', '', '20.90', '7.90-15.30', '51.5%', '1.30-4.10', '31.7%', '3.90-5.30', '73.7%', '60', '33.20']
['26', '尼基尔-亚历山大-沃克', '老鹰', '20.80', '7.00-15.30', '45.9%', '3.20-8.10', '39.9%', '3.50-3.90', '90.2%', '78', '33.40']
['27', '阿尔佩伦-申京', '火箭', '20.40', '8.10-15.60', '51.9%', '0.60-1.80', '30.5%', '3.60-5.20', '69.1%', '72', '33.30']
['28', '布兰登·米勒', '黄蜂', '20.20', '7.00-16.10', '43.5%', '3.10-8.20', '38.3%', '3.10-3.40', '89.2%', '65', '30.30']
['29', '卡尔-安东尼-唐斯', '尼克斯', '20.10', '6.90-13.80', '50.1%', '1.50-4.10', '36.8%', '4.80-5.60', '85.8%', '75', '31.00']
['29', '德斯蒙德-贝恩', '魔术', '20.10', '7.10-14.70', '48.4%', '2.00-5.20', '39.1%', '3.80-4.20', '90.8%', '82', '33.60']
['29', '拉梅洛-鲍尔', '森林狼', '20.10', '7.00-17.30', '40.7%', '3.80-10.30', '36.8%', '2.20-2.50', '89.9%', '72', '28.00']
['29', '巴姆-阿德巴约', '热火', '20.10', '6.90-15.70', '44.2%', '1.70-5.50', '31.8%', '4.50-5.80', '77.8%', '73', '32.40']
['33', '杰伦-杜伦', '活塞', '19.50', '7.50-11.50', '65%', '0.00-0.00', '0%', '4.60-6.10', '74.7%', '70', '28.20']
['34', '达龙-福克斯', '马刺', '18.60', '7.10-14.50', '48.6%', '1.80-5.50', '33.2%', '2.60-3.40', '76%', '72', '31.00']
['35', '康-克尼普尔', '黄蜂', '18.50', '6.40-13.40', '47.5%', '3.40-7.90', '42.5%', '2.40-2.80', '86.3%', '81', '31.50']
['36', '德马尔-德罗赞', '', '18.40', '6.50-13.10', '49.7%', '0.60-1.90', '32%', '4.90-5.60', '86.8%', '77', '31.20']
['37', '阿门·汤普森', '火箭', '18.30', '7.00-13.20', '53.4%', '0.30-1.50', '21.6%', '3.80-4.90', '77.9%', '79', '37.40']
['38', '埃文-莫布利', '骑士', '18.20', '7.20-13.20', '54.6%', '0.90-3.20', '29.7%', '2.80-4.60', '60.6%', '65', '31.90']
['39', '斯科蒂-巴恩斯', '猛龙', '18.10', '7.10-14.00', '50.7%', '0.90-2.80', '30.4%', '3.00-3.70', '81.5%', '80', '33.50']
['40', '萨迪克-贝', '鹈鹕', '17.70', '6.10-13.40', '45.1%', '2.10-5.70', '36.7%', '3.40-4.10', '84.1%', '72', '31.20']
['41', '莱恩-罗林斯', '雄鹿', '17.30', '6.60-13.90', '47.2%', '2.50-6.10', '40.6%', '1.70-2.10', '79.6%', '74', '32.10']
['42', '切特-霍姆格伦', '雷霆', '17.10', '6.30-11.30', '55.7%', '1.30-3.50', '36.2%', '3.30-4.10', '79.2%', '69', '28.90']
['42', '迈尔斯-布里奇斯', '太阳', '17.10', '6.20-13.50', '46%', '1.90-5.80', '33.3%', '2.70-3.30', '82.2%', '77', '31.00']
['44', '佩顿-普里查德', '凯尔特人', '17.00', '6.40-13.80', '46.4%', '2.70-7.10', '37.7%', '1.50-1.70', '89%', '79', '32.40']
['45', '斯蒂芬·卡斯尔', '马刺', '16.70', '5.70-12.00', '47.1%', '1.20-3.60', '33.2%', '4.20-5.70', '73.4%', '68', '30.00']
['45', 'OG-阿奴诺比', '尼克斯', '16.70', '5.80-12.00', '48.4%', '2.30-6.10', '38.6%', '2.70-3.30', '82.8%', '67', '33.20']
['47', '德里克-怀特', '凯尔特人', '16.50', '5.70-14.40', '39.4%', '2.70-8.30', '32.7%', '2.40-2.70', '90.2%', '77', '34.10']
['48', '伊曼纽尔-奎克利', '猛龙', '16.40', '5.70-12.90', '44.3%', '2.50-6.80', '37.4%', '2.40-3.00', '82.1%', '70', '31.90']
['49', '马塔斯·布泽利斯', '公牛', '16.30', '5.80-12.50', '46.3%', '2.20-6.40', '34.9%', '2.40-3.10', '78.6%', '77', '29.20']
['50', 'VJ-埃奇库姆', '76人', '16.00', '6.00-13.70', '43.8%', '2.00-5.60', '35.4%', '2.00-2.40', '81.8%', '75', '35.00']
相关推荐
科技道人1 小时前
记录 设置-显示大小和字体
开发语言·android设置·字体和显示大小
浮江雾1 小时前
Flutter第三节----Dart中的数据类型
android·开发语言·学习·flutter·入门·函数
想会飞的蒲公英1 小时前
回归模型怎样评估:MAE、MSE、RMSE 和 R²
人工智能·python·机器学习·数据分析·回归
weixin_422329311 小时前
AgentScope Java 项目入门 & Builder 深度解读
java·开发语言·人工智能
henrylin99991 小时前
Quant Agent 完整系统规格
python·量化
三川6982 小时前
Tkinter库的学习记录10-Message与Messagebox
python
吃饱了得干活2 小时前
别只会传 temperature!一文挖透 LangChain 模型配置:Profile、全参数与运行时动态覆盖
python·langchain
shuoyuanAI2 小时前
张家口专业的aigeo优化品牌找哪家
大数据·人工智能·python