python-NLP常用数据集0.1.022

python-NLP常用数据集0.1.022

MRPC数据集

Microsoft Research Paraphrase Corpus 3600个数据

  1. 下载地址:https://gist.github.com/W4ngatang/60c2bdb54d156a41194446737ce03e2e
  2. 格式
python 复制代码
Quality	#1 ID	#2 ID	#1 String	#2 String
1	1089874	1089925	PCCW 's chief operating officer , Mike Butcher , and Alex Arena , the chief financial officer , will report directly to Mr So .	Current Chief Operating Officer Mike Butcher and Group Chief Financial Officer Alex Arena will report to So .
1	3019446	3019327	The world 's two largest automakers said their U.S. sales declined more than predicted last month as a late summer sales frenzy caused more of an industry backlash than expected .	Domestic sales at both GM and No. 2 Ford Motor Co. declined more than predicted as a late summer sales frenzy prompted a larger-than-expected industry backlash .
1	1945605	1945824	According to the federal Centers for Disease Control and Prevention ( news - web sites ) , there were 19 reported cases of measles in the United States in 2002 .	The Centers for Disease Control and Prevention said there were 19 reported cases of measles in the United States in 2002 .
0	1430402	1430329	A tropical storm rapidly developed in the Gulf of Mexico Sunday and was expected to hit somewhere along the Texas or Louisiana coasts by Monday night .	A tropical storm rapidly developed in the Gulf of Mexico on Sunday and could have hurricane-force winds when it hits land somewhere along the Louisiana coast Monday night .
0	3354381	3354396	The company didn 't detail the costs of the replacement and repairs .	But company officials expect the costs of the replacement work to run into the millions of dollars .
1	1390995	1391183	The settling companies would also assign their possible claims against the underwriters to the investor plaintiffs , he added .	Under the agreement , the settling companies will also assign their potential claims against the underwriters to the investors , he added .
0

XNLI数据集

用户语言翻译和跨语言分类的语料库

  1. 官网地址:https://github.com/facebookresearch/XNLI
  2. 下载地址:https://dl.fbaipublicfiles.com/XNLI/XNLI-1.0.zip
  3. 注意事项:数据集有json格式的,和txt格式的
  4. 数据格式

txt格式

python 复制代码
language	gold_label	sentence1_binary_parse	sentence2_binary_parse	sentence1_parse	sentence2_parse	sentence1	sentence2	promptID	pairID	genre	label1	label2	label3	label4	label5	sentence1_tokenized	sentence2_tokenized	match
ar	neutral					وقال، ماما، لقد عدت للمنزل.	اتصل بأمه حالما أوصلته حافلة المدرسية.	1	1	facetoface	neutral	contradiction	neutral	neutral	neutral	وقال ، ماما ، لقد عدت للمنزل .	اتصل بأمه حالما أوصلته حافلة المدرسية .	True
ar	contradiction					وقال، ماما، لقد عدت للمنزل.	لم ينطق ببنت شفة.	1	2	facetoface	contradiction	contradiction	contradiction	contradiction	contradiction	وقال ، ماما ، لقد عدت للمنزل .	لم ينطق ببنت شفة .	True
ar	entailment					وقال، ماما، لقد عدت للمنزل.	أخبر أمه أنه قد عاد للمنزل.	1	3	facetoface	entailment	entailment	neutral	entailment	entailment	وقال ، ماما ، لقد عدت للمنزل .	أخبر أمه أنه قد عاد للمنزل .	True
ar	neutral	

json格式

python 复制代码
{"annotator_labels": ["neutral", "contradiction", "neutral", "neutral", "neutral"], "genre": "facetoface", "gold_label": "neutral", "language": "ar", "match": "True", "pairID": "1", "promptID": "1", "sentence1": "\u0648\u0642\u0627\u0644\u060c \u0645\u0627\u0645\u0627\u060c \u0644\u0642\u062f \u0639\u062f\u062a \u0644\u0644\u0645\u0646\u0632\u0644.", "sentence1_tokenized": "\u0648\u0642\u0627\u0644 \u060c \u0645\u0627\u0645\u0627 \u060c \u0644\u0642\u062f \u0639\u062f\u062a \u0644\u0644\u0645\u0646\u0632\u0644 .", "sentence2": "\u0627\u062a\u0635\u0644 \u0628\u0623\u0645\u0647 \u062d\u0627\u0644\u0645\u0627 \u0623\u0648\u0635\u0644\u062a\u0647 \u062d\u0627\u0641\u0644\u0629 \u0627\u0644\u0645\u062f\u0631\u0633\u064a\u0629.", "sentence2_tokenized": "\u0627\u062a\u0635\u0644 \u0628\u0623\u0645\u0647 \u062d\u0627\u0644\u0645\u0627 \u0623\u0648\u0635\u0644\u062a\u0647 \u062d\u0627\u0641\u0644\u0629 \u0627\u0644\u0645\u062f\u0631\u0633\u064a\u0629 ."}
{"annotator_labels": ["contradiction", "contradiction", "contradiction", "contradiction", "contradiction"], "genre": "facetoface", "gold_label": "contradiction", "language": "ar", "match": "True", "pairID": "2", "promptID": "1", "sentence1": "\u0648\u0642\u0627\u0644\u060c \u0645\u0627\u0645\u0627\u060c \u0644\u0642\u062f \u0639\u062f\u062a \u0644\u0644\u0645\u0646\u0632\u0644.", "sentence1_tokenized": "\u0648\u0642\u0627\u0644 \u060c \u0645\u0627\u0645\u0627 \u060c \u0644\u0642\u062f \u0639\u062f\u062a \u0644\u0644\u0645\u0646\u0632\u0644 .", "sentence2": "\u0644\u0645 \u064a\u0646\u0637\u0642 \u0628\u0628\u0646\u062a \u0634\u0641\u0629.", "sentence2_tokenized": "\u0644\u0645 \u064a\u0646\u0637\u0642 \u0628\u0628\u0646\u062a \u0634\u0641\u0629 ."}

SQuAD数据集

  1. 官网地址:https://rajpurkar.github.io/SQuAD-explorer/
  2. 下载地址:https://rajpurkar.github.io/SQuAD-explorer/
  3. 注意事项:测试集没有给出,需要在官网提交模型由平台对模型进行测试集的跑分
  4. 数据格式:点击https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v1.1.json

数据由多篇文章组成

一个title就表示一篇文章

文章里由paragraphs组成

paragraphs由多个context组成

每一个context有answers和question

部分数据:

python 复制代码
{
	"data": [{
		"title": "Super_Bowl_50",
		"paragraphs": [{
			"context": "Super Bowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) champion Denver Broncos defeated the National Football Conference (NFC) champion Carolina Panthers 24\u201310 to earn their third Super Bowl title. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the league emphasized the \"golden anniversary\" with various gold-themed initiatives, as well as temporarily suspending the tradition of naming each Super Bowl game with Roman numerals (under which the game would have been known as \"Super Bowl L\"), so that the logo could prominently feature the Arabic numerals 50.",
			"qas": [{
				"answers": [{
					"answer_start": 177,
					"text": "Denver Broncos"
				}, {
					"answer_start": 177,
					"text": "Denver Broncos"
				}, {
					"answer_start": 177,
					"text": "Denver Broncos"
				}],
				"question": "Which NFL team represented the AFC at Super Bowl 50?",
				"id": "56be4db0acb8001400a502ec"
			}, {
				"answers": [{
					"answer_start": 249,
					"text": "Carolina Panthers"
				}, {
					"answer_start": 249,
					"text": "Carolina Panthers"
				}, {
					"answer_start": 249,
					"text": "Carolina Panthers"
				}],
				"question": "Which NFL team represented the NFC at Super Bowl 50?",
				"id": "56be4db0acb8001400a502ed"
			}, {
				"answers": [{
					"answer_start": 403,
					"text": "Santa Clara, California"
				}, {
					"answer_start": 355,
					"text": "Levi's Stadium"
				}, {
					"answer_start": 355,
					"text": "Levi's Stadium in the San Francisco Bay Area at Santa Clara, California."
				}],
				"question": "Where did Super Bowl 50 take place?",
				"id": "56be4db0acb8001400a502ee"
			}, {
				"answers": [{
					"answer_start": 177,
					"text": "Denver Broncos"
				}, {
					"answer_start": 177,
					"text": "Denver Broncos"
				}, {
					"answer_start": 177,
					"text": "Denver Broncos"
				}],
				"question": "Which NFL team won Super Bowl 50?",
				"id": "56be4db0acb8001400a502ef"
			}, {
				"answers": [{
					"answer_start": 488,
					"text": "gold"
				}, {
					"answer_start": 488,
					"text": "gold"
				}, {
					"answer_start": 521,
					"text": "gold"
				}],
				"question": "What color was used to emphasize the 50th anniversary of the Super Bowl?",
				"id": "56be4db0acb8001400a502f0"
			}
相关推荐
时寒的笔记34 分钟前
js入门01
开发语言·前端·javascript
_extraordinary_39 分钟前
Java 栈和队列
java·开发语言
永远孤独的菜鸟1 小时前
# 全国职业院校技能大赛中职组“网络建设与运维“赛项项目方案
python
mit6.8241 小时前
[Meetily后端框架] 多模型-Pydantic AI 代理-统一抽象 | SQLite管理
c++·人工智能·后端·python
一眼万里*e1 小时前
Python 字典 (Dictionary) 详解
前端·数据库·python
wjs20241 小时前
Django ORM - 多表实例
开发语言
Joker—H2 小时前
【Java】JUC并发(线程的方法、多线程的同步并发)
java·开发语言·经验分享·idea
_kaika12 小时前
使用 1Panel PHP 运行环境部署 WordPress
开发语言·php
isNotNullX2 小时前
数据怎么分层?从ODS、DW、ADS三大层一一拆解!
大数据·开发语言·数据仓库·分布式·spark
wuyoula2 小时前
deepseekAI对接大模型的网页PHP源码带管理后台(可实现上传分析文件)
开发语言·php