使用datax将mysql数据抽取到hive分区表无数据

今天遇到一个问题，使用datax将mysql数据抽取到hive的分区表中，抽取完后没有数据，也不报错

具体是这样的，先常见ods层的表

sql 复制代码

create table ods.sms_coupon_full(
	`id`               string,
	`type`             string,
	`name`             string,
	`platform`         string,
	`count`            string,
	`amount`           string,
	`per_limit`        string,
	`min_point`        string,
	`start_time`       string,
	`end_time`         string,
	`use_type`         string,
	`note`             string,
	`publish_count`    string,
	`use_count`        string,
	`receive_count`    string,
	`enable_time`      string,
	`code`             string,
	`member_level`     string
)partitioned by (dt string)
stored as orc;

接着创建分区：

sql 复制代码

alter table ods.sms_coupon_full add partition (dt='2023-09-25');

使用datax抽取数据到hdfs(其实就是hive)

sql 复制代码

{
  "job": {
    "setting": {
      "speed": {
        "channel": 3
      },
      "errorLimit": {
        "record": 0,
        "percentage": 0.02
      }
    },
    "content": [
      {
        "reader": {
          "name": "mysqlreader",
          "parameter": {
            "username": "root",
            "password": "123456",
            "connection": [
              {
                "querySql": [
                  "select id,type,name,platform,count,amount,per_limit,min_point,start_time,end_time,use_type,note,publish_count,use_count,receive_count,enable_time,code,member_level from sms_coupon"
                ],
                "jdbcUrl": [
                  "jdbc:mysql://bigdata01:3306/spark_project"
                ]
              }
            ]
          }
        },
        "writer": {
          "name": "hdfswriter",
          "parameter": {
            "defaultFS": "hdfs://bigdata01:9820",
            "fileType": "orc",
            "path": "/user/hive/warehouse/ods.db/sms_coupon_full/dt=${dtime}",
            "fileName": "sms_coupon",
            "writeMode": "append",
            "column": [
              {
                "name": "id",
                "type": "bigint"
              },
              {
                "name": "type",
                "type": "int"
              },
              {
                "name": "name",
                "type": "string"
              },
              {
                "name": "platform",
                "type": "int"
              },
              {
                "name": "count",
                "type": "int"
              },
              {
                "name": "amount",
                "type": "string"
              },
              {
                "name": "per_limit",
                "type": "int"
              },
              {
                "name": "min_point",
                "type": "string"
              },
              {
                "name": "start_time",
                "type": "string"
              },
              {
                "name": "end_time",
                "type": "string"
              },
              {
                "name": "use_type",
                "type": "int"
              },
              {
                "name": "note",
                "type": "string"
              },
              {
                "name": "publish_count",
                "type": "int"
              },
              {
                "name": "use_count",
                "type": "int"
              },
              {
                "name": "receive_count",
                "type": "int"
              },
              {
                "name": "enable_time",
                "type": "string"
              },
              {
                "name": "code",
                "type": "string"
              },
              {
                "name": "member_level",
                "type": "int"
              }
            ],
            "fieldDelimiter": "\x01",
            "partition": "dt"
          }
        }
      }
    ]
  }
}

导入之后，假如查询没有数据，使用修复语句修复这个表

msck repair table ods.sms_coupon_history_incr;

然后再查询。

假如是一个普通表，没有分区，查询没有数据，可以通过如下修复：

refresh table dws.new_mid_day;