Hive建表乱码解决--设置编码格式UTF8

1.创建Hive元数据库

Hive元数据存储在MySQL中，因此需要进入MySQL中创建Hive元数据库；若已存在Hive元数据库，则修改元数据库字符格式

hive建库语句：

sql 复制代码

 create database amon DEFAULT CHARSET utf8 COLLATE utf8_general_ci;

只有修改编码后才加入的中文注释才会正常显示，修改编码前已经存在的中文注释会乱码！

sql 复制代码

##创建hive元数据库hive，并指定utf-8编码格式
mysql>create database hive DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
 
 
##修改已存在的hive元数据库，字符编码格式为utf-8
mysql>alter database hive character set utf8;     
 
 
##进入hive元数据库
mysql>use hive;
 
##查看元数据库字符编码格式
mysql>show variables like 'character_set_database';

可以看到原本编码是Hive在搭建时选择的默认格式。

2.修改Hive的元数据信息

Hive启动后，修改Hive的元数据信息，无需重启MySQL和Hive就能生效；

1).修改字段注释字符集(直接复制运行即可)

sql 复制代码

alter table hive.COLUMNS_V2 modify column COMMENT varchar(256) character set utf8

2).修改表注释字符集

sql 复制代码

alter table hive.TABLE_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8

3).修改分区表参数，以支持分区键能够用中文表示

sql 复制代码

alter table hive.PARTITION_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8
alter table hive.PARTITION_KEYS modify column PKEY_COMMENT varchar(4000) character set utf8

4).修改索引注解

sql 复制代码

alter table hive.INDEX_PARAMS modify column PARAM_VALUE varchar(4000) character set utf8;

自此重新建表，乱码问题即可解决

Hive无法创建中文分区

报错如下：

sql 复制代码

hive> alter table page_view add partition(ds='20240618开心');
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Exception thrown when executing query)

解决办法如下：

sql 复制代码

MariaDB [hive]> show create table PARTITIONS;
 
| PARTITIONS | CREATE TABLE `PARTITIONS` (
  `PART_ID` bigint(20) NOT NULL,
  `CREATE_TIME` int(11) NOT NULL,
  `LAST_ACCESS_TIME` int(11) NOT NULL,
  `PART_NAME` varchar(767) CHARACTER SET latin1 COLLATE latin1_bin DEFAULT NULL,
  `SD_ID` bigint(20) DEFAULT NULL,
  `TBL_ID` bigint(20) DEFAULT NULL,
  `LINK_TARGET_ID` bigint(20) DEFAULT NULL,
  PRIMARY KEY (`PART_ID`),
  UNIQUE KEY `UNIQUEPARTITION` (`PART_NAME`,`TBL_ID`),
  KEY `PARTITIONS_N49` (`TBL_ID`),
  KEY `PARTITIONS_N50` (`SD_ID`),
  KEY `PARTITIONS_N51` (`LINK_TARGET_ID`),
  CONSTRAINT `PARTITIONS_FK1` FOREIGN KEY (`TBL_ID`) REFERENCES `TBLS` (`TBL_ID`),
  CONSTRAINT `PARTITIONS_FK2` FOREIGN KEY (`SD_ID`) REFERENCES `SDS` (`SD_ID`),
  CONSTRAINT `PARTITIONS_FK3` FOREIGN KEY (`LINK_TARGET_ID`) REFERENCES `PARTITIONS` (`PART_ID`)
) ENGINE=InnoDB DEFAULT CHARSET=latin1 |
 
MariaDB [hive]> alter table PARTITIONS  modify column `PART_NAME` varchar(767) character set utf8;
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes
MariaDB [hive]> alter table PARTITIONS  modify column `PART_NAME` varchar(100) character set utf8;   
Query OK, 0 rows affected (0.01 sec)               
Records: 0  Duplicates: 0  Warnings: 0
 
MariaDB [hive]> alter table PARTITIONS  modify column `PART_NAME` varchar(300) character set utf8;   
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes
MariaDB [hive]> alter table PARTITIONS  modify column `PART_NAME` varchar(200) character set utf8;   
Query OK, 0 rows affected (0.00 sec)               
Records: 0  Duplicates: 0  Warnings: 0
 
MariaDB [hive]> alter table PARTITIONS  modify column `PART_NAME` varchar(250) character set utf8;  
Query OK, 0 rows affected (0.00 sec)               
Records: 0  Duplicates: 0  Warnings: 0
 
MariaDB [hive]> alter table PARTITIONS  modify column `PART_NAME` varchar(260) character set utf8;  
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes

另验证一个问题，utf8占用三个字节，之前默认是767，若指定250 * 3 = 750可以，但260*3=780不可以；

再去创建中文分区测试即可成功

sql 复制代码

alter table page_view add partition(ds='20240618开心');

插入数据

sql 复制代码

insert into page_view  partition(ds='20240618') values (1,"张三","李四") ;

查看包含中文的数据

sql 复制代码

select * from page_view where ds="20240618";

查看索引

sql 复制代码

SHOW FORMATTED INDEX ON page_view;

查看表结构

sql 复制代码

desc page_view;