Hive 源码

hive 编译

• issue

Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.5.1.1:run (default) on project hive-standalone-metastore: Error resolving artifact: com.google.protobuf:protoc:2.5.0: The following artifacts could not be resolved: com.google.protobuf:protoc:exe:osx-aarch_64:2.5.0 (absent): Could not transfer artifact com.google.protobuf:protoc:exe:osx-aarch_64:2.5.0 from/to maven-default-http-blocker (http://0.0.0.0/)

可以增加编译选项

 -Dos.arch=x86_64

可参见

https://qileq.com/article/202202230001/

• issue

ERROR Failed to execute goal on project hive-service: Could not resolve dependencies for project org.apache.hive:hive-service:jar:3.1.3: Failed to collect dependencies at org.apache.directory.server:apacheds-server-integ:jar:1.5.6 -> org.apache.directory.client.ldap:ldap-client-api:jar:0.1-SNAPSHOT: Failed to read artifact descriptor for org.apache.directory.client.ldap:ldap-client-api:jar:0.1-SNAPSHOT: The following artifacts could not be resolved: org.apache.directory.client.ldap:ldap-client-api:pom:0.1-SNAPSHOT (absent): Could not transfer artifact org.apache.directory.client.ldap:ldap-client-api:pom:0.1-SNAPSHOT from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [tbds (http://tbdsrepo.oa.com/repository/tbds/, default, disabled),

添加 repo

<repository>
    <id>Codehaus repository</id>
    <name>codehaus-mule-repo</name>
    <url>https://repository-master.mulesoft.org/nexus/content/groups/public/</url>
    <layout>default</layout>
</repository>

该问题，可以参见 这个 hive issue ： https://issues.apache.org/jira/browse/HIVE-21777

• issue

  [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile) on project hive-webhcat: Compilation failure
  [ERROR] hive/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java:[258,31] 对于FilterHolder(java.lang.Class<org.apache.hadoop.hdfs.web.AuthFilter>), 找不到合适的构造器
  [ERROR] 构造器 org.eclipse.jetty.servlet.FilterHolder.FilterHolder(org.eclipse.jetty.servlet.Source)不适用
  [ERROR] (参数不匹配; java.lang.Class<org.apache.hadoop.hdfs.web.AuthFilter>无法转换为org.eclipse.jetty.servlet.Source)
  [ERROR] 构造器 org.eclipse.jetty.servlet.FilterHolder.FilterHolder(java.lang.Class<? extends javax.servlet.Filter>)不适用
  [ERROR] (参数不匹配; java.lang.Class<org.apache.hadoop.hdfs.web.AuthFilter>无法转换为java.lang.Class<? extends javax.servlet.Filter>)
  [ERROR] 构造器 org.eclipse.jetty.servlet.FilterHolder.FilterHolder(javax.servlet.Filter)不适用
  [ERROR] (参数不匹配; java.lang.Class<org.apache.hadoop.hdfs.web.AuthFilter>无法转换为javax.servlet.Filter)
  [ERROR]

将 maven 版本 从 3.9.4 降低到 3.2.5 后该问题解决

Hive 写入

即使 设置了 set hive.exec.mode.local.auto=true; 也会回滚，当 input 文件较多时。

Query ID = hadoop_20240109112435_62531b28-9651-4d82-968d-4ff26142326d
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Cannot run job locally: Number of Input Files (= 7) is larger than hive.exec.mode.local.auto.input.files.max(= 4)