Hive 源码

hive 编译
  • issue

Failed to execute goal com.github.os72:protoc-jar-maven-plugin:3.5.1.1:run (default) on project hive-standalone-metastore: Error resolving artifact: com.google.protobuf:protoc:2.5.0: The following artifacts could not be resolved: com.google.protobuf:protoc:exe:osx-aarch_64:2.5.0 (absent): Could not transfer artifact com.google.protobuf:protoc:exe:osx-aarch_64:2.5.0 from/to maven-default-http-blocker (http://0.0.0.0/)

可以增加编译选项

 -Dos.arch=x86_64

可参见

https://qileq.com/article/202202230001/

  • issue

[ERROR] Failed to execute goal on project hive-service: Could not resolve dependencies for project org.apache.hive:hive-service:jar:3.1.3: Failed to collect dependencies at org.apache.directory.server:apacheds-server-integ:jar:1.5.6 -> org.apache.directory.client.ldap:ldap-client-api:jar:0.1-SNAPSHOT: Failed to read artifact descriptor for org.apache.directory.client.ldap:ldap-client-api:jar:0.1-SNAPSHOT: The following artifacts could not be resolved: org.apache.directory.client.ldap:ldap-client-api:pom:0.1-SNAPSHOT (absent): Could not transfer artifact org.apache.directory.client.ldap:ldap-client-api:pom:0.1-SNAPSHOT from/to maven-default-http-blocker (http://0.0.0.0/): Blocked mirror for repositories: [tbds (http://tbdsrepo.oa.com/repository/tbds/, default, disabled),

添加 repo

<repository>
    <id>Codehaus repository</id>
    <name>codehaus-mule-repo</name>
    <url>https://repository-master.mulesoft.org/nexus/content/groups/public/</url>
    <layout>default</layout>
</repository>

该问题,可以参见 这个 hive issue : https://issues.apache.org/jira/browse/HIVE-21777

  • issue

    [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.6.1:compile (default-compile) on project hive-webhcat: Compilation failure
    [ERROR] hive/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/Main.java:[258,31] 对于FilterHolder(java.lang.Class<org.apache.hadoop.hdfs.web.AuthFilter>), 找不到合适的构造器
    [ERROR] 构造器 org.eclipse.jetty.servlet.FilterHolder.FilterHolder(org.eclipse.jetty.servlet.Source)不适用
    [ERROR] (参数不匹配; java.lang.Class<org.apache.hadoop.hdfs.web.AuthFilter>无法转换为org.eclipse.jetty.servlet.Source)
    [ERROR] 构造器 org.eclipse.jetty.servlet.FilterHolder.FilterHolder(java.lang.Class<? extends javax.servlet.Filter>)不适用
    [ERROR] (参数不匹配; java.lang.Class<org.apache.hadoop.hdfs.web.AuthFilter>无法转换为java.lang.Class<? extends javax.servlet.Filter>)
    [ERROR] 构造器 org.eclipse.jetty.servlet.FilterHolder.FilterHolder(javax.servlet.Filter)不适用
    [ERROR] (参数不匹配; java.lang.Class<org.apache.hadoop.hdfs.web.AuthFilter>无法转换为javax.servlet.Filter)
    [ERROR]

将 maven 版本 从 3.9.4 降低到 3.2.5 后该问题解决

Hive 写入

即使 设置了 set hive.exec.mode.local.auto=true; 也会回滚,当 input 文件较多时。

Query ID = hadoop_20240109112435_62531b28-9651-4d82-968d-4ff26142326d
Total jobs = 3
Launching Job 1 out of 3
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Cannot run job locally: Number of Input Files (= 7) is larger than hive.exec.mode.local.auto.input.files.max(= 4)
相关推荐
武子康1 小时前
大数据-258 离线数仓 - Griffin架构 配置安装 Livy 架构设计 解压配置 Hadoop Hive
java·大数据·数据仓库·hive·hadoop·架构
18号房客8 小时前
高级sql技巧进阶教程
大数据·数据库·数据仓库·sql·mysql·时序数据库·数据库架构
Faith_xzc10 小时前
【Apache Doris】周FAQ集锦:第 26 期
大数据·数据仓库·开源·doris
筒栗子13 小时前
复习打卡大数据篇——Hadoop MapReduce
大数据·hadoop·mapreduce
金州饿霸13 小时前
Hadoop集群(HDFS集群、YARN集群、MapReduce计算框架)
大数据·hadoop·hdfs
lucky_syq14 小时前
Hive SQL和Spark SQL的区别?
hive·sql·spark
NiNg_1_23414 小时前
Hadoop中MapReduce过程中Shuffle过程实现自定义排序
大数据·hadoop·mapreduce
lucky_syq16 小时前
Spark和Hadoop之间的区别
大数据·hadoop·spark
武子康1 天前
大数据-257 离线数仓 - 数据质量监控 监控方法 Griffin架构
java·大数据·数据仓库·hive·hadoop·后端
莹雨潇潇1 天前
Hadoop完全分布式环境部署
大数据·hadoop·分布式