文章目录
将 Hadoop 编译为本地(native)代码,加速文件编码、压缩、传输
安装本地工具
bash
sudo yum install -y cmake gcc-c++ libtirpc-devel isa-l-devel snappy-devel zlib-devel openssl-devel libpmem-devel
gcc高版本(比如11)无法编译成功,会报错:ISO C++17 does not allow dynamic exception specifications
编译protoc
需使用旧版(2.5.0)protoc命令,否则报错:expected version is '2.5.0'
bash
# 下载protobuf-2.5.0源码
./configure --prefix=/apps/svr/protobuf-2.5.0
make
make check
make install
# 优先使用旧版(2.5.0)protoc命令
export PATH=/apps/svr/protobuf-2.5.0/bin:$PATH
编译选项
使用tirpc作为rpc依赖,否则报错:
rpc/types.h: No such file or directory
undefined reference to 'xdrmem_create'
bash
# 指定头文件
export CXXFLAGS="-I/usr/include/tirpc"
# 指定链接库
export LDFLAGS="-ltirpc"
maven仓库
DynamoDBLocal包需要从amazonaws下载,修改maven的settings.xml文件,使用mirror配置引导到国内可访问的下载地址:
xml
<settings>
<mirrors>
<mirror>
<mirrorOf>dynamodb-local-oregon</mirrorOf>
<id>aws</id>
<url>https://dynamodb-local.s3.amazonaws.com/release</url>
</mirror>
</mirrors>
</settings>
编译hadoop
制作包含native code的tar包
bash
mvn -Pdist,native -DskipTests -Dtar -Drequire.isal -Drequire.openssl -Drequire.snappy -Drequire.pmdk -Dmaven.javadoc.skip=true clean package -e -X
# 确认是否使用了本地的动态链接库
cd ./hadoop-dist/target/hadoop-3.1.2/bin
./hadoop checknative