一、问题背景
1.1 环境信息
| 项目 | 版本 |
|---|---|
| 操作系统 | Ubuntu 16.04 LTS |
| DPDK 版本 | 19.08.2 |
| GCC 版本 | 5.4.0 |
| Binutils | 2.26.1 |
| 目标架构 | x86_64 |
1.2 原始错误信息
使用 ./usertools/dpdk-setup.sh 编译 DPDK 时遇到以下错误:
CC portal/dpaa2_hw_dpio.o
PMDINFO portal/dpaa2_hw_dpio.o.pmd.c
CC portal/dpaa2_hw_dpio.o.pmd.o
LD portal/dpaa2_hw_dpio.o
ld: i386:x86-64 architecture of input file `portal/dpaa2_hw_dpio.o.pmd.o' is incompatible with i386 output
ld: Relocatable linking with relocations from format elf64-x86-64 (portal/dpaa2_hw_dpio.o) to format elf32-i386 (portal/dpaa2_hw_dpio.o.o) is not supported
/home/jerry/Desktop/dpdk-stable-19.08.2/mk/internal/rte.compile-pre.mk:114: recipe for target 'portal/dpaa2_hw_dpio.o' failed
make[6]: *** [portal/dpaa2_hw_dpio.o] Error 1
...
RTE_TARGET exported as x86_64-native-linuxapp-gcc
错误关键信息:
- 输入文件架构:
elf64-x86-64(64 位) - 输出格式:
elf32-i386(32 位) - 问题文件:
portal/dpaa2_hw_dpio.o(属于 FSLMC/DPAA2 驱动) - 目标架构设置:
x86_64-native-linuxapp-gcc(看起来是正确的)
二、调试过程
步骤 1:尝试修改配置文件禁用 ARM 驱动
首先尝试修改 config/common_linuxapp:
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2/usertools# sed -i 's/CONFIG_RTE_LIBRTE_FSLMC_BUS=y/CONFIG_RTE_LIBRTE_FSLMC_BUS=n/' config/common_linuxapp
sed: can't read config/common_linuxapp: No such file or directory
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2/usertools# sed -i 's/CONFIG_RTE_LIBRTE_DPAA2_PMD=y/CONFIG_RTE_LIBRTE_DPAA2_PMD=n/' config/common_linuxapp
sed: can't read config/common_linuxapp: No such file or directory
问题 :配置文件路径不对,当前在 usertools 目录下,且 DPDK 19.08 使用的是 config/common_base。
步骤 2:回到正确目录,修改正确的配置文件
cd /home/jerry/Desktop/dpdk-stable-19.08.2
ls config/
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# cd /home/jerry/Desktop/dpdk-stable-19.08.2
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# ls config/
arm64-armada-linuxapp-gcc defconfig_x86_64-native-linuxapp-clang
arm64-armada-linux-gcc defconfig_x86_64-native-linuxapp-gcc
arm64-armv8a-linuxapp-clang defconfig_x86_64-native-linuxapp-icc
arm64-armv8a-linuxapp-gcc defconfig_x86_64-native-linux-clang
arm64-armv8a-linux-clang defconfig_x86_64-native-linux-gcc
arm64-armv8a-linux-gcc defconfig_x86_64-native-linux-icc
arm64-bluefield-linuxapp-gcc defconfig_x86_x32-native-linuxapp-gcc
arm64-bluefield-linux-gcc defconfig_x86_x32-native-linux-gcc
arm64-dpaa2-linuxapp-gcc meson.build
arm64-dpaa2-linux-gcc ppc_64-power8-linuxapp-gcc
arm64-dpaa-linuxapp-gcc ppc_64-power8-linux-gcc
arm64-dpaa-linux-gcc rte_config.h
arm64-octeontx2-linuxapp-gcc rte_compatibility_defines.h
arm64-octeontx2-linux-gcc rte_config.h.in
arm64-stingray-linuxapp-gcc x86_64-native-bsdapp-clang
arm64-stingray-linux-gcc x86_64-native-bsdapp-gcc
arm64-thunderx2-linuxapp-gcc x86_64-native-freebsd-clang
arm64-thunderx2-linux-gcc x86_64-native-freebsd-gcc
arm64-thunderx-linuxapp-gcc x86_64-native-linuxapp-clang
arm64-thunderx-linux-gcc x86_64-native-linuxapp-gcc
arm64-xgene1-linuxapp-gcc x86_64-native-linuxapp-icc
arm64-xgene1-linux-gcc x86_64-native-linux-clang
arm-armv7a-linuxapp-gcc x86_64-native-linux-gcc
arm-armv7a-linux-gcc x86_64-native-linux-icc
common_base x86_x32-native-linuxapp-gcc
common_linux x86_x32-native-linux-gcc
defconfig_i686-native-linuxapp-gcc
defconfig_i686-native-linuxapp-icc
defconfig_i686-native-linux-gcc
defconfig_i686-native-linux-icc
找到 common_base 文件,查看 DPAA/FSLMC 配置状态:
grep -E "FSLMC|DPAA2" config/common_base | head -20
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# grep -E "FSLMC|DPAA2" config/common_base | head -20
# Compile NXP DPAA2 FSL-MC Bus
CONFIG_RTE_LIBRTE_FSLMC_BUS=n
# Compile Support Libraries for NXP DPAA2
CONFIG_RTE_LIBRTE_DPAA2_MEMPOOL=n
CONFIG_RTE_LIBRTE_DPAA2_USE_PHYS_IOVA=y
# Compile burst-oriented NXP DPAA2 PMD driver
CONFIG_RTE_LIBRTE_DPAA2_PMD=n
CONFIG_RTE_LIBRTE_DPAA2_DEBUG_DRIVER=n
# Compile NXP DPAA2 crypto sec driver for CAAM HW
CONFIG_RTE_LIBRTE_PMD_DPAA2_SEC=n
# Compile PMD for NXP DPAA2 event device
CONFIG_RTE_LIBRTE_PMD_DPAA2_EVENTDEV=n
# Compile PMD for NXP DPAA2 CMDIF raw device
CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV=n
# Compile PMD for NXP DPAA2 QDMA raw device
CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=n
配置文件中已经是 =n,说明问题不在这里。
步骤 3:清理并重新编译
# 清理
rm -rf build
rm -f .config
# 重新配置并编译
make config T=x86_64-native-linuxapp-gcc
make -j$(nproc)
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# rm -rf build
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# rm -f .config
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# make config T=x86_64-native-linuxapp-gcc
Configuration done using x86_64-native-linuxapp-gcc
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# make -j$(nproc)
...
CC portal/dpaa2_hw_dpio.o
PMDINFO portal/dpaa2_hw_dpio.o.pmd.c
CC portal/dpaa2_hw_dpio.o.pmd.o
LD portal/dpaa2_hw_dpio.o
ld: i386:x86-64 architecture of input file `portal/dpaa2_hw_dpio.o.pmd.o' is incompatible with i386 output
ld: Relocatable linking with relocations from format elf64-x86-64 (portal/dpaa2_hw_dpio.o) to format elf32-i386 (portal/dpaa2_hw_dpio.o.o) is not supported
/home/jerry/Desktop/dpdk-stable-19.08.2/mk/internal/rte.compile-pre.mk:114: recipe for target 'portal/dpaa2_hw_dpio.o' failed
make[4]: *** [portal/dpaa2_hw_dpio.o] Error 1
...
make: *** [all] Error 2
问题依旧 :即使 config/common_base 中已经设置为 =n,编译时仍然在尝试编译 FSLMC 驱动。
步骤 4:检查 build/.config 实际配置
关键发现:make config 生成的 build/.config 才是实际使用的配置文件。
grep -E "FSLMC|DPAA" build/.config | grep -v "^#"
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# grep -E "FSLMC|DPAA" build/.config | grep -v "^#"
CONFIG_RTE_LIBRTE_COMMON_DPAAX=y
CONFIG_RTE_LIBRTE_DPAA_BUS=y
CONFIG_RTE_LIBRTE_DPAA_MEMPOOL=y
CONFIG_RTE_LIBRTE_DPAA_PMD=y
CONFIG_RTE_LIBRTE_DPAA_HWDEBUG=n
CONFIG_RTE_LIBRTE_FSLMC_BUS=y
CONFIG_RTE_LIBRTE_DPAA2_MEMPOOL=y
CONFIG_RTE_LIBRTE_DPAA2_USE_PHYS_IOVA=y
CONFIG_RTE_LIBRTE_DPAA2_PMD=y
CONFIG_RTE_LIBRTE_DPAA2_DEBUG_DRIVER=n
CONFIG_RTE_LIBRTE_PMD_DPAA2_SEC=y
CONFIG_RTE_LIBRTE_PMD_DPAA_SEC=y
CONFIG_RTE_LIBRTE_DPAA_MAX_CRYPTODEV=4
CONFIG_RTE_LIBRTE_PMD_DPAA_EVENTDEV=y
CONFIG_RTE_LIBRTE_PMD_DPAA2_EVENTDEV=y
CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV=y
CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=y
关键发现 :build/.config 中大量的 ARM 驱动配置仍然是 =y!config/common_base 的修改没有生效。
步骤 5:直接修改 build/.config
sed -i 's/\(CONFIG_RTE_LIBRTE.*DPAA.*\)=y/\1=n/g' build/.config
sed -i 's/\(CONFIG_RTE_LIBRTE.*FSLMC.*\)=y/\1=n/g' build/.config
# 验证
grep -E "FSLMC|DPAA" build/.config | grep "=y"
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sed -i 's/\(CONFIG_RTE_LIBRTE.*DPAA.*\)=y/\1=n/g' build/.config
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sed -i 's/\(CONFIG_RTE_LIBRTE.*FSLMC.*\)=y/\1=n/g' build/.config
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# grep -E "FSLMC|DPAA" build/.config | grep "=y"
CONFIG_RTE_LIBRTE_DPAA_MEMPOOL=y
CONFIG_RTE_LIBRTE_DPAA_PMD=y
CONFIG_RTE_LIBRTE_DPAA2_MEMPOOL=y
CONFIG_RTE_LIBRTE_DPAA2_USE_PHYS_IOVA=y
CONFIG_RTE_LIBRTE_DPAA2_PMD=y
CONFIG_RTE_LIBRTE_PMD_DPAA2_SEC=y
CONFIG_RTE_LIBRTE_PMD_DPAA_SEC=y
CONFIG_RTE_LIBRTE_PMD_DPAA_EVENTDEV=y
CONFIG_RTE_LIBRTE_PMD_DPAA2_EVENTDEV=y
CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV=y
CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=y
问题 :sed 正则没有完全匹配所有配置项(有些配置项名称不包含 LIBRTE)。
步骤 6:彻底禁用所有 ARM 相关配置
需要更彻底地修改:
# 彻底删除 build 目录重新开始
rm -rf build
# 先修改 config/common_base
sed -i 's/\(CONFIG_RTE_LIBRTE.*DPAA.*\)=y/\1=n/g' config/common_base
sed -i 's/\(CONFIG_RTE_LIBRTE.*FSLMC.*\)=y/\1=n/g' config/common_base
sed -i 's/\(CONFIG_RTE_LIBRTE.*OCTEONTX.*\)=y/\1=n/g' config/common_base
sed -i 's/\(CONFIG_RTE_LIBRTE.*OTX2.*\)=y/\1=n/g' config/common_base
sed -i 's/\(CONFIG_RTE_LIBRTE.*THUNDERX.*\)=y/\1=n/g' config/common_base
# 验证
grep -E "DPAA|FSLMC|OCTEON|OTX2|THUNDER" config/common_base | grep "=y" || echo "已全部禁用"
# 重新配置
make config T=x86_64-native-linuxapp-gcc
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# rm -rf build
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sed -i 's/\(CONFIG_RTE_LIBRTE.*DPAA.*\)=y/\1=n/g' config/common_base
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sed -i 's/\(CONFIG_RTE_LIBRTE.*FSLMC.*\)=y/\1=n/g' config/common_base
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sed -i 's/\(CONFIG_RTE_LIBRTE.*OCTEONTX.*\)=y/\1=n/g' config/common_base
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sed -i 's/\(CONFIG_RTE_LIBRTE.*OTX2.*\)=y/\1=n/g' config/common_base
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sed -i 's/\(CONFIG_RTE_LIBRTE.*THUNDERX.*\)=y/\1=n/g' config/common_base
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# grep -E "DPAA|FSLMC|OCTEON|OTX2|THUNDER" config/common_base | grep "=y" || echo "已全部禁用"
已全部禁用
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# make config T=x86_64-native-linuxapp-gcc
Configuration done using x86_64-native-linuxapp-gcc
检查生成的 .config:
grep -E "DPAA|FSLMC|OCTEON|OTX2|THUNDER" build/.config | grep "=y"
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# grep -E "DPAA|FSLMC|OCTEON|OTX2|THUNDER" build/.config | grep "=y"
CONFIG_RTE_LIBRTE_COMMON_DPAAX=y
CONFIG_RTE_LIBRTE_DPAA_BUS=y
CONFIG_RTE_LIBRTE_DPAA_MEMPOOL=y
CONFIG_RTE_LIBRTE_DPAA_PMD=y
CONFIG_RTE_LIBRTE_FSLMC_BUS=y
CONFIG_RTE_LIBRTE_DPAA2_MEMPOOL=y
CONFIG_RTE_LIBRTE_DPAA2_PMD=y
CONFIG_RTE_LIBRTE_PMD_DPAA2_SEC=y
CONFIG_RTE_LIBRTE_PMD_DPAA_SEC=y
CONFIG_RTE_LIBRTE_PMD_DPAA_EVENTDEV=y
CONFIG_RTE_LIBRTE_PMD_DPAA2_EVENTDEV=y
CONFIG_RTE_LIBRTE_PMD_DPAA2_CMDIF_RAWDEV=y
CONFIG_RTE_LIBRTE_PMD_DPAA2_QDMA_RAWDEV=y
关键发现 :make config 生成的 .config 中 ARM 驱动仍然是 =y!说明配置生成过程中有其他来源覆盖了 common_base 的设置。
步骤 7:直接修改 build/.config 并编译
既然 make config 不受 common_base 控制,直接修改生成的 .config:
sed -i 's/\(CONFIG_RTE_LIBRTE.*DPAA.*\)=y/\1=n/g' build/.config
sed -i 's/\(CONFIG_RTE_LIBRTE.*FSLMC.*\)=y/\1=n/g' build/.config
sed -i 's/\(CONFIG_RTE_LIBRTE.*OCTEONTX.*\)=y/\1=n/g' build/.config
sed -i 's/\(CONFIG_RTE_LIBRTE.*OTX2.*\)=y/\1=n/g' build/.config
sed -i 's/\(CONFIG_RTE_LIBRTE.*THUNDERX.*\)=y/\1=n/g' build/.config
# 验证
grep -E "DPAA|FSLMC|OCTEON|OTX2|THUNDER" build/.config | grep "=y" || echo "配置正确"
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sed -i 's/\(CONFIG_RTE_LIBRTE.*DPAA.*\)=y/\1=n/g' build/.config
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sed -i 's/\(CONFIG_RTE_LIBRTE.*FSLMC.*\)=y/\1=n/g' build/.config
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sed -i 's/\(CONFIG_RTE_LIBRTE.*OCTEONTX.*\)=y/\1=n/g' build/.config
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sed -i 's/\(CONFIG_RTE_LIBRTE.*OTX2.*\)=y/\1=n/g' build/.config
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sed -i 's/\(CONFIG_RTE_LIBRTE.*THUNDERX.*\)=y/\1=n/g' build/.config
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# grep -E "DPAA|FSLMC|OCTEON|OTX2|THUNDER" build/.config | grep "=y" || echo "配置正确"
配置正确
配置修改成功,开始编译:
make -j$(nproc)
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# make -j$(nproc)
...
== Build drivers/mempool/ring
== Build drivers/mempool/octeontx2
== Build drivers/mempool/stack
CC rte_mempool_bucket.o
CC rte_mempool_ring.o
CC otx2_mempool.o
CC otx2_mempool_irq.o
CC otx2_mempool_ops.o
CC otx2_mempool_debug.o
CC rte_mempool_stack.o
CC octeontx_fpavf.o
PMDINFO octeontx_fpavf.o.pmd.c
CC octeontx_fpavf.o.pmd.o
LD octeontx_fpavf.o
ld: i386:x86-64 architecture of input file `octeontx_fpavf.o.pmd.o' is incompatible with i386 output
ld: Relocatable linking with relocations from format elf64-x86-64 (octeontx_fpavf.o) to format elf32-i386 (octeontx_fpavf.o.o) is not supported
...
make: *** [all] Error 2
新问题 :现在报错的是 OcteonTX 驱动(另一个 ARM 平台驱动),而且错误信息显示架构不匹配问题依然存在。
这说明有两个问题:
- 还有其他 ARM 驱动没有被禁用(OcteonTX)
- 架构不匹配的问题根源没有解决
步骤 8:深入分析架构问题
检查架构配置和链接器设置:
# 检查架构配置
echo "=== 检查架构设置 ==="
grep -E "^CONFIG_RTE_ARCH" build/.config
# 检查工具链设置
echo "=== 检查工具链设置 ==="
grep -E "TOOLCHAIN|CC=|LD=" build/.config | head -10
# 检查编译器
echo "=== 检查编译器 ==="
which gcc
gcc --version | head -1
# 检查链接器默认架构
echo "=== 检查链接器 ==="
ld -V
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# echo "=== 检查架构设置 ==="
=== 检查架构设置 ===
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# grep -E "^CONFIG_RTE_ARCH" build/.config
CONFIG_RTE_ARCH="x86_64"
CONFIG_RTE_ARCH_STRICT_ALIGN=n
CONFIG_RTE_ARCH_X86_64=y
CONFIG_RTE_ARCH_X86=y
CONFIG_RTE_ARCH_64=y
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# echo "=== 检查工具链设置 ==="
=== 检查工具链设置 ===
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# grep -E "TOOLCHAIN|CC=|LD=" build/.config | head -10
# RTE_TOOLCHAIN values are architecture we compile for. directories in mk/toolchain/
CONFIG_RTE_TOOLCHAIN="gcc"
CONFIG_RTE_TOOLCHAIN_GCC=y
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# echo "=== 检查编译器 ==="
=== 检查编译器 ===
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# which gcc
/usr/bin/gcc
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# gcc --version | head -1
gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# echo "=== 检查链接器 ==="
=== 检查链接器 ===
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# ld -V
GNU ld (GNU Binutils for Ubuntu) 2.26.1
Supported emulations:
elf_i386
i386linux
elf_iamcu
elf32_x86_64
elf_x86_64
elf_l1om
elf_k1om
i386pep
i386pe
关键发现:
- 架构配置是正确的:
CONFIG_RTE_ARCH="x86_64" - 但链接器
ld -V显示第一个支持的模拟格式是elf_i386(32 位)! - 当
ld被调用时没有指定输出格式,会使用默认的第一个格式
步骤 9:检查 PMDINFO 链接命令
查看编译脚本中 PMDINFO 相关的链接命令:
# 查看 PMDINFO 链接命令
echo "=== PMDINFO 链接命令 ==="
grep -n "PMDINFO" mk/internal/rte.compile-pre.mk
# 查看 LDFLAGS 的实际值
echo "=== 检查编译时 LDFLAGS ==="
grep "LDFLAGS" mk/target/generic/rte.vars.mk | head -10
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# echo "=== PMDINFO 链接命令 ==="
=== PMDINFO 链接命令 ===
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# grep -n "PMDINFO" mk/internal/rte.compile-pre.mk
62:PMDINFO_GEN = $(RTE_SDK_BIN)/app/dpdk-pmdinfogen $@ $@.pmd.c
63:PMDINFO_CC = $(CC) $(CPPFLAGS) $(CFLAGS) $(EXTRA_CFLAGS) -c -o $@.pmd.o $@.pmd.c
64:PMDINFO_LD = $(CROSS)ld $(LDFLAGS) -r -o $@.o $@.pmd.o $@
65:PMDINFO_TO_O = if grep -q 'RTE_PMD_REGISTER_.*(.*)' $<; then \
66: echo "$(if $V,$(PMDINFO_GEN), PMDINFO $@.pmd.c)" && \
67: $(PMDINFO_GEN) && \
68: echo "$(if $V,$(PMDINFO_CC), CC $@.pmd.o)" && \
69: $(PMDINFO_CC) && \
70: echo "$(if $V,$(PMDINFO_LD), LD $@)" && \
71: $(PMDINFO_LD) && \
77: $(PMDINFO_TO_O) && \
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# echo "=== 检查编译时 LDFLAGS ==="
=== 检查编译时 LDFLAGS ===
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# grep "LDFLAGS" mk/target/generic/rte.vars.mk | head -10
15:# - define TOOLCHAIN_LDFLAGS variable (overridden by cmdline value)
27:# - define MACHINE_LDFLAGS variable (overridden by cmdline value)
31:# - can define CPU_LDFLAGS variable (overridden by cmdline value) that
51:# - define CPU_LDFLAGS variable (overridden by cmdline or previous
63:# - define EXECENV_LDFLAGS variable (overridden by cmdline)
69:# Don't set CFLAGS/LDFLAGS flags for kernel module, all flags are
81:# merge all LDFLAGS
82:LDFLAGS := $(CPU_LDFLAGS) $(EXECENV_LDFLAGS) $(TOOLCHAIN_LDFLAGS) $(MACHINE_LDFLAGS)
83:LDFLAGS += $(TARGET_LDFLAGS)
91:# LDFLAGS += -L$(RTE_OUTPUT)/lib
关键发现 :PMDINFO_LD 命令是:
$(CROSS)ld $(LDFLAGS) -r -o $@.o $@.pmd.o $@
LDFLAGS 的值来自 CPU_LDFLAGS 等变量的合并。需要检查 CPU_LDFLAGS 的设置。
步骤 10:检查 x86_64 架构的 CPU_LDFLAGS 设置
# 查看 x86_64 架构配置文件
cat mk/arch/x86_64/rte.vars.mk
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# cat mk/arch/x86_64/rte.vars.mk
# SPDX-License-Identifier: BSD-3-Clause
# Copyright(c) 2010-2014 Intel Corporation
ARCH ?= x86_64
ARCH_DIR := x86
CROSS ?=
CPU_CFLAGS ?= -m64
CPU_LDFLAGS ?=
CPU_ASFLAGS ?= -felf64
export ARCH CROSS CPU_CFLAGS CPU_LDFLAGS CPU_ASFLAGS
RTE_OBJCOPY_TARGET = elf64-x86-64
RTE_OBJCOPY_ARCH = i386:x86-64
export RTE_OBJCOPY_TARGET RTE_OBJCOPY_ARCH
找到根因了!
CPU_LDFLAGS ?= 后面是空的!这意味着:
- 当
ld被调用时,没有-m elf_x86_64参数 - 链接器使用默认的第一个模拟格式
elf_i386(32 位) - 导致输出格式是 32 位,但输入是 64 位,产生架构不匹配错误
步骤 11:修复 CPU_LDFLAGS
修改 mk/arch/x86_64/rte.vars.mk,添加链接器格式参数:
# 备份原文件
cp mk/arch/x86_64/rte.vars.mk mk/arch/x86_64/rte.vars.mk.bak
# 修改 CPU_LDFLAGS,添加 -m elf_x86_64
sed -i 's/CPU_LDFLAGS ?=/CPU_LDFLAGS ?= -m elf_x86_64/' mk/arch/x86_64/rte.vars.mk
# 验证修改
echo "=== 验证修改 ==="
grep "CPU_LDFLAGS" mk/arch/x86_64/rte.vars.mk
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# cp mk/arch/x86_64/rte.vars.mk mk/arch/x86_64/rte.vars.mk.bak
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sed -i 's/CPU_LDFLAGS ?=/CPU_LDFLAGS ?= -m elf_x86_64/' mk/arch/x86_64/rte.vars.mk
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# echo "=== 验证修改 ==="
=== 验证修改 ===
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# grep "CPU_LDFLAGS" mk/arch/x86_64/rte.vars.mk
# - define CPU_LDFLAGS variable (overridden by cmdline or previous
CPU_LDFLAGS ?= -m elf_x86_64
export ARCH CROSS CPU_CFLAGS CPU_LDFLAGS CPU_ASFLAGS
修改成功!现在清理并重新编译:
# 清理之前失败的编译产物
make clean 2>/dev/null
# 重新编译
make -j$(nproc)
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# make clean 2>/dev/null
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# make -j$(nproc)
...
CC test_ipsec.o
MKRES test_resource_c.res.o
LD test
/usr/bin/ld: cannot find -lnuma
collect2: error: ld returned 1 exit status
...
make: *** [all] Error 2
新进展 :架构不匹配问题解决了!现在是缺少 libnuma 库。
步骤 12:安装 numa 库
# 检查 numa 库是否安装
dpkg -l | grep numa
# 检查 numa 库位置
ldconfig -p | grep numa
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# dpkg -l | grep numa
ii libnuma-dev:i386 2.0.11-1ubuntu1.1 i386 Development files for libnuma
ii libnuma1:i386 2.0.11-1ubuntu1.1 i386 Libraries for controlling NUMA policy
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# ldconfig -p | grep numa
libnuma.so.1 (libc6) => /usr/lib/i386-linux-gnu/libnuma.so.1
libnuma.so (libc6) => /usr/lib/i386-linux-gnu/libnuma.so
问题 :系统安装的是 32 位 (i386) 的 numa 库,但需要 64 位 (amd64) 的版本。
尝试安装 64 位版本:
# 安装 64 位 numa 开发库
sudo apt-get install -y libnuma-dev:amd64 libnuma1:amd64
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sudo apt-get install -y libnuma-dev:amd64 libnuma1:amd64
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Unable to locate package libnuma-dev:amd64
问题:系统没有启用 amd64 架构支持。
步骤 13:启用 amd64 架构并安装 64 位 numa 库
# 启用 amd64 架构
sudo dpkg --add-architecture amd64
sudo apt-get update
# 安装 64 位 numa 库
sudo apt-get install -y libnuma-dev:amd64 libnuma1:amd64
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sudo dpkg --add-architecture amd64
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sudo apt-get update
Hit:1 http://us.archive.ubuntu.com/ubuntu xenial InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu xenial-backports InRelease
Hit:4 http://security.ubuntu.com/ubuntu xenial-security InRelease
Get:5 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages [1,201 kB]
...
Fetched 13.0 MB in 11s (1,096 kB/s)
Reading package lists... Done
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# sudo apt-get install -y libnuma-dev:amd64 libnuma1:amd64
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
gcc-6-base:amd64 libc6:amd64 libc6-dev:amd64 libgcc1:amd64 linux-libc-dev:amd64
...
Setting up libnuma-dev:amd64 (2.0.11-1ubuntu1.1) ...
Setting up libgcc1:amd64 (1:6.0.1-0ubuntu1) ...
Processing triggers for libc-bin (2.23-0ubuntu11.3) ...
验证 64 位库安装成功:
ldconfig -p | grep numa
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# ldconfig -p | grep numa
libnuma.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libnuma.so.1
libnuma.so.1 (libc6) => /usr/lib/i386-linux-gnu/libnuma.so.1
libnuma.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libnuma.so
libnuma.so (libc6) => /usr/lib/i386-linux-gnu/libnuma.so
现在有 64 位和 32 位两个版本了。
步骤 14:继续编译
make -j$(nproc)
root@ubuntu:/home/jerry/Desktop/dpdk-stable-19.08.2# make -j$(nproc)
...
CC comp_perf_options_parse.o
CC comp_perf_test_common.o
CC comp_perf_test_verify.o
CC comp_perf_test_benchmark.o
CC main.o
CC main.o
CC evt_main.o
CC evt_options.o
CC cperf_ops.o
CC cperf_options_parsing.o
CC cperf_test_vectors.o
CC evt_test.o
CC cperf_test_throughput.o
LD dpdk-test-compress-perf
INSTALL-APP test
INSTALL-MAP test.map
...
LD dpdk-test-crypto-perf
LD dpdk-test-eventdev
INSTALL-APP dpdk-test-crypto-perf
INSTALL-MAP dpdk-test-crypto-perf.map
INSTALL-APP dpdk-test-eventdev
INSTALL-MAP dpdk-test-eventdev.map
Build complete [x86_64-native-linuxapp-gcc]
编译成功!
三、问题总结
3.1 问题根因
本次编译失败涉及 三个层面的问题:
| 层面 | 问题 | 影响 |
|---|---|---|
| 配置层 | DPDK 19.08 默认配置包含 ARM 平台驱动 | 尝试编译不兼容的 ARM 驱动 |
| 工具链层 | CPU_LDFLAGS 为空,链接器使用 32 位默认格式 |
架构不匹配错误 |
| 依赖层 | 系统仅安装 32 位 numa 库 | 链接时找不到 64 位库 |
3.2 解决方案总结
# ===== 完整修复脚本 =====
cd /home/jerry/Desktop/dpdk-stable-19.08.2
# 1. 清理所有编译产物
rm -rf build x86_64-native-linuxapp-gcc
# 2. 禁用 ARM 平台驱动
sed -i 's/\(CONFIG_RTE_LIBRTE.*DPAA.*\)=y/\1=n/g' config/common_base
sed -i 's/\(CONFIG_RTE_LIBRTE.*FSLMC.*\)=y/\1=n/g' config/common_base
sed -i 's/\(CONFIG_RTE_LIBRTE.*OCTEONTX.*\)=y/\1=n/g' config/common_base
sed -i 's/\(CONFIG_RTE_LIBRTE.*OTX2.*\)=y/\1=n/g' config/common_base
sed -i 's/\(CONFIG_RTE_LIBRTE.*THUNDERX.*\)=y/\1=n/g' config/common_base
# 3. 修复链接器架构问题
sed -i 's/CPU_LDFLAGS ?=$/CPU_LDFLAGS ?= -m elf_x86_64/' mk/arch/x86_64/rte.vars.mk
# 4. 安装 64 位 numa 库
sudo dpkg --add-architecture amd64
sudo apt-get update
sudo apt-get install -y libnuma-dev:amd64 libnuma1:amd64
# 5. 生成配置
make config T=x86_64-native-linuxapp-gcc
# 6. 确保 build/.config 中 ARM 驱动被禁用
sed -i 's/\(CONFIG_RTE_LIBRTE.*DPAA.*\)=y/\1=n/g' build/.config
sed -i 's/\(CONFIG_RTE_LIBRTE.*FSLMC.*\)=y/\1=n/g' build/.config
sed -i 's/\(CONFIG_RTE_LIBRTE.*OCTEONTX.*\)=y/\1=n/g' build/.config
sed -i 's/\(CONFIG_RTE_LIBRTE.*OTX2.*\)=y/\1=n/g' build/.config
# 7. 编译
make -j$(nproc)
3.3 为什么博客里没有这个问题?
可能的原因:
- DPDK 版本差异:不同版本的默认配置不同
- 系统环境差异:其他环境可能已安装正确依赖
- 编译方式差异:可能使用了 meson 而非 make
- Ubuntu 版本差异:不同 Ubuntu 版本的 binutils 默认行为可能不同
四、技术细节说明
4.1 PMDINFO 链接过程
DPDK 编译过程中的 PMDINFO 步骤:
源文件 (.c)
↓ gcc -c
目标文件
↓ dpdk-pmdinfogen
PMD 信息文件
↓ gcc -c
PMD 目标文件
↓ ld -r ← 这里需要指定正确的输出格式
合并目标文件
ld -r 是可重定位链接,不会生成最终可执行文件,只是合并多个目标文件。
4.2 链接器模拟格式
ld -V 输出的模拟格式列表:
| 格式 | 说明 |
|---|---|
| elf_i386 | 32 位 x86 ELF |
| elf_x86_64 | 64 位 x86_64 ELF |
| elf32_x86_64 | 32 位兼容的 x86_64 |
当不指定 -m 参数时,链接器使用列表中的第一个格式(elf_i386),这就是问题所在。
4.3 DPDK 配置优先级
config/defconfig_x86_64-native-linuxapp-gcc (最高优先级)
+
config/common_base
+
config/common_linux
↓
build/.config (实际使用的配置)
某些配置在 defconfig 文件中被显式设置,会覆盖 common_base 的值。