3D重建评估指标对比表
每个指标的具体代码位于文章末尾
| 指标 | 计算方法 | 数值范围 | 评估重点 | 优缺点 | 适用场景 |
|---|---|---|---|---|---|
| Chamfer Distance (C1) | 从预测网格到真实网格的平均距离 | [0, +∞) | 几何形状准确性 | 优点 ：直观、计算高效 缺点：对噪声敏感 | 整体形状评估 |
| Chamfer Distance (C2) | 从真实网格到预测网格的平均距离 | [0, +∞) | 几何形状完整性 | 优点 ：检测缺失部分 缺点：可能被异常值影响 | 完整性评估 |
| Normal Consistency | 对应点法向量点积的平均值 | [0, 1] | 表面细节质量 | 优点 ：反映表面光滑度 缺点：不关注几何形状 | 表面质量评估 |
| F-Score | 基于距离阈值的精确率/召回率调和平均 | [0, 100] | 高精度区域占比 | 优点 ：关注高精度区域 缺点：依赖阈值选择 | 精度评估 |
| Bounding Box IoU | 边界框交集体积/并集体积 | [0, 1] | 整体形状重叠度 | 优点 ：计算简单快速 缺点：忽略细节差异 | 粗略形状评估 |
详细指标说明
1. Chamfer Distance (C1 & C2)
python
# C1: 预测→真实
c1 = np.mean(dist_pred_to_gt) * 1000
# C2: 真实→预测
c2 = np.mean(dist_gt_to_pred) * 1000特点对比：
- C1：检测预测网格中多余的部分
- C2：检测预测网格中缺失的部分
- 理想情况：C1 ≈ C2，且都接近0
2. Normal Consistency
python
normal_consistency = np.mean(normal_pred_to_gt) + np.mean(normal_gt_to_pred)评估维度：
- 表面光滑度：法向量变化是否平滑
- 细节保持：能否保持原始表面的细节特征
- 方向一致性：表面朝向是否一致
3. F-Score
python
tau = 1e-2 # 1cm阈值
prec_tau = (dist_pred_to_gt <= tau).mean() * 100
recall_tau = (dist_gt_to_pred <= tau).mean() * 100
fscore = (2 * prec_tau * recall_tau) / (prec_tau + recall_tau)评估重点：
- 高精度区域：关注距离小于1cm的区域
- 平衡性：同时考虑精确率和召回率
- 实用性：反映实际应用中的可用性
4. Bounding Box IoU
python
iou = inter_vol / (vol1 + vol2 - inter_vol)评估范围：
- 整体形状：不考虑内部细节
- 空间位置：反映整体定位准确性
- 尺度一致性：检测尺寸是否合理
指标组合使用建议
| 评估目标 | 推荐指标组合 | 原因 |
|---|---|---|
| 整体质量 | C1 + C2 + F-Score | 全面评估几何准确性 |
| 表面质量 | Normal Consistency | 专注表面细节 |
| 快速筛选 | Bounding Box IoU | 计算快速，适合大规模筛选 |
| 高精度应用 | F-Score | 关注高精度区域 |
| 研究对比 | 全部指标 | 提供全面的评估维度 |
实际应用中的选择
- 服装重建：重点关注C1、C2和Normal Consistency
- 快速原型：使用Bounding Box IoU进行初步筛选
- 生产应用：重点关注F-Score确保高精度
- 学术研究：使用全部指标进行综合评估
这些指标各有侧重，组合使用能够全面评估3D重建的质量。
bash
import os
import torch
import scipy as sp
import numpy as np
import argparse
import trimesh
from tqdm import tqdm
def compute_iou_bbox(mesh, gt_mesh):
mesh_bounds = mesh.bounds
gt_mesh_bounds = gt_mesh.bounds
xx1 = np.max([mesh_bounds[0, 0], gt_mesh_bounds[0, 0]])
yy1 = np.max([mesh_bounds[0, 1], gt_mesh_bounds[0, 1]])
zz1 = np.max([mesh_bounds[0, 2], gt_mesh_bounds[0, 2]])
xx2 = np.min([mesh_bounds[1, 0], gt_mesh_bounds[1, 0]])
yy2 = np.min([mesh_bounds[1, 1], gt_mesh_bounds[1, 1]])
zz2 = np.min([mesh_bounds[1, 2], gt_mesh_bounds[1, 2]])
vol1 = (mesh_bounds[1, 0] - mesh_bounds[0, 0]) * (
mesh_bounds[1, 1] - mesh_bounds[0, 1]) * (mesh_bounds[1, 2] -
mesh_bounds[0, 2])
vol2 = (gt_mesh_bounds[1, 0] - gt_mesh_bounds[0, 0]) * (
gt_mesh_bounds[1, 1] - gt_mesh_bounds[0, 1]) * (gt_mesh_bounds[1, 2] -
gt_mesh_bounds[0, 2])
inter_vol = np.max([0, xx2 - xx1]) * np.max([0, yy2 - yy1]) * np.max(
[0, zz2 - zz1])
iou = inter_vol / (vol1 + vol2 - inter_vol + 1e-11)
return iou
def calculate_iou(gt, prediction):
intersection = torch.logical_and(gt, prediction)
union = torch.logical_or(gt, prediction)
return torch.sum(intersection) / torch.sum(union)
def compute_surface_metrics(mesh_pred, mesh_gt):
"""Compute surface metrics (chamfer distance and f-score) for one example.
Args:
mesh: trimesh.Trimesh, the mesh to evaluate.
Returns:
chamfer: float, chamfer distance.
fscore: float, f-score.
"""
# Chamfer
eval_points = 100000
point_gt, idx_gt = mesh_gt.sample(eval_points, return_index=True)
normal_gt = mesh_gt.face_normals[idx_gt]
point_gt = point_gt.astype(np.float32)
point_pred, idx_pred = mesh_pred.sample(eval_points, return_index=True)
normal_pred = mesh_pred.face_normals[idx_pred]
point_pred = point_pred.astype(np.float32)
dist_pred_to_gt, normal_pred_to_gt = distance_field_helper(point_pred, point_gt, normal_pred, normal_gt)
dist_gt_to_pred, normal_gt_to_pred = distance_field_helper(point_gt, point_pred, normal_gt, normal_pred)
# TODO: subdivide by 2 following OccNet
# https://github.com/autonomousvision/occupancy_networks/blob/406f79468fb8b57b3e76816aaa73b1915c53ad22/im2mesh/eval.py#L136
chamfer_l1 = np.mean(dist_pred_to_gt) + np.mean(dist_gt_to_pred)
c1 = np.mean(dist_pred_to_gt)
c2 = np.mean(dist_gt_to_pred)
normal_consistency = np.mean(normal_pred_to_gt) + np.mean(normal_gt_to_pred)
# Fscore
tau = 1e-2
eps = 1e-6
#dist_pred_to_gt = (dist_pred_to_gt**2)
#dist_gt_to_pred = (dist_gt_to_pred**2)
prec_tau = (dist_pred_to_gt <= tau).astype(np.float32).mean() * 100.
recall_tau = (dist_gt_to_pred <= tau).astype(np.float32).mean() * 100.
fscore = (2 * prec_tau * recall_tau) / max(prec_tau + recall_tau, eps)
# Following the tradition to scale chamfer distance up by 10.
return c1 * 1000., c2 * 1000., normal_consistency / 2., fscore
def distance_field_helper(source, target, normals_src=None, normals_tgt=None):
target_kdtree = sp.spatial.cKDTree(target)
distances, idx = target_kdtree.query(source, n_jobs=-1)
if normals_src is not None and normals_tgt is not None:
normals_src = \
normals_src / np.linalg.norm(normals_src, axis=-1, keepdims=True)
normals_tgt = \
normals_tgt / np.linalg.norm(normals_tgt, axis=-1, keepdims=True)
normals_dot_product = (normals_tgt[idx] * normals_src).sum(axis=-1)
# Handle normals that point into wrong direction gracefully
# (mostly due to mehtod not caring about this in generation)
normals_dot_product = np.abs(normals_dot_product)
else:
normals_dot_product = np.array(
[np.nan] * source.shape[0], dtype=np.float32)
return distances, normals_dot_product
def main(args):
input_subfolder = [x for x in sorted(os.listdir(args.input_path)) if os.path.isdir(os.path.join(args.input_path, x))]
gt_subfolder = [x for x in sorted(os.listdir(args.gt_path)) if os.path.isdir(os.path.join(args.gt_path, x))]
eval_name = args.input_path.split('/')[-1]
mean_c1_list = []
mean_c2_list = []
mean_fscore_list = []
mean_normal_consistency_list = []
iou_list = []
for pred, gt in tqdm(zip(input_subfolder, gt_subfolder)):
pred_path = [x for x in sorted(os.listdir(os.path.join(args.input_path, pred))) if
x.endswith('shoes.obj') and not x.startswith('init')and not x.startswith('.')]
if len(pred_path) == 0:
continue
mesh_pred = trimesh.load(os.path.join(args.input_path, pred, pred_path[0]))
gt_path = [x for x in sorted(os.listdir(os.path.join(args.gt_path, gt, 'clothing'))) if x.endswith('shoe.obj')and not x.startswith('.')][0]
mesh_gt = trimesh.load(os.path.join(args.gt_path, gt, 'clothing', gt_path))
pred_2_scan, scan_2_pred, normal_consistency, fscore = compute_surface_metrics(mesh_pred, mesh_gt)
iou = compute_iou_bbox(mesh_pred, mesh_gt)
#print('Chamfer: {:.3f}, {:.3f}, Normal Consistency: {:.3f}, Fscore: {:.3f}, IOU: {:.3f}'.format(pred_2_scan, scan_2_pred, normal_consistency, fscore, iou))
#print((pred_2_scan + scan_2_pred) / 2.0)
iou_list.append(iou)
mean_c1_list.append(pred_2_scan)
mean_c2_list.append(scan_2_pred)
mean_fscore_list.append(fscore)
mean_normal_consistency_list.append(normal_consistency)
mean_c1 = np.mean(mean_c1_list)
mean_c2 = np.mean(mean_c2_list)
mean_fscore = np.mean(mean_fscore_list)
mean_normal_consistency = np.mean(mean_normal_consistency_list)
mean_iou = np.mean(iou_list)
std_c1 = np.std(mean_c1_list)
std_c2 = np.std(mean_c2_list)
std_fscore = np.std(mean_fscore_list)
std_normal_consistency = np.std(mean_normal_consistency_list)
std_iou = np.std(iou_list)
print('Mean Chamfer: {:.3f} ({:.3f}), {:.3f} ({:.3f}), Normal Consistency: {:.3f} ({:.3f}), Fscore: {:.3f} ({:.3f})'
.format(mean_c1, std_c1, mean_c2, std_c2, mean_normal_consistency, std_normal_consistency, mean_fscore, std_fscore))
print('{:.3f} ({:.3f}),{:.3f} ({:.3f}),{:.3f} ({:.3f}),{:.3f} ({:.3f}),{:.3f} ({:.3f})'
.format(mean_c1, std_c1, mean_c2, std_c2, mean_normal_consistency, std_normal_consistency, mean_fscore, std_fscore, mean_iou, std_iou))
print('{:.6f}, {:.6f}, {:.6f}, {:.6f}, {:.6f}'.format(mean_c1, mean_c2, mean_normal_consistency, mean_fscore, mean_iou))
output_txt = eval_name + '.txt'
out = np.stack([mean_c1_list, mean_c2_list, mean_normal_consistency_list, mean_fscore_list], axis=1)
np.savetxt(output_txt, out, fmt='%.6f', delimiter=' ')
if __name__ == '__main__':
parser = argparse.ArgumentParser()
#parser.add_argument('-o', '--output_dir', required=True, help='Where to store the processed images and other data.')
parser.add_argument('-i', '--input_path', required=True ,type=str)
parser.add_argument('-g', '--gt_path', required=True ,type=str)
main(parser.parse_args())