1 算法原理
CBF是*Cross Bilateral Filter(交叉双边滤波)*的缩写，论文《IMAGE FUSION BASED ON PIXEL SIGNIFICANCE USING CROSS BILATERAL FILTER》。
论文中，作者使用交叉双边滤波算法对原始图像 A A A， B B B 进行处理得到细节（detail）层，通过细节层得到权重系数，最后经过权重系数对原始图像 A A A， B B B进行融合，得到最终的输出图像 F F F 。
2 算法实现
2.1 交叉双边滤波算法
双边波算法是局部的，非线性、非迭代的方法，使用高斯滤波器和颜色滤波器来对图像进行处理， 兼具平滑和保持边界的功能。这在我之前的专栏中有提到过，我在这就不在细说双边滤波，接下来说一下文中提到的交叉双边滤波。
交叉双边滤波算法考虑了 A A A图像的灰度层的相似性和几何层的闭合性来对 B B B 图像进行过滤，或者使用 B B B图像来对 A A A 图像进行过滤，输入为 A A A, B B B 两幅输入图像，假设使用 A A A 图像对 B B B图像进行过滤，论文中 σ s = 1.8 , σ r = 25 \sigma_s=1.8,\sigma_r=25 σs=1.8,σr=25 ,CBFlv滤波窗口大小为 11 ∗ 11 11*11 11∗11 那么：
B C B F ( p ) = 1 W ∑ q ∈ S ( e − ∣ p − q ∣ 2 2 σ s 2 ∗ e − ∣ A ( p ) − A ( q ) ∣ 2 2 σ r 2 ∗ B ( q ) ) B_{CBF}(p)=\frac{1}{W}\sum_{q\in S}(e^{-\frac{|p-q|^2}{2\sigma_s^2}}*e^{-\frac{|A(p)-A(q)|^2}{2\sigma_r^2}}*B(q)) BCBF(p)=W1q∈S∑(e−2σs2∣p−q∣2∗e−2σr2∣A(p)−A(q)∣2∗B(q))
W = ∑ q ∈ S e − ∣ p − q ∣ 2 2 σ s 2 ∗ e − ∣ A ( p ) − A ( q ) ∣ 2 2 σ r 2 W=\sum_{q\in S}e^{-\frac{|p-q|^2}{2\sigma_s^2}}*e^{-\frac{|A(p)-A(q)|^2}{2\sigma_r^2}} W=q∈S∑e−2σs2∣p−q∣2∗e−2σr2∣A(p)−A(q)∣2
同理，对于 A C B F ( p ) A_{CBF}(p) ACBF(p) 使用的是 B B B图像来对 A A A 图像进行过滤，由 $B_{CBF}、 A_{CBF} $以及原始图像 A A A 、 B B B ，在原始图像的基础上减去交叉双边滤波得到的图像就是细节层图像，
A D = A − A C B F , B D = B − B C B F A_D=A-A_{CBF},B_D=B-B_{CBF} AD=A−ACBF,BD=B−BCBF
文中作者给出交叉双边滤波有用的解释是：在多聚焦图像中， A A A 图像未聚焦的区域在图像 B B B 中是聚焦的区域，对图像 B B B 进行交叉双边滤波会使得对图像 B B B 的聚焦区域平滑的比图像 B B B 的非聚焦区域更强，原因是：图像 A A A 中的非聚焦区域由相近的颜色会使得过滤器更像是高斯过滤器。
最上面两幅为原始图像，中间两幅为CBF滤波后的图像，最下面两幅就是由原始图像-CBF图像得到的图像
2.2基于像素的融合策略
接下来的一步是使用细节图像计算融合权重。
选定窗口大小为 w ∗ w w∗w w∗w（论文中使用 5 ∗ 5 5*5 5∗5），来方便计算权重，这个窗口划过的区域看做矩阵 $C_h^{i,j}C_h^{i,j} $的无偏估计：
c o v a r i a n c e ( X ) = E [ ( X − E [ X ] ) ( X − E ( X ) ) T ] covariance(X)=E[(X-E[X])(X-E(X))^T] covariance(X)=E[(X−E[X])(X−E(X))T]
C h i , j = ∑ k = 1 w ( x k − x ˉ ) ( x k − x ˉ ) T w − 1 C_h^{i,j}=\frac{\sum_{k=1}^w(x_k-\bar{x})(x_k-\bar{x})^T}{w-1} Chi,j=w−1∑k=1w(xk−xˉ)(xk−xˉ)T
得到 C h i , j C_h^{i,j} Chi,j 大小为 w ∗ w w*w w∗w，然后计算矩阵 C h i , j C_h^{i,j} Chi,j 的特征值，并且将这些特征值相加就得到了水平方向区域细节强度，并记为 H d e t a i l S t r e n g t h HdetailStrength HdetailStrength ：
H d e t a i l S t r e n g t h ( i , j ) = ∑ k = 1 w e i g e n k HdetailStrength(i,j)=\sum_{k=1}^weigen_k HdetailStrength(i,j)=k=1∑weigenk
同理将 X X X的每一列看做观察量，将 X X X的行看做是变量，计算协方差矩阵 C v i , j C_v^{i,j} Cvi,j，并计算垂直方向区域细节强度 V d e t a i l S t r e n g t h VdetailStrength VdetailStrength :
V d e t a i l S t r e n g t h ( i , j ) = ∑ k = 1 w e i g e n k VdetailStrength(i,j)=\sum_{k=1}^weigen_k VdetailStrength(i,j)=∑k=1weigenk$
最终的权重为，水平和垂直方向的累加和：
w t ( i , j ) = H d e t a i l S t r e n g t ( i , j ) + V d e t a i l S t r e n g t ( i , j ) wt(i,j)=HdetailStrengt(i,j)+VdetailStrengt(i,j) wt(i,j)=HdetailStrengt(i,j)+VdetailStrengt(i,j)
得到 A A A 、 B B B 图像的权重系数后，就可以按照权重系数就行融合：
F ( i , j ) = A ( i , j ) w t a ( i , j ) + B ( i , j ) w t b ( i , j ) w t a ( i , j ) + w t b ( i , j ) F(i,j)=\frac{A(i,j)wt_a(i,j)+B(i,j)wt_b(i,j)}{wt_a(i,j)+wt_b(i,j)} F(i,j)=wta(i,j)+wtb(i,j)A(i,j)wta(i,j)+B(i,j)wtb(i,j)
3 opencv实现代码
python
import numpy as np
import cv2
import argparse
import math
cov_wsize = 5
sigmas = 1.8
sigmar = 25
ksize = 11
def gaussian_kernel_2d_opencv(kernel_size = 11,sigma = 1.8):
kx = cv2.getGaussianKernel(kernel_size,sigma)
ky = cv2.getGaussianKernel(kernel_size,sigma)
return np.multiply(kx,np.transpose(ky))
def bilateralFilterEx(img_r, img_v):
#edge solved
win_size = ksize//2
img_r_copy = None
img_v_copy = None
img_r_copy = cv2.copyTo(img_r, None)
img_v_copy = cv2.copyTo(img_v, None)
img_r_cbf = np.ones_like(img_r, dtype=np.float32)
img_v_cbf = np.ones_like(img_r, dtype=np.float32)
img_r_copy = np.pad(img_r_copy, (win_size, win_size), 'reflect')
img_v_copy = np.pad(img_v_copy, (win_size, win_size), 'reflect')
gk = gaussian_kernel_2d_opencv()
for i in range(win_size, win_size+img_r.shape[0]):
for j in range(win_size, win_size+img_r.shape[1]):
sumr1 = 0.
sumr2 = 0.
sumv1 = 0.
sumv2 = 0.
img_r_cdis = img_r_copy[i-win_size:i+win_size+1, j-win_size:j+win_size+1] *1.0- img_r_copy[i,j]*1.0
img_v_cdis = img_v_copy[i-win_size:i+win_size+1, j-win_size:j+win_size+1] *1.0- img_v_copy[i,j]*1.0
sumr1 = np.sum(np.exp(-img_v_cdis*img_v_cdis) *gk/ (2*sigmar*sigmar) )
sumv1 = np.sum(np.exp(-img_r_cdis*img_r_cdis) *gk/ (2*sigmar*sigmar) )
sumr2 = np.sum(np.exp(-img_v_cdis*img_v_cdis) *gk*img_r_copy[i-win_size:i+win_size+1, j-win_size:j+win_size+1] *1.0/ (2*sigmar*sigmar) )
sumv2 = np.sum(np.exp(-img_r_cdis*img_r_cdis) *gk*img_v_copy[i-win_size:i+win_size+1, j-win_size:j+win_size+1] *1.0/ (2*sigmar*sigmar) )
img_r_cbf[i-win_size,j-win_size] = sumr2 / sumr1
img_v_cbf[i-win_size,j-win_size] = sumv2 / sumv1
return (img_r*1. - img_r_cbf, img_v*1. - img_v_cbf)
def CBF_WEIGHTS(img_r_d, img_v_d):
win_size = cov_wsize // 2
img_r_weights = np.ones_like(img_r_d, dtype=np.float32)
img_v_weights= np.ones_like(img_v_d, dtype=np.float32)
img_r_d_pad = np.pad(img_r_d, (win_size, win_size), 'reflect')
img_v_d_pad = np.pad(img_v_d, (win_size, win_size), 'reflect')
for i in range(win_size, win_size+img_r_d.shape[0]):
for j in range(win_size, win_size+img_r_d.shape[1]):
npt_r = img_r_d_pad[i-win_size:i+win_size+1, j-win_size:j+win_size+1]
npt_v = img_v_d_pad[i-win_size:i+win_size+1, j-win_size:j+win_size+1]
npt_r_V = npt_r - np.mean(npt_r, axis=0)
npt_r_V = npt_r_V*npt_r_V.transpose()
npt_r_H = npt_r.transpose() - np.mean(npt_r, axis=1)
npt_r_H = npt_r_H*npt_r_H.transpose()
npt_v_V = npt_v - np.mean(npt_v, axis=0)
npt_v_V = npt_v_V*npt_v_V.transpose()
npt_v_H = npt_v.transpose() - np.mean(npt_v, axis=1)
npt_v_H = npt_v_H*npt_v_H.transpose()
img_r_weights[i-win_size,j-win_size] = np.trace(npt_r_H) + np.trace(npt_r_V)
img_v_weights[i-win_size,j-win_size] = np.trace(npt_v_H) + np.trace(npt_v_V)
return img_r_weights, img_v_weights
def CBF_GRAY(img_r, img_v):
img_r_d, img_v_d = bilateralFilterEx(img_r, img_v)
img_r_weights, img_v_weights = CBF_WEIGHTS(img_r_d, img_v_d)
img_fused =(img_r*1. * img_r_weights + img_v*1.*img_v_weights) /(img_r_weights+img_v_weights)
img_fused = cv2.convertScaleAbs(img_fused)
return img_fused
def CBF_RGB(img_r, img_v):
img_r_gray = cv2.cvtColor(img_r, cv2.COLOR_BGR2GRAY)
img_v_gray = cv2.cvtColor(img_v, cv2.COLOR_BGR2GRAY)
return CBF_GRAY(img_r_gray, img_v_gray)
def CBF(_rpath, _vpath):
img_r = cv2.imread(_rpath)
img_v = cv2.imread(_vpath)
if not isinstance(img_r, np.ndarray) :
print('img_r is null')
return
if not isinstance(img_v, np.ndarray) :
print('img_v is null')
return
if img_r.shape[0] != img_v.shape[0] or img_r.shape[1] != img_v.shape[1]:
print('size is not equal')
return
fused_img = None
if len(img_r.shape) < 3 or img_r.shape[2] ==1:
if len(img_v.shape) < 3 or img_v.shape[-1] ==1:
fused_img = CBF_GRAY(img_r, img_v)
else:
img_v_gray = cv2.cvtColor(img_v, cv2.COLOR_BGR2GRAY)
fused_img = CBF_GRAY(img_r, img_v)
else:
if len(img_v.shape) < 3 or img_v.shape[-1] ==1:
img_r_gray = cv2.cvtColor(img_r, cv2.COLOR_BGR2GRAY)
fused_img = CBF_GRAY(img_r_gray, img_v)
else:
fused_img = CBF_RGB(img_r, img_v)
cv2.imshow('fused image', fused_img)
cv2.imwrite("fused_image_2.jpg", fused_img)
cv2.waitKey(0)
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('-r', type=str, default='ir2.png' ,help='input IR image path', required=False)
parser.add_argument('-v', type=str, default= 'vr2.png',help='input Visible image path', required=False)
args = parser.parse_args()
CBF(args.r, args.v)
