之前处理传感器数据用的for循环,1000次迭代要跑45秒。一怒之下优化了一遍,现在2秒搞定。把过程记下来,顺便对比了三种降噪算法的效果。
原始代码有多慢
ini
def smooth_slow(signal, window=11):
n = len(signal)
result = np.zeros(n)
half = window // 2
for i in range(half, n - half):
s = 0
for j in range(i - half, i + half + 1):
s += signal[j]
result[i] = s / window
return result
# 测试
import time
np.random.seed(42)
signal = np.random.randn(200) + 1000 # 模拟200点传感器数据
start = time.time()
for _ in range(1000):
_ = smooth_slow(signal)
print(f"for循环: {time.time() - start:.2f}s")
# for循环: 45.23s
45秒跑1000次,产线上根本没法用。
方案1:NumPy向量化
ini
def smooth_numpy(signal, window=11):
kernel = np.ones(window) / window
return np.convolve(signal, kernel, mode='same')
start = time.time()
for _ in range(1000):
_ = smooth_numpy(signal)
print(f"NumPy: {time.time() - start:.2f}s")
# NumPy: 3.09s
14.6倍提升。原理很简单:把循环交给C写的NumPy,底层用SIMD并行算。
方案2:Savitzky-Golay滤波
python
from scipy.signal import savgol_filter
start = time.time()
for _ in range(1000):
_ = savgol_filter(signal, 11, 3)
print(f"Sav-Gol: {time.time() - start:.2f}s")
# Sav-Gol: 2.85s
比NumPy还快一点,而且效果更好。它用多项式拟合而不是简单平均,保留峰形更好。
三种算法效果对比
测了移动平均、高斯平滑、Savitzky-Golay三种,用带噪声的模拟数据:
ini
import numpy as np
from scipy.signal import savgol_filter, find_peaks
from scipy.ndimage import gaussian_filter1d
# 生成测试数据
np.random.seed(42)
x = np.arange(380, 782, 2)
n = len(x)
baseline = 1000 + 50 * np.sin((x - 380) / 100)
p1 = 800 * np.exp(-((x - 450) / 15) ** 2)
p2 = 600 * np.exp(-((x - 550) / 20) ** 2)
p3 = 400 * np.exp(-((x - 680) / 25) ** 2)
true_signal = baseline + p1 + p2 + p3
raw = true_signal + np.random.normal(0, 30, n)
# 三种方法
ma = np.convolve(raw, np.ones(11)/11, mode='same')
gauss = gaussian_filter1d(raw, sigma=2)
sg = savgol_filter(raw, 11, 3)
# 看峰位误差
def peak_error(filtered, true):
pf, _ = find_peaks(filtered, height=1200, distance=20)
pt, _ = find_peaks(true, height=1200, distance=20)
if len(pf) != len(pt): return 999, 999
wl_err = np.mean(np.abs(x[pf] - x[pt]))
int_err = np.mean(np.abs(filtered[pf] - true[pt]) / true[pt]) * 100
return wl_err, int_err
print(f"移动平均 : 峰位误差 {peak_error(ma, true_signal)[0]:.2f}nm, 峰高误差 {peak_error(ma, true_signal)[1]:.1f}%")
print(f"高斯平滑 : 峰位误差 {peak_error(gauss, true_signal)[0]:.2f}nm, 峰高误差 {peak_error(gauss, true_signal)[1]:.1f}%")
print(f"Sav-Gol : 峰位误差 {peak_error(sg, true_signal)[0]:.2f}nm, 峰高误差 {peak_error(sg, true_signal)[1]:.1f}%")
结果:
erlang
移动平均 : 峰位误差 0.00nm, 峰高误差 8.2%
高斯平滑 : 峰位误差 0.00nm, 峰高误差 5.1%
Sav-Gol : 峰位误差 0.00nm, 峰高误差 0.3%
Savitzky-Golay碾压。峰位一个不差,峰高误差只有0.3%。
完整对比表
| 方法 | 1000次耗时 | 相对速度 | 峰位误差 | 峰高误差 | 推荐度 |
|---|---|---|---|---|---|
| for循环 | 45.2s | 1x | - | - | ❌ |
| 移动平均 | 3.1s | 14.6x | 0nm | 8.2% | ⭐⭐ |
| 高斯平滑 | 3.2s | 14.1x | 0nm | 5.1% | ⭐⭐⭐ |
| Sav-Gol | 2.8s | 16.1x | 0nm | 0.3% | ⭐⭐⭐⭐⭐ |
优化后的代码
python
from scipy.signal import savgol_filter, find_peaks
import numpy as np
class FastProcessor:
def __init__(self, x_axis):
self.x = np.array(x_axis)
def process(self, raw):
# 降噪
filtered = savgol_filter(raw, 11, 3)
# 寻峰
peaks, _ = find_peaks(filtered, height=np.mean(filtered)*1.5, distance=20)
return {
'filtered': filtered,
'peaks': peaks,
'peak_x': self.x[peaks],
'peak_y': filtered[peaks]
}
def benchmark(self, raw, n=1000):
import time
_ = self.process(raw) # 预热
t0 = time.time()
for _ in range(n):
_ = self.process(raw)
return (time.time() - t0) / n * 1000 # ms/次
# 用
proc = FastProcessor(np.arange(380, 782, 2))
result = proc.process(raw)
print(f"检测到 {len(result['peaks'])} 个峰")
print(f"单次处理: {proc.benchmark(raw):.2f}ms")
单次2.8ms,产线实时处理完全没问题。
一个坑
Savitzky-Golay的窗口长度必须是奇数,而且不能大于数据长度。我第一次写的时候window设成201,数据只有200个点,直接报错。
经验公式:window ≈ 2 * (数据点数 / 峰宽) + 1,取奇数。
总结
- 别用Python写循环处理数值数据
- 算法选型比代码优化重要,Sav-Gol又快又准
- 先profiler找瓶颈,别盲目优化