调用python函数的不同方法效率对比测试

方法1:

用注册DuckDB自定义函数的方法, 我用的是1.3.2版的DuckDB, 在create_function时总是报错,

复制代码
  File "C:\d\pyduck.txt", line 32, in <module>
    duckdb.create_function("count_composition_ways_optimized", count_composition_ways_optimized, [VARCHAR, [VARCHAR]], BIGINT)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Unable to cast Python instance of type <class 'list'> to C++ type '?' (#define PYBIND11_DETAILED_ERROR_MESSAGES or compile in debug mode for details)

后来看到文档,
Type Annotation

When the function has type annotation it's often possible to leave out all of the optional parameters. Using DuckDBPyType we can implicitly convert many known types to DuckDBs type system.

不指定类型,反而好了。

复制代码
def count_composition_ways_optimized(a: str, b: set) -> int:
    """
    优化版本:限制子串最大长度,提高效率
    """
    n = len(a)
    if n == 0:
        return 1
    
    # 计算集合中字符串的最大长度
    max_len = max(len(word) for word in b) if b else 0
    
    dp = [0] * (n + 1)
    dp[0] = 1
    
    for i in range(1, n + 1):
        # 只检查可能的子串长度,避免不必要的检查
        for length in range(1, min(i, max_len) + 1):
            start = i - length
            if a[start:i] in b:
                dp[i] += dp[start]
            #print(i, length,start,dp)
        #print(i, dp)
    return dp[n]

a,b='bill', ['bi', 'l','ike']

print(count_composition_ways_optimized(a, b))

import duckdb
from duckdb.typing import VARCHAR, BIGINT
from duckdb import list_type, struct_type
duckdb.create_function("count_composition_ways_optimized", count_composition_ways_optimized)  #, [VARCHAR, [[VARCHAR]], BIGINT)
s="""
with recursive co as(
select trim(unnest(string_split(c, ',')))c from (from read_csv('2419-input.txt', header=0, delim='-')t(c) limit 1)), 
c2 as(select list(c) c from co), 
w as(select w from read_csv('2419-input.txt', header=0, delim='-', skip=2) t(w))
select sum(count_composition_ways_optimized(w, c)) sum_of_way from c2,w 
-- select w, count_composition_ways_optimized(w, c) sum_of_way from c2,w limit 3
"""
import time
t=time.time();print(duckdb.sql(s), time.time()-t)

上述程序的运行结果

复制代码
C:\d>python pyduck.txt
1
┌─────────────────┐
│   sum_of_way    │
│     int128      │
├─────────────────┤
│ 950763269786650 │
└─────────────────┘
 0.0038411617279052734

方法2:

用手工方法把数据拆成a、b两部分,a部分是待拆解字符串,用csv模块读取一行,b是子字符串列表。

复制代码
import sys
import time
import csv
with open('2419-input.txt', mode='r') as file:
    csvFile = csv.reader(file)
    for lines in csvFile:
        b=[i.strip() for i in lines]
        #print(lines)
        break

def count_composition_ways_optimized(a: str, b: set) -> int:
    """
    优化版本:限制子串最大长度,提高效率
    """
    n = len(a)
    if n == 0:
        return 1
    
    # 计算集合中字符串的最大长度
    max_len = max(len(word) for word in b) if b else 0
    
    dp = [0] * (n + 1)
    dp[0] = 1
    
    for i in range(1, n + 1):
        # 只检查可能的子串长度,避免不必要的检查
        for length in range(1, min(i, max_len) + 1):
            start = i - length
            if a[start:i] in b:
                dp[i] += dp[start]
    
    return dp[n]

def main():
    if len(sys.argv) != 2:
        print("用法: python script.py <文件名>")
        return
    t = time.time()
    filename = sys.argv[1]
    cnt=0
    try:
        with open(filename, 'r', encoding='utf-8') as file:
            while True:
                line = file.readline().strip()
                if line == '':
                    break
                
                cnt+=count_composition_ways_optimized(line, b)

    except FileNotFoundError:
        print(f"错误: 文件 '{filename}' 不存在")
    except Exception as e:
        print(f"错误: {e}")
    print(cnt)
    print(f"耗时: {round(time.time()-t, 4)} s")


if __name__ == "__main__":
    main()

执行结果是

复制代码
950763269786650
耗时: 0.3117 s

算得的结果相同,反而是DuckDB调用python函数更快,估计一与DuckDB并行计算有关, 二是Python确实慢。同样的程序用pypy执行,就是下面的结果

复制代码
950763269786650
鑰楁椂: 0.0885 s

补记:

我搞错了,方法1不该把计时输出放在同一个print里,这样就正常了

复制代码
t=time.time();print(duckdb.sql(s));print(time.time()-t)
0.6274137496948242
相关推荐
多加点辣也没关系几秒前
JavaScript|第40章:JavaScript 工程化实践
开发语言·javascript·ecmascript
2601_9679801120 分钟前
python pytest如何编写测试
python·测试用例·pytest·函数·assert
2601_967980111 小时前
Python中heapq模块的各种用法实例
数据结构·python·算法·应用实例·heapq模块
大胖爱学习_MJ1 小时前
超越向量检索:用 Spring Boot + Neo4j 手搓 GraphRAG 知识图谱记忆
java·后端·python
RobinDevNotes2 小时前
24GB Mac 做数字人:声音放本地,画面租云端
人工智能·python·macos·开发工具
龙腾AI白云2 小时前
数字孪生与计算机视觉:让机器看懂现场
大数据·数据库·机器学习·知识图谱·pygame
吃我一窝jio2 小时前
Java Bug复盘与性能优化实战
java·开发语言·笔记
千寻技术帮3 小时前
基于Spring Boot+Vue的农产品销售管理系统(源码+文档+数据库,附部署教程)
java·数据库·spring boot·后端·mysql·vue
学无止境小袁同学4 小时前
LlamaIndex Workflows 学习
python·llm·tensorflow·llamaindex·agent开发·多agent·workflows
Evand J4 小时前
【MATLAB例程】基于APF的多无人机动态路径规划与避障仿真。避碰及安全性能分析。附完整代码的下载链接
开发语言·matlab·无人机