GPT3.5\GPT4系列计算完整prompt token数的官方方法

前言:

ChatGPT如何计算token数?https://wtl4it.blog.csdn.net/article/details/135116493?spm=1001.2014.3001.5502https://wtl4it.blog.csdn.net/article/details/135116493?spm=1001.2014.3001.5502

GPT3.5\GPT4系列计算完整prompt token数的官方方法:

How to count tokens with tiktoken | OpenAI CookbookOpen-source examples and guides for building with the OpenAI API. Browse a collection of snippets, advanced techniques and walkthroughs. Share your own examples and guides.https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktokenhttps://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynbhttps://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb

计算的代码如下:
python 复制代码
def num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613"):
    """Return the number of tokens used by a list of messages."""
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        print("Warning: model not found. Using cl100k_base encoding.")
        encoding = tiktoken.get_encoding("cl100k_base")
    if model in {
        "gpt-3.5-turbo-0613",
        "gpt-3.5-turbo-16k-0613",
        "gpt-4-0314",
        "gpt-4-32k-0314",
        "gpt-4-0613",
        "gpt-4-32k-0613",
        }:
        tokens_per_message = 3
        tokens_per_name = 1
    elif model == "gpt-3.5-turbo-0301":
        tokens_per_message = 4  # every message follows <|start|>{role/name}\n{content}<|end|>\n
        tokens_per_name = -1  # if there's a name, the role is omitted
    elif "gpt-3.5-turbo" in model:
        print("Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.")
        return num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613")
    elif "gpt-4" in model:
        print("Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.")
        return num_tokens_from_messages(messages, model="gpt-4-0613")
    else:
        raise NotImplementedError(
            f"""num_tokens_from_messages() is not implemented for model {model}. See https://github.com/openai/openai-python/blob/main/chatml.md for information on how messages are converted to tokens."""
        )
    num_tokens = 0
    for message in messages:
        num_tokens += tokens_per_message
        for key, value in message.items():
            num_tokens += len(encoding.encode(value))
            if key == "name":
                num_tokens += tokens_per_name
    num_tokens += 3  # every reply is primed with <|start|>assistant<|message|>
    return num_tokens
相关推荐
聚客AI24 分钟前
🔥 大模型开发进阶:基于LangChain的异步流式响应与性能优化
人工智能·langchain·agent
cooldream200937 分钟前
Python 包管理新时代:深入了解 `uv` 的使用与实践
python·uv·包管理器
之歆37 分钟前
Python-魔术方法-创建、初始化与销毁-hash-bool-可视化-运算符重载-容器和大小-可调用对象-上下文管理-反射-描述器-二分-学习笔记
笔记·python·学习
CareyWYR44 分钟前
每周AI论文速递(250707-250711)
人工智能
AI街潜水的八角1 小时前
深度学习图像分类数据集—五种电器识别分类
人工智能·深度学习·分类
众链网络1 小时前
AI进化论08:机器学习的崛起——数据和算法的“二人转”,AI“闷声发大财”
人工智能·算法·机器学习
生命是有光的1 小时前
【机器学习】机器学习基础
人工智能·机器学习
平和男人杨争争1 小时前
机器学习13——支持向量机下
人工智能·机器学习·支持向量机
胖达不服输1 小时前
「日拱一码」025 机器学习——评价指标
人工智能·python·机器学习·评价指标
默辨1 小时前
机器学习(ML)、深度学习(DL)、强化学习(RL)关系和区别
人工智能·深度学习·机器学习