提示工程（Prompt Engineering）对各位小伙伴们来说是再熟悉不过了，提示词工程技术是通过编写指令词，指导开发者们调用AI基础模型（FMs）获得期望的响应。但是经常写提示词的朋友们会知道，为了获取理想的输出，我们可能需要花费数月时间不断进行实验和调整才能得到最优的提示词，同时不同基础模型的提示词最佳实践也不尽相同，这意味着我们要设计兼容不同模型类别的提示词。此外提示词通常是与特定模型和特定任务场景所匹配的，当更换不同的基础模型时，适用于其他模型的提示词性能无法得到保证。提示工程调优的巨大工作量常常拖慢了我们开发基于不同模型的生成式AI应用。
在本系列之前的上篇中，小李哥通过一个分析客服通话记录的真实场景，手把手带大家通过提示词优化功能，无痛生成完美提示词，提升生成式AI应用的表现。在本篇中我们会继续介绍如何通过API调用的方式优化提示词，并验证优化后的提示词在提示词数据集上性能基准测试结果和模型回复效果提升。
如何通过API调用的方式优化提示词
除了通过亚马逊云科技控制台中使用提示词优化功能优化提示词外，我们还可以通过亚马逊云科技提供的官方Python SDK Boto3对提示词进行优化。
首先大家确认已安装了Boto3 库，并已配置好AWS秘钥凭证，在本地访问 Amazon Bedrock 服务。大家可以通过以下的示例使用Python脚本对提示词进行优化。
python
import boto3
import json
# 创建 Amazon Bedrock 客户端
client = boto3.client('bedrock', region_name='us-east-1')
# 定义要优化的提示词
original_prompt = "请总结以下文本："
# 定义目标模型 ID，例如 'anthropic.claude-3.5-sonnet'
target_model_id = 'anthropic.claude-3.5-sonnet'
# 调用优化提示词的 API
response = client.optimize_prompt(
input={
'textPrompt': {
'text': original_prompt
}
},
targetModelId=target_model_id
)
# 输出优化后的提示词
optimized_prompt = response['optimizedPrompt']['textPrompt']['text']
print("优化后的提示词：", optimized_prompt)在上述示例脚本中，大家需要根据大家所需的配置的替换想要使用的模型"target_model_id"、所在的亚马逊云科技区域"region_name"和需要优化的提示词"original_prompt"。
优化后的提示词基准测试效果
基准测试数据集选择
接下来小李哥在多个开源的提示词基准测试数据集上运行了经过优化的提示词，与未优化的提示词的模型回复效果进行了比对，验证优化后的提示词对我们的生成式AI应用带来的性能提升。我们使用了常用的生成式AI使用场景下的基准测试数据集作为我们的提示词基准测试数据集。分别为如下三个：
文本摘要场景下，我们使用了XSUM
基于RAG（检索增强生成）的对话续写场景下，我们使用了DSTC
常规的函数调用场景下，我们使用了GLAIVE
基准测试评估指标
为了衡量优化后的提示词调用模型的回复效果上的性能提升，我们可以使用以下评估指标：
文本摘要任务使用ROUGE-2 F1
RAG对话续写任务使用HELM-F1
函数调用任务使用HELM-F1和JSON 匹配
基准测试提示词优化后的性能提升结果
在我们对优化后提示词的基准测试中，得到了如下性能提升数据：
文本摘要任务性能提升18%
对话续写任务性能提升8%
函数调用基准测试性能提升22%
基准测试的具体提示词内容如下：
1. 文本总结任务（性能提升18.04%）
原始提示词：
First, please read the article below.
{context}
Now, can you write me an extremely short abstract for it?
调优后的提示词：
<task>
Your task is to provide a concise 1-2 sentence summary of the given text that captures the main points or key information.
</task>``<context>
{context}
</context>``<instructions>
Please read the provided text carefully and thoroughly to understand its content. Then, generate a brief summary in your own words that is much shorter than the original text while still preserving the core ideas and essential details. The summary should be concise yet informative, capturing the essence of the text in just 1-2 sentences.
</instructions>``<result_format>
Summary: [WRITE YOUR 1-2 SENTENCE SUMMARY HERE]
</result_format>
2. 基于RAG的对话续（提升8.23%）
原始提示词：
Functions available:
{available_functions}
Examples of calling functions:
Input:
Functions: [{"name": "calculate_area", "description": "Calculate the area of a shape", "parameters": {"type": "object", "properties": {"shape": {"type": "string", "description": "The type of shape (e.g. rectangle, triangle, circle)"}, "dimensions": {"type": "object", "properties": {"length": {"type": "number", "description": "The length of the shape"}, "width": {"type": "number", "description": "The width of the shape"}, "base": {"type": "number", "description": "The base of the shape"}, "height": {"type": "number", "description": "The height of the shape"}, "radius": {"type": "number", "description": "The radius of the shape"}}}}, "required": ["shape", "dimensions"]}}]
Conversation history: USER: Can you calculate the area of a rectangle with a length of 5 and width of 3?
Output:
{"name": "calculate_area", "arguments": {"shape": "rectangle", "dimensions": {"length": 5, "width": 3}}}``Input:
Functions: [{"name": "search_books", "description": "Search for books based on title or author", "parameters": {"type": "object", "properties": {"search_query": {"type": "string", "description": "The title or author to search for"}}, "required": ["search_query"]}}]
Conversation history: USER: I am looking for books by J.K. Rowling. Can you help me find them?
Output:
{"name": "search_books", "arguments": {"search_query": "J.K. Rowling"}}``Input:
Functions: [{"name": "calculate_age", "description": "Calculate the age based on the birthdate", "parameters": {"type": "object", "properties": {"birthdate": {"type": "string", "format": "date", "description": "The birthdate"}}, "required": ["birthdate"]}}]
Conversation history: USER: Hi, I was born on 1990-05-15. Can you tell me how old I am today?
Output:
{"name": "calculate_age", "arguments": {"birthdate": "1990-05-15"}}
Current chat history:
{conversation_history}
Respond to the last message. Call a function if necessary.
优化后的提示词：
Task: Respond to the user's message in the given conversation by calling appropriate functions if necessary.
Instructions:
1. Review the list of available functions:
<available_functions>
{available_functions}
</available_functions>
2. Study the examples of how to call these functions:
<fewshot_examples>
<example>
H:
<context>Functions: [{"name": "calculate_area", "description": "Calculate the area of a shape", "parameters": {"type": "object", "properties": {"shape": {"type": "string", "description": "The type of shape (e.g. rectangle, triangle, circle)"}, "dimensions": {"type": "object", "properties": {"length": {"type": "number", "description": "The length of the shape"}, "width": {"type": "number", "description": "The width of the shape"}, "base": {"type": "number", "description": "The base of the shape"}, "height": {"type": "number", "description": "The height of the shape"}, "radius": {"type": "number", "description": "The radius of the shape"}}}}, "required": ["shape", "dimensions"]}}]</context>
<question>USER: Can you calculate the area of a rectangle with a length of 5 and width of 3?</question>
A:
<output>{"name": "calculate_area", "arguments": {"shape": "rectangle", "dimensions": {"length": 5, "width": 3}}}</output>
</example>
<example>
H:
<context>Functions: [{"name": "search_books", "description": "Search for books based on title or author", "parameters": {"type": "object", "properties": {"search_query": {"type": "string", "description": "The title or author to search for"}}, "required": ["search_query"]}}]</context>
<question>USER: I am looking for books by J.K. Rowling. Can you help me find them?</question>
A:
<output>{"name": "search_books", "arguments": {"search_query": "J.K. Rowling"}}</output>
</example>
<example>
H:
<context>Functions: [{"name": "calculate_age", "description": "Calculate the age based on the birthdate", "parameters": {"type": "object", "properties": {"birthdate": {"type": "string", "format": "date", "description": "The birthdate"}}, "required": ["birthdate"]}}]</context>
<question>USER: Hi, I was born on 1990-05-15. Can you tell me how old I am today?</question>
A:
<output>{"name": "calculate_age", "arguments": {"birthdate": "1990-05-15"}}</output>
</example>
</fewshot_examples>
3. Carefully read the current conversation history:
<conversation_history>
{conversation_history}
</conversation_history>
4. Analyze the last message from the user and determine if any of the available functions need to be called to provide an appropriate response.
5. If a function call is necessary, follow the format demonstrated in the examples to invoke the relevant function with the required arguments.
6. If no function call is needed, provide a direct response to the user's message.
7. Your response should be concise, relevant, and tailored to the specific context of the conversation.
8. Enclose your final response in <response></response> tags, without any additional preamble or explanation.
Provide your response immediately after these instructions, following the specified format.
3.函数调用任务 （性能提升22.03%）
原始提示词：
## Retrieved knowledge snippets
Domain: Restaurant
Entity: THE COPPER KETTLE
Review: My friend Mark took me to the copper kettle to celebrate my promotion. I decided to treat myself to Shepherds Pie. It was not as flavorful as I'd have liked and the consistency was just runny, but the servers were awesome and I enjoyed the view from the patio. I may come back to try the strawberries and cream come time for Wimbledon..
Highlight: It was not as flavorful as I'd have liked and the consistency was just runny, but the servers were awesome and I enjoyed the view from the patio.
Domain: Restaurant
Entity: THE COPPER KETTLE
Review: Last week, my colleagues and I visited THE COPPER KETTLE that serves British cuisine. We enjoyed a nice view from inside of the restaurant. The atmosphere was enjoyable and the restaurant was located in a nice area. However, the food was mediocre and was served in small portions.
Highlight: We enjoyed a nice view from inside of the restaurant.
## Dialog
user: Howdy, I'm looking for a British restaurant for breakfast.
agent: There are several British restaurants available. Would you prefer a moderate or expensive price range?
user: Moderate price range please.
agent: Five restaurants match your criteria. Four are in Centre area and one is in the West. Which area would you prefer?
user: I would like the Center of town please.
agent: How about The Copper Kettle?
user: Do they offer a good view?
agent:
Given the above information from a retrieval augmented generation (RAG) system, respond with the appropriate answer to the question from the user.
优化后的提示词：
<task_description>
You are an advanced question-answering system that utilizes information from a retrieval augmented generation (RAG) system to provide accurate and relevant responses to user queries.
</task_description>``<instructions>
1. Carefully review the provided context information:
<context>
Domain: Restaurant
Entity: THE COPPER KETTLE
Review: My friend Mark took me to the copper kettle to celebrate my promotion. I decided to treat myself to Shepherds Pie. It was not as flavorful as I'd have liked and the consistency was just runny, but the servers were awesome and I enjoyed the view from the patio. I may come back to try the strawberries and cream come time for Wimbledon..
Highlight: It was not as flavorful as I'd have liked and the consistency was just runny, but the servers were awesome and I enjoyed the view from the patio.``Domain: Restaurant
Entity: THE COPPER KETTLE
Review: Last week, my colleagues and I visited THE COPPER KETTLE that serves British cuisine. We enjoyed a nice view from inside of the restaurant. The atmosphere was enjoyable and the restaurant was located in a nice area. However, the food was mediocre and was served in small portions.
Highlight: We enjoyed a nice view from inside of the restaurant.
</context>``2. Analyze the user's question:
<question>
user: Howdy, I'm looking for a British restaurant for breakfast.``agent: There are several British restaurants available. Would you prefer a moderate or expensive price range?``user: Moderate price range please.``agent: Five restaurants match your criteria. Four are in Centre area and one is in the West. Which area would you prefer?``user: I would like the Center of town please.``agent: How about The Copper Kettle?``user: Do they offer a good view?
agent:
</question>
3. Leverage the context information and your knowledge to generate a concise and accurate answer to the user's question.
4. Ensure your response directly addresses the specific query while incorporating relevant details from the context.
5. Provide your answer in a clear and easy-to-understand manner, without any unnecessary preamble or explanation.
</instructions>
<output_format>
Answer: [Insert your concise answer here]
</output_format>
<example>
Context:
The Eiffel Tower is a wrought-iron lattice tower on the Champ de Mars in Paris, France. It is named after the engineer Gustave Eiffel, whose company designed and built the tower. Constructed from 1887 to 1889 as the centerpiece of the 1889 World's Fair, it was initially criticized by some of France's leading artists and intellectuals for its design, but it has become a global cultural icon of France and one of the most recognizable structures in the world.
Question: What is the Eiffel Tower?
Answer: The Eiffel Tower is a wrought-iron lattice tower in Paris, France, named after its designer Gustave Eiffel, and constructed as the centerpiece of the 1889 World's Fair.
</example>
我们从上述优化的提示词内容可以看到，原始提示词在不同任务场景下都经过了持续的改进，突显了提示优化功能可以有效适用于各个场景和模型。我们通过使用经过优化的提示，并结合每种模型提示词的最佳创建方法，可以获得更好的生成效果的同时可以节省大量时间和精力。