OpenClaw工具拆解之canvas+message

一、canvas 工具

1.1 工具概述

功能 :控制节点 Canvas(屏幕共享/演示)
核心特性:

  • 支持多种操作(present/hide/navigate/eval/snapshot/A2UI)
  • 支持截图(PNG/JPEG)
  • 支持 JavaScript 执行
  • 支持 A2UI 推送(JSONL 格式)

1.2 Schema 定义

位置:第 24388 行

javascript 复制代码
const CanvasToolSchema = Type.Object({
    action: stringEnum(CANVAS_ACTIONS),
    node: Type.Optional(Type.String()),
    target: Type.Optional(Type.String()),
    url: Type.Optional(Type.String()),
    x: Type.Optional(Type.Number()),
    y: Type.Optional(Type.Number()),
    width: Type.Optional(Type.Number()),
    height: Type.Optional(Type.Number()),
    javaScript: Type.Optional(Type.String()),
    outputFormat: Type.Optional(Type.String()),
    maxWidth: Type.Optional(Type.Number()),
    quality: Type.Optional(Type.Number()),
    jsonl: Type.Optional(Type.String()),
    jsonlPath: Type.Optional(Type.String())
});

1.3 完整执行代码

位置:第 24408 行

javascript 复制代码
function createCanvasTool(options) {
    const imageSanitization = resolveImageSanitizationLimits(options?.config);
    
    return {
        label: "Canvas",
        name: "canvas",
        description: "Control node canvases (present/hide/navigate/eval/snapshot/A2UI). Use snapshot to capture the rendered UI.",
        parameters: CanvasToolSchema,
        execute: async (_toolCallId, args) => {
            const params = args;
            
            // 1. 解析 action(必填)
            const action = readStringParam$1(params, "action", { required: true });
            const gatewayOpts = readGatewayCallOptions(params);
            
            // 2. 解析节点 ID
            const nodeId = await resolveNodeId(gatewayOpts, 
                readStringParam$1(params, "node", { trim: true }), true);
            
            // 3. 构建调用函数
            const invoke = async (command, invokeParams) => await callGatewayTool("node.invoke", gatewayOpts, {
                nodeId,
                command,
                params: invokeParams,
                idempotencyKey: crypto$1.randomUUID()
            });
            
            switch (action) {
                // === action: present ===
                case "present": {
                    const placement = {
                        x: typeof params.x === "number" ? params.x : void 0,
                        y: typeof params.y === "number" ? params.y : void 0,
                        width: typeof params.width === "number" ? params.width : void 0,
                        height: typeof params.height === "number" ? params.height : void 0
                    };
                    
                    const invokeParams = {};
                    const presentTarget = readStringParam$1(params, "target", { trim: true }) ?? 
                                         readStringParam$1(params, "url", { trim: true });
                    
                    if (presentTarget) {
                        invokeParams.url = presentTarget;
                    }
                    
                    if (Number.isFinite(placement.x) || Number.isFinite(placement.y) || 
                        Number.isFinite(placement.width) || Number.isFinite(placement.height)) {
                        invokeParams.placement = placement;
                    }
                    
                    await invoke("canvas.present", invokeParams);
                    
                    return jsonResult({ ok: true });
                }
                
                // === action: hide ===
                case "hide":
                    await invoke("canvas.hide", void 0);
                    return jsonResult({ ok: true });
                
                // === action: navigate ===
                case "navigate":
                    await invoke("canvas.navigate", { 
                        url: readStringParam$1(params, "url", { trim: true }) ?? 
                             readStringParam$1(params, "target", {
                                 required: true,
                                 trim: true,
                                 label: "url"
                             }) 
                    });
                    return jsonResult({ ok: true });
                
                // === action: eval ===
                case "eval": {
                    const result = (await invoke("canvas.eval", { 
                        javaScript: readStringParam$1(params, "javaScript", { required: true }) 
                    }))?.payload?.result;
                    
                    if (result) {
                        return {
                            content: [{
                                type: "text",
                                text: result
                            }],
                            details: { result }
                        };
                    }
                    
                    return jsonResult({ ok: true });
                }
                
                // === action: snapshot ===
                case "snapshot": {
                    const formatRaw = typeof params.outputFormat === "string" ? 
                        params.outputFormat.toLowerCase() : "png";
                    
                    // 调用截图
                    const payload = parseCanvasSnapshotPayload((await invoke("canvas.snapshot", {
                        format: formatRaw === "jpg" || formatRaw === "jpeg" ? "jpeg" : "png",
                        maxWidth: typeof params.maxWidth === "number" && Number.isFinite(params.maxWidth) ? 
                            params.maxWidth : void 0,
                        quality: typeof params.quality === "number" && Number.isFinite(params.quality) ? 
                            params.quality : void 0
                    }))?.payload);
                    
                    // 保存到临时文件
                    const filePath = canvasSnapshotTempPath({ 
                        ext: payload.format === "jpeg" ? "jpg" : payload.format 
                    });
                    await writeBase64ToFile(filePath, payload.base64);
                    
                    const mimeType = imageMimeFromFormat(payload.format) ?? "image/png";
                    
                    // 返回图片结果
                    return await imageResult({
                        label: "canvas:snapshot",
                        path: filePath,
                        base64: payload.base64,
                        mimeType,
                        details: { format: payload.format },
                        imageSanitization
                    });
                }
                
                // === action: a2ui_push ===
                case "a2ui_push": {
                    const jsonl = typeof params.jsonl === "string" && params.jsonl.trim() ? 
                        params.jsonl : 
                        typeof params.jsonlPath === "string" && params.jsonlPath.trim() ? 
                        await readJsonlFromPath(params.jsonlPath) : "";
                    
                    if (!jsonl.trim()) {
                        throw new Error("jsonl or jsonlPath required");
                    }
                    
                    await invoke("canvas.a2ui.pushJSONL", { jsonl });
                    
                    return jsonResult({ ok: true });
                }
                
                // === action: a2ui_reset ===
                case "a2ui_reset":
                    await invoke("canvas.a2ui.reset", void 0);
                    return jsonResult({ ok: true });
                
                // === 未知 action ===
                default:
                    throw new Error(`Unknown action: ${action}`);
            }
        }
    };
}

1.4 支持的 Actions

Action 说明 必需参数 可选参数
present 显示 Canvas 无 target/url, x, y, width, height
hide 隐藏 Canvas 无 无
navigate 导航到 URL url/target 无
eval 执行 JavaScript javaScript 无
snapshot 截图 无 outputFormat, maxWidth, quality
a2ui_push 推送 A2UI jsonl/jsonlPath 无
a2ui_reset 重置 A2UI 无 无

1.5 执行流程图

复制代码
canvas 工具调用
    ↓
1. 解析 action(必填)
    ↓
2. 解析节点 ID
    ↓
3. 根据 action 执行
    ├─ present → 显示 Canvas(URL + 位置)
    ├─ hide → 隐藏 Canvas
    ├─ navigate → 导航到新 URL
    ├─ eval → 执行 JavaScript
    │  └─ 返回执行结果
    ├─ snapshot → 截图
    │  ├─ 调用截图命令
    │  ├─ 解析 payload
    │  ├─ 保存临时文件
    │  └─ 返回图片
    ├─ a2ui_push → 推送 JSONL
    └─ a2ui_reset → 重置 A2UI
    ↓
4. 返回结果

1.6 返回结果格式

present/hide/navigate/a2ui_reset:

json 复制代码
{
  "ok": true
}

eval 成功:

json 复制代码
{
  "content": [{
    "type": "text",
    "text": "JavaScript 执行结果"
  }],
  "details": {
    "result": "JavaScript 执行结果"
  }
}

snapshot 成功:

json 复制代码
{
  "content": [
    { "type": "text", "text": "canvas:snapshot" },
    {
      "type": "image",
      "data": "base64...",
      "mimeType": "image/png"
    }
  ],
  "details": {
    "format": "png",
    "path": "/tmp/canvas-snapshot-xxx.png"
  }
}

二、message 工具

2.1 工具概述

功能 :跨渠道发送和管理消息
核心特性:

  • 支持多种消息操作(send/reply/react/edit/unsend 等)
  • 支持秘密解析(resolveSecretRefs)
  • 支持上下文装饰(toolContext)
  • 支持中止信号(abortSignal)
  • 自动剥离推理标签

2.2 Schema 构建

位置:第 102938 行

javascript 复制代码
// Schema 是动态构建的,取决于配置
const schema = options?.config ? buildMessageToolSchema({
    cfg: options.config,
    currentChannelProvider: options.currentChannelProvider,
    currentChannelId: options.currentChannelId,
    currentThreadTs: options.currentThreadTs,
    currentMessageId: options.currentMessageId,
    currentAccountId: agentAccountId,
    sessionKey: options.agentSessionKey,
    sessionId: options.sessionId,
    agentId: resolvedAgentId,
    requesterSenderId: options.requesterSenderId
}) : MessageToolSchema;

2.3 完整执行代码

位置:第 103069 行

javascript 复制代码
function createMessageTool(options) {
    // 1. 解析依赖函数
    const loadConfigForTool = options?.loadConfig ?? loadConfig;
    const resolveSecretRefsForTool = options?.resolveCommandSecretRefsViaGateway ?? resolveCommandSecretRefsViaGateway;
    const runMessageActionForTool = options?.runMessageAction ?? runMessageAction;
    
    // 2. 解析 Agent 账户 ID
    const agentAccountId = resolveAgentAccountId(options?.agentAccountId);
    const resolvedAgentId = options?.agentSessionKey ? resolveSessionAgentId({
        sessionKey: options.agentSessionKey,
        config: options?.config
    }) : void 0;
    
    // 3. 构建 Schema
    const schema = options?.config ? buildMessageToolSchema({
        cfg: options.config,
        currentChannelProvider: options.currentChannelProvider,
        currentChannelId: options.currentChannelId,
        currentThreadTs: options.currentThreadTs,
        currentMessageId: options.currentMessageId,
        currentAccountId: agentAccountId,
        sessionKey: options.agentSessionKey,
        sessionId: options.sessionId,
        agentId: resolvedAgentId,
        requesterSenderId: options.requesterSenderId
    }) : MessageToolSchema;
    
    return {
        label: "Message",
        name: "message",
        displaySummary: "Send and manage messages across configured channels.",
        description: buildMessageToolDescription({
            config: options?.config,
            currentChannel: options?.currentChannelProvider,
            currentChannelId: options?.currentChannelId,
            currentThreadTs: options?.currentThreadTs,
            currentMessageId: options?.currentMessageId,
            currentAccountId: agentAccountId,
            sessionKey: options?.agentSessionKey,
            sessionId: options?.sessionId,
            agentId: resolvedAgentId,
            requesterSenderId: options?.requesterSenderId
        }),
        parameters: schema,
        execute: async (_toolCallId, args, signal) => {
            // 4. 检查中止信号
            if (signal?.aborted) {
                const err = new Error("Message send aborted");
                err.name = "AbortError";
                throw err;
            }
            
            const params = { ...args };
            
            // 5. 剥离推理标签
            for (const field of ["text", "content", "message", "caption"]) {
                if (typeof params[field] === "string") {
                    params[field] = stripReasoningTagsFromText(params[field]);
                }
            }
            
            // 6. 解析 action(必填)
            const action = readStringParam$1(params, "action", { required: true });
            
            // 7. 解析配置(如果需要秘密解析)
            let cfg = options?.config;
            if (!cfg) {
                const loadedRaw = loadConfigForTool();
                
                const scope = resolveMessageSecretScope({
                    channel: params.channel,
                    target: params.target,
                    targets: params.targets,
                    fallbackChannel: options?.currentChannelProvider,
                    accountId: params.accountId,
                    fallbackAccountId: agentAccountId
                });
                
                const scopedTargets = getScopedChannelsCommandSecretTargets({
                    config: loadedRaw,
                    channel: scope.channel,
                    accountId: scope.accountId
                });
                
                cfg = (await resolveSecretRefsForTool({
                    config: loadedRaw,
                    commandName: "tools.message",
                    targetIds: scopedTargets.targetIds,
                    ...scopedTargets.allowedPaths ? { allowedPaths: scopedTargets.allowedPaths } : {},
                    mode: "enforce_resolved"
                })).resolvedConfig;
            }
            
            // 8. 检查显式目标要求
            if (options?.requireExplicitTarget === true && actionNeedsExplicitTarget(action)) {
                if (!(
                    typeof params.target === "string" && params.target.trim().length > 0 ||
                    typeof params.to === "string" && params.to.trim().length > 0 ||
                    typeof params.channelId === "string" && params.channelId.trim().length > 0 ||
                    Array.isArray(params.targets) && params.targets.some((value) => 
                        typeof value === "string" && value.trim().length > 0
                    )
                )) {
                    throw new Error("Explicit message target required for this run. Provide target/targets (and channel when needed).");
                }
            }
            
            // 9. 解析账户 ID
            const accountId = readStringParam$1(params, "accountId") ?? agentAccountId;
            if (accountId) params.accountId = accountId;
            
            // 10. 解析 Gateway 选项
            const gatewayResolved = resolveGatewayOptions$1({
                gatewayUrl: readStringParam$1(params, "gatewayUrl", { trim: false }),
                gatewayToken: readStringParam$1(params, "gatewayToken", { trim: false }),
                timeoutMs: readNumberParam(params, "timeoutMs")
            });
            
            const gateway = {
                url: gatewayResolved.url,
                token: gatewayResolved.token,
                timeoutMs: gatewayResolved.timeoutMs,
                clientName: GATEWAY_CLIENT_IDS.GATEWAY_CLIENT,
                clientDisplayName: "agent",
                mode: GATEWAY_CLIENT_MODES.BACKEND
            };
            
            // 11. 构建工具上下文
            const hasCurrentMessageId = typeof options?.currentMessageId === "number" || 
                                       typeof options?.currentMessageId === "string" && 
                                       options.currentMessageId.trim().length > 0;
            
            const toolContext = options?.currentChannelId || options?.currentChannelProvider || 
                               options?.currentThreadTs || hasCurrentMessageId || 
                               options?.replyToMode || options?.hasRepliedRef ? {
                currentChannelId: options?.currentChannelId,
                currentChannelProvider: options?.currentChannelProvider,
                currentThreadTs: options?.currentThreadTs,
                currentMessageId: options?.currentMessageId,
                replyToMode: options?.replyToMode,
                hasRepliedRef: options?.hasRepliedRef,
                skipCrossContextDecoration: true
            } : void 0;
            
            // 12. 执行消息操作
            const result = await runMessageActionForTool({
                cfg,
                action,
                params,
                defaultAccountId: accountId ?? void 0,
                requesterSenderId: options?.requesterSenderId,
                gateway,
                toolContext,
                sessionKey: options?.agentSessionKey,
                sessionId: options?.sessionId,
                agentId: resolvedAgentId,
                sandboxRoot: options?.sandboxRoot,
                abortSignal: signal
            });
            
            // 13. 返回结果
            const toolResult = getToolResult(result);
            if (toolResult) return toolResult;
            
            return jsonResult(result.payload);
        }
    };
}

2.4 支持的消息操作

Action 说明 必需参数
send 发送消息 text/content, target/channel
reply 回复消息 text, replyTo
react 添加表情反应 reaction, target
edit 编辑消息 text, messageId
unsend 撤回消息 messageId
delete 删除消息 messageId
pin 置顶消息 messageId
unpin 取消置顶 messageId
broadcast 广播消息 text, targets
poll 创建投票 question, options
poll-vote 投票 pollId, option
... 更多操作 ...

2.5 执行流程图

复制代码
message 工具调用
    ↓
1. 检查中止信号
    ↓
2. 剥离推理标签
    ↓
3. 解析 action(必填)
    ↓
4. 解析配置(秘密解析)
    ↓
5. 检查显式目标要求
    ↓
6. 解析账户 ID
    ↓
7. 解析 Gateway 选项
    ↓
8. 构建工具上下文
    ↓
9. 执行消息操作
    ↓
10. 返回结果

2.6 秘密解析流程

javascript 复制代码
// 1. 解析秘密作用域
const scope = resolveMessageSecretScope({
    channel: params.channel,
    target: params.target,
    targets: params.targets,
    fallbackChannel: options?.currentChannelProvider,
    accountId: params.accountId,
    fallbackAccountId: agentAccountId
});

// 2. 获取作用域目标
const scopedTargets = getScopedChannelsCommandSecretTargets({
    config: loadedRaw,
    channel: scope.channel,
    accountId: scope.accountId
});

// 3. 解析秘密引用
const cfg = (await resolveSecretRefsForTool({
    config: loadedRaw,
    commandName: "tools.message",
    targetIds: scopedTargets.targetIds,
    ...scopedTargets.allowedPaths ? { allowedPaths: scopedTargets.allowedPaths } : {},
    mode: "enforce_resolved"  // 强制解析为已解决的秘密
})).resolvedConfig;

2.7 推理标签剥离

javascript 复制代码
// 从消息内容中剥离推理标签
function stripReasoningTagsFromText(text) {
    if (!text) return text;
    
    // 移除 <thinking>...</thinking>
    text = text.replace(/<thinking>[\s\S]*?<\/thinking>/g, "");
    
    // 移除 <reasoning>...</reasoning>
    text = text.replace(/<reasoning>[\s\S]*?<\/reasoning>/g, "");
    
    return text.trim();
}

三、关键机制对比

3.1 功能定位

特性 canvas message
用途 屏幕共享/演示 跨渠道消息
目标 节点 Canvas 消息渠道
媒体支持 截图 不支持

3.2 操作类型

特性 canvas message
actions 数量 7 个 30+ 个
创建操作 present send/broadcast
执行代码 eval 不支持

3.3 安全限制

限制类型 canvas message
所有者限制 不需要 不需要
秘密解析 不需要 resolveSecretRefs
显式目标 不需要 requireExplicitTarget
中止信号 不支持 abortSignal

四、使用示例

4.1 canvas 工具调用

用户 :截取节点屏幕

大模型返回:

json 复制代码
{
  "tool_call": {
    "name": "canvas",
    "arguments": { 
      "action": "snapshot",
      "node": "my-phone",
      "outputFormat": "png"
    }
  }
}

执行结果:

json 复制代码
{
  "content": [
    { "type": "text", "text": "canvas:snapshot" },
    {
      "type": "image",
      "data": "base64...",
      "mimeType": "image/png"
    }
  ],
  "details": {
    "format": "png",
    "path": "/tmp/canvas-snapshot-xxx.png"
  }
}

4.2 message 工具调用

用户 :发送消息到飞书群

大模型返回:

json 复制代码
{
  "tool_call": {
    "name": "message",
    "arguments": { 
      "action": "send",
      "channel": "feishu",
      "target": "oc_xxx",
      "text": "大家好!"
    }
  }
}

执行结果:

json 复制代码
{
  "ok": true,
  "messageId": "om_xxx",
  "timestamp": 1711716000000
}
相关推荐
智联视频超融合平台4 分钟前
视频 AI 的误报之战,正在从调参数卷到改架构
人工智能·大模型·边缘计算·视频监控·安防
水如烟1 小时前
孤能子视角:智能的边界不在算法强弱,在可形式化性——从概率法到因果推理,选择池的边界一直在那里
人工智能
万里侯2 小时前
日志聚类之后别把 noise 扔掉:降噪应该改变视图,不抹掉证据
人工智能·微服务·ai·容器·k8s
澹锦汐2 小时前
纯前端秋日银杏与枫叶落叶色板智能聚类器:基于 K-Means 色彩量化与直方图自动配方提取
前端·人工智能·ai·语言模型·vue·react·k-means
qq_158895411113 小时前
Suno-API 是什么?官方 V6 模型与 Python / TypeScript 接入指南
python·ai音乐·suno api
Flynt9 小时前
AI 写的 Three.js 大屏,为什么越跑越卡?我在 Node 里复现了它的内存曲线
前端·ai编程·three.js
J船长10 小时前
AI 知识库的主流做法是 RAG,RAG怎么做
ai编程
小虎AI生活10 小时前
哇塞~24小时开机的WorkBuddy免费领,速来
aigc·ai编程
71777710 小时前
国内 AI 研发效能工具怎么选:2026 年基于公开信息的选型框架与 Gitee 实践
人工智能·gitee
Csvn10 小时前
换供应商要改一遍代码?把模型接入收进一层网关(C04)
人工智能·aigc·agent