Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 51 additions & 2 deletions .agents/design/core/ai/auxiliary-generation.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

## 适用范围

辅助生成用于不经过 Workflow Dispatcher、但需要复用 Chat 身份、SSE、计费、停止和 Agent Loop 的生成场景。目前的核心调用方是 Chat Agent Helper。
辅助生成用于不经过 Workflow Dispatcher、但需要复用 Chat 身份、SSE、计费、停止和 Agent Loop 的生成场景。目前的核心调用方是 Chat Agent Helper 和 Skill Edit 调试对话

它不是第二套 Workflow runtime,也不负责:

Expand All @@ -28,6 +28,10 @@
| `stop.ts` | 读取并清理统一停止标记 |
| `type.ts` | processor、用户上下文和运行结果协议 |

Skill Edit 的鉴权、消息组装、Sandbox 准备、Agent Loop runtime、ChatBox 事件和聊天持久化保留在
`packages/service/core/ai/skill/debugChat`。该目录调用辅助生成公共生命周期,但不依赖 Workflow Dispatcher
或 `agentLoopCore`。

## 执行流程

```text
Expand Down Expand Up @@ -84,6 +88,41 @@ Chat Agent Helper 读取历史时使用 `reserveTool: true`。除 interactive
- `done`、`error` 和 `aborted` 都清除该 memory,避免后续普通消息恢复陈旧暂停点。
- 通用 `saveChat` 已支持 memories;辅助生成只扩展 processor 返回协议和 Chat Agent Helper 保存调用,不修改通用保存语义。

Skill Edit 不复用 Chat Agent Helper wrapper,而是在自己的 processor 中直接调用 `runAgentLoop`,并通过公共
`AgentLoopRuntime.systemTools` 显式启用 `plan`、`ask`、`sandbox` 和 `readFile`。Skill Edit 不注册
runtime tools;Sandbox 和文件读取均使用 Agent Loop 标准 system tool 协议。

## Skill Edit 直连

Skill Edit 调试对话保留原 `/api/core/ai/skill/debugChat` 和 ChatBox SSE 协议,但移除
`workflowStart -> agent` 临时 Workflow。执行流程如下:

```text
debugChat API
|-- Skill 写权限、频控、运行中 edit sandbox 校验
|-- preChatRound 与历史/文件 URL 恢复
`-- runAuxiliaryGeneration
|-- Skill Edit processor 准备 sandbox 和当前用户上下文
|-- runAgentLoop(systemTools: plan/ask/sandbox/readFile)
|-- Skill Edit event adapter 生成 SSE、assistantResponses、nodeResponses
`-- 写入 chat round、agent providerState 和 node response rows
```

边界约束:

- Skill Edit 只依赖 Agent Loop 的 `interface` 和 Sandbox 的 `interface`,不调用 Workflow Dispatcher,
也不复用 `packages/service/core/workflow/dispatch/ai/agentLoopCore`。
- Pro 的内置 Skill prepare action 直接从 Sandbox interface 注入;Workflow 侧只保留兼容 re-export,
不维护 Skill Edit 专用 adapter。
- ChatBox 仍消费既有 answer、tool、plan、interactive、flowNodeResponse 和 duration 事件;这是传输兼容,
不代表执行经过 Workflow。
- ask 暂停时只持久化 opaque `providerState`;恢复时由 Agent Loop provider 解释。
- ask 恢复统一使用 `continuation: { type: 'ask', answer, additionalMessages }`;回答作为对应
tool response,同轮新上传的文件作为 `additionalMessages` 追加到暂停上下文,
避免把回答文本重复作为 user message。
- `read_files` 只允许读取当前聊天上下文中已授权的文件 URL,单文件失败转换为模型可见结果。
- 正常、交互暂停和 Agent Loop error 都先完成聊天与 node response 持久化,再发送 SSE `[DONE]`。

## SSE 与断流续传

- Stream key 使用 `teamId/sourceType/sourceId/chatId`,与标准 Chat source 隔离规则一致。
Expand Down Expand Up @@ -113,7 +152,17 @@ agent_runtime_stopping:<sourceType>:<sourceId>:<chatId>

## 扩展规则

- 新的辅助生成场景优先复用 `runAuxiliaryGeneration`,只新增 processor。
- 新的辅助生成场景优先复用 `runAuxiliaryGeneration`,只在所属业务域新增 processor。
- 业务事件由 processor 显式写入,不扩展通用 stream 层去理解业务配置。
- 公共生命周期需求放在本模块;单场景数据组装保留在调用方业务目录。
- source 标识统一使用 `sourceType/sourceId`,不能恢复 App-only 的 `appId` 入口。

## TODO

- [x] 将当前分支线性对齐到最新 `upstream/main`,保留旧分支恢复引用。
- [x] 在 Agent Loop 公共 Input 和两个 provider 中统一 ask `continuation` 恢复协议。
- [x] 扩展辅助生成生命周期,支持 usage 复用和 `[DONE]` 前业务持久化。
- [x] 将 Skill Debug 从临时 Workflow 改为 Skill 域内的直接 Agent Loop processor。
- [x] 将内置 Skill prepare action 下沉到 Sandbox interface,并更新 Pro Skill Debug 入口。
- [x] 覆盖消息上下文、ask 恢复、runtime/event adapter、API 收尾和错误路径测试。
- [x] 运行相关局部测试、lint 和 Pro 定向类型检查(按要求不运行全量测试)。
12 changes: 9 additions & 3 deletions packages/global/core/ai/sandbox/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,11 +57,12 @@ export const generateSandboxId = ({
// Prompt
export const SANDBOX_USER_FILES_PATH = 'user_files/';
export const SANDBOX_ENTRYPOINT_MAX_LENGTH = 16 * 1024;
export const SANDBOX_SYSTEM_PROMPT = `<sandbox_capability>
const buildSandboxSystemPrompt = (includeUserFilesPrompt: boolean) => `<sandbox_capability>
你拥有一个独立的 Linux 沙盒环境(Ubuntu 22.04),可通过 sandbox 工具操作文件和执行命令。
- 系统预装:bash / python3 / node / bun / git / curl
- 用户对话上传的文件存储在 ${SANDBOX_USER_FILES_PATH} 目录下
- 使用 ${SANDBOX_SHELL_TOOL_NAME} 执行命令、运行代码和安装依赖(apt / pip / npm)
${
includeUserFilesPrompt ? `- 用户对话上传的文件存储在 ${SANDBOX_USER_FILES_PATH} 目录下\n` : ''
}- 使用 ${SANDBOX_SHELL_TOOL_NAME} 执行命令、运行代码和安装依赖(apt / pip / npm)
- 使用 ${SANDBOX_READ_FILE_TOOL_NAME} 读取文本文件内容,可通过 offset/limit 分段读取
- 使用 ${SANDBOX_WRITE_FILE_TOOL_NAME} 创建或覆盖文本文件
- 使用 ${SANDBOX_EDIT_FILE_TOOL_NAME} 对已有文件做精确查找替换
Expand All @@ -72,3 +73,8 @@ export const SANDBOX_SYSTEM_PROMPT = `<sandbox_capability>
- HTML 等多文件预览产物必须使用相对资源路径(例如 ./assets/app.js),不要使用 /assets/app.js 这类根路径
- 若需要将生成的文件链接,可使用 ${SANDBOX_GET_FILE_URL_TOOL_NAME} 获取临时访问链接
</sandbox_capability>`;

export const SANDBOX_SYSTEM_PROMPT = buildSandboxSystemPrompt(true);

/** Skill Edit 不把对话附件写入 sandbox,附件统一通过 read_files 或多模态消息提供。 */
export const SKILL_EDIT_SANDBOX_SYSTEM_PROMPT = buildSandboxSystemPrompt(false);
14 changes: 13 additions & 1 deletion packages/global/test/core/ai/sandbox/constants.test.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
import { describe, it, expect } from 'vitest';
import { generateSandboxId } from '@fastgpt/global/core/ai/sandbox/constants';
import {
generateSandboxId,
SANDBOX_SYSTEM_PROMPT,
SKILL_EDIT_SANDBOX_SYSTEM_PROMPT
} from '@fastgpt/global/core/ai/sandbox/constants';
import { ChatSourceTypeEnum } from '@fastgpt/global/core/chat/constants';

describe('generateSandboxId', () => {
Expand Down Expand Up @@ -39,3 +43,11 @@ describe('generateSandboxId', () => {
expect(sandboxId).toBe(sandboxId.toLowerCase());
});
});

describe('sandbox system prompts', () => {
it('only advertises injected user_files to runtimes that actually mount them', () => {
expect(SANDBOX_SYSTEM_PROMPT).toContain('user_files/');
expect(SKILL_EDIT_SANDBOX_SYSTEM_PROMPT).not.toContain('user_files/');
expect(SKILL_EDIT_SANDBOX_SYSTEM_PROMPT).toContain('sandbox_read_file');
});
});
8 changes: 7 additions & 1 deletion packages/service/core/ai/auxiliaryGeneration/agentLoop.ts
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,13 @@ export async function runAuxiliaryGenerationAgentLoop({
systemPrompt,
messages,
providerState,
userAnswer
continuation:
providerState && userAnswer !== undefined
? {
type: 'ask',
answer: userAnswer
}
: undefined
}
});

Expand Down
61 changes: 40 additions & 21 deletions packages/service/core/ai/auxiliaryGeneration/service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,20 @@ export const runAuxiliaryGeneration = async <T>({
data,
histories,
usageSource,
usageId,
processor,
maxFiles,
customPdfParse,
onStreamContextReady
onStreamContextReady,
onBeforeStreamDone
}: AuxiliaryGenerationRunParams<T>): Promise<
AuxiliaryGenerationRunResult & {
streamContext: Awaited<ReturnType<typeof createAuxiliaryGenerationStream>>;
}
> => {
let stopping = false;
let stopCheckRunning = false;
let stopCheckTimer: ReturnType<typeof setInterval> | undefined;
const startedAt = Date.now();
const streamContext = await createAuxiliaryGenerationStream({
req,
Expand All @@ -46,28 +50,35 @@ export const runAuxiliaryGeneration = async <T>({
sourceId,
chatId
});
onStreamContextReady?.(streamContext);
try {
onStreamContextReady?.(streamContext);
const usageContext = await createAuxiliaryGenerationUsage({
teamId,
tmbId,
appName,
sourceType,
sourceId,
usageSource,
usageId
});
await clearAuxiliaryGenerationStop({ sourceType, sourceId, chatId });

const usageContext = await createAuxiliaryGenerationUsage({
teamId,
tmbId,
appName,
sourceType,
sourceId,
usageSource
});
await clearAuxiliaryGenerationStop({ sourceType, sourceId, chatId });
res.once('close', () => {
stopping = true;
});

res.once('close', () => {
stopping = true;
});
stopCheckTimer = setInterval(async () => {
if (stopping || stopCheckRunning) return;

const stopCheckTimer = setInterval(async () => {
if (stopping) return;
stopping = await shouldAuxiliaryGenerationStop({ sourceType, sourceId, chatId });
}, 100);
stopCheckRunning = true;
try {
const shouldStop = await shouldAuxiliaryGenerationStop({ sourceType, sourceId, chatId });
stopping = stopping || shouldStop;
} finally {
stopCheckRunning = false;
}
}, 100);

try {
const result = await processor({
query,
userAnswer,
Expand All @@ -78,6 +89,7 @@ export const runAuxiliaryGeneration = async <T>({
streamWriter: streamContext.write,
checkIsStopping: () => stopping,
usageSink: usageContext.pushUsage,
usageId: usageContext.usageId,
maxFiles,
customPdfParse,
user: {
Expand All @@ -89,15 +101,22 @@ export const runAuxiliaryGeneration = async <T>({
}
});

const durationSeconds = +((Date.now() - startedAt) / 1000).toFixed(2);
await onBeforeStreamDone?.({
result,
durationSeconds
});
streamContext.writeDone();

return {
...result,
durationSeconds: +((Date.now() - startedAt) / 1000).toFixed(2),
durationSeconds,
streamContext
};
} finally {
clearInterval(stopCheckTimer);
if (stopCheckTimer) {
clearInterval(stopCheckTimer);
}
await clearAuxiliaryGenerationStop({ sourceType, sourceId, chatId });
}
};
11 changes: 9 additions & 2 deletions packages/service/core/ai/auxiliaryGeneration/stream.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ import { getStreamResumeMirror } from '../../chat/resume';
import { createChatCompletionDeltaResponse } from '@fastgpt/global/core/ai/llm/utils';

export type AuxiliaryGenerationStreamWriter = (params: {
id?: string;
event?: `${AuxiliaryGenerationEventEnum}` | string;
data: string | object;
}) => void;
Expand Down Expand Up @@ -64,8 +65,14 @@ export const createAuxiliaryGenerationStream = async ({
}
});

const write: AuxiliaryGenerationStreamWriter = ({ event, data }) => {
const payload = typeof data === 'string' ? data : JSON.stringify(data);
const write: AuxiliaryGenerationStreamWriter = ({ id, event, data }) => {
const payload =
typeof data === 'string'
? data
: JSON.stringify({
...data,
...(id ? { responseValueId: id } : {})
});
sseContext.write({ event, data: payload });
};

Expand Down
23 changes: 18 additions & 5 deletions packages/service/core/ai/auxiliaryGeneration/type.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,11 @@
import type { NodeHttpRequest, NodeHttpResponse } from '../../../types/http';
import type { localeType } from '@fastgpt/global/common/i18n/type';
import type { ChatSourceTypeEnum } from '@fastgpt/global/core/chat/constants';
import type { AIChatItemValueItemType, ChatItemDBSchemaType } from '@fastgpt/global/core/chat/type';
import type {
AIChatItemValueItemType,
ChatHistoryItemResType,
ChatItemMiniType
} from '@fastgpt/global/core/chat/type';
import type { ChatNodeUsageType } from '@fastgpt/global/support/wallet/bill/type';
import type { UsageSourceEnum } from '@fastgpt/global/support/wallet/usage/constants';
import type { AuxiliaryGenerationChatFileType } from '@fastgpt/global/core/ai/auxiliaryGeneration/type';
Expand All @@ -20,24 +24,26 @@ export type AuxiliaryGenerationProcessorParams<T = unknown> = {
userAnswer?: string;
files: AuxiliaryGenerationChatFileType[];
data: T;
histories: ChatItemDBSchemaType[];
histories: ChatItemMiniType[];
streamWriter?: AuxiliaryGenerationStreamWriter;
requestOrigin?: string;
maxFiles?: number;
customPdfParse?: boolean;
checkIsStopping?: () => boolean;
usageSink?: (usages: ChatNodeUsageType[]) => void;
usageId: string;
user: AuxiliaryGenerationUser;
};

export type AuxiliaryGenerationProcessorResponse = {
aiResponse: AIChatItemValueItemType[];
memories?: Record<string, any>;
usage: {
usage?: {
model: string;
inputTokens: number;
outputTokens: number;
};
nodeResponses?: ChatHistoryItemResType[];
memories?: Record<string, any>;
};

export type AuxiliaryGenerationRunParams<T = unknown> = {
Expand All @@ -56,15 +62,22 @@ export type AuxiliaryGenerationRunParams<T = unknown> = {
userAnswer?: string;
files: AuxiliaryGenerationChatFileType[];
data: T;
histories: ChatItemDBSchemaType[];
histories: ChatItemMiniType[];
usageSource: UsageSourceEnum;
/** 交互续答复用上一轮 usage,避免把一次逻辑调用拆成多条计费记录。 */
usageId?: string;
processor: (
params: AuxiliaryGenerationProcessorParams<T>
) => Promise<AuxiliaryGenerationProcessorResponse>;
maxFiles?: number;
customPdfParse?: boolean;
/** SSE 创建后立即暴露给路由层,用于失败时写 error 和 flush resume。 */
onStreamContextReady?: (streamContext: AuxiliaryGenerationStreamContext) => void;
/** 公共层写结束事件前的业务收尾,例如持久化本轮聊天。 */
onBeforeStreamDone?: (params: {
result: AuxiliaryGenerationProcessorResponse;
durationSeconds: number;
}) => Promise<void> | void;
};

export type AuxiliaryGenerationRunResult = AuxiliaryGenerationProcessorResponse & {
Expand Down
26 changes: 16 additions & 10 deletions packages/service/core/ai/auxiliaryGeneration/usage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -11,20 +11,23 @@ type CreateAuxiliaryGenerationUsageParams = {
sourceType: ChatSourceTypeEnum;
sourceId: string;
usageSource: UsageSourceEnum;
usageId?: string;
};

/**
* 为辅助生成建立统一的扣费上下文。
*
* 余额校验在生成前执行;后续 processor 只需把各模型/工具用量推入 `pushUsage`。
* 余额校验在生成前执行;交互续答可以复用已有 usageId,后续 processor 只需把
* 各模型/工具用量推入 `pushUsage`。
*/
export const createAuxiliaryGenerationUsage = async ({
teamId,
tmbId,
appName,
sourceType,
sourceId,
usageSource
usageSource,
usageId: existingUsageId
}: CreateAuxiliaryGenerationUsageParams) => {
await checkTeamAIPoints(teamId);

Expand All @@ -37,14 +40,17 @@ export const createAuxiliaryGenerationUsage = async ({
if (sourceType === ChatSourceTypeEnum.skillEdit) return sourceId;
})();

const usageId = await createChatUsageRecord({
appName,
appId: usageAppId,
skillId: usageSkillId,
teamId,
tmbId,
source: usageSource
});
// 交互追问的后续轮次沿用原 usage,确保一次逻辑调用只生成一条计费记录。
const usageId =
existingUsageId ??
(await createChatUsageRecord({
appName,
appId: usageAppId,
skillId: usageSkillId,
teamId,
tmbId,
source: usageSource
}));

return {
usageId,
Expand Down
Loading
Loading