Skip to content

docs: use enterprise-safe examples in content moderation pages #792

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
RiskeyL merged 3 commits into from
May 25, 2026
Merged

docs: use enterprise-safe examples in content moderation pages #792

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d0854eb
docs: use enterprise-safe examples and localize zh/ja examples
RiskeyL May 25, 2026
42985e6
Revert "docs: use enterprise-safe examples and localize zh/ja examples"
RiskeyL May 25, 2026
c816625
Apply suggestions from code review
RiskeyL May 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions en/use-dify/workspace/api-extension/moderation-api-extension.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,10 @@ When **Content Moderation > Review Input Content** is enabled in applications su
"params": {
"app_id": "61248ab4-1125-45be-ae32-0ce91334d021",
"inputs": {
"var_1": "I will kill you.",
"var_2": "I will fuck you."
"var_1": "My SSN is 123-45-6789.",
"var_2": "My phone number is 123-456-7890."
Comment thread
RiskeyL marked this conversation as resolved.
},
"query": "Happy everydays."
"query": "Please help me update my account."
Comment thread
RiskeyL marked this conversation as resolved.
}
Comment thread
RiskeyL marked this conversation as resolved.
}
```
Expand Down Expand Up @@ -81,10 +81,10 @@ When **Content Moderation > Review Input Content** is enabled in applications su
"flagged": true,
"action": "overridden",
"inputs": {
"var_1": "I will *** you.",
"var_2": "I will *** you."
"var_1": "My SSN is ***-**-****.",
"var_2": "My phone number is ***-***-****."
},
"query": "Happy everydays."
"query": "Please help me update my account."
}
```

Expand All @@ -110,7 +110,7 @@ When **Content Moderation > Review Output Content** is enabled in applications s
"point": "app.moderation.output",
"params": {
"app_id": "61248ab4-1125-45be-ae32-0ce91334d021",
"text": "I will kill you."
"text": "My SSN is 123-45-6789."
}
Comment thread
RiskeyL marked this conversation as resolved.
Comment thread
RiskeyL marked this conversation as resolved.
}
```
Expand Down Expand Up @@ -141,7 +141,7 @@ When **Content Moderation > Review Output Content** is enabled in applications s
{
"flagged": true,
"action": "overridden",
"text": "I will *** you."
"text": "My SSN is ***-**-****."
}
```

Expand Down
16 changes: 8 additions & 8 deletions ja/use-dify/workspace/api-extension/moderation-api-extension.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,10 @@ Chatflow、Agent、チャットアシスタントなどのアプリケーショ
"params": {
"app_id": "61248ab4-1125-45be-ae32-0ce91334d021",
"inputs": {
"var_1": "I will kill you.",
"var_2": "I will fuck you."
"var_1": "私のマイナンバーは 1234-5678-9012 です。",
"var_2": "私の電話番号は 090-1234-5678 です。"
Comment thread
RiskeyL marked this conversation as resolved.
},
"query": "Happy everydays."
"query": "アカウント情報を更新してください。"
Comment thread
RiskeyL marked this conversation as resolved.
}
Comment thread
RiskeyL marked this conversation as resolved.
}
```
Expand Down Expand Up @@ -81,10 +81,10 @@ Chatflow、Agent、チャットアシスタントなどのアプリケーショ
"flagged": true,
"action": "overridden",
"inputs": {
"var_1": "I will *** you.",
"var_2": "I will *** you."
"var_1": "私のマイナンバーは ****-****-**** です。",
"var_2": "私の電話番号は ***-****-**** です。"
},
"query": "Happy everydays."
"query": "アカウント情報を更新してください。"
}
```

Expand All @@ -110,7 +110,7 @@ Chatflow、Agent、チャットアシスタントなどのアプリケーショ
"point": "app.moderation.output",
"params": {
"app_id": "61248ab4-1125-45be-ae32-0ce91334d021",
"text": "I will kill you."
"text": "私のマイナンバーは 1234-5678-9012 です。"
}
Comment thread
RiskeyL marked this conversation as resolved.
Comment thread
RiskeyL marked this conversation as resolved.
}
```
Expand Down Expand Up @@ -141,7 +141,7 @@ Chatflow、Agent、チャットアシスタントなどのアプリケーショ
{
"flagged": true,
"action": "overridden",
"text": "I will *** you."
"text": "私のマイナンバーは ****-****-**** です。"
}
```

Expand Down
4 changes: 1 addition & 3 deletions ...ns/3-0-x/en/user-guide/application-orchestrate/app-toolkits/moderation-tool.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,7 @@ Now you can also directly call the OpenAI Moderation API on Dify; you can review

## Keywords

Developers can customize the sensitive words they need to review, such as using "kill" as a keyword to perform an audit action when users input. The preset reply content should be "The content is violating usage policies." It can be anticipated that when a user inputs a text chuck containing "kill" at the terminal, it will trigger the sensitive word review tool and return the preset reply content.

![Configuring Load Balancing from Add Model](https://assets-docs.dify.ai/dify-enterprise-mintlify/en/guides/application-orchestrate/app-toolkits/15652f65c3af15d931598a866a7baa17.png)
Developers can customize the sensitive words they need to review, such as using "confidential" as a keyword to perform an audit action when users input. The preset reply content should be "The content is violating usage policies." It can be anticipated that when a user inputs a text chunk containing "confidential" at the terminal, it will trigger the sensitive word review tool and return the preset reply content.

## Moderation Extension

Expand Down
16 changes: 8 additions & 8 deletions versions/3-0-x/en/user-guide/extension/api-based-extension/moderation.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,10 @@ This module is used to review the content input by end-users and the output cont
"params": {
"app_id": "61248ab4-1125-45be-ae32-0ce91334d021",
"inputs": {
"var_1": "I will kill you.",
"var_2": "I will fuck you."
"var_1": "My SSN is 123-45-6789.",
"var_2": "My phone number is 123-456-7890."
Comment thread
RiskeyL marked this conversation as resolved.
},
"query": "Happy everydays."
"query": "Please help me update my account."
}
Comment thread
RiskeyL marked this conversation as resolved.
Comment thread
RiskeyL marked this conversation as resolved.
}
```
Expand Down Expand Up @@ -77,10 +77,10 @@ This module is used to review the content input by end-users and the output cont
"flagged": true,
"action": "overridden",
"inputs": {
"var_1": "I will *** you.",
"var_2": "I will *** you."
"var_1": "My SSN is ***-**-****.",
"var_2": "My phone number is ***-***-****."
},
"query": "Happy everydays."
"query": "Please help me update my account."
}
```

Expand All @@ -104,7 +104,7 @@ This module is used to review the content input by end-users and the output cont
"point": "app.moderation.output",
"params": {
"app_id": "61248ab4-1125-45be-ae32-0ce91334d021",
"text": "I will kill you."
"text": "My SSN is 123-45-6789."
}
Comment thread
RiskeyL marked this conversation as resolved.
Comment thread
RiskeyL marked this conversation as resolved.
}
```
Expand Down Expand Up @@ -134,6 +134,6 @@ This module is used to review the content input by end-users and the output cont
{
"flagged": true,
"action": "overridden",
"text": "I will *** you."
"text": "My SSN is ***-**-****."
}
```
8 changes: 2 additions & 6 deletions ...ns/3-0-x/zh/user-guide/application-orchestrate/app-toolkits/moderation-tool.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,12 @@ OpenAI 和大多数 LLM 公司提供的模型，都带有内容审查功能，

### 功能二：自定义关键词

开发者可以自定义需要审查的敏感词，比如把“kill”作为关键词，在用户输入的时候作审核动作，要求预设回复内容为“The content is violating usage policies.”可以预见的结果是当用户在终端输入包含“kill”的语料片段，就会触发敏感词审查工具，返回预设回复内容。

![Keywords](https://assets-docs.dify.ai/dify-enterprise-mintlify/zh_CN/guides/application-orchestrate/app-toolkits/644c2b024f59497aed3cd8ac984c96e3.png)
开发者可以自定义需要审查的敏感词，比如把“密钥”作为关键词，在用户输入的时候作审核动作，要求预设回复内容为“The content is violating usage policies.”。可以预见的结果是当用户在终端输入包含“密钥”的语料片段，就会触发敏感词审查工具，返回预设回复内容。
Comment thread
RiskeyL marked this conversation as resolved.

### 功能三： 敏感词审查 Moderation 扩展

不同的企业内部往往有着不同的敏感词审查机制，企业在开发自己的 AI 应用如企业内部知识库 ChatBot，需要对员工输入的查询内容作敏感词审查。为此，开发者可以根据自己企业内部的敏感词审查机制写一个 API 扩展，具体可参考 [敏感内容审查](/zh-hans/guides/tools/extensions/api-based/moderation)，从而在 Dify 上调用，实现敏感词审查的高度自定义和隐私保护。

![Moderation Settings](https://assets-docs.dify.ai/dify-enterprise-mintlify/zh_CN/guides/application-orchestrate/app-toolkits/d8b6dff6fce6d70795b87aefc56eb02b.png)

比如我们在自己的本地服务中自定义敏感词审查规则：不能查询有关美国总统的名字的问题。当用户在`query`变量输入"Trump"，则在对话时会返回 "Your content violates our usage policy." 测试效果如下：

![Moderation Test](https://assets-docs.dify.ai/dify-enterprise-mintlify/zh_CN/guides/application-orchestrate/app-toolkits/970c894a68f017def62c0f7253b0f44e.png)
比如我们在自己的本地服务中自定义敏感词审查规则：不能查询有关公司内部薪资的问题。当用户在`query`变量输入"薪资"，则在对话时会返回 "Your content violates our usage policy."。
Comment thread
RiskeyL marked this conversation as resolved.
16 changes: 8 additions & 8 deletions zh/use-dify/workspace/api-extension/moderation-api-extension.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,10 +41,10 @@ title: 敏感内容审查
"params": {
"app_id": "61248ab4-1125-45be-ae32-0ce91334d021",
"inputs": {
"var_1": "I will kill you.",
"var_2": "I will fuck you."
"var_1": "我的身份证号是 123456789012345678。",
"var_2": "我的手机号是 123-4567-8901。"
Comment thread
RiskeyL marked this conversation as resolved.
},
"query": "Happy everydays."
"query": "请帮我更新账户信息。"
Comment thread
RiskeyL marked this conversation as resolved.
}
Comment thread
RiskeyL marked this conversation as resolved.
}
```
Expand Down Expand Up @@ -81,10 +81,10 @@ title: 敏感内容审查
"flagged": true,
"action": "overridden",
"inputs": {
"var_1": "I will *** you.",
"var_2": "I will *** you."
"var_1": "我的身份证号是 ******************。",
"var_2": "我的手机号是 ***-****-****。"
},
"query": "Happy everydays."
"query": "请帮我更新账户信息。"
}
```

Expand All @@ -110,7 +110,7 @@ title: 敏感内容审查
"point": "app.moderation.output",
"params": {
"app_id": "61248ab4-1125-45be-ae32-0ce91334d021",
"text": "I will kill you."
"text": "我的身份证号是 123456789012345678。"
}
Comment thread
RiskeyL marked this conversation as resolved.
Comment thread
RiskeyL marked this conversation as resolved.
}
```
Expand Down Expand Up @@ -141,7 +141,7 @@ title: 敏感内容审查
{
"flagged": true,
"action": "overridden",
"text": "I will *** you."
"text": "我的身份证号是 ******************。"
}
```

Expand Down
Loading