Skip to content

docs(custom-agents): add BYOI section and improve example agent#2568

Open
rh-hemartin wants to merge 1 commit into
mainfrom
docs/custom-agents-byoi
Open

docs(custom-agents): add BYOI section and improve example agent#2568
rh-hemartin wants to merge 1 commit into
mainfrom
docs/custom-agents-byoi

Conversation

@rh-hemartin

@rh-hemartin rh-hemartin commented Jun 23, 2026

Copy link
Copy Markdown
Member

Summary

  • Add "Bringing your own identity" section explaining how to use a custom GitHub App instead of github-actions[bot]
  • Improve example agent with a concrete greeting use case
  • Fix sandbox paths in harness example

Closes #2554

Test plan

  • Review rendered docs on branch

🤖 Generated with Claude Code

@rh-hemartin rh-hemartin self-assigned this Jun 23, 2026
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Docs: add BYOI (GitHub App identity) and improve custom agent example
📝 Documentation 🕐 20-40 Minutes

Grey Divider

Description

• Document how to comment as a custom GitHub App instead of github-actions[bot].
• Refine the sample agent into a concrete “compose a greeting” workflow with schema updates.
• Fix harness/sandbox path examples and required workflow env wiring for the docs.
Diagram

graph TD
  A["GitHub Workflow"] --> B["App token step"] --> C["Run fullsend"] --> D["Post-script"] --> E["GitHub Issue"]
  C --> F["Sandbox workspace"]

  subgraph Legend
    direction LR
    _wf["Workflow step"] ~~~ _ext{{"GitHub"}} ~~~ _env[("Sandbox/env")]
  end
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Use a fine-grained PAT for GH_TOKEN
  • ➕ Simpler to set up than a GitHub App
  • ➕ Works in any workflow without extra action steps
  • ➖ Weaker security posture (long-lived secret)
  • ➖ Identity is a user, not a bot/application
  • ➖ Harder to scope and rotate safely at scale
2. Keep github-actions[bot] via GITHUB_TOKEN
  • ➕ Zero extra configuration; least moving parts
  • ➕ Permissions can be controlled via workflow permissions block
  • ➖ Cannot customize the commenting identity
  • ➖ May not match org policies requiring app-based automation identities

Recommendation: The documented approach (GitHub App + actions/create-github-app-token) is the best default for BYOI: it provides a distinct automation identity with tightly scoped, revocable permissions while integrating cleanly with the existing GH_TOKEN-based flow.

Files changed (1) +66 / -18

Documentation (1) +66 / -18
building-custom-agents.mdAdd BYOI guidance and make the example agent post a greeting comment +66/-18

Add BYOI guidance and make the example agent post a greeting comment

• Refines the example agent from a placeholder workflow into a concrete “compose a greeting” use case, including updated JSON output and schema constraints. Fixes sandbox path examples and updates the harness/workflow env variables (e.g., REPO_FULL_NAME) so the post-script can comment on the issue. Adds a new section describing how to run the workflow under a custom GitHub App identity using create-github-app-token.

docs/guides/user/building-custom-agents.md

@rh-hemartin rh-hemartin force-pushed the docs/custom-agents-byoi branch from 7be4418 to 859acdd Compare June 23, 2026 15:26
@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (3) 📘 Rule violations (0) 📜 Skill insights (1)

Context used
✅ Compliance rules (platform): 51 rules

Grey Divider


Action required

1. Missing issues write permission 🐞 Bug ≡ Correctness
Description
The guide’s workflow sets GH_TOKEN to ${{ secrets.GITHUB_TOKEN }} and the post-script calls `gh
issue comment, but the workflow permissions: block does not grant issues: write`, so posting the
greeting will fail with insufficient permissions.
Code

docs/guides/user/building-custom-agents.md[R330-334]

+# Post the greeting as a comment on the issue.
+if [[ -n "${GREETING}" && "${STATUS}" == "complete" ]]; then
+  gh issue comment "${ISSUE_KEY}" --repo "${REPO_FULL_NAME}" --body "${GREETING}"
+  echo "Posted greeting to issue #${ISSUE_KEY}"
+fi
Relevance

⭐⭐⭐ High

Repo workflows/scripts rely on gh issue comment; similar comment-posting patterns are maintained
and hardened historically.

PR-#438
PR-#2306
PR-#279

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The post-script now invokes gh issue comment, while the example workflow uses
secrets.GITHUB_TOKEN and defines a restricted permissions block that lacks issues: write. Repo
scaffold workflows that comment on issues explicitly include issues: write, demonstrating the
required permission pattern in this codebase.

docs/guides/user/building-custom-agents.md[330-334]
docs/guides/user/building-custom-agents.md[356-446]
internal/scaffold/fullsend-repo/.github/workflows/triage.yml[5-10]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The documentation’s example workflow attempts to post a GitHub issue comment using the default `GITHUB_TOKEN`, but the example `permissions:` block omits `issues: write`. When a workflow defines `permissions`, any omitted scopes are restricted, so `gh issue comment` will not be allowed.

## Issue Context
The post-script now posts a greeting comment via `gh issue comment ...`, which requires `issues: write` for `GITHUB_TOKEN`.

## Fix Focus Areas
- docs/guides/user/building-custom-agents.md[330-334]
- docs/guides/user/building-custom-agents.md[356-446]

## Proposed fix
Update the example workflow `permissions:` block to include at least:
- `issues: write`

(Optionally also `pull-requests: write` if readers will use the same pattern on PRs.)

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. BYOI steps not numbered 📜 Skill insight ✧ Quality
Description
The new "Bringing your own identity" procedure is written as prose paragraphs rather than a numbered
(ordered) list of steps. This violates the documentation requirement for procedural content to use
numbered steps, which helps ensure clarity and consistent execution.
Code

docs/guides/user/building-custom-agents.md[R476-507]

+### Bringing your own identity
+
+To make the agent comment as an application other than `github-actions` create a new application,
+install it in your repository and generate a key for it. Then upload it alongside the app id
+to your repo:
+
+```bash
+gh secret set MY_AGENT_APP_PRIVATE_KEY --repo OWNER/REPO < path/to/private-key.pem
+gh variable set MY_AGENT_APP_ID --repo OWNER/REPO --body "123456"
+```
+
+Next modify the workflow you create to run `fullsend run` to add a new step:
+
+```yaml
+- name: Generate app token
+  id: app-token
+  uses: actions/create-github-app-token@v3
+  with:
+    app-id: ${{ vars.MY_AGENT_APP_ID }}
+    private-key: ${{ secrets.MY_AGENT_APP_PRIVATE_KEY }}
+    repositories: ${{ github.event.repository.name }}
+```
+
+And then modify where the `GH_TOKEN` variable comes from:
+
+```yaml
+- name: Run my-agent
+  env:
+    GH_TOKEN: ${{ steps.app-token.outputs.token }}
+    ...
+  ```
+
Relevance

⭐⭐ Medium

No historical evidence found that docs procedures must be numbered steps; doc style changes are
inconsistently enforced.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
PR Compliance ID 1062079 requires procedural instructions in documentation guides to be expressed as
numbered steps. The "Bringing your own identity" section is a sequence of actions presented as prose
(e.g., "To make...", "Next modify...", "And then modify...") with code blocks, but no numbered step
list.

docs/guides/user/building-custom-agents.md[476-507]
Skill: writing-user-docs

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The "Bringing your own identity" section contains procedural instructions written as prose paragraphs. Procedures in guides must be expressed as numbered steps.

## Issue Context
This section instructs readers to create/install a GitHub App, store secrets/variables, add a token-generation step, and update `GH_TOKEN`. These are sequential actions and should be formatted as an ordered list.

## Fix Focus Areas
- docs/guides/user/building-custom-agents.md[476-507]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Jira path posts GitHub comment 🐞 Bug ≡ Correctness
Description
The pre-script supports ISSUE_SOURCE=jira, but the post-script always runs gh issue comment on
complete, so a Jira run will attempt to comment on GitHub using a Jira issue key and fail.
Code

docs/guides/user/building-custom-agents.md[R330-334]

+# Post the greeting as a comment on the issue.
+if [[ -n "${GREETING}" && "${STATUS}" == "complete" ]]; then
+  gh issue comment "${ISSUE_KEY}" --repo "${REPO_FULL_NAME}" --body "${GREETING}"
+  echo "Posted greeting to issue #${ISSUE_KEY}"
+fi
Relevance

⭐⭐ Medium

No clear historical evidence on Jira vs GitHub issue-source branching in docs/examples.

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The doc’s pre-script and workflow inputs explicitly support Jira, but the newly added post-script
behavior only supports GitHub comments, creating a contradiction that will cause Jira runs to fail
once the greeting-posting path executes.

docs/guides/user/building-custom-agents.md[269-279]
docs/guides/user/building-custom-agents.md[363-375]
docs/guides/user/building-custom-agents.md[330-334]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The guide claims the agent can be triggered with `issue_source: jira or github` and the pre-script has logic for Jira vs GitHub, but the post-script unconditionally posts a GitHub comment when `status == complete`. This breaks Jira runs.

## Issue Context
- Pre-script branches on `ISSUE_SOURCE` and supports Jira.
- Workflow input explicitly allows `issue_source`.
- Post-script only implements GitHub-side effects.

## Fix Focus Areas
- docs/guides/user/building-custom-agents.md[269-279]
- docs/guides/user/building-custom-agents.md[363-375]
- docs/guides/user/building-custom-agents.md[330-334]

## Proposed fix
Update the post-script example to either:
1) Only comment when `ISSUE_SOURCE == github` (and skip otherwise), or
2) Add a Jira comment implementation for the Jira path.

Minimal safe change:
```bash
if [[ "${ISSUE_SOURCE:-}" == "github" && -n "${GREETING}" && "${STATUS}" == "complete" ]]; then
 gh issue comment "${ISSUE_KEY}" --repo "${REPO_FULL_NAME}" --body "${GREETING}"
fi
```

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

4. Wrong GitHub App var name 🐞 Bug ≡ Correctness
Description
In the BYOI section, the doc instructs setting MY_AGENT_APP_ID but the git identity example uses
${{ vars.MY_AGENT_APPID }} (missing underscore), which will expand to empty and produce an invalid
noreply email.
Code

docs/guides/user/building-custom-agents.md[R510-513]

+```bash
+git config --global user.name "my-agent[bot]"
+git config --global user.email "${{ vars.MY_AGENT_APPID }}+my-agent[bot]@users.noreply.github.com"
+```
Relevance

⭐⭐⭐ High

Team frequently accepts doc correctness/consistency fixes for variable names and examples in guides.

PR-#1179
PR-#665

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The doc shows users creating MY_AGENT_APP_ID and using it in the app-token step, but later
references a different variable name (MY_AGENT_APPID) in the commit email snippet, which will not
be set if readers follow the instructions.

docs/guides/user/building-custom-agents.md[482-485]
docs/guides/user/building-custom-agents.md[489-497]
docs/guides/user/building-custom-agents.md[510-513]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The BYOI instructions define the repository variable as `MY_AGENT_APP_ID`, but the later `git config user.email` example references `vars.MY_AGENT_APPID`, which is a different name and will not resolve.

## Issue Context
This is in the optional BYOI section for configuring commit identity when using a GitHub App token.

## Fix Focus Areas
- docs/guides/user/building-custom-agents.md[482-485]
- docs/guides/user/building-custom-agents.md[510-513]

## Proposed fix
Change:
- `${{ vars.MY_AGENT_APPID }}`

To:
- `${{ vars.MY_AGENT_APP_ID }}`

So the documented variable name matches what readers are told to create and what the workflow step uses.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Qodo Logo

Comment on lines +476 to +507
### Bringing your own identity

To make the agent comment as an application other than `github-actions` create a new application,
install it in your repository and generate a key for it. Then upload it alongside the app id
to your repo:

```bash
gh secret set MY_AGENT_APP_PRIVATE_KEY --repo OWNER/REPO < path/to/private-key.pem
gh variable set MY_AGENT_APP_ID --repo OWNER/REPO --body "123456"
```

Next modify the workflow you create to run `fullsend run` to add a new step:

```yaml
- name: Generate app token
id: app-token
uses: actions/create-github-app-token@v3
with:
app-id: ${{ vars.MY_AGENT_APP_ID }}
private-key: ${{ secrets.MY_AGENT_APP_PRIVATE_KEY }}
repositories: ${{ github.event.repository.name }}
```

And then modify where the `GH_TOKEN` variable comes from:

```yaml
- name: Run my-agent
env:
GH_TOKEN: ${{ steps.app-token.outputs.token }}
...
```

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Byoi steps not numbered 📜 Skill insight ✧ Quality

The new "Bringing your own identity" procedure is written as prose paragraphs rather than a numbered
(ordered) list of steps. This violates the documentation requirement for procedural content to use
numbered steps, which helps ensure clarity and consistent execution.
Agent Prompt
## Issue description
The "Bringing your own identity" section contains procedural instructions written as prose paragraphs. Procedures in guides must be expressed as numbered steps.

## Issue Context
This section instructs readers to create/install a GitHub App, store secrets/variables, add a token-generation step, and update `GH_TOKEN`. These are sequential actions and should be formatted as an ordered list.

## Fix Focus Areas
- docs/guides/user/building-custom-agents.md[476-507]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

Comment thread docs/guides/user/building-custom-agents.md
Comment thread docs/guides/user/building-custom-agents.md
@fullsend-ai-review

fullsend-ai-review Bot commented Jun 23, 2026

Copy link
Copy Markdown

🤖 Finished Review · ❌ Failure · Started 3:30 PM UTC · Completed 3:42 PM UTC
Commit: 859acdd · View workflow run →

@codecov

codecov Bot commented Jun 23, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@fullsend-ai-review

fullsend-ai-review Bot commented Jun 23, 2026

Copy link
Copy Markdown

Review

Findings

High

  • [logic-error] docs/guides/user/building-custom-agents.md:266 — The diff removes the if [[ "${ISSUE_SOURCE}" == "jira" ]] block and its body but keeps the elif [[ "${ISSUE_SOURCE}" == "github" ]] line unchanged. After the removal, the pre-script code block starts with elif, which is a bash syntax error. The resulting script will fail to parse at runtime for any user who copies it.
    Remediation: Change the elif to if so the script reads: if [[ "${ISSUE_SOURCE}" == "github" ]]; then.

Medium

  • [permission-expansion] docs/guides/user/building-custom-agents.md:393 — The diff adds a second top-level permissions: block (with contents: read and issues: write) after the on: key, while the existing workflow example already has a permissions: block before on: (with contents: read and id-token: write). Duplicate top-level YAML keys have undefined behavior per the YAML spec; GitHub Actions typically uses the last occurrence. Users copying this example will lose id-token: write, which is required for GCP Workload Identity Federation authentication.
    Remediation: Consolidate into a single permissions: block containing all three permissions: contents: read, id-token: write, and issues: write.

  • [injection] docs/guides/user/building-custom-agents.md:326 — The post-script passes untrusted agent output (GREETING) directly to gh issue comment --body "${GREETING}". While bash double-quoting prevents shell injection, the GREETING originates from the untrusted sandbox. The post-script does not independently validate the length or content before posting. A compromised agent could inject arbitrary GitHub markdown content (@mentions, phishing links) into issue comments. The 280-char maxLength constraint is only enforced inside the sandbox validation loop, not in the post-script itself.
    Remediation: Add a length check in the post-script before posting (e.g., if [[ ${#GREETING} -gt 280 ]]; then echo 'ERROR: greeting exceeds max length'; exit 1; fi) and consider sanitizing markdown.

  • [internal-inconsistency] docs/guides/user/building-custom-agents.md:362 — The diff changes the workflow's issue_source input description from 'Issue source: jira or github' to 'Issue source: github', but the input is still declared with required: true and default: 'github'. Since Jira support was removed from the pre-script, the issue_source input is now pointless. The pre-script's elif (which is also a syntax error — see high finding) means any value other than 'github' silently skips data fetching.
    Remediation: Either remove the issue_source input entirely and hardcode the GitHub path in the pre-script, or add validation for unexpected ISSUE_SOURCE values.


Labels: PR modifies user-facing documentation under docs/guides/

Previous run

Review

Findings

High

  • [API contract violation] docs/guides/user/building-custom-agents.md:350 — The post-script example calls gh issue comment "${ISSUE_KEY}" --repo "${REPO_FULL_NAME}" --body "${GREETING}", which requires issues: write permission on the GITHUB_TOKEN. However, the example workflow's permissions block only declares contents: read and id-token: write. The default GITHUB_TOKEN will lack permission to comment on issues, causing the gh issue comment call to fail with a 403 error at runtime.
    Remediation: Add issues: write to the permissions block in the example workflow.

Labels: PR modifies user-facing documentation under docs/guides/

Previous run

Review

Findings

High

  • [runtime mechanism] docs/guides/user/building-custom-agents.md:502 — Variable name mismatch in the "Bringing your own identity" section. The variable is stored as MY_AGENT_APP_ID (gh variable set MY_AGENT_APP_ID), but referenced as MY_AGENT_APPID (${{ vars.MY_AGENT_APPID }}) in the git config email line. The missing underscore between "APP" and "ID" means vars.MY_AGENT_APPID will resolve to an empty string at runtime, producing an incorrect email address.
    Remediation: Change ${{ vars.MY_AGENT_APPID }} to ${{ vars.MY_AGENT_APP_ID }} in the git config email command.

  • [API contract violation] docs/guides/user/building-custom-agents.md:350 — The post-script example now includes gh issue comment "${ISSUE_KEY}" --repo "${REPO_FULL_NAME}" --body "${GREETING}", which requires issues: write permission. However, the example workflow's permissions block only declares contents: read and id-token: write. The GITHUB_TOKEN will lack permission to comment on issues, causing the gh issue comment call to fail with a 403 error at runtime.
    Remediation: Add issues: write to the workflow permissions block.

Medium

  • [internal-consistency] docs/guides/user/building-custom-agents.md:129 — The host_files destination paths are updated inconsistently. Three entries move from /tmp/workspace/ to /sandbox/workspace/, but the GCP credentials entry moves to /tmp/.gcp-credentials.json (not /sandbox/workspace/.gcp-credentials.json). If this is intentional (e.g., the credential file must reside at a fixed path), the document should explain why.
    Remediation: Either change the GCP credentials dest to /sandbox/workspace/.gcp-credentials.json for consistency, or add a comment explaining why it uses a different base path.

Low

  • [terminology-consistency] docs/guides/user/building-custom-agents.md:121 — The diff adds a new frontmatter field role: my-agent without explanation. The "Key frontmatter fields" table does not document this field, leaving users without context for its purpose.
    Remediation: Add the role field to the "Key frontmatter fields" table with an explanation or cross-reference to ADR-0045.

Labels: PR modifies only documentation under docs/guides/.


Labels: PR modifies user-facing documentation under docs/guides/.

Previous run

Review

Findings

High

  • [API contract violation] docs/guides/user/building-custom-agents.md:350 — The post-script example calls gh issue comment "${ISSUE_KEY}" --repo "${REPO_FULL_NAME}" --body "${GREETING}", which requires issues: write permission on the GITHUB_TOKEN. However, the example workflow's permissions block only declares contents: read and id-token: write. The default GITHUB_TOKEN will lack permission to comment on issues, causing the gh issue comment call to fail with a 403 error at runtime.
    Remediation: Add issues: write to the permissions block in the example workflow.

Labels: PR modifies user-facing documentation under docs/guides/

Previous run (2)

Review

Findings

High

  • [runtime mechanism] docs/guides/user/building-custom-agents.md:502 — Variable name mismatch in the "Bringing your own identity" section. The variable is stored as MY_AGENT_APP_ID (gh variable set MY_AGENT_APP_ID), but referenced as MY_AGENT_APPID (${{ vars.MY_AGENT_APPID }}) in the git config email line. The missing underscore between "APP" and "ID" means vars.MY_AGENT_APPID will resolve to an empty string at runtime, producing an incorrect email address.
    Remediation: Change ${{ vars.MY_AGENT_APPID }} to ${{ vars.MY_AGENT_APP_ID }} in the git config email command.

  • [API contract violation] docs/guides/user/building-custom-agents.md:350 — The post-script example now includes gh issue comment "${ISSUE_KEY}" --repo "${REPO_FULL_NAME}" --body "${GREETING}", which requires issues: write permission. However, the example workflow's permissions block only declares contents: read and id-token: write. The GITHUB_TOKEN will lack permission to comment on issues, causing the gh issue comment call to fail with a 403 error at runtime.
    Remediation: Add issues: write to the workflow permissions block.

Medium

  • [internal-consistency] docs/guides/user/building-custom-agents.md:129 — The host_files destination paths are updated inconsistently. Three entries move from /tmp/workspace/ to /sandbox/workspace/, but the GCP credentials entry moves to /tmp/.gcp-credentials.json (not /sandbox/workspace/.gcp-credentials.json). If this is intentional (e.g., the credential file must reside at a fixed path), the document should explain why.
    Remediation: Either change the GCP credentials dest to /sandbox/workspace/.gcp-credentials.json for consistency, or add a comment explaining why it uses a different base path.

Low

  • [terminology-consistency] docs/guides/user/building-custom-agents.md:121 — The diff adds a new frontmatter field role: my-agent without explanation. The "Key frontmatter fields" table does not document this field, leaving users without context for its purpose.
    Remediation: Add the role field to the "Key frontmatter fields" table with an explanation or cross-reference to ADR-0045.

Labels: PR modifies only documentation under docs/guides/.

@github-actions

github-actions Bot commented Jun 25, 2026

Copy link
Copy Markdown

Site preview

Preview: https://27dd0193-site.fullsend-ai.workers.dev

Commit: d09a9780d62ece2627456925075accc65c0097d0

@fullsend-ai-review

fullsend-ai-review Bot commented Jun 25, 2026

Copy link
Copy Markdown

🤖 Finished Review · ❌ Failure · Started 6:45 AM UTC · Completed 6:55 AM UTC
Commit: fe90512 · View workflow run →

Comment thread docs/guides/user/building-custom-agents.md
@rh-hemartin rh-hemartin force-pushed the docs/custom-agents-byoi branch from fe90512 to 3466866 Compare June 26, 2026 06:29
Add "Bringing your own identity" section explaining how to use a custom
GitHub App instead of github-actions[bot]. Also improve the example agent
with a concrete greeting use case and fix sandbox paths.

Signed-off-by: Hector Martinez <hemartin@users.noreply.github.com>

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Signed-off-by: Hector Martinez <hemartin@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Jun 26, 2026

Copy link
Copy Markdown

🤖 Finished Review · ❌ Failure · Started 6:34 AM UTC · Completed 6:50 AM UTC
Commit: d09a978 · View workflow run →

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Doc: Add custom agent identity section to building-custom-agents guide

2 participants