Skip to content

CI: make changelog automation LLM-based (Gemini) and race-safe#1082

Open
Gui-FernandesBR wants to merge 3 commits into
developfrom
enh/llm-changelog-automation
Open

CI: make changelog automation LLM-based (Gemini) and race-safe#1082
Gui-FernandesBR wants to merge 3 commits into
developfrom
enh/llm-changelog-automation

Conversation

@Gui-FernandesBR

@Gui-FernandesBR Gui-FernandesBR commented Jul 22, 2026

Copy link
Copy Markdown
Member

Problem

The post-merge Populate Changelog job (.github/workflows/changelog.yml) used a blind label→prefix mapping inline in the workflow. It:

  • Double-prefixed titles that already carried a conventional prefix — e.g. PR BUG/MNT: pre-release v1.13.0 review fixes #1074 (BUG/MNT: ...) landed as ENH: BUG/MNT: pre-release v1.13.0 review fixes (still visible in the v1.13.0 section).
  • Never deduplicated — a re-run inserted the same entry twice.
  • Only ever used Added / Changed / Fixed, never Deprecated / Removed / Security.
  • Pushed without a rebase, so it lost races against concurrent merges into develop (the "sometimes it just doesn't work").

What changed

.github/scripts/update_changelog.py — the entry is now formatted and placed by Gemini (model alias gemini-flash-latest), behind a deterministic safety net:

  • Only the [Unreleased] block is ever rewritten; released history is preserved byte-for-byte.
  • Model output is validated before use: every existing [#N](url) link must survive, the new PR must be referenced exactly once, no released ## [vX] header may leak in, and growth is bounded.
  • On any failure — API error, rejected output, or missing GEMINI_API_KEY — it falls back to a safe deterministic insert that detects an existing prefix (never producing ENH: BUG: ...) and skips duplicates.
  • Idempotent: a PR already referenced in [Unreleased] is a no-op.

changelog.yml — installs google-genai, passes PR_BODY + GEMINI_API_KEY, and the push step now rebases and retries (5×) to survive concurrent merges.

.github/scripts/test_update_changelog.py — 13 self-contained tests of the deterministic pieces (split/reassemble, dedup, anti-double-prefix, canonical section ordering, output validator). They run without network or the google-genai package. ruff check and ruff format --check are clean.

Docsdocs/development/first_pr.rst, docs/development/style_guide.rst, and .github/pull_request_template.md now state that contributors do not edit CHANGELOG.md: the LLM writes the entry after merge, so you just open and merge PRs.

⚠️ Action required: add the GEMINI_API_KEY repository secret

Before this workflow can use Gemini, a maintainer must add the API key as a repository secret:

Settings → Secrets and variables → Actions → New repository secret — name GEMINI_API_KEY.

Without it the job still runs via the deterministic fallback (no LLM). The key is only ever read from secrets.GEMINI_API_KEY; it is never hardcoded or logged.

Notes

  • Model choice: gemini-2.5-flash is now blocked for new API keys (returns 404 "no longer available to new users"), so the alias gemini-flash-latest is used to track the newest stable flash model and avoid future breakage.
  • The stray ENH: BUG/MNT: ... line already committed in the released v1.13.0 section is left untouched (released history); can be cleaned up separately if desired.
  • If two PRs edit the top of the same subsection simultaneously, the rebase may conflict and the job fails loudly (::error::) rather than corrupting the file — manual fixup needed.

🤖 Generated with Claude Code

@Gui-FernandesBR
Gui-FernandesBR requested a review from a team as a code owner July 22, 2026 01:07
@Gui-FernandesBR

Copy link
Copy Markdown
Member Author

⚠️ Action required: add the `GEMINI_API_KEY` repository secret

Before this workflow can use Gemini, a maintainer must add the API key as a repository secret:

Settings → Secrets and variables → Actions → New repository secret

  • Name: `GEMINI_API_KEY`
  • Value: (the Gemini API key)

Until the secret exists, the job still runs — it just falls back to the deterministic insert (no LLM). The key is only ever read from `secrets.GEMINI_API_KEY`, passed to the script as an env var; it is never hardcoded or printed to logs.

@codecov

codecov Bot commented Jul 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.36%. Comparing base (e0ff281) to head (c524b92).
⚠️ Report is 13 commits behind head on develop.

Additional details and impacted files
@@             Coverage Diff             @@
##           develop    #1082      +/-   ##
===========================================
+ Coverage    82.18%   82.36%   +0.18%     
===========================================
  Files          122      122              
  Lines        16355    16377      +22     
===========================================
+ Hits         13441    13489      +48     
+ Misses        2914     2888      -26     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Gui-FernandesBR and others added 3 commits July 21, 2026 22:36
The post-merge changelog job used a blind label->prefix mapping that
double-prefixed titles already carrying one (e.g. "ENH: BUG/MNT: ..."),
never deduplicated, only ever used Added/Changed/Fixed, and pushed
without a rebase (so it lost races with concurrent merges).

Replace the inline script with .github/scripts/update_changelog.py:
- Gemini (gemini-2.5-flash) formats and places the entry in the right
  subsection, reusing an existing prefix and avoiding duplicates.
- Only the [Unreleased] block is ever rewritten; released history is
  preserved byte-for-byte.
- Model output is validated (all existing links kept, new PR referenced
  exactly once, no leaked version header, bounded growth). On failure or
  a missing GEMINI_API_KEY it falls back to a safe deterministic insert
  that detects existing prefixes and skips duplicates.
- Idempotent: a PR already in [Unreleased] is a no-op.
- Push now rebases and retries to survive concurrent merges.

Add self-contained tests (.github/scripts/test_update_changelog.py)
covering the deterministic pieces; they run without network or the
google-genai package.

Requires a GEMINI_API_KEY repository secret; without it the job still
works via the deterministic fallback.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A live test showed gemini-2.5-flash returns 404 "no longer available to
new users" for freshly created API keys, which would send every run to
the deterministic fallback. Switch to the gemini-flash-latest alias so
the job tracks the newest stable flash model and does not break when a
pinned version is retired. Output validation still guards any drift.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Contributors no longer edit CHANGELOG.md by hand. Update the first-PR
guide, style guide, and PR template to make clear that after a PR is
merged an LLM workflow writes the entry automatically, so you only open
and merge PRs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Gui-FernandesBR
Gui-FernandesBR force-pushed the enh/llm-changelog-automation branch from 1ce6f94 to c524b92 Compare July 22, 2026 01:36
@autoweave-bot

Copy link
Copy Markdown

Ooh, nice work automating changelogs – looks super solid! 🤩
Explore here →

Subweave map of rocketpy-team/rocketpy#1082

Maintainer? Turn off weaves from non-maintainers →
Carefully crafted by Subweave · 🧶 used ~78k LLM tokens

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants