Nightly E2E workflow failing: E2E_API_TOKEN secret expired (project 5946, since 2026-06-27)

[padak]

Summary

The nightly E2E workflow (.github/workflows/e2e.yml, cron 17 3 * * *) has been failing every night with an authentication error at test setup. The immediate, blocking root cause is that the E2E_API_TOKEN repository secret holds an expired/invalid Storage API token for the dedicated E2E project (5946). Every credentialed test aborts during setup, so the nightly signal is currently useless.

This needs a maintainer action: mint a fresh Storage token for project 5946 and update the E2E_API_TOKEN repository secret. A token cannot be rotated by an automated agent — it is strictly a manual maintainer step.

Evidence

Latest nightly run 29890919132 (2026-07-22), from gh run view --log-failed:

"code": "INVALID_TOKEN",
"error_type": "authentication",
"message": "Invalid or expired token (token: 5946-...v0jX): Invalid access token",

The failure is in the autouse setup fixture's project add (e.g. tests/test_e2e.py:4120), so it cascades to every credentialed test class (TestFullE2E, TestE2ESyncWorkflow, TestE2EFlowOperations, ...). The latest run shows 203 Invalid access token hits.

Failure window / when the token broke

The token invalidation began on 2026-06-27. Sampling Invalid access token occurrences per scheduled run (1 = the single intentional TestE2EErrorHandling::test_add_with_invalid_token, which passes; 180+ = the token is dead and every credentialed test fails at setup):

Date	Invalid access token hits	Meaning
2026-06-25	1	token healthy (only the intentional test)
2026-06-26	1	token healthy
2026-06-27	188	token invalidated — every credentialed test fails at setup
2026-07-03	194	still dead
2026-07-13	194	still dead
2026-07-22	203	still dead

So the token likely expired or was revoked on ~2026-06-27 (a ~30-day Storage token minted late May would fit the timing).

Note: the workflow was red before the token broke too

For full accuracy: the nightly has actually been red further back than 2026-06-27 (no green scheduled run in the last 100 runs, back to 2026-06-05), but for a different, secondary reason. With a valid token (e.g. run 28217682364, 2026-06-26), the failures were individual test-body assertions — swap-schema adoption, file-list visibility, stream OTLP, etc.:

AssertionError: After swap, 'in.c-...swap.original' should adopt the schema of '...typed_copy' (VARCHAR(80)); got VARCHAR(20).
AssertionError: uploaded file must appear in list

Those are read-after-write / eventual-consistency-style flakes at the test level. Two of this family have already been fixed on the PR #516 branch (the _test_mcp_parity_commands step that could never pass since 0.73.0, and an add/delete-column read-after-DDL race, now polled). The token is the blocking issue to fix first — until it is rotated, none of these downstream test bodies are even reached, so we cannot see whether they still fail.

Fix (maintainer, manual)

1. Mint a fresh Storage API token for the dedicated E2E project:
   • Project 5946 ("kbagent E2E"), stack connection.us-east4.gcp.keboola.com.
   • This is a throwaway project whose data the suite creates/deletes, so a broad token is fine.
   • Consider a non-expiring token (or a calendar reminder to rotate) so this does not silently recur.
2. Update the repository secret: Settings → Secrets and variables → Actions → E2E_API_TOKEN. Confirm E2E_URL is still connection.us-east4.gcp.keboola.com.
3. Re-run the workflow from the Actions tab (workflow_dispatch) to confirm setup passes.

Verification that only the CI secret is stale

Local full-suite runs against the same project 5946 pass the setup / project add step and run to completion (config-dir mode, reading a locally-registered token via KBAGENT_E2E_CONFIG_DIR + KBAGENT_E2E_ALIAS, as documented in CONTRIBUTING.md). So the project and stack are healthy — only the token stored in the CI secret is stale.

---

Filed after investigating during PR #516. Root cause, failure window, and the local-vs-CI distinction verified 2026-07-22 via gh run view --log-failed.

[padak]

Token rotation confirmed working. Run 29911337559 after the secret update:

• Setup (project add) now passes — the only Invalid access token occurrence is the intentional TestE2EErrorHandling::test_add_with_invalid_token, i.e. the healthy signal (was 200+ before).
• 111 passed, 4 failed, 11 skipped — the credentialed suite runs end-to-end again.

So the root cause this issue documented (expired E2E_API_TOKEN secret, dead since 2026-06-27) is fixed.

Four pre-existing, non-token failures remain — they predate the token break and are unrelated to the v0.75.0 release; the swap-schema one is visible in the 2026-06-26 nightly (run 28217682364), when the token was still valid:

Test	Symptom
TestE2EStorageSwapTables::test_swap_in_dev_branch_exchanges_schemas	assert '20' == '80' — post-swap schema not adopted in dev branch
TestE2EStorageCloneTable::test_clone_prod_table_into_dev_branch	clone-table returns status: error (exit 1)
test_stream_otlp_e2e	source create returns status: error
TestE2EConfigSecretEncryption::test_config_create_and_update_encrypt_secret	KeyError: 'configuration' (response shape)

These look project/environment-specific (storage-branches behavior on the Snowflake E2E project, response-shape drift) rather than product regressions, but they keep the nightly red and warrant their own tracking.

[padak]

Closing: the root cause here (expired E2E_API_TOKEN secret) is fixed and confirmed by run 29911337559 — setup passes, 111/115 green. The 4 remaining pre-existing test failures are a separate concern and will be tracked + fixed on their own; they are not this token issue. Thanks for rotating the secret 🙏