Symptom
Keyless integration tests on every PR running them now fail with:
```
Certificate pinning failed: Certificate pin mismatch for fulcio.sigstore.dev:
got e30da317897121cb..., expected one of 2 configured pins
```
Surfaced on PR #115 (regorus bump — unrelated change), but the root cause is pre-existing on main. Fulcio rotated its certificate sometime between when the pin set was last updated (`2026-04-14` per code comment in `src/lib/src/signature/keyless/cert_pinning.rs:88`) and now, and the new leaf SPKI fingerprint is not in our 2-pin allowlist.
Why it bites now
This isn't a regression from the regorus bump — the Fulcio rotation is independent of any sigil change. It became visible because:
- The keyless integration test path is gated to actually run only on PRs that touch certain files. Many recent PRs (Cerisier docs, v0.8.2 release commit, audit fix-PRs) didn't trigger it.
- Today's run on a Cargo.toml/Cargo.lock change did.
Composition with audit C-4 (issue #95)
Audit finding C-4 documented that SPKI pinning today is warn-only because `ureq` doesn't expose a custom `ServerCertVerifier`. The pin mismatch is logged but the TLS handshake still completes. That's why this finding shows up as a test failure (the test asserts the mismatch should fail closed) rather than a production outage.
If #95 (migrate `ureq` → rustls-direct, enforce pinning at the TLS layer) lands first, this Fulcio rotation would have hard-broken every keyless signing operation on every consumer. Today it just breaks the test.
This is an argument for adding pin-rotation monitoring to the operations playbook before enforcement lands.
Resolution options
-
Update the pin set with the new leaf SPKI. Concrete steps:
```sh
echo | openssl s_client -connect fulcio.sigstore.dev:443 -servername fulcio.sigstore.dev 2>/dev/null \
| openssl x509 -pubkey -noout \
| openssl pkey -pubin -outform DER \
| openssl dgst -sha256 -binary \
| xxd -p -c 256
```
Verify the result independently (e.g. compare with `https://crt.sh/?q=fulcio.sigstore.dev\` or contact Sigstore maintainers), then append to the pin set in `src/lib/src/signature/keyless/cert_pinning.rs::fulcio_production()`.
-
Add an out-of-band monitoring job that runs the above command nightly and opens an issue if the observed fingerprint isn't in the configured pin set. Catches rotations before they break production.
-
Document the rotation cadence in a runbook so this isn't a surprise every time.
Out-of-scope here (tracked separately)
Repro
`gh run view --job 75605605068 --log-failed` on PR #115's CI run.
Related
Symptom
Keyless integration tests on every PR running them now fail with:
```
Certificate pinning failed: Certificate pin mismatch for fulcio.sigstore.dev:
got e30da317897121cb..., expected one of 2 configured pins
```
Surfaced on PR #115 (regorus bump — unrelated change), but the root cause is pre-existing on main. Fulcio rotated its certificate sometime between when the pin set was last updated (`2026-04-14` per code comment in `src/lib/src/signature/keyless/cert_pinning.rs:88`) and now, and the new leaf SPKI fingerprint is not in our 2-pin allowlist.
Why it bites now
This isn't a regression from the regorus bump — the Fulcio rotation is independent of any sigil change. It became visible because:
Composition with audit C-4 (issue #95)
Audit finding C-4 documented that SPKI pinning today is warn-only because `ureq` doesn't expose a custom `ServerCertVerifier`. The pin mismatch is logged but the TLS handshake still completes. That's why this finding shows up as a test failure (the test asserts the mismatch should fail closed) rather than a production outage.
If #95 (migrate `ureq` → rustls-direct, enforce pinning at the TLS layer) lands first, this Fulcio rotation would have hard-broken every keyless signing operation on every consumer. Today it just breaks the test.
This is an argument for adding pin-rotation monitoring to the operations playbook before enforcement lands.
Resolution options
Update the pin set with the new leaf SPKI. Concrete steps:
```sh
echo | openssl s_client -connect fulcio.sigstore.dev:443 -servername fulcio.sigstore.dev 2>/dev/null \
| openssl x509 -pubkey -noout \
| openssl pkey -pubin -outform DER \
| openssl dgst -sha256 -binary \
| xxd -p -c 256
```
Verify the result independently (e.g. compare with `https://crt.sh/?q=fulcio.sigstore.dev\` or contact Sigstore maintainers), then append to the pin set in `src/lib/src/signature/keyless/cert_pinning.rs::fulcio_production()`.
Add an out-of-band monitoring job that runs the above command nightly and opens an issue if the observed fingerprint isn't in the configured pin set. Catches rotations before they break production.
Document the rotation cadence in a runbook so this isn't a surprise every time.
Out-of-scope here (tracked separately)
Repro
`gh run view --job 75605605068 --log-failed` on PR #115's CI run.
Related