cocoon-worker-release-latest is not reproducible from source — TEE_IMAGE_HASH mismatch blocks all new GPU hosts

[sponomarenko]

Summary

The published cocoon-worker-release-latest.tar.xz produces a TEE_IMAGE_HASH that is not registered in the production root contract. Building from the same source commit (9060ce5) using the documented reproducible build (./scripts/build-image prod) yields a third, different hash. As a result, no one running publicly-available code can pass the worker_hash_is_valid check on-chain.

Environment

• Server: Dell PowerEdge XE9680, 8x NVIDIA H200 SXM 141GB + 4x NVSwitch
• CPU: 2x Intel Xeon Platinum 8558 (Emerald Rapids, TDX-capable)
• Host OS: Ubuntu 25.10, kernel 6.17.0-19-generic
• QEMU: 10.1.0
• mkosi: 25.3
• Cocoon release: cocoon-worker-release-9060ce5.tar.xz (commit 9060ce5, Last-Modified Sat, 11 Apr 2026 18:29:07 GMT)
• TDX: Initialized (virt/tdx: module initialized, kvm_intel.tdx=Y)
• GPU CC mode: Enabled on all 8 H200s

All GPU/TDX/Intel SGX-DCAP/PCCS infrastructure is functional. RA-TLS certificates generate correctly, seal-server attestation passes, sglang inference server boots and serves test requests.

Reproduction

Followed README.md exactly:

wget https://ci.cocoon.org/cocoon-worker-release-latest.tar.xz
tar xf cocoon-worker-release-latest.tar.xz
cd release-9060ce5
./bin/seal-server --enclave-path ./bin/enclave.signed.so &
cp worker.conf.example worker.conf
# edit worker.conf with our owner_address, gpu, model, hf_token, etc.
./scripts/cocoon-launch worker.conf

worker.conf:

[node]
type = worker
model = Qwen/Qwen3-32B@9216db5781bf21249d130ec9da846c4624c16137:a3782505b11f4cb89c220933e80c151969965ef67509bc98dd2fa20daa166728
owner_address = UQDkVjgpqzj8OuOu4_8d88vyGMrh7lFBtMqUYSJIEy82od9d
gpu = 0000:19:00.0
node_wallet_key = ...
hf_token = ...
ton_config = /opt/cocoon/release-9060ce5/mainnet.cocoon.global.config.json
root_contract_address = EQCns7bYSp0igFvS1wpb5wsZjCKCV19MD5AVzI4EyxsnU73k

Observed behavior

cocoon-worker-runner.service crashes with:

worker-runner[1651]: Check `c->root_contract_config->has_worker_hash(local_image_hash_unverified_)` failed

Computed TEE_IMAGE_HASH:

EM9UIhzi5nT5bAyN+OfQ5ctLf9e2/m7fUsf+7qQBmHs=
hex: 10cf54221ce2e674f96c0c8df8e7d0e5cb4b7fd7b6fe6edf52c7feeea401987b

On-chain check via TON Center:

curl -s "https://toncenter.com/api/v3/runGetMethod" -X POST -H "Content-Type: application/json" -d '{
  "address":"EQCns7bYSp0igFvS1wpb5wsZjCKCV19MD5AVzI4EyxsnU73k",
  "method":"worker_hash_is_valid",
  "stack":[{"type":"num","value":"7603323586114790364894868202345832479141369465407458134401589376456734120059"}]
}'
# {"gas_used":2417,"exit_code":0,"stack":[{"type":"num","value":"0x0"}]}

0x0 = not in the whitelist.

Reproducible build investigation

Following the ## Reproducible Build section of README.md:

git clone https://github.com/TelegramMessenger/cocoon.git
cd cocoon
git submodule update --init --recursive
./reprodebian/fetch-deps.sh
./scripts/build-image prod

Build completed successfully. Comparing SHA-256 of generated images vs. the public tarball:

File	Public release	Built from source	Match
OVMF.fd	7e2417e6...	7e2417e6...	✅
image.vmlinuz	f34f19b9...	f34f19b9...	✅
image.initrd	24181782...	24181782...	✅
image.cmdline	c5945795...	922c71c1...	❌
image.img	f6990698...	2aa1ba8f...	❌

Full hashes:

Public release

7e2417e61d894e8733250f23ef35fa00197846dad33e33dbbf4698613c107d02  OVMF.fd
f34f19b9c7f4b310fd3ff2948fbfdcecbfb2abf9eca32b1812d3f1e130581728  image.vmlinuz
24181782e6045b242cb31ffbb91b698314debdfdea5244d0faf044e27518edbd  image.initrd
c59457958b55c4e657b2cc052fb42f9d54bab4997984942c4064784d6adfa9d7  image.cmdline
f6990698a9778b8691cd5ed4d616aebe465173ab4a0acb5453f79a121230a425  image.img

Built from source (commit 9060ce5)

7e2417e61d894e8733250f23ef35fa00197846dad33e33dbbf4698613c107d02  OVMF.fd
f34f19b9c7f4b310fd3ff2948fbfdcecbfb2abf9eca32b1812d3f1e130581728  image.vmlinuz
24181782e6045b242cb31ffbb91b698314debdfdea5244d0faf044e27518edbd  image.initrd
922c71c1481ee2e2fc89a73e7c1edd2dd9dea5092fdbdeeb27d49fa984127c91  image.cmdline
2aa1ba8f10c7fad1fe2a63499b2a8936074c2eca3309b358be71cfd1bb138448  image.img

Diff in image.cmdline is only the roothash= parameter (which is the dm-verity hash of image.img):

- roothash=13e35597b3d085fc56c08c57d2877f26b1ef9a73b720a6f7ff134cfab0241c37 ...
+ roothash=e09d4244b88efce344265367aee8046011d3be0faa0b27ea628ab6bb870ee679 ...

So image.img is the only file that genuinely differs. Everything else (kernel, initrd, OVMF) builds bit-identical.

mkosi.conf pins everything you'd expect: Snapshot=20260408T203914Z, SourceDateEpoch=0, all NVIDIA packages pinned to 595.58.03-1, kernel pinned to 6.19.10-1~bpo13+1, mkosi commit pinned. Submodule commits are tracked. The non-reproducibility of image.img appears to be a bug in the build pipeline.

Impact

There are currently 3 distinct TEE_IMAGE_HASH values for the "same" release-9060ce5:

1. Hash A — what the Cocoon CI built and registered on-chain (unknown to us)
2. Hash B — EM9UIhzi5nT5bAyN+OfQ5ctLf9e2/m7fUsf+7qQBmHs= — what running the published tarball produces
3. Hash C — whatever local mkosi rebuild produces — what ./scripts/build-image prod produces today

Only Hash A is in the on-chain whitelist (last addWorkerType was fa928fc1b9d5945eec827efdafcc5c79052108e51dd11e98aac2b4204c09328c on 2026-04-12 07:14 UTC, which doesn't match B or C).

This means: no GPU host onboarding from the published artifacts works right now. The full attestation/cryptography stack works perfectly; the only failing check is worker_hash_is_valid against the root contract.

Suggested fixes

Any one of these would unblock onboarding:

1. Fix the reproducibility of image.img so that ./scripts/build-image prod from a tagged commit produces the exact same bytes as cocoon-worker-release-<commit>.tar.xz, and ensure the registered on-chain hash matches that build.
2. Register Hash B on-chain (the hash of the currently-published tarball: EM9UIhzi5nT5bAyN+OfQ5ctLf9e2/m7fUsf+7qQBmHs= / 0x10cf54221ce2e674f96c0c8df8e7d0e5cb4b7fd7b6fe6edf52c7feeea401987b). This will at least let anyone running the public release work.
3. Publish SHA256SUMS for each release on ci.cocoon.org and document the canonical TDX measurement so hosts can verify what they're running matches what's registered.
4. Add a CI step that, for every release, computes TEE_IMAGE_HASH from the produced artifacts and includes it in the release manifest.

Happy to share build logs, attestation reports, or run any further experiments. Thanks for the work on Cocoon, once the registration path is unblocked, this server will join the network.

@ubique @arseny30

[sponomarenko]

/cc @vdtk — you have a live worker, can you confirm whether your image hash is on-chain?

[teterivloover]

@sponomarenko Hi, I'm not sure what the issue exactly is, but we had 2 workers running on Intel TDX + H200 at the time when you created the issue, and they were performing fine. Currently we're trying to set up another worker on H100 + AMD EPYC 9124, and the issues/errors we're experiencing are exactly like the ones you've described. Even though it is not mentioned on official https://cocoon.org/gpu-owners, the worker also supports TEEs via AMD SEV-SNP - https://github.com/TelegramMessenger/cocoon/blob/main/docs/amd-tee.md , but we keep getting a similar issue 'worker-runner[1651]: Check c->root_contract_config->has_worker_hash(local_image_hash_unverified_) failed'.

Old workers run fine though, which is pretty odd, considering that they use the same worker image. Devs are not very talkative either and apart from @arseny30 i haven't found a customer support or something similar. Also the list of available models mentioned in the https://github.com/TelegramMessenger/cocoon/blob/main/docs/deployment.md or https://cocoon.wiki is misleading. A month ago the only model that was actively served by workers/used by clients was Seed-X-PPO-7B, which is not even mentioned in the docs. Now the available ones are Qwen/Qwen3-32B and Qwen/Qwen3-8B, which we were able to verify by connecting to Cocoon from client side. But only Qwen3-32B is being served by 13 workers.

Hope you find this info helpful.

[ubique]

@teterivloover Please contact me regarding AMD SEV-SNP issues.

[teterivloover]

@ubique what's the best way to contact you? me@ubique.spb.ru?

[sponomarenko]

Hi @ubique @arseny30 — adding our voice. Same has_worker_hash(local_image_hash_unverified_) failed on 8× H200 SXM + Intel TDX back in April. We verified the build is deterministic (two independent rebuilds, bit-identical image.img), so it's clearly the on-chain whitelist side, not a build problem.

We're still ready to bring this hardware online if there's a path forward. Two questions:

Is there a way to get our worker hash added to the root contract config? If yes — what's the process / who to contact / what info do you need from us?
If not currently accepting new workers — please just tell us, so we can stop spinning hardware. No hard feelings, we just need to know.
Happy to share build hashes, attestation logs, anything that helps. Either answer is fine — just looking for a definitive yes/no.

[ubique]

@sponomarenko for sure this will be solved soon, I will post an update when. Working on it.

[ubique]

@sponomarenko, @teterivloover could you please try worker from dev branch?

[teterivloover]

Our worker has sucessfully started today, when built from the dev branch.

[sponomarenko]

Thanks @ubique — we did build the worker from the dev branch (541432c).

Setup: Dell XE9680, 8×H200, Intel TDX. Everything starts — TDX VM boots, seal-server + disk encryption work, TON syncs, the root contract is read — but worker-runner aborts at the image-hash check:

WorkerRunner.cpp:268 has_worker_hash(local_image_hash_unverified_) failed
local image hash: 7BF95D5F90F54BB66566EAC2D627749A2A56619C4DB00904E1CED5FDC7C349DC
This hash isn't among the worker hashes in the root contract. We also tried the published cocoon-worker-release-latest.tar.xz (9060ce5, Apr 11) — its hash isn't whitelisted either.

@teterivloover — which dev commit did you build? Did you build the TDX image yourself with build-image prod, or use a prebuilt one? What local image hash did your worker report?

@ubique — is the dev 541432c image hash registered in the root contract? Should a self-built build-image be bit-reproducible enough to match the on-chain hash, or is there a canonical image we should use?

Happy to share full logs or provide SSH access to debug directly.

[ubique]

@sponomarenko Please send the full log of startup to me@ubique.spb.ru . Also attach output of the sev-info --get-extended-report 0000000000000000000000000000000000000000000000000000000000000000 cmd to the email. Thanks.

[sponomarenko]

@ubique done

[teterivloover]

@sponomarenko we've built from the last commit on dev branch from source. After talking to Ubique, we've managed to get it working, but I'm not sure exactly what the issue was, but now our hashes are on chain.

[sponomarenko]

@teterivloover — congrats, this is great news! And thanks for letting us know.

I'm currently going through the same struggle with Ubique (multiple
diff iterations, latest TEE_IMAGE_HASH for my build is
c0e4d1b5e20b1d843ef90e462d319b0dd2b19ed3c82c01ef07985695d8004912 — not
on chain).

Could you share what worked for you, even if you don't remember the
exact root cause? Specifically:

1. Build host environment — which Linux distro and version did you
   use? (Ubuntu 24.04? Debian Trixie? Something else?)
2. gcc / mkosi versions in your build sandbox.
3. mkosi.conf Snapshot= date you ended up using (the repo currently
   has 20260408T203914Z).
4. Steps beyond what's in reprodebian/README.txt — anything you
   had to do manually that the docs don't mention?
5. rocksdb_build_date in your final key-manager-runner — is it
   1970-01-01 00:00:00 or the real GIT_DATE?

Even partial info would help. Happy to share anything from my side.

Thanks!