From 05647b9f20028f17d5ab7e316798262f54abe06f Mon Sep 17 00:00:00 2001 From: violetbrina <9669990+violetbrina@users.noreply.github.com> Date: Mon, 13 Jul 2026 16:05:05 +1000 Subject: [PATCH 1/3] docs(hail): troubleshoot dev deploy "Unauthorized" failures Add a troubleshooting subsection to the Developer deploy section explaining that a 401 "You must be logged in to the server (Unauthorized)" in dev deploy jobs (first seen in create_test_database_server_config, recurring in create_ssl_config_hail_root) is caused by GKE invalidating the legacy admin-token service-account token. Includes local read-only diagnosis steps and the fix (delete + recreate the admin-token secret). Co-Authored-By: Claude Opus 4.8 --- hail.md | 89 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 89 insertions(+) diff --git a/hail.md b/hail.md index eee0474..658eac9 100644 --- a/hail.md +++ b/hail.md @@ -366,6 +366,95 @@ apt update && apt install -y mysql-client mysql --defaults-file=/sql-config/sql-config.cnf ``` +### Troubleshooting: dev deploy fails with `Unauthorized` + +If a dev deploy fails with an error like: + +``` +error: You must be logged in to the server (Unauthorized) +``` + +this is usually **not** a problem with the step that reports it — it means the +Kubernetes token the deploy jobs use to talk to the cluster has stopped working, so +every `kubectl` call in the deploy fails. + +You'll typically see it first in the `create_test_database_server_config` job (the +earliest step that runs `kubectl`), on its very first command: + +``` +Running: SECRET_JSON=$(kubectl get secret database-server-config -n $NAMESPACE -o json) +error: You must be logged in to the server (Unauthorized) +``` + +and it recurs in later `kubectl` steps such as `create_ssl_config_hail_root`: + +``` +error: failed to create secret Unauthorized +error: You must be logged in to the server (Unauthorized) +``` + +#### Root cause + +Deploy steps that run `kubectl` (those with a `serviceAccount:` in `build.yaml`) +authenticate as the `admin` service account in your namespace. The CI driver reads +the token from the `admin-token` secret in your namespace, builds a kubeconfig from +it, and mounts it into the job. + +That `admin-token` is a **legacy service-account token**. GKE progressively +invalidates legacy service-account tokens that haven't been used for a while (the +`LegacyServiceAccountTokenCleanUp` behaviour), so after a gap between deploys the +stored token stops authenticating and every `kubectl` call returns `401 Unauthorized`. +Re-running the deploy doesn't help: the `default_ns` step applies the secret with +`kubectl apply`, which won't regenerate the token of an already-existing secret. + +#### Diagnose locally + +Point `kubectl` at the `vdc` cluster and test the stored token directly. These are +all read-only checks: + +```bash +export NAMESPACE= +gcloud container clusters get-credentials vdc --location=australia-southeast1-b + +# The service account, RBAC and secret usually all still exist and look fine: +kubectl -n $NAMESPACE get sa admin +kubectl -n $NAMESPACE get secret admin-token +kubectl -n $NAMESPACE get role,rolebinding + +# The tell-tale check — try to authenticate as admin using the *stored* token: +TOKEN=$(kubectl -n $NAMESPACE get secret admin-token -o jsonpath='{.data.token}' | base64 -d) +kubectl auth whoami --token="$TOKEN" +``` + +If that last command prints `error: You must be logged in to the server (Unauthorized)` +instead of `system:serviceaccount:$NAMESPACE:admin`, the stored token is the problem. + +#### Fix + +Delete and recreate the `admin-token` secret so the token controller mints a fresh, +valid token: + +```bash +kubectl -n $NAMESPACE delete secret admin-token +kubectl -n $NAMESPACE apply -f - < Date: Mon, 13 Jul 2026 17:11:42 +1000 Subject: [PATCH 2/3] docs(hail): tighten Unauthorized troubleshooting section Collapse the duplicated error blocks, merge the root-cause explanation into a single paragraph, and drop the redundant read-only diagnostic checks so the section leads straight to the confirm-and-fix steps. Co-Authored-By: Claude Opus 4.8 --- hail.md | 64 ++++++++++----------------------------------------------- 1 file changed, 11 insertions(+), 53 deletions(-) diff --git a/hail.md b/hail.md index 658eac9..b94e86d 100644 --- a/hail.md +++ b/hail.md @@ -368,71 +368,31 @@ mysql --defaults-file=/sql-config/sql-config.cnf ### Troubleshooting: dev deploy fails with `Unauthorized` -If a dev deploy fails with an error like: +If deploy steps that run `kubectl` fail with: ``` error: You must be logged in to the server (Unauthorized) ``` -this is usually **not** a problem with the step that reports it — it means the -Kubernetes token the deploy jobs use to talk to the cluster has stopped working, so -every `kubectl` call in the deploy fails. +the problem usually isn't the reporting step. Those steps authenticate as the `admin` +service account using the token in your namespace's `admin-token` secret, which is a +**legacy service-account token**. GKE invalidates legacy tokens that haven't been used +for a while (`LegacyServiceAccountTokenCleanUp`), so after a gap between deploys the +stored token stops working and every `kubectl` call fails. Re-running won't fix it — +`kubectl apply` doesn't regenerate the token of an existing secret. -You'll typically see it first in the `create_test_database_server_config` job (the -earliest step that runs `kubectl`), on its very first command: - -``` -Running: SECRET_JSON=$(kubectl get secret database-server-config -n $NAMESPACE -o json) -error: You must be logged in to the server (Unauthorized) -``` - -and it recurs in later `kubectl` steps such as `create_ssl_config_hail_root`: - -``` -error: failed to create secret Unauthorized -error: You must be logged in to the server (Unauthorized) -``` - -#### Root cause - -Deploy steps that run `kubectl` (those with a `serviceAccount:` in `build.yaml`) -authenticate as the `admin` service account in your namespace. The CI driver reads -the token from the `admin-token` secret in your namespace, builds a kubeconfig from -it, and mounts it into the job. - -That `admin-token` is a **legacy service-account token**. GKE progressively -invalidates legacy service-account tokens that haven't been used for a while (the -`LegacyServiceAccountTokenCleanUp` behaviour), so after a gap between deploys the -stored token stops authenticating and every `kubectl` call returns `401 Unauthorized`. -Re-running the deploy doesn't help: the `default_ns` step applies the secret with -`kubectl apply`, which won't regenerate the token of an already-existing secret. - -#### Diagnose locally - -Point `kubectl` at the `vdc` cluster and test the stored token directly. These are -all read-only checks: +Confirm the stored token is dead: ```bash export NAMESPACE= gcloud container clusters get-credentials vdc --location=australia-southeast1-b -# The service account, RBAC and secret usually all still exist and look fine: -kubectl -n $NAMESPACE get sa admin -kubectl -n $NAMESPACE get secret admin-token -kubectl -n $NAMESPACE get role,rolebinding - -# The tell-tale check — try to authenticate as admin using the *stored* token: TOKEN=$(kubectl -n $NAMESPACE get secret admin-token -o jsonpath='{.data.token}' | base64 -d) kubectl auth whoami --token="$TOKEN" ``` -If that last command prints `error: You must be logged in to the server (Unauthorized)` -instead of `system:serviceaccount:$NAMESPACE:admin`, the stored token is the problem. - -#### Fix - -Delete and recreate the `admin-token` secret so the token controller mints a fresh, -valid token: +If this prints `Unauthorized` instead of `system:serviceaccount:$NAMESPACE:admin`, +delete and recreate the secret so a fresh token is minted: ```bash kubectl -n $NAMESPACE delete secret admin-token @@ -447,14 +407,12 @@ metadata: kubernetes.io/service-account.name: admin EOF -# Verify it now authenticates (should print system:serviceaccount:$NAMESPACE:admin): +# Verify (should now print system:serviceaccount:$NAMESPACE:admin), then re-run the deploy: sleep 3 TOKEN=$(kubectl -n $NAMESPACE get secret admin-token -o jsonpath='{.data.token}' | base64 -d) kubectl auth whoami --token="$TOKEN" ``` -Once `kubectl auth whoami` reports the `admin` service account, re-run your dev deploy. - ### Syncing local changes to pod Instead of manually dev-deploying for every change, you can synchronise your local changes with the k8s pod, using the `devbin/sync.py` script in the Hail repository. From bbec66e6a1c96555214bfc73c9a774ecdf963069 Mon Sep 17 00:00:00 2001 From: Sabrina Yan <9669990+violetbrina@users.noreply.github.com> Date: Mon, 13 Jul 2026 17:18:44 +1000 Subject: [PATCH 3/3] docs(hail): specify language on Unauthorized error fence MD040 requires fenced code blocks to declare a language. Co-Authored-By: Claude Opus 4.8 --- hail.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/hail.md b/hail.md index b94e86d..43a3300 100644 --- a/hail.md +++ b/hail.md @@ -370,7 +370,7 @@ mysql --defaults-file=/sql-config/sql-config.cnf If deploy steps that run `kubectl` fail with: -``` +```text error: You must be logged in to the server (Unauthorized) ```