feat: add gateway-api support by l0wl3vel · Pull Request #299 · metal-stack/mini-lab

l0wl3vel · 2026-05-06T12:20:19Z

Description

Add kind-cloud-controller-manager to provide Type: Loadbalancer services
Introduce envoy-gateway as the Gateway API implementation
Move metal-stack control plane kind cluster into the mini_lab_external docker network
- can't select container IP in the default docker bridge, which we need for the pre-defined *.nip.io DNS records
Kept ingress-nginx for now. Still required for Dex, Thanos, Gardener, PowerDNS

WIPs

Certificates are a bit messed up still (using default-gateway cert for grcp termination)
~~Link metal-roles pr branch to run ci in pull request~~ metal-roles PR is merged
CI is failing -> Fix route reconciliation by enabling fpm #315

Closes: #297

Requires: metal-stack/helm-charts#156 and metal-stack/metal-roles#594

Tested configurations

Sonic
Dell Sonic
Gardener (looks good, deploys correctly with metal-stack on GWAPI and Gardener components still on ingress-nginx)
Kamaji/capi-lab
- out of scope So likely a wontfix, unless it gets integreated into mini-lab. Only usable in capi-lab, which uses an old pinned version of mini-lab.

Noteworthy

- We now create a dedicated internal mini-lab docker network (mini_lab_internal) to enable explicit reproducible IP assignments for Service type: LoadBalancer. We reserve the 172.42.0.1/16 subnet for the mini_lab_internal network. mini-lab will manage the lifecycle of this network and will destroy it during `make down`.

- We made changes to the certificates. If you deployed mini-lab before you need to run `make roll-certs` to update host names in the certificates.

Used AI-Tools ✨

none

ma-hartma · 2026-05-11T07:36:52Z

Dell Sonic does actually work, but you need credentials to pull from r.metal-stack.io.

vknabel · 2026-05-28T13:47:37Z

Sadly I got the following error:

deploy-control-plane  | TASK [ansible-common/roles/helm-chart : Copy over custom helm charts] **********
deploy-control-plane  | fatal: [localhost]: FAILED! => 
deploy-control-plane  |     changed: false
deploy-control-plane  |     cmd: /usr/bin/rsync --delay-updates -F --compress --delete-after --archive --out-format='<<CHANGED>>%i
deploy-control-plane  |         %n%L' /helm-charts/charts/metal-control-plane /tmp/helm-chart
deploy-control-plane  |     msg: |-
deploy-control-plane  |         rsync: [sender] change_dir "/helm-charts/charts" failed: No such file or directory (2)
deploy-control-plane  |         rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.4.1]
deploy-control-plane  |     rc: 23

I had the following overrides metal_roles_version: gatewayapi

l0wl3vel · 2026-05-29T12:58:10Z

Sadly I got the following error:

deploy-control-plane  | TASK [ansible-common/roles/helm-chart : Copy over custom helm charts] **********
deploy-control-plane  | fatal: [localhost]: FAILED! => 
deploy-control-plane  |     changed: false
deploy-control-plane  |     cmd: /usr/bin/rsync --delay-updates -F --compress --delete-after --archive --out-format='<<CHANGED>>%i
deploy-control-plane  |         %n%L' /helm-charts/charts/metal-control-plane /tmp/helm-chart
deploy-control-plane  |     msg: |-
deploy-control-plane  |         rsync: [sender] change_dir "/helm-charts/charts" failed: No such file or directory (2)
deploy-control-plane  |         rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.4.1]
deploy-control-plane  |     rc: 23

I had the following overrides metal_roles_version: gatewayapi

@vknabel fixed in 629cb02

l0wl3vel · 2026-05-29T13:09:32Z

@Sven-Ric Would you mind taking a look at the network changes?

Sven-Ric · 2026-06-05T12:14:47Z

It seems like the kind node always ends up in the default kind network on a clean first run. The kind network is read from .env, which is written by env.sh. However the Makefile reads .env before env.sh is invoked and the kind node network falls back to default. Because .env is persistent the bug is masked on all subsequent runs.

On initial run:

# docker inspect metal-control-plane-control-plane
[
    {
        <SNIP>
        "NetworkSettings": {
            <SNIP>
            "Networks": {
                "kind": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "DriverOpts": null,
                    "GwPriority": 0,
                    "NetworkID": "6530b19e41b397d41d37f6a38d6b1bbd74c9ba2b7478df95f6a6270cc84c4d0e",
                    "EndpointID": "6d56f5f0fa83330b85e0b0ebbd04175a93d8586c48492c9b545beb7eeecce015",
                    "Gateway": "172.18.0.1",
                    "IPAddress": "172.18.0.2",
                    "MacAddress": "12:98:42:c8:e4:ec",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "fc00:f853:ccd:e793::1",
                    "GlobalIPv6Address": "fc00:f853:ccd:e793::2",
                    "GlobalIPv6PrefixLen": 64,
                    "DNSNames": [
                        "metal-control-plane-control-plane",
                        "bd976835cec0"
                    ]
                }
            }
        },
        "ImageManifestDescriptor": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "digest": "sha256:21c46cf61fd45873f89e6a1bfcba4b7904dffa84c2bec88aeeca9a0409af4725",
            "size": 743,
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        }
    }
]

On all subsequent runs:

# docker inspect metal-control-plane-control-plane
[
    {
        <SNIP>
        "NetworkSettings": {
            <SNIP>
            "Networks": {
                "mini_lab_internal": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "DriverOpts": null,
                    "GwPriority": 0,
                    "NetworkID": "2734b8f942cae84d8693ecd43ab3bb9d5cd71905faf992fbfe5c3df17ddc376b",
                    "EndpointID": "62f8b2a6eb379bb65f13f6441a9249417fc9ce754218a29b699cd7511b393d29",
                    "Gateway": "172.42.0.1",
                    "IPAddress": "172.42.0.2",
                    "MacAddress": "66:e7:b9:9c:2e:39",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "DNSNames": [
                        "metal-control-plane-control-plane",
                        "5b12fbbedfdc"
                    ]
                }
            }
        },
        "ImageManifestDescriptor": {
            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
            "digest": "sha256:21c46cf61fd45873f89e6a1bfcba4b7904dffa84c2bec88aeeca9a0409af4725",
            "size": 743,
            "platform": {
                "architecture": "amd64",
                "os": "linux"
            }
        }
    }
]

l0wl3vel · 2026-06-05T17:52:10Z

Thank you so much for checking it out @Sven-Ric. Fixed in baddf29. A few people checked this PR and it worked fine but CI was failing and I had no clue why. You saved me a lot of time 😅

Gerrit91

PR looks good to me. Thanks for the effort!

Would like to read the migration path somewhere. I guess for most operators it is sufficient to deploy a Gateway controller and then set metal_control_plane_gateway_dns: "{{ metal_control_plane_ingress_dns }} and when we remove the old metal_control_plane_ingress_dns then they need to replace the old variable in their deployment repository?

Still required for Dex, Thanos, Gardener, PowerDNS

Dex should not be used anymore, let's remove the role in another PR.

l0wl3vel · 2026-07-08T12:40:13Z

~~Tests currently do not pass until routing propagation fix lands: #315~~ done

To enable fixed IPs for the Gateway API, mini-lab needs a docker network that is not the docker default bridge. Create a dedicated `mini_lab_internal` network (172.42.0.0/16) and relocate everything off the old default network bridge (172.17.0.1 / 172.18.0.0). This shifts the control-plane ingress DNS from 172.17.0.1.nip.io to 172.42.0.1.nip.io. No Gateway API migration yet Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

Add a metal-api server cert (api./v2. nip.io SANs) for the Gateway to terminate TLS with, add it to roll_certs.sh, and update the grpc cert host to the Gateway LoadBalancer IP. Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

Add the Gateway API plumbing without wiring any application to it yet: Deploy cloud-provider-kind to provide LoadBalancer IPs. Add a `gateway` role deploying the Envoy Gateway controller, a GatewayClass, an EnvoyProxy pinned to the fixed LB IP 172.42.0.42, and the metal-control-plane Gateway. Register the role in the control-plane playbook and expose metal_control_plane_gateway_dns Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

Gerrit91

Looks really good now for me. Just some small remarks left, after this I would like to merge it.

I assume the ingress-controller is just there for Gardener exposal now? We can probably remove it in a subsequent PR and remove the ugly service status patching?

Gerrit91 · 2026-07-15T12:28:41Z

Ingress-controller NGINX should be updated to Envoy Gateway? :)

Gerrit91 · 2026-07-15T12:33:25Z

 zitadel_endpoint: zitadel.{{ metal_control_plane_namespace }}.svc.cluster.local
-zitadel_external_domain: auth.{{ metal_control_plane_ingress_dns }}
-zitadel_ingress_dns: https://{{ zitadel_external_domain }}:4443
+zitadel_external_domain: zitadel.{{ metal_control_plane_gateway_dns }}


Why exactly this renaming? We thought that "auth" would be a bit less specific because it also works if we exchange the auth provider with something else that supports OIDC.

Wanted to prevent confusion, as it is served via the same domain as the metalstack control plane api, while not being a metal-stack api component.

l0wl3vel · 2026-07-15T16:12:40Z

Looks really good now for me. Just some small remarks left, after this I would like to merge it.

I assume the ingress-controller is just there for Gardener exposal now? We can probably remove it in a subsequent PR and remove the ugly service status patching?

Thx, happy to hear that.

Gardener, PowerDNS, Minio and Thanos come to mind. There is already a note in the docs, that some dependencies still require an ingress controller and replacing ingress nginx with a supported option is recommended. Going to look into what options are suitable for replacing ingress-nginx next.

nginx follow up issue: #316

Migrate zitadel, nsq, metal-api and metal-apiserver off ingress-nginx to Gateway API. Enable http/tcp routes, point their external URLs at the Gateway LoadBalancer (172.42.0.42 / gateway_dns). Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

as we are not listening on the host we are free to use any port we like. So we switch to the default ports for https and http Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

we need to put the CA data into an env var to later access metal-api. So we run cert generation on before calling the requested make target if certs are not generated yet. Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

metal-robot Bot added this to Development May 6, 2026

l0wl3vel force-pushed the feat/gatewayapi branch 2 times, most recently from 28079c5 to f84c000 Compare May 8, 2026 14:58

l0wl3vel mentioned this pull request May 26, 2026

Use Sonic VPP image #301

Draft

9 tasks

l0wl3vel requested review from Sven-Ric and vknabel May 29, 2026 13:10

l0wl3vel force-pushed the feat/gatewayapi branch from bf2ba5a to 5aa456d Compare June 1, 2026 06:53

l0wl3vel marked this pull request as ready for review June 1, 2026 06:53

l0wl3vel requested review from a team as code owners June 1, 2026 06:53

l0wl3vel force-pushed the feat/gatewayapi branch from 5aa456d to a67cb9d Compare June 2, 2026 12:14

l0wl3vel force-pushed the feat/gatewayapi branch from e88a693 to a67cb9d Compare June 5, 2026 17:20

l0wl3vel force-pushed the feat/gatewayapi branch 4 times, most recently from 8075d54 to ffc4120 Compare June 10, 2026 04:42

Gerrit91 reviewed Jun 15, 2026

View reviewed changes

Comment thread control-plane/kind.yaml

Comment thread scripts/roll_certs.sh

Comment thread roles/gateway/tasks/main.yml Outdated

Comment thread roles/gateway/tasks/main.yml

l0wl3vel force-pushed the feat/gatewayapi branch 2 times, most recently from f80647f to 5819828 Compare July 3, 2026 12:09

l0wl3vel force-pushed the feat/gatewayapi branch from dae73d5 to a78c21e Compare July 6, 2026 11:13

l0wl3vel requested a review from Gerrit91 July 6, 2026 11:49

l0wl3vel force-pushed the feat/gatewayapi branch 3 times, most recently from adeaa03 to 8da4fe3 Compare July 8, 2026 12:39

l0wl3vel force-pushed the feat/gatewayapi branch 3 times, most recently from e15f6b4 to 8aaaac4 Compare July 13, 2026 16:07

l0wl3vel added 3 commits July 14, 2026 10:43

feat: add metal-api server certificate

b033476

Add a metal-api server cert (api./v2. nip.io SANs) for the Gateway to terminate TLS with, add it to roll_certs.sh, and update the grpc cert host to the Gateway LoadBalancer IP. Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

l0wl3vel force-pushed the feat/gatewayapi branch from 8aaaac4 to aa1926b Compare July 14, 2026 08:43

Gerrit91 reviewed Jul 15, 2026

View reviewed changes

l0wl3vel force-pushed the feat/gatewayapi branch from aa1926b to 4424515 Compare July 16, 2026 15:44

l0wl3vel added 6 commits July 16, 2026 17:47

feat: remove ingress-nginx exposed ports

71941ea

Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

feat: use a single certificate for metal-api and metal-apiserver

1bf6c8c

Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

feat: add https listeners for metal-api and metal-apiserver

3e53230

Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

feat: use 80 and 443 on gateway listeners

9bba781

as we are not listening on the host we are free to use any port we like. So we switch to the default ports for https and http Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

feat: use HTTPS endpoints for metalctl

0bd111a

we need to put the CA data into an env var to later access metal-api. So we run cert generation on before calling the requested make target if certs are not generated yet. Signed-off-by: Benjamin Ritter <benjamin.ritter@x-cellent.com>

l0wl3vel force-pushed the feat/gatewayapi branch from 4424515 to 0bd111a Compare July 16, 2026 15:47

Uh oh!

Conversation

l0wl3vel commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

WIPs

Tested configurations

Noteworthy

Used AI-Tools ✨

Uh oh!

ma-hartma commented May 11, 2026

Uh oh!

vknabel commented May 28, 2026

Uh oh!

l0wl3vel commented May 29, 2026

Uh oh!

l0wl3vel commented May 29, 2026

Uh oh!

Sven-Ric commented Jun 5, 2026

Uh oh!

l0wl3vel commented Jun 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gerrit91 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

l0wl3vel commented Jul 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gerrit91 left a comment

Choose a reason for hiding this comment

Uh oh!

Gerrit91 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Gerrit91 Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

l0wl3vel Jul 15, 2026

Choose a reason for hiding this comment

Uh oh!

l0wl3vel commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

l0wl3vel commented May 6, 2026 •

edited

Loading

l0wl3vel commented Jun 5, 2026 •

edited

Loading

l0wl3vel commented Jul 8, 2026 •

edited

Loading

l0wl3vel commented Jul 15, 2026 •

edited

Loading