feat(cubeproxy): add plaintext gRPC ingress on port 9090

[xiaojunxiang2023]

Summary

Add a plaintext HTTP/2 gRPC ingress listener on CubeProxy (default port 9090) so clients can reach sandbox gRPC services without wildcard DNS or TLS on CubeProxy.

Clients dial the CubeProxy IP and identify the target sandbox via the gRPC :authority pseudo-header, using the same <container_port>-<sandbox_id> format as host-based HTTP routing.

Changes

Area	What
CubeProxy/nginx.conf	Add grpc_backend upstream and HTTP/2 server block on port 9090
CubeProxy/Dockerfile	Expose port 9090
One-click deploy	CUBE_PROXY_GRPC_PORT templating, startup/postcheck, env.example
TKE	CLB/service exposure and security group for port 9090
Docs	gRPC access mode in https-and-domain.md; port/config in network-hardening.md, self-build-deploy.md, service-management.md
Tests	Extend test_runtime_file_safety.sh for gRPC postcheck port

Motivation

CubeProxy already supports sandbox ingress via:

• Host mode: <container_port>-<sandbox_id>.<domain> on ports 80/443
• Path mode: /sandbox/<sandbox_id>/<container_port>/...

gRPC clients often cannot rely on wildcard DNS or custom TLS. A dedicated plaintext HTTP/2 listener lets them connect directly to the CubeProxy IP.

Usage

dial: <cube-proxy-ip>:9090
:authority: <container_port>-<sandbox_id>

Example:

dial: 10.0.0.5:9090
:authority: 49983-abc123

See docs/guide/https-and-domain.md for details.

Configuration

Variable	Default	Description
CUBE_PROXY_GRPC_PORT	9090	Plaintext gRPC (HTTP/2) listen port in one-click deploy

[cubesandboxbot]

Code Review: PR #679 — Plaintext gRPC Ingress on Port 9090

Overview

The PR adds a plaintext HTTP/2 gRPC listener on CubeProxy port 9090 with corresponding deployment, Terraform, doc, and test changes. Overall well-structured — consistent error handling patterns, good defensive templating, and clean integration with existing code. Below are the noteworthy findings across all review dimensions.

---

Findings

1. CLB security group is missing port 9090 — feature will be unreachable (Critical)

The cube_proxy Kubernetes Service (tke-addons.tf:762) is annotated with tencentcloud_security_group.clb.id, but tencentcloud_security_group_rule_set.clb (main.tf, lines 420-469) only has ingress rules for ports 80, 443, 3000, and 8089 — port 9090 is absent. The CLB will drop all inbound traffic on port 9090.

The port 9090 rule was instead placed in the jumpserver security group (a bastion CVM), where it has no effect. An operator reading the jumpserver SG would reasonably believe the port is open while the actual traffic path silently drops it.

(Inlined at main.tf:309)

2. Missing grpc_buffering off; — streaming RPCs buffer indefinitely (High)

Nginx defaults grpc_buffering to on, meaning the entire upstream response is buffered in memory before delivery to the client. The gRPC server block has no grpc_buffering directive. For server-streaming RPCs like filesystem.Filesystem/WatchDir, this consumes memory proportional to uptime. The existing HTTP server blocks explicitly disable buffering for streaming endpoints via proxy_buffering off.

(Inlined at CubeProxy/nginx.conf:383)

3. specify-protocol annotation excludes 9090 in internal mode (Medium)

The base specify-protocol annotation only declares ports 80 and 443. The public mode override correctly adds 9090, but the internal mode branch doesn't override it — so port 9090 lacks an explicit TCP protocol declaration in internal mode.

(Inlined at tke-addons.tf:765)

4. Missing client IP forwarding in gRPC block (Medium)

The HTTP server blocks forward client IP via proxy_set_header X-Real-IP $remote_addr. The gRPC block uses grpc_pass and needs grpc_set_header equivalents, which are absent. This degrades audit logging and prevents IP-based access control in the upstream.

(Inlined at CubeProxy/nginx.conf:386)

5. _sidecar_resume location triplicated across server blocks (Medium)

The same location = /_sidecar_resume { internal; ... } block appears identically in the 8081, 8080, and new 9090 server blocks — a maintenance hazard. This location is protocol-agnostic and could be extracted to an include file.

(Inlined at CubeProxy/nginx.conf:370)

6. ufw firewall examples omit port 9090 (Low)

The hardening guide's ufw example allows ports 80 and 443 for cube-proxy but not 9090. Users following these rules will block gRPC traffic. Add sudo ufw allow 9090/tcp.

docs/guide/network-hardening.md:183 (ZH: docs/zh/guide/network-hardening.md:173)

7. Missing changelog entry (Low)

The v0.5.0 changelog has no mention of the gRPC ingress feature or CUBE_PROXY_GRPC_PORT variable. docs/changelog/v0.5.0.md (ZH: docs/zh/changelog/v0.5.0.md)

8. grpc_plaintext.py example lacks error handling (Low)

channel_ready_future(...).result(timeout=15) raises FutureTimeoutError/RpcError on failure, producing an unfriendly traceback. Wrap with try/except. Also consider documenting keepalive options.

(Inlined at examples/grpc-ingress/grpc_plaintext.py:50)

---

Validated as Correct

• Template validation: build-release-bundle.sh verification loop catches unsubstituted tokens at build time
• Token enforcement: gRPC metadata routing to ngx.var.http_* works with existing underscores_in_headers on;
• Upstream keepalive: keepalive 200 is appropriate for HTTP/2 multiplexing (vs 1500 for HTTP/1.1)
• /_sidecar_resume security: internal; directive correctly prevents external access
• Error reporting improvement: The readiness check now collects all failed ports into not_ready instead of dying on the first failure

[chenhengqi]

Please add an example. Do we need SDK side support?

[xiaojunxiang2023]

Please add an example. Do we need SDK side support?

Example: OK, I have added examples/grpc-ingress/. Run python grpc_plaintext.py after setting CUBE_API_URL, CUBE_TEMPLATE_ID, and CUBE_PROXY_NODE_IP. It creates a sandbox, dials CUBE_PROXY_NODE_IP:9090 with gRPC :authority set to <envd_port>-<sandbox_id> (default 49983-<id>), and verifies the channel becomes READY through CubeProxy.

SDK: No SDK changes are required for this PR. The existing Python/Go SDK continues to use HTTP/Connect on CubeProxy HTTP/HTTPS (commands, files, etc.). Port 9090 is for native gRPC clients (grpcio, grpc-go, custom tooling) that cannot use *.cube.app DNS or TLS on CubeProxy.

[chenhengqi]

cc @fslongjin @tinklone for deploy-related changes.

[chenhengqi]

Drop these four lines.

[chenhengqi]

We will remove sidecar in #705.

[xiaojunxiang2023]

Sounds good. Since there would be merge conflicts, I'll wait for #705 to be merged before opening a new PR. I'll close this PR for now.

[cubesandboxbot]

Missing grpc_buffering off; — streaming RPCs can exhaust memory

The HTTP server blocks explicitly disable buffering for known streaming endpoints (process.Start, Connect, WatchDir) via proxy_buffering off in envd_streaming_host_route.inc. The gRPC equivalent is grpc_buffering off, but it's absent here.

Nginx defaults grpc_buffering to on, meaning the entire upstream response is buffered in memory before any data reaches the client. For server-streaming RPCs like filesystem.Filesystem/WatchDir (a never-ending stream), this would consume memory proportional to uptime.

Recommend adding grpc_buffering off; to this location block (or separating streaming and unary methods into distinct locations as the HTTP ports do).

[cubesandboxbot]

Missing grpc_set_header for client IP forwarding

The HTTP server blocks forward the client IP via proxy_set_header X-Real-IP $remote_addr and proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for (lines 343-344). The gRPC block needs grpc_set_header instead of proxy_set_header since it uses grpc_pass.

Without this, the upstream envd process receives all connections as if they originated from the nginx proxy's address rather than the actual client — degrading audit logging and preventing IP-based access control.

[cubesandboxbot]

_sidecar_resume block is triplicated across server blocks

This same location = /_sidecar_resume { internal; ... } block now appears identically in the 8081 (line 132), 8080 (line 254), and the new 9090 server blocks. It's a maintenance hazard — any change to the sidecar resume protocol must be replicated in three places.

Consider extracting this into an included file (e.g. conf/includes/sidecar_resume.inc) and using include in all three server blocks. The location is protocol-agnostic and doesn't need to differ between HTTP and gRPC servers.

[cubesandboxbot]

Port 9090 is opened on the jumpserver security group but the CLB SG is missing it

The port 9090 ingress rule is added to the jumpserver security group (tencentcloud_security_group.jumpserver), which is a bastion CVM that does not run cube-proxy. Meanwhile, the CLB security group (tencentcloud_security_group_rule_set.clb, lines 420-469) lists ports 80, 443, 3000, and 8089 but not 9090.

The cube_proxy Kubernetes Service (tke-addons.tf:762) is annotated with tencentcloud_security_group.clb.id, so the CLB SG is the one that controls inbound traffic to the proxy. Without a 9090 ingress rule there, the CLB will drop all inbound traffic on port 9090.

Recommend moving this rule to the CLB SG:

ingress {
  action      = "ACCEPT"
  cidr_block  = var.enable_public_network ? "0.0.0.0/0" : "10.0.0.0/16"
  protocol    = "TCP"
  port        = "9090"
  description = "Allow cube-proxy plaintext gRPC ingress"
}

[cubesandboxbot]

channel_ready_future should handle connection failures gracefully

grpc.channel_ready_future(channel).result(timeout=15) raises grpc.FutureTimeoutError on connection timeout or grpc.RpcError on other failures. Neither is caught here. Consider wrapping in try/except with a user-friendly message.

Also consider documenting keepalive options (grpc.keepalive_time_ms) for production use, as gRPC streams may be dropped by intermediate network equipment with idle timeouts.

[cubesandboxbot]

Port 9090 missing from specify-protocol annotation in internal mode

The base annotation (line 759) only declares ports 80 and 443. The public mode override correctly adds 9090, but the internal mode branch does not override specify-protocol — so port 9090 falls back to the base annotation without TCP protocol declaration. Recommend moving 9090 into the base annotation.

[cubesandboxbot]

Missing grpc_buffering off; — streaming RPCs can exhaust memory

The HTTP server blocks explicitly disable buffering for known streaming endpoints (process.Start, Connect, WatchDir) via proxy_buffering off in envd_streaming_host_route.inc. The gRPC equivalent is grpc_buffering off, but it's absent here.

Nginx defaults grpc_buffering to on, meaning the entire upstream response is buffered in memory before any data reaches the client. For server-streaming RPCs like filesystem.Filesystem/WatchDir (a never-ending stream), this would consume memory proportional to uptime.

Recommend adding grpc_buffering off; to this location block (or separating streaming and unary methods into distinct locations as the HTTP ports do).

[cubesandboxbot]

Missing grpc_set_header for client IP forwarding

The HTTP server blocks forward the client IP via proxy_set_header X-Real-IP $remote_addr and proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for (lines 343-344). The gRPC block needs grpc_set_header instead of proxy_set_header since it uses grpc_pass.

Without this, the upstream envd process receives all connections as if they originated from the nginx proxy's address rather than the actual client — degrading audit logging and preventing IP-based access control.

[cubesandboxbot]

_sidecar_resume block is triplicated across server blocks

This same location = /_sidecar_resume { internal; ... } block now appears identically in the 8081 (line 132), 8080 (line 254), and the new 9090 server blocks. It's a maintenance hazard — any change to the sidecar resume protocol must be replicated in three places.

Consider extracting this into an included file (e.g. conf/includes/sidecar_resume.inc) and using include in all three server blocks. The location is protocol-agnostic and doesn't need to differ between HTTP and gRPC servers.