Add retry logic to SendTCP in e2e framework by markmandel · Pull Request #4465 · agones-dev/agones

markmandel · 2026-02-28T17:26:24Z

What type of PR is this?

Uncomment only one /kind <> line, press enter to put that in a new line, and remove leading whitespace from that line:

/kind breaking
/kind bug

/kind cleanup

/kind documentation
/kind feature
/kind hotfix
/kind release

What this PR does / Why we need it:

Not sure if this will solve the issues with Autopilot, but this at least gives us a step in the right direction.

Which issue(s) this PR fixes:

Work on #4464

Special notes for your reviewer:

N/A

Not sure if this will solve the issues with Autopilot, but this at least gives us a step in the right direction. Work on agones-dev#4464

markmandel · 2026-02-28T17:27:00Z

pkg/apis/agones/v1/gameserver.go

 	// make sure the container value points to a valid container
 	if !gss.HasContainer(gss.Container, false) {
-		allErrs = append(allErrs, field.Invalid(fldPath.Child("container"), gss.Container, "Could not find a container named " + gss.Container))
+		allErrs = append(allErrs, field.Invalid(fldPath.Child("container"), gss.Container, "Could not find a container named "+gss.Container))


My updated linter was failing on this one, so this got blended in.

agones-bot · 2026-02-28T18:03:04Z

Build Failed 😭

Build Id: 01a4b98e-524c-43fb-99c4-376c85bea92a

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

markmandel · 2026-02-28T18:21:04Z

Yeah there is something definitely wrong with the Autopilot clusters, even with the retry logic.

=== RUN   TestGameServerTcpUdpProtocol
=== PAUSE TestGameServerTcpUdpProtocol
=== CONT  TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:17.220" level=info msg="GameServer created, waiting for Ready" gs=game-servermssq6 test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:17.369" level=info msg="Waiting for states to match" awaitingState=Ready currentState=Creating gs=game-servermssq6 test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:18.353" level=info msg="Waiting for states to match" awaitingState=Ready currentState=Starting gs=game-servermssq6 test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:19.355" level=info msg="Waiting for states to match" awaitingState=Ready currentState=Starting gs=game-servermssq6 test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:20.354" level=info msg="Waiting for states to match" awaitingState=Ready currentState=RequestReady gs=game-servermssq6 test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:21.355" level=info msg="GameServer states match" awaitingState=Ready currentState=Ready gs=game-servermssq6 test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:21.356" level=info msg="GameServer Ready" gs=game-servermssq6
time="2026-02-28 18:00:21.356" level=info msg="GameServer created, sending UDP ping" name=game-servermssq6
time="2026-02-28 18:00:21.477" level=info msg="UDP ping passed, sending TCP ping" name=game-servermssq6
time="2026-02-28 18:00:21.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:23.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:25.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:27.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:29.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:31.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:33.598" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:35.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:37.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:39.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:41.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:43.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:45.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:47.598" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:49.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:51.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:53.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:55.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:57.598" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:00:59.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:01:01.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:01:03.598" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:01:05.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:01:07.598" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:01:09.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:01:11.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:01:13.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:01:15.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:01:17.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:01:19.597" level=info msg="could not dial address" address="35.229.225.100:7824" error="dial tcp 35.229.225.100:7824: connect: connection refused" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:01:21.478" level=info msg="Failed to send TCP packet to GameServer. Dumping Events!" gs=game-servermssq6 status="{State:Ready Ports:[{Name:gameserver-tcp Port:7824} {Name:gameserver-udp Port:7824}] Address:35.229.225.100 Addresses:[{Type:InternalIP Address:10.140.0.27} {Type:ExternalIP Address:35.229.225.100} {Type:Hostname Address:gk3-gke-autopilot-e2e-test-clu-pool-1-12d37f0a-8s2r} {Type:PodIP Address:10.32.130.86}] NodeName:gk3-gke-autopilot-e2e-test-clu-pool-1-12d37f0a-8s2r ReservedUntil:<nil> Players:0xc000746750 Counters:map[] Lists:map[] Eviction:0xc000744490}" test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:01:21.478" level=info msg="Dumping Events:" kind= test=TestGameServerTcpUdpProtocol
time="2026-02-28 18:01:21.638" level=info msg="Event!" lastTimestamp="2026-02-28 18:00:17 +0000 UTC" message="Port allocated" reason=PortAllocation test=TestGameServerTcpUdpProtocol type=Normal
time="2026-02-28 18:01:21.638" level=info msg="Event!" lastTimestamp="2026-02-28 18:00:17 +0000 UTC" message="Pod game-servermssq6 created" reason=Creating test=TestGameServerTcpUdpProtocol type=Normal
time="2026-02-28 18:01:21.638" level=info msg="Event!" lastTimestamp="2026-02-28 18:00:19 +0000 UTC" message="SDK state change" reason=RequestReady test=TestGameServerTcpUdpProtocol type=Normal
time="2026-02-28 18:01:21.638" level=info msg="Event!" lastTimestamp="2026-02-28 18:00:20 +0000 UTC" message="Address and port populated" reason=Ready test=TestGameServerTcpUdpProtocol type=Normal
time="2026-02-28 18:01:21.638" level=info msg="Event!" lastTimestamp="2026-02-28 18:00:20 +0000 UTC" message="SDK.Ready() complete" reason=Ready test=TestGameServerTcpUdpProtocol type=Normal
    gameserver_test.go:1093: Could not ping TCP GameServer: timed out attempting to send TCP message to address: context deadline exceeded
--- FAIL: TestGameServerTcpUdpProtocol (64.63s)
FAIL test/e2e.TestGameServerTcpUdpProtocol (re-run 1) (64.63s)
time="2026-02-28 18:01:21.953" level=info msg="Namespace 1772301615 is deleted"
FAIL test/e2e

I remember us having to recreate those clusters occasionally? Is that issue still a thing?

markmandel · 2026-02-28T18:22:09Z

@igooch @peterzhongyi - check above please.

markmandel · 2026-03-01T14:36:12Z

/gcbrun

agones-bot · 2026-03-01T15:05:21Z

Build Failed 😭

Build Id: a111e873-f944-40ff-9f85-1a00120301b4

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

agones-bot · 2026-03-03T18:08:29Z

Build Failed 😭

Build Id: 5ea64a27-5f5f-48c7-bb81-6ae13defbd5a

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

Sivasankaran25 · 2026-03-04T07:11:52Z

/gcbrun

agones-bot · 2026-03-04T08:08:46Z

Build Failed 😭

Build Id: faa02cc3-656d-419d-8e85-c674c6259e30

Status: FAILURE

To get permission to view the Cloud Build view, join the agones-discuss Google Group.

Sivasankaran25 · 2026-03-04T09:05:12Z

/gcbrun

agones-bot · 2026-03-04T10:00:44Z

Build Succeeded 🥳

Build Id: 4b4f53b3-587b-4067-826c-7018fd5ed68b

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.57.0-dev-dd54f32
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.57.0-dev-dd54f32
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.57.0-dev-dd54f32-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.57.0-dev-dd54f32
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.57.0-dev-dd54f32
image: us-docker.pkg.dev/agones-images/ci/agones-processor:1.57.0-dev-dd54f32
Linux C++ SDK (build): agonessdk-1.57.0-dev-dd54f32-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.57.0-dev-dd54f32.zip

A preview of the website (the last 30 builds are retained):

https://dd54f32-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4465/head:pr_4465 && git checkout pr_4465
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.57.0-dev-dd54f32

agones-bot · 2026-03-04T18:37:05Z

Build Succeeded 🥳

Build Id: 1d12e91e-3fe6-4ccd-b511-4d286a6a3cc3

The following development artifacts have been built, and will exist for the next 30 days:

image: us-docker.pkg.dev/agones-images/ci/agones-controller:1.57.0-dev-3f57ad8
image: us-docker.pkg.dev/agones-images/ci/agones-extensions:1.57.0-dev-3f57ad8
image: us-docker.pkg.dev/agones-images/ci/agones-sdk:1.57.0-dev-3f57ad8-linux
image: us-docker.pkg.dev/agones-images/ci/agones-ping:1.57.0-dev-3f57ad8
image: us-docker.pkg.dev/agones-images/ci/agones-allocator:1.57.0-dev-3f57ad8
image: us-docker.pkg.dev/agones-images/ci/agones-processor:1.57.0-dev-3f57ad8
Linux C++ SDK (build): agonessdk-1.57.0-dev-3f57ad8-linux-arch_64.tar.gz
SDK Server: agonessdk-server-1.57.0-dev-3f57ad8.zip

A preview of the website (the last 30 builds are retained):

https://3f57ad8-dot-preview-dot-agones-images.appspot.com/

To install this version:

git fetch https://github.com/googleforgames/agones.git pull/4465/head:pr_4465 && git checkout pr_4465
helm install agones ./install/helm/agones --namespace agones-system --set agones.image.registry=us-docker.pkg.dev/agones-images/ci --set agones.image.tag=1.57.0-dev-3f57ad8

Add retry logic to SendTCP in e2e framework

cd0e183

Not sure if this will solve the issues with Autopilot, but this at least gives us a step in the right direction. Work on agones-dev#4464

markmandel added the area/tests Unit tests, e2e tests, anything to make sure things don't break label Feb 28, 2026

github-actions bot added kind/cleanup Refactoring code, fixing up documentation, etc size/S labels Feb 28, 2026

markmandel commented Feb 28, 2026

View reviewed changes

Merge branch 'main' into e2e/tcp-redundancy

dd54f32

Merge branch 'main' into e2e/tcp-redundancy

3f57ad8

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add retry logic to SendTCP in e2e framework#4465

Add retry logic to SendTCP in e2e framework#4465
markmandel wants to merge 3 commits intoagones-dev:mainfrom
markmandel:e2e/tcp-redundancy

markmandel commented Feb 28, 2026

Uh oh!

markmandel Feb 28, 2026

Uh oh!

agones-bot commented Feb 28, 2026

Uh oh!

markmandel commented Feb 28, 2026

Uh oh!

markmandel commented Feb 28, 2026

Uh oh!

markmandel commented Mar 1, 2026

Uh oh!

agones-bot commented Mar 1, 2026

Uh oh!

agones-bot commented Mar 3, 2026

Uh oh!

Sivasankaran25 commented Mar 4, 2026

Uh oh!

agones-bot commented Mar 4, 2026

Uh oh!

Sivasankaran25 commented Mar 4, 2026

Uh oh!

agones-bot commented Mar 4, 2026

Uh oh!

agones-bot commented Mar 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

markmandel commented Feb 28, 2026

Uh oh!

markmandel Feb 28, 2026

Choose a reason for hiding this comment

Uh oh!

agones-bot commented Feb 28, 2026

Uh oh!

markmandel commented Feb 28, 2026

Uh oh!

markmandel commented Feb 28, 2026

Uh oh!

markmandel commented Mar 1, 2026

Uh oh!

agones-bot commented Mar 1, 2026

Uh oh!

agones-bot commented Mar 3, 2026

Uh oh!

Sivasankaran25 commented Mar 4, 2026

Uh oh!

agones-bot commented Mar 4, 2026

Uh oh!

Sivasankaran25 commented Mar 4, 2026

Uh oh!

agones-bot commented Mar 4, 2026

Uh oh!

agones-bot commented Mar 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants