Use gha oracle runners for build and test job by ddelnano · Pull Request #2277 · pixie-io/pixie

ddelnano · 2025-11-22T01:36:09Z

Summary: Use gha oracle runners for build and test job

This PR supersedes #2261.

Relevant Issues: N/A

Type of change: /kind cleanup

Test Plan: Adhoc build from latest commit passes (build link)

ddelnano · 2025-12-04T14:59:14Z

bazel/toolchain_transitions.bzl

    native.cc_binary,
    {
        "@//bazel/cc_toolchains:compiler": meta.replace_with("clang"),
+        "@//bazel/cc_toolchains:libc_version": meta.replace_with("glibc2_36"),


Without this, cc_clang_binary failed to link the stdlib or failed to find any stdlib header includes.

ddelnano · 2025-12-04T15:02:03Z

src/utils/testingutils/docker/elastic.go

-					SizeBytes: 100 * 1024 * 1024,
-				},
-			},
-		}


The container logs from a failed run are listed below:

container logs

time="2025-11-30T10:09:57Z" level=error msg="Elasticsearch container stdout:\nCreated elasticsearch keystore in /usr/share/elasticsearch/config\n{\"type\": \"server\", \"timestamp\": \"2025-11-30T10:09:02,580Z\", \"level\": \"WARN\", \"component\": \"o.e.b.Natives\", \"cluster.name\": \"docker-cluster\", \"node.name\": \"c84550e5e3b9\", \"message\": \"unable to load JNA native support library, native methods will be disabled.\", \n\"stacktrace\": [\"java.lang.UnsatisfiedLinkError: /tmp/elasticsearch-5921953616030113121/jna--1985354563/jna11826645726440770622.tmp: /tmp/elasticsearch-5921953616030113121/jna--1985354563/jna11826645726440770622.tmp: failed to map segment from shared object: Operation not permitted\",\n\"at java.lang.ClassLoader$NativeLibrary.load0(Native Method) ~[?:?]\",\n\"at java.lang.ClassLoader$NativeLibrary.load(ClassLoader.java:2440) ~[?:?]\",\n\"at java.lang.ClassLoader$NativeLibrary.loadLibrary(ClassLoader.java:2497) ~[?:?]\",\n\"at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:2694) ~[?:?]\",\n\"at java.lang.ClassLoader.loadLibrary(ClassLoader.java:2627) ~[?:?]\",\n\"at java.lang.Runtime.load0(Runtime.java:744) ~[?:?]\",\n\"at java.lang.System.load(System.java:1873) ~[?:?]\",\n\"at com.sun.jna.Native.loadNativeDispatchLibraryFromClasspath(Native.java:947) ~[jna-4.5.1.jar:4.5.1 (b0)]\",\n\"at com.sun.jna.Native.loadNativeDispatchLibrary(Native.java:922) ~[jna-4.5.1.jar:4.5.1 (b0)]\",\n\"at com.sun.jna.Native.<clinit>(Native.java:190) ~[jna-4.5.1.jar:4.5.1 (b0)]\",\n\"at java.lang.Class.forName0(Native Method) ~[?:?]\",\n\"at java.lang.Class.forName(Class.java:333) ~[?:?]\",\n\"at org.elasticsearch.bootstrap.Natives.<clinit>(Natives.java:45) [elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:104) [elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) [elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.6.0.jar:7.6.0]\"] }\n{\"type\": \"server\", \"timestamp\": \"2025-11-30T10:09:02,585Z\", \"level\": \"WARN\", \"component\": \"o.e.b.Natives\", \"cluster.name\": \"docker-cluster\", \"node.name\": \"c84550e5e3b9\", \"message\": \"cannot check if running as root because JNA is not available\" }\n{\"type\": \"server\", \"timestamp\": \"2025-11-30T10:09:02,585Z\", \"level\": \"WARN\", \"component\": \"o.e.b.Natives\", \"cluster.name\": \"docker-cluster\", \"node.name\": \"c84550e5e3b9\", \"message\": \"cannot install system call filter because JNA is not available\" }\n{\"type\": \"server\", \"timestamp\": \"2025-11-30T10:09:02,585Z\", \"level\": \"WARN\", \"component\": \"o.e.b.Natives\", \"cluster.name\": \"docker-cluster\", \"node.name\": \"c84550e5e3b9\", \"message\": \"cannot register console handler because JNA is not available\" }\n{\"type\": \"server\", \"timestamp\": \"2025-11-30T10:09:02,586Z\", \"level\": \"WARN\", \"component\": \"o.e.b.Natives\", \"cluster.name\": \"docker-cluster\", \"node.name\": \"c84550e5e3b9\", \"message\": \"cannot getrlimit RLIMIT_NPROC because JNA is not available\" }\n{\"type\": \"server\", \"timestamp\": \"2025-11-30T10:09:02,586Z\", \"level\": \"WARN\", \"component\": \"o.e.b.Natives\", \"cluster.name\": \"docker-cluster\", \"node.name\": \"c84550e5e3b9\", \"message\": \"cannot getrlimit RLIMIT_AS because JNA is not available\" }\n{\"type\": \"server\", \"timestamp\": \"2025-11-30T10:09:02,586Z\", \"level\": \"WARN\", \"component\": \"o.e.b.Natives\", \"cluster.name\": \"docker-cluster\", \"node.name\": \"c84550e5e3b9\", \"message\": \"cannot getrlimit RLIMIT_FSIZE because JNA is not available\" }\n{\"type\": \"server\", \"timestamp\": \"2025-11-30T10:09:02,725Z\", \"level\": \"ERROR\", \"component\": \"o.e.b.ElasticsearchUncaughtExceptionHandler\", \"cluster.name\": \"docker-cluster\", \"node.name\": \"c84550e5e3b9\", \"message\": \"uncaught exception in thread [main]\", \n\"stacktrace\": [\"org.elasticsearch.bootstrap.StartupException: ElasticsearchException[failed to bind service]; nested: AccessDeniedException[/opt/elasticsearch/volatile/data/nodes];\",\n\"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"Caused by: org.elasticsearch.ElasticsearchException: failed to bind service\",\n\"at org.elasticsearch.node.Node.<init>(Node.java:615) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"... 6 more\",\n\"Caused by: java.nio.file.AccessDeniedException: /opt/elasticsearch/volatile/data/nodes\",\n\"at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]\",\n\"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]\",\n\"at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) ~[?:?]\",\n\"at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389) ~[?:?]\",\n\"at java.nio.file.Files.createDirectory(Files.java:693) ~[?:?]\",\n\"at java.nio.file.Files.createAndCheckIsDirectory(Files.java:800) ~[?:?]\",\n\"at java.nio.file.Files.createDirectories(Files.java:786) ~[?:?]\",\n\"at org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:274) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:211) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:271) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.node.Node.<init>(Node.java:277) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.0.jar:7.6.0]\",\n\"... 6 more\"] }\n" time="2025-11-30T10:09:57Z" level=error msg="Elasticsearch container stderr:\nOpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.\nuncaught exception in thread [main]\nElasticsearchException[failed to bind service]; nested: AccessDeniedException[/opt/elasticsearch/volatile/data/nodes];\nLikely root cause: java.nio.file.AccessDeniedException: /opt/elasticsearch/volatile/data/nodes\n\tat java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)\n\tat java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)\n\tat java.base/sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:389)\n\tat java.base/java.nio.file.Files.createDirectory(Files.java:693)\n\tat java.base/java.nio.file.Files.createAndCheckIsDirectory(Files.java:800)\n\tat java.base/java.nio.file.Files.createDirectories(Files.java:786)\n\tat org.elasticsearch.env.NodeEnvironment.lambda$new$0(NodeEnvironment.java:274)\n\tat org.elasticsearch.env.NodeEnvironment$NodeLock.<init>(NodeEnvironment.java:211)\n\tat org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:271)\n\tat org.elasticsearch.node.Node.<init>(Node.java:277)\n\tat org.elasticsearch.node.Node.<init>(Node.java:257)\n\tat org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221)\n\tat org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221)\n\tat org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349)\n\tat org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)\n\tat org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)\n\tat org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)\n\tat org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)\n\tat org.elasticsearch.cli.Command.main(Command.java:90)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)\n\tat org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)\nFor complete error details, refer to the log at /opt/elasticsearch/volatile/logs/docker-cluster.log\n" 2025/11/30 10:09:57 Cannot start elasticsearch: reached retry deadline

I opted to remove these since the lack of VM runner access made this difficult to debug.

Instead of removing this, specifying the tmpfs mount permissions got the original test passing again.

Signed-off-by: Koray Oksay <koray.oksay@gmail.com>

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>

… start Signed-off-by: Dom Del Nano <ddelnano@gmail.com>

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>

…artup Signed-off-by: Dom Del Nano <ddelnano@gmail.com>

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>

ddelnano force-pushed the ddelnano/update-gha-oracle-runners branch from 5e9f851 to 53592ff Compare November 27, 2025 13:48

ddelnano force-pushed the ddelnano/update-gha-oracle-runners branch from 53592ff to 8f1d8c4 Compare December 4, 2025 14:52

ddelnano commented Dec 4, 2025

View reviewed changes

ddelnano force-pushed the ddelnano/update-gha-oracle-runners branch from 8f1d8c4 to e943e96 Compare December 4, 2025 15:54

ddelnano marked this pull request as ready for review December 4, 2025 22:31

ddelnano requested review from a team as code owners December 4, 2025 22:31

ddelnano mentioned this pull request Dec 4, 2025

update oracle runners with vm ones #2261

Closed

JamesMBartlett approved these changes Dec 4, 2025

View reviewed changes

vihangm approved these changes Dec 5, 2025

View reviewed changes

ddelnano force-pushed the ddelnano/update-gha-oracle-runners branch from 71e9a5f to 442f86e Compare December 5, 2025 17:10

koksay and others added 8 commits December 5, 2025 17:11

update oracle runners with vm ones

87400e9

Signed-off-by: Koray Oksay <koray.oksay@gmail.com>

Don't run with remote execution. Build all bpf tests #ci:bpf-build

6bd9571

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>

Remove es tmpfs mounts that cause AccessDenfiedException on container…

6ba104e

… start Signed-off-by: Dom Del Nano <ddelnano@gmail.com>

Ensure that cc_clang_binary rules have access to sysroot glibc

103b873

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>

Ensure the path.data and path.logs ES env vars are removed too

9ea8740

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>

Remove branch trigger used for testing

4a99482

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>

Add tmpfs mounts back with proper permissions to fix elasticsearch st…

0fa9e12

…artup Signed-off-by: Dom Del Nano <ddelnano@gmail.com>

Fix linting issues

68178d1

Signed-off-by: Dom Del Nano <ddelnano@gmail.com>

ddelnano force-pushed the ddelnano/update-gha-oracle-runners branch from 442f86e to 68178d1 Compare December 5, 2025 17:11

ddelnano merged commit 2ae9747 into pixie-io:main Dec 5, 2025
27 checks passed

ddelnano deleted the ddelnano/update-gha-oracle-runners branch December 5, 2025 18:07

ddelnano mentioned this pull request Dec 10, 2025

Investigate ASAN test timeouts on CNCF runners #2295

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use gha oracle runners for build and test job#2277

Use gha oracle runners for build and test job#2277
ddelnano merged 8 commits intopixie-io:mainfrom
ddelnano:ddelnano/update-gha-oracle-runners

ddelnano commented Nov 22, 2025 •

edited

Loading

Uh oh!

ddelnano Dec 4, 2025

Uh oh!

ddelnano Dec 4, 2025

Uh oh!

ddelnano Dec 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

ddelnano commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ddelnano Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

ddelnano Dec 4, 2025

Choose a reason for hiding this comment

Uh oh!

ddelnano Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ddelnano commented Nov 22, 2025 •

edited

Loading