Fix istio proxy build after moving Envoy to hermteic toolchain

[krinkinmu]

What this PR does / why we need it:

It makes changes to the Istio proxy Bazel configs to migrate to using Envoy's hermetic LLVM toolchain for builds. The "hermetic" here means that as part of the build process Bazel will download the LLVM toolchain, including compiler, linker and libraries, and will use them to build.

One thing worth calling out specifically, is that Envoy will also download a sysroot that packages glibc vers 2.28, which is, I think, the same version of glibc that we are currently using, so we still should be able to run the resulting Envoy binary outside of the container environment on old OSes.

NOTE: llvm toolchain itself is not statically built, unfortunately, and not all required libraries come with the toolchain. One library specifically that I noticed might cause problems is libncurses. Clang in the hermetic llvm toolchain is built against libncurses version 5, while some major distributions (i.e., Ubuntu) moved towards version 6.

It does not matter when building inside a container - we use a pretty old base image that should have the required version of the libncurses library, however building things outside the container may require jumping through a few hoops to install the right libraries if the hermetic toolchain is used.

I don't think there is a sensible way to workaround this issue in the hermetic toolchain itself (I wish LLVM folks just released a static version of the toolchain, but alas they don't do that). So IMO the best path forward would be to just allow easy switching between hermetic toolchain and host toolchain, which this PR does not implement yet. However, if we ok in principle with this approach, I can probably add support for that.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

[istio-testing]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[istio-policy-bot]

😊 Welcome @krinkinmu! This is either your first contribution to the Istio proxy repo, or it's been
a while since you've been here.

You can learn more about the Istio working groups, Code of Conduct, and contribution guidelines
by referring to Contributing to Istio.

Thanks for contributing!

Courtesy of your friendly welcome wagon.

[krinkinmu]

/test all

[howardjohn]

Does this mean you can finally just run bazel build and it works??? amazing!!

[krinkinmu]

Does this mean you can finally just run bazel build and it works??? amazing!!

Almost, but there are corner cases. E.g., llvm tools themselves are not built statically, so they still depend on some external libraries, but at the very least installing clang should not be necessary.

And we still need to figure out if we want to go this way or rely on a local toolchain.

[zirain]

/test all

[zirain]

@krinkinmu can you update your PR base on envoyproxy/envoy#42572?

[krinkinmu]

@zirain will do that later today

[krinkinmu]

/test all

[krinkinmu]

Interesting... It works locally, but not on CI. I need to dig a bit more.

[krinkinmu]

/test all

[krinkinmu]

At least it's building now...

[krinkinmu]

@zirain it looks like it builds. I'd need to double check what it's actually building and that the version of GLIBC it needs is indeed 2.28, but on the surface it looks ok.

[krinkinmu]

Aha... I missed something in the configuration apparently - it didn't really use the hermetic sysroot and instead used host GLIBC, which is ok when building insider a container, but not what we actually wanted.

[krinkinmu]

/test all

[krinkinmu]

I tested it locally and it builds using the right sysroot with glibc 2.28

[krinkinmu]

/test all

[krinkinmu]

/test all

[kyessenov]

Yes, of course, using upstream clang would give you clang 18 and better compilation.

[krinkinmu]

A small update on the issue with libncurses library.

It looks like this is the only weird dependency that causes this problem and LLVM folks said that they removed this dependency in the later version of clang, so say with clang-19 we wouldn't have this problem. However, fixing the latest 18.1.8 release of clang-18 or doing another release (e.g. 18.1.9) is unlikely (LLVM folks made some changes to the process in LLVM 19, so they are not particularly keen on trying to release 18 again).

So two options that make sense to try IMO:

1. We can try to switch to clang-19, but that's obviously a divergence from upstream Envoy, so we should try to do that in upstream Envoy.
2. When building locally and not on CI, just use the non-hermetic toolchain available in the host system.

I think if the option 1 works, it would be the best fix - no divergence, newer compiler, should work the same way on CI and on a local computer.

[krinkinmu]

/test all

[krinkinmu]

Regarding libncurses problem - we discussed in Slack a bit and it seems to be preferable to just ignore it for now. The issue only affects those who build proxy outside of the container and for folks advanced enough to go that path it shouldn't be too difficult to go through a few additional hoops to install libncurses5 if it's not available.

Eventually we will move to a newer version of clang and newer versions shouldn't have that problem anymore, so eventually we wouldn't need to worry about it at all.

[krinkinmu]

/test release-test-arm64

[krinkinmu]

/test release-test-arm64

[fjglira]

/test release-test-arm64

[keithmattix]

/test release-test-arm64

[krinkinmu]

/test release-test-arm64

[krinkinmu]

/test release-test-arm64

[krinkinmu]

/test release-test-arm64

[krinkinmu]

/test release-test-arm64

[krinkinmu]

/test all

[krinkinmu]

/test release-test

[zirain]

/retest

[zirain]

Thanks for taking this forward.

[zirain]

Regarding libncurses problem - we discussed in Slack a bit and it seems to be preferable to just ignore it for now. The issue only affects those who build proxy outside of the container and for folks advanced enough to go that path it shouldn't be too difficult to go through a few additional hoops to install libncurses5 if it's not available.

Eventually we will move to a newer version of clang and newer versions shouldn't have that problem anymore, so eventually we wouldn't need to worry about it at all.

this's fine for most of the cases.

it would be better if you can add a hint(use clang-19+ when you're building proxy outside of devcontainer) in README.

[dhawton]

/retest