Tigera operator (post-migration) fails to work if some pods have ips not in cluster pod CIDR

[taxilian]

First, let me clarify that I'm aware that my cluster is misconfigured =] Unfortunately it seems there is no way to fix that without completely rebuilding the cluster, which I don't have time to do right now.

I did not realize that calico's IPPool resources still needed to be within the cluster pod CIDR, so I'm using pools that are outside of it -- e.g. 172.25.64.0/20 when my cluster CIDR is 10.172.0.0/16. I didn't realize it mattered, and since I'm using ToR bgp peering 99% of everything still works fine. However, after migrating to the tigera operator (during which I attempted to change my cluster CIDR and discovered that it's either not possible or at least more difficult than expected) the operator won't update anything because it's constantly in an "error" state:

Could not resolve CalicoNetwork IPPool and kubeadm configuration: IPPool 172.25.64.0/20 is not within the platform's configured pod network CIDR(s) [10.172.0.0/16 2607:fa18:1000:21::10:0/108]

I'm using VPNs and linking multiple clusters together, so unfortunately moving my IPPools isn't an option.

Expected Behavior

A warning should be thrown, but there should be a way to tell it "yeah, I know this is wrong, but that's how everything is set up so please ignore it"

I absolutely 100% agree that there should be warnings to tell uneducated folks like me that they are Doing Something Stupid, but since it can actually work it should let you do it if you really want.

Current Behavior

The Tigera operator is completely nonfunctional due to the error state.

Steps to Reproduce (for bugs)

1. Have a cluster with IPPools that are not in the cluster pod CIDR
2. Upgrade to the calico operator, doing whatever you have to to get it to install =]

Context

I think I've clarified this above, but it seems like it should be an easy fix.

Your Environment

Kubeadm bare metal cluster, 7 servers; opnsense ToR routers with bgp peering. Dual IPv4/IPv6 stack.

[tmjd]

As a work around for you right now you should be able to set the CIDRs in the Installation resource to match the kubeadm configuration. This will be fine because the operator currently does not reconcile the IPPools defined in the Installation resource with the existing IPPools, so since the default IPPools are already created, it will not try to change/add/update the existing IPPools.
You should make sure the IPPools you define in the Installation resource do match the existing IPPools fields encapsulation and natOutgoing fields.

[caseydavenport]

This will be more important long term as well. Kubernetes is adding the ability to have multiple cluster CIDRs in the near future, so we'll need to keep our logic insync with that.

Might be worth converting this to a warning sooner rather than later, and stop blocking the operator from doing its work. I don't think we have a great mechanism to implement that right now, but it would be useful.

[tmjd]

I'm not sure where we could put a warning that the user would see. Is this a use case for adding another "tigerastatus" column/field that would report this issue but nothing would be blocked on it?
I feel like this wouldn't be noticed then unless there was some other issue, but maybe that would be good enough to give the user a pointer to what might be wrong in the case of something else being broken.

[taxilian]

What issues does it actually cause? Being the ignorant sap that I was I set up my cluster in such a way that I can't resolve this problem without rebuilding the full cluster, so I've been operating for some time with my ippools all outside the cluster pod CIDR -- I would probably have seen a warning in a tigerastatus update if it were there, but otherwise you're right I wouldn't have seen anything, partly because so far it's all working reasonably fine. I do sometimes have an odd issue, which might or might not be related to this. Anyway, not saying that it isn't a problem, but it's clearly not a "end of all things" type issue so it does not seem like it should prevent upgrade or install.

…

On Wed, Dec 15, 2021 at 11:50 AM Erik Stidham ***@***.***> wrote: I'm not sure where we could put a warning that the user would see. Is this a use case for adding another "tigerastatus" column/field that would report this issue but nothing would be blocked on it? I feel like this wouldn't be noticed then unless there was some other issue, but maybe that would be good enough to give the user a pointer to what might be wrong in the case of something else being broken. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1512 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABWYTT6YCMQZY2RKTMV66LURDPQXANCNFSM5DZ5IJDQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[tmjd]

What issues does it actually cause?

The issues that would happen is that traffic may get NAT'ed when it shouldn't be which would cause problems with policy not applying correctly (block traffic that should be allowed but since it is coming from an unexpected IP it is blocked).

[github-actions]

This issue is stale because it is kind/enhancement or kind/bug and has been open for 180 days with no activity.

[caseydavenport]

I think we should relax this error - or at least make the check opt-out. This has historically been rather important, as @tmjd mentions it can cause unexpected NAT that can manifest in unexpected ways.

There are new ways the upsteram k8s can use to detect locality of network traffic, though. So this restriction is no longer strictly correct all the time.

[caseydavenport]

Worth noting that you can set ipPools: [] in your installation to bypass this check. It does require that your IP pools are managed externally to the operator Installation CR, though.

operator/pkg/controller/ippool/defaults.go

Lines 334 to 336 in 3ba1a56

	} else if len(i.Spec.CalicoNetwork.IPPools) == 0 {
	// Empty IPPools list so nothing to do.
	return nil

[github-actions]

This issue is stale because it is kind/enhancement or kind/bug and has been open for 180 days with no activity.