OSDOCS#12501: Guidance for clusters that span data centers

Author: kowen-rh

_topic_maps/_topic_map.yml

@@ -2495,6 +2495,8 @@ Topics:
     File: etcd-disaster-recovery
 - Name: Enabling etcd encryption
   File: etcd-encrypt
+- Name: Guidance for clusters that span data centers
+  File: etcd-guidance-span
 # - Name: Setting up fault-tolerant control planes that span data centers
 #  File: etcd-fault-tolerant
 # - Name: Scaling a cluster to 4 or 5 control-plane nodes

etcd/etcd-guidance-span.adoc

@@ -0,0 +1,49 @@
+:_mod-docs-content-type: ASSEMBLY
+[id="etcd-guidance-span"]
+include::_attributes/common-attributes.adoc[]
+= Guidance for clusters that span data centers
+:context: etcd-guidance-span
+
+toc::[]
+
+Red Hat strongly recommends a deployment model where {product-title} clusters are deployed within a data center, but also acknowledges that there can be scenarios where a provider can use a deployment model where a cluster can span across data centers. This document outlines considerations when exploring the use of cluster deployments that span many data centers and describes important metrics that affect the supportability of such deployments. The design of such deployments should adhere to these guidelines for the product to function optimally and ensure the highest quality of support with the appropriate product support subscriptions.
+
+[WARNING]
+====
+A cluster deployment that spans many data centers extends the cluster as a single failure domain across locations and should not be considered a replacement for a disaster recovery plan.
+====
+
+Clusters with cluster deployments that span many data centers are bound by standard Red Hat {product-title} support guidance. See the link:https://access.redhat.com/support/policy/updates/openshift[Red Hat {product-title} Lifecycle] and link:https://access.redhat.com/support/offerings/production/soc[Red Hat Production Support Scope of Coverage] for more information.
+
+It is not recommended to deploy an {product-title} cluster that spans many sites. If you need to be in many data centers or regions, deploy one cluster per region or site and use tools such as Red Hat Advanced Cluster Management for Kubernetes (ACM) to manage these clusters and deployments.
+
+Some {product-title} platforms have specific support for many data center deployments. Check the platform-specific product documentation and release notes for details. Other platforms can span data centers, depending on the quality of the network connectivity between nodes. For more information, see link:https://access.redhat.com/articles/7010406[Understanding etcd and the tunables/conditions affecting performance].
+
+When implementing a cluster deployment that spans many data centers, you should strive to implement the practices detailed in link:https://access.redhat.com/articles/3221001[Red Hat {product-title} High Availability, and Recommended Practices]. An alternative to multisite deployments is to deploy one {product-title} cluster per site, managed by ACM.
+
+// Deployment caveats for multisite clusters
+include::modules/etcd-deployment-caveats-span.adoc[leveloffset=+1]
+
+// Infrastructure as a Service (IaaS) or cloud provider considerations
+include::modules/etcd-iaas-cloud-provider-considerations-span.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+.Additional resources
+* link:https://access.redhat.com/articles/7010220[Understanding and Validating MTU setting with {product-title} 4.x]
+* link:https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.19/html-single/configuring_openshift_data_foundation_disaster_recovery_for_openshift_workloads/index[Configuring {rh-storage} Disaster Recovery for OpenShift Workloads]
+
+// Site recommendations
+include::modules/etcd-site-recommendations-span.adoc[leveloffset=+1]
+
+// Requirements for etcd, networking, and storage
+include::modules/etcd-requirements-etcd-hardware-span.adoc[leveloffset=+1]
+
+[role="_additional-resources"]
+.Additional resources
+* link:https://access.redhat.com/articles/7010406[Understanding etcd and the tunables/conditions affecting performance]
+* link:https://access.redhat.com/articles/7010220[Understanding and Validating MTU setting with {product-title} 4.x]
+* link:https://access.redhat.com/articles/7010406#etcd-peer-round-trip-time-impacts-on-performance-6[etcd peer round trip time]
+* link:https://access.redhat.com/articles/7010406#effects-of-database-size-on-etcd-7[etcd database size]
+
+// Workload placement considerations
+include::modules/etcd-workload-placement-considerations-span.adoc[leveloffset=+1]

modules/etcd-deployment-caveats-span.adoc

@@ -0,0 +1,20 @@
+// Module included in the following assemblies:
+//
+// * etcd/etcd-guidance-span.adoc
+
+:_mod-docs-content-type: REFERENCE
+[id="deployment-caveats-span_{context}"]
+= Deployment caveats for spanned clusters
+
+The guidance provided in this documentation focuses on general aspects of a cluster deployment that spans data centers. Some caveats to remember:
+
+* Although the designs for deployments that span data centers are not bound by any special support requirements, these clusters do have additional inherent complexities that can require additional consideration or support involvement (time to identify, remediate and resolve issues) when compared to a standard single-site cluster.
+* Applications might not work well or not work at all in clusters with high Kube API latency or low transaction rates.
+* Layered products, such as storage providers, have lower latency requirements. In those cases, the latency limits are dictated by the architectures that are supported by the layered product.
+* The failure scenarios are amplified with stretched control planes, and the way they are affected is specific to the deployment. Because of this, before using a deployment that spans data centers on a production environment, the organization should test and document the behavior of the cluster during disruptions such as:
+** When there is a network partition leaving one, two, or all control plane nodes isolated
+** When there are MTU mismatches on the transport network among the control plane nodes
+** When there is a sustained spike in latency as a Day 2 event towards one or more of the control plane nodes
+** When there is a considerable change in jitter due to network congestion, misconfiguration, or lack of QoS, an intermediate network device causing packet errors, and others
+* Clusters deployed across many sites, network infrastructures, storage infrastructures, or other components inherently have a higher number of points of failure. Network disruptions or splits become a larger threat to such clusters especially, putting the nodes at risk of losing contact with each other. These multisite clusters must be designed with the potential for such failures in mind. Organizations deploying multisite clusters should extensively test failure scenarios, and should consider whether the cluster has protection from all points of failure. Consult with Red Hat Support for assistance in considering the important aspects of a resilient High Availability cluster design.
+* In some cases, GEO awareness is a requirement or issue that must be solved to minimize latency, so a proper implementation of a Global Service Load Balancing (GSLB) method must be available.

modules/etcd-iaas-cloud-provider-considerations-span.adoc

@@ -0,0 +1,16 @@
+// Module included in the following assemblies:
+//
+// * etcd/etcd-guidance-span.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="iaas-cloud-provider-considerations-span_{context}"]
+= Infrastructure as a Service (IaaS) and cloud provider considerations
+
+This guidance applies to any infrastructure provider for which {product-title} control plane nodes are supported by the user-provisioned infrastructure installer (platform=none) or the agent-based installer (platform=metal) using the ”User Managed Network” option. Installer-provisioned infrastructure installers are not covered by these guidelines, however, where possible, installer-provisioned infrastructure deployments will span zones or availability zones on cloud or IaaS providers if possible by following these or similar guidelines. This means infrastructure provider-specific integrations will not be available (for example, integration with Cloud provider services such as storage services and load balancers). Provider-specific services might still be used as external services.
+
+Using different infrastructure platform providers for control plane nodes is discouraged (for example, mixing nodes across IaaS, cloud, and bare metal as control plane nodes). Consider the following guidelines when such combinations are needed:
+
+* The minimum effective MTU across the infrastructure should be the maximum MTU used for the deployment. Using a lower MTU is acceptable. See _Understanding and Validating MTU setting with {product-title} 4.x_ for more information.
+* The combined disk and network latency and jitter must maintain an etcd peer round trip time of less than 100ms. This is not the same as the network round trip time.
+* Layered products might have lower latency requirements. In those cases, the latency limits are dictated by the requirements of the architecture supported by the layered product. For example, {product-title} cluster deployments that span data centers with {rh-storage-first} must have a latency requirement of less than 10ms RTT. For those cases, follow the specific product guidance.
+* For guidance on cluster deployments that span data centers using {rh-storage} as the storage provider, see _Configuring {rh-storage} Disaster Recovery for OpenShift Workloads_.

modules/etcd-requirements-etcd-hardware-span.adoc

@@ -0,0 +1,31 @@
+// Module included in the following assemblies:
+//
+// * etcd/etcd-guidance-span.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="requirements-etcd-hardware-span_{context}"]
+= Requirements for etcd, networking, and storage
+
+Consider the following requirements for clusters that span data centers.
+
+== etcd requirements
+
+There is a large list of factors and considerations that go into planning an etcd cluster deployment. When planning an {product-title} cluster that spans data centers, you need to plan for situations that will likely stress or push etcd to the edge of its operational limits.
+
+See _Understanding etcd and the tunables/conditions affecting performance_ for more details on how to maintain operational capabilities and reduce service-affecting events and instability of the cluster.
+
+== Network requirements
+
+The chosen network topology must yield direct IP connectivity between nodes. The minimum effective MTU across the infrastructure should be the maximum MTU used for the deployment. Using a lower MTU is acceptable.
+
+For more information, see _Understanding and Validating MTU setting with {product-title} 4.x_. The latency needs are ultimately defined by the services that use the network. See the sections related to etcd and storage for more details on requirements.
+
+In addition to the base networking requirements, you need to think about how applications will be accessed. A top-level Global Service Load Balancing (GSLB) method will be needed outside of {product-title} to enable external traffic to connect to the {product-title} control plane services and ingress controllers.
+
+== Storage requirements
+
+When considering a cluster deployment that spans data centers, special consideration needs to be given to the selected storage integration to ensure that it also meets multisite requirements, as it pertains to accessibility from all sites, fault tolerance, high availability, and so on.
+
+An object storage solution should be used for the registry, and this storage solution needs to be in addition to any PV storage integration used for application volumes or workloads. This object storage solution should also have the same special considerations given to accessibility from all sites, fault tolerance, high availability, and so on.
+
+Because disk I/O is a critical factor in the health of etcd database, it is required that they are deployed on a high speed, low latency media. See etcd guidance on _etcd peer round trip time_ and _etcd database size_ for more details on the exact requirements to meet.

modules/etcd-site-recommendations-span.adoc

@@ -0,0 +1,11 @@
+// Module included in the following assemblies:
+//
+// * etcd/etcd-guidance-span.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="site-recommendations-span_{context}"]
+= Site recommendations
+
+Assuming each site gets one control plane member, you theoretically define three sites, which is what Red Hat recommends. This allows for one data center to go into an inactive state and the cluster still maintains quorum and operational consistency.
+
+When this assumption is not met, attention should be given to the desired and actual fault tolerance state of the cluster, as it will often outline or dictate the operational capabilities (uptime and stability) of the deployment.

modules/etcd-workload-placement-considerations-span.adoc

@@ -0,0 +1,11 @@
+// Module included in the following assemblies:
+//
+// * etcd/etcd-guidance-span.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="workload-placement-considerations-span_{context}"]
+= Workload placement considerations
+
+With multisite clusters, administrators and developers must take special considerations into account to ensure that critical workloads are scheduled or placed based on the proper hardware or hosts within the topology of the cluster. This ensures that the applications and services are Highly Available and Fault Tolerant based on the topology of the cluster's deployment.
+
+Without considering this, it is possible for {product-title} to schedule workloads on hosts within the cluster so that a Single Point of Failure (SPoF) is created for {product-title} infrastructure services and other application services if there is a data center outage.