OSDOCS-14356-New: Added bond best practices info to networking docs

Author: dfitzmau

_topic_maps/_topic_map.yml

@@ -1739,6 +1739,8 @@ Topics:
     File: verifying-connectivity-endpoint
   - Name: Changing the cluster network MTU
     File: changing-cluster-network-mtu
+  - Name: Network bonding consideration
+    File: network-bonding-considerations
   - Name: Using Stream Control Transmission Protocol
     File: using-sctp
   - Name: Associating secondary interfaces metrics to network attachments

installing/installing_bare_metal/bare-metal-postinstallation-configuration.adoc

@@ -34,9 +34,6 @@ include::modules/creating-manifest-file-customized-br-ex-bridge.adoc[leveloffset
 
 * xref:../../installing/installing_bare_metal/bare-metal-expanding-the-cluster.adoc#bare-metal-expanding-the-cluster[Expanding the cluster]
 
-// Enabling OVS balance-slb mode for your cluster
-include::modules/enabling-OVS-balance-slb-mode.adoc[leveloffset=+1]
-
 // Services for a user-managed load balancer
 include::modules/nw-osp-services-external-load-balancer.adoc[leveloffset=+1]
 

installing/installing_bare_metal/ipi/ipi-install-installation-workflow.adoc

@@ -38,9 +38,6 @@ include::modules/creating-manifest-file-customized-br-ex-bridge.adoc[leveloffset
 // Scale each machine set to compute nodes
 include::modules/creating-scaling-machine-sets-compute-nodes-networking.adoc[leveloffset=+2]
 
-// Enabling OVS balance-slb mode for your cluster
-include::modules/enabling-OVS-balance-slb-mode.adoc[leveloffset=+1]
-
 // Establishing communication between subnets
 include::modules/ipi-install-establishing-communication-between-subnets.adoc[leveloffset=+1]
 

installing/installing_bare_metal/upi/installing-bare-metal-network-customizations.adoc

@@ -79,9 +79,6 @@ include::modules/creating-manifest-file-customized-br-ex-bridge.adoc[leveloffset
 // Scale each machine set to compute nodes
 include::modules/creating-scaling-machine-sets-compute-nodes-networking.adoc[leveloffset=+2]
 
-// Enabling OVS balance-slb mode for your cluster
-include::modules/enabling-OVS-balance-slb-mode.adoc[leveloffset=+1]
-
 include::modules/installation-infrastructure-user-infra.adoc[leveloffset=+1]
 
 [role="_additional-resources"]

installing/installing_bare_metal/upi/installing-bare-metal.adoc

@@ -100,9 +100,6 @@ include::modules/creating-manifest-file-customized-br-ex-bridge.adoc[leveloffset
 // Scale each machine set to compute nodes
 include::modules/creating-scaling-machine-sets-compute-nodes-networking.adoc[leveloffset=+2]
 
-// Enabling OVS balance-slb mode for your cluster
-include::modules/enabling-OVS-balance-slb-mode.adoc[leveloffset=+1]
-
 include::modules/installation-infrastructure-user-infra.adoc[leveloffset=+1]
 
 [role="_additional-resources"]

installing/installing_bare_metal/upi/installing-restricted-networks-bare-metal.adoc

@@ -94,9 +94,6 @@ include::modules/creating-manifest-file-customized-br-ex-bridge.adoc[leveloffset
 // Scale each machine set to compute nodes
 include::modules/creating-scaling-machine-sets-compute-nodes-networking.adoc[leveloffset=+2]
 
-// Enabling OVS balance-slb mode for your cluster
-include::modules/enabling-OVS-balance-slb-mode.adoc[leveloffset=+1]
-
 include::modules/installation-infrastructure-user-infra.adoc[leveloffset=+1]
 
 [role="_additional-resources"]

modules/enabling-OVS-balance-slb-mode.adoc

@@ -1,26 +1,27 @@
 // Module included in the following assemblies:
 //
-// IPI
-// * installing/installing_bare_metal/ipi/ipi-install-installation-workflow.adoc
-// * installing/installing_bare_metal/ipi/bare-metal-postinstallation-configuration.adoc
-// UPI
-// * installing/installing_bare_metal/upi/installing-bare-metal-network-customizations.adoc
-// * installing/installing_bare_metal/upi/installing-restricted-networks-bare-metal.adoc
-// * installing/installing_bare_metal/upi/installing-bare-metal.adoc
+// * networking/advanced_networking/network-bonding-considerations.adoc
 
 :_mod-docs-content-type: PROCEDURE
 [id="enabling-OVS-balance-slb-mode_{context}"]
 = Enabling OVS balance-slb mode for your cluster
 
 You can enable the Open vSwitch (OVS) `balance-slb` mode on infrastructure where your cluster runs so that two or more physical interfaces can share their network traffic. A `balance-slb` mode interface provides source load balancing (SLB) capabilities for a cluster that runs virtualization workloads, where the interface can act independently without needing to communicate with a network switch.
 
-Currently, source load balancing works by assigning a Media Access Control (MAC) address and a virtual local area network (vLAN), if required, to a bond interface, such as `br-phy`. Because of the shared MAC address and vLAN between interfaces, using `balance-slb` mode to share pod traffic has no benefit.
+Currently, source load balancing works by assigning a Media Access Control (MAC) address and a virtual local area network (VLAN), if required, to a bond interface, such as `br-phy`. Because of the shared MAC address and VLAN between interfaces, using `balance-slb` mode to share pod traffic has no benefit.
 
 The following diagram shows `balance-slb` mode on a simple cluster infrastructure layout. Virtual machines (VMs) connect to specific localnet `NetworkAttachmentDefinition` (NAD) custom resource definition (CRDs), `NAD 0` or `NAD 1`. Each NAD provides VMs with access to network traffic, such as VLAN ID tags. A `br-ex` OVS bridge receives traffic from VMs and passes the traffic to the next OVS bridge, `br-phy`. The `br-phy` bridge functions as the controller for the SLB bond. The SLB bond balances traffic from different VM ports over the physical interface links, such as `eno0` and `eno1`. Additionally, ingress traffic from either physical interface can pass through the set of OVS bridges to reach the VMs.
 
 .OVS `balance-slb` mode ` operating on a localnet with two NADs
 image::552_OpenShift_slb_mode_0625.png[OVS `balance-slb` mode ` operating on a localnet with two NADs]
 
+[NOTE]
+====
+Consider that an OVS bond with `balance-slb` mode enabled might experience issues if the bond forwards unknown unicast traffic from one physical network interface controller (NIC) into the phsycial network through another NIC. This situation can result in an Layer 2 loop, or _bridge loop_, that in turn causes _MAC flapping_. 
+
+MAC flapping causes the same MAC address to exist in many network locations for a period of time. For example, a remote switch does not learn the MAC address for the destination of a unicast packet and this causes the packet to exist on all links available on the SLB bond configuration. As a workaround for this issue, you can set the bond to `active-backup` mode during MAC address assignment and then switch the bond to use `balance-slb` mode.
+====
+
 You can integrate the `balance-slb` mode interface into primary or secondary network types by using OVS bonding. Note the following points about OVS bonding:
 
 * Supports the OVN-Kubernetes CNI plugin and easily integrates with the plugin.
@@ -67,13 +68,13 @@ networkConfig:
         enabled: false
 # ...
 ----
-<1> The interface for the provisioned network interface card (NIC). 
+<1> The interface for the provisioned network interface controller (NIC). 
 <2> The first bonded interface that pulls in the Ignition config file for the bond interface.
 <3> The second bonded interface is part of a minimal configuration that pulls ignition during cluster installation.
 
 . Define each network interface in a `MachineConfig` manifest file:
 +
-.Example `MachineConfig` manifest file that defines multiple network interfaces
+.Example `MachineConfig` manifest file that defines many network interfaces
 [source,yaml]
 ----
 # ...

modules/installation-network-user-infra.adoc

@@ -87,19 +87,23 @@ endif::ibm-z[]
 ifndef::ibm-z[]
 During the initial boot, the machines require an IP address configuration that is set either through a DHCP server or statically by providing the required boot options. After a network connection is established, the machines download their Ignition config files from an HTTP or HTTPS server. The Ignition config files are then used to set the exact state of each machine. The Machine Config Operator completes more changes to the machines, such as the application of new certificates or keys, after installation.
 
+Use a DHCP server for the long-term management of the machines for your cluster. Ensure that the DHCP server is configured to provide persistent IP addresses, DNS server information, and hostnames to the cluster machines. As a cluster administrator, ensure that you reserve the following IP addresses components that interact with the DHCP server:
+
+* Two unique virtual IP (VIP) addresses. One VIP address for the API endpoint and one VIP address for the wildcard ingress endpoint.
+* One IP address for the provisioner node. 
+* An IP address for each control plane node.
+* An IP address for each compute node.
+If you have multiple network interfaces that interact with a bonded interface, reserve the same IP addresses for these multiple network interfaces so to ensure better load balancing, fault tolerance, and bandwidth capabilites for your cluster network infrastructure. 
+
 [NOTE]
 ====
-* It is recommended to use a DHCP server for long-term management of the cluster machines. Ensure that the DHCP server is configured to provide persistent IP addresses, DNS server information, and hostnames to the cluster machines.
+* Consider using a DHCP server for long-term management of the cluster machines. Ensure that the DHCP server is configured to provide persistent IP addresses, DNS server information, and hostnames to the cluster machines.
 
 * If a DHCP service is not available for your user-provisioned infrastructure, you can instead provide the IP networking configuration and the address of the DNS server to the nodes at {op-system} install time. These can be passed as boot arguments if you are installing from an ISO image. See the _Installing {op-system} and starting the {product-title} bootstrap process_ section for more information about static IP provisioning and advanced networking options.
 ====
 endif::ibm-z[]
 
-The Kubernetes API server must be able to resolve the node names of the cluster
-machines. If the API servers and worker nodes are in different zones, you can
-configure a default DNS search zone to allow the API server to resolve the
-node names. Another supported approach is to always refer to hosts by their
-fully-qualified domain names in both the node objects and all DNS requests.
+The Kubernetes API server must be able to resolve the node names of the cluster machines. If the API servers and worker nodes are in different zones, you can configure a default DNS search zone to allow the API server to resolve the node names. Another supported approach is to always refer to hosts by their fully-qualified domain names in both the node objects and all DNS requests.
 endif::azure,gcp[]
 
 ifndef::ibm-z,azure[]
@@ -114,9 +118,7 @@ endif::ibm-z,azure[]
 [id="installation-network-connectivity-user-infra_{context}"]
 == Network connectivity requirements
 
-You must configure the network connectivity between machines to allow {product-title} cluster
-components to communicate. Each machine must be able to resolve the hostnames
-of all other machines in the cluster.
+You must configure the network connectivity between machines to allow {product-title} cluster components to communicate. Each machine must be able to resolve the hostnames of all other machines in the cluster.
 
 This section provides details about the ports that are required.
 

modules/installation-user-infra-machines-static-network.adoc

@@ -220,7 +220,9 @@ ifndef::ibm-z-kvm[]
 [discrete]
 === Bonding multiple network interfaces to a single interface
 
-Optional: You can bond multiple network interfaces to a single interface by using the `bond=` option. Refer to the following examples:
+As an optional configuration, you can bond multiple network interfaces to a single interface by using the `bond=` option. To apply this configuration to your cluster, complete the procedure steps for each node that runs on your cluster.
+
+.Procedure
 
 * The syntax for configuring a bonded interface is: `bond=<name>[:<network_interfaces>][:options]`
 +
@@ -287,9 +289,9 @@ ifndef::ibm-z[]
 [discrete]
 === Bonding multiple SR-IOV network interfaces to a dual port NIC interface
 
-Optional: You can bond multiple SR-IOV network interfaces to a dual port NIC interface by using the `bond=` option.
+As an optional configuration, you can bond multiple SR-IOV network interfaces to a dual port NIC interface by using the `bond=` option.
 
-On each node, you must perform the following tasks:
+.Procedure
 
 ifndef::installing-ibm-power[]
 . Create the SR-IOV virtual functions (VFs) following the guidance in link:https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/configuring_and_managing_virtualization/managing-virtual-devices_configuring-and-managing-virtualization#managing-sr-iov-devices_managing-virtual-devices[Managing SR-IOV devices]. Follow the procedure in the "Attaching SR-IOV networking devices to virtual machines" section.
@@ -308,13 +310,19 @@ The following examples illustrate the syntax you must use:
 
 * When you create a bonded interface using `bond=`, you must specify how the IP address is assigned and other information for the bonded interface.
 
-** To configure the bonded interface to use DHCP, set the bond's IP address to `dhcp`. For example:
+** To configure the bonded interface to use DHCP, set the `ip` parameter to `dhcp` as demonstrated in the following example:
 +
 [source,terminal]
 ----
 bond=bond0:eno1f0,eno2f0:mode=active-backup
 ip=bond0:dhcp
+fail_over_mac=0
 ----
++
+[NOTE]
+====
+The `fail_over_mac=0` default setting ensures that all secondary interfaces for a bond use the MAC address of that bond. Do not specify `1`, `active`, or `2`, follow`, because Red{nbsp}Hat does not support these configuration values.
+====
 
 ** To configure the bonded interface to use a static IP address, enter the specific IP address you want and related information. For example:
 +

modules/nw-ovs-bonding.adoc

@@ -0,0 +1,26 @@
+// Module included in the following assemblies:
+//
+// * networking/advanced_networking/network-bonding-considerations.adoc
+
+:_mod-docs-content-type: CONCEPT
+[id="nw-ovs-bonding_{context}"]
+= Open vSwitch (OVS) bonding
+
+With an OVS bonding configuration on your network, each physical interface acts as a port and connects to a specific bond. A bond then connects to a virtual switch or an OVS bridge. This connection layout provides increased bandwidth and fault tolerance capabilities for traffic that runs on your network. 
+
+Consider the following architectural layout for OVS bridges that interact with OVS interfaces:
+
+* A network interface uses a bridge MAC address for managing protocol-level traffic and other administrative tasks, such as IP address assignment.
+* The physical MAC addresses of physical interfaces do not handle traffic.
+* OVS handles all MAC address management at the OVS bridge level.
+
+This layout simplies bond interface management as bonds acts as data paths where centralized MAC address managements happens at the OVS bridge level.
+
+You can select the following OVS bonding modes for your network:
+
+* `active-backup` mode provides link aggregation capabilities for your network, where one physical interface acts as the active port while other physical interfaces act as standby ports. This mode provides fault tolerance connections for your network.
+* `kernel-bonding` mode is a built-in Linux kernel function where link aggregation can exist among many Ethernet interfaces to create a single logical physical interface. This mode does not give the same level of customization as supported OVS mode, such as `balance-slb` mode.
+* `balance-slb` mode, where an interface provides source load balancing (SLB) capabilities for a cluster that runs virtualization workloads. The interface can act independently without needing to communicate with a network switch.
+
+For `kernel-bonding` mode, the bond interfaces exist outside, which means they are not in the data path, of the bridge interface. Network traffic in this mode is not sent or received on the bond interface port but instead requires additional bridging capabilities for MAC address assignment at the kernel level. For `active-backup` and `balance-slb` modes, the bond interfaces exist in the same data path as the OVS bridge interface, so the OVS bridge can manage bonding logic instead of the physcial interfaces manages traffic. 
+