Ref: https://portal.nutanix.com/page/documents/details?targetId=Files-v3_6:fil-files-high-availability-c.html

When nodes are mistakenly configured with incorrect IP addresses and are on a different network, the most effective way to address this is to re-run the Foundation process. By using Foundation, the nodes can be completely re-imaged with the correct network settings and then properly integrated into a new cluster. This ensures that all hardware and software configurations align with the intended network and cluster settings, avoiding potential conflicts and issues that might arise from manual adjustments.
References:
* Nutanix Bible: Cluster Initialization and Foundation
* Nutanix University: Using Foundation for Node Deployment

Given the critical message about the NIC being disconnected or the switchport failing, the consultant should first verify the bonding mode of the NICs, especially since only two out of four 10GbE NICs are connected. In an AHV cluster, the NICs can be configured in different bonding modes such as active-active or active-backup. Ensuring that the NICs are configured for active-backup can provide redundancy by allowing one NIC to take over if the other fails, which seems to be necessary given the critical error indicating a possible disconnection or switchport failure. References: Nutanix Bible (https://www.nutanixbible.com/), Nutanix University (https://university.nutanix.com/)

In a situation where traffic is not being balanced across the physical NICs and the Top-of-Rack (ToR) switches have been configured with LACP (Link Aggregation Control Protocol), the appropriate AHV network configuration to resolve this issue within Prism Element is "Active-Active." This setting enables both NICs to actively handle traffic, which, combined with LACP, allows for load balancing and redundancy, effectively utilizing all available network paths. This setup ensures that traffic is spread evenly across the physical NICs, maximizing network throughput and resilience.
References:
* Nutanix Prism User Guide
* Nutanix AHV Administration Guide
In a scenario where a new AHV cluster is installed and the traffic is not being balanced across the physical NICs despite the ToR switches being configured with LACP, the appropriate network option to set within Prism Element is Active-Active. This configuration allows for a traffic-hashing algorithm, such as balance-TCP, to split traffic between multiple links in an active-active fashion. Since the uplinks appear as a single L2 link, the algorithm can balance the traffic among bond members without any regard for switch MAC address tables1.
References:
* Nutanix Community: AHV Networking | Enable, Disable and Verify LACP1
* Nutanix Support & Insights: Layer 2 Network Management2
* Nutanix Support & Insights: Enabling LAG and LACP on the ToR Switch (AHV Only)