RFC-0047-flagcx-support [commenting]

[MC952-arch]

Proposal to Add FlagCX backend to torch.distributed for Heterogeneous Cross-chip Communication.

Rendered version: https://github.com/MC952-arch/rfcs/blob/flagcx-support/RFC-0047-flagcx-support.md

[meta-cla]

Hi @MC952-arch!

Thank you for your pull request and welcome to our community.

Action Required

In order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you.

Process

In order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA.

Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with CLA signed. The tagging process may take up to 1 hour after signing. Please give it that time before contacting us about it.

If you have received this in error or have any questions, please contact us at cla@meta.com. Thanks!

[meta-cla]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks!

[albanD]

cc @wconstab @d4l3k

[d4l3k]

Hi, have you looked into implementing this as an out of tree backend (either for ProcessGroup or for torchcomms)?

[MC952-arch]

Hi, have you looked into implementing this as an out of tree backend (either for ProcessGroup or for torchcomms)?

Yes — we already have one. There's a working out-of-tree backend at plugin/torch/ that implements ProcessGroupFlagCX as a c10d::Backend subclass, installable as a standalone Python package. The RFC is about bringing it optionally in-tree, and the reason comes down to what FlagCX actually does that no existing in-tree backend does.

Comparison with UCC

UCC is the closest comparison. It's a solid multi-vendor framework, but its heterogeneous support works by decomposing collectives into homogeneous subgroups: CL hier splits ranks into NODE / NODE_LEADERS / NET teams, each backed by a single TL (tl/nccl for NVIDIA, tl/rccl for AMD, etc.). Cross-vendor traffic still routes through tl/ucp over the network. There's no way to put NVIDIA + Ascend + Cambricon ranks in one process group and get vendor-native intra-cluster acceleration plus direct device-buffer RDMA for the cross-cluster hops.

That's exactly what FlagCX does.

Heterogeneous communicator topology. At initialization time, FlagCX tries to detect vendor strings across all ranks, partitions them into clusters (same-vendor groups), and builds both a homoComm (vendor-native CCL: NCCL, HCCL, CNCL, etc.) and a heteroComm (FlagCX's own RDMA transport) inside a single communicator. One AllReduce call uses NCCL within the NVIDIA cluster, HCCL within the Ascend cluster, and FlagCX's RDMA transport between them — all under one process group, transparent to the caller.

Device-buffer RDMA. Cross-cluster transfers go through GDR-allocated buffers and IB RDMA directly — no D2H staging, no host collective, no H2D copy back. The IB adaptor has a full retransmission protocol (sequence numbers, SACK bitmaps, RTT estimation) for adaptive routing fabrics.

Vendor coverage. FlagCX supports 11 backends: NVIDIA, AMD, Ascend, Iluvatar Corex, Cambricon, Metax, Musa, Kunlunxin, DU, TSMicro, Enflame. That covers a much broader AI chip ecosystem that UCC doesn't reach.

Why optionally in-tree

The out-of-tree plugin works today — torch.distributed.init_process_group(backend="flagcx") runs fine with it.
Making it optionally in-tree provides the same benefits demonstrated by UCC, while preserving flexibility:

• CI coverage so regressions get caught on PRs
• Improved compatibility guarantees across PyTorch versions, which is essential for heterogeneous training/inferencing using FlagCX as backend
• Reduced version-skew for users on heterogeneous clusters

Users who do not enable or install FlagCX see no change.