feat: support CCL AllToAll

[GordonYang1]

Summary

This PR adds AllToAll support to the shared CCL backend abstraction by composing provider-native grouped point-to-point operations through the existing NCCL and MCCL layers for the public infinicclAllToAll() API. It also keeps inter-only communicators on the existing OpenMPI staging path during mixed-backend bootstrap flows, corrects the shared OpenMPI/MPICH movement path to use byte counts for all data types, defines zero-count behavior, makes the existing MPI example validate every received source block and propagate failures, and includes CCL-only plus OpenMPI-assisted AllToAll example programs.

Changes

• Public API and Dispatch

  • Enable the existing infinicclAllToAll() API for configured CCL backends through generated bridge dispatch without changing its public signature.
  • Use a matching native CCL intra communicator when available, and otherwise delegate to the existing OpenMPI provider when the communicator has only an OpenMPI inter communicator.
  • Return success for a zero-count AllToAll after validating the communicator and data type, without requiring non-null buffers or entering a backend.
  • Reject non-zero in-place buffers on the native CCL path with NotSupported.

• Common CCL Implementation

  • Add a shared provider-oriented CCL AllToAll implementation using one grouped Send/Recv pair per peer, including self.
  • Validate the native communicator backend, device, rank, world size, and handle before dispatch.
  • Map InfiniCCL data types through the configured provider and check per-peer and total byte-size calculations for size_t overflow.
  • Preserve the first point-to-point error, continue issuing operations for the remaining peers, and always call GroupEnd after a successful GroupStart.

• Existing CCL Provider Bindings

  • Extend the existing NCCL and MCCL API wrappers with GroupStart, GroupEnd, Send, and Recv bindings.
  • Register AllToAll with the existing NCCL and MCCL provider layers without introducing a dependency on a vendor-specific AllToAll entry point.

• MPI Correctness and Safety

  • Treat AllToAll as a movement operation in the shared OpenMPI/MPICH implementation by using MPI_BYTE with count * type_size bytes per peer.
  • Add communicator, world-size, size_t multiplication, total-buffer-size, and MPI int count-range checks.
  • Manage host staging buffers with automatic cleanup across success and error paths.
  • Validate every received source block in the existing MPI example and exchange validation status so failures propagate through the process exit code.

• Examples and Validation

  • Add a thread-per-GPU single-node CCL AllToAll example using one shared unique ID and rank-based communicator initialization.
  • Add an OpenMPI-assisted CCL example that first uses the public AllToAll API through the OpenMPI inter communicator to distribute the native unique ID, then initializes the native CCL communicator for grouped GPU point-to-point exchange.
  • Validate the complete out-of-place receive layout with source/destination-specific values on every rank.
  • Propagate validation failures across all ranks through shared atomic state in the thread-per-GPU example and a second public AllToAll status exchange in the process-based examples.
  • Report per-peer and per-rank data sizes, rank-0 average time, algorithmic bandwidth as world_size * peer_bytes / time, and bus bandwidth as algorithmic bandwidth multiplied by (world_size - 1) / world_size.
  • Parse numeric options without exceptions and guard example buffer-size calculations against overflow.

Platform and Backend Affected

Platform

• CPU
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU
• HYGON DCU

Backend

• OpenMPI
• MPICH
• NCCL
• MCCL

Performance Impact

• No performance impact
• Performance improved
• Performance regression possible

This adds GPU-native NCCL and MCCL paths for AllToAll, avoiding the existing MPI host-staging path when a supported CCL backend and matching native communicator are available. The native implementation uses grouped point-to-point operations because neither provider exposes a vendor AllToAll entry point, while the shared MPI fallback remains host-staged with corrected byte-count handling. Other collective operations are intended to remain unchanged. The validation-log timing output is reference data rather than a quantified cross-backend performance comparison.

Known Issues & Future Work

• The CCL collective backend on this branch now covers AllReduce and AllToAll; other CCL collective operations remain future work.
• AllToAll inherits the backend/device combinations and data type support of the existing CCL providers; this PR does not add a new provider or device integration.
• The native CCL path uses one grouped Send/Recv pair per peer, including self, rather than a vendor-specific AllToAll entry point. The number of point-to-point operations therefore grows linearly per rank and quadratically across the communicator.
• Non-zero in-place buffers are not supported by the native CCL path.
• The server examples validate out-of-place Float32 payloads on the default stream. Additional data types, non-default streams, and runtime coverage on Iluvatar and Moore Threads remain future work.
• The shared OpenMPI/MPICH fallback remains host-staged with per-call allocations and rejects per-peer byte counts above the MPI int range; chunked transfers remain future work.

Test Results

The implementation commit is a4ada89dffe7b22afd595124df304efc0f5ce7ac, a single commit directly based on ef4045a2d99837c75c2acd90aae57dacb83172d2. The attached composite evidence contains exactly 15 locally hash-verified canonical logs with Correct: YES 17 times and Correct: NO 0 times. The extra two positive results come from the three expected validation cases in the broadcast log.

All four current-commit AllToAll paths passed:

Path	Test topology	Data per peer / rank	Time	Alg BW	Bus BW
Pure NCCL	8 NVIDIA A100 GPUs	4.00 / 32.00 MiB	0.204 ms	164.54 GB/s	143.97 GB/s
OpenMPI + NCCL hybrid	8 NVIDIA A100 GPUs	4.00 / 32.00 MiB	0.211 ms	159.40 GB/s	139.48 GB/s
Heterogeneous OpenMPI	3 NVIDIA A100 + 3 MetaX C550 GPUs	1.00 / 6.00 MiB	11.216 ms	0.56 GB/s	0.47 GB/s
Pure MCCL	3 MetaX C550 GPUs	4.00 / 12.00 MiB	0.169 ms	74.65 GB/s	49.76 GB/s

The reported times are the rank-0 averages over 20 profiled payload calls after 2 warm-up calls. They are included as execution evidence, not as a cross-platform performance comparison or an all-rank aggregate.

• The current-commit NVIDIA NCCL and OpenMPI+NCCL invocations each passed both selected targets, 2/2 and 2/2, on 8 GPUs.
• The heterogeneous evidence combines the current-commit reduced AllToAll run with eight unchanged baseline targets from ec6caa83cbb1abb02043988f8a7c1977e0b04990, using the same parent commit. The current AllToAll run used NVIDIA GPUs 0-2 and a MetaX configuration restricted to physical GPUs 5-7.
• The pure MCCL evidence combines a current-commit reduced 3-GPU AllToAll pass with an 8-GPU standalone AllReduce pass from ec6caa8. The current MCCL run was restricted to physical GPUs 5-7, but it completed too quickly for mx-smi to capture the live placement, so only the configured placement is claimed.
• The standalone MCCL AllReduce log contains seven vendor-internal queue-creation Retrying diagnostics followed by a successful recovery, Correct: YES, matching 36.00 expected/actual values, and safe finalization. These messages are not harness retries.

The difference from ec6caa8 to the current commit is limited to local performance reporting in the three AllToAll examples. This evidence therefore establishes current-commit success for all four AllToAll paths and regression coverage for every attached target, but it does not claim a single strict current-commit 8-GPU 15/15 matrix run.

Test Involved Platform

• CPU
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU
• HYGON DCU

Test Involved Backend

• OpenMPI
• MPICH
• NCCL
• MCCL

Pure CCL (NCCL) on single-node NVIDIA:
ccl_all_reduce.log
ccl_all_to_all.log

CCL + MPI on single-node NVIDIA:
ccl_mpi_hybrid_all_reduce.log
ccl_mpi_hybrid_all_to_all.log

MPI on Heterogeneous Cluster:
mpi_all_gather.log
mpi_all_reduce.log
mpi_all_to_all.log
mpi_broadcast.log
mpi_gather.log
mpi_reduce.log
mpi_reduce_scatter.log
mpi_scatter.log
mpi_send_recv.log

Pure CCL (MCCL) on single-node MetaX — composite evidence:
ccl_all_reduce.log
ccl_all_to_all.log

---

Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

• PR title follows Conventional Commits (e.g. feat: …, fix(nccl): …).
• Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
• Each commit message follows Conventional Commits.
• Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
• No stray merge commits from master — the branch is rebased cleanly on top of the current master.
• No fixup! / squash! / wip commits remain.

Scope and Design

• Changes are minimal — no unrelated modifications were introduced (CONTRIBUTING.md §Code/General).
• No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
• No unrelated formatting churn that would obscure the diff.
• Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene

• The code is self-explanatory; comments were added only where the intent or rationale is non-obvious (CONTRIBUTING.md §Code/General).
• Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
• No trailing whitespace, inconsistent indentation, or mixed formatting styles remain.
• Identifiers referenced in comments or error messages are wrapped in Markdown backticks (e.g. the `AllReduce` implementation) (CONTRIBUTING.md §Code/General).
• All comments and error messages are in English (CONTRIBUTING.md §Code/General).
• Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

• Code follows the Google C++ Style Guide strictly.
• clang-format (version 16, per .github/workflows/clang-format.yml) has been run against all modified applicable files; the diff is clean.
• No exceptions are thrown. Error paths use assert with messages that include at least __FILE__, __LINE__, and __func__ (CONTRIBUTING.md §C++).
• Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
• N/A- Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++).
• Exactly one blank line between classes, between classes and functions, and between functions (CONTRIBUTING.md §C++).
• Exactly one blank line between members (functions and variables) within a class (CONTRIBUTING.md §C++).
• Exactly one blank line before and after the contents of a namespace (CONTRIBUTING.md §C++).

Python Specific (if Python files changed)

• N/A- Code is PEP 8 compliant; ruff check passes cleanly on CI (see `.github/workflows/ruff.yml).
• N/A- ruff format --check passes cleanly — if not, run ruff format and commit the result.
• N/A- Comments are complete English sentences, starting with a capital letter and ending with punctuation; Markdown backticks are used for code references (CONTRIBUTING.md §Python).
• N/A- Framework-specific conventions (e.g. lowercase pytest.skip messages without terminal period) are honored where applicable (CONTRIBUTING.md §Python).
• N/A- No blank line between the function signature and the body when there is no docstring or comment (CONTRIBUTING.md §Python).
• N/A- A blank line is present before and after if, for, and similar control-flow statements (CONTRIBUTING.md §Python).
• N/A- A blank line appears before each return, except when it directly follows a control-flow statement (CONTRIBUTING.md §Python).
• N/A- Docstrings (if any) follow PEP 257 (CONTRIBUTING.md §Python).
• N/A- Type hints are added / kept consistent with the surrounding code.

Testing

• All applicable example programs have been built and tested successfully on at least one supported heterogeneous cluster setup.

Build, CI, and Tooling

• N/A- New backends or devices have been added to auto-detection in CMakeLists.txt under if(AUTO_DETECT_DEVICES) or to if(AUTO_DETECT_BACKENDS) if applicable.
• Both CI workflows (clang-format.yml, ruff.yml) are green locally (or expected to be green on CI).

Documentation

• N/A- README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
• N/A- Any user-visible breaking change is called out explicitly under "Summary" and in the commit/PR title with a ! or BREAKING CHANGE: footer.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• N/A- Third-party code is license-compatible and attributed.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.