feat: support CCL Gather

[GordonYang1]

Summary

This PR adds Gather support to the shared CCL backend abstraction by composing provider-native grouped point-to-point operations through the existing NCCL and MCCL layers for the existing public infinicclGather() API. It also keeps inter-only communicators on the existing OpenMPI staging path during mixed-backend bootstrap flows, hardens the shared OpenMPI/MPICH movement path with communicator validation, byte-count and overflow checks, and automatic host-buffer cleanup, defines zero-count behavior, makes the existing MPI example validate every received source block and propagate failures, and includes CCL-only plus OpenMPI-assisted Gather examples with correctness and communication-bandwidth reporting.

Changes

• Public API and Dispatch

  • Enable the existing infinicclGather() API for configured CCL backends through generated bridge dispatch without changing its public signature.
  • Use a matching native CCL intra communicator when available, and otherwise delegate to the existing OpenMPI provider when the communicator has only an OpenMPI inter communicator.
  • Return success for a zero-count Gather after validating the communicator, data type, and root, without requiring non-null buffers or entering a backend.
  • Require a send buffer on every rank for non-zero operations while requiring the receive buffer only on the root rank.

• Common CCL Implementation

  • Add a shared provider-oriented CCL Gather implementation using grouped point-to-point operations because the providers do not expose a native Gather collective.
  • Send one contribution from every rank, including the root, and receive every peer block at the root in rank order within one provider group.
  • Validate the native communicator backend, device, rank, world size, root, handle, and data type before dispatch.
  • Check per-rank and total byte-size calculations for size_t overflow.
  • Preserve the first point-to-point error and always call GroupEnd after a successful GroupStart.

• Existing CCL Provider Bindings

  • Extend the existing NCCL and MCCL API wrappers with GroupStart, GroupEnd, Send, and Recv bindings.
  • Register Gather with the existing NCCL and MCCL provider layers without introducing a dependency on a vendor-specific Gather entry point.

• MPI Correctness and Safety

  • Treat Gather as a movement operation in the shared OpenMPI/MPICH implementation by using MPI_BYTE with count * type_size bytes per rank, preserving every data type including Float16 and BFloat16.
  • Add communicator, rank, world-size, MPI int count-range, and total-buffer-size validation.
  • Allocate the receive staging buffer only on the root and manage all host staging buffers with automatic cleanup across success and error paths.
  • Validate every received source-rank block in the existing MPI example and broadcast the validation status so failures propagate through every process exit code.

• Examples, Validation, and Metrics

  • Add a thread-per-GPU single-node CCL Gather example using one shared unique ID and rank-based communicator initialization.
  • Add an OpenMPI-assisted CCL example that first validates the OpenMPI fallback through an inter-only communicator, then initializes a native CCL communicator and validates the grouped point-to-point path.
  • Validate the complete out-of-place rank-ordered receive layout in all three Gather examples.
  • Report per-rank and root-total data sizes, root-rank average time, algorithm bandwidth as world_size * rank_bytes / time, and bus bandwidth as algorithm bandwidth multiplied by (world_size - 1) / world_size.
  • Parse numeric inputs without exceptions and guard example buffer-size calculations against overflow.

Platform and Backend Affected

Platform

• CPU
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU
• HYGON DCU

Backend

• OpenMPI
• MPICH
• NCCL
• MCCL

Performance Impact

• No performance impact
• Performance improved
• Performance regression possible

This adds GPU-native NCCL and MCCL paths for Gather, avoiding the existing MPI host-staging path when a supported CCL backend and matching native communicator are available. The native path composes grouped point-to-point operations because neither provider exposes a vendor Gather collective, while the shared MPI fallback remains host-staged with byte-count movement semantics and additional validation and cleanup. Other collective operations are intended to remain unchanged. The validation-log timings below are execution references rather than a quantified cross-backend performance comparison.

Known Issues & Future Work

• The CCL collective backend on this branch adds Gather alongside the existing AllReduce; other independently developed CCL collectives are outside this branch.
• Gather inherits the backend/device combinations and data type support of the existing CCL providers; this PR does not add a new provider or device integration.
• The native CCL path uses grouped point-to-point operations rather than a vendor Gather entry point. Each rank sends once, while the root receives one block per rank, so root-side work and storage grow linearly with communicator size.
• The server examples validate out-of-place Float32 payloads with root rank 0 on the default stream. Additional data types, in-place layouts, non-default streams, and runtime coverage on Iluvatar and Moore Threads remain future work.
• The shared OpenMPI/MPICH fallback remains host-staged with per-call allocations and rejects per-rank byte counts above the MPI int range; chunked transfers remain future work.

Test Results

The implementation commit is 773a7d32846a547c83effb5f0db9f5d8a208424e, a single commit directly based on ef4045a2d99837c75c2acd90aae57dacb83172d2. The attached evidence contains exactly 15 hash-verified canonical logs from the current commit: all 15 targets passed, with Correct: YES 17 times and Correct: NO 0 times. The extra two positive results are the additional expected validation cases in the broadcast log.

All four Gather paths passed with a 1,048,576-element Float32 contribution per rank (4.00 MiB), 2 warm-up iterations, and 20 profiled iterations:

Path	Test topology	Data per rank / total at root	Time	Alg BW	Bus BW
Pure NCCL	8 NVIDIA A100 GPUs	4.00 / 32.00 MiB	0.141 ms	237.33 GB/s	207.66 GB/s
OpenMPI + NCCL hybrid	8 NVIDIA A100 GPUs	4.00 / 32.00 MiB	0.140 ms	239.26 GB/s	209.35 GB/s
Heterogeneous OpenMPI	8 NVIDIA A100 + 8 MetaX C550 GPUs	4.00 / 64.00 MiB	39.644 ms	1.69 GB/s	1.59 GB/s
Pure MCCL	8 MetaX C550 GPUs	4.00 / 32.00 MiB	0.131 ms	255.98 GB/s	223.98 GB/s

The reported times are the root rank's elapsed time averaged over the 20 profiled calls after the 2 warm-up calls. Algorithm bandwidth uses the total bytes gathered at the root, and bus bandwidth applies the (world_size - 1) / world_size correction factor. The values were independently recomputed while accounting for the displayed time being rounded to three decimal places. They are included as execution evidence, not as a cross-platform benchmark comparison.

• The pure NCCL and OpenMPI+NCCL configurations each passed both selected targets, 2/2 and 2/2, on all 8 A100 GPUs. The hybrid Gather log also confirms that the OpenMPI fallback passed before the native NCCL communicator was initialized.
• The heterogeneous OpenMPI configuration passed all 9 MPI targets on 16 ranks. Ranks 0-7 mapped to NVIDIA devices 0-7, and ranks 8-15 mapped to MetaX devices 0-7.
• The pure MCCL configuration passed both selected targets on all 8 MetaX C550 GPUs, with ranks 0-7 mapped to devices 0-7.
• All six formal invocations returned zero. There were no harness retries, MetaX vendor queue retries, timeouts, missing canonical logs, or recorded finalization errors.

Test Involved Platform

• CPU
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU
• HYGON DCU

Test Involved Backend

• OpenMPI
• MPICH
• NCCL
• MCCL

Pure CCL (NCCL) on single-node NVIDIA:
ccl_all_reduce.log
ccl_gather.log

CCL + MPI on single-node NVIDIA:
ccl_mpi_hybrid_all_reduce.log
ccl_mpi_hybrid_gather.log

MPI on Heterogeneous Cluster:
mpi_all_gather.log
mpi_all_reduce.log
mpi_all_to_all.log
mpi_broadcast.log
mpi_gather.log
mpi_reduce.log
mpi_reduce_scatter.log
mpi_scatter.log
mpi_send_recv.log

Pure CCL (MCCL) on single-node MetaX:
ccl_all_reduce.log
ccl_gather.log

---

Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

• PR title follows Conventional Commits (e.g. feat: …, fix(nccl): …).
• Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
• Each commit message follows Conventional Commits.
• Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
• No stray merge commits from master — the branch is rebased cleanly on top of the current master.
• No fixup! / squash! / wip commits remain.

Scope and Design

• Changes are minimal — no unrelated modifications were introduced (CONTRIBUTING.md §Code/General).
• No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
• No unrelated formatting churn that would obscure the diff.
• Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene

• The code is self-explanatory; comments were added only where the intent or rationale is non-obvious (CONTRIBUTING.md §Code/General).
• Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
• No trailing whitespace, inconsistent indentation, or mixed formatting styles remain.
• Identifiers referenced in comments or error messages are wrapped in Markdown backticks (e.g. the `AllReduce` implementation) (CONTRIBUTING.md §Code/General).
• All comments and error messages are in English (CONTRIBUTING.md §Code/General).
• Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

• Code follows the Google C++ Style Guide strictly.
• clang-format (version 16, per .github/workflows/clang-format.yml) has been run against all modified applicable files; the diff is clean.
• No exceptions are thrown. Error paths use assert with messages that include at least __FILE__, __LINE__, and __func__ (CONTRIBUTING.md §C++).
• Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
• N/A- Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++).
• Exactly one blank line between classes, between classes and functions, and between functions (CONTRIBUTING.md §C++).
• Exactly one blank line between members (functions and variables) within a class (CONTRIBUTING.md §C++).
• Exactly one blank line before and after the contents of a namespace (CONTRIBUTING.md §C++).

Python Specific (if Python files changed)

• N/A- Code is PEP 8 compliant; ruff check passes cleanly on CI (see `.github/workflows/ruff.yml).
• N/A- ruff format --check passes cleanly — if not, run ruff format and commit the result.
• N/A- Comments are complete English sentences, starting with a capital letter and ending with punctuation; Markdown backticks are used for code references (CONTRIBUTING.md §Python).
• N/A- Framework-specific conventions (e.g. lowercase pytest.skip messages without terminal period) are honored where applicable (CONTRIBUTING.md §Python).
• N/A- No blank line between the function signature and the body when there is no docstring or comment (CONTRIBUTING.md §Python).
• N/A- A blank line is present before and after if, for, and similar control-flow statements (CONTRIBUTING.md §Python).
• N/A- A blank line appears before each return, except when it directly follows a control-flow statement (CONTRIBUTING.md §Python).
• N/A- Docstrings (if any) follow PEP 257 conventions.
• N/A- Type hints are added / kept consistent with the surrounding code.

Testing

• All applicable example programs have been built and tested successfully on at least one supported heterogeneous cluster setup.

Build, CI, and Tooling

• N/A- New backends or devices have been added to auto-detection in CMakeLists.txt under if(AUTO_DETECT_DEVICES) or to if(AUTO_DETECT_BACKENDS) if applicable.
• Both CI workflows (clang-format.yml, ruff.yml) are green locally (or expected to be green on CI).

Documentation

• N/A- README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
• N/A- Any user-visible breaking change is called out explicitly under "Summary" and in the commit/PR title with a ! or BREAKING CHANGE: footer.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• N/A- Third-party code is license-compatible and attributed.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.