feat: support CCL Scatter

[GordonYang1]

Summary

This PR adds Scatter support to the shared CCL backend abstraction by composing provider-native grouped point-to-point operations through the existing NCCL and MCCL layers for the existing public infinicclScatter() API. It also keeps inter-only communicators on the existing OpenMPI staging path during mixed-backend bootstrap flows, hardens the shared OpenMPI/MPICH movement path with communicator validation, byte-count and overflow checks, and automatic host-buffer cleanup, defines zero-count behavior, makes the existing MPI example validate every destination block and propagate failures, and includes CCL-only plus OpenMPI-assisted Scatter examples with correctness and communication-bandwidth reporting.

Changes

• Public API and Dispatch

  • Enable the existing infinicclScatter() API for configured CCL backends through generated bridge dispatch without changing its public signature.
  • Use a matching native CCL intra communicator when available, and otherwise delegate to the existing OpenMPI provider when the communicator has only an OpenMPI inter communicator.
  • Return success for a zero-count Scatter after validating the communicator, data type, and root, without requiring non-null buffers or entering a backend.
  • Require a receive buffer on every rank for non-zero operations while requiring the send buffer only on the root rank.

• Common CCL Implementation

  • Add a shared provider-oriented CCL Scatter implementation using grouped point-to-point operations because the providers do not expose a native Scatter collective.
  • Send one distinct rank-ordered block from the root to every rank, including the root, and receive one block from the root on every rank within one provider group.
  • Validate the native communicator backend, device, rank, world size, root, handle, and data type before dispatch.
  • Check per-rank and total byte-size calculations for size_t overflow.
  • Preserve the first point-to-point error and always call GroupEnd after a successful GroupStart.

• Existing CCL Provider Bindings

  • Extend the existing NCCL and MCCL API wrappers with GroupStart, GroupEnd, Send, and Recv bindings.
  • Register Scatter with the existing NCCL and MCCL provider layers without introducing a dependency on a vendor-specific Scatter entry point.

• MPI Correctness and Safety

  • Treat Scatter as a movement operation in the shared OpenMPI/MPICH implementation by using MPI_BYTE with count * type_size bytes per rank, preserving every data type including Float16 and BFloat16.
  • Add communicator, rank, world-size, MPI int count-range, and total-buffer-size validation.
  • Allocate the send staging buffer only on the root and manage all host staging buffers with automatic cleanup across success and error paths.
  • Validate each destination rank's complete receive block in the existing MPI example and exchange validation reports so failures propagate through every process exit code.

• Examples, Validation, and Metrics

  • Add a thread-per-GPU single-node CCL Scatter example using one shared unique ID and rank-based communicator initialization.
  • Add an OpenMPI-assisted CCL example that first validates the OpenMPI fallback through an inter-only communicator, then initializes a native CCL communicator and validates the grouped point-to-point path.
  • Validate the complete out-of-place rank-specific receive block in all three Scatter examples.
  • Report per-rank and root-total data sizes, root-rank average time, algorithm bandwidth as world_size * rank_bytes / time, and bus bandwidth as algorithm bandwidth multiplied by (world_size - 1) / world_size.
  • Parse numeric inputs without exceptions and guard example buffer-size calculations against overflow.

Platform and Backend Affected

Platform

• CPU
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU
• HYGON DCU

Backend

• OpenMPI
• MPICH
• NCCL
• MCCL

Performance Impact

• No performance impact
• Performance improved
• Performance regression possible

This adds GPU-native NCCL and MCCL paths for Scatter, avoiding the existing MPI host-staging path when a supported CCL backend and matching native communicator are available. The native path composes grouped point-to-point operations because neither provider exposes a vendor Scatter collective, while the shared MPI fallback remains host-staged with byte-count movement semantics and additional validation and cleanup. Other collective operations are intended to remain unchanged. The validation-log timings below are execution references rather than a quantified cross-backend performance comparison.

Known Issues & Future Work

• The CCL collective backend on this branch adds Scatter alongside the existing AllReduce; other independently developed CCL collectives are outside this branch.
• Scatter inherits the backend/device combinations and data type support of the existing CCL providers; this PR does not add a new provider or device integration.
• The native CCL path uses grouped point-to-point operations rather than a vendor Scatter entry point. The root sends one block per rank while every rank receives once, so root-side work and storage grow linearly with communicator size.
• The server examples validate out-of-place Float32 payloads with root rank 0 on the default stream. Additional data types, roots, in-place layouts, non-default streams, and runtime coverage on Iluvatar and Moore Threads remain future work.
• The shared OpenMPI/MPICH fallback remains host-staged with per-call allocations and rejects per-rank byte counts above the MPI int range; chunked transfers remain future work.

Test Results

The implementation commit is baadfba338f4564e0da038c6d74c96033df0e965, a single commit directly based on ef4045a2d99837c75c2acd90aae57dacb83172d2. The attached evidence contains exactly 15 hash-verified canonical logs from the current commit: all 15 targets passed, with Correct: YES 17 times and Correct: NO 0 times. The extra two positive results are the additional expected validation cases in the broadcast log.

All four Scatter paths passed with a 1,048,576-element Float32 block per rank (4.00 MiB), 2 warm-up iterations, and 20 profiled iterations:

Path	Test topology	Data per rank / total at root	Time	Alg BW	Bus BW
Pure NCCL	8 NVIDIA A100 GPUs	4.00 / 32.00 MiB	0.108 ms	309.66 GB/s	270.95 GB/s
OpenMPI + NCCL hybrid	8 NVIDIA A100 GPUs	4.00 / 32.00 MiB	0.314 ms	106.78 GB/s	93.43 GB/s
Heterogeneous OpenMPI	8 NVIDIA A100 + 8 MetaX C550 GPUs	4.00 / 64.00 MiB	62.226 ms	1.08 GB/s	1.01 GB/s
Pure MCCL	8 MetaX C550 GPUs	4.00 / 32.00 MiB	0.159 ms	211.10 GB/s	184.71 GB/s

The reported times are the root rank's elapsed time averaged over the 20 profiled calls after the 2 warm-up calls. Algorithm bandwidth uses the total bytes scattered by the root, and bus bandwidth applies the (world_size - 1) / world_size correction factor. The values were independently recomputed while accounting for the displayed time being rounded to three decimal places. They are included as execution evidence, not as a cross-platform benchmark comparison.

• The pure NCCL and OpenMPI+NCCL configurations each passed both selected targets, 2/2 and 2/2, on all 8 A100 GPUs. The hybrid Scatter log also confirms that the OpenMPI fallback passed before the native NCCL communicator was initialized.
• The heterogeneous OpenMPI configuration passed all 9 MPI targets on 16 ranks. Ranks 0-7 mapped to NVIDIA devices 0-7, and ranks 8-15 mapped to MetaX devices 0-7.
• The pure MCCL configuration passed both selected targets on all 8 MetaX C550 GPUs. Scatter and the AllReduce baseline ran as two independent single-target invocations.
• All five formal invocations returned zero. There were no harness retries, MetaX vendor queue retries, timeouts, missing canonical logs, or recorded finalization errors.

Test Involved Platform

• CPU
• NVIDIA GPU
• Iluvatar GPU
• MetaX GPU
• Moore Threads GPU
• Cambricon MLU
• HYGON DCU

Test Involved Backend

• OpenMPI
• MPICH
• NCCL
• MCCL

Pure CCL (NCCL) on single-node NVIDIA:
ccl_all_reduce.log
ccl_scatter.log

CCL + MPI on single-node NVIDIA:
ccl_mpi_hybrid_all_reduce.log
ccl_mpi_hybrid_scatter.log

MPI on Heterogeneous Cluster:
mpi_all_gather.log
mpi_all_reduce.log
mpi_all_to_all.log
mpi_broadcast.log
mpi_gather.log
mpi_reduce.log
mpi_reduce_scatter.log
mpi_scatter.log
mpi_send_recv.log

Pure CCL (MCCL) on single-node MetaX:
ccl_all_reduce.log
ccl_scatter.log

---

Checklist

Every contributor must verify every item below before requesting
review. Tick each box only after the check has actually been performed —
do not tick speculatively. If an item truly does not apply, replace the
checkbox with N/A and briefly explain why in an inline comment.

Title, Branch, and Commits

• PR title follows Conventional Commits (e.g. feat: …, fix(nccl): …).
• Branch name follows <type>/xxx-yyyy-zzzz where <type> matches the PR title's Conventional Commits type and words are joined with hyphens (see CONTRIBUTING.md §Branches).
• Each commit message follows Conventional Commits.
• Small PR is a single squashable commit; or, for a large PR, every commit is meaningful, well-formed, and independently reviewable (see CONTRIBUTING.md §Pull Requests).
• No stray merge commits from master — the branch is rebased cleanly on top of the current master.
• No fixup! / squash! / wip commits remain.

Scope and Design

• Changes are minimal — no unrelated modifications were introduced (CONTRIBUTING.md §Code/General).
• No dead code, commented-out blocks, debug prints, printf/std::cout/print(...) left behind, or TODO without an owner and issue link.
• No unrelated formatting churn that would obscure the diff.
• Public API changes (if any) are intentional, documented, and reflected in affected callers/tests.

General Code Hygiene

• The code is self-explanatory; comments were added only where the intent or rationale is non-obvious (CONTRIBUTING.md §Code/General).
• Every modified or added file ends with a single trailing newline (CONTRIBUTING.md §Code/General).
• No trailing whitespace, inconsistent indentation, or mixed formatting styles remain.
• Identifiers referenced in comments or error messages are wrapped in Markdown backticks (e.g. the `AllReduce` implementation) (CONTRIBUTING.md §Code/General).
• All comments and error messages are in English (CONTRIBUTING.md §Code/General).
• Comments and error messages are complete sentences — capitalized first letter, terminal punctuation — unless the language/framework convention says otherwise (CONTRIBUTING.md §Code/General; §Python).

C++ Specific (if C++ files changed)

• Code follows the Google C++ Style Guide strictly.
• clang-format (version 16, per .github/workflows/clang-format.yml) has been run against all modified applicable files; the diff is clean.
• No exceptions are thrown. Error paths use assert with messages that include at least __FILE__, __LINE__, and __func__ (CONTRIBUTING.md §C++).
• Error and warning message wording follows the LLVM Coding Standards (CONTRIBUTING.md §C++).
• N/A- Constructor initializer list order matches member declaration order (CONTRIBUTING.md §C++).
• Exactly one blank line between classes, between classes and functions, and between functions (CONTRIBUTING.md §C++).
• Exactly one blank line between members (functions and variables) within a class (CONTRIBUTING.md §C++).
• Exactly one blank line before and after the contents of a namespace (CONTRIBUTING.md §C++).

Python Specific (if Python files changed)

• N/A- Code is PEP 8 compliant; ruff check passes cleanly on CI (see `.github/workflows/ruff.yml).
• N/A- ruff format --check passes cleanly — if not, run ruff format and commit the result.
• N/A- Comments are complete English sentences, starting with a capital letter and ending with punctuation; Markdown backticks are used for code references (CONTRIBUTING.md §Python).
• N/A- Framework-specific conventions (e.g. lowercase pytest.skip messages without terminal period) are honored where applicable (CONTRIBUTING.md §Python).
• N/A- No blank line between the function signature and the body when there is no docstring or comment (CONTRIBUTING.md §Python).
• N/A- A blank line is present before and after if, for, and similar control-flow statements (CONTRIBUTING.md §Python).
• N/A- A blank line appears before each return, except when it directly follows a control-flow statement like if or for.
• N/A- Docstrings (if any) follow PEP 257 conventions.
• N/A- Type hints are added / kept consistent with the surrounding code.

Testing

• All applicable example programs have been built and tested successfully on at least one supported heterogeneous cluster setup.

Build, CI, and Tooling

• N/A- New backends or devices have been added to auto-detection in CMakeLists.txt under if(AUTO_DETECT_DEVICES) or to if(AUTO_DETECT_BACKENDS) if applicable.
• Both CI workflows (clang-format.yml, ruff.yml) are green locally (or expected to be green on CI).

Documentation

• N/A- README.md, CONTRIBUTING.md, or inline docs updated when behavior, build flags, or developer workflow changed.
• N/A- Any user-visible breaking change is called out explicitly under "Summary" and in the commit/PR title with a ! or BREAKING CHANGE: footer.

Security and Safety

• No secrets, access tokens, internal URLs, customer data, or personal hardware identifiers have been committed.
• N/A- Third-party code is license-compatible and attributed.
• No unsafe pointer arithmetic, uninitialized reads, or missing bounds checks were introduced.