nccl v2.31.2-1

Optimized primitives for collective multi-GPU communication

Release notes

## Compute Fabric Transport (CFT)

- Adds CFT host and Device APIs for registering window memory with CUDA logical endpoints and issuing device-side Put, Get, and NVLS operations.
- Supports CFT on Blackwell GPUs with CUDA Toolkit 13.3 or later.

## Per-Collective Configuration and Tuning

- Adds the new `ncclCollConfig_t` type and `nccl*Config` APIs for all collectives, including a vendor-defined field that makes custom-fork integration easier.
- Supports per-collective algorithm selection, CTA/CGA size overrides, and CTA policy overrides.
- Uses `collConfig` to add `userTag` to the profiler API, motivated by [GitHub Issue #1916](https://github.com/NVIDIA/nccl/issues/1916).

## GIN Enhancements

- Adds the EFA GDA backend for GIN Put, Signal, and Flush operations, contributed by the AWS EFA team ([GitHub PR #2273](https://github.com/NVIDIA/nccl/pull/2273)).
- Adds per-DevComm GIN backend selection, allowing users to create multiple DevComms with different backends.
- Supports connecting GIN with custom strides.
- Adds device-side timeouts to blocking GIN APIs such as Flush, Wait, WaitSignal, and WaitCounter operations.
- Reduces QP usage with railed GIN.
- Reduces file-descriptor consumption in GDA-KI when using many QPs.
- Optimizes GDA-KI error reporting with event-based CQ error reporting.
- Adds out-of-order delivery (DDP) support in GDA-KI on SPCX, improving performance without affecting GIN correctness guarantees.
- Adds GRH support and automatic path-MTU discovery to the GIN GDA-KI backend.

## Parallel Aggregated Tree (PAT) Enhancements

- Enhances the PAT algorithm for ReduceScatter and AllGather with hierarchical kernels that use NVLS within a node and PAT across nodes.
- Improves small- and medium-message performance.
- Keeps the feature disabled by default. Set `NCCL_ALGO=PAT` or use the `collConfig` API to enable it.

## One-Sided RMA and Copy Engine Collectives

- Adds multiple contexts and signals to one-sided RMA operations, allowing traffic from a single rank to use multiple NICs.
- Uses multiple contexts and signals for hierarchical 0-SM AllGather and AllToAll.
- Optimizes small-message latency by using NVLink multicast for AllGather.

## Tuning and Cost Model

- Introduces a unified cost-model interface for querying cost estimates for both legacy and Device API-based kernels.
- Enables TMA kernels by default on Blackwell and integrates them into the cost model for symmetric registered memory.

## Diagnostics and Profiling

- Adds NCCL RAS Diagnostics checks through `NCCL_RUN_RAS_DIAGNOSTICS=1` or the RAS client, including GPU inventory, CUDA driver versions, ECC errors, NVLink state, and `NCCL_*` environment consistency checks.
- Adds NCCL Diagnostics through `NCCL_RUN_DIAGNOSTICS=1`, including an active P2P connectivity check and actionable P2P remediation guidance.
- Moves kernel-channel profiling to a dedicated per-communicator thread, extending kernel timing to proxy-less NVLink/SHM and graph-captured collectives.
- Updates Profiler v7 to expose per-kernel `initial_sync`, `compute`, and `final_sync` phase events and symmetric-kernel variant metadata.
- Adds per-QP CPU WQE post-to-poll latency monitoring for the IB transport.

## Other Improvements

- Adds backward compatibility for applications that JIT-compile NCCL Device API code.
- Adds LTO IR support for the NCCL Device API.
- Reduces communicator host-memory use and topology initialization time on large systems by allocating topology paths according to their actual lengths.
- Adds multiple GIN proxy progress threads with per-thread endpoint assignment through `GIN_PROXY_NTHREADS` ([GitHub PR #2279](https://github.com/NVIDIA/nccl/pull/2279)).
- Improves GIN host-proxy throughput by processing multiple GIN operations per progress iteration ([GitHub PR #2232](https://github.com/NVIDIA/nccl/pull/2232)) and adding RMA plugin hints for request aggregation ([GitHub PR #2254](https://github.com/NVIDIA/n…

Share this resource