NVIDIA GPUs Plug‑and‑Play on Macs and Zero‑Copy RDMA
Review RDMA zero‑copy GPU memory sharing on macOS to evaluate inference performance.
Test RDMA zero‑copy GPU memory sharing on your mac cluster to evaluate performance gains.
Summary
Blackwell RTX PRO 5000 72GB was detected on macOS via PCIe over Thunderbolt 5, and TinyGPU's DriverKit extension loaded successfully, but NVIDIA's GSP firmware failed during initialization with an RPC 4097 error. The NOCAT error FBFLCN UNRECOGNIZED_CLIENT indicates the GPU's memory fabric does not recognize the TB5 PCIe peer, a known issue affecting all NVIDIA GPUs on TB5 enclosures. Meanwhile, the macOS RDMA subsystem accepts Metal GPU buffers and IOSurface objects for zero‑copy network transfers, a feature not documented elsewhere. Hidden ibv_reg_dmabuf_mr symbols in libibverbs.dylib confirm that Apple has a full DMA‑BUF RDMA registration pipeline, though the API is not exposed. VM‑mapped memory passes ibv_reg_mr while heap allocations fail, differing from Linux's behavior. A triple‑registered 64MB buffer demonstrates that Metal writes are immediately visible to RDMA, proving zero‑copy GPU to RDMA is possible. The author seeks help with TinyGPU GSP firmware init, Ghidra analysis of ibv_reg_dmabuf_mr, and potential GPUDirect RDMA between NVIDIA eGPU VRAM and TB5 RDMA controller.
Key changes
- Blackwell RTX PRO 5000 detected on macOS via PCIe TB5, but GSP firmware fails with RPC 4097.
- NOCAT error FBFLCN UNRECOGNIZED_CLIENT indicates TB5 PCIe topology issue.
- RDMA subsystem accepts Metal GPU buffers and IOSurface for zero‑copy transfers.
- Hidden ibv_reg_dmabuf_mr symbols in libibverbs.dylib enable GPUDirect RDMA.
- VM‑mapped memory passes ibv_reg_mr; heap allocations fail.
- Triple‑registered 64MB buffer shows zero‑copy between RDMA and Metal.
- Potential GPUDirect RDMA between NVIDIA eGPU VRAM and TB5 RDMA controller.
- TinyGPU GSP firmware init on TB5 enclosures remains unresolved.