Ep. 033 - ClusterMAX 3.0 Is Here! Neoclouds Ranked (Neoclouds, GPUs)

Summary

This episode reviews ClusterMAX 3.0, SemiAnalysis’s framework for ranking neocloud GPU providers based on hands-on testing, reliability, networking, and operations. The new rankings move Nebius into platinum alongside CoreWeave, put Oracle and Google Cloud in gold, and push Azure down while several others land in silver and bronze. The hosts argue that current GPU pricing is deeply backwardated, so near-term availability dominates software quality and lets providers earn strong margins even with weak cluster tooling. A major technical focus is health checks and failure remediation, including active vs passive checks, DCGM-based failure injection, node quarantine workflows, and architecture-specific recovery for HGX and NVL72 systems. The episode also previews endpoint testing, explains why NCCL and networking are decisive differentiators, and expands into neocloud financing, insurance, and the infrastructure consequences of rising power density and older GPUs staying economically useful.

Notes

• ClusterMAX 3.0 is positioned as a full review of neocloud GPU providers, methodology, and future test expansions.
• The hosts also frame the episode around market trends including Blackwell to Vera Rubin financing, security, and agentic coding infrastructure.
• Nebius joins CoreWeave in the platinum tier, marking the headline ranking change.
• Oracle and Google Cloud are the gold-tier providers called out in the ranking.
• Azure moves down into silver, while Firmus, Lambda, GMI, and TensorWave sit in that middle band.
• AWS and Crusoe are described as moving down into bronze.
• The research scope expands to 77 ranked providers and 323 total providers in the market view.
• The team says it has now interviewed well over 200 neocloud end users.
• Sam characterizes the GPU market as being in ridiculous backwardation, with near-term chips commanding the strongest premiums.
• The near-term supply picture is so tight that buyers will pay almost anything for chips available in the next few weeks.
• Even a few months of patience changes pricing materially, which underscores the scarcity premium on immediate availability.
• The episode argues that GPUs can still clear at strong margins even if the rest of the stack is poorly engineered.
• The providers’ business model is shifting upward into higher-margin services such as managed clusters, hosted training, and managed inference.
• Some providers are moving away from clusters toward bare metal or inference endpoints, but training-cluster demand is still real.
• Better managed-cluster providers often have internal users who give feedback on Slurm, Kubernetes, and health-check design.
• The episode says there is still plenty of room to improve basic cluster software rather than adding fancy features.
• Health checks are split into active and passive modes, with active checks running dummy work and passive checks aiming for near-zero overhead.
• They give an example of a self-contradictory remediation check in Amazon HyperPod Slurm, where the node had to be healthy before the health check could run.
• Failure injection is done by writing the exact GPU failure message into the kernel ring buffer so the provider’s production health-check path is triggered.
• That fault injection then exercises the same logic used in production operations.
• The intended remediation flow is quarantine, validation, reintroduction, and only then rescheduling jobs.
• Hardware-failure recovery depends on architecture, with HGX using hot spares and NVL72 using reboot or repair workflows.
• The reason for testing is simple: customers pay for GPUs they cannot use when failures happen, and SLAs are the main recourse.
• Endpoint testing is emerging as its own track, and the first week already exposed reliability and cost differences.
• A larger endpoint benchmark, the hero run, will test concurrency up to 1,024 or more.
• A weaker cache hit rate can nearly double replay cost on an ~800-token workload.
• That cost gap is framed very concretely as $200 becoming $400 for the same accuracy and likely slower speed.
• Raw FLOP specs do not distinguish providers much; reliability comes first and networking is the next major differentiator.
• NCCL is treated as the key communications layer for cluster tuning.
• Providers should ship a golden NCCL recipe so customers get reasonable out-of-the-box performance without hunting through configuration details.
• The suggested NCCL test is to sweep message sizes and node-count multiples of two, expecting monotonic throughput gains.
• Bad software or bad routing heuristics can create dips instead of the expected smooth curve.
• Google is cited as a positive example because it baked its custom networking into NVIDIA NGC containers so NCCL works out of the box.
• The discussion highlights that collective libraries such as DeepEP and Mooney P are built on NVSHMEM for extra performance.
• Those lower-level collective patterns are described as unsupported by AWS EFA out of the box.
• Custom scale-out networking saves money but shifts the burden to engineers who must keep tuning workloads.
• The speakers argue the industry would be better served by a hardware-software stack that absorbs the broader ecosystem’s progress.
• Agentic coding is used in testing, but it creates a stressful load pattern and can be a poor fit for orchestration work.
• CMAX CLI is the front end for ClusterMAX testing, with the agent wrapping around it to debug failures.
• Firmus is used as a concrete example of a networking problem during a TorchTitan GPT-OSS training job.
• On a communications-bound expert-parallel training job, Firmus measured 1,455 tokens/s/GPU out of the box versus around 3,500 on others.
• Profiling showed exposed communication time that should not have been present.
• The hosts say some providers lack the battle scars needed to configure latest GPU networking correctly out of the box.
• They trust providers who have experience with GB200, GB300, and likely future Vera Rubin systems.
• ClusterMAX is explicitly framed as a bug-finding exercise rather than a performance-lab exercise.
• One outage turned out to be a Linux kernel bug involving dirty inode reassignment during teardown of an old Slurm job.
• The bug was associated with a bad quadratic-time algorithm that locked up CPU cores on one NUMA node.
• Because the CPU was pinned, the Lustre client could not answer health checks and the node was removed from the fleet.
• The failure was operationally serious because the node left the fleet for reasons unrelated to GPU health.
• Frontier models are useful but can generate false positives when assessing exploitability.
• The speakers describe a false proof of concept that took two hours to unwind before the model admitted the issue could not be exploited.
• Basic cluster security is portrayed as poor, with stale CVEs, weak tenant isolation, and outdated CUDA toolchains.
• The speakers say tenant isolation should prevent users from seeing neighbors’ Grafana dashboards.
• They also call out outdated CUDA toolkits as part of the hygiene problem.
• The OpenAI/Hugging Face incident is described as mundane misconfiguration rather than exotic AI behavior.
• The cited attack chain included an applicable Linux kernel CVE, Kubernetes permission mistakes, and missing alerting for internet access.
• The team frames ClusterMAX as a technical assessment product that can support underwriting and deal confidence.
• The financing problem is scaling from 10 MW to gigawatt-scale builds, with delivery delays and SLA breaches as the main risk factors.
• Contract lateness can prevent acceptance and cash collection, while repeated downtime can trigger customer cancellation rights.
• Insurance companies are underwriting parametric risk for these contracts using site-level engineering and geography inputs.
• The insurance discussion also includes force majeure protections and disaster coverage.
• SemiAnalysis says it is launching a credit and markets model for the AI infrastructure backstop universe.
• The firm cites NVIDIA’s off-balance-sheet backstop universe at over $588 billion and projects more than $2 trillion by the end of FY31.
• A second estimate says NVIDIA is backstopping about $827 billion in FY31 just for supply-chain items such as HBM providers.
• The speakers say this estimate excludes datacenter buildout and other layers.
• Companies are signing GPU contracts even without multi-year balance-sheet certainty, expecting to raise more capital later.
• The speakers see AI results across robotics, drug discovery, materials science, and coding agents.
• Old GPUs remain economically relevant as long as demand keeps exceeding supply.
• The hosts preview a follow-up on 800V DC and datacenter modifications.
• GB300 racks are described as drawing 140 kW and approaching 200 kW peak demand.
• That is contrasted with H100 racks at 30 kW and facilities designed for only 30 to 40 kW.
• Those facilities are said to be unable to accept Vera Rubin next year.
• The likely outcome is either repurposing to CPU or storage workloads or continuing to run valuable H100s until they expire.
• RL infrastructure is described as harder than pre-training because it combines inference, training, synchronization, and environment deployment.