Ep. 033 - ClusterMAX 3.0 Is Here! Neoclouds Ranked (Neoclouds, GPUs) | Sam Harshe, Pratt Bhatt, Jordan Nanos

Summary

Jordan Nanos, Sam Harshe, and Pratt Bhatt walk through ClusterMAX 3.0, a broader NeoCloud ranking effort that now covers 77 tested providers and 323 total providers in the market view. The core thesis is that the deciding factors in GPU clouds are not headline FLOPS, but reliability, health checks, failure recovery, NCCL/network tuning, endpoint cache behavior, and security hygiene. They also frame ClusterMAX as a technical underwriting tool for financing, insurance, SLAs, and delivery risk as customers and lenders push neoclouds from megawatts toward gigawatt scale. The discussion closes with hosted training and RL, where they argue that environment management, trainer-inference consistency, and end-to-end operational support make the stack substantially harder than it first appears.

Notes

• ClusterMAX 3.0 is the episode focus and the corresponding article/review was published on Wednesday, September 23rd.
• The new ranking covers 77 providers.
• The market view expands to 323 total providers, and the research incorporates well over 200 end-user interviews.
• GPU pricing is described as sharply backwardated: immediate access commands a premium, while waiting a few months can produce a large discount.
• Managed cluster quality is framed as a software-and-ops problem on top of expensive hardware.
• Providers that use their own clusters internally tend to have better feedback loops for Slurm/Kubernetes layers and health-check design.
• Health checks are split into active and passive forms.
• Active health checks run dummy workloads and are usually preemptible when the cluster is idle.
• Bad health checks can be worse than no health checks if they interfere with jobs or depend on a node already being healthy before they can run.
• Failure testing is done by injecting synthetic GPU-failure signals into the kernel ring buffer.
• The test measures detection time, node drain time, replacement time, and recovery workflow behavior.
• HGX systems can rely on hot spares, while rack-scale architectures may need reboot or repair workflows.
• Endpoint testing is emerging as a separate focus area because providers can have very different cache-aware routing behavior even when the API looks simple.
• A 75% cache-hit rate on an ~800-token replay nearly doubled cost versus providers closer to 99%.
• Reliability comes first, and networking is framed as the next most important differentiator after basic uptime.
• Providers should ship a golden recipe for NVIDIA’s communications library so performance is reasonable out of the box.
• Message-size sweeps should produce smooth, monotonic throughput curves instead of pathological dips that require manual tuning.
• Open-source collective libraries matter, but they are only one layer in scale-out networking performance.
• Managed inference cannot rely on vLLM or SGLang alone; providers still need substantial custom layers and kernels.
• Agentic coding can help with cluster debugging, but blindly relying on it can create false confidence and bad orchestration.
• Frontier models can produce false positives when used for vulnerability exploitation or incident analysis.
• Security is still weak across GPU clusters, with badly secured systems, old CVEs, and poor defaults.
• ClusterMAX is also an underwriting tool for neocloud financing, acceptance risk, and SLA risk.
• Late delivery and SLA violations are the two main financing risks, and both can trigger cancellation dynamics.
• Parametric insurance depends on cluster-level resilience evidence: redundant power, cooling, internet, parts depots, staffing, and disaster exposure.
• SemiAnalysis has developed example SLAs and contracts to standardize terms like node, rack, site, cluster, downtime, and force majeure.
• Customers sometimes spend their entire seed round on compute, making cluster selection existential for early startups.
• Nvidia’s off-balance-sheet backstop exposure is already over $588 billion and is forecast to exceed $2 trillion by 2031.
• Pratt’s model estimates about $827 billion of Nvidia backstops in fiscal 31, focused only on supply-chain items like HBM.
• Amazon and Google are also putting balance sheets to work to backstop AI infrastructure demand.
• Older GPUs still retain strong terminal value: H100s remain in demand on four-year contracts and A100s from 2020 have not been given up by OpenAI.
• GB300 racks push data-center power density far above H100-era facilities.
• RL and hosted training are operationally harder than pretraining because they combine inference, training, and sandbox deployment.
• Hosted training must manage environments, inference nodes, trainer nodes, and trainer-inference consistency across heterogeneous hardware.
• Pratt argues hosted training becomes meaningful only if the provider handles the full workflow end to end, including environment creation and deployment support.
• Forward-deployed support and business understanding may be required for successful RL hosted training.
• Frameworks such as Miles and Prime RL are making the hosted-training stack more usable.
• Trainer-inference mismatch can compound over roughly a hundred steps and collapse training.
• Sam wants fewer obscure bugs and a cluster experience that simply works, with topology awareness and reliable health checks.