The GPU Power-Performance Curve Most Clusters Ignore | Researcher Conversations at GTC

Summary

This episode focuses on Pebble’s approach to data center energy optimization for AI workloads, with an emphasis on maximizing tokens per watt rather than simply increasing power. The core technical claim is that GPU power/performance is non-linear: beyond a point, extra power produces diminishing or saturated token output. Pebble says inference is often memory-bound, with energy spent moving weights into HBM while SMs are underutilized, especially when prefill waits on decode. Its system ingests telemetry from inference servers and NVIDIA GPUs, deploys via Helm on Kubernetes, and dynamically tunes per-GPU power caps and clock frequencies after learning workload behavior for a few days. The company is also exploring grid-responsive data centers and flexible power arrangements, claiming there is 100 gigawatts of flexible power in the U.S. that AI clusters could access through curtailment tradeoffs.

Notes

• Pebble positions itself as a data center energy optimization company for AI factories, targeting higher tokens per watt.
• The company says it has demonstrated optimizations on both inference clusters and training clusters.
• A core finding is that GPU cluster power/performance is not linear.
• Pebble argues that at some point on the power-performance curve, more power saturates or reduces efficiency instead of increasing tokens.
• Inference inefficiency is partly physical: GPU voltage, power, and clock frequency relationships are non-linear.
• Pebble frames inference as memory-bound rather than compute-bound in many cases.
• The company says power is often spent reading model weights into HBM.
• It says SMs may be underutilized, especially when prefill is waiting on decode.
• Pebble’s tuning logic looks at workload type and current load, since inference traffic can be bursty, spiky, or random.
• It models the relationship between workload, power, and performance in tokens or latency before choosing operating points.
• The main control levers are per-GPU power caps and clock frequencies.
• The same per-GPU approach applies to both single-GPU and multi-GPU workloads.
• Observability comes from inference-server telemetry such as vLLM metrics, plus GPU power data from NVIDIA devices and exporters.
• Pebble deploys as a Helm install on Kubernetes.
• The deployment spins up DaemonSets and other Kubernetes objects to read telemetry and compute an optimal clock frequency and power cap.
• Pebble says it has not yet gone into kernel code optimization directly.
• Longer term, it wants to optimize kernels for energy, moving deeper than its current systems-level approach.
• The optimization system is dynamic and keeps learning power caps and clock frequencies over time.
• After installation, Pebble waits a few days to learn workload characteristics before applying power caps.
• The delay is intended to avoid breaking SLAs or compliance SLOs.
• Pebble is exploring flexible power for data centers, rather than treating them as static grid consumers.
• The idea is to make AI clusters grid-responsive and able to curtail consumption during peak periods.
• Pebble says there are 100 gigawatts of flexible power in the U.S. that could be accessed with flexibility tradeoffs.
• A product goal is to lower AI cluster power consumption with minimal impact on SLOs and performance.
• Pebble’s approach is to tune power and frequency at the workload level rather than indiscriminately increasing GPU power budgets.
• The discussion implies a systems-level optimization stack that combines workload telemetry, GPU-level control, and cautious rollout.
• The economic implication is that energy efficiency can be improved without changing model output, by exploiting inefficient operating points.
• The broader infrastructure thesis is that flexible data centers could access large amounts of grid capacity if they can curtail during peaks.