Who Feeds the GPUs? Inside AI's Hidden $30B Layer | Renen Hallak, VAST Data

Summary

This episode is a deep dive into the hidden storage and software layer that feeds modern AI systems. Renen Hallak makes the case that AI factories will become standard for enterprises, not just frontier labs, and that the infrastructure must be redesigned for massive, heterogeneous data and low-latency inference rather than old database assumptions. He explains VAST Data’s “shared everything” architecture, its DASE-based operating system, and why training and inference create very different reliability, routing, and memory requirements. The conversation also covers confidential AI with Nvidia, encrypted inference inside enterprise premises or VPCs, and Hallak’s view that sovereign AI, agent policy, and model ownership will matter increasingly. He closes with a bullish but unsettling view: demand is accelerating so fast that exabytes are becoming planning units, and the next decade could bring more change than the previous millennium.

Notes

• VAST’s thesis is that AI needs a new software layer between GPUs and models, not just more chips.
• Hallak says the old data-center stack is being replaced by 500-kilowatt racks, faster networks, and large SSDs.
• Modern AI workloads are data-heavy and multimodal, not just rows and columns.
• Hallak says enterprises will need AI factories whether they use hyperscalers, AI clouds, or private environments.
• At scale, Hallak says it is cheaper to buy GPUs than rent them, and enterprise IP will be distilled into models and weights.
• He frames the software-infrastructure layer as the way to make AI usable for regulated enterprises.
• The physical AI factory can live on-prem, in colo, or in an AI cloud, but ownership of information, models, weights, and agents matters most.
• Hallak expects agents to become persistent, experience-based “artificial people” that organizations depend on.
• He argues enterprises should not expose proprietary information to external model providers.
• VAST’s trust-layer argument is that infrastructure should enable confidential AI and future intelligence marketplaces.
• The new launch is about model management: deciding which models handle which tasks and pairing that with confidential computing.
• Nvidia hardware support helps VAST keep enterprise data and model weights hidden during inference.
• Hallak says VAST began when neural nets started to look practically useful in 2015.
• The company was founded because existing infrastructure was not scalable or performant enough for the next AI wave.
• Shared-nothing systems break at AI scale because communication costs grow quadratically and clusters cannot stay resilient at huge node counts.
• VAST’s architecture places SSDs and metadata across the network so all nodes can see all information as if locally attached.
• VAST says its parts are based on DASE and that it cannot simply bolt open source on top of the system.
• A Singapore customer example: camera feeds are vectorized into trillions of vectors, which breaks conventional database assumptions.
• Hallak argues S3-era cloud primitives were built for the old stack, not for AI-native workloads.
• Customers typically start with a new AI use case and then expand access to more of their dataset.
• VAST claims no churn, with customers often doubling or tripling stored data each year.
• Inference is operationally harder than training because it must be low-latency, always-on, distributed, and edge-close.
• The platform must track raw data, generated data, input data, weights, and conversational context.
• Access control starts with Active Directory and ACLs, then extends to agents.
• Policy must also govern tool calls, conversations, observability, and actions in physical systems.
• Hallak points to a recent multi-agent math result as evidence that collaboration can unlock new capability.
• The protected-data feature is called Enclave, though Hallak says he does not love the name.
• VAST’s confidential AI runs inference inside the enterprise’s premises or owned VPC while encrypting weights end to end.
• Hallak says the new capability is not a breakthrough in cryptography; it is a plumbing and ecosystem problem now being solved for GPUs.
• He says VAST’s trust layer should sit low in the stack, like operating-system safety historically did.
• Demand is accelerating so fast that AI-cloud customers keep revising storage needs upward.
• He says the limiting factors are land, power, chips, and fab timelines, with some AI clouds sold out for 18 months.
• VAST says its profitability comes from software margins and the AI market’s roughly 10x-every-two-years growth.
• Neo clouds need power, hardware, and money; winners build before demand arrives.
• Hallak rejects the idea that neo clouds are only financial conduits for Nvidia hardware; he says the operational skill set compounds.
• He frames sovereign AI as a basic utility need, similar to electricity or water.
• Hallak says hardware commoditizes quickly, while software infrastructure captures most of the value.
• Nvidia is VAST’s main partner, but there is no exclusivity and no formal legal arrangement beyond the investment.
• Hallak says the fastest way to move is to find the limiting factor, remove it, and repeat.
• He says CEOs need direct access to people on the ground and a flat org, or bottlenecks will hide.
• Hallak predicts AI will keep advancing beyond human intelligence and reshape governance, money, and ownership.
• VAST’s endgame is to manage all the data while keeping the system safe and not infrastructure-limited.