Training a 400B Model on 2,048 Blackwell GPUs for $20M | Researcher Conversations at GTC

Summary

Lucas Atkins, CTO and head of research at RCAI, says the company shifted from mainly post-training on open models into pre-training its own base models in 2025 because downstream quality was capped by upstream base-model quality, and because some customers became unwilling to build on Chinese pre-trained models for legal and compliance reasons. He argues that open weights matter for sovereignty, transparency, and safer diffusion of mitigation techniques, and that western open-model labs should optimize for reliability, speed, cost, and the 80% of economically viable tasks rather than trying to match frontier labs on raw capability. Atkins also describes RC’s modular research structure, where staff can move across pre-training, data, architecture, mid-training, and SFT as bottlenecks arise. On the engineering side, he says Trinity used B300s mainly for speed and availability, even though the ecosystem lacked mature benchmarks and sparse-kernel tooling. He closes with a strong org-design view: compute is the hard resource constraint, talent is more manageable, and breakthrough research should stay in a small, opinionated room rather than a giant team.

Notes

• Lucas Atkins is CTO and head of research at RCAI, with a background in open-source AI and post-training before moving deeper into pre-training.
• RC says it started pre-training work in early 2025 and that it materialized in July 2025.
• RC frames the business goal as being the most economically viable choice for developers, startups, and enterprises.
• He argues that downstream performance hit a ceiling determined by the quality of the base model.
• He says customer preferences shifted once most sub-20B models came from China and legal/compliance teams objected to relying on those base models.
• Owning the whole stack is presented as necessary to offer the best services and build the best products.
• He says the West lacks a company whose existence depends solely on base-model quality.
• Owning the full stack also improves downstream customization because pre-training data can inform post-training data.
• He argues that the last 10% of pre-training is especially consequential for post-training quality.
• Open weights are framed as a sovereignty issue: complete control over the pipeline, data, and product behavior.
• He uses an invoice-processing workflow to illustrate why full-stack control matters for enterprise automation.
• He says frontier models are getting more expensive and slower while many business tasks need better tool use, speed, and cost.
• A practical customization target is a 4B to 8B parameter model trained on customer data and tuned for future skills.
• He says frontier-lab incentives can diverge from what consumers, developers, startups, and enterprises want.
• Open models are defended as a way to keep risks transparent and mitigations widely understood.
• He points to open source as evidence that collaborative, transparent work can drive broader progress in AI.
• He warns that commercializing everything can turn AI into a capital, talent, and compute arms race that slows compounding breakthroughs.
• If continuous learning is solved only inside OpenAI, Anthropic, or Google, diffusion to the wider ecosystem will be slower.
• He says talent is easier to solve than compute because compute ultimately depends on funding and revenue.
• People want to work on open weights because the work is visible and publicly appreciated.
• Even with access to $2 trillion, he says he would keep the research team under 30 people.
• RC’s lab structure is intentionally modular so people can move across the full training stack as needed.
• He argues open-weight labs should not define success as matching frontier labs.
• RC’s goal is to be reliable, fast, inexpensive, and strong on the 80% of economically viable tasks.
• RC says training-paradigm shifts are becoming more infrastructure-intensive than post-training.
• RC wants a loose, modular research structure so it can adapt without overspending relative to revenue.
• Trinity was named partly because the collaboration involves three companies and the first generation had three models.
• Daytology handles pre-training data curation and Prime Intellect handles infrastructure scale-up and GPU management.
• B300s were chosen because they were available and RC wanted pre-training to take one month instead of three.
• B300 adoption was challenging because the ecosystem had little at-scale benchmarking and limited tooling.
• RC leaned on other labs’ artifacts and the DeepSeek sparse-kernel ecosystem to bootstrap Hopper performance.
• Hopper-trained models provided a benchmark for throughput on a model of this size and scale.
• The speaker says task gains can be uneven: large improvements on frontier math may translate into small gains on product tasks like front-end generation.
• RC wants a research org that stays loose and modular enough to follow training shifts without turning into a giant burn rate.