How to Build Long-Horizon AI Agents — Mitch Troyanovsky, Basis

Summary

In this episode, Basis co-founder Mitch Troyanovsky lays out a practical framework for building long-horizon AI agents that can operate for minutes, hours, or even days without losing coherence. He argues that accounting is a useful testbed because it compresses messy real-world economic activity into structured outputs, but it also exposes why outcome-only evaluation breaks down. The conversation traces the history of agent failures and capability jumps, highlighting how long context, reasoning models, and better post-training changed what’s possible. Mitch then digs into Basis’s approach: behavior specs, judge models, primary-source verification, ontology design, and treating runtime context as a form of training data. The core strategic message is that reliability and embedded workflows matter more than a secret model trick, and durable moats in AI will mostly be business moats.

Notes

• Basis builds autonomous accounting agents end to end, not just chatbots that answer questions.
• The office workflow uses quiet microphones so employees can whisper high-context instructions to AI tools.
• Mitch frames accounting as one of the largest knowledge-work professions and as a compression layer over the economy.
• Long-horizon means coherence over roughly 10, 20, 30+ minutes and beyond the immediate context window.
• Long-horizon agents need sub-agents, compaction, and other ways to preserve state once they move beyond working memory.
• The early agent wave failed because models had small context windows, weak long-context attention, and poor reasoning.
• Mitch identifies Opus 3, o1, and o3 as the main capability jumps for long-context understanding and reasoning.
• Reasoning models matter because they can allocate compute unevenly and self-heal better than per-token generation.
• OpenAI’s 2023 process-supervision work is described as a precursor to later verifiable-reward methods.
• The long-horizon capability chart is treated as noisy and hard to measure today.
• The models are being trained to build theory of mind over their own future selves and notes.
• Coding wins first because it is runtime-verifiable and cheap to check.
• But code quality still depends on taste and engineering judgment beyond tests.
• Agents outside code struggle because they often do not know what good looks like and cannot verify themselves cheaply.
• For tax work, some checks are deterministic, while others require accountant judgment or judge models.
• Tax-data generation is a bottleneck because real artifacts are private, scarce, and hard to synthesize at scale.
• A 1065 can require 20+ hours of human work, and a trajectory can run into thousands of inference steps.
• Outcome-only evaluation is not enough: passing 100 evals does not prove production generalization.
• Basis insists on primary-source verification, not pretraining recall or blog-level answers.
• Behavior specs are markdown files that define desired agent behavior and can be judged against trajectories.
• Behavior design is a combined effort between accountants and ML researchers, with an accounting Product Ops team building rubrics.
• Behaviors are not shown to the agent directly; they guide engineering choices and grading.
• Basis wants as few behaviors as possible because each behavior creates maintenance burden and duplicate state.
• The judge is itself an agent, so trajectory labeling is costly and needs filtering.
• Context engineering is runtime training data, but with far less data than model post-training.
• Ontology is the world the agent lives in: structure, metadata, embeddings, graphs, and concepts.
• Canonical documentation matters because agents need one source of truth to reconstruct compacted context.
• Basis’s open-source BrainTrust project turns behaviors into markdown specs and example judges.
• Deployment is described as onboarding 300 brilliant alien employees rather than shipping a narrow tool.
• Self-improving agent systems are a research direction, but current agents are still much worse at engineering agent systems than at regular software.
• The bitter lesson will likely swallow the current harness, but not immediately.
• Business moats, not secret technical tricks, are the durable source of value in AI.
• Founders should treat AI as a paradigm shift and reason from changes in intelligence, not from Twitter noise.
• The episode emphasizes that process supervision and workflow fidelity matter more than raw accuracy for production trust.