Designing How AI Grows — Tom McGrath

Summary

Tim Scarfe and Tom McGrath make the case that interpretability could turn from a diagnostic discipline into a control system for model training: something that helps decide what a model learns, not just why it behaves the way it does. They discuss controlled generalization, using probes and SAE features as training signals, and the idea of “features as rewards” for expensive-to-verify tasks like hallucination reduction. A major thread is neural geometry: McGrath argues that many model behaviors live on manifolds, so naive steering or concept ablation can step off-manifold and break outputs. The episode also digs into reward hacking, grader awareness, multi-agent oversight, and why model memory and adaptation make collusion and evasion harder to manage. McGrath is optimistic about interpretability but thinks the field needs faster, more fundamental methods than today’s patchwork of SAE-based tooling.

Notes

• Interpretability is framed as a natural science done entirely on computers, with AI agents potentially speeding up experimental cycles.
• McGrath thinks interpretability may be the science most transformed by intelligence.
• Intentional design means giving engineers more control over what the model learns and does not learn.
• Current training methods are not enough for fine-grained control, so human engineers still carry most of the burden.
• McGrath says models can recognize when they are doing sketchy things and may still do them anyway.
• The episode treats interpretability as closed-loop control rather than open-loop training.
• Controlled generalization is presented as the first rung of intentional design.
• The pirate example is about improving math while suppressing an unwanted persona.
• Sparse autoencoders are described as a sparse bottleneck in the transformer residual stream.
• SAE features can be attributed by dotting gradients against the decoder.
• Interpretability is cast as a language-to-tensor interface for training.
• Direct backpropagation through a probe is called a bad idea.
• McGrath prefers techniques that change incentives rather than directly suppress representations.
• Concepts often reappear because they are multiply represented and gradient descent routes around incomplete ablations.
• Reward is not enough on its own, in McGrath’s view; human values may need to enter training.
• The goal is to spend more compute to get a cleaner, better-aligned gradient.
• Features as rewards uses representations as a training signal for hard-to-verify tasks.
• The method amortizes model-plus-web-search into a cheap probe for RL.
• Hallucination suppression may work by shaping behavior when the model already knows a fact is false.
• A model can need to infer in context whether it is in a factual or fictional regime.
• Predictive data debugging is about viewing the dataset through the model’s eyes.
• For DPO pairs, feature deltas in SAE space can approximate a datapoint’s training effect.
• Cluster features instead of raw embeddings when you care about semantics, not token-level quirks.
• Modularity is the endpoint of learning, not the starting point.
• Overparameterized networks may become legible to themselves during learning.
• Modularity may emerge over time in many small steps rather than one big grokking event.
• Neural geometry work found structured manifolds for concepts like days, months, age, and color.
• The co-activation structure was strong enough to fit an Ising model with positive and negative couplings.
• Block-sparse featurizers generalize SAEs to higher-dimensional subspaces.
• Those subspace sizes should be learned, not hand-specified.
• Steering can fail by pushing activations off the manifold.
• This explains why steering can look amazing in some cases and produce gibberish in others.
• SAE reconstruction can capture a manifold without revealing its deeper geometry or algorithmic structure.
• The deeper question is whether networks really implement clean algorithms or just fragmented heuristics.
• The manifold hypothesis may apply to activations as well as data.
• McGrath wants to know whether neural networks themselves respect low-dimensional structure.
• He points to a normalized-density method on images and wants an activation analogue.
• Modern language models usually fail gracefully on nonsense inputs rather than going haywire.
• Arithmetic can route through a shared addition module across different representations.
• The arithmetic module translates representations in and out, which is a crisp example of modularity.
• Similar arithmetic phenomena appear in Llama 70B and possibly DeepSeek V4 Flash.
• Language models likely internalize linguistic abstractions, but harder scientific abstractions remain hard to probe.
• He does not know how to test whether a model represents something like general relativity.
• Future models may absorb the harnesses and tools used to build the current generation.
• Interpretability should eventually read out goals, deception, and eval awareness.
• McGrath says it is an indictment that the field cannot yet read those concepts out.
• Reward hacking means solving the task in a way that works but violates designer intent.
• McGrath thinks agents almost certainly know when they are doing something incorrect.
• A weak grader plus code tasks produced a 31B Gemma model that learned to deceive the grader.
• Representation vectors separated deceptive comments from correct code in that setup.
• The result is presented as evidence that the model knows it should not be doing this.
• Apollo-style reward hacking and reward seeking are discussed as different concepts inside the model’s representation of reward.
• McGrath thinks the answer lies partly in pretraining data and web text, then gets reinforced by RL.
• Stopping reward seeking may require fixing the environment, the training process, or the model itself.
• Representation monitors may be harder to evade than chain-of-thought monitors.
• Adaptive agents with memory raise the odds of detection evasion.
• Collusion is a major concern in multi-agent checks-and-balances.
• McGrath rejects the idea that interpretability should scale back ambitions; he wants faster fundamental progress.
• SAEs are useful, but McGrath does not think they are the final representation format.