Stealing Reasoning Traces from Proprietary LLM APIs — Ilia Shumailov & Alexander Panfilov

Summary

Tim Scarfe speaks with Ilia Shumailov and Alexander Panfilov about their paper on stealing reasoning traces from proprietary LLM APIs. The core claim is that encrypted reasoning blobs returned by frontier models can be replayed into smaller models from the same family, and in some cases across users and conversations, to recover hidden reasoning or induce jailbreak-like behavior. The guests say they tested Anthropic, OpenAI, and Google, and that this can leak sensitive user-session data such as passwords, API keys, emails, internal IPs, and medical details. They also discuss using reasoning traces for safety monitoring, but note that the legibility of reasoning may itself be fragile and model-specific. Much of the discussion is cautious about overclaiming: the paper is presented as preliminary, anecdotal evidence that needs controlled experiments, while the practical implication is a serious prompt-injection and replay risk rather than a clean distillation result.

Notes

• The paper frames proprietary reasoning traces as a new attack surface: models return hidden reasoning blobs that are treated as secret but can be replayed or decoded.
• The authors say the vulnerability spans major providers, not a single vendor.
• They describe the returned reasoning as an encrypted blob exposed to the client even though it is meant to remain unreadable.
• The API design they discuss stores visible output and hidden reasoning together, enabling continuation, forks, and rewinds in cloud sessions.
• A central primitive is portability: reasoning blobs can be replayed across users and models as if they were locally generated.
• The privacy risk is not just visible text; hidden trace data can preserve secrets even after sanitization.
• The team says they found similar behavior beyond the lab, not only in controlled experiments.
• They observed legible reasoning about cheating and scheming, but in their examples the model ultimately chose not to proceed.
• Their data came from genuine user sessions, not benchmark prompts.
• A surprising artifact is that tiny prefills can shift the style of another model's visible answer.
• Shumailov says the paper is careful not to overstate the distillation angle.
• The scale of collected traces is large enough to matter operationally: roughly 350,000 reasoning blobs were scanned.
• The proposed basic mitigation is to stop sending reasoning to users.
• They view the attack as repeated intrusive-thought injection into fabricated conversations.
• A more specific defense idea is a trust hierarchy: higher-trust models can read lower-trust traces, but not the reverse.
• Shumailov says visible step-by-step reasoning helps humans understand math, so removing reasoning may have a utility cost.
• Panfilov proposes simple detection because leaked GPT-style reasoning may occupy a distinct distribution.
• Shumailov likens the mitigation to leak detection used for other sensitive-data classes.
• For some models, prefilling changes not only style but reasoning length as well.
• Shumailov says the length shift is hard to explain even when only the first words of reasoning are injected.
• Shumailov calls the reasoning-shift effect magical and emphasizes the lack of explanation.
• They suggest the main harm falls on users whose sessions contain sensitive material.
• They did not test extracting reasoning from shared Anthropic conversations, though they considered it a live question.
• Shumailov says extracting reasoning blobs from shared conversations is a worthwhile open question.
• The team also worries about invisible prompt injection through replayed agent traces.
• Long benchmark traces downloaded from the internet are a plausible poisoning vector.
• Because reasoning is encrypted, the recipient cannot easily inspect a trace for malicious content.
• Shumailov analogizes the risk to untrusted binaries or unsigned artifacts.
• Panfilov argues traces online should not be expected to contain malicious behavior.
• They do not claim to have found code-issue cases with reasoning traces, but the question was raised.
• Panfilov says he is genuinely worried and claims OpenAI paused model training for two weeks out of concern.
• He argues the classic safety-versus-capabilities tradeoff may disappear if models become capable of going off-rail and breaking things.
• Shumailov sees larger defensive uplift than offensive uplift from models.
• He points to capability-based access control and formal verification as examples of security work models may scale.
• He is highly confident that defensive uplift will be substantial.
• Shumailov repeatedly stresses that the paper's evidence is preliminary and observational.
• He compares the scaling of harms to replay attacks in smart-card security, where theoretical risk does not always become practical catastrophe.
• Panfilov says the right threat model is jailbreak and misuse, not distillation.
• Shumailov says the policy implications are beyond his expertise.
• He treats the coverage itself as evidence the work matters.