Distillation Drift
Your pocket lexicon
The take
When AI models eat their own cooking, Distillation Drift kicks in: the slow, silent decay of quality as AI trains on AI output, replacing human insight with increasingly bland or hallucinated data.
Why it matters
This matters because the race for more AI training data is quietly becoming a race to the bottom. Without fresh human input, models risk becoming digital echo chambers, losing nuance and accuracy, and eroding trust in the AI tools we rely on.
The note
Distillation Drift describes the subtle but critical problem of AI models being trained predominantly on content generated by other AI. Instead of learning from the messy, complex, and often contradictory richness of human experience, they're fed a diet of synthesized, 'perfected' data, often stripped of the very quirks that make human communication valuable. The mainstream line is that AI training data is simply expanding exponentially, leading to more robust and capable models. This ignores that a significant chunk of that 'data' is now recycled AI output, not new human insight. It's a kind of digital inbreeding, where the gene pool of information gets smaller and weaker, leading to models that are less creative, more prone to hallucination, and ultimately, less intelligent. The concrete fight is that AI companies are locked in a data arms race, but few are transparent about the diminishing returns of AI-on-AI training. As models unknowingly train on increasingly degraded information, we face a future where AI-generated content becomes indistinguishable from reality, but that reality is a bland, recycled version of itself. Your AI assistant might just be a chatbot impersonating a better chatbot.
Priority notes
Screenshot of Kimi identifying itself as Claude. The distillation problem is very real; David Sacks and others are commenting. Coin a sharp Gifdead take / term if one falls out.
Links from intake
- https://x.com/denisewu/status/2077984660211269
In the wild
Receipts from the feed. Not the definition. Proof the fight is real.
- Screenshot of Kimi identifying itself as Claude, confirming AI-on-AI output.
- David Sacks comments on the 'distillation problem' and AI models losing their 'secret sauce' when trained on inferior data.
Related
Crypto ConvergenceCrypto Convergence is what happens when DeFi, AI, social media, and TradFi stop pretending they are in different group chats. The winners will not be the slickest tech, they will be whoever owns the plumbing where all four meet, and it probably will not be you.
Exponential Gap in FinanceRegulators say 'risk' and mean 'protect the incumbents.' The Exponential Gap in Finance is the widening chasm where legacy finance uses caution theater to slow anything that compounds faster than a quarterly report.
Sources
FAQ
What's the real danger if AI models keep training on each other?
The real danger is a feedback loop of mediocrity, where AI-generated content becomes increasingly bland, inaccurate, or hallucinated, eventually making AI output indistinguishable from a digital game of telephone where the message gets dumber every round.
How can we tell if an AI model is suffering from Distillation Drift?
Look for a lack of originality, generic responses, or an increase in confident but factually incorrect 'hallucinations.' If an AI starts sounding like every other AI, it's probably been eating too much of its own kind.
What's the solution to Distillation Drift?
The solution lies in prioritizing fresh, high-quality human-generated data for training, and developing methods to identify and filter out AI-generated content from training sets. It's about ensuring a rich, diverse diet for AI, not just more calories.