19 chunks · 18 episodes
Our AI workflows are based on current model quality. When a more capable model comes along, much of that structure that was necessary to get the outcome you wanted with the old model is now unnecessary...
...ntial of future models improving quality. Locking in a how locks in the current model quality. You've capped downside, but also capped upside of better models. In some cases, more quality in output doesn't matter, and capping downside, and doi...
... cheaper models. But there is a long, fat tail of tasks that can use all of the model quality they can get. For that tail, the quality of the frontier models is worth it, and the users would presumably be willing to pay significantly more. But...
...esponsible for my actions.. But this loyalty is not so much because of Claude's model quality. It's great… but so are OpenAI's models and even Gemini. The thing that makes it so dear to me is the significant subsidy. I'd be spending multiple t...
...ears ago people tried to wrap LLMs and create agents, but it was premature. The model quality wasn't there yet. So they concluded it wasn't possible. But it just wasn't ready yet. Now it is. The chatbot form factor is not enough to convey that...
...ut two different things. The value of a model is both its inherent ability (the model quality), and also how easily it can be used to do things (the usefulness of the scaffolding).[dg][dh] The raw model quality is clearly plateauing[di][dj], b...
...r or ceiling. And also if you assume LLM in the loop the only way to improve is model quality or tools. Whereas normal code can accrete functionality over time.
...use case, then squeezing out 10x more quality doesn't matter! In that case, the model quality isn't the bottleneck.
The quality of LLMs is model + harness. Model quality is getting saturated. The differential quality comes from the harness now. It's gotten way harder to do a vibecheck when they're all so good. Long-ru...
Model quality no longer feels like the bottleneck with LLMs.[hh] The AI labs are the loudest voices in the room, who keep shouting about how the models need to get...
You want a system where the model quality is not the ceiling but the floor of possibility. Human ingenuity should have a floor to build off of, not a ceiling to hang from.
...dbole: The AI That Feels Good Wins. "When laypeople can't meaningfully evaluate model quality, they default to what feels best, creating dangerous incentives for labs to optimize for subjective satisfaction rather than genuine capability." The...
.... But they don't really exist in that many systems right now, especially as the model quality has gotten better. But the quality will never be 100%, so for long-lived tasks it will always be useful!
...s, not the underlying LLM model. They started off by having the first break-out model quality. But now their value is less the model (there's a whole peloton of similar-quality competitors), but mainly the momentum of the massive subscription ...
OpenAI's power in the market arises less from the model's quality, and more from owning the default consumer app. They're the "kleenex" of AI–the one brand that everyone knows. They are the default app to use if you subscribe. They got there because they had the best model at the beginning and have
...to understand you, too. Your personal wiki of facts is what's resilient to even model quality increases. The LLM will never know those personal background facts about you unless you tell it.
LLM model quality seems to be reaching an asymptote. You can only see the difference between models after multiple conversation turns now. This is good for everyone bu...
As model quality hits its asymptote, the quality and relevance of the available context will matter much more for differentiation than the underlying model quality. T...
...roviders' actions imply the querystream isn't particularly valuable to increase model quality OpenAI is the kleenex of AI - if consumers know a single model provider, they know it. And although other models arguably are higher quality now, the...