Bits and Bobs 10/20/25

This is one of the illusions that happens with logarithmic-value-for-exponential-cost curves.

The initial ramp of value is so extreme that it feels like it will ramp up to infinity.

But actually it approaches an asymptote.

I hadn't realized how much of ChatGPT usage is free users.

It's easy to get momentum in a business by selling dollar bills for 90 cents.

For their business success, it's imperative for them to get higher engagement / stickiness.

A crucial feature: businesses that use it can subsidize inference for users.

This is a classic aggregator play to lock in an ecosystem based on an early advantage.^[hp]

The subsidy makes it increasingly impossible for other providers to compete.

The bet is that by having predatory pricing (aka "dumping") they can force other businesses out and then corner the market.

They're already doing this by their intense subsidy of inference on ChatGPT.

OpenAI is betting that their overall story and momentum is strong enough that they'll be able to continue raising capital when others can't.

But a stable equilibrium to me seems to be Anthropic and Google staying in the race indefinitely.

That is a very different end state equilibrium.

In that world, there's not a duopoly but a triopoly.

A duopoly is effectively like a monopoly.^[hq]

The appearance of competition without actual variance.

But a triopoly is different, there's always an odd man out.

That odd man out is incentivized to do different plays that the stronger players wouldn't.

That changes the dynamic so the bigger players have to compete, too.

Triopolies are radically better equilibrium^[hr][hs] for the market.

We're still waiting for the "montage" moment for LLMs.

My vote for the montage capability is LLMs being able to create software.

Code, or just english descriptions.

In the past few months, as an industry our minds have continually been blown by how powerful it is.

We keep finding new ways to get even more out of this ability, ever more easily.

Claude Code has only been out for 8 months.

This revolution is just getting started.

We keep on discovering new orders of magnitude more powerful techniques for getting more value out of it.

MCP felt cool, but it had a low ceiling and was easy to overwhelm the context window.

Really what matters it's the ability of LLMs to do tool calling.

More generally: to create software, to do things, whether in code, or with tool calling.

Catastrophically powerful.

The default manifestation of it is a nuclear bomb.

But if you can figure out how to harness it in a nuclear reactor you could create near limitless energy.

If you could figure out how to get the "catastrophic" downside part capped so it was safe for the mass market, you could change the world.

The ability for LLMs to create software, to do things.

Giving just a teensy bit of structure to the LLM (in that case, a Typescript schema for state), and allowing a hierarchy of english language instructions that the LLM could peek inside if it wanted but not be bothered with by default.

An example of efficiently unlocking the LLM's ability to create software.

That also causes traffic jams.

Impeccable first-order thinking.

Non-existent second-order thinking.^[hv]

But also fast twitch can't get more leverage.

In lots of tech companies today, they use Slack, never email (too slow).

That means they can theoretically go faster to respond quickly to things that happen in the market... but also they are now default at that speed all the time, and unable to think deeply.

It looks great but you can break it easily.

You need an external ground truthing process.

There needs to be a quality control entity in the middle that isn't eager for your approval.

When you create something with AI, you make something you think is great.

The calibrated judgment becomes extremely important, instead of blindly accepting and making towers of slop.

Are you happily making things... that others are actually choosing to use?

That last bit is the key question.

The limiting factor often isn't what we think it is.

For example, making a movie faster is not about making more images.

It's about getting things in front of the director for approval.

The meta-thing is the system that generates the thing.^[hz]

If you can get it to work well, then you improve not only the thing, but a whole class of things.

For example, if you have a compiler, optimizations improve everything it compiles.

Another example: in search quality, don't check in a configuration file of synonyms, check in the process to generate that file based on an analysis of the querystream.

Another example: never touch the code directly, touch the specs and LEARNINGS.md that you give to the LLM to generate the code.

The models will likely centralize, due to capital requirements.^[ia]

But that doesn't really matter if there are three or more high-quality ones that are easy for an application layer to swap between.

That's why OpenAI is desperately vertically integratin^[ib]g and storing state, to avoid being a swappable component lower in the stack.

Catastrophically powerful.

Small startups can try dangerous things because they don't have much to lose.

LLMs are catastrophically powerful for building software.

The benefit of this ability accrues to the entities that have lower downside risk.

That means startups as a class have an advantage.^[ic]

Many startups that use it will blow themselves up.

But some will get lucky.

We'll see swarms of "fast-fashion" startups and apps.

Leverage gives you speed for risk.

The risk is hard to see, the speed is easy to see.

You borrow from the future to go fast today.

If it works, it works great.

If it doesn't work, it works terribly.

Lots of things are levered in ways that aren't obvious.

A dependency brings in complexity and risk from the rest of the ecosystem.

LLMs are great at writing code on demand.

If there are lots of examples of a given library, you can have it distill a custom one on demand, just for you, perfectly fit to your purpose.

No dependency risk!

You can have more of them than before.

It used to be that there was a difference between integration hard and algorithmically hard engineering.

But that distinction was pre-LLMs.

So the difficulty of executing now comes down entirely to novelty.

Graphviz, when rendered, makes sense to humans to understand flows.

LLMs can understand it based just on the markup.

A boundary object for flow graphs between LLMs and humans.

They used to be hard to understand how to use, but LLMs are great at them.

Giving agents isolation, so if they blow up something, they don't blow up everything else, allows them to move fast.

A powerful piece of infrastructure will be containerization++.

Not just containerizing the code, but also the data and resources.

Imagine if you had a copy-on-write filesystem and data layer.

Irresponsibility is the unlock.

Take shortcuts now that might blow up later.

Every year for decades is supposed to be the year of the Linux desktop.

Linux has never become mainstream for desktop use.

But it quietly spread literally everywhere else in computing.

Maybe the change is happening, just not in the place you're looking for it.

The AI submarine.

Probably a reactive JSON graph^[id][ie].

You'd need links to reference other parts of the graph.

Home-cooked software has to happen in a personal kitchen, not some factory kitchen owned by a corporation.

The people who could do so effectively were able to 10x their capacity.

But now you can offload thinking.

"Go think about this question and tell me what the options are."

That's a step change.

Seems like a combination of:

A catastrophically powerful combination.

This kind of looks like RL if you squint.

RL researchers might say this is an under-powered hack to get something like RL.

But it's different, because it has a swarm of human curation and judgment in the loop.

Distributed caching.

Reverse engineering software requires incredible patience.

LLMs have infinite patience.

To open a restaurant requires navigating a bureaucratic maze.

Talking to people who have done it before, scrutinizing overwhelming, poorly documented, kafkaesque processes that use arcane jargon.

It requires a knowledge of that jargon and infinite patience.

Something that LLMs have!

LLMs can help you navigate these kinds of processes more easily.

They effectively help metabolize arcane knowledge and allow people to operationalize it more easily.

Serializing intuition across the brain barrier is an extremely lossy and expensive process.

Once crossing the single-mind threshold you also start having coordination costs, which can balloon massively.

Alfred North Whitehead: "Civilization advances by extending the number of important operations which we can perform without thinking of them."

LLMs make orders of magnitude more ideas fit inside one mind.^[ig]

Rhymes with complex vs complicated.

A version of the last-mile problem.

The human needs to poke them and structure them to do the finishing steps.

"The first 90% is done, now it's time for the remaining 90%."

You get the experience of going really fast without necessarily getting that much closer to the end.

A version of the last-mile problem.

Are agents bad at finishing real-world tasks due to a lack of metis?

We aren't seeing a Cambrian explosion of apps because everyone has apps 90% of done.

Some things are way easier than before, some aren't.

A jagged frontier.

Going from demo to production is just as hard as it once was, if not worse.

For example, a 50% decline for Lovable from June to September.

Of course, external data is of very poor quality.

But it does track to me as being plausible.

It turns out that as a non-technical person you can't ship a production product even if the LLM writes the code for you.

There's more to shipping a production app than writing the code.

The clean up from "demoable" to "usable" (especially to make it not just usable but also safe) is a huge amount of work, that LLMs don't do a great job at unless you tell them to.

Many of the vibe coded apps that succeed in the market get taken down by security issues.

The Lovable founder responded with stats showing continued growth.

Those prominent contributors have a brand of significant quality for code he's hand-written.

Instead of muddying that brand, he has a separate one for things he vibecoded and are thus more "use at your own risk".

Verifiers of taste, or of quality, or correctness.^[ij]

This is where humans typically still need to be in the loop.

If they diverge, you get a rat's nest.

If they converge, you get a boring, over-saturated middle.

LLMs converge their output.

So now the middle of the distribution of output will be over-saturated.

That will push out the differentiation to the tails.

This was already happening before LLMs, due to the zero cost of content distribution.

But LLMs turbocharge it.

It's extremely expensive and manual today.

But they are much higher signal than quantitative user research.

Quantitative research requires you to ask just the right questions in just the right way.

If you ask them wrong, you'll generate faux insights.

Harder to find some classes of disconfirming evidence.

You can't discover your unknown unknowns.

You can't learn unexpected things as easily.

But LLMs can do qualitative nuance at quantitative scale.^[ik]

"Slop → Aura

innovative → institutional

personalized → individual

generative → creative

dissociative → enigmatic

monetizable → valuable

platform distribution → dark social distribution

mechanical → artisanal

scalable → singular

disposable → canonical

dead internet theory → dark forest theory"

I wonder: what if some of this AGI talk is in some ways a long con?

First and most obviously, the more that people believe it, the more they can attract insane amounts of capital.

Secondly, this massive, society-defining outcome overshadows the more mundane, everyday concerns of power concentration in a hyper-aggregator.

Everyone's fretting about this infinite outcome that might never come, instead of the almost-certainly-will-happen hyper-aggregation and power centralized in one company.

It's that they shouldn't have to care about it all because what happens by default is aligned with their expectations and interests.

Kindle's takes 9 hours to read out loud.

Most are much worse.

The answer is no, in my opinion.

Going down the path of chatbots / LLM-as-friend / hyper-scale leads to a path to inevitable enshittification.

Just in case there was any doubt that the sycosocial relationships inherent to chatbots are bad for you.

But unlike many characters, it was not authored.

Instead, it emerged.

A very different dynamic.

Yet another way that the implicit frame of Chatbots as "LLMs as friends" is unsettling and potentially dangerous.

It leans into hyper addiction and engagement even more.

This seems like a cynical growth play... not something you do if you had runaway momentum.

Also, do you really want this company to have records of all of your intimate interactions?

The first milestone is "first triangle."

The next milestone is "first teapot."

When you render the teapot, people see the teapot.

But the demo is that you made the system to render the teapot, not the teapot!

They understand what they see, not how it works.

Invest effort to optimize it in proportion to how often it's used.

For example, in V8, the first pass is a quick-and-dirty compile.

Optimize it from generalized/sloppy to specific/tight in proportion to how many times it runs.

The same is true for workflows that use LLMs.

If it's going to run once, just have an LLM execute an english language prompt.

But if you're going to run it thousands of times, have the LLM write deterministic code.

Right now we don't have enough real-world last-mile use of AI workflows so we're in the cheap/sloppy phase of deployment, not yet to the auto-tightening parts.

Inter: You need to coordinate with others.

Intra: each step is dependent on the others and possibly needs to change.

To me this implies that the current model of software is stagnant.

The personal OODA loop is the classic Douglas Adams quote:

But sometimes the situation really does have a faster OODA loop than before.

The ability fo LLMs to write software is truly a fundamental speed limit that has changed.

If it's not you… is it the singularity?

"I can absorb this new change once there's a breather" …but the breather never comes.

Geek machismo leads to a kind of Robert Moses effect with much more leverage.

Putting on blinders and steamrolling the world.^[il]

They think they're being hyper rational.

In reality they've hollowed themselves out.

The heart is where the meaning comes from.

To be in harmony with the world you need both.

But so do Radagasts.

Bureaucracy is about the status quo and downside capping.

Innovation is about upside.

Both the Saruman and Radagast magic are about innovation and upside.

The lack of magic, the dull, dreary, mundane company man, is about not innovation.

A lack of conflict-aversion as a super power to dominate over others.

Finance and titans of industry typically have this power.

It's an inherently antisocial power.

When there's more friction it can create different pockets of smaller winners.

More variation leads to more diversity; more adaptability and resilience in the ecosystem.

One empire can spread throughout a vast plain; mountainous terrain often has fractal chiefdoms.

Somewhat surprisingly, low friction leads to hyper optimization leads to centralization leads to hollowness and fragility of the system.

The system is overfit.

Whether it's overfit is whether you have a model that doesn't generalize in time to new scenarios since it fits to what is temporary noise.

Overfit structures are 'fit' but not in a robust way.

Even if you were exactly right about prediction for today you'd be wrong tomorrow.

The world changes in ways you can't predict, fundamentally

This riff and the next 10 were based on a talk I attended this week by Emmet Shear.

It's overly optimized.

Modernity is about the systematizing of accuracy.

You create a meta model of the world, and then every process relentlessly optimizes it.

If there's no slack in the system you are definitionally overfit.

It's that it's impossible.

You can't take a photograph of Métis^[jm].^[jn]

The more complexity you add, the more you overfit.

You explain what you see well, but also over-explain noise.

You're now overfit, poorly fit to novel inputs.

If you update your model by n bits it had better give you more than n bits of accuracy or you're falling behind.

The unknown unknowns.

The best you can say is "I'm still alive so whatever I did in the past must have worked…"

The more connected, the more overfit the system gets.

By every metric things look great.

But everyone can tell something is off.

It's hollow, hyper.

Manic trips end in one of two ways: you calm down or the universe calms you down.

But that's not true.

Local connections are healthier and more nourishing.

They allow more diversity, variance, and adaptive capacity of the system.

Don't follow high frequency information sources.

Talk to your friends and stay local, it's good for you and good for society.

In ML, one way of regularizing is keeping track of, for each weight, how precise you believe it to be.

Another approach is simply gradient clipping.

It's less precise individually, but on average, stochastically it is apparently equivalent.

If it goes faster, it gets taxed, at a rate that goes up quadratically.

The main knob to control is the tax rate.

This would naturally tilt information to local sources.

Downward pressure on inequality.

You could argue that a system of tariffs (consistently and thoughtfully applied) is a form of gradient clipping in this context.

Obviously, an unworkable proposal, but still a fascinating thought experiment.

No variance to select over!

Global connection makes the world boring and wildly unequal.

A day after it happens, not as-it-happens.

That gives some time for synthesis and distillation.

It also gives some time for perspective: what was the stack rank of most important things that happened yesterday?

Online sources can have infinite content, updated infinitely quickly.

But newspapers have scarce space and have a built in time delay.

They naturally regularize the signal.

Even better would be once-a-week roll-ups of curated what matters most.

Before they were a signal of quality because they took a long time to write.

That meant that professors only had time for a handful of them, so the students they agreed to do it for were at the top of their distribution.

But now professors can do them 100x faster, so they can do many more.

Some new signal will emerge as a quality signal.

Quality signals come from scarcity.

One way to bring back scarcity: have professors publish the names of their top recommendations.

It locks you on a hyper-growth-or-bust trajectory.

Even if you don't want to get on that trajectory, if your competitor does it, they will dominate you in the market.

That means that even if everyone would prefer not to, everyone must.

Anywhere where there's no moat (bits, not atoms) and some kind of compounding effects, you have to play.

It gave an interesting report.^[ip]

This is the kind of question I wouldn't have even bothered asking before.

Unless the world quickly disabuses you of that notion.

This is what Simon calls an inferred definition.

The process of ping-ponging ideas back and forth unearths mutually interesting ideas.

Each volley is:

An iterative "yes, and" that zeroes in on the most interesting thread.

The fact that two people find the thread interesting makes it an order of magnitude more likely another person would, too.

Everything needs to be in the critical point to be able to surf.

Control is impossible and attempting to have it will tire you out.

A great piece from Every that calls it Rugged Flexibility.

But the abstraction misses some of the grit and texture of the real world.

The reason things are hard in practice is because that grit is load bearing.

Rocket science is mostly just the fundamental dynamics.

Problems that are complex are almost all grit and texture.

There are things that LLMs could do that you shouldn't do.

But then you could end up like Blackberry.

In an infinite game, rule number one is to stay in the game.

Finite game within (zero sum), infinite game for the collective (positive sum).

Finite games that are run too hot will steal all the oxygen.

For example: the idea of America is the infinite game.

The hyper era has much more efficient competition.

Finite games hollow out infinite ones at faster and faster speeds.

The antidote is trust in the collective and long-term goals.

When individuals trust each other to behave as a collective they believe in, they will take actions that don't follow Goodhart's Law and don't destroy the collective.

That gives you the best of both worlds.

Similar to Francis Frei's observation that diverse teams that trust one another are the way to get reliably great results.

No vantage point to look non-locally.

Past that you need something to give you leverage.

Typically that requires reducing nuance to numbers.

Once you do that, Goodhart's law starts showing up.