Bits and Bobs 7/14/25

"The app will bend to your will."

Being a PM is hard!

Thinking through the implications of features in your software.

It takes taste, judgement, experience, savviness.

An observation this week: "I can see a future where it's much faster for all employees to use an IDE with .md files + git instead of Notion, Google Docs or Office."

Contains this definition from Tyler Cowen: "Context is that which is scarce."

Infinite software.

Software for every need, just waiting to be discovered.

Gets at the distinction between tinkering and vibe-coding.

It's about how active the stance is.

Are you pair programming with the LLM or outsourcing your thinking to it?

Collaborative curiosity.^[et]

This distinction also gets at why Bret Victor doesn't like LLMs: it moves people to an outsourced thinking position.

Think of what WebSim was at the beginning.

A software experience that hallucinates everything on demand.

It looks like normal software superficially, but underneath it's 100% LLM.

It seems like that kind of software might be too squishy.

Like jello.

Impossible to pin to the wall and get it to do structured things consistently.

Still, a fascinating experiment in the era of infinite software!

But I don't think it goes far enough.

It still has the LLM at the root of the loop, calling the shots, deciding what to rely on.

But any system with an LLM in the driver's seat is prone to prompt injection.

Why not have codegenned code be the root of the loop?

This is the source of the "logarithmic value / exponential cost" curve I've talked about in the past.

Strength of momentum and height of ceiling are disjoint.

MCP has strong momentum but a low ceiling.

The system seizes up, frozen, constrained in a straight jacket.

MCP assumes high levels of trust that are not warranted.

Fine for savvy users tinkering on the command line, a non-starter for everyone else.

Humans did that for the web (deciding what to link to) and writing (deciding what to bother writing in the first place).

LLMs can't vouch for new things in an authentic way.

When human publishing slows down, how will we synthesize new knowledge?

Context is a stepping stone that gives you leverage.

Good context is great.

Bad context is terrible.

When LLMs do it themselves they do middling or even bad context and that spirals.^[fa]

That's why things like Claude.md are not a hack.

They are the human making higher level contextual assertions.

Claude leaving Claude.md notes to itself help improve quality in your codebase is an example of a coactive surface.

The human can then co-create that context, editing, curating, adding.

How can we generalize that interaction?

It can be derived from it.

But it is, as the name implies, contextual.

There are many questions that can't be answered that way!

I found the HackerNews thread interesting.

The Supabase team was discussing the mitigations they were taking to make MCP infiltrations less likely, including by automatically extending prompts with stronger instructions.

Many of the comments were pointing out that it doesn't make sense to treat MCP as the security boundary.

Finally people are starting to wake up to the LLM not being able to be used as a security boundary.

The problem wasn't prompt injection per se, it was just a poorly configured and secured system.

Still, I imagine we'll see a lot of these kinds of things with companies eager to integrate AI into their publicly-exposed systems.

Chromium is easy to fork.

However, browser distribution is extremely expensive.

The only player that could plausibly have the juice to do it in the last decade is OpenAI.

OpenAI will try to redefine the browser category.

They'll say "you can use any model! Look, it's open!"

But no one changes the defaults, of course.

Then the model is beholden to the aggregated UX and consumer demand of the aggregator.

But it turns out that AI is the feature that is so important that everything else is secondary.

An AI UX that is better than what an extension can do is worth it.

It's not an IDE value prop. It's a totally new pacelayer on top.

"Neighbor" is what communities call their members.

Prosocial tech builds communities.

Antisocial tech builds dependencies.

Antisocial disengages us from what matters.

Prosocial tech strengthens the social fabric: the invisible bonds between us.

Antisocial tech harvests those bonds for engagement metrics.

Context collapse flattens the world.

Makes it efficient but inhuman.

A zombie.

We are fractal.

We contain multitudes!

We are different people in different contexts.^[fb][fc][fd]

Prosocial tech must grapple with that fact.

It's about the whole modern societal context.

We live in the Antisocial Age.

Prosocial is about becoming the version of yourself you want to be, engaged with the community that surrounds you.

Prosocial systems bloom with emergent collective intelligence.

There is intrinsic value in knowing other people were involved to create the value; and that your actions improve the system for the collective, too.

At some point that is actually more important than a marginal increase in quality that might come from the collective intelligence.

Utopias are static end-states where all problems have been solved.

They're impossible.

Protopia is a term coined by Kevin Kelly.

Rather than perfection, protopia describes a state of continuous, incremental progress.

In a protopia, things get a little better each day through small improvements, but problems still exist and new challenges emerge alongside solutions.

But because of the hyper financialization and legibility baked in, it was like Goodhart's law on hyperdrive.

The humanity was driven out of the system.

Web3 was, accidentally, even more antisocial than Web2 ended up being.

The fastest one-ply thinkers.

That's what gives us Goodhart's law at the level of society; everyone thinking a single ply and not thinking at all about the implications of their actions.

One-ply thinkers who are handsomely rewarded might not even realize there's anything at all they're missing.

You need to be able to think systematically, but also situated, about what people really want.

"Alignment is defined as reducing the misalignment between the value system of the LLM and the value system of who pays to make it / keep it running.

Either there is a single absolute value system, a sort of platonic eigenvector of goodness, or, there isn't.

If there isn't, then alignment for me is propaganda for you.

ust looking at the history of political parties and newspaper editorial rooms in history it suggests that "eigenvector alignment" is a non-starter.

So we're left with just another "what's good for me is evil for you", as predicted."

As a friend observed, "It turns out we should have been more worried about the alignment of the leaders of the AI labs than about the alignment of the models."^[fe]

But now, with LLMs, you can.

It only took us a couple thousand years.

In a safe system, code that touches something so sensitive should isolate itself.

In today's code, that isolation has to be done by construction, by security experts at design time.

To have a system capable of working with infinite software, it will have to be antifragile; auto-isolating code that has been tainted by sensitive data.

That makes any given observation swamped by the noise and thus not identifying, but allows the overall true shape of the distribution to be visible at scale.

This exploits the "consistent bias at enough scale can emerge even from noisy streams" phenomena that powers all emergent phenomena–including evolutionary gradients.

Very easy to have one or the other, much harder to have both.

The former is the primary use case.

The latter is the secondary use case, the bonus.

You can't sell a product to someone based on a secondary use case.

But you can make a product that sticks with people because of the secondary use case.

If it doesn't exist, is it because no one else thought of it?

Or because everyone who tried it failed, or found it not worth the cost.

Novelty has this problem.

You need to convince yourself it's something that everyone else missed, otherwise it's likely just not viable.

Greatness is about the ceiling.

Goodness is about the floor.

If it's already great, being good as well is useful.

See what the LLM expects to be there and do it.

Infill product development.

Do the obvious thing that's missing.

It's about gap-filling.

Doesn't make your thing great, but it can make it good.

I would have expected it to be a structural limitation.

Turns out their body plan could have gone even bigger… but the carnivores' body plan couldn't go any bigger, so there was no pressure for the herbivores to get even bigger.

The precise approach plate is extremely finicky, and it's very easy to get wrong.

Garmin apparently has a product that will tweak the approach plate for landing your particular plane in these particular conditions.

This gives a sense for the value of context.

If you're flying based on instruments (e.g. auto-curated context), then it's imperative you trust that the curator didn't make a mistake.

If they think you're a 747 but you're a Cessna, you're going to crash.

The distiller could do a bad job distilling the wrong things, or think you have different goals, desires, or context than you actually do.

But it's not good for execution.

People who prefer order of magnitude analysis are terrible estimators for real world processes.

If you only like nudges, flexible, open, option value, you end up confusing people, creating eddy currents.

Some situations require laminar flow for things to cohere.

Trust is an emergent social phenomena.

What you want is scaled trust.

Some infrastructure levers trust, and some destroys trust.

Infrastructure that helps your trust go further helps society thrive.

The LLM has come.

When will our Martin Luther emerge?

There are two stable strategies at the extreme.

Everything in between slides down the slope towards one of the extremes.

R-selected systems are open-ended.

K-selected systems are close-ended.

The benefits of the two types of systems are powerful, and more powerful when in juxtaposition to the other.

That's why the world tends to strirate in interleaving layers of r- and k-selected systems.

The world pushes from mush to posterized, fractal, swirling interleaving of open and closed systems.

There's many more ways to be wrong than right.

If you care about being right you have to invest much more effort per statement.

If you don't care about being right you can overwhelm those who do.

R-selected vs k-selected.

The result is that we are awash in a tsunami of bullshit.

This was a minor annoyance but now that we're in the Cacophonous Age it's the driving asymmetry of our time.

But by incentivizing it you modify the dynamics and break the causal model.

Since the causal model is implied, you don't even think about what you're doing as you smash it to smithereens.

Alignment over competence.

This emerges fundamentally from the emergent imperative to "act like your boss is right".

If you don't abide by this rule, you have a structurally larger chance of being knocked out of the game.

A consistent bias that emerges clear as day if you average all of the noisy examples together.

The choice of metrics is endogenous to the system.

If every update to the manager the team chooses different metrics to highlight, even if they're "doing well as reported by the metrics" they're actually doing poorly.

Consistent progress on consistent metrics shows that things are going well.

At a certain point however the metric decoheres from success and becomes an accidental end in and of itself.

At the beginning you want a consistent metric; at the end that's dangerous.

But CUJs are linear; it's impossible to capture the combinatorial feature set of the system.

All of the paths that aren't covered by CUJs rot.

This can easily lead to most of the product rotting while the part the product owners measure seems to be doing better than ever.

An emergent system can easily be constructed where caretakers who intrinsically care and are given space to follow their gut gives great results

A provocative but generative frame.

Ads, it posits, are a runaway steamroller that feeds on the host but will eventually kill it.

Goodhart's law: the swarm will inescapably cheat and kill the collective.

The swarm and the collective must be kept in harmonious balance.

If you allow the swarm to swarm unimpeded it devours the soul of the system.

The spirit is about an infinite, a prosocial stance, trust, a positive sum, an ends.

A focus on the letter of the law is about means, "if it's legal it's allowed"^[ff]

For example, consider someone being frustrated that a billionaire pays few taxes due to a loophole, and the billionaire being upset that anyone is mad they are following their incentives within the rules that were given.

From the perspective of the tax payer: "whatever is legal is allowed."

From the collective perspective "whatever is not aligned with the intention is immoral."

These two perspectives are distinct!

It's impossible for them to be fully aligned, for the same reason it's impossible for the incentives of the collective and the individual to ever be perfectly aligned.

They give a point for high-quality things to congeal and emerge.

Folksonomies work because their UX allows the potential for schelling points to emerge.

Convex systems are auto-decohering.

An infinite difference.

The difference cannot be seen from a static snapshot.

It can only be seen from the system in motion.

Even then you have to carefully study the flow patterns of micro movements.

The small biases and asymmetries, beneath the distraction of the noise.

I was reminded of this by this essay.

On a slippery slope, each step is fine, but suddenly you look up and go, "wait...".

By that time your ego is so tied to it it's hard to admit it, that you followed your incentives to a place you would have never gone intentionally.

If they realize they've done bad things, they can see how the incentives created a situation where they were channeled into it emergently.

But when we look at others we can see clearly how they are the baddies; intrinsically bad.

Imagine if the baddie did what you are about to do and see how you'd feel about it if they did it.

If you feel bad about it, don't do it!

This helps invert the fundamental attribution error and help you understand the emergent effects of your actions–especially the bad ones.

We move to where the economy wants us, undermining family and community.

Despite all of the noise in the real world of specific examples, there's a consistent bias.

A structural difference.

What effect might this have at scale in society?^{[fg][fh][fi][fj]}

A small consistent bias over noisy data of high gumption people who decide to emigrate.

People know that if they come here and succeed they can be accepted as truly American, not American with an asterisk.

It's the best place for high potential people to try their hand at striving for the top.

That asymmetry gives a massive advantage to the US.

We're currently breaking that engine.^[fk]

It will be years before it's obvious what we've lost.

Goodhart's law on steroids.

Paperclip maxing ourselves to death.

Heaven help us.

Over long enough time scales, culture is downstream of incentives.

Incentives are downstream of business model.

When in the niche, you get the benefits of being famous.

But you can step outside of the niche and get rid of the downsides of being famous.

For a small enough niche, everyone can be niche famous.

Power is often invisible but extremely important.

So the more powerful will say "why do I get increased standards? It's unfair!"

Because the power is invisible to them ("that's just the way it is, people respect me") but the constraint is visible to them, it feels unfair, even thought it's not.

If you broke it, you're responsible for it.

You own it even if you don't want to.

Subversive and understated but worldly and clever.

No individual subset of it is dangerous, it's only between the lines you can see how much it questions the status quo.

The things you choose to do despite being costly, that are "irrational.'

They're infinite, they're emergently meaningful.

Emergent components can't be seen concretely; if you keep cutting them down into smaller and smaller pieces, the magic evaporates and you'll never see it.

If you don't believe in emergence you can't believe in infinite things.

A means is just the sum of its parts. Finite.