Bits and Bobs 6/9/25

I can see why the provider is going for it, but I can't see why it's good for users.

Your user experience, your memory, and your model will all be tied together.^[jl][jm]

The power of the memory will make it harder and harder for other experiences to compete.

It will accelerate the aggregation around ChatGPT.

Good for OpenAI, bad for everyone else.

What could possibly go wrong?

"Autonomic Systems will be key to the next phase of AI. This is not about parameter count and activation functions or transformers vs RNNs, it's the vast machinery of ordinary code wrapping the models, taking their output and bending it back as input for the next go round."

"Everyone was watching the magician's right hand, expecting recursive self-improvement to involve AI updating its own weights, but the real trick was hidden in the left hand—using the LLM to write code for the Autonomic System that envelops the model itself. "

"The Autonomic System surrounding an ML model is like the structure of culture and society around a human mind. Laws and markets and your mother's expectations are not themselves part of your neural net, but without them shaping your inputs, your outputs would be very different."

"Human history began when WE got Super-Autonomic Systems. After we evolved intelligence, the IQ of a normal human was about the same for many millennia. The big jumps in civilization were because of structure and peripherals: fire, writing, machines, the rule of law, code..."^[jp]

Only if they have your personal, unique background bringing novelty to the question will you get novel answers^[jq][jr].

That's one of the reasons for the Hayes Valley demo fatigue.

And yet it's what everyone demos^[jt]!

If you're going to have a memex or an exocortex, it's imperative it's aligned with your intention: your agency and your aspirations.

"Thinking doesn't matter without a notion of salience."

Implies to me that in the era of intelligence too cheap to meter, what will matter is the curated context.

An ontology is like a map, a useful distillation and structure to help you achieve a task.

The task you're trying to achieve sets the context for what dimensions are most useful and which are extraneous.

It's much easier to start a task if you can just dump unstructured information and then structure it continuously over time.

But in mechanistic systems, to get useful insights you needed structure: fitting your information into a given ontology.

In mechanistic systems, that had to happen up front, putting a damper on the very first step.

But LLMs can do qualitative insights at quantitative scale.

That means that more interfaces can allow a flexible data entry with clean up later.

Especially if the LLM can help post-host structure.

Post-hoc ontologization.^[ju][jv]

If it gets past a certain point, it doesn't reflect reality and you go bankrupt.

It decoheres, untethered.

Their tinkering to solve their own problems also implicitly helps others, too.

In mechanistic systems, protocols must be extremely precisely defined, with formal rules.

This formalization has exponential cost with scope.

The larger the formalization process, the harder it is to coordinate to consensus, super-linearly.

This double super-linear blowup makes standardization and coordination extremely hard.

But LLMs are able to take whatever sloppy data you can give it.

That allows way easier interoperability than before, because you can keep the protocol more informal.

Systems of record have been too close-ended, and too much effort to maintain.

Information aggregation is most viable centralized with a clear economic value.

Data brokers are very interested in forming a searchable information set on you.

Perhaps it's just banal enough that it's not worth it for a user to pay for it.

Perhaps LLMs will make it easy enough for users to do and take back our power.

Claude Code does this a lot, spinning up little sub-agents that are isolated from the main flow often.

The agent is a secure little compute environment that bounds what data it can see.

That makes it so the reasoning doesn't muddy up the main context window, and vice versa.

In complex adaptive systems, boundaries always emerge to handle the compounding cacophony.

This reverse engineering of Claude Code by Simon Willison is fascinating.

Imagine the system having to figure out who my husband is each time.

But if it ever figures it out (or if I ever tell it), it can simply make a note that my husband is Daniel.

From that point on, it can take that knowledge for granted.

That pre-existing knowledge becomes like a platform, allowing jumping off to farther and farther afield insights.

If you have confidence that information you put into the system will be useful to you in the future, then you'll do it.

Even better if you have confidence that it will become more useful the more you put in, and the more powerful the system becomes.

The lower the overhead and annoyance of putting information in, and the more likely something useful comes out later, the more you'll be willing to invest time in putting things into the system.

They only allow a subset of allow-listed MCP instances for certain use cases.

This will quickly evolve into a kind of app-store distribution system.

But this is also inevitable given the security and privacy implications of MCP.

MCP is extraordinarily dangerous–not only potentially malicious integrations, but also prompt injection within the data in those integrations.

When you run a local client with MCP integrations, it's clearly your fault if an MCP integration bites you.

But if you're a less-savvy consumer using a feature of a popular chat app and MCP bites you, you're more likely to blame the chat app creator.

MCP is not the right way to solve the integration problem for AI.

They show them not because they think users will want them, or even because they think some users might answer differently.

Even if 99.99% of users will answer 'Yes', they still show them.

Because they want to make sure that if something bad happens they can't be sued.

By asking permission they've moved the liability onto the user.

We've seen this movie before: new integration tech, huge promise, completely bonkers security assumptions.

We already know how this movie ends.

If you want to dig in more, I fed my recent Bits and Bobs into ChatGPT's Deep Research and it gave a more in depth report diving into the parallels.

For example, here a non-technical person is livestreaming his vibe-coding of service, but leaving open many significant security issues.

In this Hacker News thread about how Claude Code will route around restrictions the user set on `rm`, most of the response is, "yeah but of course it can, the user should not be surprised."

People reacted to the Github prompt injection attack by saying "well the user shouldn't have granted such a broadly scoped key."

MCP and LLMs make it so more and more people can put themselves in real danger and not realize it.

The answer is not to blame the users.

That's like blaming people who use Q-tips to clean their ears.

The protections around LLMs cannot contain their power. How would you contain them?

The model of "if the user clicked a permission prompt it's on them for getting pwned" is insufficient in a world of LLMs.

They're simply too powerful to be contained by our previous half-assed containment mechanisms.

Imagine if OpenAI had been in a position where they simply didn't have the logs.

But aggregators must have logs, it's their business imperative.

Try launching a new messaging app without E2EE today.

Even though users don't even understand what that technical word means.

WhatsApp and iMessage set the bar and now it's table stakes.

Once there's an existence proof of a thing that has no downsides but is more private, users will demand it.

E2EE is one of those things that doesn't add friction, it just makes it better, it's good privacy technology.

So you can have lower friction, which gives more usage.

People's naive view of privacy technology is "more annoying prompts that get in the way."

That's bad privacy technology.

One of the web's superpowers.

It might be more powerful than you in speed or capability… or both.

It's not just being outgunned, but not even being aware of how outgunned you are.

It might be able to control you in ways you might not be able to sense, let alone understand.

True for scramblers in Blindsight, but also true with anything that might be an order of magnitude more capable than you.

It could be playing you and you'd have no idea.

A relationship with an asymmetrically powerful intelligence is all-encompassing, it's more of an environment.

If you're below the API, you are abstracted away to the rest of the system, more grist for the mill.

Whether this new world is good or bad for you largely reduces to if you're above or below the API.

If you are under the API, you are operating at its whim.

Even if you have a system that is working for you, helping you optimize what information to pay attention to and what tasks to prioritize, you could easily become "below the API" to the outside world.

By relying on the system, you have become totally captured by it.

They happen in contexts where it's a task you have to do, not one you want to do.

We do highly repetitive tasks in physical hobbies (e.g. knitting) but rarely in intellectual hobbies.

Mindless, addicting games are one possible exception.

The metacrap fallacy was "Once users have put enough meta-structure on their data, all kinds of automatic things will become possible."^[ka][kb]

But fitting things into an ontology up front is a massive amount of work, and the benefit is only theoretical and indirect.

So the direct cost beat the indirect benefit and made it so no one ever did it.

But now LLMs can be used to auto-structure information after the fact.

LLMs don't get bored, so they could do the structuring even if a human would die of boredom.

It's totally possible that the reason the metacrap fallacy was true was not "it wouldn't be valuable if you didn't have structure in the information" but rather "it's too much of a pain in the butt to structure things."

Imagine tagging a person, and being about to put on the tag #husband, and seeing in the UI that there are ten times more uses of #spouse.

It makes sense to tag #spouse because that will overlap with other programs and usage.

That logic of "is this a close enough match to be worth doing the more popular tag" can be done by a user or an LLM equivalently.

One of Gmail's insights was "if search is fast and storage is cheap, search, don't sort."

LLMs make sifting through massive information fast.

What's the equivalent insight?

The question is what do real humans in that situation think?

If somehow you could have an emergent, real process of those decisions across the anonymous swarm of real users, you'd get very useful suggestions.

If the LLM doesn't use your context for personalization, it feels like you're in its world.

How do you create a safe way for services to know you without being dangerous or creepy?

Like "How come you have red hair but everyone else in your family doesn't?"

LLMs should have at least that much tact, but sometimes they don't.

If someone asks "What's the most embarrassing thing you know about me?", the LLM should at least first check "can other people see your screen?"

Don't fall for it!

Compared to people building a crappy app they think will be the next big thing.

The former cares about having their data first.

Vibe coding to add features to a thing you already are using.

The mechanistic loops are formal graphs of computation, which may inside them have LLMs calls, but which are sandboxed and limited.

There is an agent loop but it makes a compute graph to execute that calls tools and also sub-agents whose job is to construct another graph to execute.

The agent doesn't execute, it makes a graph to execute.

The core ranking function would be "if this were to run how likely would the user be to accept its suggestions in this moment?"

It binds together incentives and desires.

Helping people know about things they need.

But advertising's benefit is balanced in favor of the advertiser, not the user.

What would a personalization system look like that was balanced to the benefit of the user?

Suggestions that are perfectly aligned with users' incentives would be much better than advertising for end users.

But also pretty good for advertisers, too, because it would be so aligned with users.

But software at scale is dangerous.

That's because software is turing complete, it can do things.

Only the owner of the origin is allowed to decide what they do.

They put users into a consumer vs creator stance by default.

Good, but only to a limit.

Could start as Google Keep but coactive and turing complete.

From there it could grow to cover everything.

You don't have to think about it, your intentions are well met.

Imagine: infinite software with a disappearing interface.

Your data comes alive with an interface that melts away.

Because no other can never be fully aligned with you, there must be some separation.

If it has its own perspective on the world, there's a principal agent problem.

This gets especially pronounced if the agent is more powerful than you.

"I can't do that, Dave."

Untrusted code - Creates open-endedness.

Sensitive data - Can work on real and specific things, not just generic things.

Network access - Can interact with the rest of the world, not just an island or dead-end.

No reason to start, because it's clearly not the future.

But we'd get a lot of value if we could have an open ended tool that could do a bunch of those things.

It mattered that it was a web page, not what was on it.

The browser / platform is open-ended and blooms with possibility.

As long as the first users of the first web site like it, the open endedness will carry the system through.

A trap: running at a use case and making it so great that even people who hate doing it will like it.

A very hard bar to hit.

Related to the tyranny of the marginal user, but before you even get to PMF.

Imagine a horizontal system that can do what nobody else does: chain all of the experiences together.

Each vertical app has to get data on its own, at high friction.

That sets a very high floor, lots of stuff that's not viable.

The value of the new system is that it's horizontal.

Any vertical slice will not show it off.

The power would be its open-endedness and horizontality. ^[ke][kf]

But there's no viable alternatives.

For most people, LLMs are a "heck yes" but ChatGPT itself is not a "heck yes".^[kg]

It's a "this is the best way to use LLMs today"

They'll jump to the better way of interacting with LLMs when they can.

"In the heyday of European imperialism, conquistadors and merchants bought entire islands and countries in exchange for coloured beads. In the twenty-first century our personal data is probably the most valuable resource most humans still have to offer, and we are giving it to the tech giants in exchange for email services and funny cat videos."

Without it, the feedback cycles are eons from the perspective of the AI being trained.

Areas that can be simulated well will have computers get radically better quickly.

But areas that aren't possible to simulate, e.g. complex phenomena with interactions, will remain somewhat difficult.

Humans can be good at them: just do an action and see how the distributed computer of the real world responds.

But LLMs can't do that, and are limited to simulations.

In some contexts, the simulations are good enough already (e.g. protein folding) but in other contexts, they're nowhere near good enough.

The service buys the credit reports of nurses on the platform, and offers nurses who have debt a lower fee for completing jobs.

This is gross–it's not about "how able is this person to pay back a debt to me" but about "what is the lowest price I can pay this person."

Capitalism, like any swarming phenomena, is powerful, but has monkey paw kinds of dynamics due to Goodhart's Law.

The frame of "Uber for" is inherently short-term and transactional.

What's gross is this powerful entity squeezing less powerful entities.

What if the gig workers had agents that collectively bargained on their behalf?^[kh]

Goodhart's law arises because the metric must be a proxy.

Because it's a proxy, there are ways to "cheat" that improve the proxy but not the reality.

If the members of the swarm are not fully committed to the good of the collective (if there's any principal agent problem) then they will be optimized to cheat because it is a cheaper way to improve the proxy.

You can get the swarm to not Goodhart's law if all of the members feel an infinite connection to the collective over their individual incentives.

This can happen if they view themselves as fundamentally a subcomponent of the swarm, and only secondarily as an individual.

That can also happen for example if they are perfect clones of one another.

A worker bee.

The coordination cost goes away because it's no longer a swarm.

But is it possible for that non-swarm intelligence to be resilient and adaptive enough?

If everyone is a clone then the system has no resilience, you have systemic collapse risk.

Coordination cost scales at a super-linear rate, but intelligence in an individual scales at linear rate.

The main difference in both is: Is the 3P stuff on the inside or outside?

People use "platform" to talk about any of these meanings (less often library), but they have very different power dynamics.

"Platform" means "a thing that the things on top could not exist without."

The AI in this case is just capitalism.

Both are swarm intelligences.

Powerful but impossible to fully align.

Monkey paw intelligence.

Be careful what you wish for!

Every system follows the path of its incentives.

If you work at an engagement-maxing company, you'll tweak to add a little more engagement, a little more addictive, a little more manipulation, over and over again.

No individual step is terrible, just a teensy tweak,

But after a decade you look up and realize how far the system has gone.

Unfortunately if everyone does it it's a humanity-destroying cycle.

How can we make caring cool again?^[ki]

But they feel like that's the future that will happen because those groups have power and are highly motivated to achieve their vision.

Everyone else just gets resigned.

Just a "screw it I might as well optimize for my short term self interest."

They disengage.

Scattered among hobby projects, homesteading, not a coherent movement together.

We all see the machine, but see we can't win individually, so we just retreat to the woods.^[kj]

It's impossible to have one centralized vision for us because we don't want one centralized thing.

But what if there's one open system we can all rally around?

I watched it on a long-haul flight.

Maybe it was just the altitude making me more susceptible to hokey things.

Imagine if you made a machine that grew out of your control that changed all of society and made you billions of dollars… and then realized it was antisocial.

I didn't watch Black Mirror for the past few years because it was too hard to watch.

Now I see that was the coward's way out. I need to watch and I need it to affect me.

Everyone who is in a position of power in tech should feel compelled to watch Black Mirror.

If you can't bear to watch it, maybe you shouldn't work in tech, or you need to align what you work on with your values.

Compute your own destiny.

I will not be fracked.

Own your digital life.

Gardens, not plantations.

Jailbreak your digital soul.

"Marshall McLuhan wrote that every augmentation is also an amputation"

"Today, quite suddenly, billions of people have access to AI systems that provide augmentations, and inflict amputations, far more substantial than anything McLuhan could have imagined."

You need both intimacy and agency for a system to be healthy.

It's notable that the person I linked to last week, Derek Thompson, is a proponent of abundance but also notes the lack of intimacy.

As our tools got better, our belonging got worse.^[kl][km]

We're missing the meso scale of communities.

The meso scale of communities includes people who challenge you or you don't like.

Mark Fisher has a notion of hauntology, a sense of unease and nostalgia, in the absence of a compelling vision of the future.

1) "I like…" - Positive feedback

2) "I wish…" - Constructive feedback with a built-in vector of improvement

3) "I wonder…" - Open-ended jumping off points from this work

The collective is antifragile because the swarm of individuals can't die as long as any of them are alive.

As long as one cell survives the mold survives.

This property also means they're very hard to control; there is no single leverage point.

A single leverage point makes a system easier to control… and also easier to kill.

Emergence creates detail.

A small genotype creates a universe of phenotypes.

Most phenotypes that can be built can't be expressed in genotypes.

The subset that can are what can "unfold" or emerge.

This subset is "alive," auto-catalyzing.

They are beautiful and rare and yet nearly everything we see is caused by one of them.

"Adaptation requires two things: mutation and selection. Mutation produces variety and deviation; selection kills off the least functional mutations. Our old, craft-based, pre-computer system of professional practice-in medicine and in other fields-was all mutation and no selection. There was plenty of room for individuals to do things differently from the norm; everyone could be an innovator. But there was no real mechanism for weeding out bad ideas or practices.

Computerization, by contrast, is all selection and no mutation. Leaders install a monolith, and the smallest changes require a committee decision, plus weeks of testing and debugging to make sure that fixing the daylight-saving-time problem, say, doesn't wreck some other, distant part of the system."

All animal cells are remarkably similar.

But at the macro scale the animals that are made up of cells are wildly diverse.

A boring lower level gives rise to tons of innovation at the layers on top.

It needs to be sharp, not dull.

That point is a nucleation site where the energy can attract and condense.

Sometimes though you can overlearn.

You've overperfected your knowledge for what you need.

Too much fidelity, not enough doing.

The doing is what ground truths and helps figure out where your mental model is incorrect.

It's the same asymptotic curve of perfection that is a bad idea in building products.

Overfitting to a simulation.

The smarter you are the more you trust your simulation.

Your simulation is always wrong.

It's easy to accidentally breathe your own exhaust and get high on it.

20% is the fun open-ended research.

When you're in a big open ended domain it's possible to do the fun part the entire time.

But only if you do the grind do you get to where you need to be.

If the grind work is invisible to you, you'll think "we're one week out" from the breakthrough forever.

The research stuff gives the feeling of insight without seeing if it actually works in practice for real use cases.

If you're working with someone who's only as good (or not quite as good) as you on the dimensions you're an expert on, but spikes in dimensions you don't sense, then you'll undercount their insights.

You'll see the banality of their insights on the dimensions you sense, but miss the novel insights on the dimensions you don't sense.

Aha moments that change the world are what matter.

The feeling is "I will explode if I don't get this out of my head right this second."

Writing is a kind of self-soothing behavior I do to calm the pain of not writing an idea down.

Later, I can clean up whatever I wrote up in that mode into something a bit more presentable.

These explosions of writing happen in little bursts, when my schedule gives a small window for them to happen.

You might call this the Mulan maneuver, from how she solved the task to get to the top of the pole at the training camp.

Another form of leaning into the weight you're saddled with is to use it to do a slingshot maneuver.

After enlightenment, you carry the water and you chop the wood.