Bits and Bobs 12/15/25

Google's Agentic AI wipes user's entire HDD without permission in catastrophic failure.

Zero-Click Agentic Browser Attack Can Delete Entire Google Drive Using Crafted Emails.

MCP Sampling as a new Prompt Injection vector.

ZDNet: Scammers are poisoning AI search results to steer you straight into their traps.

The UK's NCSC warns: Prompt injection is not SQL injection (it may be worse)

Gartner recommends that cybersecurity teams block AI browsers.

Just like software ate the world.

I heard Brendan deliver this as a speech and loved it so much I encouraged him to post it publicly.

Humans are in the world: they have a feedback loop to see how their actions affect the world, to be able to update their mental model.

LLMs are not in the world, not directly.

They have no direct learning loop.

They don't learn or change from loop to loop.

The most obvious versions of agents… and the least useful.

Real agents have to accumulate internal state.^[eh]

Otherwise, it's just a chatbot.

The context is things it learned, or things the user told it, or the new tools it created.

Each tool it makes for one purpose is a stepping stone to use in the future, too.

A little cached toe hold.

As you accumulate those toeholds, now and into the future, you get compounding benefits.

If you aren't using tools like Claude Code every day, you will have no idea what the fuss is about.

Claude Code applied to your daily life is insanely powerful.

How do you take the magic of what's possible with Claude Cde on the terminal and make it more mass market?

Each additional minute really could create levered value.

It's "rational" to do that one more action!

It's just that it never stops.

It reduces inhibition, not unlike alcohol.

"Go for it! What's the worst that could happen?"

You can really get yourself in trouble, because you'll attempt things you never would have attempted before.

AI-generated ads do better than both human-created and human-assisted-with-AI ads.

But if you disclose it's generated by AI, it does worse.

Reminds me of the old psych paper that participants rated random nonsense sentences as more meaningful than human-crafted ones.

Engines got better at a steady pace for a hundred or more years.

It wasn't until it hit a certain quality improvement that it led to discontinuous adoption by the car.

At a certain point of steady improvement you cross a replacement threshold for most people.

A gradual process can still lead to discontinuous adoption.

Exoskeleton is both support and containment.

LLMs benefit from both.

LLMs aren't great at constraints, but old-school code is great at it.

The process of discovering useful applications is a diffusion process.

The raw model performance is clearly seeing a plateau.

But the interesting applications of LLMs are just at the very beginning.

What you thought would take a month will take an hour.

The LLMs will say "this will take weeks" and then it's done in 9 minutes.

We're in a totally new world.

People who know the tech world deeply and aren't using Claude Code won't realize how much has changed.

This was even when there was a direct advantage to collaborating.

These are trivially, obviously true in real world situations, so we never noticed that "trade is good" is downstream of these assumptions.

But for LLMs these assumptions don't obviously hold.

This calculus does change if agents start storing individual state.

Markets are extremely expensive.

But in some cases, the ability to get components that automatically get better is worth it.

Now LLMs can do that for you!

I heard a story about a collective superstition around CalFresh.

Counties across the state believed a certain action was illegal… but it was actually a misunderstanding from a 30 year old defunct administrative rule.

But convincing everyone it was wrong took painstaking county-by-county discussions.

Also known, in any system, as a "liability sink."

I like Ethan Mollick's colorful frame: "sin eater."

The entity that eats the sin allows progress to be made.

They agree to own the downside if they're wrong.

Now it's more important than ever before, since LLMs allow brainless execution more cheaply than ever before.

Places where you know all of the people… or could.

Where there's some gatekeeping happening, some curation of people.

A friend predicted that in 2026 we'll see the first "Human only" spaces.

How can you create cozy, resonant spaces.

These spaces resist the Optimization Ratchet by being hidden from it.

But intrinsic personalization is empowering.

The question is: what is the goal of the entity doing the personalization?

Is it to optimize your response towards their goal?

Or is it to make it better aligned with your goal?

Extrinsic personalization is optimization.

When optimization is applied to you, it's manipulation.

There are no network effects, but there is utility for the very first user.

The Ouija Board effect comes from social residue accreting.

All of the little choices of users in the system leave extremely potent residue.

Each bit of residue is a mote of cosmic dust.

Not that useful on its own.

But enough of them together it can ignite into a star.

In a scaled system, the data the residue is attached to is less important than the residue itself.

They see the wrong things in their history and get confused.

A self-poisoning doom spiral.

The more they get off track, the more that's in the context to confuse them.

The word "not" is easy to miss, so every bit of "Don't do X" just makes them see X again.

If you sprinkle little iron filaments around, they snap into beautiful patterns.

Those patterns make the hidden patterns of language and human thought, omnipresent but hidden, visible.

Not far-field, only things directly touching it.

A camera you don't know if it's looking.

A touch sensor is only getting information if something is touching it.

Social networks were prosocial only in the first ply.

But in the other plies they weren't.

Whatever the cheapest thing you can get away with.

Inductively knowable UX is where at each layer you can pull back and understand what it does.

There's no magic, you can peel all the way back to the core of the system.

The UX complexity reveals itself to you incrementally as you dig deeper.

Design is not a one way process in a system.

Not just the people who create it.

Mentions Resonant Computing and Common Tools!

It's only at scale that resonance is not the default.

They a/b test various UX interventions, to help spread best practices.

Added to my reading list!

They run experiments to see how "elastic" a given user's demand is for staples.

If you don't notice that the prices are higher, they keep them higher for you.

The definition of hollowness created by optimization.

Hollow things aren't actually fun.

This applies to sports, for example.

Basketball is considering moving out the three point line because that's where everyone just shoots from.

The game has gotten so optimized that it's not fun anymore.

They do their work to support their cause.

The fact that it happens to be a part of the ecosystem is just a happenstance.

The ecosystem benefits from the focused effort of those users.

Developer implies a level of technical expertise and motivation that's not required anymore.

Creator just implies someone who invests enough time to build something.

The bar of motivation to become a creator is an order of magnitude lower than motivation bar for developer.

Code has never been as easy to create as it is now.

SGML had been around for awhile.

Systems worked very hard to not have dead links.

The web said, "you know what, a dead link isn't that big of a deal."

That allowed for a loose, good enough, self-healing system.

If you're building software, you're stuck within your island.

Have to create your whole society inside.

If you're using someone else's software, it can't access other things of yours.

Seems to me that we're past peak big app.

They casually mention a "leadgen" tool called RB2B.

It looks at the anonymous visitors to your site and attempts to figure out their email address.

It does this by combining cookies and identifying actions on other sites in the network.

Gross!

Train an agent to use statistics in whatever way is natural to it.

The affordances that humans and LLMs need might be disjoint.

Style, no substance.

Hollow.

Most of those swings will just be delusions.

But some of them, in retrospect, will turn out to have been innovations.

An innovation requires doing something that goes against the status quo.

But some delusions, if they turn out to be useful and actually be viable, are revealed to be innovations.

Hallucinations, delusions, are where the capacity to imagine something different comes from.

Without hallucination, nothing outside of the status quo would ever happen.

Everything would just pull towards the heat death of uniformity.

They have more volume per unit surface area than smaller organizations.

Many more interactions are inside the bubble than outside of it.

It's where the organization meets the external world that ground truth is found.

A 50,000 person organization with billions of dollars of revenue a year can be insulated from the ground truth for a very long time.

A 5 person seed stage startup can't be insulated from the ground truth for very long at all.

It's rude to point it out.

Another relevant question: how invested are you in the emperor being ground-truthed?

If he's just some other random person, then pointing it out is all downside.

If you feel tied into his fortunes–you do well if he does well–then making sure he's ground-truthed becomes more important.

If you hired 50 brilliant, autonomous people and gave no goal, the organization would just diffuse.

A consistent goal is required for the various components to start to accrete into something larger.

Once accretion happens it tends to self-accelerate.

Goals can also emerge bottom-up, as some random bit of noise turns out to work, and other agents see it work and start orienting toward it.

Something that is autonomous has to be liable for downside.

The famous policy: "A computer cannot be held accountable, therefore a human must make every management decision."

Greatness requires audacity.

Something that pushes beyond the status quo.

Found via this excellent article on castles in the middle ages.

Mutual knowledge is knowledge that everyone individually has.

Common knowledge is knowledge that everyone and individually has… and everyone knows everyone knows.

In the mutual state, you don't know if an action you take will be supported by others.

Before it's common knowledge, spontaneous coordination is hard.

The emperor having no clothes is an example.

Bubbles are another example.

It's hard to experiment and see if it's no longer load bearing.

So it can persist for a long time even if it's no longer relevant.

Chaos monkey testing finds the old vestigial rules.

Everyone is naked and they don't know they're naked.

The outsider knows they are naked, and knows what will happen when the outside world inevitably encroaches.

The outsider wants to prepare them for it but if they look like the serpent the naive residents will have them killed.

But perhaps being an apple farmer can help.

A few nuggets:

Daniel Messier has a critique.

"So much of the rebellion is against the loss of dignity.

Not just paycheck.

it's Musk, Altman, etc. saying 'you and the work you do are photo-copyable'

'We're going to photocopy your work and you'll be poor and we'll be rich and we'll spend our wealth electing people who will rage culture wars against you and when you've finally had enough and rebel, we'll just go to our NZ bunkers and you can live in your rebellion."

Bleak, I can see why people would push back on that!

When you offer the product you aren't saying "this is perfect" but "this might be useful."

Users can self-select into using it.

If it's not quite good enough for them then the user can make it better.

It's easier to grow an open single thing that works into a consortium than to retroactively make a consortium out of competing options.

The game theory is very different.

"Wait, that was recorded?"

Unrecorded 1:1s are magic, because there's only the two of you.

That allows capping the downside if there's some misunderstanding or tattling.

But discovering that it was recorded collapses the magic.

It's like in the movie Onward where if you look down you'll fall into the chasm.

You find disconfirming evidence, but are willing to be convinced.

Skeptical and incurious is a toxic combination.

For example: getting out of a warm shower.

Do a thing that doesn't have a downside now but will soon (like making the water colder).

It's easier to start the clock because it doesn't hurt now.

Horses are skittish.

Donkeys DGAF.

That we can recognize and internalize hard edges squiggles.

That our eyes can saccade so precisely on each word.

But in another way, we read so much, that of course it must have fit into something we were naturally suited to do.

But if it didn't work we wouldn't keep doing it.

We had the latent ability to read before we discovered it through a process of diffusion.

What other latent abilities do we have, just waiting to be discovered?

Taking that first step to thaw it out is important.

It requires making yourself vulnerable.

Without vulnerability, nothing can grow.