Bits and Bobs 1/6/26

"You can patch a bug, you can't patch a brain."

About 11 minutes in they reference me!

"This paper from Stanford and Harvard explains why most "agentic AI" systems feel impressive in demos and then completely fall apart in real use.

The core argument is simple and uncomfortable: agents don't fail because they lack intelligence.

They fail because they don't adapt."

"Making ai models is less like training an animal intelligence and more like summoning ghosts."^[dc]

"google is trying to do to openai what facebook ended up doing to snap which is to first decelerate growth substantially (which kills a lot of momentum & morale) & then unleashing integrations at scale rapidly by leveraging distribution advantages."

"Prompt injection, much like scams and social engineering on the web, is unlikely to ever be fully 'solved.'"

Big Tech is owned by someone else.

My Tech is owned by me.

"Let's spend years plugging holes in V8, splitting browser components to separate processes and improving sandboxing and then just plug in LLM with debugging enabled into Chrome. Great idea. Last time we had such a great idea it was lead in gasoline."

Docker Fixes 'Ask Gordon' AI Flaw That Enabled Metadata-Based Attacks.

Turning AI Safeguards Into Weapons with HITL Dialog Forging.

Excellent show.

There's an obvious metaphor for people who use AI and simply do whatever it tells them to do.

Compare to people who use AI to multiply their agency.

Looks similar, but wildly different.

So if it makes an error it will tend to make it again.

Because that error is put into the context.

The most coherent belief to amend silently assumes that the error is right.

The more errors it makes the more deeply ingrained the errors get.

On the one hand, the idea that we've been at the hand-crafted / high friction style of software to date does resonate.

But unlike in the industrial revolution where everything was made more similar, the industrial revolution of software will allow software that is fractally personalized.

The aggregator endstate.

If you have the mindset of "That might not work, let's not try it." then you won't unlock their full power.

AI is really, really unpopular with the American public.

But everyone keeps on acting like most people love it.^[dh][di][dj]

It just does what you expect.

Then as you peel back layers of understanding there's no magic.

but you don't need to know the lower layers, they just make sense if you ever did peel them back.

So if it's collected in a context where only superficial things can be extracted and then it's put in a context where it can be deeply understood, it feels like a betrayal.

That's why Google's data is a blessing and a curse in an era of LLMs.

They're sitting on a trove of data for each user… but if they preprocessed everyone's decades of emails it would feel like a crazy beytral, an invasion.

We've explored every nook and cranny of that simple software landscape.

If your architecture doesn't matter, just build as quickly as possible and get as many users as possible as quickly as possible.

But if your architecture does matter, getting users on the wrong architecture could be deadly.

We can't imagine it being any other way.

So we just stopped imagining.

For the world to become better, you have to first imagine that it could be better.^[ej][ek][el]

A used system however will tend to increase in performance.

Users simply won't tolerate their usage patterns being slow.

When it's just a demo, users have to remember to actively demand performance.

When it's something you're using, every time you use it you want it to be faster.

They're trying to get everyone to create their own software.

That will never happen… software requires thinking at least like a PM.

But it's a necessity if software can't be safely shared by strangers.

To climb the right hill, you'd have to make software that could be safely shared by strangers.

Whereas if it assumes normal compute sweetened with LLMs there's no floor or ceiling.

And also if you assume LLM in the loop the only way to improve is model quality or tools.

Big companies can't bet on two contradictory things.

What if the most powerful use of LLMs that is not chatbots?

For example, performance work should come after required architectural changes.

When you have users is when you're forced to start climbing the hill.

If you have users prematurely, it forces you optimize prematurely, meaning you likely climb the wrong hill.

But a useless slog is a waste of time.

The whole question becomes: is it useful or not?

But they can't do multi-month slogs to non-obvious endpoints.

Other teams might go slower on simple projects, but are capable of achieving more complex projects.

If they achieve something useful, that gives them a moat.

Anyone who wants to catch them would have to go through a similar slog.

Always striving to go above and beyond.

Other cultures are more willing to merely satisfice.

Then make the practical marketable.

The system builder, when they try to build product, will see the problem and try to solve it properly.

The product builder doesn't care about the system, they just want the product to work.

It happens because the combinatorics makes even a small incidence rate quickly saturate the network.

That also means that statements like "Nearly every academic paper is less than 6 citations away from a retracted paper" are not as interesting.

I didn't think it was about tipping–if most tips are in the same narrow band (15-20%), that doesn't explain much difference in speed.

But the existence of tips–even with a narrow band–incentivizes turning as many tables as possible.

So even if the average "salary" comes out the same with the default tipping rate, the one with tipping will incentivize faster table service than the one without tips, which doesn't reward based on how many tables were finished.

"I got mine, screw you."

Not caring about the externalities is selfishness.

You'll cut corners.

When you're fighting for honor, you'll optimize resonantly.

You won't take shortcuts.

Slytherin for example is ambitious.

Ambition unchecked by shame will take shortcuts with significant externalities.

Imagine a choice in front of you.

A question gives you something to do.

Curious people always generate another question.

A great question is

Preferential attachment is multiplicative vs additive.

That's why we don't yet young children make substantive decisions.

No one.

It emerges, bottoms up.

Based on the stories people choose to repeat.

The ones that loom largest.

But that's an illusion because only the top down outcomes can be thought of as "decisions".

We think naturally in narrative so top down decisions compress way easier and thus feel more obvious and tidy.

Also even bottom up things could look like top down.

Over time, over multiple iterations, it continuously morphs.

Otherwise you get a discordant mess.

The alignment of vision is critical.

The best predictor: was it filed in the Romance section or the Drama section?