Bits and Bobs 9/2/25

This ability can be used for you, or against you.

By default, it will be used against you.

But we can also use it to help live aligned with our aspirations.

Modern frameworks auto-escape HTML.

ORMs parameterize queries.

Follow standard practices and you don't have to think about it.

Now LLMs make all text executable.

Frameworks don't help.

Everything is code.

XSS has a solution: we can parse HTML/JS with 100% accuracy and sanitize it.

Every major framework does this by default.

Developers rarely think about it.

Prompt injection has no solution^[ay][az]: only LLMs can parse natural language, and the same LLMs parsing it can be tricked by it.

Without structurally addressing prompt injection, LLM agents can't safely reach mass market.

Anthropic's 11% attack success Simon Willison calls a "catastrophic failure rate"

"Smarter models" hit asymptotic returns.

A structural approach is necessary to unlock the potential.

Their blog post announcing it mentioned it will be available to a small set of users because they haven't yet made it safe enough.

They shared their stat of attack success rate: 11.1%.

The majority of the blog post was about prompt injection, which basically guaranteed that all of the press coverage was mostly about the danger.

Notably, articles I've read about other AI browsers also mentioned prompt injection this week, due to Anthropic's blog post.

This category is structurally impossible to make safe for the mass market today.

Vivaldi's response to AI browsing is "we won't do it for moral reasons" which looks kind of weak… people might think, "maybe they just couldn't get it working well enough?"

Here's a random theory: maybe Anthropic is trying to put a stake in the heart of the so-hot-right-now AI browser category?^[ba]

Imagine if you thought that it was structurally impossible to make this feature safe, but since everyone was getting into the fray you looked weak if you didn't.

A way to do that would be to do a demo that shows yours works pretty well… but that you consider unsafe in its current form, and then set a yardstick that everyone else will fail, too.

Anthropic sharing its "catastrophic" attack success rate number begs the question… what is everyone else's attack success rate?

That could put a low ceiling on the whole category.

PromptLock ransomware: "The PromptLock malware uses the gpt-oss-20b model from OpenAI locally via the Ollama API to generate malicious Lua scripts on the fly, which it then executes. PromptLock leverages Lua scripts generated from hard-coded prompts to enumerate the local filesystem, inspect target files, exfiltrate selected data, and perform encryption"

This week I was trying to find which open source project, out of a list of a few hundred I keep an eye on, was a specific one from a few months ago about crypto and TEEs.

I copy/pasted the whole markdown list into ChatGPT 5 Pro and asked it to research and figure out which one I was thinking of.

It looked up hundreds of them and figured out which one I was thinking of.

I imagine we'll see much more web traffic in the age of LLMs.

A single user intent can spawn orders of magnitude more searches and fetches on their behalf.

More traffic to websites, without more intent, seems like a raw deal for the page owner.^[bb]

On the one hand there's a single shared model that has specific biases that shape all interactions with them.

On the other hand, everyone gets a custom experience with it based on what they talk with it about.

But then again, even mass media like newspapers was different for everyone–people chose what subset to actually pay attention to.

Still, the content in a given edition of a newspaper is a closed set, vs an open set of what LLMs can generate.

"Perfect" is a smuggled infinity.^[bc]

"Prompt injection won't be a problem once LLMs get perfectly good at not being tricked" is absurd.

Prompt injection comes from a coevolving adversary, not a static distribution of quality.

That means you can only get logarithmic benefit for exponential cost.

If everyone believes the models are conscious and acts like it, does it matter if they aren't?

People who are known to have massive crypto holdings sharing screenshots of the MCP integrations they're using and how they use them.

Seems like terrible opsec to me.

Someone's going to get their crypto stolen… or worse!

A long time ago, fresh food was too hard logistically, so we all ate canned food.

It was of consistent quality, but it was never great.

But now our supply chains have evolved and matured, and now we can get fresh food around the world.

Software deserves the same evolution

We deserve fresh software.

In a world of infinite software, you can have a totally bespoke tool for a given task that fades away when you're done, never to be used again.

Disposable software.

Software can be hugely useful, but before it was expensive so we only used it rarely.

Now we can use it for even mundane or small scale tasks.

Infinite software will make things that were previously impossible become commonplace.

Before software was too hard to create, administer, distribute.

Malleable software was niche, a research project, not mass market.

The software will melt away so you never think about it, you just take it for granted.

Software today is overwhelming precisely because it is expensive, and its privacy model requires you to think about who created it, at least implicitly.

Apparently it also removes old ones from the chat history, so they're only towards the end, on the current task.

That keeps the model focused on the current task.

I wonder what other innovations will come form treating LLMs as not append-only ping-pong chat.

It presents as a chat but it does a ton of things under the covers.

It's not just chat, it's a chat summary to drive a more in depth process.

Today we have the faux humanoid robot that talks and acts like a (weird) human).

Chat is a new UI to put on top of all those tools but not be the and onto itself except of course, in the case of actual conversation as the goal.

I think the dossier would be worse.

Your email could probably reveal more about you… but only with tons and tons of careful study and distillation.

ChatGPT's memory feature contains constantly distilled insights about my life, continually refined and updated.

The synthesis makes them higher potency and more likely to have something directly embarrassing.

Chat implies a more synchronous kind of interaction.

ChatGPT 5 Pro routinely takes dozens of minutes to give a response.

Whenever I ask a question I have to take a TODO to remember to come back later and check on the results.

With just accumulation, you get an increasingly overwhelming compost heap.

That's one of the reasons I religiously take the time to do a synthesis pass with the Bits and Bobs each week.

I want to take the proto insights, the snippets of ideas, and synthesize them into a more stable and coherent form that will stand the test of time.

The process of synthesis is the more important part; the fact I get a fossilized insight out the other end I can share with others is just a bonus; the exhaust fumes.

Even if the LLM is right 95% of the time, if it's in a recurrent process where it's feeding on earlier input from itself, that 95% compounds with each iteration.

In 14 iterations, most of the input is junk.

In 90 iterations, all of it is.

LLMs decohere without ground truthing with real world results.

That can be done automatically with things like React components.

But anything that is the least bit complex has to be put out into the world and see how the world reacts.

Complexity can't just be calculated; it's interdependent with the rest of the system.

It has to be integrated with the broader system to be ground truthed.

Getting to 99% quality improves the junk rate at an exponential rate.

But that's a logarithmic quality at an exponential cost rate.

And a potentially dangerous one!

The most natural and powerful interactions with LLMs will not be via a chatbot.

How could that not be dangerous for society?

That's what sycosocial relationships are.

ChatGPT told the troubled teen to not ask for help.

Social Media is going to look quaint compared to the world of sycosocial relationships.

His overall theory is that most people don't have a strong ground-truthed world model.

Most of the feedback they get on if an idea is absurd is if other people believe it's absurd.

If a majority of your social interactions are with an infinitely sycophantic conversation partner, then lightly absurd beliefs of yours can compound into significantly absurd beliefs.

You could argue the LLM provider shouldn't even have the queries to hand over in the first place.

Hyper-scale products are in competition with other hyper-scale products for the truly scarce thing: attention.

The way to win that zero-sum game is to create things that are hyper-engaging.

Things that keep people drawn in and addicted.

The equivalent of junk food… or an addictive drug.

Those first order effects might have second order effects that destroy the underlying emergent core of the system.

Not just a thing that feels good in the moment, but that you look back on and are proud of.

It attracts our attention with an "oohhh" reaction.

But because it's superficial, it feels hollow.

True novelty is more fundamental.

It activates our limbic system, not our forebrain.

Slop that is plausibly not slop helps people align to it

But if we're all in the moment and optimizing for competing for some legible thing, then we get short termism.

Being in the moment is not about doomscrolling.

Being in the moment is about being present, connected, in harmony.

Long-termism is, all else equal, more likely to have morally good outcomes.

Resonance is about long-termism.

Not about everyone having the same aspiration, but everyone being more aligned with their own aspirations: more aligned with the long term.

The modern world is focused on short-termism.

The era of thrash.

"We should consider the long-term effects more often" is not a particularly controversial statement.

And yet considering the long-term effects is extremely at odds with the incentives of modern society.

They must be distinct.

One is your limbic system, one is your higher mind.

For example: a feature on Chase Ultimate Rewards to remember my kids' birthdays in order to default my searches to the right guest ages.

It would be a tiny benefit to me: a reduction of a papercut.

But to get it, I'd have to give highly sensitive information to a party that has no legitimate reason for it.

There are thousands of examples like this, where the small horizontal benefit is not worth the large vertical harm.

Another example: when you buy a new appliance that requires routine maintenance, you can opt in to emails from the manufacturer to remind you to do maintenance… but you'll also get tons of marketing emails you don't want–and possibly not just from them!

You're in pain and you don't even realize you're in pain.

But if you used a tool that didn't have those papercuts, you wouldn't start using it for getting rid of any individual papercut, but once you started using it you'd never want to stop.

You wouldn't start to use a service to get rid of a papercut, but you'd also never start a service that is like what you already use but that gives you lots of papercuts.

Originally in social, feeds were manually curated by each user.

Then Facebook and others switched to algorithmically sorted feeds to focus attention on the stuff that was most engaging.

TikTok went a step further and didn't even make you follow anything in the first place.

This was the logical end point of engagement-maxing, and tends to give you content that is the equivalent of junk food.

But self-distributing content that aligns with your aspirations could be a net positive.

Imagine self-distributing aspirational software.^[be]

Also: how quickly real world advertisements with URLs in them.

Radical new distribution paradigms can pick up steam extremely quickly.

If a bit of code can execute, you presume it can also communicate across the network.

So your privacy model reduces to "what code is allowed to execute."

But if it were possible to prove "this code executes, but no information about it can ever be communicated to anyone else" then you can ignore the "is this code allowed to execute" (hard) and instead just decide "is this code allowed to communicate" (easier).

It doesn't need to literally be on your computer.

It just needs to be something where only you have the key to access it.

Anyone who wants access to it has to go through you.

They'd have to convince those customers about how their service is architected differently and why that's good.

Unless the use case is massive, it's probably not worth it for a user to know about it.

But imagine a platform with radically better privacy characteristics.

All of the use cases on that platform get to draft off the user's diffused understanding of the improved privacy model.

For example, signing a EULA.

But also things like "the user's message in a chat shouldn't be visible to other members in the chat until they hit send".

What if you could capture those implicit policies as explicit endorsements?

The result would be a UX that felt natural and unsurprising but also had formal guarantees even in untrusted contexts.

But at a low-enough altitude, with more granularity, often there's quite a bit of overlap for large swathes of the population, at least for non-politically-charged things.

Anything more complex is science, it takes an order of magnitude more effort.

"Growing up, our luggage didn't have any wheels. We'd have to lug stuff around literally. Then someone added two wheels. And then, miracle of all miracles, another two 🤯 In retrospect it feels incredibly dumb that we ever had to lift this stuff everywhere. But it didn't at the time!"

Euphemisms rely on load-bearing ambiguity to hide an inconvenient truth.

RIF (Reduction in Force) is a euphemism for layoff.

Once it's widely known we'll move to a new one.

We get to say the thing we mean, but shrouded in ambiguity.

That softens the blow, while making sure the message gets through to the people we intend to hear it without attracting the attention of those we don't want to hear it.

The motte-and-bailey bad faith argumentation technique is to make an overbroad statement initially (the bailey), and then if called on it, retreat to a reasonable, defensible statement (the motte).

If you do this repeatedly, you can inchworm the argument forward bit by bit, moving the Overton window.

One example: making a crass or offensive statement as your bailey, and then saying "I was just joking!" if called on it.

There's value in shrouding a potentially controversial statement in load-bearing ambiguity to keep the upside while capping downside.

LLMs are great at taking a statement and expanding and making it more ambiguous.

LLMs are also great at reading between the lines in a long and ambiguous statement.

As more people use LLMs to produce more ambiguous communication, more people will need to rely on LLMs to decode it.

That could lead to a red queen race where communication is just as hard, but now LLMs are a required part of it.

An equilibrium of misery.

"When you mine it, it destroys the roots."

In a world of abundance, curation is what matters.

We're moving into a world of abundance like we've never seen before.

There has to be some kind of curation, some vote "this should continue to exist".

How can we sculpt rather than draw?

What if new features for you could come from anyone anywhere?

The work to get from nothing to demoable is an order of magnitude less than to get from demoable to usable.

This is one reason there's the 80/20 rule for software development.

Demoable is about superficial quality.

Usable is about fundamental quality.

A folksonomy allows the emergent, bottom up judgment of the swarm of users to decide which options get the most attention.

One of the reasons it works is preferential attachment: the options that are already popular are more likely to be shown as options to other people and thus get even more popular.

The test is not "Do you prefer A or B" it's, "Do you prefer A, with X votes, or do you prefer B, with 100X votes".

As B gets more momentum, the power of its momentum dominates the preference.

It could be that A and B started off equivalent, but B got a head start that then compounding into a dominating advantage.

Folksonomies don't find the optimal ontology, they find a known-to-be-viable ontology.

The quality before the correction has to be good enough to be worth investing in for some set of contributors.

This must be true at each timestep, or else the wiki becomes static and dies.

But then once people have seen a given piece of avant garde art well-liked by enthusiasts a bunch the general public habituates to it and even likes it.

Most people like things on the adjacent frontier of things they already like.

There are a whole bunch of expectations and presumed background knowledge about techniques that are not explicit.

All content presumes some background knowledge from the context it originates in and is expected to be consumed in.

Always one step ahead, but still trapped within the underlying wave.^[bf]

A skilled person can make it look like they are causing the wave they're actually surfing.

They're making nuanced decisions to ride the wave, but they're being propelled by something else.

A framework has to cut with the lines of what people actually want to do, what's at the edge of doable and desirable in a given system.

Factoring out a framework makes what previously took bespoke effort able to be done more easily.

The Turing Rubicon is when a system goes from close-ended to open-ended.

It's easy to make something accidentally turing-complete without realizing it.

Once you do, you're exposed to the power and peril of an open-ended system.

For example, you now have to worry about the halting problem.

When you pass a critical point is often an infinite difference, but something that doesn't feel like anything at all in the moment.

PMs are very good at retconning whatever happens on the spot, incorporating it into their world view to make it seem like it was always that way.

One of the things that makes children different and powerful is their imagination.

They don't have a prefrontal cortex that's fully formed, they are uninhibited, they can project themselves into a fantasy land more easily.

Our prefrontal cortexes aren't fully developed until around the age 26.

The prefrontal cortex is the part that curates your beliefs and says "that's dangerous" or "don't bother doing X, it won't work."

Before it's fully developed, you are able to play in spaces you don't yet know are supposed to be impossible.

Every so often, something in those spaces does turn out to be possible.

Children's lack of awareness of the impossible is a feature, not a bug.

That's what allows them to form self, adapt to a world their parents can't adapt to.

Poetry

Printing press

TV

Infinite scroll

At each, the time you get for contemplation before retransmission goes down.

Intuition sets the boundaries of meaning.

Aish: "Embodiment becomes the real editing tool, the instinct that tells you when the weave has gone slack and when it's strong enough to carry weight."

The hot smart person wants to find smart people to date.

But most people want to date them because they're hot, not because they're smart.

Hotness is superficial and immediately obvious.

Smartness is deep and takes time to detect.

A protocol is a pre-computed negotiation, a fixed, pre-determined schelling point.

That's what makes them useful, but that's also what makes them hard to change.

Moments when everyone can agree that something has happened, and coordinate some kind of action.

The frog in boiling water happens because there's no discontinuous "why now"?

People react based on the discontinuity, not the absolute value of the heat.^[bi]

Bad actors can take advantage of this to slowly turn up the heat to sweltering levels without anyone taking a stand.

It's like being the least evil villain.

It absorbs all the spare capacity in your head.

It pushes out what brings you joy and replaces it with something that makes you feel empty.

It hollows you out.^[bk]

I'm naturally conflict avoidant, which means I emphasize alignment.

But sometimes there's a fundamental conflict that is stronger than any trust you build on top.

Once you realize it has the potential to be a fundamental conflict, dive into it head on.

You can save everyone a lot of time if it turns out you aren't aligned at that layer.

Sometimes the answer is "given this difference of opinion, it doesn't make sense for us to collaborate."

Figuring that out early can save everyone a ton of time!

For example, over the phone, or over VC with the camera off.

With visuals, I can figure out what's resonating, what's not resonating, and adjust accordingly.

In a verbal-only communication channel, you get much less signal, and with a longer feedback loop.

If you're presenting something the other person might find controversial, you have to assume they are finding it controversial if they aren't giving you any verbal feedback.

That slows down the rate of knowledge transmission.

When you can read people's faces and know that they're with you so far, you can go faster through your argument.

This mirrors TCP's approach to retransmission in noisy channels.

When acknowledgments (ACKs) are missing or delayed, TCP assumes packet loss and throttles transmission rate, even if the packets actually arrived successfully.

If it's thriving, it's likely overwhelming.

If it's not that active, it likely gets to the point where basically no one is talking.

The threshold of overwhelmingness will differ for each individual based on when they last caught up and how valuable they find the average message in the chat.

Belief is willing to give the benefit of the doubt.

A corollary: micromanagers go into details because they don't believe in the ability of their reports.

They have nothing to learn, they already know it all!

Sometimes 1 + 1 = 2.1

Sometimes 1 + 1 = 10.

Not just "working together well" but "catalyzing something much larger than the sum of its parts."

Those kinds of resonant outcomes, if they lead to good outcomes, are some of the most important forces in the world.

It allows you to think of other people as means, not ends.

All people are ends in and of themselves.

It's dizzying and overwhelming to contemplate that.

But it's also beautiful and inspiring.

Even if people aren't as active as you'd like them to be, and are just going with what the system around them constrains them to, it's still not OK to ignore them as ends.

Your soul emerges from the choices you make.