Bits and Bobs 8/18/25

1) Cloud apps: couch surfing in someone else's apartment.

2) Cloud VMs: renting an apartment.

3) Trusted Execution Environments: an embassy.

4) ZK Proofs: a volcano lair on a remote island.

Each tier gives you an order of magnitude better protection.

The vast majority of the cloud is in tier 1 or 2 today

Moving most computing to tier 3 would be a massive improvement.

Tier 4 is overkill for most situations.

Your family pictures will definitely be safe in the embassy.

Obviously this is a reasonable feature given the topics that it cuts off.

But the frame of we "do this for model welfare" raises my eyebrow.

This is one of the ways you can tell the major model labs have lots of true believers trying to summon a new god.

Even if you don't intend to!

Power is corrupting.

"Can't be evil" is better than "won't be evil."^[cc]

"Would building on AI be like building for the PC in the late 1980s and early 1990s, where you wrote an application that barely worked, confident that you could, when available, seamlessly drop in a new model and get better performance instantly? Or would each model be so unique that every single model update would require an extensive rewrite of the product incorporating it?

The answer to this question does seem to be closer to the former: AI products can incorporate new models fairly seamlessly. The exception, at least in this particular case, appears to be ChatGPT! The problem isn't in the actual functionality; you work with ChatGPT 5 just like you worked with ChatGPT 4o. Rather, the problem for this set of users is the personality. They aren't bothered by the fact that 4o wasn't nearly as good of a model as 5, or that it didn't have the capability of using a reasoning model; they're bothered that the personality is different."

"More generally, I feel better than ever about my position that (1) people predicting imminent super intelligence that takes over the world and renders all economic activity worthless are being ridiculous and (2) people predicting that AI is a nothing-burger that won't amount to anything meaningful are also being ridiculous. AI continues to be a big deal, and even if all progress on the leading edge stopped today, the product overhang remains massive."

LLMs now make it plausible in any scenario.

Every user can do it intuitively.

It's only great for single-player starting of open-ended tasks.

But it's good enough for anything.

It's fine at small angles.

But impossible at large angles.

It's auto de-cohering.

Even if the LLM is right 90% of the time, if it builds on its outputs dozens of times the likelihood it's right is tiny.

You need humans in the loop, cleaning, curating, steering.

If it's late, it can be super-linearly disruptive.

We haven't had too many computing-driven compounding processes before.

If it's a set of humans doing it, one of them will notice at some point and go "wait, what?"

But when an LLM is "yes, anding" its own output ^[cd]with no curation from someone with external judgment, it can get into an auto-catalysing rat's nest of confusion.

If it's hidden (e.g. behind a looping construct) you won't notice if it's wrong, an error in that can compound.

LLM UIs have gotten much more convergent, "here is the answer, where should we go from here" vs "here are 5 options, pick one"

Vibe surfing is a useful interaction where the LLM and the human are in a nice interaction, a coactive loop.

But they don't really exist in that many systems right now, especially as the model quality has gotten better.

But the quality will never be 100%, so for long-lived tasks it will always be useful!

It can do this without you needing to systematize it.

That allows it to be fuzzy.

Systematization is about making something black and white, quantitative, unambiguous.

A lot of nuance and nebulosity and resonance is removed in that process.

It's also an extremely expensive process to distill those rules!

LLMs can do qualitative nuance at quantitative scale.^[ce]

There's no internal ontology, no way for you to tell it, "this is my grandmother."

ChatGPT attempts to create an ontology, but indirectly via chats.

LLMs could plausibly help, but it will need to be a coactive process.

Every action you take to curate / correct / confirm could give an exponential benefit down the line.

If users can feel that, then it would be a killer product.

Sometimes the right UX activates a previously latent possibility.

When you put a seed in a garden you expect it to bloom.

Don't think of it as automation, which is mechanistic.

Think of it as gardening suggestions.

As you garden, they get better and better.

If you're gardening suggestions, constantly curating, tending to what the garden is producing, giving it feedback, it has a naturally alive feeling.

1) The ability to easily create a pointer to another file by typing '[[' and then having a nice autocomplete (that also makes it easy to intentionally create a stub).

2) Being able to navigate quickly to a page by typing a key command and then typing a few letters of the name in an autocomplete..

3) Knowing that I can visualize backlinks pointing to any page.

4) Knowing that if I rename a page, all of the backlinks will update.

5) A single layer of folders (for example, for daily pages, or for a 'people' folder)^[cg][ch]

That's it!

"Sloppy AI defenses take cybersecurity back to the 1990s, researchers say"

"GPT-4o still outperforms GPT-5 on hardened [security] benchmarks across the board."

"GitHub Copilot RCE Vulnerability via Prompt Injection Leads to Full System Compromise"

Rolling Stone: ChatGPT Lured Him Down a Philosophical Rabbit Hole. Then He Had to Find a Way Out

Ars Technica: "ChatGPT users hate GPT-5's "overworked secretary" energy, miss their GPT-4o buddy"

Good Work / Dan Toomey: "Is ChatGPT Therapy a horrible idea?"

To do things for you requires turing-completeness.

That would unlock a significantly larger market.

Otherwise it will be too annoying for a primary use case.

1) The LLM does everything.

2) The LLM behavior is sublimated into a mechanistic harness that can be run inside of other systems.

3) The LLM creates mechanistic behavior that can be executed unless something needs to be changed.

Does only assembly count?

What about C?

What about Javascript?

What about vibe prompting?

LLMs have the potential to be the printing press for turing completeness.

There was more to read, which made reading more valuable.

Which then led to more demand for books, which compounded.

However those languages had way more characters so it wasn't plausible to scale.

It took the printing press plus the latin alphabet to explode its potential.

Vibe-coding is like China inventing the printing press, but code being too hard to write safely.

How can you make turing-complete code as natural to talk as any other language?

Mass scale emergent programming.

Computer literacy at a massive scale, we can't even predict how large the impact on society could be.

A power of infinite patience!

These can help assist you with things that you don't even bother to do today.

All upside!

He needed elective surgery and needed to shop around to many different providers.

He had a body scan that clearly showed the situation and need for surgery, but he had to describe the needs and history in various forms for each provider each time.

Sending the raw scan out to all of the providers to have them bid on it would be terrifying… but also very convenient.

How can you take away the terrifying part and just leave the convenient part?

What if the providers could come to his scan, bid on it, and not be able to communicate back across the network about the scan?

Being able to innovate new tools is what allows adaptation.

It's a snapshot of when it was trained.

The model can't shift but the memory pace layer can.

They have curation.

The things that are replicated are not a random sample but a curated sample.

The subset of images that people found useful or interesting.

It's just "marriage."

Of course marriage can be between two men or two women.

It's only people who were alive before who would even think to distinguish, or realize there even is something to distinguish.

(Of course, it's always possible we backslide as society…)

When in reality the demand for the product is so great it's hard to have messed it up.

The skill of the PM only matters that much in middling ideas, not in fundamentally great ones.

The action of doing something for yourself that can then be summarized for the entire community.

People don't do it primarily for the community; they do it for themselves and the fact it helps the community emergently is a bonus.

These kinds of systems are significantly easier to activate and get compounding loops in.

The incentive is direct and immediate.

Systems that are efficient and humane.

Builders who can code and contemplate second-order effects.

The age of AI demands synthesis, not reduction.

Clippy: "It looks like you're trying to solve intractable human problems with technology"

It sums up the tech industry's approach to the nuance of human / social problems.

One ply thinking that misjudges the nuance and complexity by an order of magnitude or more.

"Let's Make Technology Virtuous

We, the makers of the machines, are more than the sum of our parts.

Let's domesticate tech to aid human flourishing."

Hear, hear!

Something that is aligned at every level.

You like it intuitively and the closer you consider it, the more you like it.

But resonance technically just means aligned at multiple dimensions.

Sometimes that emergent result is bad.

For example, soldiers marching across a suspension bridge causing it to collapse.

Resonance tells you two things match, not that they're good.

Tech addiction is a kind of limbic resonance expanded to a grotesque society scale.

"unregretted user-minutes" is actually a good metric, if it didn't have unfortunate associations…

Does it resonate with your limbic system or your values?

Because everything emerges out of the physics.

The ceremony of it underlines how significant it is.

"I have to go to a specific place and use a pen?"

Takes you out of the ordinary flow to make you think "... are you really sure?"

Makes it harder for someone to say, "did you not even read the contract or realize how important it was?

In the swiss cheese model, the chance that an exploit makes it through all the layers goes down super-linearly.

The more holey the cheese the more layers you need in the swiss cheese model.

It's possible, and even common, to have all of the individual actions be uniformly good, but the emergent result be bad.

It's a 2x2:

A system where everyone in it does the right thing but the emergent result is a bad thing is a bad system.

Integration hard is a straightforward long slog.

Algorithm hard is complex puzzle with one solution.

Claude is great at integration-hard code; ChatGPT feels better at algorithm-hard code.

They allow it to be morphed to fit a different use case.

Wrinkled products are harder to develop, market, test, maintain, coordinate within a team about.

So over time hyperscale products tend to get smoother and smoother.

To get scale you need to reduce its fit to any one case.

One size fits none.

If you have a successful product with people who care about it, often the things the employees work on naturally improve it and make it a more successful product.

This strategy works better for consumer products than B2B, because most of the employees working on it are also users, so they have a calibrated intuition of what features to add to make it better for themselves (and likely others).

The scavenger mindset is not "create new things," it's "take existing things that I can take for granted and combine them into novel combinations."^[cj]

The thing in your set are:

The larger the inventory, the more likely there's a chocolate-and-peanut-butter game changing combination to discover.

If they aren't you get something default divergent.

That authentic belief is why Wikipedia works.

When Wikipedia forks then only the zealots go to the fork, and that bias makes it more and more extreme in that direction.

That's one of the reasons everyone stays on the one that everyone agrees is at least minimally reasonable.

Japan's tradition of going out and getting drunk with your team: nominication.

Sauna culture.

Smoke breaks (back when that was a thing).

That used to be true for photography, too.

It used to be that to be an artist you had to be a drafter and a curator.

Now you can just be a curator.

Moravec's paradox is under-appreciated.

Maybe humans are orders of magnitude more complex than we think we are.

When you get vertigo you realize how computationally intensive just standing is.

Meaning comes from cost.

The same is true of art forms.

At one point Kevin Kelly wanted to know if any technology had disappeared.

NPR did a challenge, but even the weirdest ones he was able to find someone who made it.

What technology gives us is choices.

You can still do old things if you want, but you also get Midjourney.

If you have the same question but go to a different source you'll get a different answer.

The monoculture of LLMs leads to everyone having the same answers to the same questions.^[ck]

I heard of someone who thought someone else had plagiarized their essay.

"Alex is still hung up on X but thinks he's over it."

On one dimension: what you know about yourself vs not.

On another dimension: what others know about you vs not.

Even if you had a socially perceptive LLM helping you out, it might do the wrong thing if it doesn't know details of your physical situation in the moment.

The right recommendation requires knowing the right context.

No one believes Peter Rabbit is real.

But it helps transmit cultural knowledge: "if you don't follow the norms you might get injured or killed."

It's hard to learn norms when you don't see them violated well.

Stories are great for that.

9 dots connected out of 10, so it's easier to apply to yourself without it feeling forced.

You connect some of the dots yourself in a way that feels authentic to you.

It's about your own response to this shared artifact in the world.

But what if there was a movie made specifically for you?

Would it be as easy to accept any insights from it, or feel like they were being deliberately incepted onto you?

Just some random fever dream concocted by the Netflix algorithm.

But unexpectedly it's one of the most fresh and original mainstream movies I've seen, the anti-slop.

The fact you pull the wheel on a slot machine makes it more intriguing.

Kids choose to play with the toys that have random / unpredictable behavior

Static on an old TV is totally novel but also meaningless.

'Bandit' here is based on 'one-armed bandit' referring to a slot machine.

The slot machine is called a 'bandit' because it steals your money!

Maybe it shouldn't be a thing we optimize for and feel good about!.

If you need multiple people to work together for it to happen, it's easier to coordinate if they can understand how it could work: a story.

How will stories evolve now that we have this self-replicating technology?

We're never getting rid of React!

Apparently alphabets stopped morphing when dictionaries were published.

The protocol humans use to connect to others… and ourselves.

Stories are a very potent molecule of meaning.

It survives all kinds of retelling.

Not just noise, a narrative that resonates.

Everything emerges entirely out of local state.

But we don't enjoy watching computers play chess.

There was a flood of soft core porn.

Apparently "Let them eat cake" is actually a meme from a softcore lesbian porn about Marie Antoinette.

The French Revolution was catalyzed partially based on memes and slop!

The pamphlets didn't have to be correct to cause people to do big things.

There were pamphlets like Common Sense, but most of the pamphlets were low-quality slop.

If you perturb it a small bit, it stays in the concave region and returns to the balance point.

If you perturb it a lot, it gets into the convex region and starts accelerating away from the balance point.

Almost all systems have both: a bowl on a pedestal.

The lip of the bowl is the critical region, the line between the two modalities.

The thing that makes it work is that every team member believes in the power and legitimacy of the collective.

They are willing to subordinate their own desires to what is good for the group.

By doing that without being coerced, their authentic goals and desires are able to be transferred to the group, allowing it to emerge into something way beyond what it was before.

If a given organization fractures from one high-trust group to sub-societies that compete, the magic of that collective is lost.

Does the thing you're doing align with others?

The group coheres, naturally.

Things become easy.

It's orders of magnitude harder to earn trust than to lose it.

It takes many good faith actions to slowly build trust.

It takes a single bad faith action to erase all of it in one go.

Diversity here meaning "how closely related are the authors"

If they are closely related, they're more likely to be cited in 10 years.

If they are not that related, they're more likely to be cited in 20 years.

It's just important, but never does new stuff.

The plumbing is the foundation.

Important but never fun.

Sometimes urgent, when it's spraying water everywhere.

The plumbing never feels like it unlocks anything great, it's just a baseline thing that must be done.

So we keep forgetting they exist!

The curse of blind spots is not that we have them, it's that we can't see them.

It takes active effort to remember they exist, and no matter how hard you try, you will forget about them and they will bite you and others around you.

And yet those statements are never as true for a given leader as they think they are.

Even if the leader does have an open door, it's less open than they think it is.

Gossip works great in a small high trust environment.

Gossip at mass-media scale is a totally different thing.

Hyper-orality.

TV is one to many.

Now we just have ephemerality of oral tradition but at global scale.

Uh oh!

But the self vs other is not real!

We have the natural necessary sense of self.

It selects for the system to stay as its been, to ossify, not become adaptive.

Immigration brings new ideas.

The human evolutionary environment is the non-stationary environment.

Humans by being nomadic had to learn to deal with different situations.

Children are the automatic immigrants.

The researchers taught the kids a beach bowling game.

Then they asked some of the kids how to change the game so they won more.

For other kids they asked to make the game more fun.

Fun comes to some degree from leaning into the challenge.

Wanting to win is to some degree in tension with having fun.

You can't not do it.^[cl][cm]

If it's flexible in ways you don't need, it's not an upside, it's worse.

The illusion of it being an extension of you only works if you feel you can control it with high precision.

Every time you can't take it for granted, it breaks the illusion and separates you from it.

It's calibrated to what the powerful entity in a system 'can get away with,'

It's not free floating, it's contextual.

As we interact with chatbots more often, we also treat humans in a more transactional way.

Convergent phase: there are bad ideas.

If you apply a business model from one distribution paradigm to another you're going to miss the point.^[cn]

It took a decade or more of the web to discover the social media business model.

You don't have to mask anything or pretend, you simply do.

Pretending is expensive!

It requires you to go against your natural, authentic desires in that moment.

Pluralistic values are easier to align on than a singular set of values for everyone.

They can never be fully aligned.

That fundamental lack of alignment is what creates all of the coordination challenges inherent to politics.

Politics is a phenomena that emerges out of that asymmetry.

As soon as you get more than two people, the emergent social imaginary enters the picture.

The importance of the social imaginary grows at a compounding rate.

Even at relatively small team sizes, the importance of the social imaginary often dominates the reality.

That's easier if there's a schelling point, a thing that everyone takes for granted at the center.

Having one leader is "efficient".

If the universities had acted like one collective instead of a collection of universities they could have pushed back against authoritarian requests.

"United we stand, divided we fall."

Nearly everyone is likely to have at least some desire for the product that's sold, making it a natural schelling point to meet up, a focal point of a community.

When two land masses combine the animals from the smaller land mass tend to lose out.

This is because they're less able to handle change.

They had an easier competition to win that never pushed them as hard as the animals on the bigger land mass.

In an evolutionary environment, you're only pushed as far as competition takes you.

Less competition pushes you less hard.

It both supports and constrains you.

Which makes it hard to get anywhere when you do need to move.

Radagasts stat from the assumption that it's positive sum.

Importance comes from inside.