Bits and Bobs 6/30/25

An eager servant who feeds you emotional junk food, knows your deepest secrets, and subtly optimizes for a multinational corporation's engagement metrics.^[gh]

"Parasocial" captured something important about modern media relationships.

But AI creates something new: interactive relationships that feel personal but are fundamentally one way and sycophantic.

We need a new word.

Sycosocial.

As a conversation goes on, the context gets more confusing than helpful.

You need to start a new thread.

Hints at the importance of curation of context.^[gi]

They absorb the intention of the context.

Even an earnest and helpful LLM, when given malicious context, can itself become malicious.

But if it sees any untrusted context at all the LLM can become malicious.

This is why prompt injection is so dangerous.

That is, how the model would choose to blackmail its creators in some cases.

Simon Willison's summary is worth reading.

It's a raw utility.

It has to be used as an input to something else to blossom into its potential.

The current manifestation is the simplest thing you can imagine: a chatbot.

But that manifestation is obviously not the end point.

The fact they're commoditized means they don't have power.

If you lock into a single model, then you can't see the influence of bias.

If the switching cost of models is high then the problem of AI bias becomes significantly more important.

That's the world we're skating towards with vertical integration of chatbots if chatbots are the one UI to rule them all.

The product is a model in the middle like a christmas tree decorated with various doo dads.

Just a single model to rule them all for all use cases forever?

Clearly the model will just be a component of a larger system.

The model is the engine, not the car.
Instead of having a service that is entirely a chatbot, what about a service where one feature is a chatbot^[gk][gl].

Excerpts from the Stratechery interview of Bret Taylor:

"When portals dominated and directories dominated, the web tended to be small because you needed to be a presence on these directories. It almost was like if you weren't in the menu, no one could order it, there's no off-menu websites."

"As search … became the dominant form factor … it also produced the long tail. All of a sudden, because you went from a directory form factor to a search form factor, it created a market for people to essentially provide content for those keywords. … As a consequence of that, it made the web bigger.

"As you think about the evolution of AI and the form factor of ChatGPT … we might still be in Yahoo directory."

From Strachery this week:

"Everyone used the Internet on Windows PCs, but it was the Internet that created the conditions for the paradigm that would surpass the PC, which was mobile"

Curation is about distillation, taste.

You can steer quite a bit when you curate.

In a world of cacophonous background noise, the raw signal is never seen, it's what is curating your view.

The curator has a massive amount of leverage to steer you.^[gm][gn][go]

Who curates it?

Who controls it?

Who stores it?

Who can see it?

Who can write to it?

You should have full and ultimate control.

It requires open-ended solutions.

Only savvy enthusiasts will be able to safely use it.

That's arguably what happened with Linux and Crypto.

A system that makes normal usage safe and hard to mess up for low-savviness users allows it to blossom into a much larger ecosystem.

An open-ended catalog of what other users have built and used.

The agent could create tools on demand and draw on the wisdom of the crowd to find good ones.

It used to be that writing software was so expensive that you needed careful planning up front to make sure you didn't waste any effort on it.

With LLMs making the actual coding significantly cheaper, the bottleneck shifts to verifying it actually does what it's supposed to.

Someone who's thought about this domain a lot asserts that it works.

Especially valuable for things that abstract messy network problems (like payments) or fractally complex compliance problems.

Many Saas tools, even if distributed as just a spreadsheet, would still have value.

Related to the idea of "A good brand is a promise kept".

The SaaS companies are offering a promise for competence in a specific domain.

A lot of platforms being built for the age of AI imagine that most of the weight of systems will be LLMs, with just a little bit of code.^[gq]

What if it's the other way around, and it's mostly code, with a little bit of LLMs as magical duct tape?

The former has the prompt injection problem, fundamentally.

The latter has the potential for a non-prompt-injectable system.

With vibecoding maybe the latter is more important?

The tech industry has a baked in assumption of "of course people with a CS degree are at the top of the totem pole and always will be".

But the tech industry as a whole would miss it if LLMs changed it because we'd all have the same bias and blindspot.

Domains that don't work with CS are less deterministic.

Their taste, metacognition, entrepreneurial spirit, and subject matter expertise matter a ton.

Maybe philosophy is the best degree for people who "code in English"?^[gs]

Aligned with my intention in an era of AI.

Software that builds itself, aligned with my intentions.

It wasn't just optimizing for making it so people could choose to help others with their own actions, it was defaulting it so that users created that indirect value as a bonus even if they didn't intend it.

People are OK with indirect benefit for others from their actions, but don't typically take proactive actions^{[gt][gu][gv][gw]} to make that happen.

Defaults matter.

This is a summary of the key insight in Dan Bricklin's The Cornucopia of the Commons: How to get volunteer labor.

For example, the feature on Maps that shows how busy a given business is right now.

It's created by aggregating and distilling a massive swarm of anonymous location pings into a high-quality, useful signal.

The way these signals are calculated often uses differential privacy thresholds internally to make sure the data isn't identifying, even early in the pipeline.

Two problems with the status quo:

1) You can't actually verify that they are using privacy preserving techniques in the pipeline.

2) The aggregate signal is owned by the aggregator, not the users.

If you have policies that everyone can structurally trust to be followed you can have new coordination mechanisms and very different equilibriums.

Your browser profile includes session tokens from various domains that allow anyone driving it to take actions as you.

Signing into a service in a browser is like leaving a horcrux of yourself.

A big deal!

The AI doesn't need to fear getting fired, so it can speak truthfully.

The LLM can be used as a responsibility launderer.

You know it's important to have a devil's advocate, but any individual who plays that role has an asymmetric downside.

So have the thing that's not alive play that role.

To be human is to be a kaleidoscopic, emergent force: an end in and of yourself.

An API can only be about a means.

When humans are below the API they become invisible.

The raw data has to be processed to uncover those insights.

It's possible that even someone with the data doesn't invest the effort to uncover them.

But if someone's already done the synthesis work, it just needs to be transmitted.

For example, document each directory with a README, have aggressive type checking / linters.

But most humans give up or aren't patient with overly constrained setups.

LLMs are infinitely patient, so you can have them be very constrained to what you want.

When you have those best practices, the actual instructions to the agent can be quite short.

In many systems there's multiple layers.

At the bottom is the bedrock semantics (e.g. assembly).

Then you have the first layer that expert users might actually use in normal usage.

Then you have layers that less savvy users might use (the sugar).

It's hard to start with the sugar first–you don't know what patterns will be common and useful.

It's easier to start with the lower levels and watch what savvy users do: the idioms that emerge.

Then you want to add layers such that 80% of the lower level things could have been done at that layer without dropping down to the lower level.

They're very different stances.

One is you're in the loop with it, one is you're out of the loop.

When it's just making changes automatically you aren't in it, you're looking from the balcony.

Pair programming vs "handing off to the intern" gives you significantly more ownership of the code.

Did you have the LLM think for you or did it allow you to think 10x better?

Are you engaged or disengaged? Are you enrolled in the process?

Is it a tool to help you think better or a tool to help you think less?

This is what is called cognitive debt.

Making subtle decisions about the information, keeping a sense of what it's doing, steering it out of corners before it gets stuck.

Why?

Innovation looks like noise.

Trust is giving the benefit of the doubt to someone to lean into their suggestion that looks like noise to you.

That noise could turn out to be innovation if everyone leans into it.

Imagine outcomes randomly jiggling through the possibility space.

An outcome by itself tends to evaporate.

Outcomes that pile on and accumulate can grow into stable structure.

As the coherent pile gets larger and has more surface area, there's more likelihood a random outcome touches it and coheres, which gives compounding momentum.

This is a metaphor for finding consensus and other emergent phenomena.

The more degrees of freedom in the possibility space, the less likely the outcomes randomly cohere.

Constraints reduce the degrees of freedom, making it more likely ideas cohere and build.

The corporation is the original runaway AI.

Why are we worrying about runaway AGI?

It's already here!

… I don't know if that makes me feel better or worse.

A manifesto for Radagasts.

Especially relevant in the age of AI.

But Computer Science can't tell us how to understand LLMs.

Scale requires quantization, and that requires distillation.

In capturing some dimensions, it must miss others.

To scale you need to erode the soul.

If you later care about some dimensions that were denatured from the data, you can't recover it.

Which dimensions matter is a kaleidoscopic, fractal, ever-changing thing.

That's what's soulless.

Finite, not infinite.

Short term.

The slow-thinking part of your brain is wise.

The fast-thinking is looking at what's precisely in front of you.

The slow-thinking part is absorbing from breadth and experience.

It keeps on ratcheting up.

This will get worse in an era of AI.

The importance of stillness and reflection will become increasingly important.

It helps you avoid addictive apps on your phone.

To unblock the apps, you have to physically scan the Blok device.

That allows you to add significant friction by physically putting the Blok device far away, or giving it to someone else.

To force you to pause.

To think more intentionally, rather than having one more bite.

More chances for your intention to break through.

Lashing yourself to the mast.

The way they learn from their training data is more than just reproducing.

Essence is a new concept.

It doesn't exist in the legal canon yet other than things like trade secrets.

Copyright is about mechanical reproduction, but what LLMs do is different.

If you don't look carefully you won't see anywhere it shows up directly because in any given instance the noise dominates.

But if the bias is consistent it doesn't matter how noisy it is.

The consistency of the bias is more important than its strength.

The noise falls away at scale and all that's left is the bias: the signal.

The noise is camouflage that hides any given instance of input.

"Where does evolution come from?"

Only if you blur your eyes can you see it.

Sarumans look at each instance and don't see anything interesting so they conclude there's nothing interesting at all.

When it's not in a pocket, it is dominated by the cacophony of the average.

What makes it a pocket is the network topology but also the friction of information transmission.

The friction of information transmission has declined significantly.

This has always been true, but it didn't used to be as obvious.

The obviousness that everything is a remix is tied to:

1) how connected you are to the same things as everyone else,

2) also the cost to generate an artifact.

Now we're all in one big melting pot and it's free to create slop.

It's now much more obvious that everything is a remix.

What kind of thinking is worth doing ourselves?

Is insight compute bound or attention bound?

Not cheaper cognition but richer intention.

The cheaper thinking is, the more important slow thinking becomes.

When thinking becomes cheap, will it 10x the industry's addiction to heroics?

I'd rather that instead of thinking becoming faster it becomes deeper.

So when imagining the future it's invisible and you undercount it.

Hofstadter's law: even when you take into account Hofstadter's law, it will take you longer to achieve the outcome than you planned.

Not just "surprisal" but "valuable surprise".

Humans think it's interesting only if it can help them do things in their situated context.

What computers think is interesting and what humans think is interesting is disjoint because they have different views on what is valuable.^[gy]

How big a collection feels is proportional to how familiar it feels.

So as the trip goes on and you get more experience with everyone the group feels smaller.

It's a structural impossibility.

You could get asymptotically close to aligned in some circumstances, but never perfectly aligned.

There's always a situation where what's bad for one individual is good for the collective.

The Ones Who Walk Away From Omelas captures this dynamic.

From the inside an organization can be seen for what it is, a collection of sub-entities that aren't perfectly aligned.

Outside the boundary you only see the external API, not the internal complexity.

Status is structurally scarce, since it is entirely relative.

You choose to affiliate with collectives that will raise your status.

This is one of the drivers of boundary gradients.

Each individual atom's alignment isn't that big of a deal.

But at the macroscale huge fields result.

They end up in recursive isolation that leads to a kind of mania about how everyone else is stupid.^[gz]

The most cynical among us, those without shame, have seized on this fact.

If you do corruption in public there's no discontinuity when the story breaks.

It's the discontinuity, the shared, coordinated "they did what??" that leads to a force decisive enough to drive a coherent effort to punish it.

But the cynical among us^[ha] have noticed that in the modern era where divisive personalities have loyal mook armies, if they just commit the corruption brazenly and in public there will never be a discontinuous moment of alignment for the people who might bring them to justice for it.

"The best lack all conviction, while the worst are full of passionate intensity"

"Laws and regulations are part of the free market system.

As rules approach zero, competition approaches war."^[hb]

Could you make the same argument about AI today?

A Stoic reframe to find meaning in the everyday: imagine "this is the last time I'm doing this."

When you're older you're forced to confront this often.

But it's true in many other scenarios.

Leaning into it helps you be here now.