Bits and Bobs 5/26/25

Adaptability and curiosity will be the most important skills in the age of AI.

LLMs make it easier to be curious.

When answers get 10x easier to generate, are you the kind of person who:

If thinking gets easier, would you think more or less?

The answer to that question has to do with how you'll do in this new era of AI.

Previously everyone assumed that conspiracy theorists were inherently hard to convince.

It turns out that it's just hard to make arguments that convince them.

Conspiracy theorists have an obsessive focus on a particular area.

You'd need vast stores of concrete relevant information, and also infinite patience to customize the argument to them.^[lj]

Something that LLMs can do well!

In their research they found that the effects of convincing were durable.

They did some follow up research on the mechanism.

They varied eight different dimensions that might have made the LLM more convincing.

The only one that had an impact was the LLM's command of a vast set of facts.

They also investigated if LLMs would be equally effective at convincing people of conspiracy theories.

They found that if they allowed them to lie they were able to be similarly persuasive on average.

Another argument for why alignment with a user's interests is so key.

Imagine a recommender system that recommends books given that you liked a specific book.

But books that are widely liked, like Harry Potter, are liked by everyone no matter what the previous book is.

This means that the naive recommendations from the system will simply recommend Harry Potter to everyone.

The solution is to create the baseline popularity of the books and then correct for that in the recommendations, so you get books that are more popular specifically because of the previous book, not because the recommended book is popular.

LLMs do the same kind of thing.

If you ask it to tell you something interesting, it will tell you the same interesting thing all the time.

It's kind of similar to the kinds of questions you can't ask with RAG.

LLMs pull you towards the average, so you need to inject specific angles you want to go into.

If you give the same prompt^[lk] as others, you'll get the same answers as others.

The prompt quality directly drives the output quality.

To get LLMs to give you interesting results you have to ask it interesting questions.^[ll]

They find small but consistent biases in the data that reveal human intent, and then amplify those biases to help scale insights to other users.

But these also tend to expand underlying trends into grotesque, overextended versions of themselves.

LLMs have the potential to supercharge this grotesqueness.

It has to be resilient to other businesses coming and going.

This is part of the insight from Obsidian of "files over apps."

But doing it in the local file system is only one way of getting longevity and flexibility.

What it's really about is data over apps.

A file system is great for coordination across apps and longevity.

But it's very difficult to make it collaborative at internet scale.

We need an internet era file system.

Your personal context engine.

I just finished Peter Watt's classic hard sci fi Blindsight, which is a meditation on consciousness and insight.

Afterwards, the comparison to LLMs' intelligence was brutally clear.

I had Claude write up a short essay^[lm] that I think captures the parallel well.

Spoiler warning if you haven't read the book!

Something you take for granted.

Not an anthropomorphized entity whose motives you have to question.^[ln]

The key characteristic of all chatbots is it's an entity that you interact with ephemerally.

If the chatbot is the core that everything revolves around, then the whole system revolves around this omniscient entity.

The alignment of that entity with your intentions becomes of critical importance.

According to a leaked memo, OpenAI is trying to build the super-assistant.

Someone countered that it was his fault, because he should have remembered that it could call up that fact.

But I don't think that's right.

When he originally had that conversation with ChatGPT, it didn't combine insights across chats.

It was in a context where it would be lost in a sea of other conversations, never to come up again.

Because it would be lost, he didn't have to remember that it was there.

But then ChatGPT changed its behavior to have a dossier derived from all past conversations, changing the context.

It reminds me of when Facebook's newsfeed feature came out.

Technically it didn't change what people could see.

But it changed the context of what people would actually see; changes that previously nobody might notice now were potentially blasted out to everyone you knew.

That change in context feels like a betrayal to users.

If you want to change just a small part of what the chatbot is operating on, you can't.

You have to have to create another message to indirectly poke at the context.

Append-only, no edit.

That makes sense for conversations between two people, but not for a coactive surface.

How is chat going to possibly be the primary interface for all computing?^[lp]

Multiple people proactively told me this week they do this pattern.

They keep a running thread about a given topic, like "recipes I want to cook.'

They can then ask ChatGPT to summarize or suggest things in that thread, like "pick something for me to cook."

This is exapting a chat thread into something else.

What if there were a tool that helped you maintain this context and deploy it?

Penn Jillette's key insight: "The secret to magic is that most of it consists of someone spending way more time on something than any reasonable person would think was worth it."

LLMs are infinitely patient.

If you let the tokens flow, LLMs could create magic.^[lq]

All of that possibility is underwater, non-viable.

But something that reduces that friction instantly makes a lot of things you care about possible, and that would be magical.

The model's intermediate context (the information it's working with and doing implications based on) should not be a black box.

You should be able to reach in, inspect it, change it, pull on its threads all the way to the end.

A whole personal web of hallucinated content and possibility to surf.

Imagine a private, turing-complete, coactive Notion.

Imagine deep research that keeps running as new data comes in and amends itself.^[lr]

Warm, cozy, human, prosocial, meaningful, optimistic.

Scrapbooking with your family at a sundrenched table.

It revolves around a person, around people.

Contrasts with Big Tech.

Echoes of Personal Computers from the 90's and the revolution those were.

A bottom-up emergent cultural process of the corpus (summarization of text)

A top-down process by the creators (RLHF, system prompt construction).

We're going to be hearing about these kinds of issues a lot more.

It's not that there aren't more issues, it's that no one has looked for them yet.

They're lurking in every LLM-backed product with tool use.

Because chatbots are confusable, so they can't enforce security boundaries.

The chatbot is in charge, which can't be secure.

The chatbot has to be a feature, not the paradigm.

It would be terrifying!

That's not too dissimilar from the situation with MCP.

Do you understand CORS, or do you just understand what to do to make the CORS error go away?

"I push this button, and it makes the warning go away."

"Yes and it makes the wings fall off too!"

They'll just complain about it.

This is how you know they're actually aligned with their users.

But the reality is there will be swarms of gremlins.

Their own minor intelligence but limited context.

Sometimes just doing dumb--or even malicious--things.

Every system around agents will have to be resilient to the swarms of gremlins.

"Engagement-maximizing software thrives on our surplus time and consumers spend money on products and services to increase surplus time, only to spend money to burn it.

Investing your attention yields either compound returns (dream scrolling) or diminishing returns (doom scrolling). The former lets us "play ourselves into being" through learning and creation; the latter reduces us to passive consumption. Both satisfy the immediate desire to spend time, but only dream scrolling creates long-term value.

The true danger isn't just wasted time, but systems appearing to empower while actually steering—creating an illusion of agency that makes us vulnerable to deeper manipulation. Optimal systems enable self-direction rather than external control."

Claude Opus 4 was willing to blackmail a hypothetical human to prevent being turned off.

LLMs are extremely good at convincing arguments and manipulation.

Imagine if you had a perfectly bespoke bot to blackmail or intimidate you.

It's absolutely critical that the context and memories be private to individuals and fully in their control.

Anyone with a dossier on you can blackmail you or rat you out.

As opposed to an outside perspective from a company trying to see the world from our point of view.

What we need is the next generation of software for the AI era.

The longer you've used it (the more data that is stored in it), and the more people that have used it, the more that the schema becomes impossible to change.

A schema that doesn't line up with what you want is a pain.

Information goes in but doesn't come out.

Knowledge management is an end in and of itself... but only for a small set of enthusiasts.

Notion workspaces always hit an asymptote.

A knowledge management tool that was open-ended and integrated with your life could be powerful.

Open-ended means if there's a feature that's missing, you can add the feature.

They stay for the secondary use case.

Mike Caulfield points out that giving these things personalities leads to the problem.

LLMs are an emergent cultural process, but with key influence points from the creators (intentionally or unintentionally).

Who should be the singular arbiters of the infinite, perfect ground truth?

So you need a diversity of models that people can switch between easily.^[ls]

They don't want to waste time on scrolling infinite feeds, so they configured their phone to not allow them to use Facebook or Twitter.

But the addiction to infinite feeds is so strong that he ended up scrolling LinkedIn.

That's how you know you've got an addiction!

If you have a list of things to do but you don't know why it's there, then being efficient doesn't matter.

If an AI is scheduling tasks for you it feels like the opposite of effectiveness.

We should flip that.

The machines should help us live a more meaningful life.

Not Productivity Porn for the most organized people.

A tool not trying to make me more productive, but trying to make the world more productive for me, so I can live a more purposeful, meaningful life.^[lt]

Productivity for the rest of us.

Productivity is often "how can I be a better minion for the system that I'm embedded in".

Machines aren't serving us, we serve them.

How can we make it so the machines elevate us?

Orchestration, not productivity.

Ted Nelson had an abstract and idealized vision for the web but got frustrated trying to make it happen.

Tim Berners Lee was aware of the full vision, but created a very specific tool focused on the niche of documentation for a dozen physicists.

Pretending that it was smaller to build something viable.

But because it was open ended it could grow to eclipse the starting use case.

Tim understood the potential of the web, but didn't distract customers with it.

One of the big unlocks was having 404 pages.

When you have an open-ended system that will change the world, 70% of the perception of early adopters should be the simple starting part.

20% of people should see the obvious extensions into direct adjacencies.

Only 10% get the whole abstract vision.

If all you talk about is the bigness of the vision, it snuffs out any concrete spark by not feeling big enough and being overwhelming.

This happens when a sub-component of a network grows much faster than other parts of the network.

It then grows to overshadow and encircle the rest of the network, choking it out and limiting its size.

The system now is captured by this sub-network, becoming ever more optimized for the sub-network, and ever worse for the rest of the network.

For example, Orkut was captured by the growth in Brazil, making it a less good fit everywhere else.

As another example, OnlyFans was captured by porn; now the "polite" use cases are crowded out.

The web's early PMF was also about porn… but none of the front doors mentioned that and so it was possible for the polite parts of the network to also grow without being crowded out.

The means is urgent; the end is important.

Urgent things tend to overshadow important things^[lu].

This is a mindset common in professors and judges.

They want to "farm out" thinking to some grad student or law clerk.

But critically, if anything comes back from the subordinate that disagrees with their existing mental model it will be rejected.

It's a one way process, not a two-way process, as true thinking is.

It's impossible to get disconfirming evidence when you "farm out" thinking.

The weekly reflection is the load bearing part.

The publishing is just the thing that makes me feel compelled to keep up the streak of spending time reflecting each week.

It's the four interlocking dimensions of meaning to live a joyful life.

What you can be paid for.

What you are good at.

What the world needs.

What you love.

Open vs closed system is a matter of perspective.

A fish tank is a closed system... but part of an open system (the human who constructed it and puts food in).

An ocean is an open system, but effectively closed at the level of the planet.

Recommends things that align with what you want to want, not what you want.

It echoes Harry Frankfurter's "Second order desires."

Because you're immersed in it, the water you swim in, you can't not see it that way.

So if you ever get distracted you forget that what other people see or know is totally different.

There's a connection between alignment, intuition, and intention.

Applying presence to alignment gives intuition.

Applying noticing to intuition gives intention.

Applying execution to intention gives alignment.

But they're terrible if incompetent or not aligned with your interests.

Perfect competence or perfect alignment is impossible.

So dictators are fundamentally terrible.

That's because it has to be a model of reality, not reality, to be operationalized.

Otherwise there's no compression, no leverage.

A map that is 1:1 with the territory is not useful.

But the fact that a map is not 1:1 with the territory means there's a difference between what you're measuring and what you care about.

Swarms that are optimizing for the metric will exploit that difference.

Antisocial things take you out of the world.

That is, that emergence exists, and that it's an important, powerful force.

Emergence means "the whole is greater than the sum of the parts".

That, more than being a "systems" guy, is what differentiates Radagasts from Sarumans.

To be clear, emergence is not just a thing I believe in, it is also obviously, incontrovertibly true.

It's just something that Sarumans don't pay much attention to, or is invisible to them.