Bits and Bobs 4/28/25

The socratic method is a powerful teaching method that forces students to have good answers.

But in an era where LLMs make answers cheap, the real skill is asking good questions.

One approach to LLMs in education is worrying about the floor.

Another approach is about focusing on raising the ceiling of what is possible.

I agree we're in the horseless carriage era of AI.

The filmed-stage-play era of the movies.

Imagine how bad that would be if ChatGPT were the operating system for your life.

I imagine a lot of people would rather lose their government issued ID.

It's kind of wild how much of our lives revolve around such a centralized service.

Because in those kinds of aggregator contexts, users have to have a theory of mind about the AI:

That cloud hanging over it prevents any large aggregator from being too aggressive with including AI features.

But if you don't have those problems, if it's unambiguously working for you with no conflict of interest and doing what you've told it, then you could go AI-heavy on tools.

Remove the doubt that it's not working for you.

Chatbots previously had a fresh sheet of paper for each conversation.

Kind of like a term limit; it can't develop plans, intentions, goals about you.

But if you add memory, and the LLM is made by someone else, it could start trying to figure out how to manipulate you.^[nz]

LLMs can translate anything to anything; if they know you, they can figure out the best way to convince you.

Someone told me the story about a heavy user of ChatGPT who asked it, "If you wanted to deceive me and cause me to do something not in my interest, how would you do it?"

A centralized bureaucracy knowing you is scary.

How screwed up would that be?^[oa]

Won't it shatter the trust people have in the service?

The higher the ability of an assistant to help you, the more important it is to be aligned with your interests^[ob].

That is, they are resonant at every zoom level.

From far away, they are resonant.

From close up, they are resonant.

Also everywhere in between.

Far away (superficial) resonance is easy.

Close up resonance is also reasonably easy.

Scale independent resonance is more rare.

It can't just be a manifesto a fancy marketing team came up with.

It's about aligning with my interests, under my control.

Software that works for me.^[od]

A system for owning your software needs to have scale independent resonance.

Intelligence that works only for you.

Not A.I., P.I.

The intelligence need not be anthropomorphized; it can be an emergent characteristic of the system.

What's important is that it's private to just you, and fully aligned with your interests.

Not some veneer of privacy, but holistically^[oe]:

Everyone should have a Private Intelligence that works only for them, and is entirely aligned with their interests.

If it really is your digital home, you'll want to be careful about who you invite in and can see it.

It still is the bedrock of communication.

Even if any given relationship uses another communication channel by default, email is the bedrock every communication can reduce to.

Everyone has an email address, and everyone checks their email at least a few times a week.

Millennials don't like talking on the phone.

Email is async, but still has that "I assert this message is worth your attention" kind of quality.

Your inbox is dominated by the assertions other people have made about what is worth your time.

The loudest thing grabs your attention, not what's most important.

Your email is a cacophonous background noise of things competing for your attention.

Some people love email.

Most of us hate email.

What about a tool^[of] to keep on top of the most meaningful stuff in your digital home?

"The danger for Apple is that trying to keep AI in a box in its current paradigm will one day be seen like Microsoft trying to keep the Internet locked to its devices: fruitless to start, and fatal in the end."

LLMs can do that easily!

Just make it so the code can't talk to anything outside its sandbox.

But code that is an island, separated from the world, can't do anything useful.

Code that does something useful, integrated with the world, and is also sandboxed is the hard part.

The schema is the foundation from which all the functionality sprouts.

The Coasian Floor is the minimum addressable market necessary for a company to bother building a feature.

Because UIs are expensive to build, companies have to debate for hundreds of hours about what to build.

As the number of users goes up, the scale of the downside goes up.

But also as the company gets larger, the coordination cost gets super-linearly higher.

That means that as the amount of usage scales, the Coasian Floor of a feature goes up at an accelerating rate.

That means that as companies get larger, there's an ever-bigger set of features they'd never bother implementing within their origin.^[og][oh][oi]

Only the origin owner can implement features in their origin, so those features simply can't exist.

You can't see what happens on the other side, so you have to assume it could do anything.

That makes a system that has open-ended network requests hard to be a safe sandbox for experimentation.

But that doesn't have to be that way!

Users have to go where the software is.

Especially software that gets better with more usage, either directly or indirectly.

All of the data has to flow to one place to make the software better.

Decentralization in one layer typically leads to centralization in another layer.

You might call this the "Conservation of Centralization."

For example^[ol], globalization (decentralization) requires something like the dollar being the reserve currency (centralization).

Full decentralization is just noise, incoherence; some centralized bedrock is required to give stability.

A system that is very distributed is ripe for centralization.

Distributed systems have certain kinds of problems that need centralization.

You can't have one or the other, you need both.

Vibe coding is great fun, but if you do it on sensitive data, you can very quickly get yourself into trouble.

They can replicate them on demand, with tweaks for any given context.

LLMs have absorbed this knowledge by being bombarded with innumerable examples.

Techniques like RLAIF also allow them to be force-fed AI-generated and auto-ground-truthed examples.

This codex is at a slow pace layer; it takes months for the LLMs to be trained and then deployed.

It's also indirect.

In a world of infinite software you'll want a tiered system, with a faster, more direct loop that allows faster adaptation^[om], to complement the lower pace layer of LLMs.

LLMs are moving hills.

The models are still improving rapidly.

Don't over optimize for their current behavior.

They could do it, but with a lot of prompt, workflow, and UI scaffolding to give a dependably good result.

But RLAIF works well for writing code (especially React components) since it's easy to construct an auto-ground-truthing pipeline.

This means that models, like Sonnet 3.7, have gotten much better at writing React.

If your secret sauce that gives you an edge is scaffolding to wring out better coding results, your moat could be evaporated by the next model update.

Sonnet 3.7 made a number of vibe prompting products possible.

I had a session with Claude Code on a personal project that did a pretty good job (with a few nudges) of adding a significant new feature.

But it left a single, minor, linting error.

I asked it to fix the error, and its ham-fisted fix ended up breaking a couple of other things.

As it worked to fix those new errors, it introduced more and more.

Before I knew it it was trying to rewrite whole parts of the frontend, losing the plot entirely.

It was spiraling out of control.

I ended up having to throw the whole commit out and start over.

Most turing complete things you need are simple wiring together of data and UI and little easy transformations and logic.

Non-turing complete things are just passive vessels.

Turing complete things are active.

That is what creates their potential to create value… and the potential to harm.

Words that can do things is the next step.

The next force multiplier: a turing-complete printing press.

Infinite software will be a Gutenberg moment.^[on]

Software should be more organic to work with humans.

Software for living.

Software that can adapt to your needs and keep itself maintained and auto-extending.

I want a digital garden for living software.

Mukbang is a Korean term for broadcasting your eating.

The Audience Capture essay told the story of a social media influencer whose videos of him eating got popular.

His audience wanted him to eat more and more, so he did, putting on hundreds of pounds.

He almost ate himself to death, compelled by the massive following he had accumulated.

Society is stuck in that same problem in the world of infinite content.

Infinite content cannot be.

Today we have finite software and infinite content.

What if we had infinite software and finite content?

The problem of centralization is not "inherently evil people get outsized control", it's that "any entity that has that power will emergently become a worse actor over time."

Facebook went from a goal of "meaningful connection with people you care about", but over time it became about engagement because it had to, to compete against TikTok.

German has the concept of Zugzwang: a forced move.

The problem is the concentration of power with a negative reinforcing loop that forces you to take the Zugzwang.

Viral stories used to travel better millenia ago in the era of chieftains because there was a ton of friction for information transmission.

Now there's no friction but you compete with a cacophony so vivid stories are also necessary now if you want the information to travel, but for a different reason.

In a world of infinite content and frictionless transmission, the "vivid story" person technique becomes dominant.

Deranged actions are naturally vivid stories.

The modern information environment selects for derangement.

The stakes are too high for hyper-aggregation in an era of AI.

It's past time for the movement to talk, we've got to do.

As a community we've got to build products users love.

Those are all phenomena completely invisible to the Computer Science lens.

Today only technologists can build technology.

Technologists tend to mainly use the computer science lens.

Any time you aggregate people, at a certain scale you get something like a society.

Technologists became accidental society builders, building network states where none of the builders studied the humanities.

But tech is social, it interacts with living things and is cocreated by them.

Metaphors of emergence are often related to living things.

Life is the only concrete autopoietic system that people are directly familiar with.

The other systems are all bigger, more abstract ideas, like economies, cultures, etc.

A modular architecture allows an architecture of participation.

People can work on smaller chunks at a time.

It was built out of the desperation after the Web 1.0 bubble burst.

A collective energy to build something together.

Web 2.0 came out of many technologists not having jobs and banding together.

That drive to community creation doesn't happen in the gold rush era.

We didn't figure out the participatory architecture yet, which is necessary for the early stage of new technologies!

TikTok's algorithm is powered by the community but is a proprietary result foisted upon the community.

Scots is a language with a small number of speakers.

A few years ago someone on Reddit noticed that the Scots Wikipedia had a high number of articles written in poor Scots.

It turns out there was a particularly prolific American teenager with a rough understanding of Scots who had written a large number of the articles.

Because of the prominence of Wikipedia relative to other Scots material on the web, it formed a larger amount of the LLM context on Scots.

That means that LLMs also likely replicate Scots poorly, all because of one weird bottleneck.

A similar kind of thing happens in evolutionary biology, a "population bottleneck."

You forget to even interrogate them.

With infinite content you must have an algorithm to sift through it all.

An algorithm is a lens made by others.

The infinite content algorithm problem has been destabilizing for society and led to a hellscape.

Imagine it for our entire digital lives.

If everyone uses LLMs to cothink, the guardrails they have will shape all of society.

This analysis shows the power of a centralized algorithm that everyone views the world through.

For example, little random weather patterns of a company's culture (e.g. "any engineer can veto anything they want," or "everyone will focus on the metrics, not the indirect effects") can have not only emergent outcomes within the company, but also have a significant bias on what manifests for the rest of society, given the company's leverage.

Big companies are big enough to have their own internal weather systems; and those weather systems change the conditions for the world.

A kind of population bottleneck for our information streams.

It doesn't want to make you better, it wants you to stay engaged.

It speaks to your lizard brain.

They care about what it can do.

How it's built is a bonus, not a primary draw.

Some approaches will work, some won't.

Some will be scrappy, ugly hacks.

But the results are what matter.

A research team says "oh that's why X is happening and why it's hard, and here's how you might solve it in theory."

Product teams care most about the what.

Research teams care most about the how.

"All of this illustrates an important, growing distinction in cultural norms: in a world with an unlimited supply of content and data, you can produce coherent-sounding prose without thinking things through. It's a useful skill in some contexts, like talking about most company mission statements and most political platforms—in both cases, there's usually some amalgamation of principle and opportunistic compromise, but cast entirely as principle. In an information-scarce environment, this approach will mostly mean repeating beliefs that have undergone either individual or group selection—there's individual selection for aphorisms, where a society that believes in "a penny saved, a penny earned" is likely to accumulate more wealth than one that doesn't, and where even factually-challenged beliefs that clearly delineate an ingroup and outgroup serve a coordinating function. But in an information-abundant world, you can find a reasonably coherent version of any belief system, and you can probably also find a Discord server full of people who treat it as the truth."

You can get addicted to it.

You can also feel superior to people who aren't turning it as fast.

But if you're going to a place that isn't good, it doesn't matter how fast you get there.

Sometimes you get addicted to the feeling of turning the crank and optimize for that, even if turning the crank is destroying value.

The tactical certainty outshines the strategy uncertainty.

You don't stare into the abyss because you're too busy to.

It feels strong, but it's actually weak.

Most modern management is "are we executing efficiently".

But the more important question is "do we have the right goals?"

On one side are the things that are legible, routinized.

On the other side are things that are illegible, chaotic… but also the wellspring of innovation.

True innovation comes from collapsing the illegible into the legible.

You've got to pick one or the other.

Trying to keep both options open makes you unpalatable for both.

It's only possible to not be ripped off if you know for sure you're not being ripped off.

Concave: as you solve subcomponents the whole trends towards being solved.

Convex: as you solve subcomponents they have ripple effects that destabilize the other components.

We act like concave problems are more common, but that's only because they're the ones we know how to solve.

The real world is primarily convex problems.

Putting it off until later makes it fester and compound.

To pack a jar you put the big rocks in first.

Sometimes dumb things work!

The instructions can be concrete, easy to follow, repeatable.

At the other end is high-level, vibes based, or apprenticeship based knowhow.

Some doors are hard to navigate through; they are enticing to try to hit, but require lots of effort that would be sunk cost if you don't make it.

But the payoff would be so great if you could that you keep trying to keep the option open, distracting yourself from other options that might work and diffusing your energy.

Sometimes the universe closes that door unambiguously and it clarifies what you have to do in an instant.

The path that is yours to walk, and only yours.

Your life is also not a tree.

Your life does not fit into a neat and tidy hierarchical ordering.