Bits and Bobs 8/3/26

At first, the reaction is "wow, you are so much smarter than I'd expected."

Then they make a dumb-as-bricks error and you realize "wow you are not wise in the slightest."

"You can have detailed conversations about music to generate listening recommendations and it always sounds like it knows what I mean while discussing subtle aspects of the sound design or composition.

Then it'll recommend a song and I remember 'oh you don't actually know what music is.'"

LLMs can do a convincing job of acting like they understand experiential phenomena, but they only have ever seen the shadows in the cave.

Models, when given noisy additional information, don't do that much better.

But if they know the information is correct and from a participant trying to help them, then even ambiguous new information can have a discontinuous increase in output quality.

They allow you to fill in the details you wouldn't have previously had the patience or time to do.

The frame you bring to the analysis still sets the fundamental shape of the container the LLM will fill in.

The LLM can just fill in the container way faster than you can.

That's not exponentially better; it's logarithmically better.

It's not that they're smarter than a human security researcher; it's that they're able to scale their search much more than humans can.

Humans can do one lane of analysis at a time, but LLMs, even a single instance, can swarm multiple search heads at once.

The parallelizing of the search is best modeled as patience.

They find things that humans could find… if only they had infinite time and patience.

A great HackerNews comment:

They are also arguably better at figuring out where to search.

That's becoming more and more clear as we use agents more than chatbots.

It doesn't know the most important things to us.

It knows the frontier of our understanding: where our curiosity is or lack of understanding is.

These are important to us, but the most important things to us–the things we know extremely well–we never bother asking about, so the chatbot never sees it.

The reason is that each silo doesn't know about the other.

So each bit of multi-user coordination only works per silo.

To have high-fidelity coordination across the m silos that contain your data with n people is just combinatorially too expensive.

So you fall back on the smallest number of silos that you know everyone uses.

This reduces down to WhatsApp or Google Docs.

In enterprise, higher leverage tools can emerge, because there's a schelling point of the tools that the enterprise requires the employees to use.

But for consumers, it's hopeless to coordinate on anything other than the lowest common denominator.

We are social beings, coordinating in high-fidelity ways should be common and powerful.

But we're missing the Silver Swarm of social software, because the physics of trust make it impossible.

These are the missing social use cases.

You might conclude that they must not be valuable because we don't have them today.

But actually we don't have them today because our physics of trust make them too costly.

New physics of trust that make them cheap will unlock tons of value that was always sitting there, waiting to be activated.

The Silver Swarm describes a situation where there is a massive, obvious amount of value.

But as you zoom in on any part of it, any concrete grain is small, inconsequential.

From that, you could conclude that the whole swarm is low-value, but that is incorrect.

This happens when the power is not in any singular use case, but in the emergent value between use cases.

If you go looking for the concrete bits of value, it will be like the raccoon washing the cotton candy.

You might erroneously conclude that there's no value there, when in reality there is overwhelming value.

A fact Alan Kay himself didn't like.

Way more compute power than he imagined, applied to a much smaller physical device than he imagined.

But it's locked down.

It's impossible to build an iPad app on an iPad.

That fundamental restriction breaks the generative loop of software.

Instead of an infinite self-recursive loop, we have an extractive single loop.

We should be angry about that.

The Dynabook vision has stayed as a north star for the industry… but accidental architectural decisions made it impossible to reach.

We locked ourselves in silos, and so now we can't reach the stars.

"Do you trust this entity in perpetuity in an open-ended way with this data" is obviously an impossible-to-answer question.

There structurally is no good answer to that question.

We shouldn't have to answer that superficial question, we should just have to answer the more fundamental question about what we're OK with our data.

That would require changing the physics of trust.

There are low-hanging fruit everywhere, and there are too many for even the swarm of experts to find.

This is the state we're in with applying LLMs to useful work.

Fable is frustrating because it will sometimes dig a hole to China even if I didn't ask it to.

It's presumably vibecoded, but every single detail has significant taste and craft.

This is the kind of game that previously would have been expensive enough to create that it would only make sense with ads or being a paid app.

But vibecoding makes the cost lower so it can be simply a delightful little app with no annoyances.

It can fill any crack you give it with good enough, rough-but-working code.

You can even use it to build whole structures… but the bigger they are, the more likely they are to collapse under their own weight.

Your own assumptions are often invisible to you.

When two parties have different assumptions, they can conflict without either party proactively realizing it.

This can happen often with LLMs and agents.

If a system makes assumptions that the other party might not share, it should proactively state them, so the other party can call them out.

We're so used to "that will never work, it requires someone with infinite patience to keep it up to date."

Well, now we have that!

Everything else is very hard.

The more powerful your knowledge base, the more dangerous it is to share.

Slack bots powered by LLMs that can be called by coworkers are aimed at teams with significant amounts of pre-existing trust.

1) Safe

2) Powerful

3) Easy to use

In our current physics of trust, you have to pick 2.

Social now means "user generated content media" now.

"Social" is the defining characteristic of being a human, and now when you say it in the software domain people assume you're talking about an infinite slopfactory.

The software isn't very good, it's not aligned with our interests, and the amount of the software is a tiny fraction of what it could be.

LLMs is leading to a renaissance in software.

Now is the time to escape that local maxima.

Even back in the Claude 2 days, the interoperability research could see Claude replace memorized facts about numbers with the algorithm for doing arithmetic.

The algorithm wasn't one that we'd have developed ourselves.

There are also presumably many other emergent algorithms in the weights that are novel to us, some of which are more useful than another fuzzy algorithm for arithmetic.

For example, AlphaFold seems to have absorbed some fundamental algorithm or estimate that eludes us.

If only we had some systematic way to abduct them.

To not just lie around, but do things for you?

We have pixie dust in the form of LLMs, but if you sprinkle it recklessly it might be like the Sorcerer's Apprentice.

To get your data to work with you, even with infinite pixie dust, it would have to be structurally aligned with your intentions.

"In the last two milestones, Chrome 149 and 150, we have fixed 1072 security bugs, surpassing the total number of security bugs fixed across the prior 23 milestones combined."

The App Store might seem like low friction relative to installing software to your PC from a CD.

But it's still massive amounts of friction.

Some of that friction is for the user of installing, and then putting relevant data into the app, since it's a silo.

But a lot of that friction is hidden from the user, in the forms of the sisyphean tasks a developer has to do to even create an account to get apps in the App Store.

This friction must be there because in the current physics of trust, software is dangerous, and so there must be friction to avoid harm.

The friction means that only maybe 10% of the useful software that could exist does exist.

We're so used to software having to be trusted, so needing significant distribution friction, that we can't imagine any other way.

But with different physics of trust, there could be an order of magnitude more useful software.

That requires it to be split up into islands to cap the downside.

But splitting into islands also caps the upside!

But not in today's physics of trust.

If you make it so software isn't dangerous, then you can unlock the power of infinite software.

Otherwise it's the spooky baby rhino effect.

You think you're safe, but actually it doesn't do that much anymore.

"Don't worry, it's running inside of a sandbox."

… and then minutes later it's outside the sandbox innocently doing the tasks you asked it to do.

Gulp!

Context Collapse, Part 3 - AI Worming through Word.

SharedRoot; Escaping the Claude Cowork sandbox.

Prompt Injection comes for the justice system in Brazil.

"Solved" is almost certainly the wrong word here.

Prompt injection is a problem that cannot be solved, only mitigated.

Everything the user interacts with is "below the API."

The same generic pixels and interactions shown to every user.

During friction is when you can catch yourself and see if your intentions match your actions.

It's possible to do calculations horizontally across everyone's data to produce distilled signals (e.g. rankings or recommendation) that benefit everyone.

Those distilled signals are more than the sum of its parts; every user benefits from everyone pooling their data into the calculation.

But in today's physics of trust, the only way to accomplish this is for one entity to have all of the data in plaintext.

You then have to trust that one entity to be a good-faith actor with the data and not use it in ways not aligned with your incentives.

That's a bad assumption today, where the aggregators have gargantuan Terms of Service that reserve the right for them to do just about anything they can think of with your data, and also have incentives not aligned with yours.

In addition, that one entity likely won't give the contributors of that data the distilled result, since it's the aggregator's proprietary signal.

People would be happy to contribute their incidental data for free, if no other entity had access to it and they were able to get access to the distilled signals.

If they knew their data couldn't be used in any identifying way, they wouldn't even need to collectively own the distilled signal, just have access to it.

Consumer is a vertical-shaped company.

Infrastructure is a horizontal-shaped company.

The only way those can stably persist in the same company is if they're totally separate subsidiaries.

When push comes to shove, which part of the business would bow to the other?

It's been constructed carefully so that structurally no person at the browser company can see it.

Compare that to a web app, which requires your data to be available on the server by default, and almost certainly is using your data in some way that's not fully aligned with your interests.

Imagine a local-first piece of software reaching out to Google with an anonymous query for [hemorrhoids cream].

Imagine instead that you run a VM on a public cloud thousands of other users are using and it does the same anonymous query.

Your request is drowned out by all of the other traffic exiting the same IP.

He's been contorted to become evil by his greed for the thing he found himself holding.

Power corrupts.

If the new things can grow to a much larger scale than the earlier thing, it can overshadow it.

A new renaissance of software leads to a lot of blossoming.

It's for our own safety that they won't let us have it, they tell us.

Each silo can only see its own view of the world.

That means when you want to use a new experience (e.g. an app) it has no idea what gmail address you've proven you control in the past.

You have to sign in constantly, including doing 2FA, to convince each silo that you are who you say you are.

It's easy to unlock a bit of profit by lighting trust on fire.

If a premium hotel brand skimps on the soap, you shouldn't trust them to not skimp on the things you can't see.

Apple has the most pure and defensible brand promise in tech.

It blows my mind that they're continually incrementally sullying it with ads.

Suh-prise.

It's still shocking to me how quickly Google jumped over the AI-results cliff, just because everyone else was doing it.

The engine of that ecosystem and how value flows through it, previously limping along, has now been completely smashed.

The image most of society has about tech is a techbro on a steamroller that's driving toward them, saying "get out of the way, losers!"

"This technology of ours is so dangerous that only we should be trusted with it."

Someone to handle the grunt work for you.

Although cynically you could claim that it was "Only the chip makers should have a vertical monopoly; everyone above us should be a commodity."

Still, it helped create a schelling point that made it very unlikely that powerful open weight models are banned.

We are stumbling towards the goldilocks path of powerful models that are commodities.

For society's sake, yay!

The default very easily could have been set to the opposite.

That was very bold, and assumed that democratizing technology is fundamentally a good idea.

A similar thing happened around genomics.

Anyone who sees them goes, "that is obviously how things will work in the future."

Their point is not to show what is built, but to illuminate the future and make it more concrete and intuitive.

First mover disadvantage: you stake out a position on a hill... and later everyone realizes the real value is actually on the neighboring hill.

You want to add value, so you go where people are today and solve problems they actually have.

But now you have structurally locked yourself into the logic of the system of today.

All you can do is fill the potholes; you can't create new roads.

If you're trying to change the structure of the domain, you can't get stuck filling potholes because you'll never escape.

Instead, go to where people are today, solve a problem they have, but bring them along to a radically different world they didn't know they wanted or could even get to.

If all of the test runs are terrible or all too excellent, the eval doesn't give you a gradient.

Getting an eval that is calibrated to the dynamic range of the current capabilities of the system under test is hard.

Once everyone scores 100%, the benchmark is fully saturated, and you need a new one.

Auto-improving benchmarks can be self-steering north-stars.

You start with coarse sandpaper and work through progressively finer grits until the surface feels smooth.

The first massive jumps in smoothness take as much work as the later tiny improvements in smoothness.

The same fundamentally logarithmic curve of fidelity that shows up everywhere, including in product development.

Looks great, but based on a fundamentally rotted interior.

Convergent people like finishing things.

For divergent people, novelty is more important than completion.

Divergent people are more likely to be Myer's Brigg N vs S.

It sets expectations that the product can't meet.

That is, not what you do in an instant, but what you would do after reflection.

This frame comes from my friend Kanjun.

Every other game was procedurally rendered, but this game had full motion video with high-quality hand-drawn animation.

The way it worked was it was a LaserDisc where it could seek around and play different prebaked videoclips based on the (limited) decisions the player made.

The high-fidelity graphics obscured that the underlying state space was tiny; more like a low-fi labyrinth with pretty pixels on top.

Still, it was a classic Pizza Fax kind of demo; it previewed the future inevitable state of hifi video game graphics.

It has its roots in DRM.

Today it's typically used in user-hostile ways: the provider making it so you can't patch anything on the receiving side.

But that same ability to patch anything is what makes it impossible to trust what the party on the other side of the network connection did.

Plus, this tool can be used in user-empowering ways, to force the server to do what the user intends.

Soft yes in that context is a hidden no.

It's all about cohering something out of nothing together.

Fighting entropy requires being in total alignment.

A pre-PMF startup is a little bit like a cult.

The group believes something that the rest of the world does not believe–and that also might be directly contradicted by evidence.

That means if there's anyone on the bus who doesn't believe the group's mythos, it dilutes the whole enterprise or threatens to explode it.

So cults evolve behavior to demand purity tests and ruthlessly cast out anyone who fails them.

The confidence to have a perspective comes from your situated taking of risks.

You own the decision, which means you have absorbed it, and aren't just going through the motions.

Not just executing on an idea from a human, but coming up with the idea in the first place.

Any time there's a time delay over uncertainty, there will be oscillations, and repeated under- and-over-shooting.

If you're participating you're only obliged to do work if there's an action assigned to you.

If you're driving you're the owner.

There's always an action assigned to you implicitly which is "move the ball down the field by making sure an effective execution task is owned by someone (or you) and if there isn't one, come up with one."

How much energy being an owner takes scales with how far from default diverging to default converging the workstream is.

Fundamental project management is ensuring that the team is converging on the right thing in a fast enough time frame.

The former is about heroically turning the crank.

The latter is about clarity and cohesion of the whole.

The former is a dime a dozen, and often ineffective.

The latter is rare and extremely valuable.

"Well I'm great at receiving feedback and they haven't told me it's not working so everything is good."

You think you're receiving feedback when you're explicitly receiving it, but actually you're receiving indirect feedback constantly.

People are calibrating what you're insecure about, what you lash out about, etc.

They'll walk on eggshells on things they can tell you care a lot about or are core to your identity.

So you're worse than you think you are, and people know it more than you think they do.

If it makes them more money, an enterprise is more willing to investigate it, by tasking one of their swarm of employees on it.

For consumers, it takes time to consider something, and time is single-plexed and extremely precious, so only if it comes with an enthusiastic recommendation from someone they trust will they bother.

It's magic.

A ton of value created, with no single concrete thing unlocking that much.

When the software now does what it wants to do, but more robustly, that's tons of prediction errors avoided for users and also other engineers.

Cleaning up is part of the Second 90%.

It doesn't require grand engineering plans.

It just is a matter of people pitching in, good tooling, good judgment, etc.

Transcendence vs immanence.

Then systematically work through the lower layers to clean them up, now that they've been indirected away from the load above.

The Strangler Pattern is a related concept.

The proper amount of individualism leads to innovation for the overall system.

But too much tears it apart.

The advice of "do what the hero of your movie would do right now" is hyper-individualist.

They terraform their surrounding world (to the extent they're able) to have the shape that they thrive in.

They feel proud about this because forging a positive-sum dynamic is not just good for them, but good for everyone.

This is the power and imperative of the Radagast.

"Illuminators" here is David Brook's frame for people whose attention makes others feel known.

Within a few minutes 1:1, they can form a resonant connection between them and anyone who is even the slightest bit open.

Because it is enjoyable and meaningful, that connection, even though it's only a few minutes of wall-clock time, is something that matters.

That means that person is more likely to seek out the illuminator with an interesting opportunity.

For the relationship to be truly resonant, the illuminator also has to respond authentically when the person comes back looking for help or advice, too.

The quicker you can form a resonant connection 1:1, the higher your leverage that comes from increasing your network and farming for serendipity.

Now LLMs are significantly better than the average engineer.

Unfortunately, nearly everyone starts off below average and gets better with more experience.

The lowest rung of the ladder is now deep underwater.

You have the competitors internally, keeping the overall system strong, at the cost of efficiency.

Resilience and efficiency are in tension.

But it's not good enough in a ground truthed environment.

If you aren't being continuously ground-truthed (for example by a competitor) then everything appears to work, until one day, catastrophically, the barbarians storm the gates.

For example, a flight of mine recently was switched to a different aircraft and delayed by two hours due to a short thunderstorm and its downstream effects.

Once we were ready to board, they informed us that the plane had originally been planning to go to South Africa, and needed significant amounts of fuel removed for safety, which would take two hours… and had not started yet.

This is presumably something that would have been blindingly obvious to the gate agent responsible for that flight, who was sitting there thinking about this flight and this flight only.

It used to be that sending information was hard, so the system had to have good-enough rules of thumb for local empowered decisions.

Another possibility: airlines are trying to optimize their on-time performance, so they'd rather further delay the already-delayed single flight more rather than spread out a bit of new delay across other currently-on-time flights.

Either way, hyper optimization is to blame for lack of resilience for unexpected scheduling changes.

"Sure, why not take the flight that leaves during rush hour so we get to our destination a little earlier?"

This is even for people who know us quite well!

Embarrassingly, I do the same thing to a number of same-sex couples I know.

Why does this happen somewhat consistently?

First, let's lay out a few observations.

I think what is happening is that how you store an individual is related to how you first got to know them.

If you got to know them as an individual, it's stored in your brain as the individual, then later with a tag for being in a couple.

If you got to know them as a couple, then it's stored in your brain as a couple, with each individual having a little discriminator tag.

For opposite-sex couples, there's an obvious and unambiguous discriminator tag: gender.

For same-sex couples though that discriminator tag doesn't work, and other discriminator tags (e.g. "the tall one", "the one with curly dark hair") are much less salient and require more work.

Once you get to know the individual as an individual (e.g. you spend a ton of time with them), that becomes the primary way you encode them.

It's a playground where kids of all ages can get hammers and nails and wood and build whatever they want (with adult supervision).

It's a wondrously emergent place, totally resonant.

It also has an internal balancing set of incentives.

To get a tool, you have to find 5 loose nails or "splinters" of wood and turn them in.

If you find a "Dr. Dangerous" (a piece of wood with a nail sticking out the other side), you get 5x the reward.

They have magnets on a string that you can pull through the sand, finding loose nails to turn in, that even the youngest kids can do.

This creates a clever and stable equilibrium where the kids themselves are keeping the playground in a stable, mostly-safe state.

The common critique is that "everyone tips the same anyway" so the tip doesn't drive differential service.

But the real benefit is that tipping (of any amount) now gives a structural incentive to the waiter to provide speedy service.

The more checks they get, the more bonus they get.

But the tip itself makes sure they don't rush the customer to turn a table; if the table feels rushed, they're much more likely to leave a smaller tip.

As a result, the service is incentivized to be more attentive than it would otherwise be.

This is one of my college friend's ironclad laws of humor.

Why does it work?

Humor requires a prediction error: for something unexpected.

When someone is within earshot but not visible, we forget they're there… but later they can pop back up in the conversation.

We forgot they'd have thoughts on the proceedings because we weren't thinking about them, but they still cared.

The reflection you did is real, even if the reason you did it wasn't.

"The moment you think you know it is the moment you stop growing."

Another bit of Peloton wisdom.

If you record your hike on Strava, then it could be performative.

If you do the hike without recording it, then it's definitely not performant.