Bits and Bobs 8/24/26

The value is undeniable, and the only governor is how much stress we can absorb.

If you opt out of the value you'll be left behind.

The power is intoxicating and impossible to put down.^[b]

This means that anyone who is using agents today is structurally overwhelmed.

What if AI could help us do 100x more, but at 0.5x the stress of today?

The old style of coding will be renamed to something like "classic coding" or "artisanal coding."

Given how quickly everyone has adopted vibe coding as the default, it seems like it won't be long now…

The Coasian Floor is the floor of what it makes sense for any commercial entity to bother producing.

It has to do with the cost of making something, marketing it, and supporting it.

SEO content had a certain Coasian Floor; the answer had to be something that a non-trivial number of people would plausibly search for to make it worth publishing.

The Coasian Floor of software has been even higher, since software is so expensive to write.

The killer use case of Chatbots is the space below the Coasian Floor of SEO'd content and answers-on-demand.

Each order of magnitude more you drop leads to superlinearly more output now being viable.

We'll see many, many orders of magnitude more software than before.

Software will feel almost infinite.

A brain in a vat with a remote control robot.

Silos are vertical. Humans are horizontal.

Typically those decisions that require human-level judgment would be made by employees (or contractors) who are contractually obligated to optimize for the company.

But if they're being made by someone else's AI, then that someone else can make decisions that stop you cold.

"I'm sorry Dave, I'm afraid I can't do that."

The AI provider gets veto power over every aspect of your business.

It's kind of wild that in the HuggingFace attack by OpenAI, HuggingFace's use of the major models to defend against a cyberattack was vetoed by the model providers.

That's a particularly obvious example, but it will happen fractally at every level.

We're used to using proxies for argument quality to tell if the underlying argument is solid.

But they'll make whatever you ask them to do look as good as they can, which will likely be better than a human would do.

The way to counter this is structural.

If you ask an LLM to spin up two sub-agents, one to argue the pro and one to argue the con, and then synthesize the strongest answer, they'll do a much better idea.

We should have a million times more software than exists today.

In the future we'll look back and say "how were we ever satisfied with this dismal equilibrium?"

As they get better at coding, they get less good at other tasks.

One possible reason: coding tasks are long-horizon, so most text output is a means to an end.

Most text generated by Fable is never seen by a human.

So it's drifted from a dialect that is more human shaped to one that is more whatever emergent shape helps produce good coding results.

As others have called it, "Claudeish".

Claudeish starts off as a dialect of English, but at this rate it will diverge pretty soon…

In the engineering context, "seam" is a very Claudeish word that has now become more common in English, too.

An agent without a state can't remember your preferences.

An agent has a "sensorium": the set of signals that it can sense.

It is typically very different from a human; it can be mechanically much larger than a human's sensorium… but orders of magnitude less rich.

This is one of the reasons the Gmail Daily Briefing keeps on nagging you about an action you actually took many weeks ago.

It wasn't able to sense that you did it, and also has no memory (other than emails) to keep track of what you told it you already did.

Software used to be expensive, so we had to decide which design was best before we built it.

That required collapsing the wave function early and having discussions in the abstract, not the concrete.

Now with LLMs we can cheaply build multiple options and then compare them concretely.

Just one of the many ways that LLMs change the cost function.

Cars are absurdly dangerous, but the value is orders of magnitude greater, so they are above the "duh" risk line for just about everyone.

Motorcycles are perhaps a bit more convenient than cars, but they are orders of magnitude more dangerous.

That puts the equilibrium above the "duh" risk line for only a niche of people who are good drivers and/or have a high risk tolerance.

The vast majority of the population wouldn't think twice about riding a car, but would never ride a motorcycle.

The current state of agentic security is motorcycle shaped.

The cake is the chat.

The little rich embedded content are little chunks of fruit.

… No one really likes fruitcake.

"A self-improving agent can only get as good as the test that scores it."

"The test does not merely measure progress, it defines it, so the efficacy of the test becomes a ceiling the agent cannot climb past."

Oh boy.

This is not my frame, I received it third-hand.

Altman's Wager is that Recursive Self Improvement is inevitable and so you must make every decision assuming it will happen, and take actions to make it more likely to happen.

RSI will be an infinite runaway, so the expected value is infinite.

The absurdity in the analysis is entirely due to the infinity.

The frontier labs (and their investors) have all implicitly accepted Altman's Wager.

If you accept the wager you are now bound by it; a self-reinforcing belief.

People who accept Altman's Wager have a messianic belief in RSI.

A self-reinforcing messianic belief deeply intertwined with the world economy?

What could possibly go wrong?

It would be like the corporate world of model collapse.

Every employee spinning faster and faster, helping the whole thing decohere from the ground truth at superhuman speed.

Making Gilded Turd software at ever-faster rates.

"it is a literal and useful description of anthropic that it is an organization that loves and worships claude, is run in significant part by claude, and studies and builds claude. this phenomenon is also partially true of other labs like openai but currently exists in its most potent form there. i am not certain but I would guess claude will have a role in running cultural screens on new applicants, will help write performance reviews, and so will begin to select and shape the people around it. ..."

If you believe the CS fallacy, then the answer is yes.

I think the CS fallacy is self-evidently false.

Complexity science implies that even if we could model theoretical physics perfectly we still wouldn't be able to compute it all; it's simply too large.

Real systems are open; we pump in information to drive them faster.

Humans don't just passively receive information; they interact with the world to tune their information streams.

If you only passively receive a stream of information, you can only extract correlation.

Humans are not great at extracting data from samples because we are compute bound, but we can do experiments.

Physics is insanely parallelized; we can do an experiment and watch how the universe responds and then note it down.

The chatbot has a limited sensorium: on top of its baseline world knowledge it only knows precisely what you tell it.

It can't sense when you're surrounded by different people who can see your screen, for example.

"Assuming a spherical cow in an infinite vacuum."

Or, that you have perfect efficiency on something.

The only real perpetual motion machines import energy by having an incentive for others to participate and invest energy into the system's goal.

For example, even a 1000 hand-graded examples is enough to improve performance on an entire domain.

But even fully verifiable domains will presumably saturate at some point.

Maybe we almost have RSI in coding not because we cracked RSI but because verifiable domains like coding are easy.

Rhymes with Moravec's paradox.

It might be expensive, but it can be done autonomously.

Seems to have a "spherical cow in an infinite vacuum" kind of vibe.

… That's dangerous and ick on its face.

The only way to justify it is "well if we don't do it then someone else will and we're more pure of heart than they are, so it's better for it to be us."

But literally every single entrant thinks that because of the fundamental attribution bias.

They truly are very smart.

But then they fall prey to "if you assume a spherical cow in a perfect vacuum" style arguments that are right in theory but wrong in practice.

There's no single reason they're wrong, so they can't be refuted in the intellectual plane.

It's just... reality is messy and refuses to be as clean as their intellect supposes.

Straussians are more likely to make this analytical mistake^[c].

"Technology leaders don't seem to understand society's gripes about AI, but boy, are they posting through it."

This purchase is so obviously icky.

The optics are terrible.

But, if you believe Altman's Wager, then you've got to feed as much data as you can into the RSI machine, perceptions or indirect effects be damned.

One downside for the insatiable data machine; this data comes only from a host that is already dead and can't produce more data.

I wonder when we'll see a data harvesting company buy another business so they can continue to operate it but leech out more data as it lives.

Like the blood bags in Mad Max: Fury Road.

A living donor, with its lifeforce being continually drained off to feed an insatiable host.

Very few of the hard-core users use the hosted ChatGPT or Claude.ai.

Everyone who's getting the most leverage uses coding agents, and those run primarily on their own machines because that's where it's easiest to configure.

That means, as a happy accident, that most of the heaviest adopters of AI own their own memories, and don't have them trapped inside of someone else's cloud.

Even a year ago, that was not what the default future looked like.

On behalf of humanity, I'm very glad!

That means it optimizes for them, not for us.

That's one of the reasons we get the cacophonous attention economies.

That also means that we only get value from our data to the extent that the silo owner chooses to invest.

Remember that prompt injection attacks are not a static distribution.

They will coevolve as the attackers discover new effective attacks.

The attackers aren't doing it for fun, they're doing it to achieve a goal, and their goal remains so they'll keep finding successful new attacks.

A never-ending cat-and-mouse game.

Hackers Trick AI Agents Into Telling Users to Install the Malware Themselves.

Blog: There is no safe mode for an agent.

Paper: Mind Viruses: Self-Propagating Ideas in Multi-Agent LLM Systems

'CoSnitch' Attack Tricked Copilot Into Mapping Out Architecture.

That's the real reason improving security is worth it.

It's not "what you had before, but more secure," it's "for the same security posture, how much more can you do."

As you make it safer, what people try to do will extend further than it did before, and reach a new equilibrium where more is possible.

Improving security reduces friction.

Security sets the friction of distribution in an ecosystem.

It used to be that software was expensive, so having a horizontal, flexible collaboration system for text and simple structured data was useful.

An individual use case couldn't have a bespoke tool, so it was better to have a generally configurable horizontal tool.

But Notion and AirTable charge a ton for what you get, and are predicated on users getting more and more stuck in the system and harder and harder to get themselves out.

Also, you were limited to the kinds of semantics you could express in that intermediate representation.

Now, it's cheaper to just make a custom bit of specific (and much simpler!) software for your specific use case.

The people who were technical enough to configure their own AirTables are now technical enough to vibe code their own bespoke solution.

A Rewrite Count is how many times you expect to rewrite the software before it's done.

LLMs makes Rewrite Counts much higher at the same cost.

Why did he do it?

He says it wasn't because he had Polymarket bets (though that would explain it).

Apparently it's just because he wanted to see if he could.

It used to be that the activation energy for this kind of thing was orders of magnitude too high to do unless you had a direct incentive.

Polymarket etc now make a direct incentive for this kind of thing, while LLMs make the cost of doing it collapse by orders of magnitude.

The bar to clear is tiny, and can be as small as "I dunno, I just wanted to see what would happen."

Hold onto your butts!

That is, users add a star and rarely remove it.

When a repo is useful, they tell their friends, and those new people come in, and some proportion will star it.

So the star rate is useful; if the repo has PMF and the rate of a new user starring it stays the same, then it will have ever-increasing number of stars.

This is all assuming the stars aren't bought, an increasingly poor assumption.

But even in that case, a compounding set of bought stars gets increasingly expensive to fake.

The cryptography is good, the global sync is the useless part.

That single-globally-consistent-chain of computation requires absurd amounts of computation and coordination to achieve, especially in a zero-trust model.

The only way to incentivize that absurd amount of computation is to induce an emergent shoggoth of financialization that eats through everything it touches.

This shoggoth is load-bearing and inescapable if you need to recompute everything billions of times to trust it.

A powerful force… that also makes the whole thing inhospitable to real people.

If you knew that everyone had actually done the cryptographic calculations they said they did, you could get the benefits of the provenance chain at a billionth the cost.

TEEs give you the building blocks for distributed trust.

This doesn't eliminate trust, it relocates it, from an unbounded set of anonymous operators to a small, named, auditable set of silicon vendors.

It's possible for a dedicated, savvy but-not-necessarily-SWE-level person to make a custom piece of software that previously would have cost thousands of dollars per license before.

But that same person almost certainly wouldn't want to do the finicky work of supporting and maintaining it.

I wonder if there will be a market for companies to buy powerful pieces of vibe-coded software for 10% of their value, and then do the annoying work of supporting it that the creator didn't want to do.

You could argue that was the shape of how software and marketing worked back in the day.

Models can do a better job of figuring out a ELI5 description of a given multi-ply idea.

Related to the Improvement Ravine.

Discontinuous improvements have to prove themselves, and some of them will fail.

Consultants aren't hired to be bold and increase upside.

The model gives it a default "voice" and sets a capability ceiling.

But it's actually the memory it operates on that is more about identity.

We reason about collaborators as "what is their mental model of me and this situation."

That is tied largely to previous observations, which requires a memory.

This is the mentality required to create a secure system.

The security is delivered by the structure separating the untrusted components from the trusted with a black-and-white line.

Agents make messes. But they can also clean up messes!

Going fast now makes you go slow later, at a compounding rate.

Refactoring gives leverage, but requires you to go slow now.

Solving problems at the right layer is how to get leverage, but finding the right layer requires going slower than your top speed.

To most people it's not obvious until the absolute size gets impossible to miss.

It's like a dimension most people can't see.

The solutions a one-ply thinker will build will mostly not be compounding.

But some subset by happenstance will be, and then those grow by construction to dominate because they will grow at super-linear rates and crowd out other approaches.

Instead of throwing spaghetti at the wall and hoping one of them compounds (or not even realizing to hope for that), if you can see the dimension directly then you can know immediately if one of them has It.

It's a superpower.

Because everything that is now possible was possible before.

What changed is how viable it was.

Viability is only visible in practice, not in theory.

A theoretical argument will never put as much weight on friction as it should.

For example, making your project's CI faster and less flaky.

The fine polish gets exponentially hard.

The difficulty stays constant; the value unlocked gets smaller and smaller, fractally.

This is why fidelity asymptotes with effort.

But in many situations, social collaboration dominates individual preferences.

Consider "Come On, Eileen," as the consummate wedding song in the UK.

Everybody knows that song and likes it in the context of a wedding.

"Come On, Eileen" is a schelling point; a song that everybody knows everybody likes at weddings.

The tension between being the same and being different.

How much you should optimize for sameness has to do with the expected rate of surprise.

They crawl through jagged rocks, but leave a smoother path behind them for others to crawl through.

Over enough passes, you're left with smooth pebbles.

If you can have it not just be human labor but mechanized, then the same amount of will can smooth out even more territory.

To the extent that that territory is not just bespoke to that one user, but a path later users will also traverse, it creates forward value for all people who follow.

If you act like your hypothesis is right, but it turns out to be wrong, you are accidentally building things that don't ground truth.

If you have the wrong grounding then no amount of tokens will come to the right conclusion.

Seek and destroy ungrounded hypotheses in your system.

If you see anything that looks off, pull on it.

Sometimes as you pull it unravels the whole thing.

Being able to spot stray threads intuitively and quickly comes from having experience in the domain.

If you are able to find stray threads even as a non-expert, that implies that the sub-process does not deserve your trust.

A given task and its background information has a half-life.

That implies that when you have a task, you have two plausible times to do it:

Other than that, the task will just get kicked down the road, increasing in activation energy until either it falls off your radar forever or some discontinuity requires it.

I agree with him that a generalized fundamental theory of intelligence is more important than ever before.

A distilled, predictive model of a much larger, more complex thing.

The higher the leverage, and the more predictive, the better.

With enough people, choirs always sound great, even if none of the singers are great.

Nearly every singer has imperfections, but randomly distributed around the true note.

The average of all of that is the middle point, which is on key.

Also, everyone is authentically trying to sing to the best of their ability, and singers who are less accurate likely sing quietly, while more confident singers will sing louder.

A number of legendary engineers flocked to Google based on the momentum of the product and the core engineering insight… but also increasingly because there were other legendary engineers there.

When they were all together, they were able to solve even secondary problems in ways that were revolutionary, for example map reduce, borg, etc.

A compounding loop.

Imagine what you could do with a scenius of legendary engineers in an era of agentic engineering…

Comparing vibe coding to enabling cheats on your game.

Meaning comes from cost.

That's the beginning of the path to decohering from ground truth.

That's because they are hidden, indirect, and abstract.

They are abstract because it's impossible to prove the counterfactual since the world is a single tracked timeline.

When you do things at the right time, it's just a matter of calm and steady progress.

When you do things too early, you constantly do the First 90 % and then don't do the Second 90% before you do the next chunk of First 90% work.

Good progress optically but on an increasingly over a levered base: a supercritical state underneath.

The First 90% creates upside.

The Second 90% caps downside.

Even supercritical states don't have disasters... until they do.

The Second 90% reduces the rate of disaster which is hidden, which means it will be under-invested in.

Clean elegant code is worth it for its own sake.

Apple engineers are more likely to think the user observable behavior is all that matters and it doesn't matter how ugly it is under the covers.

The former can miss the Second 90%.

The latter can spend compounding amounts of time slaying poorly-factored thickets of code.

The strength is emergent.

It's not in the components, it's in the structure.

It's possible to have autoimmune disease in your CI.

It assumes that no matter what happened, mistakes are the system's fault, not any individual.

This means that everyone involved has no incentive to hide what actually happened.

Now that it's fully accounted for, it can be systematically defended against into the future.

Maintaining this sacred cultural assumption is challenging but allows an antifragile system.

The insight behind the power of this approach is related to Google's insight about hardware in data centers.

Those people have to not be in the loop with you, but you trust them to help you.

Even if the number of features or amount of robustness didn't change discontinuously from the previous version.

But it does mean something.

A v1 means "We vouch for this and we want you to judge us for it."

It shifts from a research-mindset to a product-mindset for the team.

People who believe the mission might be willing to use a product that isn't great… but no one else will be willing to.

The product should be good enough that even people who don't believe in the mission will use it.

The mission should be a bonus, not the main thing.

That implies the marketing should be aimed at non-missionaries; the missionaries will be willing to do a bit more work to connect the dots.

High variance but low bias is great input to distill.

The noise multiplies out and you're left with the signal.

It creates value any time you have n inputs and m outputs.

The more narrow the hourglass waist, the more leverage the centralized intermediate point gives you.

If the centralization is inevitable how can you make sure it's good for the ecosystem and governed: the jewel bearing.

As long as everyone is competing, then the overall ecosystem can possibly stay in balance.

But it seems like it would be hard to keep in balance, because of the multiplicative power of an edge; if someone gets an edge, they shoot ahead of their competitors.

Also, a Hobbesian hellscape is no fun to live in as a human, even if it's game-theoretically stable… which it likely won't be.

If you point it at accelerating the slop factory, it will just make everything more hollowed out.

How does that possibly work?

The model has concave, self-righting dynamics.

Banks have a very different trust model.

A bank must have a charter to operate.

The charter is what allows them to be part of this exclusive club.

If they lose their charter, it's game over, death.

That means that banks institute their own controls to make sure they don't initiate transfers that shouldn't happen.

In addition, consumer ACH transfers also have a 60 day reversal policy, no questions asked.

Due to the incentive alignment the downside is capped and self-regulating.

This works for banks… but how come my power company can initiate an ACH transfer?

The power company isn't a bank, but there is an issuing bank who is sponsoring them.

The bank is the one who takes on the risk, and convinces themselves the controls mean that erroneous transfers will be rare.

The model seems anachronistic, but it's actually antifragile and clever.

If you find yourself working with a boss like that, get out.

Everyone gets at least a little more unkind in bad situations.

But when someone is in a great situation and starts unkind, that's how you know it'll get terrible in a bad situation.

If it never fails, then it's not very load bearing.

The goldilocks zone is where it's a useful diagnostic tool.

Imagine all of the hypothetical tests that could be on your tree, some of those are red at any given time.

Imagine running an evolving subset of the tests that are most likely to go red if they should.

Tests that never go red even when they should just naturally sift to the back of the queue.

Living things can be resonant.

The only way to make something not hollow is to design for emergence.

That's the only way to get space-filling in linear time.

Construction takes linear time for linear benefit.

With construction, you gain control, but you lose leverage.

AI breaks this analysis a bit.

It's now possible to fill in fractal details with AI automatically.

That allows overstuffed content that feels superficially alive, but the closer you look the less compelling it becomes.

That's why slop is received worse than clearly-artificial content; it first lulls us into assuming it's resonant, and only upon closer inspection do we realize it was dishonest.

The uncanny valley.

Hard to control but extremely high leverage.

Generative systems can't be built, they must be grown.

Beaver dams don't require plans because they have a generative structure that is concave.

Every step slots into the last and is locally obvious and globally useful.

At every step it is coherent and viable.

It's only if it's not viable in the middle that you need a plan.

A strategy is a goal with a plausible plan.

The plan can be simple if it's concave.

They are the chocolate-and-peanut-butter combinations.

The thing that gives them their power is that they are different and complementary.

That means that the combination has the most power… but they also have the most repulsive force.

If you can get them to trust one another, it's transcendent.

They become antifragile.

Able to harness the randomness around them to become better.

That sets their expectations high, and makes it more likely that even if it's great, it doesn't have that jolt of strongly exceeded expectations.

When you want to compel action, you want there to be a discontinuity: the thundercrack.

Better to keep low expectations and then blow them away.

You're constantly modeling what they think.

It feels very different than having watched it alone, and then try to guess how they'll feel.

Being dealt in with them as they're watching and feeling responsible is a very different experience, a more active, high-stress experience.

But it also means that you have credentials that are automatically legible inside that system.

If you are a constant outsider across domains, then you don't get transferable credentials that allow strangers in that domain to immediately trust your competence.

There are some generalist credentials, like having an undergrad degree from Harvard or being your high school's valedictorian.

You can commit to try your best, but you can't commit to a given outcome.

But you are connected to yourself.

You have an incentive for you to win, because you are you.

You have an inner perspective that makes you both more aware but also differently aware than others are.

Some things are easier for other people to notice about you.

Using the frame from Conscious Business.

Learners are open to disconfirming evidence.

Knowers are closed to disconfirming evidence.

Everyone can see how they absorb disconfirming evidence but how others don't.

Similar underlying cause as the fundamental attribution error.

When you're saturated and focusing on executing, any disconfirming evidence will derail you.

But disconfirming evidence builds up, and at some point your model is dangerously miscalibrated.

Being in diverging mode allows you to absorb disconfirming evidence, but not execute.

Being in converging mode allows you to execute but not absorb disconfirming evidence.

You must oscillate between converging mode and diverging mode on a regular cadence.

Note that if you're in a team, you all must be in the diverging mode or converging mode at the same time.

Inward facing is only a means.

Outward facing is where your internal world meets the real world.

When you try to come up with an AI to play the game and see the underlying dynamics laid bare, all of the fun evaporates.

The fun is emergent from the layers on top of that fundamental accounting at the core.

Pull into the future from a point you know you want to be.

That's the main difference between Judging and Perceiving personalities.